[
https://issues.apache.org/jira/browse/FLINK-40120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095409#comment-18095409
]
Gyula Fora commented on FLINK-40120:
------------------------------------
Thanks [~WeiqingYang] . To me it seems like the concrete gap exists for the
RowData serializer / snapshot top level. We simply need the names here as an
array in addition to the existing content. For nested fields we already have
the names available through the logical types I believe, and other major types
like Pojo / Avro are well covered.
In addition to adding this info to the SerializerSnapshot, the Serializer
itself may also need it but that's somewhat of an implementation detail. Also
for the serializer/snapshot we should add a public getter to make it accessible
but that we can also add when the first time we implement a feature needing it.
> Persist RowData field names in RowDataSerializerSnapshot
> --------------------------------------------------------
>
> Key: FLINK-40120
> URL: https://issues.apache.org/jira/browse/FLINK-40120
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
> Reporter: Weiqing Yang
> Priority: Major
>
> {\{RowDataSerializerSnapshot}} currently persists only the field *types* of a
> \{{RowData}} state serializer, not the field *names*. Multiple efforts need
> the names persisted:
> - FLIP-599 (State Catalog) needs the field names to infer and expose a
> readable schema for keyed state.
> - FLIP-527 (State Schema Evolution for RowData) needs them for name-based
> compatibility and migration.
> This issue covers the small, self-contained, shared building block: *persist
> the RowData field names in the serializer snapshot, written unconditionally
> and decoupled from any schema-evolution enable/disable logic.*
> *Scope (in):*
> - Add \{{String[] fieldNames}} to \{{RowDataSerializer}} (name-based
> metadata; a lightweight alternative to carrying the full \{{RowType}}).
> - Bump \{{RowDataSerializerSnapshot}} from V3 to V4, appending the field
> names after the existing field-types section. Names are written
> *unconditionally* (not gated on any feature flag).
> - V4 readers remain able to read V3 snapshots (field names absent -> null),
> so existing savepoints continue to restore.
> - Field names cover nested ROW types recursively.
> *Scope (out - intentionally not here):*
> - No change to \{{resolveSchemaCompatibility}} behavior: with only this
> change, a field-type-array difference stays \{{incompatible()}} exactly as
> today. This issue adds no migration/evolution behavior; it only makes the
> names available.
> - The schema-evolution opt-in and the compatibility/migration logic that
> consumes the names are owned by FLIP-527 and are out of scope here.
> Decoupling the "names persisted" metadata from the "evolution permitted"
> signal is precisely why this is a standalone change.
> - Scoped to \{{RowDataSerializer}}, since its snapshot is the one that
> persists field types without names. Field names for other serializer types,
> if the State Catalog needs them, are separate follow-ups (each serializer
> owns its own snapshot format).
> *Compatibility:*
> - Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4
> reads V3 with names absent. Runtime behavior is unchanged for all existing
> jobs (names are persisted but not yet consumed by any decision).
> Both FLIP-527 and FLIP-599 independently proposed the same V3 -> V4
> name-storage bump; landing it once as shared infrastructure lets whichever
> FLIP lands first establish a single format that serves both, and avoids two
> conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS] thread.
> Links:
> - FLIP-599: State Catalog -
> https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
> - FLIP-527: State Schema Evolution for RowData -
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
