[
https://issues.apache.org/jira/browse/FLINK-40120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095495#comment-18095495
]
Weiqing Yang commented on FLINK-40120:
--------------------------------------
Hi [~gyfora], I opened a PR for this:
https://github.com/apache/flink/pull/28717. It persists the flat top-level
field names in RowDataSerializerSnapshot (V3→V4, V4 still reads V3 → null),
written unconditionally and evolution-free.
I deliberately kept the public getter out of this PR so this change stays
names-only. The getter can be added later once a feature actually needs it. I
filed sub-task FLINK-40122 for that; I can put up the getter PR once this one
lands, defer it further, or add it to this PR if you’d prefer to keep them
together.
Thanks!
> Persist RowData field names in RowDataSerializerSnapshot
> --------------------------------------------------------
>
> Key: FLINK-40120
> URL: https://issues.apache.org/jira/browse/FLINK-40120
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
> Reporter: Weiqing Yang
> Priority: Major
> Labels: pull-request-available
>
> `RowDataSerializerSnapshot` currently persists only the field *types* of a
> `RowData` state serializer, not the field *names*. Multiple efforts need the
> names persisted:
> - FLIP-599 (State Catalog) needs the field names to infer and expose a
> readable schema for keyed state.
> - FLIP-527 (State Schema Evolution for RowData) needs them for name-based
> compatibility and migration.
> This issue covers the small, self-contained, shared building block: **persist
> the RowData field names in the serializer snapshot, written unconditionally
> and decoupled from any schema-evolution enable/disable logic.**
> Scope (in):
> - Add `String[] fieldNames` to `RowDataSerializer` (name-based metadata; a
> lightweight alternative to carrying the full `RowType`).
> - Bump `RowDataSerializerSnapshot` from V3 to V4, appending the field names
> after the existing field-types section. Names are written **unconditionally**
> (not gated on any feature flag).
> - V4 readers remain able to read V3 snapshots (field names absent -> `null`),
> so existing savepoints continue to restore.
> - Only the top-level `RowType` field names are persisted (a flat `String[]`).
> Nested `ROW` field names already survive today via the persisted field
> logical types, so no recursion is needed.
> Scope (out — intentionally not here):
> - No change to `resolveSchemaCompatibility` behavior: with only this change,
> a field-type-array difference stays `incompatible()` exactly as today. This
> issue adds no migration/evolution behavior; it only makes the names available.
> - The schema-evolution opt-in and the compatibility/migration logic that
> consumes the names are owned by FLIP-527 and are out of scope here.
> Decoupling the "names persisted" metadata from the "evolution permitted"
> signal is precisely why this is a standalone change.
> - Scoped to `RowDataSerializer`, since its snapshot is the one that persists
> field types without names. Field names for other serializer types, if the
> State Catalog needs them, are separate follow-ups (each serializer owns its
> own snapshot format).
> Compatibility:
> - Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4
> reads V3 with names absent. Runtime behavior is unchanged for all existing
> jobs (names are persisted but not yet consumed by any decision).
> Motivation for splitting this out: both FLIP-527 and FLIP-599 independently
> proposed the same V3 -> V4 name-storage bump; landing it once as shared
> infrastructure lets whichever FLIP lands first establish a single format that
> serves the catalog's schema-inference needs and schema evolution's needs, and
> avoids two conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS]
> thread.
> Links:
> - FLIP-599: State Catalog —
> https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
> - FLIP-527: State Schema Evolution for RowData —
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
