[
https://issues.apache.org/jira/browse/FLINK-40120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095439#comment-18095439
]
Weiqing Yang edited comment on FLINK-40120 at 7/10/26 11:23 PM:
----------------------------------------------------------------
Thanks [~gyfora], that lines up with what I found in the snapshot format, so
I'm aligned on all of it.
Top-level scope is confirmed. RowDataSerializerSnapshot Java-serializes each
field's whole LogicalType, so any nested ROW field already carries its names
through restore. Only the outermost row's names are dropped, where
RowDataSerializer(RowType) keeps getChildren() and discards the wrapper. So a
flat top-level String[] written alongside the existing content covers the gap
with no recursion needed. I'll narrow the ticket description accordingly. Names
only (the types are already persisted), and RowData only since Pojo/Avro carry
names already.
Agreed the serializer-side field is an implementation detail, and on the public
getter. It makes sense to add it with the first feature that needs it (FLIP-527
/ FLIP-599) rather than ship an unused accessor now. This first cut just
persists the names (V3→V4, written unconditionally, V4 still reads V3) and
stays evolution-free with no resolveSchemaCompatibility change, making it a
clean shared building block for both use cases.
I'll take the first cut and put up a draft PR shortly, and ask for your review.
Thanks,
Weiqing
was (Author: weiqingyang):
Thanks [~gyfora], that lines up with what I found in the snapshot format, so
I'm aligned on all of it.
Top-level scope is confirmed. RowDataSerializerSnapshot Java-serializes each
field's whole LogicalType, so any nested ROW field already carries its names
through restore. Only the outermost row's names are dropped, where
RowDataSerializer(RowType) keeps getChildren() and discards the wrapper. So a
flat top-level String[] written alongside the existing content covers the gap
with no recursion needed. I'll narrow the ticket description accordingly. Names
only (the types are already persisted), and RowData only since Pojo/Avro carry
names already.
Agreed the serializer-side field is an implementation detail, and on the public
getter. It makes sense to add it with the first feature that needs it (FLIP-527
/ the catalog) rather than ship an unused accessor now. This first cut just
persists the names (V3→V4, written unconditionally, V4 still reads V3) and
stays evolution-free with no resolveSchemaCompatibility change, making it a
clean shared building block for both use cases.
I'll take the first cut and put up a draft PR shortly, and ask for your review.
Thanks,
Weiqing
> Persist RowData field names in RowDataSerializerSnapshot
> --------------------------------------------------------
>
> Key: FLINK-40120
> URL: https://issues.apache.org/jira/browse/FLINK-40120
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
> Reporter: Weiqing Yang
> Priority: Major
>
> `RowDataSerializerSnapshot` currently persists only the field *types* of a
> `RowData` state serializer, not the field *names*. Multiple efforts need the
> names persisted:
> - FLIP-599 (State Catalog) needs the field names to infer and expose a
> readable schema for keyed state.
> - FLIP-527 (State Schema Evolution for RowData) needs them for name-based
> compatibility and migration.
> This issue covers the small, self-contained, shared building block: **persist
> the RowData field names in the serializer snapshot, written unconditionally
> and decoupled from any schema-evolution enable/disable logic.**
> Scope (in):
> - Add `String[] fieldNames` to `RowDataSerializer` (name-based metadata; a
> lightweight alternative to carrying the full `RowType`).
> - Bump `RowDataSerializerSnapshot` from V3 to V4, appending the field names
> after the existing field-types section. Names are written **unconditionally**
> (not gated on any feature flag).
> - V4 readers remain able to read V3 snapshots (field names absent -> `null`),
> so existing savepoints continue to restore.
> - Only the top-level `RowType` field names are persisted (a flat `String[]`).
> Nested `ROW` field names already survive today via the persisted field
> logical types, so no recursion is needed.
> Scope (out — intentionally not here):
> - No change to `resolveSchemaCompatibility` behavior: with only this change,
> a field-type-array difference stays `incompatible()` exactly as today. This
> issue adds no migration/evolution behavior; it only makes the names available.
> - The schema-evolution opt-in and the compatibility/migration logic that
> consumes the names are owned by FLIP-527 and are out of scope here.
> Decoupling the "names persisted" metadata from the "evolution permitted"
> signal is precisely why this is a standalone change.
> - Scoped to `RowDataSerializer`, since its snapshot is the one that persists
> field types without names. Field names for other serializer types, if the
> State Catalog needs them, are separate follow-ups (each serializer owns its
> own snapshot format).
> Compatibility:
> - Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4
> reads V3 with names absent. Runtime behavior is unchanged for all existing
> jobs (names are persisted but not yet consumed by any decision).
> Motivation for splitting this out: both FLIP-527 and FLIP-599 independently
> proposed the same V3 -> V4 name-storage bump; landing it once as shared
> infrastructure lets whichever FLIP lands first establish a single format that
> serves the catalog's schema-inference needs and schema evolution's needs, and
> avoids two conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS]
> thread.
> Links:
> - FLIP-599: State Catalog —
> https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
> - FLIP-527: State Schema Evolution for RowData —
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
