Weiqing Yang created FLINK-40120:
------------------------------------
Summary: Persist RowData field names in RowDataSerializerSnapshot
Key: FLINK-40120
URL: https://issues.apache.org/jira/browse/FLINK-40120
Project: Flink
Issue Type: Improvement
Components: Table SQL / Runtime
Reporter: Weiqing Yang
{\{RowDataSerializerSnapshot}} currently persists only the field *types* of a
\{{RowData}} state serializer, not the field *names*. Multiple efforts need the
names persisted:
- FLIP-599 (State Catalog) needs the field names to infer and expose a readable
schema for keyed state.
- FLIP-527 (State Schema Evolution for RowData) needs them for name-based
compatibility and migration.
This issue covers the small, self-contained, shared building block: *persist
the RowData field names in the serializer snapshot, written unconditionally and
decoupled from any schema-evolution enable/disable logic.*
*Scope (in):*
- Add \{{String[] fieldNames}} to \{{RowDataSerializer}} (name-based metadata;
a lightweight alternative to carrying the full \{{RowType}}).
- Bump \{{RowDataSerializerSnapshot}} from V3 to V4, appending the field names
after the existing field-types section. Names are written *unconditionally*
(not gated on any feature flag).
- V4 readers remain able to read V3 snapshots (field names absent -> null), so
existing savepoints continue to restore.
- Field names cover nested ROW types recursively.
*Scope (out - intentionally not here):*
- No change to \{{resolveSchemaCompatibility}} behavior: with only this change,
a field-type-array difference stays \{{incompatible()}} exactly as today. This
issue adds no migration/evolution behavior; it only makes the names available.
- The schema-evolution opt-in and the compatibility/migration logic that
consumes the names are owned by FLIP-527 and are out of scope here. Decoupling
the "names persisted" metadata from the "evolution permitted" signal is
precisely why this is a standalone change.
- Scoped to \{{RowDataSerializer}}, since its snapshot is the one that persists
field types without names. Field names for other serializer types, if the State
Catalog needs them, are separate follow-ups (each serializer owns its own
snapshot format).
*Compatibility:*
- Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4
reads V3 with names absent. Runtime behavior is unchanged for all existing jobs
(names are persisted but not yet consumed by any decision).
Both FLIP-527 and FLIP-599 independently proposed the same V3 -> V4
name-storage bump; landing it once as shared infrastructure lets whichever FLIP
lands first establish a single format that serves both, and avoids two
conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS] thread.
Links:
- FLIP-599: State Catalog -
https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
- FLIP-527: State Schema Evolution for RowData -
https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
