[
https://issues.apache.org/jira/browse/FLINK-40120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiqing Yang updated FLINK-40120:
---------------------------------
Description:
`RowDataSerializerSnapshot` currently persists only the field *types* of a
`RowData` state serializer, not the field *names*. Multiple efforts need the
names persisted:
- FLIP-599 (State Catalog) needs the field names to infer and expose a readable
schema for keyed state.
- FLIP-527 (State Schema Evolution for RowData) needs them for name-based
compatibility and migration.
This issue covers the small, self-contained, shared building block: **persist
the RowData field names in the serializer snapshot, written unconditionally and
decoupled from any schema-evolution enable/disable logic.**
Scope (in):
- Add `String[] fieldNames` to `RowDataSerializer` (name-based metadata; a
lightweight alternative to carrying the full `RowType`).
- Bump `RowDataSerializerSnapshot` from V3 to V4, appending the field names
after the existing field-types section. Names are written **unconditionally**
(not gated on any feature flag).
- V4 readers remain able to read V3 snapshots (field names absent -> `null`),
so existing savepoints continue to restore.
- Only the top-level `RowType` field names are persisted (a flat `String[]`).
Nested `ROW` field names already survive today via the persisted field logical
types, so no recursion is needed.
Scope (out — intentionally not here):
- No change to `resolveSchemaCompatibility` behavior: with only this change, a
field-type-array difference stays `incompatible()` exactly as today. This issue
adds no migration/evolution behavior; it only makes the names available.
- The schema-evolution opt-in and the compatibility/migration logic that
consumes the names are owned by FLIP-527 and are out of scope here. Decoupling
the "names persisted" metadata from the "evolution permitted" signal is
precisely why this is a standalone change.
- Scoped to `RowDataSerializer`, since its snapshot is the one that persists
field types without names. Field names for other serializer types, if the State
Catalog needs them, are separate follow-ups (each serializer owns its own
snapshot format).
Compatibility:
- Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4
reads V3 with names absent. Runtime behavior is unchanged for all existing jobs
(names are persisted but not yet consumed by any decision).
Motivation for splitting this out: both FLIP-527 and FLIP-599 independently
proposed the same V3 -> V4 name-storage bump; landing it once as shared
infrastructure lets whichever FLIP lands first establish a single format that
serves the catalog's schema-inference needs and schema evolution's needs, and
avoids two conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS]
thread.
Links:
- FLIP-599: State Catalog —
https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
- FLIP-527: State Schema Evolution for RowData —
https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
was:
{\{RowDataSerializerSnapshot}} currently persists only the field *types* of a
\{{RowData}} state serializer, not the field *names*. Multiple efforts need the
names persisted:
- FLIP-599 (State Catalog) needs the field names to infer and expose a readable
schema for keyed state.
- FLIP-527 (State Schema Evolution for RowData) needs them for name-based
compatibility and migration.
This issue covers the small, self-contained, shared building block: *persist
the RowData field names in the serializer snapshot, written unconditionally and
decoupled from any schema-evolution enable/disable logic.*
*Scope (in):*
- Add \{{String[] fieldNames}} to \{{RowDataSerializer}} (name-based metadata;
a lightweight alternative to carrying the full \{{RowType}}).
- Bump \{{RowDataSerializerSnapshot}} from V3 to V4, appending the field names
after the existing field-types section. Names are written *unconditionally*
(not gated on any feature flag).
- V4 readers remain able to read V3 snapshots (field names absent -> null), so
existing savepoints continue to restore.
- Field names cover nested ROW types recursively.
*Scope (out - intentionally not here):*
- No change to \{{resolveSchemaCompatibility}} behavior: with only this change,
a field-type-array difference stays \{{incompatible()}} exactly as today. This
issue adds no migration/evolution behavior; it only makes the names available.
- The schema-evolution opt-in and the compatibility/migration logic that
consumes the names are owned by FLIP-527 and are out of scope here. Decoupling
the "names persisted" metadata from the "evolution permitted" signal is
precisely why this is a standalone change.
- Scoped to \{{RowDataSerializer}}, since its snapshot is the one that persists
field types without names. Field names for other serializer types, if the State
Catalog needs them, are separate follow-ups (each serializer owns its own
snapshot format).
*Compatibility:*
- Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4
reads V3 with names absent. Runtime behavior is unchanged for all existing jobs
(names are persisted but not yet consumed by any decision).
Both FLIP-527 and FLIP-599 independently proposed the same V3 -> V4
name-storage bump; landing it once as shared infrastructure lets whichever FLIP
lands first establish a single format that serves both, and avoids two
conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS] thread.
Links:
- FLIP-599: State Catalog -
https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
- FLIP-527: State Schema Evolution for RowData -
https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
> Persist RowData field names in RowDataSerializerSnapshot
> --------------------------------------------------------
>
> Key: FLINK-40120
> URL: https://issues.apache.org/jira/browse/FLINK-40120
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
> Reporter: Weiqing Yang
> Priority: Major
>
> `RowDataSerializerSnapshot` currently persists only the field *types* of a
> `RowData` state serializer, not the field *names*. Multiple efforts need the
> names persisted:
> - FLIP-599 (State Catalog) needs the field names to infer and expose a
> readable schema for keyed state.
> - FLIP-527 (State Schema Evolution for RowData) needs them for name-based
> compatibility and migration.
> This issue covers the small, self-contained, shared building block: **persist
> the RowData field names in the serializer snapshot, written unconditionally
> and decoupled from any schema-evolution enable/disable logic.**
> Scope (in):
> - Add `String[] fieldNames` to `RowDataSerializer` (name-based metadata; a
> lightweight alternative to carrying the full `RowType`).
> - Bump `RowDataSerializerSnapshot` from V3 to V4, appending the field names
> after the existing field-types section. Names are written **unconditionally**
> (not gated on any feature flag).
> - V4 readers remain able to read V3 snapshots (field names absent -> `null`),
> so existing savepoints continue to restore.
> - Only the top-level `RowType` field names are persisted (a flat `String[]`).
> Nested `ROW` field names already survive today via the persisted field
> logical types, so no recursion is needed.
> Scope (out — intentionally not here):
> - No change to `resolveSchemaCompatibility` behavior: with only this change,
> a field-type-array difference stays `incompatible()` exactly as today. This
> issue adds no migration/evolution behavior; it only makes the names available.
> - The schema-evolution opt-in and the compatibility/migration logic that
> consumes the names are owned by FLIP-527 and are out of scope here.
> Decoupling the "names persisted" metadata from the "evolution permitted"
> signal is precisely why this is a standalone change.
> - Scoped to `RowDataSerializer`, since its snapshot is the one that persists
> field types without names. Field names for other serializer types, if the
> State Catalog needs them, are separate follow-ups (each serializer owns its
> own snapshot format).
> Compatibility:
> - Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4
> reads V3 with names absent. Runtime behavior is unchanged for all existing
> jobs (names are persisted but not yet consumed by any decision).
> Motivation for splitting this out: both FLIP-527 and FLIP-599 independently
> proposed the same V3 -> V4 name-storage bump; landing it once as shared
> infrastructure lets whichever FLIP lands first establish a single format that
> serves the catalog's schema-inference needs and schema evolution's needs, and
> avoids two conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS]
> thread.
> Links:
> - FLIP-599: State Catalog —
> https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
> - FLIP-527: State Schema Evolution for RowData —
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
