[
https://issues.apache.org/jira/browse/FLINK-40120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095389#comment-18095389
]
Weiqing Yang edited comment on FLINK-40120 at 7/10/26 7:03 PM:
---------------------------------------------------------------
Filed to decouple persisting the RowData field names from FLIP-527's
schema-evolution opt-in, per the FLIP-599 [DISCUSS] thread. The intent is a
small, standalone change that both the State Catalog and schema evolution can
build on: persist the field names in {{RowDataSerializerSnapshot}} (V3 -> V4),
written unconditionally, with no compatibility/migration behavior change.
Hi [~gyfora], before I put up a first cut, a couple of scope questions for the
State Catalog side:
- *Serializer scope:* does the catalog need field names only from {{RowData}}
keyed state, or a more general mechanism across other structured serializers
too? Names for other serializer types would each be a separate change (each
serializer owns its own snapshot format), and most non-RowData structured
serializers already carry names (e.g. POJO), so {{RowDataSerializer}} looks
like the concrete gap — but I'd like to confirm that matches the catalog's need.
- *What to persist:* names only, or names plus anything else? Note V4 already
carries the field *types* (from V3) alongside the new names, so the snapshot
would expose both name and type per field, including nested {{ROW}}
recursively. Is that sufficient for the catalog's schema inference?
I can take a first cut on the RowData piece once the above is confirmed, and
open a draft PR linked here. FLIP-527's in-flight PR will then rebase onto this
so there's a single V3 -> V4 format.
was (Author: weiqingyang):
Filed to decouple persisting the RowData field names from FLIP-527's
schema-evolution opt-in, per the FLIP-599 [DISCUSS] thread. The intent is a
small, standalone change that both the State Catalog and schema evolution can
build on: persist the field names in {{RowDataSerializerSnapshot}} (V3 -> V4),
written unconditionally, with no compatibility/migration behavior change.
[~gyfora] Before I put up a first cut, a couple of scope questions for the
State Catalog side:
- *Serializer scope:* does the catalog need field names only from {{RowData}}
keyed state, or a more general mechanism across other structured serializers
too? Names for other serializer types would each be a separate change (each
serializer owns its own snapshot format), and most non-RowData structured
serializers already carry names (e.g. POJO), so {{RowDataSerializer}} looks
like the concrete gap — but I'd like to confirm that matches the catalog's need.
- *What to persist:* names only, or names plus anything else? Note V4 already
carries the field *types* (from V3) alongside the new names, so the snapshot
would expose both name and type per field, including nested {{ROW}}
recursively. Is that sufficient for the catalog's schema inference?
I can take a first cut on the RowData piece once the above is confirmed, and
open a draft PR linked here. FLIP-527's in-flight PR will then rebase onto this
so there's a single V3 -> V4 format.
> Persist RowData field names in RowDataSerializerSnapshot
> --------------------------------------------------------
>
> Key: FLINK-40120
> URL: https://issues.apache.org/jira/browse/FLINK-40120
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
> Reporter: Weiqing Yang
> Priority: Major
>
> {\{RowDataSerializerSnapshot}} currently persists only the field *types* of a
> \{{RowData}} state serializer, not the field *names*. Multiple efforts need
> the names persisted:
> - FLIP-599 (State Catalog) needs the field names to infer and expose a
> readable schema for keyed state.
> - FLIP-527 (State Schema Evolution for RowData) needs them for name-based
> compatibility and migration.
> This issue covers the small, self-contained, shared building block: *persist
> the RowData field names in the serializer snapshot, written unconditionally
> and decoupled from any schema-evolution enable/disable logic.*
> *Scope (in):*
> - Add \{{String[] fieldNames}} to \{{RowDataSerializer}} (name-based
> metadata; a lightweight alternative to carrying the full \{{RowType}}).
> - Bump \{{RowDataSerializerSnapshot}} from V3 to V4, appending the field
> names after the existing field-types section. Names are written
> *unconditionally* (not gated on any feature flag).
> - V4 readers remain able to read V3 snapshots (field names absent -> null),
> so existing savepoints continue to restore.
> - Field names cover nested ROW types recursively.
> *Scope (out - intentionally not here):*
> - No change to \{{resolveSchemaCompatibility}} behavior: with only this
> change, a field-type-array difference stays \{{incompatible()}} exactly as
> today. This issue adds no migration/evolution behavior; it only makes the
> names available.
> - The schema-evolution opt-in and the compatibility/migration logic that
> consumes the names are owned by FLIP-527 and are out of scope here.
> Decoupling the "names persisted" metadata from the "evolution permitted"
> signal is precisely why this is a standalone change.
> - Scoped to \{{RowDataSerializer}}, since its snapshot is the one that
> persists field types without names. Field names for other serializer types,
> if the State Catalog needs them, are separate follow-ups (each serializer
> owns its own snapshot format).
> *Compatibility:*
> - Backward compatible. Snapshot format moves V3 -> V4 by appending names; V4
> reads V3 with names absent. Runtime behavior is unchanged for all existing
> jobs (names are persisted but not yet consumed by any decision).
> Both FLIP-527 and FLIP-599 independently proposed the same V3 -> V4
> name-storage bump; landing it once as shared infrastructure lets whichever
> FLIP lands first establish a single format that serves both, and avoids two
> conflicting snapshot bumps. Discussed on the FLIP-599 [DISCUSS] thread.
> Links:
> - FLIP-599: State Catalog -
> https://lists.apache.org/thread/xhojhcr0pkg7j8dblnt2crrdkkjz76vo
> - FLIP-527: State Schema Evolution for RowData -
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
