Hi David, Thanks for the thoughtful edge cases. Quick take on each.
1. Castable / potentially lossless type changes (for example to STRING or VARIANT): the current FLIP deliberately rejects all existing-field type changes and fails closed, to keep this first version conservative. Allowing selected non-narrowing or otherwise safe casts is a reasonable extension, but I'd prefer to keep it out of this scope and treat it as a follow-up. 2. Comments / field descriptions: from the RowData state serializer perspective, these don't affect compatibility. The serializer compatibility is driven by field names and logical types, not table or column comments. So adding, removing, or changing a comment should be compatible and require no state migration. 3. Removing primary-key / unique constraints: this FLIP is scoped to the RowData value serializer's field schema. Key and constraint changes are a separate concern. RowData used as a state key is explicitly out of scope, and key-schema changes remain incompatible and are rejected by the state backend. Also, changing constraints can affect planning and operator state layout, so I would not include that in this FLIP's migration guarantees. 4. Reduction of included metadata columns: if a metadata column is virtual/pruned and does not reach state, there is no state serializer change to migrate. If it is included in the RowData stored in state, then removing it is a field removal, which this FLIP rejects as incompatible and fails closed. Adding a nullable metadata column that reaches state would follow the same rule as adding any nullable field. 5. Shadowing, and whether both table definitions need the flag: the opt-in is a job-level/table config (`table.exec.state.schema-evolution.enabled`), not a per-table property, so it is not tied to individual table definitions. Shadowing is a plan-time concern: a temporary table with the same identifier hides the permanent one, and the query resolves to the temporary table. The operator's state serializer is then built from whichever table the query resolves to. Schema evolution runs at restore time by comparing the savepoint's state serializer snapshot with the current job's serializer. So the two compose cleanly: shadowing picks the current schema, and evolution migrates the savepoint state to it if the resolved schema change is within the supported RowData value-schema changes. These read as refinements and future scope rather than blockers, so I'll keep the vote (link <https://lists.apache.org/thread/5pxzlry1hblddz0qhzbm1qml4yyjy1wn>) open. But please let me know if any of them feel like they should change the current design. Happy to dig further. Best, Weiqing On Tue, Jul 21, 2026 at 1:32 AM David Radley <[email protected]> wrote: > Hi, > I support this proposal. I am thinking of edge cases around this , to run > by you: > > * > something is castable to another type without narrowing - could also be > allowed e.g. casting to a string or variant. > * > How adding removing or changing comments fits in > * > Removing. Primary key / unique constraints. > * > Reduction of included metadata columns > * > It would be worth checking how this effects > https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/table/common/#shadowing > . Do both able definitions need the flag on, what happens if the permanent > table is different from the temporary table. > > > Kind regards , David. > > > From: Weiqing Yang <[email protected]> > Date: Thursday, 16 July 2026 at 06:07 > To: [email protected] <[email protected]> > Subject: [EXTERNAL] Re: [DISCUSS] State Schema Evolution for RowData > > Thanks Shengkai and Leonard for the reviews and the +1s (and everyone for > the earlier discussion)! > > If there are no further concerns, I'll start a [VOTE] thread in a few days. > > Best, > Weiqing > > On Mon, Jul 13, 2026 at 10:39 PM Leonard Xu <[email protected]> wrote: > > > Hi Weiqing, > > > > > > > > Thanks for driving this FLIP, +1 for the proposal. > > > > I think this is a very useful feature for many production SQL jobs. In > > practice, it is common for input schemas to evolve as business > requirements > > change, especially for CDC / dimension-table scenarios where adding > > nullable fields or evolving nested ROW structures should not necessarily > > require users to drop state or reprocess all historical data. > > > > The current scoped and opt-in design looks reasonable to me. It keeps the > > migration behavior safe by default, while still giving users a path to > > handle backward-compatible RowData schema changes when they understand > the > > state mapping of their job. > > > > Overall, I see this as an important step towards better schema evolution > > support for Flink SQL jobs. > > > > > > > > Best, > > > > Leonard > > > > > > > 2026 7月 14 10:42 上午，Shengkai Fang <[email protected]> 写道： > > > > > > +1 for the proposal. > > > > > > Best, > > > Shengkai > > > > > > Weiqing Yang <[email protected]> 于2026年7月9日周四 14:14写道： > > > > > >> Hi all, > > >> > > >> I'd like to revive this discussion. Since the last round I've > re-aligned > > >> the proposal with current master, folded in all of the feedback, > > mirrored > > >> the proposal to a cwiki FLIP page (as Shengkai suggested), and opened > a > > >> draft PR so the design can be reviewed against working code. > > >> > > >> Proposal doc (updated): link > > >> < > > > https://docs.google.com/document/d/1WtAxp-jAVTLMOfWNldLCAoK137P0ZCMxR8hOZGcMxuc/edit > > > > > >> cwiki FLIP-527: link > > >> < > > > https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData > > > > > >> Draft PR (implements the proposal; kept in draft until we converge): > > link > > >> <https://github.com/apache/flink/pull/28678> > > >> > > >> Here's what changed, mapped to the feedback: > > >> > > >> 1. Single object-level migration hook (Zakelly, Hangxiang). The design > > >> converged on one default method on TypeSerializerSnapshot: T > > >> migrate(TypeSerializerSnapshot<T> oldSerializerSnapshot, T value). It > is > > >> invoked on the new snapshot and receives the old one (the > > >> oldSerializerSnapshot naming Hangxiang suggested), with an identity > > >> default. Because it works on the already-deserialized object rather > > than on > > >> raw bytes, the same method covers value, list-element, and map-value > > >> migration uniformly. This supersedes the earlier byte-level > > >> migrateState(in, out) form and removes the need for a separate > > >> migrateElement — which answers the migrateElement question you both > > raised > > >> (Zakelly, Hangxiang): the object hook handles list elements and map > > values > > >> with the same method, so no dedicated element hook is needed. It also > > >> addresses Hangxiang's point that a common interface shouldn't grow > > methods > > >> most implementations won't use, as they inherit the identity default > and > > >> are entirely unaffected. > > >> > > >> 2. The SchemaEvolutionSerializer-on-the-compatibility-result > alternative > > >> (Hangxiang). This is written up under Rejected Alternatives: > > >> TypeSerializerSchemaCompatibility is a result holder rather than an > > >> executor, and a migration is a distinct transform role (old serialized > > >> bytes -> new layout), so a default method on the snapshot is the > > smaller, > > >> more targeted change. > > >> > > >> 3. Field metadata (Shengkai). Adopted String[] fieldNames on > > >> RowDataSerializer instead of the full RowType. This approach is > > lightweight > > >> and exactly enough for name-based mapping. > > >> > > >> 4. The opt-in, and whether it is necessary (Shengkai, Gabor; and the > > >> question I left open last July). The feature is gated by > > >> table.exec.state.schema-evolution.enabled (default false), scoped > under > > >> table.exec.state.* to signal it is Table/RowData-specific for now — > > which > > >> directly addresses Gabor's point that the config name should make > clear > > it > > >> is RowData-specific rather than a generic state-evolution switch. On > > >> whether the opt-in is necessary: it is the fail-closed safety switch > for > > >> exactly the case Shengkai raised, where a SQL change can silently > shift > > >> operator-internal buffers (e.g. inserting SUM(d) before SUM(c)) even > > when > > >> field names appear to match. With the option off, serializers are > built > > >> name-less and behavior is byte-for-byte as today; enabling it is a > > >> deliberate per-job confirmation that the change preserves state > mapping. > > >> > > >> 5. Scope and a concrete example (Shengkai, Hongshun). The FLIP now > leads > > >> with the primary case Shengkai steered toward — the SQL is unchanged > and > > >> the input schema evolves backward-compatibly — with a worked > end-to-end > > >> example: a Kafka fact stream joined with a CDC dimension whose nested > > >> profile ROW gains a nullable field. Because the join buffers the whole > > >> dimension row in keyed state, the evolved ROW actually reaches state > (a > > >> leaf projection like metadata.userId would be column-pruned and would > > not), > > >> so the savepoint restore exercises exactly this feature. That also > > serves > > >> as the connector example Hongshun asked for. > > >> > > >> On Shengkai's broader concern (July 23) that the feature may not be > > >> accessible to users because few understand the SQL operator state > > >> structure: the reframing above is my attempt to address it — by > leading > > >> with the SQL-unchanged case and a concrete join example rather than > the > > >> operator-internal view. Shengkai, I'd especially welcome your read on > > >> whether this framing now makes the feature's applicability clear. > > >> > > >> Two boundaries I've made explicit this round: > > >> - RocksDB is the initial target backend; the ForSt sync backend can > > >> follow, and ForSt async is out of scope. > > >> - RowData nested below a composite serializer (List or Tuple, e.g. > > >> interval- and outer-join buffers) is out of scope for now and fails > > closed > > >> (rejected on restore, never mis-migrated); propagating migration > through > > >> composite serializers is a planned follow-up. > > >> > > >> The proposal doc and cwiki have the full details, worked examples, the > > >> V3->V4 snapshot compatibility story, and rejected alternatives. > > Feedback is > > >> very welcome. If the direction looks good after this round, I'll > start a > > >> VOTE. > > >> > > >> Thanks again for all the input, > > >> Weiqing > > >> > > >> > > >> On Tue, Aug 19, 2025 at 7:34 AM Gabor Somogyi < > [email protected] > > > > > >> wrote: > > >> > > >>> Hi Weiqing, > > >>> > > >>> I've just read through the whole FLIP and +1 on the direction. > > >>> > > >>> I've a comment apart from the other pending items. Namely the > > >>> configuration is > > >>> `state.schema-evolution.enable` which implied to me that it's a > generic > > >>> state evolution > > >>> feature but it's limited to Row data. Maybe we can mark that it's Row > > >>> data specific. > > >>> I'm pretty sure that we're going to add further types but not all. > > >>> > > >>> BR, > > >>> G > > >>> > > >>> On 2025/04/26 05:45:32 Weiqing Yang wrote: > > >>>> Hi all, > > >>>> > > >>>> I’d like to initiate a discussion about enhancing state schema > > evolution > > >>>> support for RowData in Flink. > > >>>> > > >>>> *Motivation* > > >>>> > > >>>> Flink applications frequently need to evolve their state schema as > > >>> business > > >>>> requirements change. Currently, when users update a Table API or SQL > > job > > >>>> with schema changes involving RowData types (particularly nested > > >>>> structures), they encounter serialization compatibility errors > during > > >>> state > > >>>> restoration, causing job failures.The issue occurs because existing > > >>> state > > >>>> migration mechanisms don't properly handle RowData types during > schema > > >>>> evolution, preventing users from making backward-compatible changes > > >>> like: > > >>>> > > >>>> - > > >>>> > > >>>> Adding nullable fields to existing structures > > >>>> - > > >>>> > > >>>> Reordering fields within a row while preserving field names > > >>>> - > > >>>> > > >>>> Evolving nested row structures > > >>>> > > >>>> This limitation impacts production applications using Flink's Table > > >>> API, as > > >>>> the RowData type is central to this interface. Users are forced to > > >>> choose > > >>>> between maintaining outdated schemas or reprocessing all state data > > when > > >>>> schema changes are required. > > >>>> > > >>>> Here’s the proposal document: Link > > >>>> < > > >>> > > > https://docs.google.com/document/d/1WtAxp-jAVTLMOfWNldLCAoK137P0ZCMxR8hOZGcMxuc/edit?tab=t.0 > > >>>> > > >>>> Your feedback and ideas are welcome to refine this feature. > > >>>> > > >>>> Thanks, > > >>>> Weiqing > > >>>> > > >>> > > >> > > > > > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > Winchester, Hampshire SO21 2JN >
