Weiqing Yang created FLINK-40298:
------------------------------------
Summary: Opt-in name-based state schema evolution for RowData
Key: FLINK-40298
URL: https://issues.apache.org/jira/browse/FLINK-40298
Project: Flink
Issue Type: Sub-task
Components: API / Type Serialization System, Table SQL / Runtime
Reporter: Weiqing Yang
The core of FLIP-527. Adds:
* A new option table.exec.state.schema-evolution.enabled (Boolean, default
false).
* Name-based compatibility in
RowDataSerializer.RowDataSerializerSnapshot.resolveSchemaCompatibility: when
the field layouts differ but the difference is a supported backward-compatible
change, it returns compatibleAfterMigration() instead of incompatible(). Fields
are matched by the names persisted by FLINK-40120. Added fields must be
nullable, removed fields and changed leaf types are rejected, and nested ROW
fields are validated by recursion.
* An override of migrate that remaps a row into the new layout: fields placed
by name, added fields null-filled, row kind preserved, nested ROW values
remapped recursively.
The option is carried as an in-memory, non-persisted flag on the RowData
serializer rather than as anything written into the snapshot. The flag is set
only on a serializer that is a state's own value serializer, at the point the
state descriptor initializes it. Consequently a RowData serializer that sits
below a composite serializer (List or Tuple, as in interval and outer join
buffers), or one an operator pre-builds itself, never carries the flag and is
rejected on restore rather than migrated. This matters for correctness:
migration is not propagated into nested serializers by composite snapshots, so
those shapes must fail closed. A test asserts this rejection explicitly.
Snapshots written before field names were persisted have no names and are
likewise rejected.
Depends on FLINK-40296.
FLIP-527:
https://cwiki.apache.org/confluence/spaces/FLINK/pages/353601981/FLIP-527+State+Schema+Evolution+for+RowData
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
