Hi Ramin,
thank you for proposing this FLIP. It nicely extends the existing
semantics and improves the experience esp. for the CONTINUOUS mode. But
even in FULL mode, people don't want to pay for full processing and
would like to have more control over what will be reprocessed in case of
a query evolution. Sometimes also to skip malformed data.
I was heavily involved in the FLIP discussions and it took us quite a
few iterations to end up with this concise syntax that nicely fits into
Flink semantics of savepoints and time travelling (FLIP-308), while
keeping new developments such as PTFs (FLIP-440) in the picture as well.
+1 to this proposal.
Cheers,
Timo
On 01.12.25 18:23, Ramin Gharib wrote:
Hi everyone,
I would like to start a discussion on FLIP-557: Granular Control over Data
Reprocessing and State Retention in Materialized Table Evolution [1].
Currently, ALTER MATERIALIZED TABLE forces a full job restart and discards
state, which is inefficient for many evolution scenarios. FLIP-557 proposes
decoupling data scope from state management by introducing two new optional
clauses:
1. START_MODE*:* Controls the data processing window (e.g., FROM_BEGINNING,
RESUME_OR_...).
2. STATE_RETENTION*:* Controls how existing state is handled (e.g., NONE,
PTF_ONLY).
This gives users explicit control over cost and correctness during table
evolution.
For more details, please refer to the FLIP [1].
Looking forward to your feedback and thoughts!
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-557%3A+Granular+Control+over+Data+Reprocessing+and+State+Retention+in+Materialized+Table+Evolution
Best regards,
Ramin Gharib
