[
https://issues.apache.org/jira/browse/FLINK-40656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Fang updated FLINK-40656:
----------------------------------
Description:
h2. Description
A failed schema change can leave the distributed `SchemaCoordinator` in
`EVOLVING`. During global failover, the coordinator instance is reused, but
`resetToCheckpoint()` restores only the schema manager. Replayed schema
requests then fail with `Unexpected evolving status: EVOLVING`, exhausting the
restart budget even after the sink becomes available again.
h2. Steps to reproduce
# Start a Fluss-to-Fluss CDC pipeline and complete a checkpoint.
# Stop the sink Fluss Coordinator, add `extra_v2 STRING` to the source table,
and write rows using the new column.
# Let the failed sink schema update trigger failover, then restart the sink
Fluss Coordinator.
# Schema requests are replayed, but the CDC job repeatedly fails with
`Unexpected evolving status: EVOLVING`; the target column and subsequent data
remain unsynchronized.
h2. Expected behavior
Checkpoint restore should discard in-flight schema evolution state and allow
schema requests to be replayed successfully after the sink recovers. The
previous schema-change worker must exit before restored state is installed or a
new metadata-apply call starts.
h2. Proposed fix
* Quiesce previous coordinator work at the shared reset boundary, then restore
durable schema state and reinitialize transient protocol state in both
coordinator implementations.
* Treat null checkpoint state as an empty coordinator state, and reject
requests belonging to a reset attempt.
The same-instance coordinator regression reproduces the exception on main
commit `b11d6fdf6346520e720396f25e5b5162ce5435b9`. This is distinct from
FLINK-37578 (successful evolution response ordering) and FLINK-37713
(preserving restored schemas during initial startup).
was:
h2. Description
A failed schema change can leave the distributed `SchemaCoordinator` in
`EVOLVING`. During global failover, the coordinator instance is reused, but
`resetToCheckpoint()` restores only the schema manager. Replayed schema
requests then fail with `Unexpected evolving status: EVOLVING`, exhausting the
restart budget even after the sink becomes available again.
h2. Steps to reproduce
# 1. Start a Fluss-to-Fluss CDC pipeline and complete a checkpoint.
# 2. Stop the sink Fluss Coordinator, add `extra_v2 STRING` to the source
table, and write rows using the new column.
# 3. Let the failed sink schema update trigger failover, then restart the sink
Fluss Coordinator.
# 4. Schema requests are replayed, but the CDC job repeatedly fails with
`Unexpected evolving status: EVOLVING`; the target column and subsequent data
remain unsynchronized.
h2. Expected behavior
Checkpoint restore should discard in-flight schema evolution state and allow
schema requests to be replayed successfully after the sink recovers. The
previous schema-change worker must exit before restored state is installed or a
new metadata-apply call starts.
h2. Proposed fix
- Quiesce previous coordinator work at the shared reset boundary, then restore
durable schema state and reinitialize transient protocol state in both
coordinator implementations.
- Treat null checkpoint state as an empty coordinator state, and reject
requests belonging to a reset attempt.
The same-instance coordinator regression reproduces the exception on main
commit `b11d6fdf6346520e720396f25e5b5162ce5435b9`. This is distinct from
FLINK-37578 (successful evolution response ordering) and FLINK-37713
(preserving restored schemas during initial startup).
> Distributed schema coordinator can remain EVOLVING after checkpoint restore
> ---------------------------------------------------------------------------
>
> Key: FLINK-40656
> URL: https://issues.apache.org/jira/browse/FLINK-40656
> Project: Flink
> Issue Type: Bug
> Reporter: Xiaobing Fang
> Priority: Major
>
> h2. Description
> A failed schema change can leave the distributed `SchemaCoordinator` in
> `EVOLVING`. During global failover, the coordinator instance is reused, but
> `resetToCheckpoint()` restores only the schema manager. Replayed schema
> requests then fail with `Unexpected evolving status: EVOLVING`, exhausting
> the restart budget even after the sink becomes available again.
> h2. Steps to reproduce
> # Start a Fluss-to-Fluss CDC pipeline and complete a checkpoint.
> # Stop the sink Fluss Coordinator, add `extra_v2 STRING` to the source table,
> and write rows using the new column.
> # Let the failed sink schema update trigger failover, then restart the sink
> Fluss Coordinator.
> # Schema requests are replayed, but the CDC job repeatedly fails with
> `Unexpected evolving status: EVOLVING`; the target column and subsequent data
> remain unsynchronized.
> h2. Expected behavior
> Checkpoint restore should discard in-flight schema evolution state and allow
> schema requests to be replayed successfully after the sink recovers. The
> previous schema-change worker must exit before restored state is installed or
> a new metadata-apply call starts.
> h2. Proposed fix
> * Quiesce previous coordinator work at the shared reset boundary, then
> restore durable schema state and reinitialize transient protocol state in
> both coordinator implementations.
> * Treat null checkpoint state as an empty coordinator state, and reject
> requests belonging to a reset attempt.
> The same-instance coordinator regression reproduces the exception on main
> commit `b11d6fdf6346520e720396f25e5b5162ce5435b9`. This is distinct from
> FLINK-37578 (successful evolution response ordering) and FLINK-37713
> (preserving restored schemas during initial startup).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
