Vinay Sagar Gonabavi created FLINK-40990:
--------------------------------------------
Summary: MySQL CDC source suppresses terminal missing-schema
errors and keeps checkpointing without CDC
Key: FLINK-40990
URL: https://issues.apache.org/jira/browse/FLINK-40990
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.6.0
Environment: Linux, Java 17, Flink 2.2.0, MySQL 8.0, and a Flink CDC
3.6.0-based connector. The upstream error-suppression branch was also checked
at commit {{{}1c9f2aa091c6f71868006668aae25f1dccb9710a{}}}; runtime
reproduction on unmodified upstream is pending.
Reporter: Vinay Sagar Gonabavi
h3. Problem
An included table with a missing reader schema raises a terminal binlog
processing error, but the MySQL source can remain {{RUNNING}} and continue
checkpointing while emitting no further CDC records, including records for
other captured tables.
Observed with a CDC 3.6.0-based connector. The same error-suppression branch
remains in current upstream {{master}}
({{{}1c9f2aa091c6f71868006668aae25f1dccb9710a{}}}); current-master runtime
verification is pending.
h3. Reproduction
# Capture two tables, {{inventory.orders}} and {{{}inventory.healthy{}}}, and
save a known-good savepoint.
# Restore capturing only {{{}inventory.healthy{}}}, then take another
savepoint. Normal filtering removes {{orders}} from the reader's schema map.
# Restore that savepoint capturing both tables, with
{{scan.newly-added-table.enabled=false}} and
{{{}scan.binlog.newly-added-table.enabled=true{}}}.
# Write changes to both tables. Disable automatic restarts
({{{}restart-strategy.type=none{}}}) to make terminal failure observable.
*Actual:* the source logs {{{}whose schema isn't known to this connector{}}},
stops processing CDC for both tables, and remains {{RUNNING}} with successful
checkpoints.
*Expected:* with the default {{fail}} inconsistent-schema policy, the original
error must reach Flink's failure handling. Explicit {{{}warn{}}}/{{{}ignore{}}}
policies should retain their existing behavior.
h3. Cause and proposed fix
[MySqlErrorHandler returns without recording the producer
failure|https://github.com/apache/flink-cdc/blob/1c9f2aa091c6f71868006668aae25f1dccb9710a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/debezium/task/context/MySqlErrorHandler.java#L63-L84],
although [the streaming source clears its event handlers after that
error|https://github.com/apache/flink-cdc/blob/1c9f2aa091c6f71868006668aae25f1dccb9710a/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/io/debezium/connector/mysql/MySqlStreamingChangeEventSource.java#L436-L447].
Propagate the failure through the existing producer-error path. Prevent later
callbacks from advancing offsets or emitting heartbeats, reject buffered
batches after failure, and prevent failed bounded backfill from emitting a
successful end watermark. Preserve the original cause and include
binlog-position diagnostics.
This reports failure propagation, not automatic discovery or repair of a
missing schema. It is distinct from FLINK-34884, which concerns discovering
schemas when adding tables on restore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
