谢云流 created FLINK-41197:
---------------------------
Summary: Multi-table DROP may bypass tables.exclude when binlog
newly-added table discovery is enabled
Key: FLINK-41197
URL: https://issues.apache.org/jira/browse/FLINK-41197
Project: Flink
Issue Type: Bug
Components: Flink CDC
Reporter: 谢云流
h3. Summary
When scan.binlog.newly-added-table.enabled is enabled, a MySQL schema-change
record that contains multiple table names in a single source.table value may
pass the subscription filter as one combined table identifier. Parsing the DROP
statement afterwards can then emit a DropTableEvent for a table explicitly
excluded by tables.exclude.
This report is based on prior component-level observations with constructed
schema-change records. A standalone public reproducer and validation against
real MySQL binlogs are still being prepared. The exact affected upstream
release(s), MySQL version and complete environment details will be provided
after verification.
h3. Triggering conditions
* MySQL Pipeline Source has scan.binlog.newly-added-table.enabled set to true.
* The inclusion pattern matches the combined table-name value.
* tables.exclude precisely excludes one of the individual tables.
* A multi-table DROP schema-change record represents the table names as a
comma-separated source.table value.
Relevant configuration fragment:
{code:yaml}
scan.binlog.newly-added-table.enabled: true
tables.exclude: test_db.excluded_table
{code}
The inclusion pattern must be supplied with the reproducer. The fragment above
is not a complete runnable pipeline.
Illustrative SQL for the scenario to be validated:
{code:sql}
DROP TABLE test_db.captured_table, test_db.excluded_table;
{code}
Constructed metadata used in the component scenario:
{code:text}
source.db = test_db
source.table = captured_table,excluded_table
{code}
The SQL above has not yet been verified to produce exactly this metadata in a
real upstream MySQL/Debezium run.
h3. Prior component-level observations
The combined table-name value can match the inclusion pattern while failing to
match the precise exclusion for test_db.excluded_table. The record is then
accepted and the DDL parser emits individual DropTableEvent instances,
including the excluded table's event.
A single-table excluded DROP was filtered in the component control cases.
Therefore this report is specifically about multi-table records under binlog
newly-added table discovery, rather than all DROP events bypassing exclusion.
h3. Expected behavior
Apply inclusion and exclusion rules to each affected table. Emit the DROP event
for the subscribed table and suppress the DROP event for the excluded table,
without losing the subscribed table's event from the same statement.
h3. Suspected root cause
Record-level filtering uses a combined table identifier, while DDL parsing
produces per-table events. The exclusion rules do not appear to be reapplied to
each parsed event.
h3. Reproducer and validation status
A minimal public reproducer, full configuration, precise versions and raw event
metadata will be added. Please treat the combined source.table value as a
constructed component input until the real-binlog path is verified. We are not
claiming end-to-end reproduction on all upstream releases.
For the reproducer, the important assertions are that:
* No DropTableEvent is emitted for the excluded table.
* The DropTableEvent for the subscribed table is retained.
* The single-table excluded DROP control remains filtered.
h3. Related work
FLINK-36945 and PR #3876 discuss related multi-table RENAME metadata/filtering
behavior. They are referenced as related work; this report does not assume that
the RENAME fix covers multi-table DROP.
[FLINK-36945|https://issues.apache.org/jira/browse/FLINK-36945]
[PR #3876|https://github.com/apache/flink-cdc/pull/3876]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
