yux created FLINK-36184:
---------------------------
Summary: Transform Operator swallows schema changes from tables
not present in transform rules
Key: FLINK-36184
URL: https://issues.apache.org/jira/browse/FLINK-36184
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.2.0, cdc-3.3.0
Reporter: yux
Currently, tables that are not present in transform blocks should be treated as
if there's such a dummy fallback block:
transform:
- source-table: "\.*.\.*" # capture all unmentioned tables
projection: "*" # keep all columns
# without filtering any rows
There's a bug in #3557's implementation, where schema change events should be
filtered out if there's no wildcard (*) in transform rules. However, it also
filters out those tables that are not defined in transform rules, which causes
schema change events lost with the following example:
transform:
- source-table: foo.bar.baz # Another table that doesn't really matter
projection: ...
Here, since there's one transform block, TransformOperator will be added into
operator chain. Now let's perform some schema change events in another table
(like db.table), it will be filtered out since TransformOperator regards it as
a asterisk-less table and does not require schema change events.
By checking if a table is transformed or not, we could set hasAsterisk flag map
correctly and resolve this problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
