peiyu created FLINK-40148:
-----------------------------
Summary: TableIdRouter resolves wrong sink table when source regex
has multi-digit suffix and sink-table has no back-reference
Key: FLINK-40148
URL: https://issues.apache.org/jira/browse/FLINK-40148
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.6.0
Environment: flink 1.20.3
flinkcdc 3.6.0
Reporter: peiyu
Fix For: cdc-3.7.0
## Problem
TableIdRouter.resolveReplacement can produce incorrect sink table names when
the source-table
regex contains a capturing group whose value may span multiple characters (e.g.
multi-digit
suffixes like `13`, `14`, `16`) and the sink-table does not use a `$N`
back-reference.
In this case the unmatched characters after the captured prefix are appended to
the configured
sink-table, producing wrong sink names. For example, given:
route:
- source-table: db_6.table_([1-9]|1[0-6])
sink-table: new_db_6.table_merged
`db_6.table_13` is wrongly routed to `new_db_6.table_merged3` (instead of the
configured
`new_db_6.table_merged`), `db_6.table_14` to `new_db_6.table_merged4`,
`db_6.table_16` to
`new_db_6.table_merged6`, etc. Tables 1-9 are not affected because the regex
consumes the
entire single-digit suffix.
## Reproduction
1. Configure a pipeline with the route above.
2. Run the job against a source that has tables `db_6.table_1` …
`db_6.table_16`.
3. Observe in the logs / sink database that tables 10-16 are written to sink
tables named
`new_db_6.table_merged0`, `new_db_6.table_merged1`,
`new_db_6.table_merged3`, … instead of
the single configured sink table.
A unit test that reproduces the bug (without the fix) is added in this ticket;
it fails before
the patch and passes after.
## Root Cause
`TableIdRouter.resolveReplacement` builds a `Matcher` against the source table
id and calls
`Matcher.find()` followed by `Matcher.replaceAll(sinkTable)`:
Matcher matcher = route.f0.matcher(originalTable.toString());
if (matcher.find()) {
return TableId.parse(matcher.replaceAll(route.f1));
}
Two issues combine to produce the bug:
1. `find()` only matches a *prefix* of the source table id. With the regex
`([1-9]|1[0-6])`, the alternation `[1-9]` wins first, matching only `1` from
input `13`.
2. `Matcher.replaceAll` internally calls `reset()` and iterates with `find()`,
so it only
replaces the prefix matched in step 1 and leaves the trailing characters of
the source
table id (`0`, `3`, `4`, `5`, `6`) appended to the sink-table. When the
sink-table has no
`$N` reference, those trailing characters end up in the result.
The same root cause also makes `find()`-based capturing groups return only the
first matched
digit when the sink-table *does* use `$1`, so the bug is not strictly limited
to back-reference
free sinks.
## Proposed Fix
Switch to `Matcher.matches()` to ensure the entire source table id is consumed,
and use
`Matcher.appendReplacement` / `Matcher.appendTail` to perform the `$N`
substitution on the
matched region only — this avoids the trailing-characters-left-over problem of
`replaceAll`.
## Tests
- Add `TableIdRouterTest#testRouteWithoutBackReferenceWithMultiDigitSuffix` to
lock down the
reported scenario (sink-table has no back-reference).
- Add `TableIdRouterTest#testRouteWithBackReferenceAndMultiDigitCapture` to
cover the related
case where sink-table uses `$1` and the captured group must be the full
multi-digit value.
- All existing tests in `TableIdRouterTest`, `TableIdRouterMatchModeTest`, and
`SchemaDerivatorTest` continue to pass (27 tests total).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
