fxbing commented on PR #4494:
URL: https://github.com/apache/flink-cdc/pull/4494#issuecomment-5199610939
Ran this PR (5a7ce328) end-to-end as a Fluss→Fluss pipeline, built against
apache/fluss main (22cb1969). Build is fine and the PR's own tests are green
(74 UT + 43 IT). Two issues showed up in E2E:
1. Source deserializer doesn't handle ARRAY / MAP / ROW
FlussRecordDeserializer.convertFlussField covers the 16 scalar types and
throws for everything else:
java.lang.UnsupportedOperationException: Unsupported Fluss data type for
deserialization: ArrayType
at
...FlussRecordDeserializer.convertFlussField(FlussRecordDeserializer.java:396)
at
...FlussRecordDeserializer.convertFlussRowToCdcRecord(FlussRecordDeserializer.java:201)
Repro: a source table with tags ARRAY<STRING> (same for MAP / ROW) — the
job fails on the first record and goes into a restart loop. The sink side
already supports these types via CdcAsFlussArray /
CdcAsFlussMap / CdcAsFlussRow, so the gap looks source-only.
2. CreateTableEvent isn't reconciled against an existing sink table
If the sink table already exists with fewer columns than the source,
FlussMetaDataApplier treats CreateTableEvent as create-if-not-exists and leaves
the physical schema as is. Upstream-only columns are then
silently dropped while the job stays RUNNING with no error.
Repro: sink (id, payload), source (id, payload, extra_col) → the row lands
as (1, 'cold-start') and extra_col is lost. The coordinator does pass the full
schema down:
Step 3.5 - Corresponding schema changes are: [CreateTableEvent{...
`extra_col` STRING ...}]
A worse variant: when the sink is missing a middle column (source (id,
name, sale_quantity) vs sink (id, sale_quantity)), the write still succeeds
with name projected away rather than failing fast.
Would it make sense for the metadata applier to reload the real TableInfo
on CreateTableEvent, append missing nullable trailing columns, and fail fast
when the difference can't be reconciled that way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
