Xiaobing Fang created FLINK-41434:
-------------------------------------
Summary: Fluss CDC source rebuilds the column mapping for every
record with an older schema
Key: FLINK-41434
URL: https://issues.apache.org/jira/browse/FLINK-41434
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Reporter: Xiaobing Fang
After a column is appended to a Fluss table, a source reader can see the new
schema and continue receiving records written with the previous schema.
To reproduce, use a 453-column table, append one nullable column, and let the
same reader process a 454-column record followed by multiple 453-column records.
For every older record, FlussRecordDeserializer rebuilds both Fluss Schema
objects and recomputes the same column mapping. Building the schemas also
repeats duplicate-name checks whose cost grows quadratically with the number of
columns. In a local benchmark alternating the two versions, deserialization
consumed 551 µs of thread CPU per record, compared with 3.26 µs when all
records used the latest version.
The reader should compute the mapping once for each source and target schema
version pair and reuse it for subsequent records. Older records should continue
to emit null for the appended column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
