junmuz opened a new pull request, #9531:
URL: https://github.com/apache/paimon/pull/9531
Add two new configurations for lookup and full-compaction changelog
producers to filter out UPDATE_BEFORE (-U) and DELETE (-D) records from
changelog files.
### Purpose
When using lookup or full-compaction changelog producers, the generated
changelog files contain complete CDC records including UPDATE_BEFORE (-U) and
DELETE (-D) entries. However, many downstream consumers don't need these
retraction records:
- Upsert sinks (e.g. databases with primary keys) overwrite by key and
don't need -U records to correctly apply updates. This is quite important if
you are migrating away from Kafka. The existing `ignore-update-before` option
is only applicable for data writes.
- `change log-producer.ignore-delete` was added to keep it in sync with
`ignore-delete` pattern.
- Storage-constrained environments where changelog files contribute
significant overhead — filtering unnecessary row kinds reduces changelog file
size and I/O.
These options give users fine-grained control over what gets included in the
changelog output.
Note: These options should not be used when downstream consumers rely on
retraction semantics (e.g. Flink retract-based aggregations or joins), as
missing -U or -D records will produce incorrect results.
### Tests
- Existing ChangelogMergeTreeRewriterTest and
LookupMergeTreeCompactRewriterTest pass with updated constructor signatures
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
