lszskye opened a new pull request, #248:
URL: https://github.com/apache/paimon-cpp/pull/248
### Purpose
Linked issue: #174
Support `input` and `lookup` changelog producers for primary-key tables.
This change:
- Writes raw input changelog records into changelog files while write
process.
- Rolls input changelog files by target file size, consistently with Java
Paimon.
- Produces lookup changelogs during compaction by comparing level-0 records
with existing records.
- Supports lookup changelog with:
- several merge engines;
- deletion vectors enabled;
- `ignore_previous_files`;
- row deduplication and ignored comparison fields;
- schema evolution and external data paths.
- Persists keys in first-row lookup SST files so excluded higher-level
records can be detected.
- Supports shared shredding when writing input and compaction changelog
files.
- Supports configurable changelog file prefix, format, and compression.
- Reads changelog manifests during streaming scans.
- Excludes level-0 files from the initial full scan for lookup changelog
tables to avoid duplicate emission.
- Adds order-independent equality comparison for MAP values.
- Keeps `full-compaction` changelog producer unsupported.
### Tests
Added and updated unit and integration coverage for:
- input and lookup changelog stream reads;
- insert, update, and delete changelogs;
- first-row and deletion-vector lookup;
- row deduplication and ignored fields;
- schema evolution and external paths;
- configurable changelog format, compression, and prefix;
- shared-shredding changelog files;
- nested row and order-independent MAP equality;
- changelog manifest scanning.
### API and Format
This change adds public option constants for:
- `changelog-producer.row-deduplicate`
- `changelog-producer.row-deduplicate-ignore-fields`
- `changelog-file.prefix`
- `changelog-file.format`
- `changelog-file.compression`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
