SteNicholas opened a new issue, #1320:
URL: https://github.com/apache/paimon-rust/issues/1320

   ### Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-rust/issues) and found nothing 
similar.
   
   
   ### Motivation
   
   Rust can read tables that use `changelog-producer=full-compaction`, but it 
cannot produce their changelog.
   
   Reading already works. `StreamTableScan` follows changelog files for every 
producer other than `none` 
([`stream_table_scan.rs#L154-L158`](https://github.com/apache/paimon-rust/blob/062f1bcda0fd74f777b8ce4235292db6efd4456a/crates/paimon/src/table/stream_table_scan.rs#L154-L158)).
 For `full-compaction`, its initial full state reads only the last level, as in 
Java 
([`#L392-L405`](https://github.com/apache/paimon-rust/blob/062f1bcda0fd74f777b8ce4235292db6efd4456a/crates/paimon/src/table/stream_table_scan.rs#L392-L405),
 #1313).
   
   The write side has these gaps:
   
   - `ensure_supported` rejects primary-key compaction unless 
`changelog-producer` is `none`, `input` or `lookup` 
([`compact.rs#L1222-L1237`](https://github.com/apache/paimon-rust/blob/062f1bcda0fd74f777b8ce4235292db6efd4456a/crates/paimon/src/table/compact.rs#L1222-L1237)).
 Manual compaction and `CALL sys.compact` therefore fail on these tables.
   - Writes still succeed, but the writer's automatic compaction round is 
skipped silently because the mode is outside the compaction API 
([`auto_maintenance.rs#L27-L32`](https://github.com/apache/paimon-rust/blob/062f1bcda0fd74f777b8ce4235292db6efd4456a/crates/paimon/src/table/table_commit/auto_maintenance.rs#L27-L32)).
 `docs/src/compaction.md` documents this: "Table modes unsupported by the 
compaction API retain their write behavior without automatic compaction", and 
"The `full-compaction` changelog producer is unsupported."
   - `full-compaction.delta-commits` is not read anywhere.
   
   As a result, when only Rust writers (Rust, Python, DataFusion, C or Go) 
write to a `full-compaction` table, level-0 files are never compacted and no 
changelog is generated. A streaming reader, in Java or Rust, sees nothing after 
its initial full state until a Java job runs a full compaction.
   
   | | Java | Rust today |
   |---|---|---|
   | Write | Supported | Supported |
   | Compaction below the max level | Universal compaction, no changelog | 
Rejected (manual) / skipped (automatic) |
   | Compaction into the max level | Changelog `+I` / `-U` / `+U` / `-D` | 
Rejected / skipped |
   | Full-compaction trigger | `full-compaction.delta-commits` (Flink), 
`compact_strategy => 'full'` | None |
   | Streaming read | Max level, then changelog | Supported |
   
   ### Solution
   
   Port Java `FullChangelogMergeTreeCompactRewriter` and 
`FullChangelogMergeFunctionWrapper` to the primary-key compaction executor, and 
add a way to request a full compaction.
   
   **Changelog semantics** (`FullChangelogMergeFunctionWrapper`): for each key, 
compare the merged result with the key's record from the max level. By the 
level invariant there is at most one such record.
   
   - No max-level record, and the merged result is an add: `+I` (merged).
   - A max-level record, and the merged result is a retract: `-D` (old).
   - A max-level record, and the merged result is an add whose value differs, 
or row deduplication is disabled: `-U` (old), `+U` (merged).
   - Only the max-level record: no changelog.
   
   Changelog is produced only when the output level is the max level 
(`rewriteChangelog`). Compaction into lower levels behaves like `none`. 
Upgrading a file into the max level produces changelog without rewriting the 
data file (`UpgradeStrategy.CHANGELOG_NO_REWRITE`). Java asserts that deletes 
are dropped from a max-level result. The Rust executor currently sets 
`drop_delete` only for deletion-vector tables 
([`primary_key.rs#L381`](https://github.com/apache/paimon-rust/blob/062f1bcda0fd74f777b8ce4235292db6efd4456a/crates/paimon/src/table/compact/primary_key.rs#L381)),
 so this path has to guarantee it.
   
   **Options.** `changelog-producer.row-deduplicate`, 
`changelog-producer.row-deduplicate-ignore-fields`, 
`changelog-producer.ignore-update-before` and 
`changelog-producer.ignore-delete` are already parsed for the lookup producer 
(`ChangelogOptions` in 
[`lookup.rs#L32`](https://github.com/apache/paimon-rust/blob/062f1bcda0fd74f777b8ce4235292db6efd4456a/crates/paimon/src/table/compact/primary_key/lookup.rs#L32)).
 The full-compaction rewriter can share that parsing.
   
   **Invariant to keep.** Once the compaction API accepts a `full-compaction` 
table, every task whose output level is the max level must go through the 
changelog rewriter. The universal planner already promotes to the max level 
when all runs are selected 
([`planner.rs#L215-L216`](https://github.com/apache/paimon-rust/blob/062f1bcda0fd74f777b8ce4235292db6efd4456a/crates/paimon/src/table/compact/primary_key/planner.rs#L215-L216)).
 Accepting the mode in `ensure_supported` or the planner without the rewriter 
would silently drop changes from the stream.
   
   **Suggested steps**
   
   1. **Rewriter and manual full compaction.** Add the rewriter, a full mode on 
`CompactBuilder` that compacts all runs of each selected bucket into the max 
level (the counterpart of Java `compact(partition, bucket, true)`), and accept 
`full-compaction` in `ensure_supported` and the planner for the merge engines 
Java allows. Ordinary automatic rounds then also produce changelog whenever 
their output reaches the max level, as Java writers do.
   2. **Triggers.** Honor `full-compaction.delta-commits` in automatic 
compaction, and add `compact_strategy => 'full'` / `'minor'` to `CALL 
sys.compact`, matching the Java procedures.
   
   **Tests**
   
   - Changelog rows for `deduplicate`, `partial-update` and `aggregation` 
across insert, update, delete and no-op update.
   - `changelog-producer.row-deduplicate` (with and without ignore fields), 
`ignore-update-before` and `ignore-delete`.
   - Upgrading a single level-0 file into the max level emits `+I` changelog 
without rewriting the data file.
   - Compaction below the max level emits no changelog.
   - End to end: a Rust `StreamTableScan` over a Rust-written table sees the 
changelog after a full compaction.
   
   ### Anything else?
   
   **Out of scope, as in Java:** deletion vectors (Java `SchemaValidation` 
already rejects `full-compaction` with DV, and Rust mirrors that), `first-row` 
(only `none` / `lookup`), and postpone buckets. The Flink-only 
`changelog-producer.compaction-interval` is also out of scope.
   
   **Open questions**
   
   1. API shape for the full mode on `CompactBuilder`, for example 
`with_full_compaction()` or a strategy enum shared with `CALL sys.compact`.
   2. Java's `GlobalFullCompactionSinkWrite` remembers buckets written since 
the last full compaction in Flink operator state, and full-compacts them every 
`full-compaction.delta-commits` commits (default 1 for this producer). Rust's 
automatic round only sees the buckets changed by the current commit. Should 
step 2 full-compact only the current commit's buckets when the commit count 
reaches the threshold, or should it reconstruct the written buckets from 
snapshots since the last full compaction?
   3. For `partial-update` and `aggregation`, should rounds below the max level 
keep the existing `none` / `input` raw-state restrictions 
(`plain_state_mergeable`), with only max-level rounds exempt?
   4. Java rejects `full-compaction.delta-commits` with 
`changelog-producer=lookup` and on unaware- or dynamic-bucket append tables 
(`SchemaValidation`). Should Rust add the same validation in step 2?
   
   Related: #1247 (compaction API), #1252 (lookup changelog producer), #1313 
(streaming startup levels), #1312 (`CALL sys.compact`).
   
   Java reference: `FullChangelogMergeTreeCompactRewriter`, 
`FullChangelogMergeFunctionWrapper`, `ChangelogMergeTreeRewriter`, 
`MergeTreeCompactManagerFactory#createRewriter`, 
`GlobalFullCompactionSinkWrite`.
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to