foosekis4j opened a new pull request, #11211: URL: https://github.com/apache/paimon/pull/11211
### Purpose Primary-key compaction is Paimon's heaviest maintenance work, and it scales with CPU threads per write subtask (#10133). This PR adds an optional module, `paimon-gpu`, that runs that compaction on an NVIDIA GPU through the `CompactRewriterFactory` hook from #9762. Paimon still selects inputs, schedules, and commits. Only the merge of a compaction's files moves to the device. **Stacked on #11209.** The first commit here is #11209's (`MergeTreeCompactRewriter#writerFactory()`); this PR adds the second. **How it works.** For each batch of sections: 1. cuDF reads the Parquet files. 2. cuDF sorts the rows by key and `_SEQUENCE_NUMBER`. 3. A TornadoVM kernel keeps the last row of each key, and drops retractions when the output level drops deletes. 4. The survivors are gathered and written by cuDF. The `DataFileMeta` comes from the written footer through Paimon's own Parquet stats extractor. Ranges, delete counts and value stats match what `MergeTreeCompactRewriter` writes. **Scope.** - `deduplicate` tables without `sequence.field`, `ignore-delete`, record expiration, thin mode, deletion vectors, lookup changelog or file indexes. - Parquet with zstd, snappy or no compression; INT/BIGINT keys; INT, BIGINT, DOUBLE, DATE and STRING values; local files. - Everything else, and any device failure or failed result check, is handed to Paimon's rewriter. Nothing is ever half-written. **Build.** Only `-Pgpu` (JDK 22+) builds the module, so default builds are unchanged. TornadoVM 8.1 (Apache 2.0) is a `provided` dependency, supplied by its SDK at run time. Paimon's `_VALUE_KIND` is an `INT(8)` column, and compactions mix files written by parquet-mr and by libcudf. TornadoVM's cuDF tasks needed fixes for both: beehive-lab/TornadoVM#1329. **Results.** RTX 4090 with a 32-core host, JDK 21. A full compaction of a 32-bucket table (50M keys; 93.3M records in 800 files, merged into 49.4M rows; BIGINT key plus INT, DOUBLE, BIGINT, INT and STRING values; zstd). Each run starts from the same files. | Rewriter | Writers | Time | |---|---:|---:| | Paimon | 1 | 45.81 s | | Paimon | 8 | 8.20 s | | Paimon | 16 | 6.58 s (fastest CPU) | | Paimon | 32 | 8.48 s | | GPU | 1 | **3.78 s** (12.1x one writer, 1.74x the fastest CPU) | Both produce the same table: same row count and same order-independent digest. One GPU writer is enough, which leaves the host's cores to ingestion and queries. The numbers come from the same code packaged out of tree, with each configuration run once: https://github.com/beestream-ai/jitPaimon. ### Tests - `KvLayoutTest` (no device): which tables are taken or refused, and how columns are placed. 9/9. - `GpuCompactionITCase`: compacts two copies of a table, one with Paimon's rewriter and one on the device. It covers: - a full compaction; - three rounds of appends plus non-full compactions (level-1 outputs keep retractions, and the inputs mix Paimon-written and device-written files); - a final full compaction. After each step it requires the same rows and the same data-file metadata, and it asserts that no compaction fell back to the CPU. The only allowed difference: libcudf writes a zero DOUBLE minimum as `-0.0`, as the Parquet format recommends, where parquet-mr writes `0.0`. It is skipped without a TornadoVM JVM. It passes on an RTX 4090: ```bash mvn -Pgpu -pl paimon-gpu test -Dtest=GpuCompactionITCase \ -Dpaimon.gpu.testArgs="@$TORNADOVM_HOME/tornado-argfile -Dtornado.device.memory=8GB" ``` `mvn -Pgpu -pl paimon-gpu verify` passes checkstyle, spotless and RAT. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: partially reviewed - Prompt Summary: port an out-of-tree GPU compaction rewriter (TornadoVM + cuDF) into an optional Paimon module, with CPU-vs-GPU equivalence tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
