foosekis4j opened a new pull request, #11211:
URL: https://github.com/apache/paimon/pull/11211

   ### Purpose
   
   Primary-key compaction is Paimon's heaviest maintenance work, and it scales 
with CPU threads per write subtask (#10133). This PR adds an optional module, 
`paimon-gpu`, that runs that compaction on an NVIDIA GPU through the 
`CompactRewriterFactory` hook from #9762. Paimon still selects inputs, 
schedules, and commits. Only the merge of a compaction's files moves to the 
device.
   
   **Stacked on #11209.** The first commit here is #11209's 
(`MergeTreeCompactRewriter#writerFactory()`); this PR adds the second.
   
   **How it works.** For each batch of sections:
   1. cuDF reads the Parquet files.
   2. cuDF sorts the rows by key and `_SEQUENCE_NUMBER`.
   3. A TornadoVM kernel keeps the last row of each key, and drops retractions 
when the output level drops deletes.
   4. The survivors are gathered and written by cuDF.
   
   The `DataFileMeta` comes from the written footer through Paimon's own 
Parquet stats extractor. Ranges, delete counts and value stats match what 
`MergeTreeCompactRewriter` writes.
   
   **Scope.**
   - `deduplicate` tables without `sequence.field`, `ignore-delete`, record 
expiration, thin mode, deletion vectors, lookup changelog or file indexes.
   - Parquet with zstd, snappy or no compression; INT/BIGINT keys; INT, BIGINT, 
DOUBLE, DATE and STRING values; local files.
   - Everything else, and any device failure or failed result check, is handed 
to Paimon's rewriter. Nothing is ever half-written.
   
   **Build.** Only `-Pgpu` (JDK 22+) builds the module, so default builds are 
unchanged. TornadoVM 8.1 (Apache 2.0) is a `provided` dependency, supplied by 
its SDK at run time. Paimon's `_VALUE_KIND` is an `INT(8)` column, and 
compactions mix files written by parquet-mr and by libcudf. TornadoVM's cuDF 
tasks needed fixes for both: beehive-lab/TornadoVM#1329.
   
   **Results.** RTX 4090 with a 32-core host, JDK 21. A full compaction of a 
32-bucket table (50M keys; 93.3M records in 800 files, merged into 49.4M rows; 
BIGINT key plus INT, DOUBLE, BIGINT, INT and STRING values; zstd). Each run 
starts from the same files.
   
   | Rewriter | Writers | Time |
   |---|---:|---:|
   | Paimon | 1 | 45.81 s |
   | Paimon | 8 | 8.20 s |
   | Paimon | 16 | 6.58 s (fastest CPU) |
   | Paimon | 32 | 8.48 s |
   | GPU | 1 | **3.78 s** (12.1x one writer, 1.74x the fastest CPU) |
   
   Both produce the same table: same row count and same order-independent 
digest. One GPU writer is enough, which leaves the host's cores to ingestion 
and queries. The numbers come from the same code packaged out of tree, with 
each configuration run once: https://github.com/beestream-ai/jitPaimon.
   
   ### Tests
   
   - `KvLayoutTest` (no device): which tables are taken or refused, and how 
columns are placed. 9/9.
   - `GpuCompactionITCase`: compacts two copies of a table, one with Paimon's 
rewriter and one on the device. It covers:
     - a full compaction;
     - three rounds of appends plus non-full compactions (level-1 outputs keep 
retractions, and the inputs mix Paimon-written and device-written files);
     - a final full compaction.
   
     After each step it requires the same rows and the same data-file metadata, 
and it asserts that no compaction fell back to the CPU. The only allowed 
difference: libcudf writes a zero DOUBLE minimum as `-0.0`, as the Parquet 
format recommends, where parquet-mr writes `0.0`. It is skipped without a 
TornadoVM JVM. It passes on an RTX 4090:
   
   ```bash
   mvn -Pgpu -pl paimon-gpu test -Dtest=GpuCompactionITCase \
       -Dpaimon.gpu.testArgs="@$TORNADOVM_HOME/tornado-argfile 
-Dtornado.device.memory=8GB"
   ```
   
   `mvn -Pgpu -pl paimon-gpu verify` passes checkstyle, spotless and RAT.
   
   ---
   **AI Disclosure**
   - Model: Claude Opus 5.5
   - Platform/Tool: Claude Code
   - Human Oversight: partially reviewed
   - Prompt Summary: port an out-of-tree GPU compaction rewriter (TornadoVM + 
cuDF) into an optional Paimon module, with CPU-vs-GPU equivalence tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to