foosekis4j opened a new pull request, #18898: URL: https://github.com/apache/iceberg/pull/18898
This adds `iceberg-tornadovm`, an optional module that runs Iceberg's per-row work on an NVIDIA GPU while planning, delete-file resolution, validation and commits stay Iceberg's, through its public APIs: merge-on-read scans with position deletes, deletion vectors and equality deletes (filters, grouped aggregates, materialization), compaction (bin-pack, sorted, z-order), equality deletes converted to deletion vectors, delete by key as deletion vectors, and NDV theta sketches. GPU code is Java compiled to CUDA by TornadoVM, with cuDF for Parquet; a CPU twin runs the same algorithms on the host. `iceberg-tornadovm-spark` plugs it into Spark 4.2 through the providers of #18897 (the first commit here) and a planner strategy for aggregates. Both modules join the build only when `TORNADOVM_HOME` is set, so default builds and CI are unchanged. Opened to discuss whether, and in what shape, this belongs in Iceberg. Test plan: every device result is checked against `IcebergGenerics` and the CPU twin (`TestDeviceMergeOnRead` 27, `TestDeviceEvolutionAndStorage` 5, `TestStarAggregation` 6), and the Spark paths against Spark with the GPU off (`TestSparkGpuReads` 3, `TestSparkGpuAggregates` 6, `TestSparkGpuRewrite` 1); all pass with TornadoVM 8.1.0 on an RTX 4090. Without `TORNADOVM_HOME`, `iceberg-spark-4.2` builds and its JMH sources compile as before. On 100M rows after merge-on-read deletes, a grouped aggregate runs in 256-292 ms on the GPU against 862-1582 ms for the CPU twin on 32 threads, same answers. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: partially reviewed - Prompt Summary: Port the GPU merge-on-read and maintenance modules onto Iceberg main and Spark 4.2, clean them for upstream, and open them as one pull request. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
