Hi all, I'm Michalis Papadimitriou from the TornadoVM team https://github.com/beehive-lab/TornadoVM). TornadoVM compilesplain Java methods into GPU kernels; latest is release 8.0.0 under Apache 2.0
We have built a proof of concept, iceberg-tornadovm: a library on Iceberg 1.12's public APIs that runs merge-on-read scans and table maintenance on NVIDIA GPUs. Iceberg still plans and commits (planFiles, RewriteFiles, RowDelta with their usual validations); the GPU only does the per-row work in between. It needs no engine and no change to Iceberg. What it covers today: - Merge-on-read scans with position deletes, equality deletes and v3 deletion vectors applied on the GPU - Equality deletes -> deletion vectors (v3) or file-scoped position deletes (v2) - Delete by key, written as deletion vectors - NDV theta sketches into Puffin - Compaction: bin-pack, sorted and Z-order Every operation has a CPU version using the same algorithm, as a reference. Early results (Spark 3.5.9, Iceberg 1.12): - 3.9-4.8x faster merge-on-read scans at 5B rows (2x A10 vs 128 CPU threads), scaling ~2x from one GPU to two - Equality-delete -> DV conversion 106x cheaper than Spark compaction (TPC-DS SF100: 0.77 s vs 81.9 s) - 14x faster NDV statistics than compute_table_stats (2B rows) - 1.8-8.6x faster compaction than Spark on CPU We think that v4 decision to forbid new equality deletes (#17783), converting existing ones becomes a required work, which is where the GPU can help the most. We'd like to discuss the possibility of an upstream path , and would value the community's view on: 1. Home: should this stay out of tree, following the direction taken with Comet (#15674), or is there interest in a contrib/optional module? 2. Hooks: the FileFormat API (#12774) covers much of what we need. Are there other extension points (e.g. a pluggable rewrite-group runner or delete applier) the community would accept, so accelerators don't need forks? 3. Validation: which benchmarks or correctness suites would you want to see before discussing this further? Any other input it would be great. We're happy to share the code and benchmark harness, or to present at a community sync if that's useful. Thanks, Michalis Papadimitriou TornadoVM team
