mbutrovich commented on code in PR #17162: URL: https://github.com/apache/iceberg/pull/17162#discussion_r3570946768
########## site/docs/blog/posts/2026-07-10-accelerating-iceberg-rust-development-with-datafusion-comet.md: ########## @@ -0,0 +1,213 @@ +--- +date: 2026-07-10 +title: Accelerating Apache Spark Queries (and Iceberg Rust Development) with Apache DataFusion Comet +slug: accelerating-iceberg-rust-development-with-datafusion-comet # this is the blog url +authors: + - mbutrovich +categories: + - blog +--- + +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +<!-- more --> + +Apache Iceberg provides a universal table format that serves as a foundation for modern data +lakehouse +platforms. With Iceberg, users store their tables with the benefit of being able to access +and modify their data from a number of different query engines. +[Apache Spark](https://spark.apache.org) is the engine most closely associated with Iceberg. The +[Iceberg Java repository](https://github.com/apache/iceberg), the *de facto* reference +implementation of the Iceberg specification, ships Spark as its most mature integration. It is also the +engine most teams rely on for table maintenance like compaction and snapshot expiration. +In addition to Java, the Iceberg community maintains a number +of other Iceberg implementations like [C++](https://github.com/apache/iceberg-cpp), +[Go](https://github.com/apache/iceberg-go), and [Rust](https://github.com/apache/iceberg-rust). +These other implementations benefit not only from the Iceberg specification, but also the lessons +learned and design decisions of the Java project's community. The Java repository's extensive +test suites, for instance, include nearly 10,000 correctness tests driven by Spark (as of Iceberg +1.11 with Spark 4.1). Each implementation maintains its own test suite and can look to Iceberg Java +as a reference for both correct behavior and test coverage. None of them, however, can run Java's +tests directly against their own code. + +While Spark remains a powerful and robust engine, a number of projects exist to accelerate its +JVM-backed execution. One such solution is +[Apache DataFusion Comet](https://datafusion.apache.org/comet/), which Apple donated in 2024 +as a subproject of the [Apache DataFusion](https://datafusion.apache.org) query engine. Comet's +native execution engine runs CPU-bound jobs faster and IO-bound jobs with +fewer resources, giving users control over how they want to optimize their Spark jobs. As we will +see, Comet does more than speed up queries: the same design that makes it fast also makes it a tool +for accelerating Iceberg Rust's development. + +## Accelerating Spark Queries with Comet + +Comet builds upon several related Apache projects including DataFusion (for its efficient operator +implementations like joins and aggregations), [Arrow-rs](https://github.com/apache/arrow-rs) +(for its standardized in-memory format and robust Parquet reader), and, somewhat surprisingly, +both the Java and the Rust implementations of Iceberg. To accelerate Spark queries, Comet +intercepts execution +at the physical plan level. After Spark has parsed, planned, and optimized a user's query, +Comet's JVM code runs as one final optimizer rule to convert Spark plan nodes to Comet plan nodes. +These Comet plan nodes have a superpower: they execute in DataFusion's Rust engine over columnar Arrow +data. + +<figure markdown="span">{ width="750" }<figcaption>Comet converts a Spark physical plan into an equivalent DataFusion physical plan.</figcaption></figure> + +So how does Comet use *both* Iceberg libraries to accelerate Spark queries over Iceberg tables? +As previously mentioned, Iceberg provides robust integrations with Spark, enabling users to query +their Iceberg tables regardless of the Spark API they are using (*e.g.*, SQL, Scala, or PySpark). +Iceberg relies on Spark's +[`Data Source v2`](https://spark.apache.org/docs/4.2.0-preview4/sql-data-sources-v2.html) API to +integrate with query planning, a process that +Apache Iceberg PMC member Russell Spitzer recently described in a talk titled +["An Extremely Technical Overview of How Apache Iceberg Planning Actually Works"](https://www.youtube.com/watch?v=kJaD0WuQ1Bg). +The short version of the talk is that given a query reading an Iceberg table, the Java +library inspects table metadata (*e.g.*, version history, schema, statistics, file layout) to +construct `FileScanTask` objects. These objects describe the low-level operations (*e.g.*, file paths +and byte ranges) needed to read the table and feed data to downstream query operators. + +Comet still relies on Iceberg Java for this planning. Acceleration is possible because Iceberg Rust +has its own `FileScanTask`, so Comet uses it as the common abstraction between the two libraries: it +takes the `FileScanTask` objects that Iceberg Java produced and hands them to Iceberg Rust, which +reads the described files into the in-memory Arrow batches that feed the rest of the plan. + +<figure markdown="span">{ width="750" }<figcaption>Comet translates Iceberg Java's <code>FileScanTask</code> objects into Iceberg Rust's <code>FileScanTask</code> objects.</figcaption></figure> + +To measure Comet's impact on real workloads, the AWS Data on EKS team benchmarked Comet against +Spark alone on the TPC-DS 3 TB workload over Iceberg tables. Comet completed the suite roughly +40% faster (2,803.80s versus 4,665.47s) and accelerated 102 of the 103 TPC-DS queries, with only +a single query regressing. See the +[full benchmark writeup](https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-datafusion-comet-benchmark) +for the complete methodology and per-query results. + +<figure markdown="span">{ width="500" }<figcaption>TPC-DS 3 TB (Iceberg) on AWS EKS: Spark with Comet completes the suite ~40% faster.</figcaption></figure> + +Raw speed only matters if the answers are correct. Comet prioritizes correctness and compatibility +with the libraries it accelerates. In addition to its own exhaustive test suites, Comet goes +further by running Iceberg Java's Spark test suites with Comet enabled as regression tests, +continuously checking the native path against the same corpus that guards the reference +implementation. + +Comet does not yet accelerate all Iceberg table reads. For example, Comet currently falls back to Review Comment: Comet doesn't accelerate writes at all right now. The iceberg-rust write path is even more immature. Maybe I can raise that as future green pastures to use this development approach to bootstrap as well (phrased better). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
