mbutrovich commented on code in PR #17162: URL: https://github.com/apache/iceberg/pull/17162#discussion_r3579431416
########## site/docs/blog/posts/2026-07-13-accelerating-iceberg-rust-development-with-datafusion-comet.md: ########## @@ -0,0 +1,229 @@ +--- +date: 2026-07-13 +title: Accelerating Apache Spark Queries (and Iceberg Rust Development) with Apache DataFusion Comet +slug: accelerating-iceberg-rust-development-with-datafusion-comet # this is the blog url +authors: + - mbutrovich +categories: + - blog +--- + +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +Apache Iceberg's ecosystem spans multiple query engines and language implementations that work +together to give users a consistent experience across the data lakehouse. This post explores one +integration within that ecosystem, [Iceberg Rust](https://github.com/apache/iceberg-rust) and +[Apache DataFusion Comet](https://datafusion.apache.org/comet/), and the two benefits their +relationship brings. +Comet accelerates Apache Spark's reads over Iceberg tables by running them natively through Iceberg +Rust. That same integration turns Iceberg Java's nearly 10,000 Spark tests into a differential-testing +harness whose benefits run both ways: Iceberg Rust gets exercised against a broad corpus of +real-world scenarios, and the comparison has even caught bugs in Iceberg Java. The resulting fixes +land upstream and benefit every project built on these libraries, not just Comet, as the Iceberg and +DataFusion communities build on each other's strengths. + +<!-- more --> + +## Background + +Apache Iceberg provides a universal table format that serves as a foundation for modern data +lakehouse +platforms. With Iceberg, users store their tables with the benefit of being able to access +and modify their data from a number of different query engines. +The +[Iceberg Java repository](https://github.com/apache/iceberg), the *de facto* reference +implementation of the Iceberg specification, ships a mature [Apache Spark](https://spark.apache.org) +integration. Beyond querying their data, teams also use Spark for table maintenance like compaction and +snapshot expiration. +In addition to Java, the Iceberg community maintains a number +of other Iceberg implementations like [C++](https://github.com/apache/iceberg-cpp), +[Go](https://github.com/apache/iceberg-go), [Python](https://github.com/apache/iceberg-python), and +[Rust](https://github.com/apache/iceberg-rust). +These other implementations benefit not only from the Iceberg specification, but also the lessons +learned and design decisions of the Java project's community. The Java repository's extensive +test suites, for instance, include nearly 10,000 correctness tests driven by Spark (as of Iceberg +1.11 with Spark 4.1). Each implementation maintains its own test suite and can look to Iceberg Java +as a reference for both correct behavior and test coverage. None of them, however, can run Java's +tests directly against their own code. + +While Spark remains widely used for working with Iceberg, a number of projects exist to accelerate +its JVM-backed execution. One such solution is Comet, which Apple donated in 2024 as a subproject of +the [Apache DataFusion](https://datafusion.apache.org) query engine. Comet's native execution engine +aims to run CPU-bound jobs faster and IO-bound jobs with fewer resources. As we will see, it does +more than speed up queries: the same design that makes it fast also makes it a tool for accelerating +Iceberg Rust's development. + +## Accelerating Spark Queries with Comet + +Comet builds upon several related Apache projects including DataFusion (for its efficient operator +implementations like joins and aggregations), [Arrow-rs](https://github.com/apache/arrow-rs) +(for its standardized in-memory format and robust Parquet reader), and both the Java and Rust +implementations of Iceberg. To accelerate Spark queries, Comet +intercepts execution +at the physical plan level. After Spark has parsed, planned, and optimized a user's query, +Comet's JVM code runs as one final optimizer rule to convert Spark plan nodes to Comet plan nodes. +These Comet plan nodes have a superpower: they execute in DataFusion's Rust engine over columnar Arrow +data. + +<figure markdown="span">{ width="750" }<figcaption>Comet converts a Spark physical plan into an equivalent DataFusion physical plan.</figcaption></figure> + +So how does Comet use *both* Iceberg libraries to accelerate Spark queries over Iceberg tables? +As previously mentioned, Iceberg provides robust integrations with Spark, enabling users to query +their Iceberg tables regardless of the Spark API they are using (*e.g.*, SQL, Scala, or PySpark). +Iceberg relies on Spark's +[`Data Source v2`](https://spark.apache.org/docs/4.2.0-preview4/sql-data-sources-v2.html) API to Review Comment: I think I found a suitable replacement, thanks for pushing me to take another look at the link! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
