This is an automated email from the ASF dual-hosted git repository.
andygrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new d4fbc1663a docs: update Gluten comparison with AWS Labs benchmark and
Comet strengths (#4873)
d4fbc1663a is described below
commit d4fbc1663ad39a415c66090c82081abb40062ef6
Author: Andy Grove <[email protected]>
AuthorDate: Thu Jul 9 10:51:58 2026 -0600
docs: update Gluten comparison with AWS Labs benchmark and Comet strengths
(#4873)
---
docs/source/about/gluten_comparison.md | 112 +++++++++++++++++++++++++++------
1 file changed, 94 insertions(+), 18 deletions(-)
diff --git a/docs/source/about/gluten_comparison.md
b/docs/source/about/gluten_comparison.md
index 3e59feffb7..1d18c5edc4 100644
--- a/docs/source/about/gluten_comparison.md
+++ b/docs/source/about/gluten_comparison.md
@@ -24,8 +24,6 @@ between them. This document is likely biased because the
Comet community maintai
We recommend trying out both Comet and Gluten to see which is the best fit for
your needs.
-This document is based on Comet 0.16.0 and Gluten 1.6.0.
-
## Architecture
Comet and Gluten have very similar architectures. Both are Spark plugins that
translate Spark physical plans to
@@ -53,10 +51,32 @@ the choice of implementation language (Rust vs C++) and
this may be the main fac
choosing a solution. For users wishing to implement UDFs in Rust, Comet would
likely be a better choice. For users
wishing to implement UDFs in C++, Gluten would likely be a better choice.
+The choice of language also has implications for robustness. Rust provides
memory safety guarantees at compile
+time, eliminating entire classes of bugs such as use-after-free, buffer
overflows, and data races that a C++ engine
+must guard against manually at runtime. For a component that runs inside every
Spark executor and processes
+untrusted data, this reduces the risk of memory-corruption crashes and
security vulnerabilities. DataFusion achieves
+this safety without a garbage collector, so there is no additional runtime
overhead compared to C++.
+
If users are just interested in speeding up their existing Spark jobs and do
not need to implement UDFs in native
code, then we suggest benchmarking with both solutions and choosing the
fastest one for your use case.
-
+## Community and Governance
+
+Comet is developed within the Apache DataFusion project, and its native
execution is built directly on the
+DataFusion query engine. DataFusion is a broad, vendor-neutral community
governed by the Apache Software Foundation
+and used by many downstream projects beyond Comet. This means that
engine-level improvements such as new operators,
+optimizer rules, and performance work are shared across the whole DataFusion
ecosystem: work done for other
+DataFusion users benefits Comet, and work done for Comet benefits them in turn.
+
+Velox is also an open-source project with contributors from multiple
organizations, but its development has been
+primarily driven by Meta. Contributing to Velox additionally requires signing
a Contributor License Agreement: the
+Velox [contributing guide] instructs contributors to sign the Meta [CLA]
before their contributions can be accepted.
+Comet and DataFusion follow the standard Apache Software Foundation
contribution model and do not require a per-contributor
+CLA. Users evaluating long-term adoption may want to weigh the governance
model, contribution process, and breadth of
+the community behind each engine alongside the technical differences.
+
+[contributing guide]:
https://github.com/facebookincubator/velox/blob/main/CONTRIBUTING.md
+[CLA]: https://code.facebook.com/cla
## Spark Version Support
@@ -117,22 +137,76 @@ expression dialect (RE2 vs `java.util.regex`), NaN
handling, and timestamp encod
[Velox backend limitations]:
https://apache.github.io/gluten/velox-backend-limitations.html
-## Performance
+## Codegen Dispatch
+
+Comet has a feature called the codegen dispatcher that has no direct
equivalent in Gluten. When an expression has a
+native implementation with known semantic differences from Spark, or has no
native implementation at all, Comet can
+run Spark's own generated code for that expression inside the native pipeline.
Data is passed to the JVM in Arrow
+format, evaluated using Spark's byte-exact logic, and returned to the native
pipeline for the rest of the query.
+
+This matters for two reasons:
+
+- **Correctness by default.** Expressions that are only partially compatible
with Spark (for example, regular
+ expressions, JSON functions, and some datetime and array functions) route
through the codegen dispatcher by
+ default, so they produce results that match Spark exactly. The faster native
path is opt-in per expression for
+ users who accept its differences.
+- **Fewer full fallbacks.** Because the dispatcher keeps evaluation inside the
native pipeline instead of falling
+ back to whole-stage Spark execution, a single unsupported or incompatible
expression does not force an entire
+ query stage back onto vanilla Spark. This keeps more of the plan running
natively than a strict native-or-fallback
+ model would.
-When running a benchmark derived from TPC-H on a single node against local
Parquet files, we see that both Comet
-and Gluten provide an impressive speedup when compared to Spark. Comet
provides a 2.4x speedup compares to a 2.8x speedup
-with Gluten.
+By contrast, Gluten falls back to vanilla Spark for expressions and operators
that its Velox backend does not
+support. See the [Comet Compatibility Guide] for more detail on how the
codegen dispatcher works and which
+expressions use it.
+
+## Regular Expression Compatibility
+
+Regular expressions are a good example of where the codegen dispatcher gives
Comet an architectural advantage.
+Spark's regex semantics come from the JVM's `java.util.regex` engine, which
supports features such as
+backreferences and lookaround. Because Comet's codegen dispatcher runs Spark's
own generated Java code, Comet can
+evaluate `rlike`, `regexp_replace`, `regexp_extract`, `regexp_extract_all`,
`regexp_instr`, and `split` with 100%
+compatibility with Spark by default, including every pattern feature the JVM
engine supports. Comet also offers a
+faster native Rust regex engine as an opt-in per expression for users who can
accept its semantic differences.
+
+Gluten's Velox backend evaluates regular expressions using the C++ RE2 engine.
RE2 uses a different dialect from
+`java.util.regex` and deliberately omits features such as backreferences and
lookaround, so it cannot fully match
+Spark's regex semantics regardless of configuration. This is a fundamental
consequence of using a native C++ regex
+engine rather than the JVM engine, and it is documented in the [Gluten Velox
limitations] page. Comet's codegen
+dispatcher sidesteps this class of incompatibility entirely by keeping Spark's
own engine in the loop.
+
+## Scala and Java UDF Support
+
+The same codegen dispatcher lets Comet run Spark's existing Scala and Java
scalar UDFs inside the native pipeline
+without any rewrite. A UDF's compiled code executes on Arrow data passed from
the native pipeline, and, crucially,
+the presence of a UDF does not force the enclosing operator back to vanilla
Spark: the surrounding native operators
+keep running natively while only the UDF itself is evaluated on the JVM. This
covers functions registered via
+`udf(...)`, `spark.udf.register(...)`, and SQL `CREATE FUNCTION ... AS
'com.example.MyUDF'`, including UDFs over
+complex nested types and UDFs composed with other Catalyst expressions and
higher-order functions.
+
+This means users can adopt Comet without giving up their existing Scala and
Java UDFs, and without rewriting them in
+the engine's native language. Gluten's Velox backend supports native C++ UDFs
but does not run arbitrary existing
+JVM UDFs in the native pipeline, so plans containing them typically fall back
to Spark. See the
+[Comet Scala and Java UDF guide] for the full list of supported and
unsupported cases.
+
+[Comet Scala and Java UDF guide]: /user-guide/latest/scala_java_udfs.md
+
+## Performance
-Gluten is currently faster than Comet for this particular benchmark, but we
expect to close that gap over time.
+AWS Labs published an [independent benchmark] comparing Comet 0.16.0 with
Gluten 1.6.0 on a TPC-DS 3TB workload,
+running on Amazon EKS with Graviton4 instances. The study concluded that the
two accelerators deliver similar
+overall performance, with Gluten finishing roughly 9% faster than Comet across
the full query set.
-Although TPC-H is a good benchmark for operators such as joins and aggregates,
it doesn't necessarily represent
-real-world queries, especially for ETL use cases. For example, there are no
complex types involved and no string
-manipulation, regular expressions, or other advanced expressions. We recommend
running your own benchmarks based
-on your existing Spark jobs.
+[independent benchmark]:
https://awslabs.github.io/data-on-eks/docs/benchmarks/spark-gluten-velox-comet-benchmark
-
+The headline number masks wide per-query variation: some queries were
significantly faster on Gluten (for example,
+large fact-table joins where Gluten uses a shuffled hash join strategy), while
others were significantly faster on
+Comet (for example, CPU-bound scan and aggregate pipelines). We expect Comet
performance to continue improving over
+time and for this gap to close.
-The scripts that were used to generate these results can be found
[here](https://github.com/apache/datafusion-comet/tree/main/benchmarks/tpc).
+Although TPC-DS and TPC-H are good benchmarks for operators such as joins and
aggregates, they don't necessarily
+represent real-world queries, especially for ETL use cases. For example, there
are limited complex types involved
+and little string manipulation, regular expressions, or other advanced
expressions. We recommend running your own
+benchmarks based on your existing Spark jobs.
## Ease of Development & Contributing
@@ -141,7 +215,9 @@ management capabilities vs the complexities around
installing C++ dependencies.
## Summary
-Comet and Gluten are both good solutions for accelerating Spark jobs. Comet
currently has an edge for users on
-Spark 4.0 with ANSI mode enabled, and for users who want a fully native
Iceberg scan path. Gluten currently leads
-on TPC-H performance and offers broader native integration with Delta Lake,
Hudi, and Paimon, plus a second backend
-in ClickHouse. We recommend trying both to see which is the best fit for your
needs.
+Comet and Gluten are both good solutions for accelerating Spark jobs, and
independent benchmarking shows they
+deliver similar overall performance. Comet currently has an edge for users on
Spark 4.0 with ANSI mode enabled, for
+users who want a fully native Iceberg scan path, and through its codegen
dispatcher, which keeps partially compatible
+or unsupported expressions running inside the native pipeline rather than
falling back to Spark. Gluten holds a
+small performance lead in the AWS Labs TPC-DS benchmark and offers broader
native integration with Delta Lake, Hudi,
+and Paimon, plus a second backend in ClickHouse. We recommend trying both to
see which is the best fit for your needs.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
