This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6051-667d5435f3aa85be0a1ffd1c8a3dc24885f5c353 in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit d62d3382ac5db67f185dc9f77544a5cefd2002ab Author: Andy Grove <[email protected]> AuthorDate: Sun Sep 20 13:21:58 2026 +0000 docs: recommend setting spark.executor.memoryOverhead alongside off-heap memory (#6051) * docs: recommend setting spark.executor.memoryOverhead alongside off-heap memory * docs: qualify the memoryOverhead recommendation so it cannot shrink the container Setting spark.executor.memoryOverhead replaces Spark's derived value rather than adding to it (ResourceProfile.calculateOverHeadMemory uses getOrElse), so a flat 2g recommendation reduces the allowance for executors large enough that the factor already derives more than that. Describe the derivation accurately, including the 0.4 factor for non-JVM apps on Kubernetes in cluster mode and the configurable floor on Spark 4.0+, scope 2g to small executors, and point larger deployments at spark.executor.memoryOverheadFactor. --- README.md | 3 +- docs/source/index.md | 3 +- docs/source/user-guide/latest/iceberg.md | 6 ++-- docs/source/user-guide/latest/installation.md | 6 ++-- docs/source/user-guide/latest/pyarrow-udfs.md | 1 + docs/source/user-guide/latest/tuning.md | 52 +++++++++++++++++++++++++++ 6 files changed, 65 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 6eccd07816..6c57277eee 100644 --- a/README.md +++ b/README.md @@ -104,7 +104,8 @@ $SPARK_HOME/bin/spark-shell \ --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \ --conf spark.comet.explain.fallback.enabled=true \ --conf spark.memory.offHeap.enabled=true \ - --conf spark.memory.offHeap.size=4g + --conf spark.memory.offHeap.size=4g \ + --conf spark.executor.memoryOverhead=2g ``` For full installation instructions, published jar downloads, and configuration reference, see the diff --git a/docs/source/index.md b/docs/source/index.md index 166fb6884d..facd9b8f0b 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -38,7 +38,8 @@ Runs your existing Spark queries on the Apache DataFusion native engine, no code <span class="term-line term-indent">--conf spark.plugins=org.apache.spark.CometPlugin \</span> <span class="term-line term-indent">--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \</span> <span class="term-line term-indent">--conf spark.memory.offHeap.enabled=true \</span> -<span class="term-line term-indent">--conf spark.memory.offHeap.size=4g</span> +<span class="term-line term-indent">--conf spark.memory.offHeap.size=4g \</span> +<span class="term-line term-indent">--conf spark.executor.memoryOverhead=2g</span> <span class="term-line term-spacer"></span> <span class="term-line term-comment">// Your existing queries now run on the DataFusion native engine</span> <span class="term-line"><span class="term-prompt">scala></span> spark.sql(<span class="term-str">"SELECT category, COUNT(*) FROM events GROUP BY category"</span>).show()</span> diff --git a/docs/source/user-guide/latest/iceberg.md b/docs/source/user-guide/latest/iceberg.md index 0b98a39356..2fed6d3503 100644 --- a/docs/source/user-guide/latest/iceberg.md +++ b/docs/source/user-guide/latest/iceberg.md @@ -45,7 +45,8 @@ $SPARK_HOME/bin/spark-shell \ --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \ --conf spark.comet.explain.fallback.enabled=true \ --conf spark.memory.offHeap.enabled=true \ - --conf spark.memory.offHeap.size=2g + --conf spark.memory.offHeap.size=2g \ + --conf spark.executor.memoryOverhead=2g ``` Catalog configuration is standard Iceberg-on-Spark and independent of Comet. The native reader has been tested with Hadoop, Hive, and REST catalogs. The example above uses a Hadoop catalog. For the full catalog configuration reference, see Iceberg's [Spark catalog configuration](https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs). @@ -138,7 +139,8 @@ $SPARK_HOME/bin/spark-shell \ --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \ --conf spark.comet.explain.fallback.enabled=true \ --conf spark.memory.offHeap.enabled=true \ - --conf spark.memory.offHeap.size=2g + --conf spark.memory.offHeap.size=2g \ + --conf spark.executor.memoryOverhead=2g ``` Note that REST catalogs require explicit namespace creation before creating tables: diff --git a/docs/source/user-guide/latest/installation.md b/docs/source/user-guide/latest/installation.md index c32f35fcf8..9ed0e6bfe3 100644 --- a/docs/source/user-guide/latest/installation.md +++ b/docs/source/user-guide/latest/installation.md @@ -117,7 +117,8 @@ $SPARK_HOME/bin/spark-shell \ --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \ --conf spark.comet.explain.fallback.enabled=true \ --conf spark.memory.offHeap.enabled=true \ - --conf spark.memory.offHeap.size=4g + --conf spark.memory.offHeap.size=4g \ + --conf spark.executor.memoryOverhead=2g ``` <!-- ENDIF --> @@ -170,7 +171,8 @@ $SPARK_HOME/bin/spark-shell \ --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \ --conf spark.comet.explain.fallback.enabled=true \ --conf spark.memory.offHeap.enabled=true \ - --conf spark.memory.offHeap.size=4g + --conf spark.memory.offHeap.size=4g \ + --conf spark.executor.memoryOverhead=2g ``` ### Verify Comet enabled for Spark SQL query diff --git a/docs/source/user-guide/latest/pyarrow-udfs.md b/docs/source/user-guide/latest/pyarrow-udfs.md index a39235314d..5aab9a61b4 100644 --- a/docs/source/user-guide/latest/pyarrow-udfs.md +++ b/docs/source/user-guide/latest/pyarrow-udfs.md @@ -113,6 +113,7 @@ spark = SparkSession.builder \ .config("spark.comet.exec.pyarrowUDF.enabled", "true") \ .config("spark.memory.offHeap.enabled", "true") \ .config("spark.memory.offHeap.size", "2g") \ + .config("spark.executor.memoryOverhead", "2g") \ .getOrCreate() df = spark.read.parquet("data.parquet") diff --git a/docs/source/user-guide/latest/tuning.md b/docs/source/user-guide/latest/tuning.md index 98056ec860..84a3a16ede 100644 --- a/docs/source/user-guide/latest/tuning.md +++ b/docs/source/user-guide/latest/tuning.md @@ -78,7 +78,13 @@ cases, it may be possible to reduce the amount of memory allocated to Spark so t the same or lower than the original configuration. In other cases, enabling Comet may require allocating more memory than before. See the [Determining How Much Memory to Allocate] section for more details. +Comet needs two things configured: an off-heap pool for it to draw its reservations from, and enough executor memory +overhead to cover the part of its footprint that no pool tracks. See [Configuring Comet Memory] and +[Configuring Executor Memory Overhead]. + [Determining How Much Memory to Allocate]: #determining-how-much-memory-to-allocate +[Configuring Comet Memory]: #configuring-comet-memory +[Configuring Executor Memory Overhead]: #configuring-executor-memory-overhead ### Configuring Comet Memory @@ -132,6 +138,52 @@ need to spill or have a single spillable operator. [shuffle]: #shuffle [Advanced Memory Tuning]: #advanced-memory-tuning +### Configuring Executor Memory Overhead + +Enabling off-heap memory is not sufficient on its own. Comet also needs room in +`spark.executor.memoryOverhead`. + +`spark.memory.offHeap.size` is a budget, and the cluster manager already sizes the executor +container to include it, so the memory that Comet's operators explicitly reserve has room. What does +not have room is everything Comet allocates without reserving it — the untracked categories listed +under [Configuring Comet Memory]. Those allocations are made by the Rust global allocator and live +in the native heap, outside the JVM heap and outside Spark's off-heap allocations, and nothing in +the container sizing accounts for them. The same applies to Comet's JVM-side Arrow buffers. + +`spark.executor.memoryOverhead` is the only slack the container has for this, and the JVM's own +non-heap usage — metaspace, code cache, thread stacks, GC structures — is already drawing on it. + +Work out what the executor already gets before choosing a value. When +`spark.executor.memoryOverhead` is unset, Spark derives the overhead as +`max(spark.executor.memoryOverheadFactor * spark.executor.memory, 384 MiB)`. The factor defaults to +`0.1`, except for PySpark and SparkR applications submitted to Kubernetes in cluster mode, where it +defaults to `0.4`. On Spark 4.0 and later the floor is configurable through +`spark.executor.minMemoryOverhead`. Setting `spark.executor.memoryOverhead` **replaces** the derived +value rather than adding to it, so a value below what is derived today shrinks the container instead +of growing it. + +For a small executor, `2g` is a reasonable starting point. A 4 GiB executor derives only 409 MiB, so +this is a real increase: + +``` +spark.executor.memoryOverhead=2g +``` + +A 32 GiB executor, on the other hand, already derives 3276 MiB, and the same setting would take away +1228 MiB. For executors that large, either pick an absolute value above what is derived today, or +raise `spark.executor.memoryOverheadFactor` instead so that the overhead keeps scaling with executor +size: + +``` +spark.executor.memoryOverheadFactor=0.2 +``` + +Raise the value further if executors are killed by the cluster manager (on Kubernetes, +`ExecutorLostFailure` with exit code 137) rather than failing with a task-level out-of-memory error. + +Note that on Kubernetes and YARN the overhead is added to the container size, so raising it reduces +how many executors fit on a node. + ### Determining How Much Memory to Allocate Generally, increasing the amount of memory allocated to Comet will improve query performance by reducing the --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
