This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 1542a476ff Publish built docs triggered by
d62d3382ac5db67f185dc9f77544a5cefd2002ab
1542a476ff is described below
commit 1542a476ffffb943e7649f09b8406087ab7c60c0
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Sun Sep 20 16:18:29 2026 +0000
Publish built docs triggered by d62d3382ac5db67f185dc9f77544a5cefd2002ab
---
_sources/index.md.txt | 3 +-
_sources/user-guide/latest/iceberg.md.txt | 6 ++-
_sources/user-guide/latest/installation.md.txt | 6 ++-
_sources/user-guide/latest/pyarrow-udfs.md.txt | 1 +
_sources/user-guide/latest/tuning.md.txt | 52 ++++++++++++++++++++++++++
index.html | 3 +-
searchindex.js | 2 +-
user-guide/latest/iceberg.html | 6 ++-
user-guide/latest/installation.html | 6 ++-
user-guide/latest/pyarrow-udfs.html | 1 +
user-guide/latest/tuning.html | 41 ++++++++++++++++++++
11 files changed, 116 insertions(+), 11 deletions(-)
diff --git a/_sources/index.md.txt b/_sources/index.md.txt
index 166fb6884d..facd9b8f0b 100644
--- a/_sources/index.md.txt
+++ b/_sources/index.md.txt
@@ -38,7 +38,8 @@ Runs your existing Spark queries on the Apache DataFusion
native engine, no code
<span class="term-line term-indent">--conf
spark.plugins=org.apache.spark.CometPlugin \</span>
<span class="term-line term-indent">--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
\</span>
<span class="term-line term-indent">--conf spark.memory.offHeap.enabled=true
\</span>
-<span class="term-line term-indent">--conf spark.memory.offHeap.size=4g</span>
+<span class="term-line term-indent">--conf spark.memory.offHeap.size=4g
\</span>
+<span class="term-line term-indent">--conf
spark.executor.memoryOverhead=2g</span>
<span class="term-line term-spacer"></span>
<span class="term-line term-comment">// Your existing queries now run on the
DataFusion native engine</span>
<span class="term-line"><span class="term-prompt">scala></span>
spark.sql(<span class="term-str">"SELECT category, COUNT(*) FROM events GROUP
BY category"</span>).show()</span>
diff --git a/_sources/user-guide/latest/iceberg.md.txt
b/_sources/user-guide/latest/iceberg.md.txt
index ed23bfb918..19f0f80f77 100644
--- a/_sources/user-guide/latest/iceberg.md.txt
+++ b/_sources/user-guide/latest/iceberg.md.txt
@@ -45,7 +45,8 @@ $SPARK_HOME/bin/spark-shell \
--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
\
--conf spark.comet.explain.fallback.enabled=true \
--conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=2g
+ --conf spark.memory.offHeap.size=2g \
+ --conf spark.executor.memoryOverhead=2g
```
Catalog configuration is standard Iceberg-on-Spark and independent of Comet.
The native reader has been tested with Hadoop, Hive, and REST catalogs. The
example above uses a Hadoop catalog. For the full catalog configuration
reference, see Iceberg's [Spark catalog
configuration](https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs).
@@ -138,7 +139,8 @@ $SPARK_HOME/bin/spark-shell \
--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
\
--conf spark.comet.explain.fallback.enabled=true \
--conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=2g
+ --conf spark.memory.offHeap.size=2g \
+ --conf spark.executor.memoryOverhead=2g
```
Note that REST catalogs require explicit namespace creation before creating
tables:
diff --git a/_sources/user-guide/latest/installation.md.txt
b/_sources/user-guide/latest/installation.md.txt
index f7ee94edd8..978a6532bc 100644
--- a/_sources/user-guide/latest/installation.md.txt
+++ b/_sources/user-guide/latest/installation.md.txt
@@ -116,7 +116,8 @@ $SPARK_HOME/bin/spark-shell \
--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
\
--conf spark.comet.explain.fallback.enabled=true \
--conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=4g
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.executor.memoryOverhead=2g
```
@@ -147,7 +148,8 @@ $SPARK_HOME/bin/spark-shell \
--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
\
--conf spark.comet.explain.fallback.enabled=true \
--conf spark.memory.offHeap.enabled=true \
- --conf spark.memory.offHeap.size=4g
+ --conf spark.memory.offHeap.size=4g \
+ --conf spark.executor.memoryOverhead=2g
```
### Verify Comet enabled for Spark SQL query
diff --git a/_sources/user-guide/latest/pyarrow-udfs.md.txt
b/_sources/user-guide/latest/pyarrow-udfs.md.txt
index a39235314d..5aab9a61b4 100644
--- a/_sources/user-guide/latest/pyarrow-udfs.md.txt
+++ b/_sources/user-guide/latest/pyarrow-udfs.md.txt
@@ -113,6 +113,7 @@ spark = SparkSession.builder \
.config("spark.comet.exec.pyarrowUDF.enabled", "true") \
.config("spark.memory.offHeap.enabled", "true") \
.config("spark.memory.offHeap.size", "2g") \
+ .config("spark.executor.memoryOverhead", "2g") \
.getOrCreate()
df = spark.read.parquet("data.parquet")
diff --git a/_sources/user-guide/latest/tuning.md.txt
b/_sources/user-guide/latest/tuning.md.txt
index 98056ec860..84a3a16ede 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -78,7 +78,13 @@ cases, it may be possible to reduce the amount of memory
allocated to Spark so t
the same or lower than the original configuration. In other cases, enabling
Comet may require allocating more memory
than before. See the [Determining How Much Memory to Allocate] section for
more details.
+Comet needs two things configured: an off-heap pool for it to draw its
reservations from, and enough executor memory
+overhead to cover the part of its footprint that no pool tracks. See
[Configuring Comet Memory] and
+[Configuring Executor Memory Overhead].
+
[Determining How Much Memory to Allocate]:
#determining-how-much-memory-to-allocate
+[Configuring Comet Memory]: #configuring-comet-memory
+[Configuring Executor Memory Overhead]: #configuring-executor-memory-overhead
### Configuring Comet Memory
@@ -132,6 +138,52 @@ need to spill or have a single spillable operator.
[shuffle]: #shuffle
[Advanced Memory Tuning]: #advanced-memory-tuning
+### Configuring Executor Memory Overhead
+
+Enabling off-heap memory is not sufficient on its own. Comet also needs room in
+`spark.executor.memoryOverhead`.
+
+`spark.memory.offHeap.size` is a budget, and the cluster manager already sizes
the executor
+container to include it, so the memory that Comet's operators explicitly
reserve has room. What does
+not have room is everything Comet allocates without reserving it — the
untracked categories listed
+under [Configuring Comet Memory]. Those allocations are made by the Rust
global allocator and live
+in the native heap, outside the JVM heap and outside Spark's off-heap
allocations, and nothing in
+the container sizing accounts for them. The same applies to Comet's JVM-side
Arrow buffers.
+
+`spark.executor.memoryOverhead` is the only slack the container has for this,
and the JVM's own
+non-heap usage — metaspace, code cache, thread stacks, GC structures — is
already drawing on it.
+
+Work out what the executor already gets before choosing a value. When
+`spark.executor.memoryOverhead` is unset, Spark derives the overhead as
+`max(spark.executor.memoryOverheadFactor * spark.executor.memory, 384 MiB)`.
The factor defaults to
+`0.1`, except for PySpark and SparkR applications submitted to Kubernetes in
cluster mode, where it
+defaults to `0.4`. On Spark 4.0 and later the floor is configurable through
+`spark.executor.minMemoryOverhead`. Setting `spark.executor.memoryOverhead`
**replaces** the derived
+value rather than adding to it, so a value below what is derived today shrinks
the container instead
+of growing it.
+
+For a small executor, `2g` is a reasonable starting point. A 4 GiB executor
derives only 409 MiB, so
+this is a real increase:
+
+```
+spark.executor.memoryOverhead=2g
+```
+
+A 32 GiB executor, on the other hand, already derives 3276 MiB, and the same
setting would take away
+1228 MiB. For executors that large, either pick an absolute value above what
is derived today, or
+raise `spark.executor.memoryOverheadFactor` instead so that the overhead keeps
scaling with executor
+size:
+
+```
+spark.executor.memoryOverheadFactor=0.2
+```
+
+Raise the value further if executors are killed by the cluster manager (on
Kubernetes,
+`ExecutorLostFailure` with exit code 137) rather than failing with a
task-level out-of-memory error.
+
+Note that on Kubernetes and YARN the overhead is added to the container size,
so raising it reduces
+how many executors fit on a node.
+
### Determining How Much Memory to Allocate
Generally, increasing the amount of memory allocated to Comet will improve
query performance by reducing the
diff --git a/index.html b/index.html
index 80db24bc21..6c19c2c4be 100644
--- a/index.html
+++ b/index.html
@@ -408,7 +408,8 @@ Runs your existing Spark queries on the Apache DataFusion
native engine, no code
<span class="term-line term-indent">--conf
spark.plugins=org.apache.spark.CometPlugin \</span>
<span class="term-line term-indent">--conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
\</span>
<span class="term-line term-indent">--conf spark.memory.offHeap.enabled=true
\</span>
-<span class="term-line term-indent">--conf spark.memory.offHeap.size=4g</span>
+<span class="term-line term-indent">--conf spark.memory.offHeap.size=4g
\</span>
+<span class="term-line term-indent">--conf
spark.executor.memoryOverhead=2g</span>
<span class="term-line term-spacer"></span>
<span class="term-line term-comment">// Your existing queries now run on the
DataFusion native engine</span>
<span class="term-line"><span class="term-prompt">scala></span>
spark.sql(<span class="term-str">"SELECT category, COUNT(*) FROM events GROUP
BY category"</span>).show()</span>
diff --git a/searchindex.js b/searchindex.js
index 5cae72553b..55f2d6fb31 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [73, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/iceberg.html b/user-guide/latest/iceberg.html
index 56dcef4917..916b2fc357 100644
--- a/user-guide/latest/iceberg.html
+++ b/user-guide/latest/iceberg.html
@@ -798,7 +798,8 @@ matching your Spark and Scala versions (Comet also ships
Spark 3.5 / Scala 2.13
<span class="w"> </span>--conf<span class="w">
</span>spark.shuffle.manager<span
class="o">=</span>org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager<span
class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.comet.explain.fallback.enabled<span class="o">=</span><span
class="nb">true</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.enabled<span class="o">=</span><span
class="nb">true</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.size<span class="o">=</span>2g
+<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.size<span class="o">=</span>2g<span class="w">
</span><span class="se">\</span>
+<span class="w"> </span>--conf<span class="w">
</span>spark.executor.memoryOverhead<span class="o">=</span>2g
</pre></div>
</div>
<p>Catalog configuration is standard Iceberg-on-Spark and independent of
Comet. The native reader has been tested with Hadoop, Hive, and REST catalogs.
The example above uses a Hadoop catalog. For the full catalog configuration
reference, see Iceberg’s <a class="reference external"
href="https://iceberg.apache.org/docs/latest/spark-configuration/#catalogs";>Spark
catalog configuration</a>.</p>
@@ -886,7 +887,8 @@ configure Spark to use a REST catalog with Comet’s native
Iceberg scan:</p>
<span class="w"> </span>--conf<span class="w">
</span>spark.shuffle.manager<span
class="o">=</span>org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager<span
class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.comet.explain.fallback.enabled<span class="o">=</span><span
class="nb">true</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.enabled<span class="o">=</span><span
class="nb">true</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.size<span class="o">=</span>2g
+<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.size<span class="o">=</span>2g<span class="w">
</span><span class="se">\</span>
+<span class="w"> </span>--conf<span class="w">
</span>spark.executor.memoryOverhead<span class="o">=</span>2g
</pre></div>
</div>
<p>Note that REST catalogs require explicit namespace creation before creating
tables:</p>
diff --git a/user-guide/latest/installation.html
b/user-guide/latest/installation.html
index f36d55f8ad..57fca0f0d5 100644
--- a/user-guide/latest/installation.html
+++ b/user-guide/latest/installation.html
@@ -919,7 +919,8 @@ and pick the jar with the newest timestamp. Then use it as
described in
<span class="w"> </span>--conf<span class="w">
</span>spark.shuffle.manager<span
class="o">=</span>org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager<span
class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.comet.explain.fallback.enabled<span class="o">=</span><span
class="nb">true</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.enabled<span class="o">=</span><span
class="nb">true</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.size<span class="o">=</span>4g
+<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.size<span class="o">=</span>4g<span class="w">
</span><span class="se">\</span>
+<span class="w"> </span>--conf<span class="w">
</span>spark.executor.memoryOverhead<span class="o">=</span>2g
</pre></div>
</div>
</section>
@@ -945,7 +946,8 @@ source releases, or from the latest code in the GitHub
repository.</p>
<span class="w"> </span>--conf<span class="w">
</span>spark.shuffle.manager<span
class="o">=</span>org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager<span
class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.comet.explain.fallback.enabled<span class="o">=</span><span
class="nb">true</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.enabled<span class="o">=</span><span
class="nb">true</span><span class="w"> </span><span class="se">\</span>
-<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.size<span class="o">=</span>4g
+<span class="w"> </span>--conf<span class="w">
</span>spark.memory.offHeap.size<span class="o">=</span>4g<span class="w">
</span><span class="se">\</span>
+<span class="w"> </span>--conf<span class="w">
</span>spark.executor.memoryOverhead<span class="o">=</span>2g
</pre></div>
</div>
<section id="verify-comet-enabled-for-spark-sql-query">
diff --git a/user-guide/latest/pyarrow-udfs.html
b/user-guide/latest/pyarrow-udfs.html
index f995c7487d..a650f2a195 100644
--- a/user-guide/latest/pyarrow-udfs.html
+++ b/user-guide/latest/pyarrow-udfs.html
@@ -883,6 +883,7 @@ worker. Both confs can be set independently.</p>
<span class="o">.</span><span class="n">config</span><span
class="p">(</span><span
class="s2">"spark.comet.exec.pyarrowUDF.enabled"</span><span class="p">,</span>
<span class="s2">"true"</span><span class="p">)</span> \
<span class="o">.</span><span class="n">config</span><span
class="p">(</span><span class="s2">"spark.memory.offHeap.enabled"</span><span
class="p">,</span> <span class="s2">"true"</span><span class="p">)</span> \
<span class="o">.</span><span class="n">config</span><span
class="p">(</span><span class="s2">"spark.memory.offHeap.size"</span><span
class="p">,</span> <span class="s2">"2g"</span><span class="p">)</span> \
+ <span class="o">.</span><span class="n">config</span><span
class="p">(</span><span class="s2">"spark.executor.memoryOverhead"</span><span
class="p">,</span> <span class="s2">"2g"</span><span class="p">)</span> \
<span class="o">.</span><span class="n">getOrCreate</span><span
class="p">()</span>
<span class="n">df</span> <span class="o">=</span> <span
class="n">spark</span><span class="o">.</span><span class="n">read</span><span
class="o">.</span><span class="n">parquet</span><span class="p">(</span><span
class="s2">"data.parquet"</span><span class="p">)</span>
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index fc3dc849fa..cfbaf18085 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -825,6 +825,9 @@ disabled even when configuration overrides are enabled.</p>
cases, it may be possible to reduce the amount of memory allocated to Spark so
that overall memory allocation is
the same or lower than the original configuration. In other cases, enabling
Comet may require allocating more memory
than before. See the <a class="reference internal"
href="#determining-how-much-memory-to-allocate">Determining How Much Memory to
Allocate</a> section for more details.</p>
+<p>Comet needs two things configured: an off-heap pool for it to draw its
reservations from, and enough executor memory
+overhead to cover the part of its footprint that no pool tracks. See <a
class="reference internal" href="#configuring-comet-memory">Configuring Comet
Memory</a> and
+<a class="reference internal"
href="#configuring-executor-memory-overhead">Configuring Executor Memory
Overhead</a>.</p>
<section id="configuring-comet-memory">
<h3>Configuring Comet Memory<a class="headerlink"
href="#configuring-comet-memory" title="Link to this heading">#</a></h3>
<p>Comet shares an off-heap memory pool with Spark. The size of the pool is
@@ -866,6 +869,43 @@ when there is sufficient memory in order to leave enough
memory for other operat
<p>The <code class="docutils literal notranslate"><span
class="pre">greedy_unified</span></code> pool type implements a greedy
first-come first-serve limit. This pool works well for queries that do not
need to spill or have a single spillable operator.</p>
</section>
+<section id="configuring-executor-memory-overhead">
+<h3>Configuring Executor Memory Overhead<a class="headerlink"
href="#configuring-executor-memory-overhead" title="Link to this
heading">#</a></h3>
+<p>Enabling off-heap memory is not sufficient on its own. Comet also needs
room in
+<code class="docutils literal notranslate"><span
class="pre">spark.executor.memoryOverhead</span></code>.</p>
+<p><code class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code> is a budget, and the
cluster manager already sizes the executor
+container to include it, so the memory that Comet’s operators explicitly
reserve has room. What does
+not have room is everything Comet allocates without reserving it — the
untracked categories listed
+under <a class="reference internal"
href="#configuring-comet-memory">Configuring Comet Memory</a>. Those
allocations are made by the Rust global allocator and live
+in the native heap, outside the JVM heap and outside Spark’s off-heap
allocations, and nothing in
+the container sizing accounts for them. The same applies to Comet’s JVM-side
Arrow buffers.</p>
+<p><code class="docutils literal notranslate"><span
class="pre">spark.executor.memoryOverhead</span></code> is the only slack the
container has for this, and the JVM’s own
+non-heap usage — metaspace, code cache, thread stacks, GC structures — is
already drawing on it.</p>
+<p>Work out what the executor already gets before choosing a value. When
+<code class="docutils literal notranslate"><span
class="pre">spark.executor.memoryOverhead</span></code> is unset, Spark derives
the overhead as
+<code class="docutils literal notranslate"><span
class="pre">max(spark.executor.memoryOverheadFactor</span> <span
class="pre">*</span> <span class="pre">spark.executor.memory,</span> <span
class="pre">384</span> <span class="pre">MiB)</span></code>. The factor
defaults to
+<code class="docutils literal notranslate"><span
class="pre">0.1</span></code>, except for PySpark and SparkR applications
submitted to Kubernetes in cluster mode, where it
+defaults to <code class="docutils literal notranslate"><span
class="pre">0.4</span></code>. On Spark 4.0 and later the floor is configurable
through
+<code class="docutils literal notranslate"><span
class="pre">spark.executor.minMemoryOverhead</span></code>. Setting <code
class="docutils literal notranslate"><span
class="pre">spark.executor.memoryOverhead</span></code>
<strong>replaces</strong> the derived
+value rather than adding to it, so a value below what is derived today shrinks
the container instead
+of growing it.</p>
+<p>For a small executor, <code class="docutils literal notranslate"><span
class="pre">2g</span></code> is a reasonable starting point. A 4 GiB executor
derives only 409 MiB, so
+this is a real increase:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">spark</span><span
class="o">.</span><span class="n">executor</span><span class="o">.</span><span
class="n">memoryOverhead</span><span class="o">=</span><span
class="mi">2</span><span class="n">g</span>
+</pre></div>
+</div>
+<p>A 32 GiB executor, on the other hand, already derives 3276 MiB, and the
same setting would take away
+1228 MiB. For executors that large, either pick an absolute value above what
is derived today, or
+raise <code class="docutils literal notranslate"><span
class="pre">spark.executor.memoryOverheadFactor</span></code> instead so that
the overhead keeps scaling with executor
+size:</p>
+<div class="highlight-default notranslate"><div
class="highlight"><pre><span></span><span class="n">spark</span><span
class="o">.</span><span class="n">executor</span><span class="o">.</span><span
class="n">memoryOverheadFactor</span><span class="o">=</span><span
class="mf">0.2</span>
+</pre></div>
+</div>
+<p>Raise the value further if executors are killed by the cluster manager (on
Kubernetes,
+<code class="docutils literal notranslate"><span
class="pre">ExecutorLostFailure</span></code> with exit code 137) rather than
failing with a task-level out-of-memory error.</p>
+<p>Note that on Kubernetes and YARN the overhead is added to the container
size, so raising it reduces
+how many executors fit on a node.</p>
+</section>
<section id="determining-how-much-memory-to-allocate">
<h3>Determining How Much Memory to Allocate<a class="headerlink"
href="#determining-how-much-memory-to-allocate" title="Link to this
heading">#</a></h3>
<p>Generally, increasing the amount of memory allocated to Comet will improve
query performance by reducing the
@@ -1220,6 +1260,7 @@ fallback to Spark works, see <a class="reference
internal" href="understanding-c
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#adaptive-partial-aggregation">Adaptive Partial Aggregation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#memory-tuning">Memory Tuning</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#configuring-comet-memory">Configuring Comet Memory</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#configuring-executor-memory-overhead">Configuring Executor Memory
Overhead</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#determining-how-much-memory-to-allocate">Determining How Much Memory to
Allocate</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#batch-size">Batch Size</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#limiting-spill-disk-usage">Limiting Spill Disk Usage</a></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
