This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 6219c644e Publish built docs triggered by
90f8096ed340d7e035efdc6e12eff566a8111e6c
6219c644e is described below
commit 6219c644ebd41993fcc47f7f836986151d63068f
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Jul 24 11:47:01 2026 +0000
Publish built docs triggered by 90f8096ed340d7e035efdc6e12eff566a8111e6c
---
_sources/user-guide/tuning-guide.md.txt | 43 ++++++++++++++++++++++-----------
searchindex.js | 2 +-
user-guide/tuning-guide.html | 39 ++++++++++++++++++++----------
3 files changed, 56 insertions(+), 28 deletions(-)
diff --git a/_sources/user-guide/tuning-guide.md.txt
b/_sources/user-guide/tuning-guide.md.txt
index 4100e0b5a..6e0ba0c51 100644
--- a/_sources/user-guide/tuning-guide.md.txt
+++ b/_sources/user-guide/tuning-guide.md.txt
@@ -102,10 +102,18 @@ decreasing the vcore count may help.
## Configuring Executor Memory Pool
-By default the executor uses DataFusion's unbounded memory pool, so spillable
-operators (sort, hash join, hash aggregate) grow until the host runs out of
-memory. To bound executor memory and let those operators spill to disk under
-pressure, pass `--memory-pool-size` when starting the executor:
+By default the executor auto-sizes a bounded memory pool from the detected
+host or cgroup memory limit, so spillable operators (sort, hash join, hash
+aggregate) spill to disk under pressure instead of growing until the host
+runs out of memory. The pool is sized to `--memory-pool-fraction` (default
+`0.70`, i.e. 70%) of the detected limit, so headroom is left for untracked
+overhead such as in-flight Arrow batches, shuffle writer buffers, and
+fragmentation. When running under a cgroup-limited container (the common
+Kubernetes/Docker case), the cgroup limit is used in preference to host
+memory when it is the tighter of the two.
+
+To use a specific budget instead of auto-detection, pass `--memory-pool-size`
+when starting the executor:
```sh
ballista-executor --memory-pool-size 8GB --vcores 8
@@ -113,20 +121,27 @@ ballista-executor --memory-pool-size 8GB --vcores 8
The argument accepts human-readable sizes (`8GB`, `512MiB`) or a plain byte
count. SI suffixes (`KB`/`MB`/`GB`) are powers of 10; IEC suffixes
-(`KiB`/`MiB`/`GiB`) are powers of 2.
-
-The total budget is divided equally across the executor's vcores: each task
-receives its own `FairSpillPool` of size `memory_pool_size / vcores`.
-With `--memory-pool-size 8GB --vcores 8`, every task sees a 1 GB
-pool, fully isolated from other tasks. Idle slots do not lend their share to
-busy ones, which keeps task memory predictable at the cost of some unused
-capacity when the executor is under-utilized.
+(`KiB`/`MiB`/`GiB`) are powers of 2. Pass `--memory-pool-size 0` to opt out of
+the pool entirely and run DataFusion's unbounded pool, matching the executor's
+pre-auto-sizing behavior.
+
+Whether the budget comes from auto-detection or `--memory-pool-size`, it is
+divided equally across the executor's vcores: each task receives its own
+`FairSpillPool` of size `memory_pool_size / vcores`. With
+`--memory-pool-size 8GB --vcores 8`, every task sees a 1 GB pool, fully
+isolated from other tasks. Idle slots do not lend their share to busy ones,
+which keeps task memory predictable at the cost of some unused capacity when
+the executor is under-utilized.
The executor refuses to start if the per-task share would round to zero (i.e.
`memory_pool_size < vcores`).
-When `--memory-pool-size` is not set, the executor behaves as before with no
-memory pool installed.
+Use `--memory-pool-fraction` to tune how much of the detected memory limit
+the auto-sized pool uses; it accepts a value in `(0.0, 1.0]` and defaults to
+`0.70`. It is ignored when `--memory-pool-size` is set.
+
+Because operators spill to disk rather than fail, make sure the executor's
+`--work-dir` has enough scratch space to absorb spills under memory pressure.
## Join Strategy
diff --git a/searchindex.js b/searchindex.js
index 188213351..c0fc675de 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"AQE join behavior
(opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution
(AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution
(Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a
persona":[[25,"adding-a-persona"]],"Aggregate
Functions":[[49,"aggregate-functions"]],"Apache DataFusion
Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"AQE join behavior
(opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution
(AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution
(Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a
persona":[[25,"adding-a-persona"]],"Aggregate
Functions":[[49,"aggregate-functions"]],"Apache DataFusion
Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index 8c31faeb6..25b29af04 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -477,26 +477,39 @@ decreasing the vcore count may help.</p>
</section>
<section id="configuring-executor-memory-pool">
<h2>Configuring Executor Memory Pool<a class="headerlink"
href="#configuring-executor-memory-pool" title="Link to this heading">#</a></h2>
-<p>By default the executor uses DataFusion’s unbounded memory pool, so
spillable
-operators (sort, hash join, hash aggregate) grow until the host runs out of
-memory. To bound executor memory and let those operators spill to disk under
-pressure, pass <code class="docutils literal notranslate"><span
class="pre">--memory-pool-size</span></code> when starting the executor:</p>
+<p>By default the executor auto-sizes a bounded memory pool from the detected
+host or cgroup memory limit, so spillable operators (sort, hash join, hash
+aggregate) spill to disk under pressure instead of growing until the host
+runs out of memory. The pool is sized to <code class="docutils literal
notranslate"><span class="pre">--memory-pool-fraction</span></code> (default
+<code class="docutils literal notranslate"><span
class="pre">0.70</span></code>, i.e. 70%) of the detected limit, so headroom is
left for untracked
+overhead such as in-flight Arrow batches, shuffle writer buffers, and
+fragmentation. When running under a cgroup-limited container (the common
+Kubernetes/Docker case), the cgroup limit is used in preference to host
+memory when it is the tighter of the two.</p>
+<p>To use a specific budget instead of auto-detection, pass <code
class="docutils literal notranslate"><span
class="pre">--memory-pool-size</span></code>
+when starting the executor:</p>
<div class="highlight-sh notranslate"><div
class="highlight"><pre><span></span>ballista-executor<span class="w">
</span>--memory-pool-size<span class="w"> </span>8GB<span class="w">
</span>--vcores<span class="w"> </span><span class="m">8</span>
</pre></div>
</div>
<p>The argument accepts human-readable sizes (<code class="docutils literal
notranslate"><span class="pre">8GB</span></code>, <code class="docutils literal
notranslate"><span class="pre">512MiB</span></code>) or a plain byte
count. SI suffixes (<code class="docutils literal notranslate"><span
class="pre">KB</span></code>/<code class="docutils literal notranslate"><span
class="pre">MB</span></code>/<code class="docutils literal notranslate"><span
class="pre">GB</span></code>) are powers of 10; IEC suffixes
-(<code class="docutils literal notranslate"><span
class="pre">KiB</span></code>/<code class="docutils literal notranslate"><span
class="pre">MiB</span></code>/<code class="docutils literal notranslate"><span
class="pre">GiB</span></code>) are powers of 2.</p>
-<p>The total budget is divided equally across the executor’s vcores: each task
-receives its own <code class="docutils literal notranslate"><span
class="pre">FairSpillPool</span></code> of size <code class="docutils literal
notranslate"><span class="pre">memory_pool_size</span> <span
class="pre">/</span> <span class="pre">vcores</span></code>.
-With <code class="docutils literal notranslate"><span
class="pre">--memory-pool-size</span> <span class="pre">8GB</span> <span
class="pre">--vcores</span> <span class="pre">8</span></code>, every task sees
a 1 GB
-pool, fully isolated from other tasks. Idle slots do not lend their share to
-busy ones, which keeps task memory predictable at the cost of some unused
-capacity when the executor is under-utilized.</p>
+(<code class="docutils literal notranslate"><span
class="pre">KiB</span></code>/<code class="docutils literal notranslate"><span
class="pre">MiB</span></code>/<code class="docutils literal notranslate"><span
class="pre">GiB</span></code>) are powers of 2. Pass <code class="docutils
literal notranslate"><span class="pre">--memory-pool-size</span> <span
class="pre">0</span></code> to opt out of
+the pool entirely and run DataFusion’s unbounded pool, matching the executor’s
+pre-auto-sizing behavior.</p>
+<p>Whether the budget comes from auto-detection or <code class="docutils
literal notranslate"><span class="pre">--memory-pool-size</span></code>, it is
+divided equally across the executor’s vcores: each task receives its own
+<code class="docutils literal notranslate"><span
class="pre">FairSpillPool</span></code> of size <code class="docutils literal
notranslate"><span class="pre">memory_pool_size</span> <span
class="pre">/</span> <span class="pre">vcores</span></code>. With
+<code class="docutils literal notranslate"><span
class="pre">--memory-pool-size</span> <span class="pre">8GB</span> <span
class="pre">--vcores</span> <span class="pre">8</span></code>, every task sees
a 1 GB pool, fully
+isolated from other tasks. Idle slots do not lend their share to busy ones,
+which keeps task memory predictable at the cost of some unused capacity when
+the executor is under-utilized.</p>
<p>The executor refuses to start if the per-task share would round to zero
(i.e.
<code class="docutils literal notranslate"><span
class="pre">memory_pool_size</span> <span class="pre"><</span> <span
class="pre">vcores</span></code>).</p>
-<p>When <code class="docutils literal notranslate"><span
class="pre">--memory-pool-size</span></code> is not set, the executor behaves
as before with no
-memory pool installed.</p>
+<p>Use <code class="docutils literal notranslate"><span
class="pre">--memory-pool-fraction</span></code> to tune how much of the
detected memory limit
+the auto-sized pool uses; it accepts a value in <code class="docutils literal
notranslate"><span class="pre">(0.0,</span> <span
class="pre">1.0]</span></code> and defaults to
+<code class="docutils literal notranslate"><span
class="pre">0.70</span></code>. It is ignored when <code class="docutils
literal notranslate"><span class="pre">--memory-pool-size</span></code> is
set.</p>
+<p>Because operators spill to disk rather than fail, make sure the executor’s
+<code class="docutils literal notranslate"><span
class="pre">--work-dir</span></code> has enough scratch space to absorb spills
under memory pressure.</p>
</section>
<section id="join-strategy">
<h2>Join Strategy<a class="headerlink" href="#join-strategy" title="Link to
this heading">#</a></h2>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
