This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f297e73e2 Publish built docs triggered by
752d4de136e81418f217a133beef51119bf5592e
f297e73e2 is described below
commit f297e73e257863b54f0ec1c2b948c151e5e58e53
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Jul 20 02:54:18 2026 +0000
Publish built docs triggered by 752d4de136e81418f217a133beef51119bf5592e
---
_sources/user-guide/configs.md.txt | 18 +++++------
_sources/user-guide/tuning-guide.md.txt | 49 +++++++++++++++++++++--------
searchindex.js | 2 +-
user-guide/configs.html | 20 +++---------
user-guide/tuning-guide.html | 55 ++++++++++++++++++++-------------
5 files changed, 84 insertions(+), 60 deletions(-)
diff --git a/_sources/user-guide/configs.md.txt
b/_sources/user-guide/configs.md.txt
index b9e44f6dc..83f0351db 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -94,16 +94,14 @@ The following session-level keys control Ballista's shuffle
behavior. See
the [tuning guide](tuning-guide.md#shuffle-implementation) for an
explanation of the sort-based (default) and hash-based shuffle writers.
-| key | type | default |
description
|
-| --------------------------------------------- | ------- | --------- |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| ballista.shuffle.max_concurrent_read_requests | UInt64 | 64 |
Maximum number of concurrent fetch requests the shuffle reader will issue.
|
-| ballista.shuffle.force_remote_read | Boolean | false | Forces
the shuffle reader to fetch every partition through Arrow Flight, even when the
data is local. Intended for testing. |
-| ballista.shuffle.remote_read_prefer_flight | Boolean | false | For
remote reads, prefer the Arrow Flight reader over the block reader. The block
reader is generally faster.
|
-| ballista.shuffle.sort_based.enabled | Boolean | true |
Enables the sort-based shuffle writer (consolidated data file per input
partition with an index, instead of one file per (input partition, output
partition) pair). |
-| ballista.shuffle.sort_based.buffer_size | UInt64 | 1048576 |
Per-partition buffer size in bytes for the sort-based writer (1 MiB default).
|
-| ballista.shuffle.sort_based.memory_limit | UInt64 | 268435456 | Total
in-memory budget across all output-partition buffers for the sort-based writer
(256 MiB default). |
-| ballista.shuffle.sort_based.spill_threshold | Utf8 | "0.8" |
Fraction of `memory_limit` at which the largest buffers spill to disk. Must be
in the range 0–1.
|
-| ballista.shuffle.sort_based.batch_size | UInt64 | 8192 | Target
row count when coalescing buffered batches before they are written or spilled.
|
+| key | type | default
| description
|
+| ------------------------------------------------------- | ------- |
--------- |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| ballista.shuffle.max_concurrent_read_requests | UInt64 | 64
| Maximum number of concurrent fetch requests the shuffle reader will issue.
|
+| ballista.shuffle.force_remote_read | Boolean | false
| Forces the shuffle reader to fetch every partition through Arrow Flight,
even when the data is local. Intended for testing.
|
+| ballista.shuffle.remote_read_prefer_flight | Boolean | false
| For remote reads, prefer the Arrow Flight reader over the block reader. The
block reader is generally faster.
|
+| ballista.shuffle.sort_based.enabled | Boolean | true
| Enables the sort-based shuffle writer (consolidated data file per input
partition with an index, instead of one file per (input partition, output
partition) pair).
|
+| ballista.shuffle.sort_based.batch_size | UInt64 | 8192
| Target row count when coalescing buffered batches before they are written or
spilled.
|
+| ballista.shuffle.sort_based.memory_limit_per_task_bytes | UInt64 |
268435456 | Per-task buffered-bytes budget at which the sort-based writer
spills to disk (256 MiB default), counted independently of the runtime memory
pool. Set to `0` to disable the per-task budget and spill only under
memory-pool pressure — safe only with a bounded memory pool, otherwise the
writer never spills and may run out of memory. |
## Ballista Scheduler Configuration Settings
diff --git a/_sources/user-guide/tuning-guide.md.txt
b/_sources/user-guide/tuning-guide.md.txt
index c2daa312b..b9f3d5d7f 100644
--- a/_sources/user-guide/tuning-guide.md.txt
+++ b/_sources/user-guide/tuning-guide.md.txt
@@ -133,17 +133,42 @@ write latency.
### Sort-based shuffle (default)
-The sort-based writer accumulates batches in a per-output-partition
-in-memory buffer. When the total buffered size crosses a threshold, the
-largest buffers are spilled to disk. After the input stream finishes, the
-remaining in-memory data and any spilled batches are merged and written
-into a single consolidated Arrow IPC file per input partition, alongside
-an index file that lets readers seek directly to a given output partition.
+The sort-based writer accumulates incoming batches in memory, tracking each
+row's output partition. It spills the buffered batches to disk when either of
+two independent triggers fires:
+
+- the runtime memory pool rejects a growth request (the executor is under
+ memory pressure — see [Configuring Executor Memory
Pool](#configuring-executor-memory-pool)), or
+- the per-task buffered-bytes budget
+ (`ballista.shuffle.sort_based.memory_limit_per_task_bytes`) is reached. This
+ budget is counted independently of the memory pool, so it bounds the writer's
+ memory even when the pool is unbounded (the default). Setting it to `0`
+ disables this trigger, leaving memory-pool pressure as the sole spill signal.
+
+**Warning:** setting `memory_limit_per_task_bytes` to `0` while the executor
+uses the default unbounded memory pool disables both spill triggers. The writer
+then buffers the entire task's shuffle output in memory and never spills, which
+can exhaust the executor and cause an out-of-memory failure on large inputs.
+Only use `0` together with a bounded memory pool (see
+[Configuring Executor Memory Pool](#configuring-executor-memory-pool)), so
+pool pressure still forces spilling.
+
+After the input stream finishes, the remaining in-memory data and any spilled
+batches are written into a single consolidated Arrow IPC file per input
+partition, alongside an index file that lets readers seek directly to a given
+output partition.
This produces `2 × N` files instead of `N × M`, coalesces small batches
to a target size before writing, and bounds shuffle memory use via
spilling — at the cost of higher write latency than the hash writer.
+Worst-case sort-shuffle memory per executor is approximately
+`vcores × memory_limit_per_task_bytes`, since one writer task can run per
+core. Lower the per-task budget on memory-constrained executors to spill
+sooner, or raise it to keep more data in memory and reduce spill I/O. Setting
it
+to `0` removes the budget entirely and is safe only with a bounded memory pool
+(see the warning above).
+
### Hash-based shuffle (opt-in)
The hash-based writer hashes each incoming `RecordBatch` and immediately
@@ -164,13 +189,11 @@ let session_config = SessionConfig::new_with_ballista()
The following session-level keys tune its behavior:
-| key | type | default |
description
|
-| ------------------------------------------- | ------- | --------- |
---------------------------------------------------------------------------------------------------------
|
-| ballista.shuffle.sort_based.enabled | Boolean | true | Enables
the sort-based shuffle writer.
|
-| ballista.shuffle.sort_based.buffer_size | UInt64 | 1048576 |
Per-partition buffer size in bytes (1 MiB default).
|
-| ballista.shuffle.sort_based.memory_limit | UInt64 | 268435456 | Total
in-memory budget across all output-partition buffers (256 MiB default).
|
-| ballista.shuffle.sort_based.spill_threshold | Utf8 | "0.8" | Fraction
of `memory_limit` at which the largest buffers begin spilling to disk. Must be
in the range 0–1. |
-| ballista.shuffle.sort_based.batch_size | UInt64 | 8192 | Target
row count when coalescing buffered batches before they are written or spilled.
|
+| key | type | default
| description
|
+| ------------------------------------------------------- | ------- |
--------- |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| ballista.shuffle.sort_based.enabled | Boolean | true
| Enables the sort-based shuffle writer.
|
+| ballista.shuffle.sort_based.batch_size | UInt64 | 8192
| Target row count when coalescing buffered batches before they are written or
spilled.
|
+| ballista.shuffle.sort_based.memory_limit_per_task_bytes | UInt64 |
268435456 | Per-task buffered-bytes budget at which the writer spills to disk
(256 MiB default). Counted independently of the runtime memory pool. Set to `0`
to spill only under memory pressure — safe only with a bounded memory pool,
otherwise the writer never spills and may run out of memory. |
## Adaptive Query Execution (Experimental)
diff --git a/searchindex.js b/searchindex.js
index 8c0089ef2..fb16d1532 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"AQE join behavior
(opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution
(AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution
(Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a
persona":[[25,"adding-a-persona"]],"Aggregate
Functions":[[49,"aggregate-functions"]],"Apache DataFusion
Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"AQE join behavior
(opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution
(AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution
(Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a
persona":[[25,"adding-a-persona"]],"Aggregate
Functions":[[49,"aggregate-functions"]],"Apache DataFusion
Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index ef39c293e..399e49d71 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -484,25 +484,15 @@ explanation of the sort-based (default) and hash-based
shuffle writers.</p>
<td><p>true</p></td>
<td><p>Enables the sort-based shuffle writer (consolidated data file per input
partition with an index, instead of one file per (input partition, output
partition) pair).</p></td>
</tr>
-<tr class="row-even"><td><p>ballista.shuffle.sort_based.buffer_size</p></td>
+<tr class="row-even"><td><p>ballista.shuffle.sort_based.batch_size</p></td>
<td><p>UInt64</p></td>
-<td><p>1048576</p></td>
-<td><p>Per-partition buffer size in bytes for the sort-based writer (1 MiB
default).</p></td>
+<td><p>8192</p></td>
+<td><p>Target row count when coalescing buffered batches before they are
written or spilled.</p></td>
</tr>
-<tr class="row-odd"><td><p>ballista.shuffle.sort_based.memory_limit</p></td>
+<tr
class="row-odd"><td><p>ballista.shuffle.sort_based.memory_limit_per_task_bytes</p></td>
<td><p>UInt64</p></td>
<td><p>268435456</p></td>
-<td><p>Total in-memory budget across all output-partition buffers for the
sort-based writer (256 MiB default).</p></td>
-</tr>
-<tr
class="row-even"><td><p>ballista.shuffle.sort_based.spill_threshold</p></td>
-<td><p>Utf8</p></td>
-<td><p>“0.8”</p></td>
-<td><p>Fraction of <code class="docutils literal notranslate"><span
class="pre">memory_limit</span></code> at which the largest buffers spill to
disk. Must be in the range 0–1.</p></td>
-</tr>
-<tr class="row-odd"><td><p>ballista.shuffle.sort_based.batch_size</p></td>
-<td><p>UInt64</p></td>
-<td><p>8192</p></td>
-<td><p>Target row count when coalescing buffered batches before they are
written or spilled.</p></td>
+<td><p>Per-task buffered-bytes budget at which the sort-based writer spills to
disk (256 MiB default), counted independently of the runtime memory pool. Set
to <code class="docutils literal notranslate"><span class="pre">0</span></code>
to disable the per-task budget and spill only under memory-pool pressure — safe
only with a bounded memory pool, otherwise the writer never spills and may run
out of memory.</p></td>
</tr>
</tbody>
</table>
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index 4ae2b2907..349dcf7c2 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -487,15 +487,38 @@ available, with different trade-offs around file count,
memory use, and
write latency.</p>
<section id="sort-based-shuffle-default">
<h3>Sort-based shuffle (default)<a class="headerlink"
href="#sort-based-shuffle-default" title="Link to this heading">#</a></h3>
-<p>The sort-based writer accumulates batches in a per-output-partition
-in-memory buffer. When the total buffered size crosses a threshold, the
-largest buffers are spilled to disk. After the input stream finishes, the
-remaining in-memory data and any spilled batches are merged and written
-into a single consolidated Arrow IPC file per input partition, alongside
-an index file that lets readers seek directly to a given output partition.</p>
+<p>The sort-based writer accumulates incoming batches in memory, tracking each
+row’s output partition. It spills the buffered batches to disk when either of
+two independent triggers fires:</p>
+<ul class="simple">
+<li><p>the runtime memory pool rejects a growth request (the executor is under
+memory pressure — see <a class="reference internal"
href="#configuring-executor-memory-pool"><span class="xref myst">Configuring
Executor Memory Pool</span></a>), or</p></li>
+<li><p>the per-task buffered-bytes budget
+(<code class="docutils literal notranslate"><span
class="pre">ballista.shuffle.sort_based.memory_limit_per_task_bytes</span></code>)
is reached. This
+budget is counted independently of the memory pool, so it bounds the writer’s
+memory even when the pool is unbounded (the default). Setting it to <code
class="docutils literal notranslate"><span class="pre">0</span></code>
+disables this trigger, leaving memory-pool pressure as the sole spill
signal.</p></li>
+</ul>
+<p><strong>Warning:</strong> setting <code class="docutils literal
notranslate"><span class="pre">memory_limit_per_task_bytes</span></code> to
<code class="docutils literal notranslate"><span class="pre">0</span></code>
while the executor
+uses the default unbounded memory pool disables both spill triggers. The writer
+then buffers the entire task’s shuffle output in memory and never spills, which
+can exhaust the executor and cause an out-of-memory failure on large inputs.
+Only use <code class="docutils literal notranslate"><span
class="pre">0</span></code> together with a bounded memory pool (see
+<a class="reference internal" href="#configuring-executor-memory-pool"><span
class="xref myst">Configuring Executor Memory Pool</span></a>), so
+pool pressure still forces spilling.</p>
+<p>After the input stream finishes, the remaining in-memory data and any
spilled
+batches are written into a single consolidated Arrow IPC file per input
+partition, alongside an index file that lets readers seek directly to a given
+output partition.</p>
<p>This produces <code class="docutils literal notranslate"><span
class="pre">2</span> <span class="pre">×</span> <span
class="pre">N</span></code> files instead of <code class="docutils literal
notranslate"><span class="pre">N</span> <span class="pre">×</span> <span
class="pre">M</span></code>, coalesces small batches
to a target size before writing, and bounds shuffle memory use via
spilling — at the cost of higher write latency than the hash writer.</p>
+<p>Worst-case sort-shuffle memory per executor is approximately
+<code class="docutils literal notranslate"><span class="pre">vcores</span>
<span class="pre">×</span> <span
class="pre">memory_limit_per_task_bytes</span></code>, since one writer task
can run per
+core. Lower the per-task budget on memory-constrained executors to spill
+sooner, or raise it to keep more data in memory and reduce spill I/O. Setting
it
+to <code class="docutils literal notranslate"><span
class="pre">0</span></code> removes the budget entirely and is safe only with a
bounded memory pool
+(see the warning above).</p>
</section>
<section id="hash-based-shuffle-opt-in">
<h3>Hash-based shuffle (opt-in)<a class="headerlink"
href="#hash-based-shuffle-opt-in" title="Link to this heading">#</a></h3>
@@ -527,25 +550,15 @@ and merging of the sort-based writer is unnecessary
overhead:</p>
<td><p>true</p></td>
<td><p>Enables the sort-based shuffle writer.</p></td>
</tr>
-<tr class="row-odd"><td><p>ballista.shuffle.sort_based.buffer_size</p></td>
+<tr class="row-odd"><td><p>ballista.shuffle.sort_based.batch_size</p></td>
<td><p>UInt64</p></td>
-<td><p>1048576</p></td>
-<td><p>Per-partition buffer size in bytes (1 MiB default).</p></td>
+<td><p>8192</p></td>
+<td><p>Target row count when coalescing buffered batches before they are
written or spilled.</p></td>
</tr>
-<tr class="row-even"><td><p>ballista.shuffle.sort_based.memory_limit</p></td>
+<tr
class="row-even"><td><p>ballista.shuffle.sort_based.memory_limit_per_task_bytes</p></td>
<td><p>UInt64</p></td>
<td><p>268435456</p></td>
-<td><p>Total in-memory budget across all output-partition buffers (256 MiB
default).</p></td>
-</tr>
-<tr class="row-odd"><td><p>ballista.shuffle.sort_based.spill_threshold</p></td>
-<td><p>Utf8</p></td>
-<td><p>“0.8”</p></td>
-<td><p>Fraction of <code class="docutils literal notranslate"><span
class="pre">memory_limit</span></code> at which the largest buffers begin
spilling to disk. Must be in the range 0–1.</p></td>
-</tr>
-<tr class="row-even"><td><p>ballista.shuffle.sort_based.batch_size</p></td>
-<td><p>UInt64</p></td>
-<td><p>8192</p></td>
-<td><p>Target row count when coalescing buffered batches before they are
written or spilled.</p></td>
+<td><p>Per-task buffered-bytes budget at which the writer spills to disk (256
MiB default). Counted independently of the runtime memory pool. Set to <code
class="docutils literal notranslate"><span class="pre">0</span></code> to spill
only under memory pressure — safe only with a bounded memory pool, otherwise
the writer never spills and may run out of memory.</p></td>
</tr>
</tbody>
</table>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
