This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a22d29bc6 Publish built docs triggered by
6540d0851d7604fd759b41594d9652e5a4432489
a22d29bc6 is described below
commit a22d29bc62a0c6316b9861c4eb99d18174ff0d19
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue Jul 21 17:59:17 2026 +0000
Publish built docs triggered by 6540d0851d7604fd759b41594d9652e5a4432489
---
_sources/user-guide/configs.md.txt | 3 +--
_sources/user-guide/tuning-guide.md.txt | 38 ++++++++----------------------
searchindex.js | 2 +-
user-guide/configs.html | 11 +++------
user-guide/tuning-guide.html | 41 ++++++++-------------------------
5 files changed, 25 insertions(+), 70 deletions(-)
diff --git a/_sources/user-guide/configs.md.txt
b/_sources/user-guide/configs.md.txt
index 83f0351db..8bc7276f4 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -92,14 +92,13 @@ let expected = [
The following session-level keys control Ballista's shuffle behavior. See
the [tuning guide](tuning-guide.md#shuffle-implementation) for an
-explanation of the sort-based (default) and hash-based shuffle writers.
+explanation of the sort-based shuffle writer.
| key | type | default
| description
|
| ------------------------------------------------------- | ------- |
--------- |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
| ballista.shuffle.max_concurrent_read_requests | UInt64 | 64
| Maximum number of concurrent fetch requests the shuffle reader will issue.
|
| ballista.shuffle.force_remote_read | Boolean | false
| Forces the shuffle reader to fetch every partition through Arrow Flight,
even when the data is local. Intended for testing.
|
| ballista.shuffle.remote_read_prefer_flight | Boolean | false
| For remote reads, prefer the Arrow Flight reader over the block reader. The
block reader is generally faster.
|
-| ballista.shuffle.sort_based.enabled | Boolean | true
| Enables the sort-based shuffle writer (consolidated data file per input
partition with an index, instead of one file per (input partition, output
partition) pair).
|
| ballista.shuffle.sort_based.batch_size | UInt64 | 8192
| Target row count when coalescing buffered batches before they are written or
spilled.
|
| ballista.shuffle.sort_based.memory_limit_per_task_bytes | UInt64 |
268435456 | Per-task buffered-bytes budget at which the sort-based writer
spills to disk (256 MiB default), counted independently of the runtime memory
pool. Set to `0` to disable the per-task budget and spill only under
memory-pool pressure — safe only with a bounded memory pool, otherwise the
writer never spills and may run out of memory. |
diff --git a/_sources/user-guide/tuning-guide.md.txt
b/_sources/user-guide/tuning-guide.md.txt
index b9f3d5d7f..5391c399d 100644
--- a/_sources/user-guide/tuning-guide.md.txt
+++ b/_sources/user-guide/tuning-guide.md.txt
@@ -127,11 +127,12 @@ scheduler or executors.
Ballista exchanges data between query stages by writing the output of each
upstream task to local files, which downstream tasks read either from disk
-(when co-located) or over Arrow Flight. Two shuffle implementations are
-available, with different trade-offs around file count, memory use, and
-write latency.
+(when co-located) or over Arrow Flight. Ballista uses a sort-based shuffle
+writer that bounds file count and memory use. Single-partition stages (a
+query's final output, coalesce, and broadcast-build stages) are written
+directly to one file.
-### Sort-based shuffle (default)
+### Sort-based shuffle
The sort-based writer accumulates incoming batches in memory, tracking each
row's output partition. It spills the buffered batches to disk when either of
@@ -160,7 +161,7 @@ output partition.
This produces `2 × N` files instead of `N × M`, coalesces small batches
to a target size before writing, and bounds shuffle memory use via
-spilling — at the cost of higher write latency than the hash writer.
+spilling.
Worst-case sort-shuffle memory per executor is approximately
`vcores × memory_limit_per_task_bytes`, since one writer task can run per
@@ -169,31 +170,12 @@ sooner, or raise it to keep more data in memory and
reduce spill I/O. Setting it
to `0` removes the budget entirely and is safe only with a bounded memory pool
(see the warning above).
-### Hash-based shuffle (opt-in)
-
-The hash-based writer hashes each incoming `RecordBatch` and immediately
-encodes the per-partition slices to Arrow IPC, streaming them into one
-file per `(input_partition, output_partition)` pair. Nothing is buffered
-in memory across batches.
-
-This is simple and low latency, but for `N` input partitions and `M`
-output partitions it produces `N × M` files. Wide shuffles can therefore
-generate a large number of small files. Consider switching to the
-hash-based writer for narrow shuffles where the additional buffering
-and merging of the sort-based writer is unnecessary overhead:
-
-```rust
-let session_config = SessionConfig::new_with_ballista()
- .set_bool("ballista.shuffle.sort_based.enabled", false);
-```
-
The following session-level keys tune its behavior:
-| key | type | default
| description
|
-| ------------------------------------------------------- | ------- |
--------- |
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
-| ballista.shuffle.sort_based.enabled | Boolean | true
| Enables the sort-based shuffle writer.
|
-| ballista.shuffle.sort_based.batch_size | UInt64 | 8192
| Target row count when coalescing buffered batches before they are written or
spilled.
|
-| ballista.shuffle.sort_based.memory_limit_per_task_bytes | UInt64 |
268435456 | Per-task buffered-bytes budget at which the writer spills to disk
(256 MiB default). Counted independently of the runtime memory pool. Set to `0`
to spill only under memory pressure — safe only with a bounded memory pool,
otherwise the writer never spills and may run out of memory. |
+| key | type | default
| description
|
+| ------------------------------------------------------- | ------ | ---------
|
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| ballista.shuffle.sort_based.batch_size | UInt64 | 8192
| Target row count when coalescing buffered batches before they are written or
spilled.
|
+| ballista.shuffle.sort_based.memory_limit_per_task_bytes | UInt64 | 268435456
| Per-task buffered-bytes budget at which the writer spills to disk (256 MiB
default). Counted independently of the runtime memory pool. Set to `0` to spill
only under memory pressure — safe only with a bounded memory pool, otherwise
the writer never spills and may run out of memory. |
## Adaptive Query Execution (Experimental)
diff --git a/searchindex.js b/searchindex.js
index 99301185b..47c61d5e4 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"AQE join behavior
(opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution
(AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution
(Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a
persona":[[25,"adding-a-persona"]],"Aggregate
Functions":[[49,"aggregate-functions"]],"Apache DataFusion
Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"AQE join behavior
(opt-in)":[[27,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution
(AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution
(Experimental)":[[50,"adaptive-query-execution-experimental"]],"Adding a
persona":[[25,"adding-a-persona"]],"Aggregate
Functions":[[49,"aggregate-functions"]],"Apache DataFusion
Ballista":[[26,null]],"Apache DataFusion Ballista 0.10.0
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0 Changelog" [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 399e49d71..20a94b68d 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -454,7 +454,7 @@
<h2>Shuffle Settings<a class="headerlink" href="#shuffle-settings" title="Link
to this heading">#</a></h2>
<p>The following session-level keys control Ballista’s shuffle behavior. See
the <a class="reference internal"
href="tuning-guide.html#shuffle-implementation"><span class="std
std-ref">tuning guide</span></a> for an
-explanation of the sort-based (default) and hash-based shuffle writers.</p>
+explanation of the sort-based shuffle writer.</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
<tr class="row-odd"><th class="head"><p>key</p></th>
@@ -479,17 +479,12 @@ explanation of the sort-based (default) and hash-based
shuffle writers.</p>
<td><p>false</p></td>
<td><p>For remote reads, prefer the Arrow Flight reader over the block reader.
The block reader is generally faster.</p></td>
</tr>
-<tr class="row-odd"><td><p>ballista.shuffle.sort_based.enabled</p></td>
-<td><p>Boolean</p></td>
-<td><p>true</p></td>
-<td><p>Enables the sort-based shuffle writer (consolidated data file per input
partition with an index, instead of one file per (input partition, output
partition) pair).</p></td>
-</tr>
-<tr class="row-even"><td><p>ballista.shuffle.sort_based.batch_size</p></td>
+<tr class="row-odd"><td><p>ballista.shuffle.sort_based.batch_size</p></td>
<td><p>UInt64</p></td>
<td><p>8192</p></td>
<td><p>Target row count when coalescing buffered batches before they are
written or spilled.</p></td>
</tr>
-<tr
class="row-odd"><td><p>ballista.shuffle.sort_based.memory_limit_per_task_bytes</p></td>
+<tr
class="row-even"><td><p>ballista.shuffle.sort_based.memory_limit_per_task_bytes</p></td>
<td><p>UInt64</p></td>
<td><p>268435456</p></td>
<td><p>Per-task buffered-bytes budget at which the sort-based writer spills to
disk (256 MiB default), counted independently of the runtime memory pool. Set
to <code class="docutils literal notranslate"><span class="pre">0</span></code>
to disable the per-task budget and spill only under memory-pool pressure — safe
only with a bounded memory pool, otherwise the writer never spills and may run
out of memory.</p></td>
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index 349dcf7c2..295373b9c 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -482,11 +482,12 @@ scheduler or executors.</p>
<h2>Shuffle Implementation<a class="headerlink" href="#shuffle-implementation"
title="Link to this heading">#</a></h2>
<p>Ballista exchanges data between query stages by writing the output of each
upstream task to local files, which downstream tasks read either from disk
-(when co-located) or over Arrow Flight. Two shuffle implementations are
-available, with different trade-offs around file count, memory use, and
-write latency.</p>
-<section id="sort-based-shuffle-default">
-<h3>Sort-based shuffle (default)<a class="headerlink"
href="#sort-based-shuffle-default" title="Link to this heading">#</a></h3>
+(when co-located) or over Arrow Flight. Ballista uses a sort-based shuffle
+writer that bounds file count and memory use. Single-partition stages (a
+query’s final output, coalesce, and broadcast-build stages) are written
+directly to one file.</p>
+<section id="sort-based-shuffle">
+<h3>Sort-based shuffle<a class="headerlink" href="#sort-based-shuffle"
title="Link to this heading">#</a></h3>
<p>The sort-based writer accumulates incoming batches in memory, tracking each
row’s output partition. It spills the buffered batches to disk when either of
two independent triggers fires:</p>
@@ -512,29 +513,13 @@ partition, alongside an index file that lets readers seek
directly to a given
output partition.</p>
<p>This produces <code class="docutils literal notranslate"><span
class="pre">2</span> <span class="pre">×</span> <span
class="pre">N</span></code> files instead of <code class="docutils literal
notranslate"><span class="pre">N</span> <span class="pre">×</span> <span
class="pre">M</span></code>, coalesces small batches
to a target size before writing, and bounds shuffle memory use via
-spilling — at the cost of higher write latency than the hash writer.</p>
+spilling.</p>
<p>Worst-case sort-shuffle memory per executor is approximately
<code class="docutils literal notranslate"><span class="pre">vcores</span>
<span class="pre">×</span> <span
class="pre">memory_limit_per_task_bytes</span></code>, since one writer task
can run per
core. Lower the per-task budget on memory-constrained executors to spill
sooner, or raise it to keep more data in memory and reduce spill I/O. Setting
it
to <code class="docutils literal notranslate"><span
class="pre">0</span></code> removes the budget entirely and is safe only with a
bounded memory pool
(see the warning above).</p>
-</section>
-<section id="hash-based-shuffle-opt-in">
-<h3>Hash-based shuffle (opt-in)<a class="headerlink"
href="#hash-based-shuffle-opt-in" title="Link to this heading">#</a></h3>
-<p>The hash-based writer hashes each incoming <code class="docutils literal
notranslate"><span class="pre">RecordBatch</span></code> and immediately
-encodes the per-partition slices to Arrow IPC, streaming them into one
-file per <code class="docutils literal notranslate"><span
class="pre">(input_partition,</span> <span
class="pre">output_partition)</span></code> pair. Nothing is buffered
-in memory across batches.</p>
-<p>This is simple and low latency, but for <code class="docutils literal
notranslate"><span class="pre">N</span></code> input partitions and <code
class="docutils literal notranslate"><span class="pre">M</span></code>
-output partitions it produces <code class="docutils literal notranslate"><span
class="pre">N</span> <span class="pre">×</span> <span
class="pre">M</span></code> files. Wide shuffles can therefore
-generate a large number of small files. Consider switching to the
-hash-based writer for narrow shuffles where the additional buffering
-and merging of the sort-based writer is unnecessary overhead:</p>
-<div class="highlight-rust notranslate"><div
class="highlight"><pre><span></span><span class="kd">let</span><span class="w">
</span><span class="n">session_config</span><span class="w"> </span><span
class="o">=</span><span class="w"> </span><span
class="n">SessionConfig</span><span class="p">::</span><span
class="n">new_with_ballista</span><span class="p">()</span>
-<span class="w"> </span><span class="p">.</span><span
class="n">set_bool</span><span class="p">(</span><span
class="s">"ballista.shuffle.sort_based.enabled"</span><span
class="p">,</span><span class="w"> </span><span class="kc">false</span><span
class="p">);</span>
-</pre></div>
-</div>
<p>The following session-level keys tune its behavior:</p>
<div class="pst-scrollable-table-container"><table class="table">
<thead>
@@ -545,17 +530,12 @@ and merging of the sort-based writer is unnecessary
overhead:</p>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p>ballista.shuffle.sort_based.enabled</p></td>
-<td><p>Boolean</p></td>
-<td><p>true</p></td>
-<td><p>Enables the sort-based shuffle writer.</p></td>
-</tr>
-<tr class="row-odd"><td><p>ballista.shuffle.sort_based.batch_size</p></td>
+<tr class="row-even"><td><p>ballista.shuffle.sort_based.batch_size</p></td>
<td><p>UInt64</p></td>
<td><p>8192</p></td>
<td><p>Target row count when coalescing buffered batches before they are
written or spilled.</p></td>
</tr>
-<tr
class="row-even"><td><p>ballista.shuffle.sort_based.memory_limit_per_task_bytes</p></td>
+<tr
class="row-odd"><td><p>ballista.shuffle.sort_based.memory_limit_per_task_bytes</p></td>
<td><p>UInt64</p></td>
<td><p>268435456</p></td>
<td><p>Per-task buffered-bytes budget at which the writer spills to disk (256
MiB default). Counted independently of the runtime memory pool. Set to <code
class="docutils literal notranslate"><span class="pre">0</span></code> to spill
only under memory pressure — safe only with a bounded memory pool, otherwise
the writer never spills and may run out of memory.</p></td>
@@ -724,8 +704,7 @@ processes. The default is <code class="docutils literal
notranslate"><span class
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#configuring-executor-memory-pool">Configuring Executor Memory
Pool</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#join-strategy">Join Strategy</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#shuffle-implementation">Shuffle Implementation</a><ul class="nav
section-nav flex-column">
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#sort-based-shuffle-default">Sort-based shuffle (default)</a></li>
-<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#hash-based-shuffle-opt-in">Hash-based shuffle (opt-in)</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#sort-based-shuffle">Sort-based shuffle</a></li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#adaptive-query-execution-experimental">Adaptive Query Execution
(Experimental)</a><ul class="nav section-nav flex-column">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
