This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 2c07fae8a Publish built docs triggered by
ac41a22245b563ec7bfba53e2d4b0783aa5c5318
2c07fae8a is described below
commit 2c07fae8ad7aa7f29bdc11c29c975d7d451fb614
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Sat Jul 11 13:48:26 2026 +0000
Publish built docs triggered by ac41a22245b563ec7bfba53e2d4b0783aa5c5318
---
_sources/contributors-guide/architecture.md.txt | 62 +++++++++++++++++++++++++
_sources/user-guide/tuning-guide.md.txt | 12 ++---
contributors-guide/architecture.html | 62 +++++++++++++++++++++++++
searchindex.js | 2 +-
user-guide/tuning-guide.html | 12 ++---
5 files changed, 137 insertions(+), 13 deletions(-)
diff --git a/_sources/contributors-guide/architecture.md.txt
b/_sources/contributors-guide/architecture.md.txt
index b25321d04..8a7ff1063 100644
--- a/_sources/contributors-guide/architecture.md.txt
+++ b/_sources/contributors-guide/architecture.md.txt
@@ -199,3 +199,65 @@ and [ShuffleReaderExec] operators.
[shufflewriterexec]:
https://github.com/apache/datafusion-ballista/blob/main/ballista/core/src/execution_plans/shuffle_writer.rs
[shufflereaderexec]:
https://github.com/apache/datafusion-ballista/blob/main/ballista/core/src/execution_plans/shuffle_reader.rs
+
+## Adaptive Query Execution (AQE)
+
+The scheduling described above is _static_: the full set of query stages is
computed once, at job
+submission time, from the plan-time cost estimates. Adaptive Query Execution
(AQE) is an experimental
+alternative in which the stage DAG is built _incrementally_, re-optimizing the
remaining plan using the
+exact row counts and byte sizes observed as each shuffle stage completes. It
is disabled by default and
+enabled with the `ballista.planner.adaptive.enabled` configuration setting.
+
+### Static vs adaptive execution graphs
+
+Job scheduling revolves around an `ExecutionGraph`, which translates a
physical plan into a set of stages.
+There are two implementations, selected by configuration when a job is
submitted:
+
+- **`StaticExecutionGraph`** — the default. All stages are planned up front by
the `DistributedPlanner`,
+ which returns a static list of stages.
+- **`AdaptiveExecutionGraph`** — the adaptive counterpart. It is driven by the
`AdaptivePlanner`, which runs
+ a set of pluggable physical optimizer rules after each stage completes and
returns only the stages that are
+ currently runnable. Because decisions are deferred, stages come back already
resolved, so the
+ `UnResolved` stage state used by the static path is not needed here.
+
+Keeping the two graphs side by side lets the adaptive implementation mature
without destabilizing the static
+path.
+
+### How re-optimization works
+
+As each shuffle stage finishes, its exact output statistics are attached to
the remaining plan and the
+optimizer rules are re-run. This is the key difference from DataFusion's
normal single-shot physical
+optimization, where each rule runs once per plan. Running the full rule set
repeatedly means the rules must be
+**idempotent** — otherwise they would keep inserting redundant exec nodes on
each pass.
+
+Two adaptive optimizations are implemented today:
+
+- **Join reordering** — runtime row counts are used so the smaller side drives
the join.
+- **Empty stage elimination** — when a completed stage produces zero rows, its
downstream exchange is
+ replaced with an empty execution node and emptiness is propagated up the
plan so dependent stages are
+ skipped entirely.
+
+### Code layout
+
+The adaptive scheduler code lives under [`ballista/scheduler/src/state/aqe`],
with the
+`AdaptiveExecutionGraph` in `mod.rs`, the planner in `planner.rs`, and the
optimizer rules in
+`optimizer_rule.rs`.
+
+### Current limitations
+
+The adaptive path currently covers the happy path only. Known gaps, each with
a tracking issue:
+
+- Executor failure handling on the AQE path ([#1986])
+- Dynamic coalescing of shuffle partitions ([#1987])
+- Switching from hash join to sort-merge join based on runtime statistics
([#1988])
+- Switching from streaming aggregation to hash aggregation based on runtime
statistics ([#1989])
+
+Design and progress are tracked in the AQE epic ([#1359]). See also the
+[AQE section of the tuning guide](../user-guide/tuning-guide.md) for the
user-facing description.
+
+[`ballista/scheduler/src/state/aqe`]:
https://github.com/apache/datafusion-ballista/tree/main/ballista/scheduler/src/state/aqe
+[#1359]: https://github.com/apache/datafusion-ballista/issues/1359
+[#1986]: https://github.com/apache/datafusion-ballista/issues/1986
+[#1987]: https://github.com/apache/datafusion-ballista/issues/1987
+[#1988]: https://github.com/apache/datafusion-ballista/issues/1988
+[#1989]: https://github.com/apache/datafusion-ballista/issues/1989
diff --git a/_sources/user-guide/tuning-guide.md.txt
b/_sources/user-guide/tuning-guide.md.txt
index 7b6f38130..b7906b166 100644
--- a/_sources/user-guide/tuning-guide.md.txt
+++ b/_sources/user-guide/tuning-guide.md.txt
@@ -218,14 +218,14 @@ implemented:
The implementation covers the happy path only. The following are known to be
missing or incomplete:
-- Executor failure handling on the AQE path
-- Dynamic coalescing of shuffle partitions
-- Switching from hash join to sort-merge join based on runtime statistics
-- Switching from streaming aggregation to hash aggregation based on runtime
statistics
+- Executor failure handling on the AQE path
([#1986](https://github.com/apache/datafusion-ballista/issues/1986))
+- Dynamic coalescing of shuffle partitions
([#1987](https://github.com/apache/datafusion-ballista/issues/1987))
+- Switching from hash join to sort-merge join based on runtime statistics
([#1988](https://github.com/apache/datafusion-ballista/issues/1988))
+- Switching from streaming aggregation to hash aggregation based on runtime
statistics ([#1989](https://github.com/apache/datafusion-ballista/issues/1989))
Until these gaps are closed, AQE should be used for testing and experimentation
-rather than production workloads. See [issue
#387](https://github.com/apache/datafusion-ballista/issues/387)
-for the tracking issue and ongoing work.
+rather than production workloads. See [issue
#1359](https://github.com/apache/datafusion-ballista/issues/1359)
+for the tracking epic and ongoing work.
## Push-based vs Pull-based Task Scheduling
diff --git a/contributors-guide/architecture.html
b/contributors-guide/architecture.html
index f5148806d..9027a56dd 100644
--- a/contributors-guide/architecture.html
+++ b/contributors-guide/architecture.html
@@ -549,6 +549,61 @@ stage. This mechanism is known as an Exchange or a
Shuffle. The logic for this c
and <a class="reference external"
href="https://github.com/apache/datafusion-ballista/blob/main/ballista/core/src/execution_plans/shuffle_reader.rs";>ShuffleReaderExec</a>
operators.</p>
</section>
</section>
+<section id="adaptive-query-execution-aqe">
+<h2>Adaptive Query Execution (AQE)<a class="headerlink"
href="#adaptive-query-execution-aqe" title="Link to this heading">#</a></h2>
+<p>The scheduling described above is <em>static</em>: the full set of query
stages is computed once, at job
+submission time, from the plan-time cost estimates. Adaptive Query Execution
(AQE) is an experimental
+alternative in which the stage DAG is built <em>incrementally</em>,
re-optimizing the remaining plan using the
+exact row counts and byte sizes observed as each shuffle stage completes. It
is disabled by default and
+enabled with the <code class="docutils literal notranslate"><span
class="pre">ballista.planner.adaptive.enabled</span></code> configuration
setting.</p>
+<section id="static-vs-adaptive-execution-graphs">
+<h3>Static vs adaptive execution graphs<a class="headerlink"
href="#static-vs-adaptive-execution-graphs" title="Link to this
heading">#</a></h3>
+<p>Job scheduling revolves around an <code class="docutils literal
notranslate"><span class="pre">ExecutionGraph</span></code>, which translates a
physical plan into a set of stages.
+There are two implementations, selected by configuration when a job is
submitted:</p>
+<ul class="simple">
+<li><p><strong><code class="docutils literal notranslate"><span
class="pre">StaticExecutionGraph</span></code></strong> — the default. All
stages are planned up front by the <code class="docutils literal
notranslate"><span class="pre">DistributedPlanner</span></code>,
+which returns a static list of stages.</p></li>
+<li><p><strong><code class="docutils literal notranslate"><span
class="pre">AdaptiveExecutionGraph</span></code></strong> — the adaptive
counterpart. It is driven by the <code class="docutils literal
notranslate"><span class="pre">AdaptivePlanner</span></code>, which runs
+a set of pluggable physical optimizer rules after each stage completes and
returns only the stages that are
+currently runnable. Because decisions are deferred, stages come back already
resolved, so the
+<code class="docutils literal notranslate"><span
class="pre">UnResolved</span></code> stage state used by the static path is not
needed here.</p></li>
+</ul>
+<p>Keeping the two graphs side by side lets the adaptive implementation mature
without destabilizing the static
+path.</p>
+</section>
+<section id="how-re-optimization-works">
+<h3>How re-optimization works<a class="headerlink"
href="#how-re-optimization-works" title="Link to this heading">#</a></h3>
+<p>As each shuffle stage finishes, its exact output statistics are attached to
the remaining plan and the
+optimizer rules are re-run. This is the key difference from DataFusion’s
normal single-shot physical
+optimization, where each rule runs once per plan. Running the full rule set
repeatedly means the rules must be
+<strong>idempotent</strong> — otherwise they would keep inserting redundant
exec nodes on each pass.</p>
+<p>Two adaptive optimizations are implemented today:</p>
+<ul class="simple">
+<li><p><strong>Join reordering</strong> — runtime row counts are used so the
smaller side drives the join.</p></li>
+<li><p><strong>Empty stage elimination</strong> — when a completed stage
produces zero rows, its downstream exchange is
+replaced with an empty execution node and emptiness is propagated up the plan
so dependent stages are
+skipped entirely.</p></li>
+</ul>
+</section>
+<section id="code-layout">
+<h3>Code layout<a class="headerlink" href="#code-layout" title="Link to this
heading">#</a></h3>
+<p>The adaptive scheduler code lives under <a class="reference external"
href="https://github.com/apache/datafusion-ballista/tree/main/ballista/scheduler/src/state/aqe";><code
class="docutils literal notranslate"><span
class="pre">ballista/scheduler/src/state/aqe</span></code></a>, with the
+<code class="docutils literal notranslate"><span
class="pre">AdaptiveExecutionGraph</span></code> in <code class="docutils
literal notranslate"><span class="pre">mod.rs</span></code>, the planner in
<code class="docutils literal notranslate"><span
class="pre">planner.rs</span></code>, and the optimizer rules in
+<code class="docutils literal notranslate"><span
class="pre">optimizer_rule.rs</span></code>.</p>
+</section>
+<section id="current-limitations">
+<h3>Current limitations<a class="headerlink" href="#current-limitations"
title="Link to this heading">#</a></h3>
+<p>The adaptive path currently covers the happy path only. Known gaps, each
with a tracking issue:</p>
+<ul class="simple">
+<li><p>Executor failure handling on the AQE path (<a class="reference
external"
href="https://github.com/apache/datafusion-ballista/issues/1986";>#1986</a>)</p></li>
+<li><p>Dynamic coalescing of shuffle partitions (<a class="reference external"
href="https://github.com/apache/datafusion-ballista/issues/1987";>#1987</a>)</p></li>
+<li><p>Switching from hash join to sort-merge join based on runtime statistics
(<a class="reference external"
href="https://github.com/apache/datafusion-ballista/issues/1988";>#1988</a>)</p></li>
+<li><p>Switching from streaming aggregation to hash aggregation based on
runtime statistics (<a class="reference external"
href="https://github.com/apache/datafusion-ballista/issues/1989";>#1989</a>)</p></li>
+</ul>
+<p>Design and progress are tracked in the AQE epic (<a class="reference
external"
href="https://github.com/apache/datafusion-ballista/issues/1359";>#1359</a>).
See also the
+<a class="reference internal" href="../user-guide/tuning-guide.html"><span
class="std std-doc">AQE section of the tuning guide</span></a> for the
user-facing description.</p>
+</section>
+</section>
</section>
@@ -617,6 +672,13 @@ and <a class="reference external"
href="https://github.com/apache/datafusion-bal
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#shuffle">Shuffle</a></li>
</ul>
</li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#adaptive-query-execution-aqe">Adaptive Query Execution (AQE)</a><ul
class="nav section-nav flex-column">
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#static-vs-adaptive-execution-graphs">Static vs adaptive execution
graphs</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#how-re-optimization-works">How re-optimization works</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#code-layout">Code layout</a></li>
+<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#current-limitations">Current limitations</a></li>
+</ul>
+</li>
</ul>
</nav></div>
diff --git a/searchindex.js b/searchindex.js
index 36beefa4a..92b81df85 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles":{"AQE join behavior
(opt-in)":[[25,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution
(Experimental)":[[47,"adaptive-query-execution-experimental"]],"Aggregate
Functions":[[46,"aggregate-functions"]],"Apache DataFusion
Ballista":[[24,null]],"Apache DataFusion Ballista 0.10.0
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0
Changelog":[[1,null]],"Apache DataFusion Ballista 0.12.0
Changelog":[[2,null]],"Apache DataFusion Ballista 0.5.0 Changelog":[[ [...]
\ No newline at end of file
+Search.setIndex({"alltitles":{"AQE join behavior
(opt-in)":[[25,"aqe-join-behavior-opt-in"]],"Adaptive Query Execution
(AQE)":[[21,"adaptive-query-execution-aqe"]],"Adaptive Query Execution
(Experimental)":[[47,"adaptive-query-execution-experimental"]],"Aggregate
Functions":[[46,"aggregate-functions"]],"Apache DataFusion
Ballista":[[24,null]],"Apache DataFusion Ballista 0.10.0
Changelog":[[0,null]],"Apache DataFusion Ballista 0.11.0
Changelog":[[1,null]],"Apache DataFusion Ballista 0.12. [...]
\ No newline at end of file
diff --git a/user-guide/tuning-guide.html b/user-guide/tuning-guide.html
index f5fe67091..98de61e7c 100644
--- a/user-guide/tuning-guide.html
+++ b/user-guide/tuning-guide.html
@@ -605,14 +605,14 @@ is propagated up the plan so downstream stages are
skipped entirely.</p></li>
<p>The implementation covers the happy path only. The following are known to be
missing or incomplete:</p>
<ul class="simple">
-<li><p>Executor failure handling on the AQE path</p></li>
-<li><p>Dynamic coalescing of shuffle partitions</p></li>
-<li><p>Switching from hash join to sort-merge join based on runtime
statistics</p></li>
-<li><p>Switching from streaming aggregation to hash aggregation based on
runtime statistics</p></li>
+<li><p>Executor failure handling on the AQE path (<a class="reference
external"
href="https://github.com/apache/datafusion-ballista/issues/1986";>#1986</a>)</p></li>
+<li><p>Dynamic coalescing of shuffle partitions (<a class="reference external"
href="https://github.com/apache/datafusion-ballista/issues/1987";>#1987</a>)</p></li>
+<li><p>Switching from hash join to sort-merge join based on runtime statistics
(<a class="reference external"
href="https://github.com/apache/datafusion-ballista/issues/1988";>#1988</a>)</p></li>
+<li><p>Switching from streaming aggregation to hash aggregation based on
runtime statistics (<a class="reference external"
href="https://github.com/apache/datafusion-ballista/issues/1989";>#1989</a>)</p></li>
</ul>
<p>Until these gaps are closed, AQE should be used for testing and
experimentation
-rather than production workloads. See <a class="reference external"
href="https://github.com/apache/datafusion-ballista/issues/387";>issue #387</a>
-for the tracking issue and ongoing work.</p>
+rather than production workloads. See <a class="reference external"
href="https://github.com/apache/datafusion-ballista/issues/1359";>issue #1359</a>
+for the tracking epic and ongoing work.</p>
</section>
</section>
<section id="push-based-vs-pull-based-task-scheduling">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
