This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 8d380c85c7 Publish built docs triggered by
cf639fdec71e0337b42bf0068adbb3d1813edde3
8d380c85c7 is described below
commit 8d380c85c722525d5eb7a531e97ccd48ed6b5877
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Sat Sep 12 22:47:26 2026 +0000
Publish built docs triggered by cf639fdec71e0337b42bf0068adbb3d1813edde3
---
_sources/user-guide/latest/tuning.md.txt | 40 ++++++++++++++++++++++++++++++++
searchindex.js | 2 +-
user-guide/latest/tuning.html | 34 +++++++++++++++++++++++++++
3 files changed, 75 insertions(+), 1 deletion(-)
diff --git a/_sources/user-guide/latest/tuning.md.txt
b/_sources/user-guide/latest/tuning.md.txt
index ceec8b1a15..25b805d1cc 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -31,6 +31,46 @@ It is recommended that `COMET_WORKER_THREADS` be set to the
number of executor c
in some environments, such as Kubernetes, where the number of cores allocated
to a pod will already be equal to the
number of executor cores.
+## Adaptive Partial Aggregation
+
+For high-cardinality grouping, Comet can bypass partial hash aggregation when
it is not
+reducing the number of rows enough. This currently applies only to fused
native shuffle-writer
+plans whose partial aggregates are grouping-only or single-argument `COUNT`.
Low-cardinality
+inputs continue to aggregate normally. The SQL metric `rows bypassing partial
aggregation`
+shows whether skipping occurred.
+
+Eligibility is conservative for the whole fused native plan: any unsupported
partial accumulator,
+Spark `PartialMerge`, or mixed-mode aggregate disables skipping in that plan.
Multi-argument
+`COUNT` and other accumulators are not admitted. Distribution-required
grouping-only stages
+still fully deduplicate, and non-native-shuffle plans retain ordinary
aggregation.
+The DataFusion testing configuration override does not bypass these safety
checks.
+
+DataFusion 55 defaults to probing after 100,000 input rows per partial
aggregation
+partition and skipping when the number of groups divided by input rows exceeds
`0.8`.
+To experiment with these thresholds, enable
`spark.comet.exec.respectDataFusionConfigs`,
+a development and testing option that defaults to `false`. For example, the
following
+SQL settings pass through the default threshold values, which you can adjust:
+
+```sql
+SET spark.comet.exec.respectDataFusionConfigs=true;
+SET
spark.comet.datafusion.execution.skip_partial_aggregation_probe_rows_threshold=100000;
+SET
spark.comet.datafusion.execution.skip_partial_aggregation_probe_ratio_threshold=0.8;
+```
+
+A lower row threshold allows an earlier decision; a lower ratio threshold makes
+skipping more likely. Skipping can increase the number of partial states
emitted
+and the amount of shuffle data, so measure the effect on your workload.
+
+To disable skipping, keep `spark.comet.exec.respectDataFusionConfigs=true` and
set
+the ratio threshold above the maximum possible groups/input-rows ratio:
+
+```sql
+SET
spark.comet.datafusion.execution.skip_partial_aggregation_probe_ratio_threshold=1.1;
+```
+
+These settings only tune eligible plans. Unsupported accumulators and modes
remain
+disabled even when configuration overrides are enabled.
+
## Memory Tuning
It is necessary to specify how much memory Comet can use in addition to memory
already allocated to Spark. In some
diff --git a/searchindex.js b/searchindex.js
index 432c1aa640..601b83f9de 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [72, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[56, "id1"]], "%": [[54, "id1"]], "&":
[[44, "id1"]], "*": [[54, "id2"]], "+": [[54, "id3"]], "-": [[54, "id4"]], "/":
[[54, "id5"]], "1. Format Your Code": [[41, "format-your-code"]], "1. Install
Comet": [[62, "install-comet"], [72, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[28, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[41, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[62,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index 02122ec67e..2c137713bd 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -786,6 +786,39 @@ and <code class="docutils literal notranslate"><span
class="pre">COMET_MAX_BLOCK
in some environments, such as Kubernetes, where the number of cores allocated
to a pod will already be equal to the
number of executor cores.</p>
</section>
+<section id="adaptive-partial-aggregation">
+<h2>Adaptive Partial Aggregation<a class="headerlink"
href="#adaptive-partial-aggregation" title="Link to this heading">#</a></h2>
+<p>For high-cardinality grouping, Comet can bypass partial hash aggregation
when it is not
+reducing the number of rows enough. This currently applies only to fused
native shuffle-writer
+plans whose partial aggregates are grouping-only or single-argument <code
class="docutils literal notranslate"><span class="pre">COUNT</span></code>.
Low-cardinality
+inputs continue to aggregate normally. The SQL metric <code class="docutils
literal notranslate"><span class="pre">rows</span> <span
class="pre">bypassing</span> <span class="pre">partial</span> <span
class="pre">aggregation</span></code>
+shows whether skipping occurred.</p>
+<p>Eligibility is conservative for the whole fused native plan: any
unsupported partial accumulator,
+Spark <code class="docutils literal notranslate"><span
class="pre">PartialMerge</span></code>, or mixed-mode aggregate disables
skipping in that plan. Multi-argument
+<code class="docutils literal notranslate"><span
class="pre">COUNT</span></code> and other accumulators are not admitted.
Distribution-required grouping-only stages
+still fully deduplicate, and non-native-shuffle plans retain ordinary
aggregation.
+The DataFusion testing configuration override does not bypass these safety
checks.</p>
+<p>DataFusion 55 defaults to probing after 100,000 input rows per partial
aggregation
+partition and skipping when the number of groups divided by input rows exceeds
<code class="docutils literal notranslate"><span class="pre">0.8</span></code>.
+To experiment with these thresholds, enable <code class="docutils literal
notranslate"><span
class="pre">spark.comet.exec.respectDataFusionConfigs</span></code>,
+a development and testing option that defaults to <code class="docutils
literal notranslate"><span class="pre">false</span></code>. For example, the
following
+SQL settings pass through the default threshold values, which you can
adjust:</p>
+<div class="highlight-sql notranslate"><div
class="highlight"><pre><span></span><span class="k">SET</span><span class="w">
</span><span class="n">spark</span><span class="p">.</span><span
class="n">comet</span><span class="p">.</span><span class="k">exec</span><span
class="p">.</span><span class="n">respectDataFusionConfigs</span><span
class="o">=</span><span class="k">true</span><span class="p">;</span>
+<span class="k">SET</span><span class="w"> </span><span
class="n">spark</span><span class="p">.</span><span class="n">comet</span><span
class="p">.</span><span class="n">datafusion</span><span
class="p">.</span><span class="n">execution</span><span class="p">.</span><span
class="n">skip_partial_aggregation_probe_rows_threshold</span><span
class="o">=</span><span class="mi">100000</span><span class="p">;</span>
+<span class="k">SET</span><span class="w"> </span><span
class="n">spark</span><span class="p">.</span><span class="n">comet</span><span
class="p">.</span><span class="n">datafusion</span><span
class="p">.</span><span class="n">execution</span><span class="p">.</span><span
class="n">skip_partial_aggregation_probe_ratio_threshold</span><span
class="o">=</span><span class="mi">0</span><span class="p">.</span><span
class="mi">8</span><span class="p">;</span>
+</pre></div>
+</div>
+<p>A lower row threshold allows an earlier decision; a lower ratio threshold
makes
+skipping more likely. Skipping can increase the number of partial states
emitted
+and the amount of shuffle data, so measure the effect on your workload.</p>
+<p>To disable skipping, keep <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.respectDataFusionConfigs=true</span></code> and set
+the ratio threshold above the maximum possible groups/input-rows ratio:</p>
+<div class="highlight-sql notranslate"><div
class="highlight"><pre><span></span><span class="k">SET</span><span class="w">
</span><span class="n">spark</span><span class="p">.</span><span
class="n">comet</span><span class="p">.</span><span
class="n">datafusion</span><span class="p">.</span><span
class="n">execution</span><span class="p">.</span><span
class="n">skip_partial_aggregation_probe_ratio_threshold</span><span
class="o">=</span><span class="mi">1</span><span class="p">.</span><sp [...]
+</pre></div>
+</div>
+<p>These settings only tune eligible plans. Unsupported accumulators and modes
remain
+disabled even when configuration overrides are enabled.</p>
+</section>
<section id="memory-tuning">
<h2>Memory Tuning<a class="headerlink" href="#memory-tuning" title="Link to
this heading">#</a></h2>
<p>It is necessary to specify how much memory Comet can use in addition to
memory already allocated to Spark. In some
@@ -1160,6 +1193,7 @@ fallback to Spark works, see <a class="reference
internal" href="understanding-c
<nav class="bd-toc-nav page-toc"
aria-labelledby="pst-page-navigation-heading-2">
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#configuring-tokio-runtime">Configuring Tokio Runtime</a></li>
+<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#adaptive-partial-aggregation">Adaptive Partial Aggregation</a></li>
<li class="toc-h2 nav-item toc-entry"><a class="reference internal nav-link"
href="#memory-tuning">Memory Tuning</a><ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#configuring-comet-memory">Configuring Comet Memory</a></li>
<li class="toc-h3 nav-item toc-entry"><a class="reference internal nav-link"
href="#determining-how-much-memory-to-allocate">Determining How Much Memory to
Allocate</a></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
