This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d99c83e8d6 Publish built docs triggered by
198109160221e41cc97ecd7e415971006fa3b637
d99c83e8d6 is described below
commit d99c83e8d66867b33ecb4d1c0ec89004cea80484
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Wed Jul 22 13:18:44 2026 +0000
Publish built docs triggered by 198109160221e41cc97ecd7e415971006fa3b637
---
_sources/contributor-guide/native_shuffle.md.txt | 4 +++-
_sources/user-guide/latest/configs.md.txt | 1 +
contributor-guide/native_shuffle.html | 4 +++-
searchindex.js | 2 +-
user-guide/latest/configs.html | 8 ++++++--
5 files changed, 14 insertions(+), 5 deletions(-)
diff --git a/_sources/contributor-guide/native_shuffle.md.txt
b/_sources/contributor-guide/native_shuffle.md.txt
index c46b8b45a8..4d6421de5e 100644
--- a/_sources/contributor-guide/native_shuffle.md.txt
+++ b/_sources/contributor-guide/native_shuffle.md.txt
@@ -214,7 +214,9 @@ sizes.
Native shuffle uses DataFusion's memory management with spilling support:
- **Memory pool**: Tracks memory usage across the shuffle operation.
-- **Spill threshold**: When buffered data exceeds the threshold, partitions
spill to disk.
+- **Spill triggers**: Partitions spill to disk when the memory pool denies an
allocation, or
+ when the buffered bytes reach `spark.comet.shuffle.maxBufferBytes`. That
config defaults to
+ 0, which disables the fixed limit and leaves memory pressure as the only
trigger.
- **Per-partition spilling**: Each partition has its own spill file. Multiple
spills for a
partition are concatenated when writing the final output.
- **Scratch space**: Reusable buffers for partition ID computation to reduce
allocations.
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index 7e5f520e4f..c695d00185 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -112,6 +112,7 @@ These settings can be used to determine which parts of the
plan are accelerated
| `spark.comet.native.shuffle.partitioning.range.enabled` | Whether to enable
range partitioning for Comet native shuffle. | true |
| `spark.comet.native.shuffle.partitioning.roundrobin.enabled` | Whether to
enable round robin partitioning for Comet native shuffle. This is disabled by
default because Comet's round-robin produces different partition assignments
than Spark. Spark sorts rows by their binary UnsafeRow representation before
assigning partitions, but Comet uses Arrow format which has a different binary
layout. Instead, Comet implements round-robin as hash partitioning on all
columns, which achieves the sam [...]
| `spark.comet.native.shuffle.partitioning.roundrobin.maxHashColumns` | The
maximum number of columns to hash for round robin partitioning. When set to 0
(the default), all columns are hashed. When set to a positive value, only the
first N columns are used for hashing, which can improve performance for wide
tables while still providing reasonable distribution. | 0 |
+| `spark.comet.shuffle.maxBufferBytes` | Maximum number of bytes that the
native shuffle writer will buffer in memory before spilling to disk. The
default of 0 disables this limit, in which case spilling is triggered only when
the memory pool denies an allocation. Setting a limit caps the writer's memory
footprint independently of the size of the memory pool, at the cost of more
frequent spilling. | 0b |
| `spark.comet.shuffle.preferDictionary.ratio` | The ratio of total values to
distinct values in a string column to decide whether to prefer dictionary
encoding when shuffling the column. If the ratio is higher than this config,
dictionary encoding will be used on shuffling string column. This config is
effective if it is higher than 1.0. Note that this config is only used when
`spark.comet.exec.shuffle.mode` is `jvm`. | 10.0 |
| `spark.comet.shuffle.sizeInBytesMultiplier` | Comet reports smaller sizes
for shuffle due to using Arrow's columnar memory format and this can result in
Spark choosing a different join strategy due to the estimated size of the
exchange being smaller. Comet will multiple sizeInBytes by this amount to avoid
regressions in join strategy. | 1.0 |
<!-- prettier-ignore-end -->
diff --git a/contributor-guide/native_shuffle.html
b/contributor-guide/native_shuffle.html
index 9666f40a41..e31bf064ca 100644
--- a/contributor-guide/native_shuffle.html
+++ b/contributor-guide/native_shuffle.html
@@ -798,7 +798,9 @@ sizes.</p>
<p>Native shuffle uses DataFusion’s memory management with spilling
support:</p>
<ul class="simple">
<li><p><strong>Memory pool</strong>: Tracks memory usage across the shuffle
operation.</p></li>
-<li><p><strong>Spill threshold</strong>: When buffered data exceeds the
threshold, partitions spill to disk.</p></li>
+<li><p><strong>Spill triggers</strong>: Partitions spill to disk when the
memory pool denies an allocation, or
+when the buffered bytes reach <code class="docutils literal notranslate"><span
class="pre">spark.comet.shuffle.maxBufferBytes</span></code>. That config
defaults to
+0, which disables the fixed limit and leaves memory pressure as the only
trigger.</p></li>
<li><p><strong>Per-partition spilling</strong>: Each partition has its own
spill file. Multiple spills for a
partition are concatenated when writing the final output.</p></li>
<li><p><strong>Scratch space</strong>: Reusable buffers for partition ID
computation to reduce allocations.</p></li>
diff --git a/searchindex.js b/searchindex.js
index 8d48050344..c5116646cf 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[53, "id1"]], "%": [[51, "id1"]], "&":
[[41, "id1"]], "*": [[51, "id2"]], "+": [[51, "id3"]], "-": [[51, "id4"]], "/":
[[51, "id5"]], "1. Format Your Code": [[38, "format-your-code"]], "1. Install
Comet": [[59, "install-comet"], [69, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[26, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[38, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[59,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[53, "id1"]], "%": [[51, "id1"]], "&":
[[41, "id1"]], "*": [[51, "id2"]], "+": [[51, "id3"]], "-": [[51, "id4"]], "/":
[[51, "id5"]], "1. Format Your Code": [[38, "format-your-code"]], "1. Install
Comet": [[59, "install-comet"], [69, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[26, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[38, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[59,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index c278deab22..6c4936ca04 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -1015,11 +1015,15 @@ under the License.
<td><p>The maximum number of columns to hash for round robin partitioning.
When set to 0 (the default), all columns are hashed. When set to a positive
value, only the first N columns are used for hashing, which can improve
performance for wide tables while still providing reasonable
distribution.</p></td>
<td><p>0</p></td>
</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.shuffle.preferDictionary.ratio</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.shuffle.maxBufferBytes</span></code></p></td>
+<td><p>Maximum number of bytes that the native shuffle writer will buffer in
memory before spilling to disk. The default of 0 disables this limit, in which
case spilling is triggered only when the memory pool denies an allocation.
Setting a limit caps the writer’s memory footprint independently of the size of
the memory pool, at the cost of more frequent spilling.</p></td>
+<td><p>0b</p></td>
+</tr>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.shuffle.preferDictionary.ratio</span></code></p></td>
<td><p>The ratio of total values to distinct values in a string column to
decide whether to prefer dictionary encoding when shuffling the column. If the
ratio is higher than this config, dictionary encoding will be used on shuffling
string column. This config is effective if it is higher than 1.0. Note that
this config is only used when <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.shuffle.mode</span></code> is <code
class="docutils literal notranslate"><s [...]
<td><p>10.0</p></td>
</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.shuffle.sizeInBytesMultiplier</span></code></p></td>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.shuffle.sizeInBytesMultiplier</span></code></p></td>
<td><p>Comet reports smaller sizes for shuffle due to using Arrow’s columnar
memory format and this can result in Spark choosing a different join strategy
due to the estimated size of the exchange being smaller. Comet will multiple
sizeInBytes by this amount to avoid regressions in join strategy.</p></td>
<td><p>1.0</p></td>
</tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
