This is an automated email from the ASF dual-hosted git repository.
github-actions[bot] pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 611d3d898e Publish built docs triggered by
32e2d7c4a82fafd70ba023dd848c6076609f3c9a
611d3d898e is described below
commit 611d3d898eed755d20f7312d1a8e35ebdb4b2a1d
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue Jul 21 13:11:03 2026 +0000
Publish built docs triggered by 32e2d7c4a82fafd70ba023dd848c6076609f3c9a
---
_sources/contributor-guide/jvm_shuffle.md.txt | 13 +++++--------
_sources/user-guide/latest/configs.md.txt | 4 +---
contributor-guide/jvm_shuffle.html | 7 -------
searchindex.js | 2 +-
user-guide/latest/configs.html | 18 +++++-------------
5 files changed, 12 insertions(+), 32 deletions(-)
diff --git a/_sources/contributor-guide/jvm_shuffle.md.txt
b/_sources/contributor-guide/jvm_shuffle.md.txt
index 2145c82eba..e67f79a524 100644
--- a/_sources/contributor-guide/jvm_shuffle.md.txt
+++ b/_sources/contributor-guide/jvm_shuffle.md.txt
@@ -184,14 +184,11 @@ Selection logic in
`CometShuffleManager.shouldBypassMergeSort()`:
- `CometShuffleMemoryAllocator`: Custom allocator for off-heap memory pages
- Memory is allocated in pages; when allocation fails, writers spill to disk
- `CometDiskBlockWriter` coordinates spilling across all partition writers
(largest first)
-- Async spilling is supported via `ShuffleThreadPool`
## Configuration
-| Config | Description
|
-| ----------------------------------------------- |
----------------------------------- |
-| `spark.comet.columnar.shuffle.async.enabled` | Enable async spill writes
|
-| `spark.comet.columnar.shuffle.async.thread.num` | Threads per writer for
async |
-| `spark.comet.columnar.shuffle.batch.size` | Rows per Arrow batch
|
-| `spark.comet.columnar.shuffle.spill.threshold` | Row count threshold for
spill |
-| `spark.comet.exec.shuffle.compression.codec` | Compression codec (zstd,
lz4, etc.) |
+| Config | Description
|
+| ---------------------------------------------- |
----------------------------------- |
+| `spark.comet.columnar.shuffle.batch.size` | Rows per Arrow batch
|
+| `spark.comet.columnar.shuffle.spill.threshold` | Row count threshold for
spill |
+| `spark.comet.exec.shuffle.compression.codec` | Compression codec (zstd,
lz4, etc.) |
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index f79e45df39..7e5f520e4f 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -99,10 +99,8 @@ These settings can be used to determine which parts of the
plan are accelerated
<!-- prettier-ignore-start -->
| Config | Description | Default Value |
|--------|-------------|---------------|
-| `spark.comet.columnar.shuffle.async.enabled` | Whether to enable
asynchronous shuffle for Arrow-based shuffle. | false |
-| `spark.comet.columnar.shuffle.async.max.thread.num` | Maximum number of
threads on an executor used for Comet async columnar shuffle. This is the upper
bound of total number of shuffle threads per executor. In other words, if the
number of cores * the number of shuffle threads per task
`spark.comet.columnar.shuffle.async.thread.num` is larger than this config.
Comet will use this config as the number of shuffle threads per executor
instead. | 100 |
-| `spark.comet.columnar.shuffle.async.thread.num` | Number of threads used for
Comet async columnar shuffle per shuffle task. Note that more threads means
more memory requirement to buffer shuffle data before flushing to disk. Also,
more threads may not always improve performance, and should be set based on the
number of cores available. | 3 |
| `spark.comet.columnar.shuffle.batch.size` | Batch size when writing out
sorted spill files on the native side. Note that this should not be larger than
batch size (i.e., `spark.comet.batchSize`). Otherwise it will produce larger
batches than expected in the native operator after shuffle. | 8192 |
+| `spark.comet.columnar.shuffle.max.writers.per.executor` | Maximum number of
concurrent hash-based shuffle writers per executor. Comet's hash-based (bypass
merge sort) columnar shuffle allocates one writer per partition, so `executor
cores * numPartitions` writers can be active concurrently. When that product
exceeds this cap, Comet falls back to sort-based shuffle to avoid OOM. | 100 |
| `spark.comet.exec.shuffle.compression.codec` | The codec of Comet native
shuffle used to compress shuffle data. lz4, zstd, and snappy are supported.
Compression can be disabled by setting spark.shuffle.compress=false. | lz4 |
| `spark.comet.exec.shuffle.compression.zstd.level` | The compression level to
use when compressing shuffle files with zstd. | 1 |
| `spark.comet.exec.shuffle.convertFromSparkPlan.enabled` | When enabled,
Comet will convert a Spark `ShuffleExchangeExec` to a Comet columnar shuffle
even when its child is a non-Comet (Spark) plan. Disable to leave such shuffles
as native Spark shuffles, restricting Comet shuffle to cases where the child is
already a Comet plan. | true |
diff --git a/contributor-guide/jvm_shuffle.html
b/contributor-guide/jvm_shuffle.html
index b4238f24e5..84b9dfce31 100644
--- a/contributor-guide/jvm_shuffle.html
+++ b/contributor-guide/jvm_shuffle.html
@@ -780,7 +780,6 @@ to protobuf and implements the partitioning logic in native
code.</p>
<li><p><code class="docutils literal notranslate"><span
class="pre">CometShuffleMemoryAllocator</span></code>: Custom allocator for
off-heap memory pages</p></li>
<li><p>Memory is allocated in pages; when allocation fails, writers spill to
disk</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">CometDiskBlockWriter</span></code> coordinates spilling across all
partition writers (largest first)</p></li>
-<li><p>Async spilling is supported via <code class="docutils literal
notranslate"><span class="pre">ShuffleThreadPool</span></code></p></li>
</ul>
</section>
<section id="configuration">
@@ -792,12 +791,6 @@ to protobuf and implements the partitioning logic in
native code.</p>
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.async.enabled</span></code></p></td>
-<td><p>Enable async spill writes</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.async.thread.num</span></code></p></td>
-<td><p>Threads per writer for async</p></td>
-</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.batch.size</span></code></p></td>
<td><p>Rows per Arrow batch</p></td>
</tr>
diff --git a/searchindex.js b/searchindex.js
index 75694070db..8d48050344 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"!": [[53, "id1"]], "%": [[51, "id1"]], "&":
[[41, "id1"]], "*": [[51, "id2"]], "+": [[51, "id3"]], "-": [[51, "id4"]], "/":
[[51, "id5"]], "1. Format Your Code": [[38, "format-your-code"]], "1. Install
Comet": [[59, "install-comet"], [69, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[26, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[38, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[59,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"!": [[53, "id1"]], "%": [[51, "id1"]], "&":
[[41, "id1"]], "*": [[51, "id2"]], "+": [[51, "id3"]], "-": [[51, "id4"]], "/":
[[51, "id5"]], "1. Format Your Code": [[38, "format-your-code"]], "1. Install
Comet": [[59, "install-comet"], [69, "install-comet"]], "1. Native Operators
(nativeExecs map)": [[26, "native-operators-nativeexecs-map"]], "2. Build and
Verify": [[38, "build-and-verify"]], "2. Clone Iceberg and Apply Diff": [[59,
"clone-iceberg-and-apply- [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index f8df7199f1..c278deab22 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -963,22 +963,14 @@ under the License.
</tr>
</thead>
<tbody>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.async.enabled</span></code></p></td>
-<td><p>Whether to enable asynchronous shuffle for Arrow-based shuffle.</p></td>
-<td><p>false</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.async.max.thread.num</span></code></p></td>
-<td><p>Maximum number of threads on an executor used for Comet async columnar
shuffle. This is the upper bound of total number of shuffle threads per
executor. In other words, if the number of cores * the number of shuffle
threads per task <code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.async.thread.num</span></code> is
larger than this config. Comet will use this config as the number of shuffle
threads per executor instead.</p></td>
-<td><p>100</p></td>
-</tr>
-<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.async.thread.num</span></code></p></td>
-<td><p>Number of threads used for Comet async columnar shuffle per shuffle
task. Note that more threads means more memory requirement to buffer shuffle
data before flushing to disk. Also, more threads may not always improve
performance, and should be set based on the number of cores available.</p></td>
-<td><p>3</p></td>
-</tr>
-<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.batch.size</span></code></p></td>
+<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.batch.size</span></code></p></td>
<td><p>Batch size when writing out sorted spill files on the native side. Note
that this should not be larger than batch size (i.e., <code class="docutils
literal notranslate"><span class="pre">spark.comet.batchSize</span></code>).
Otherwise it will produce larger batches than expected in the native operator
after shuffle.</p></td>
<td><p>8192</p></td>
</tr>
+<tr class="row-odd"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.max.writers.per.executor</span></code></p></td>
+<td><p>Maximum number of concurrent hash-based shuffle writers per executor.
Comet’s hash-based (bypass merge sort) columnar shuffle allocates one writer
per partition, so <code class="docutils literal notranslate"><span
class="pre">executor</span> <span class="pre">cores</span> <span
class="pre">*</span> <span class="pre">numPartitions</span></code> writers can
be active concurrently. When that product exceeds this cap, Comet falls back to
sort-based shuffle to avoid OOM.</p></td>
+<td><p>100</p></td>
+</tr>
<tr class="row-even"><td><p><code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.shuffle.compression.codec</span></code></p></td>
<td><p>The codec of Comet native shuffle used to compress shuffle data. lz4,
zstd, and snappy are supported. Compression can be disabled by setting
spark.shuffle.compress=false.</p></td>
<td><p>lz4</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
