This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a0562eda Publish built docs triggered by
14494d3a06338b28ce8ad31d032ac60b75f4c227
a0562eda is described below
commit a0562eda8b79269797bbbe925e896fc545b03385
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Thu May 9 00:39:10 2024 +0000
Publish built docs triggered by 14494d3a06338b28ce8ad31d032ac60b75f4c227
---
_sources/user-guide/configs.md.txt | 2 +-
user-guide/configs.html | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/_sources/user-guide/configs.md.txt
b/_sources/user-guide/configs.md.txt
index 02ecbd69..22a7a098 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -29,7 +29,7 @@ Comet provides the following configuration settings.
| spark.comet.columnar.shuffle.async.enabled | Whether to enable asynchronous
shuffle for Arrow-based shuffle. By default, this config is false. | false |
| spark.comet.columnar.shuffle.async.max.thread.num | Maximum number of
threads on an executor used for Comet async columnar shuffle. By default, this
config is 100. This is the upper bound of total number of shuffle threads per
executor. In other words, if the number of cores * the number of shuffle
threads per task `spark.comet.columnar.shuffle.async.thread.num` is larger than
this config. Comet will use this config as the number of shuffle threads per
executor instead. | 100 |
| spark.comet.columnar.shuffle.async.thread.num | Number of threads used for
Comet async columnar shuffle per shuffle task. By default, this config is 3.
Note that more threads means more memory requirement to buffer shuffle data
before flushing to disk. Also, more threads may not always improve performance,
and should be set based on the number of cores available. | 3 |
-| spark.comet.columnar.shuffle.enabled | Force Comet to only use columnar
shuffle for CometScan and Spark regular operators. If this is enabled, Comet
native shuffle will not be enabled but only Arrow shuffle. By default, this
config is false. | false |
+| spark.comet.columnar.shuffle.enabled | Whether to enable Arrow-based
columnar shuffle for Comet and Spark regular operators. If this is enabled,
Comet prefers columnar shuffle than native shuffle. By default, this config is
true. | true |
| spark.comet.columnar.shuffle.memory.factor | Fraction of Comet memory to be
allocated per executor process for Comet shuffle. Comet memory size is
specified by `spark.comet.memoryOverhead` or calculated by
`spark.comet.memory.overhead.factor` * `spark.executor.memory`. By default,
this config is 1.0. | 1.0 |
| spark.comet.debug.enabled | Whether to enable debug mode for Comet. By
default, this config is false. When enabled, Comet will do additional checks
for debugging purpose. For example, validating array when importing arrays from
JVM at native side. Note that these checks may be expensive in performance and
should only be enabled for debugging purpose. | false |
| spark.comet.enabled | Whether to enable Comet extension for Spark. When this
is turned on, Spark will use Comet to read Parquet data source. Note that to
enable native vectorized execution, both this config and
'spark.comet.exec.enabled' need to be enabled. By default, this config is the
value of the env var `ENABLE_COMET` if set, or true otherwise. | true |
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 0344865a..795a315c 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -318,8 +318,8 @@ under the License.
<td><p>3</p></td>
</tr>
<tr class="row-even"><td><p>spark.comet.columnar.shuffle.enabled</p></td>
-<td><p>Force Comet to only use columnar shuffle for CometScan and Spark
regular operators. If this is enabled, Comet native shuffle will not be enabled
but only Arrow shuffle. By default, this config is false.</p></td>
-<td><p>false</p></td>
+<td><p>Whether to enable Arrow-based columnar shuffle for Comet and Spark
regular operators. If this is enabled, Comet prefers columnar shuffle than
native shuffle. By default, this config is true.</p></td>
+<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.columnar.shuffle.memory.factor</p></td>
<td><p>Fraction of Comet memory to be allocated per executor process for Comet
shuffle. Comet memory size is specified by <code class="docutils literal
notranslate"><span class="pre">spark.comet.memoryOverhead</span></code> or
calculated by <code class="docutils literal notranslate"><span
class="pre">spark.comet.memory.overhead.factor</span></code> * <code
class="docutils literal notranslate"><span
class="pre">spark.executor.memory</span></code>. By default, this config is
1.0.</p></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
