This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new afb52eb05 Publish built docs triggered by
badbd376898cb4f80b5136a90697019ad11a2e90
afb52eb05 is described below
commit afb52eb05b6c665cd91bcedd90b36e7f996de6ae
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Fri Mar 21 19:51:34 2025 +0000
Publish built docs triggered by badbd376898cb4f80b5136a90697019ad11a2e90
---
_sources/user-guide/configs.md.txt | 8 ++++----
_sources/user-guide/tuning.md.txt | 17 ++++++-----------
searchindex.js | 2 +-
user-guide/configs.html | 10 +++++-----
user-guide/tuning.html | 15 +++++----------
5 files changed, 21 insertions(+), 31 deletions(-)
diff --git a/_sources/user-guide/configs.md.txt
b/_sources/user-guide/configs.md.txt
index 32c6a3db8..d0f95480d 100644
--- a/_sources/user-guide/configs.md.txt
+++ b/_sources/user-guide/configs.md.txt
@@ -54,7 +54,7 @@ Comet provides the following configuration settings.
| spark.comet.exec.hashJoin.enabled | Whether to enable hashJoin by default. |
true |
| spark.comet.exec.initCap.enabled | Whether to enable initCap by default. |
false |
| spark.comet.exec.localLimit.enabled | Whether to enable localLimit by
default. | true |
-| spark.comet.exec.memoryPool | The type of memory pool to be used for Comet
native execution. Available memory pool types are 'greedy', 'fair_spill',
'greedy_task_shared', 'fair_spill_task_shared', 'greedy_global',
'fair_spill_global', and `unbounded`. For off-heap types are 'unified' and
`fair_unified`. | greedy_task_shared |
+| spark.comet.exec.memoryPool | The type of memory pool to be used for Comet
native execution. When running Spark in on-heap mode, available pool types are
'greedy', 'fair_spill', 'greedy_task_shared', 'fair_spill_task_shared',
'greedy_global', 'fair_spill_global', and `unbounded`. When running Spark in
off-heap mode, available pool types are 'unified' and `fair_unified`. The
default pool type is `greedy_task_shared` for on-heap mode and `unified` for
off-heap mode. For more information, [...]
| spark.comet.exec.project.enabled | Whether to enable project by default. |
true |
| spark.comet.exec.replaceSortMergeJoin | Experimental feature to force Spark
to replace SortMergeJoin with ShuffledHashJoin for improved performance. This
feature is not stable yet. For more information, refer to the Comet Tuning
Guide (https://datafusion.apache.org/comet/user-guide/tuning.html). | false |
| spark.comet.exec.shuffle.compression.codec | The codec of Comet native
shuffle used to compress shuffle data. lz4, zstd, and snappy are supported.
Compression can be disabled by setting spark.shuffle.compress=false. | lz4 |
@@ -71,9 +71,9 @@ Comet provides the following configuration settings.
| spark.comet.explain.verbose.enabled | When this setting is enabled, Comet
will provide a verbose tree representation of the extended information. | false
|
| spark.comet.explainFallback.enabled | When this setting is enabled, Comet
will provide logging explaining the reason(s) why a query stage cannot be
executed natively. Set this to false to reduce the amount of logging. | false |
| spark.comet.expression.allowIncompatible | Comet is not currently fully
compatible with Spark for all expressions. Set this config to true to allow
them anyway. For more information, refer to the Comet Compatibility Guide
(https://datafusion.apache.org/comet/user-guide/compatibility.html). | false |
-| spark.comet.memory.overhead.factor | Fraction of executor memory to be
allocated as additional memory for Comet when running in on-heap mode or when
using the `fair_unified` pool in off-heap mode. For more information, refer to
the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 0.2 |
-| spark.comet.memory.overhead.min | Minimum amount of additional memory to be
allocated per executor process for Comet, in MiB, when running in on-heap mode
or when using the `fair_unified` pool in off-heap mode. For more information,
refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 402653184b |
-| spark.comet.memoryOverhead | The amount of additional memory to be allocated
per executor process for Comet, in MiB, when running in on-heap mode or when
using the `fair_unified` pool in off-heap mode. This config is optional. If
this is not specified, it will be set to `spark.comet.memory.overhead.factor` *
`spark.executor.memory`. For more information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | |
+| spark.comet.memory.overhead.factor | Fraction of executor memory to be
allocated as additional memory for Comet when running in on-heap mode. For more
information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 0.2 |
+| spark.comet.memory.overhead.min | Minimum amount of additional memory to be
allocated per executor process for Comet, in MiB, when running in on-heap mode.
For more information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 402653184b |
+| spark.comet.memoryOverhead | The amount of additional memory to be allocated
per executor process for Comet, in MiB, when running in on-heap mode. This
config is optional. If this is not specified, it will be set to
`spark.comet.memory.overhead.factor` * `spark.executor.memory`. For more
information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | |
| spark.comet.metrics.updateInterval | The interval in milliseconds to update
metrics. If interval is negative, metrics will be updated upon task completion.
| 3000 |
| spark.comet.nativeLoadRequired | Whether to require Comet native library to
load successfully when Comet is enabled. If not, Comet will silently fallback
to Spark when it fails to load the native lib. Otherwise, an error will be
thrown and the Spark job will be aborted. | false |
| spark.comet.parquet.enable.directBuffer | Whether to use Java direct byte
buffer when reading Parquet. | false |
diff --git a/_sources/user-guide/tuning.md.txt
b/_sources/user-guide/tuning.md.txt
index 1a17f4ccc..7a745d5fc 100644
--- a/_sources/user-guide/tuning.md.txt
+++ b/_sources/user-guide/tuning.md.txt
@@ -108,34 +108,29 @@ resource managers respect Apache Spark memory
configuration before starting the
Comet implements multiple memory pool implementations. The type of pool can be
specified with `spark.comet.exec.memoryPool`.
-The valid pool types are:
+The valid pool types for off-heap mode are:
- `unified` (default when `spark.memory.offHeap.enabled=true` is set)
- `fair_unified`
+Both of these pools share off-heap memory between Spark and Comet. This
approach is referred to as
+unified memory management. The size of the pool is specified by
`spark.memory.offHeap.size`.
+
The `unified` pool type implements a greedy first-come first-serve limit. This
pool works well for queries that do not
-need to spill or have a single spillable operator. The size of the pool is
specified by `spark.memory.offHeap.size`
-and the pool interacts with Spark's memory pool, effectively sharing the
off-heap memory between Spark and Comet. This
-approach is sometimes referred to as unified memory management.
+need to spill or have a single spillable operator.
The `fair_unified` pool type prevents operators from using more than an even
fraction of the available memory
(i.e. `pool_size / num_reservations`). This pool works best when you know
beforehand
the query has multiple operators that will likely all need to spill. Sometimes
it will cause spills even
when there is sufficient memory in order to leave enough memory for other
operators.
-The pool size configuration for the `fair_unified` pool, is a little more
complex. The total pool size is computed by
-multiplying `spark.memory.offHeap.size` by
`spark.comet.memory.overhead.factor` with the minimum amount being
-`spark.comet.memory.overhead.min`. It is also possible to manually specify
`spark.comet.memoryOverhead` instead to
-override this default behavior. Note that the `fair_unified` pool does not use
unified memory management to interact
-with Spark's memory pools, which is why the allocation defaults to a fraction
of off-heap memory.
-
### Configuring On-Heap Memory Pools
When running in on-heap mode, Comet will use its own dedicated memory pools
that are not shared with Spark.
The type of pool can be specified with `spark.comet.exec.memoryPool`. The
default setting is `greedy_task_shared`.
-The valid pool types are:
+The valid pool types for on-heap mode are:
- `greedy`
- `greedy_global`
diff --git a/searchindex.js b/searchindex.js
index 6917c25cb..2ca4949ce 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
diff --git a/user-guide/configs.html b/user-guide/configs.html
index 230363898..edd91ab80 100644
--- a/user-guide/configs.html
+++ b/user-guide/configs.html
@@ -444,8 +444,8 @@ TO MODIFY THIS CONTENT MAKE SURE THAT YOU MAKE YOUR CHANGES
TO THE TEMPLATE FILE
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.exec.memoryPool</p></td>
-<td><p>The type of memory pool to be used for Comet native execution.
Available memory pool types are ‘greedy’, ‘fair_spill’, ‘greedy_task_shared’,
‘fair_spill_task_shared’, ‘greedy_global’, ‘fair_spill_global’, and <code
class="docutils literal notranslate"><span class="pre">unbounded</span></code>.
For off-heap types are ‘unified’ and <code class="docutils literal
notranslate"><span class="pre">fair_unified</span></code>.</p></td>
-<td><p>greedy_task_shared</p></td>
+<td><p>The type of memory pool to be used for Comet native execution. When
running Spark in on-heap mode, available pool types are ‘greedy’, ‘fair_spill’,
‘greedy_task_shared’, ‘fair_spill_task_shared’, ‘greedy_global’,
‘fair_spill_global’, and <code class="docutils literal notranslate"><span
class="pre">unbounded</span></code>. When running Spark in off-heap mode,
available pool types are ‘unified’ and <code class="docutils literal
notranslate"><span class="pre">fair_unified</span></cod [...]
+<td><p>default</p></td>
</tr>
<tr class="row-even"><td><p>spark.comet.exec.project.enabled</p></td>
<td><p>Whether to enable project by default.</p></td>
@@ -512,15 +512,15 @@ TO MODIFY THIS CONTENT MAKE SURE THAT YOU MAKE YOUR
CHANGES TO THE TEMPLATE FILE
<td><p>false</p></td>
</tr>
<tr class="row-even"><td><p>spark.comet.memory.overhead.factor</p></td>
-<td><p>Fraction of executor memory to be allocated as additional memory for
Comet when running in on-heap mode or when using the <code class="docutils
literal notranslate"><span class="pre">fair_unified</span></code> pool in
off-heap mode. For more information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
+<td><p>Fraction of executor memory to be allocated as additional memory for
Comet when running in on-heap mode. For more information, refer to the Comet
Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
<td><p>0.2</p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.memory.overhead.min</p></td>
-<td><p>Minimum amount of additional memory to be allocated per executor
process for Comet, in MiB, when running in on-heap mode or when using the <code
class="docutils literal notranslate"><span
class="pre">fair_unified</span></code> pool in off-heap mode. For more
information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
+<td><p>Minimum amount of additional memory to be allocated per executor
process for Comet, in MiB, when running in on-heap mode. For more information,
refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
<td><p>402653184b</p></td>
</tr>
<tr class="row-even"><td><p>spark.comet.memoryOverhead</p></td>
-<td><p>The amount of additional memory to be allocated per executor process
for Comet, in MiB, when running in on-heap mode or when using the <code
class="docutils literal notranslate"><span
class="pre">fair_unified</span></code> pool in off-heap mode. This config is
optional. If this is not specified, it will be set to <code class="docutils
literal notranslate"><span
class="pre">spark.comet.memory.overhead.factor</span></code> * <code
class="docutils literal notranslate"><span class="pr [...]
+<td><p>The amount of additional memory to be allocated per executor process
for Comet, in MiB, when running in on-heap mode. This config is optional. If
this is not specified, it will be set to <code class="docutils literal
notranslate"><span class="pre">spark.comet.memory.overhead.factor</span></code>
* <code class="docutils literal notranslate"><span
class="pre">spark.executor.memory</span></code>. For more information, refer to
the Comet Tuning Guide (https://datafusion.apache.org/com [...]
<td><p></p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.metrics.updateInterval</p></td>
diff --git a/user-guide/tuning.html b/user-guide/tuning.html
index 8df974620..bb9b0fe98 100644
--- a/user-guide/tuning.html
+++ b/user-guide/tuning.html
@@ -507,30 +507,25 @@ resource managers respect Apache Spark memory
configuration before starting the
<section id="configuring-off-heap-memory-pools">
<h3>Configuring Off-Heap Memory Pools<a class="headerlink"
href="#configuring-off-heap-memory-pools" title="Link to this
heading">¶</a></h3>
<p>Comet implements multiple memory pool implementations. The type of pool can
be specified with <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool</span></code>.</p>
-<p>The valid pool types are:</p>
+<p>The valid pool types for off-heap mode are:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span
class="pre">unified</span></code> (default when <code class="docutils literal
notranslate"><span class="pre">spark.memory.offHeap.enabled=true</span></code>
is set)</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">fair_unified</span></code></p></li>
</ul>
+<p>Both of these pools share off-heap memory between Spark and Comet. This
approach is referred to as
+unified memory management. The size of the pool is specified by <code
class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code>.</p>
<p>The <code class="docutils literal notranslate"><span
class="pre">unified</span></code> pool type implements a greedy first-come
first-serve limit. This pool works well for queries that do not
-need to spill or have a single spillable operator. The size of the pool is
specified by <code class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code>
-and the pool interacts with Spark’s memory pool, effectively sharing the
off-heap memory between Spark and Comet. This
-approach is sometimes referred to as unified memory management.</p>
+need to spill or have a single spillable operator.</p>
<p>The <code class="docutils literal notranslate"><span
class="pre">fair_unified</span></code> pool type prevents operators from using
more than an even fraction of the available memory
(i.e. <code class="docutils literal notranslate"><span
class="pre">pool_size</span> <span class="pre">/</span> <span
class="pre">num_reservations</span></code>). This pool works best when you know
beforehand
the query has multiple operators that will likely all need to spill. Sometimes
it will cause spills even
when there is sufficient memory in order to leave enough memory for other
operators.</p>
-<p>The pool size configuration for the <code class="docutils literal
notranslate"><span class="pre">fair_unified</span></code> pool, is a little
more complex. The total pool size is computed by
-multiplying <code class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code> by <code class="docutils
literal notranslate"><span
class="pre">spark.comet.memory.overhead.factor</span></code> with the minimum
amount being
-<code class="docutils literal notranslate"><span
class="pre">spark.comet.memory.overhead.min</span></code>. It is also possible
to manually specify <code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code> instead to
-override this default behavior. Note that the <code class="docutils literal
notranslate"><span class="pre">fair_unified</span></code> pool does not use
unified memory management to interact
-with Spark’s memory pools, which is why the allocation defaults to a fraction
of off-heap memory.</p>
</section>
<section id="configuring-on-heap-memory-pools">
<h3>Configuring On-Heap Memory Pools<a class="headerlink"
href="#configuring-on-heap-memory-pools" title="Link to this heading">¶</a></h3>
<p>When running in on-heap mode, Comet will use its own dedicated memory pools
that are not shared with Spark.</p>
<p>The type of pool can be specified with <code class="docutils literal
notranslate"><span class="pre">spark.comet.exec.memoryPool</span></code>. The
default setting is <code class="docutils literal notranslate"><span
class="pre">greedy_task_shared</span></code>.</p>
-<p>The valid pool types are:</p>
+<p>The valid pool types for on-heap mode are:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span
class="pre">greedy</span></code></p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">greedy_global</span></code></p></li>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
