This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c881825b4 Publish built docs triggered by
68d756a72f78545e4cfccbc059ed70248ca4642c
c881825b4 is described below
commit c881825b4c7aa32a6b1e32ca06f99572f3dfe81e
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue Oct 14 17:13:07 2025 +0000
Publish built docs triggered by 68d756a72f78545e4cfccbc059ed70248ca4642c
---
_sources/user-guide/latest/configs.md.txt | 7 +-
_sources/user-guide/latest/tuning.md.txt | 99 ++++----------------------
searchindex.js | 2 +-
user-guide/latest/configs.html | 62 +++++++----------
user-guide/latest/tuning.html | 112 +++++-------------------------
5 files changed, 58 insertions(+), 224 deletions(-)
diff --git a/_sources/user-guide/latest/configs.md.txt
b/_sources/user-guide/latest/configs.md.txt
index 13efe7024..918fbc4b1 100644
--- a/_sources/user-guide/latest/configs.md.txt
+++ b/_sources/user-guide/latest/configs.md.txt
@@ -49,8 +49,8 @@ Comet provides the following configuration settings.
| spark.comet.exec.globalLimit.enabled | Whether to enable globalLimit by
default. | true |
| spark.comet.exec.hashJoin.enabled | Whether to enable hashJoin by default. |
true |
| spark.comet.exec.localLimit.enabled | Whether to enable localLimit by
default. | true |
-| spark.comet.exec.memoryPool | The type of memory pool to be used for Comet
native execution. When running Spark in on-heap mode, available pool types are
'greedy', 'fair_spill', 'greedy_task_shared', 'fair_spill_task_shared',
'greedy_global', 'fair_spill_global', and `unbounded`. When running Spark in
off-heap mode, available pool types are 'greedy_unified' and `fair_unified`.
The default pool type is `greedy_task_shared` for on-heap mode and `unified`
for off-heap mode. For more infor [...]
-| spark.comet.exec.onHeap.enabled | Whether to allow Comet to run in on-heap
mode. Required for running Spark SQL tests. | false |
+| spark.comet.exec.memoryPool | The type of memory pool to be used for Comet
native execution when running Spark in off-heap mode. Available pool types are
'greedy_unified' and `fair_unified`. For more information, refer to the Comet
Tuning Guide (https://datafusion.apache.org/comet/user-guide/tuning.html). |
fair_unified |
+| spark.comet.exec.memoryPool.fraction | Fraction of off-heap memory pool that
is available to Comet. Only applies to off-heap mode. For more information,
refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 1.0 |
| spark.comet.exec.project.enabled | Whether to enable project by default. |
true |
| spark.comet.exec.replaceSortMergeJoin | Experimental feature to force Spark
to replace SortMergeJoin with ShuffledHashJoin for improved performance. This
feature is not stable yet. For more information, refer to the Comet Tuning
Guide (https://datafusion.apache.org/comet/user-guide/tuning.html). | false |
| spark.comet.exec.shuffle.compression.codec | The codec of Comet native
shuffle used to compress shuffle data. lz4, zstd, and snappy are supported.
Compression can be disabled by setting spark.shuffle.compress=false. | lz4 |
@@ -69,9 +69,6 @@ Comet provides the following configuration settings.
| spark.comet.expression.allowIncompatible | Comet is not currently fully
compatible with Spark for all expressions. Set this config to true to allow
them anyway. For more information, refer to the Comet Compatibility Guide
(https://datafusion.apache.org/comet/user-guide/compatibility.html). | false |
| spark.comet.logFallbackReasons.enabled | When this setting is enabled, Comet
will log warnings for all fallback reasons. | false |
| spark.comet.maxTempDirectorySize | The maximum amount of data (in bytes)
stored inside the temporary directories. | 107374182400b |
-| spark.comet.memory.overhead.factor | Fraction of executor memory to be
allocated as additional memory for Comet when running Spark in on-heap mode.
For more information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 0.2 |
-| spark.comet.memory.overhead.min | Minimum amount of additional memory to be
allocated per executor process for Comet, in MiB, when running Spark in on-heap
mode. For more information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | 402653184b |
-| spark.comet.memoryOverhead | The amount of additional memory to be allocated
per executor process for Comet, in MiB, when running Spark in on-heap mode.
This config is optional. If this is not specified, it will be set to
`spark.comet.memory.overhead.factor` * `spark.executor.memory`. For more
information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html). | |
| spark.comet.metrics.updateInterval | The interval in milliseconds to update
metrics. If interval is negative, metrics will be updated upon task completion.
| 3000 |
| spark.comet.native.shuffle.partitioning.hash.enabled | Whether to enable
hash partitioning for Comet native shuffle. | true |
| spark.comet.native.shuffle.partitioning.range.enabled | Whether to enable
range partitioning for Comet native shuffle. | true |
diff --git a/_sources/user-guide/latest/tuning.md.txt
b/_sources/user-guide/latest/tuning.md.txt
index 585d5a9d5..8015e1d3b 100644
--- a/_sources/user-guide/latest/tuning.md.txt
+++ b/_sources/user-guide/latest/tuning.md.txt
@@ -36,41 +36,15 @@ than before. See the [Determining How Much Memory to
Allocate] section for more
[Determining How Much Memory to Allocate]:
#determining-how-much-memory-to-allocate
-Comet supports Spark's on-heap (the default) and off-heap mode for allocating
memory. However, we strongly recommend
-using off-heap mode. Comet has some limitations when running in on-heap mode,
such as requiring more memory overall,
-and requiring shuffle memory to be separately configured.
+### Configuring Comet Memory
-### Configuring Comet Memory in Off-Heap Mode
-
-The recommended way to allocate memory for Comet is to set
`spark.memory.offHeap.enabled=true`. This allows
-Comet to share an off-heap memory pool with Spark, reducing the overall memory
overhead. The size of the pool is
+Comet shares an off-heap memory pool with Spark. The size of the pool is
specified by `spark.memory.offHeap.size`. For more details about Spark
off-heap memory mode, please refer to
[Spark documentation]. For full details on configuring Comet memory in
off-heap mode, see the [Advanced Memory Tuning]
section of this guide.
[Spark documentation]: https://spark.apache.org/docs/latest/configuration.html
-### Configuring Comet Memory in On-Heap Mode
-
-```{warning}
-Support for on-heap memory pools is deprecated and will be removed from a
future release.
-```
-
-Comet is disabled by default in on-heap mode, but can be enabled by setting
`spark.comet.exec.onHeap.enabled=true`.
-
-When running in on-heap mode, Comet memory can be allocated by setting
`spark.comet.memoryOverhead`. If this setting
-is not provided, it will be calculated by multiplying the current Spark
executor memory by
-`spark.comet.memory.overhead.factor` (default value is `0.2`) which may or may
not result in enough memory for
-Comet to operate. It is not recommended to rely on this behavior. It is better
to specify `spark.comet.memoryOverhead`
-explicitly.
-
-Comet supports native shuffle and columnar shuffle (these terms are explained
in the [shuffle] section below).
-In on-heap mode, columnar shuffle memory must be separately allocated using
`spark.comet.columnar.shuffle.memorySize`.
-If this setting is not provided, it will be calculated by multiplying
`spark.comet.memoryOverhead` by
-`spark.comet.columnar.shuffle.memory.factor` (default value is `1.0`). If a
shuffle exceeds this amount of memory
-then the query will fail. For full details on configuring Comet memory in
on-heap mode, see the [Advanced Memory Tuning]
-section of this guide.
-
[shuffle]: #shuffle
[Advanced Memory Tuning]: #advanced-memory-tuning
@@ -93,7 +67,7 @@ Baseline Spark Performance
Comet Performance
-- Comet requires at least 5 GB of RAM in off-heap mode and 6 GB RAM in on-heap
mode, but performance at this level
+- Comet requires at least 5 GB of RAM, but performance at this level
is around 340 seconds, which is significantly faster than Spark with any
amount of RAM
- Comet running in off-heap with 8 cores completes the benchmark in 295
seconds, more than 2x faster than Spark
- It is worth noting that running Comet with only 4 cores and 4 GB RAM
completes the benchmark in 520 seconds,
@@ -114,11 +88,11 @@ Workarounds for this problem include:
## Advanced Memory Tuning
-### Configuring Off-Heap Memory Pools
+### Configuring Comet Memory Pools
Comet implements multiple memory pool implementations. The type of pool can be
specified with `spark.comet.exec.memoryPool`.
-The valid pool types for off-heap mode are:
+The valid pool types are:
- `fair_unified` (default when `spark.memory.offHeap.enabled=true` is set)
- `greedy_unified`
@@ -126,67 +100,18 @@ The valid pool types for off-heap mode are:
Both of these pools share off-heap memory between Spark and Comet. This
approach is referred to as
unified memory management. The size of the pool is specified by
`spark.memory.offHeap.size`.
-The `greedy_unified` pool type implements a greedy first-come first-serve
limit. This pool works well for queries that do not
-need to spill or have a single spillable operator.
+Comet's memory accounting isn't 100% accurate and this can result in Comet
using more memory than it reserves,
+leading to out-of-memory exceptions. To work around this issue, it is possible
to
+set `spark.comet.exec.memoryPool.fraction` to a value less than `1.0` to
restrict the amount of memory that can be
+reserved by Comet.
-The `fair_unified` pool type prevents operators from using more than an even
fraction of the available memory
+The `fair_unified` pool types prevents operators from using more than an even
fraction of the available memory
(i.e. `pool_size / num_reservations`). This pool works best when you know
beforehand
the query has multiple operators that will likely all need to spill. Sometimes
it will cause spills even
when there is sufficient memory in order to leave enough memory for other
operators.
-### Configuring On-Heap Memory Pools
-
-```{warning}
-Support for on-heap memory pools is deprecated and will be removed from a
future release.
-```
-
-When running in on-heap mode, Comet will use its own dedicated memory pools
that are not shared with Spark.
-
-The type of pool can be specified with `spark.comet.exec.memoryPool`. The
default setting is `greedy_task_shared`.
-
-The valid pool types for on-heap mode are:
-
-- `greedy`
-- `greedy_global`
-- `greedy_task_shared`
-- `fair_spill`
-- `fair_spill_global`
-- `fair_spill_task_shared`
-- `unbounded`
-
-Pool types ending with `_global` use a single global memory pool between all
tasks on same executor.
-
-Pool types ending with `_task_shared` share a single memory pool across all
attempts for a single task.
-
-Other pool types create a dedicated pool per native query plan using a
fraction of the available pool size based on number of cores
-and cores per task.
-
-The `greedy*` pool types use DataFusion's [GreedyMemoryPool], which implements
a greedy first-come first-serve limit. This
-pool works well for queries that do not need to spill or have a single
spillable operator.
-
-The `fair_spill*` pool types use DataFusion's [FairSpillPool], which prevents
spillable reservations from using more
-than an even fraction of the available memory sans any unspillable reservations
-(i.e. `(pool_size - unspillable_memory) / num_spillable_reservations`). This
pool works best when you know beforehand
-the query has multiple spillable operators that will likely all need to spill.
Sometimes it will cause spills even
-when there was sufficient memory (reserved for other operators) to avoid doing
so. Unspillable memory is allocated in
-a first-come, first-serve fashion
-
-The `unbounded` pool type uses DataFusion's [UnboundedMemoryPool], which
enforces no limit. This option is useful for
-development/testing purposes, where there is no room to allow spilling and
rather choose to fail the job.
-Spilling significantly slows down the job and this option is one way to
measure the best performance scenario without
-adjusting how much memory to allocate.
-
-[GreedyMemoryPool]:
https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.GreedyMemoryPool.html
-[FairSpillPool]:
https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.FairSpillPool.html
-[UnboundedMemoryPool]:
https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.UnboundedMemoryPool.html
-
-### Configuring spark.executor.memoryOverhead in On-Heap Mode
-
-In some environments, such as Kubernetes and YARN, it is important to
correctly set `spark.executor.memoryOverhead` so
-that it is possible to allocate off-heap memory when running in on-heap mode.
-
-Comet will automatically set `spark.executor.memoryOverhead` based on the
`spark.comet.memory*` settings so that
-resource managers respect Apache Spark memory configuration before starting
the containers.
+The `greedy_unified` pool type implements a greedy first-come first-serve
limit. This pool works well for queries that do not
+need to spill or have a single spillable operator.
## Optimizing Joins
diff --git a/searchindex.js b/searchindex.js
index 8a5d4d55f..add786aee 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[12, "install-comet"]],
"2. Clone Spark and Apply Diff": [[12, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[12, "run-spark-sql-tests"]], "ANSI Mode": [[17,
"ansi-mode"], [56, "ansi-mode"]], "ANSI mode": [[30, "ansi-mode"], [43,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "Accelerating Apache Iceberg
Parquet Scans using Comet (Experimental)": [[22, null], [35, n [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[12, "install-comet"]],
"2. Clone Spark and Apply Diff": [[12, "clone-spark-and-apply-diff"]], "3. Run
Spark SQL Tests": [[12, "run-spark-sql-tests"]], "ANSI Mode": [[17,
"ansi-mode"], [56, "ansi-mode"]], "ANSI mode": [[30, "ansi-mode"], [43,
"ansi-mode"]], "API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "Accelerating Apache Iceberg
Parquet Scans using Comet (Experimental)": [[22, null], [35, n [...]
\ No newline at end of file
diff --git a/user-guide/latest/configs.html b/user-guide/latest/configs.html
index 97bea64ae..d8518ba6a 100644
--- a/user-guide/latest/configs.html
+++ b/user-guide/latest/configs.html
@@ -688,12 +688,12 @@ under the License.
<td><p>true</p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.exec.memoryPool</p></td>
-<td><p>The type of memory pool to be used for Comet native execution. When
running Spark in on-heap mode, available pool types are ‘greedy’, ‘fair_spill’,
‘greedy_task_shared’, ‘fair_spill_task_shared’, ‘greedy_global’,
‘fair_spill_global’, and <code class="docutils literal notranslate"><span
class="pre">unbounded</span></code>. When running Spark in off-heap mode,
available pool types are ‘greedy_unified’ and <code class="docutils literal
notranslate"><span class="pre">fair_unified</spa [...]
-<td><p>default</p></td>
+<td><p>The type of memory pool to be used for Comet native execution when
running Spark in off-heap mode. Available pool types are ‘greedy_unified’ and
<code class="docutils literal notranslate"><span
class="pre">fair_unified</span></code>. For more information, refer to the
Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
+<td><p>fair_unified</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.exec.onHeap.enabled</p></td>
-<td><p>Whether to allow Comet to run in on-heap mode. Required for running
Spark SQL tests.</p></td>
-<td><p>false</p></td>
+<tr class="row-even"><td><p>spark.comet.exec.memoryPool.fraction</p></td>
+<td><p>Fraction of off-heap memory pool that is available to Comet. Only
applies to off-heap mode. For more information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
+<td><p>1.0</p></td>
</tr>
<tr class="row-odd"><td><p>spark.comet.exec.project.enabled</p></td>
<td><p>Whether to enable project by default.</p></td>
@@ -767,95 +767,83 @@ under the License.
<td><p>The maximum amount of data (in bytes) stored inside the temporary
directories.</p></td>
<td><p>107374182400b</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.memory.overhead.factor</p></td>
-<td><p>Fraction of executor memory to be allocated as additional memory for
Comet when running Spark in on-heap mode. For more information, refer to the
Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
-<td><p>0.2</p></td>
-</tr>
-<tr class="row-even"><td><p>spark.comet.memory.overhead.min</p></td>
-<td><p>Minimum amount of additional memory to be allocated per executor
process for Comet, in MiB, when running Spark in on-heap mode. For more
information, refer to the Comet Tuning Guide
(https://datafusion.apache.org/comet/user-guide/tuning.html).</p></td>
-<td><p>402653184b</p></td>
-</tr>
-<tr class="row-odd"><td><p>spark.comet.memoryOverhead</p></td>
-<td><p>The amount of additional memory to be allocated per executor process
for Comet, in MiB, when running Spark in on-heap mode. This config is optional.
If this is not specified, it will be set to <code class="docutils literal
notranslate"><span class="pre">spark.comet.memory.overhead.factor</span></code>
* <code class="docutils literal notranslate"><span
class="pre">spark.executor.memory</span></code>. For more information, refer to
the Comet Tuning Guide (https://datafusion.apache.o [...]
-<td><p></p></td>
-</tr>
-<tr class="row-even"><td><p>spark.comet.metrics.updateInterval</p></td>
+<tr class="row-odd"><td><p>spark.comet.metrics.updateInterval</p></td>
<td><p>The interval in milliseconds to update metrics. If interval is
negative, metrics will be updated upon task completion.</p></td>
<td><p>3000</p></td>
</tr>
-<tr
class="row-odd"><td><p>spark.comet.native.shuffle.partitioning.hash.enabled</p></td>
+<tr
class="row-even"><td><p>spark.comet.native.shuffle.partitioning.hash.enabled</p></td>
<td><p>Whether to enable hash partitioning for Comet native shuffle.</p></td>
<td><p>true</p></td>
</tr>
-<tr
class="row-even"><td><p>spark.comet.native.shuffle.partitioning.range.enabled</p></td>
+<tr
class="row-odd"><td><p>spark.comet.native.shuffle.partitioning.range.enabled</p></td>
<td><p>Whether to enable range partitioning for Comet native shuffle.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.nativeLoadRequired</p></td>
+<tr class="row-even"><td><p>spark.comet.nativeLoadRequired</p></td>
<td><p>Whether to require Comet native library to load successfully when Comet
is enabled. If not, Comet will silently fallback to Spark when it fails to load
the native lib. Otherwise, an error will be thrown and the Spark job will be
aborted.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.parquet.enable.directBuffer</p></td>
+<tr class="row-odd"><td><p>spark.comet.parquet.enable.directBuffer</p></td>
<td><p>Whether to use Java direct byte buffer when reading Parquet.</p></td>
<td><p>false</p></td>
</tr>
-<tr
class="row-odd"><td><p>spark.comet.parquet.read.io.adjust.readRange.skew</p></td>
+<tr
class="row-even"><td><p>spark.comet.parquet.read.io.adjust.readRange.skew</p></td>
<td><p>In the parallel reader, if the read ranges submitted are skewed in
sizes, this option will cause the reader to break up larger read ranges into
smaller ranges to reduce the skew. This will result in a slightly larger number
of connections opened to the file system but may give improved
performance.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.parquet.read.io.mergeRanges</p></td>
+<tr class="row-odd"><td><p>spark.comet.parquet.read.io.mergeRanges</p></td>
<td><p>When enabled the parallel reader will try to merge ranges of data that
are separated by less than ‘comet.parquet.read.io.mergeRanges.delta’ bytes.
Longer continuous reads are faster on cloud storage.</p></td>
<td><p>true</p></td>
</tr>
-<tr
class="row-odd"><td><p>spark.comet.parquet.read.io.mergeRanges.delta</p></td>
+<tr
class="row-even"><td><p>spark.comet.parquet.read.io.mergeRanges.delta</p></td>
<td><p>The delta in bytes between consecutive read ranges below which the
parallel reader will try to merge the ranges. The default is 8MB.</p></td>
<td><p>8388608</p></td>
</tr>
-<tr
class="row-even"><td><p>spark.comet.parquet.read.parallel.io.enabled</p></td>
+<tr
class="row-odd"><td><p>spark.comet.parquet.read.parallel.io.enabled</p></td>
<td><p>Whether to enable Comet’s parallel reader for Parquet files. The
parallel reader reads ranges of consecutive data in a file in parallel. It is
faster for large files and row groups but uses more resources.</p></td>
<td><p>true</p></td>
</tr>
-<tr
class="row-odd"><td><p>spark.comet.parquet.read.parallel.io.thread-pool.size</p></td>
+<tr
class="row-even"><td><p>spark.comet.parquet.read.parallel.io.thread-pool.size</p></td>
<td><p>The maximum number of parallel threads the parallel reader will use in
a single executor. For executors configured with a smaller number of cores, use
a smaller number.</p></td>
<td><p>16</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.parquet.respectFilterPushdown</p></td>
+<tr class="row-odd"><td><p>spark.comet.parquet.respectFilterPushdown</p></td>
<td><p>Whether to respect Spark’s PARQUET_FILTER_PUSHDOWN_ENABLED config. This
needs to be respected when running the Spark SQL test suite but the default
setting results in poor performance in Comet when using the new native scans,
disabled by default</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.regexp.allowIncompatible</p></td>
+<tr class="row-even"><td><p>spark.comet.regexp.allowIncompatible</p></td>
<td><p>Comet is not currently fully compatible with Spark for all regular
expressions. Set this config to true to allow them anyway. For more
information, refer to the Comet Compatibility Guide
(https://datafusion.apache.org/comet/user-guide/compatibility.html).</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.scan.allowIncompatible</p></td>
+<tr class="row-odd"><td><p>spark.comet.scan.allowIncompatible</p></td>
<td><p>Some Comet scan implementations are not currently fully compatible with
Spark for all datatypes. Set this config to true to allow them anyway. For more
information, refer to the Comet Compatibility Guide
(https://datafusion.apache.org/comet/user-guide/compatibility.html).</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.scan.enabled</p></td>
+<tr class="row-even"><td><p>spark.comet.scan.enabled</p></td>
<td><p>Whether to enable native scans. When this is turned on, Spark will use
Comet to read supported data sources (currently only Parquet is supported
natively). Note that to enable native vectorized execution, both this config
and ‘spark.comet.exec.enabled’ need to be enabled.</p></td>
<td><p>true</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.scan.preFetch.enabled</p></td>
+<tr class="row-odd"><td><p>spark.comet.scan.preFetch.enabled</p></td>
<td><p>Whether to enable pre-fetching feature of CometScan.</p></td>
<td><p>false</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.scan.preFetch.threadNum</p></td>
+<tr class="row-even"><td><p>spark.comet.scan.preFetch.threadNum</p></td>
<td><p>The number of threads running pre-fetching for CometScan. Effective if
spark.comet.scan.preFetch.enabled is enabled. Note that more pre-fetching
threads means more memory requirement to store pre-fetched row groups.</p></td>
<td><p>2</p></td>
</tr>
-<tr class="row-even"><td><p>spark.comet.shuffle.preferDictionary.ratio</p></td>
+<tr class="row-odd"><td><p>spark.comet.shuffle.preferDictionary.ratio</p></td>
<td><p>The ratio of total values to distinct values in a string column to
decide whether to prefer dictionary encoding when shuffling the column. If the
ratio is higher than this config, dictionary encoding will be used on shuffling
string column. This config is effective if it is higher than 1.0. Note that
this config is only used when <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.shuffle.mode</span></code> is <code
class="docutils literal notranslate"><s [...]
<td><p>10.0</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.comet.shuffle.sizeInBytesMultiplier</p></td>
+<tr class="row-even"><td><p>spark.comet.shuffle.sizeInBytesMultiplier</p></td>
<td><p>Comet reports smaller sizes for shuffle due to using Arrow’s columnar
memory format and this can result in Spark choosing a different join strategy
due to the estimated size of the exchange being smaller. Comet will multiple
sizeInBytes by this amount to avoid regressions in join strategy.</p></td>
<td><p>1.0</p></td>
</tr>
-<tr
class="row-even"><td><p>spark.comet.sparkToColumnar.supportedOperatorList</p></td>
+<tr
class="row-odd"><td><p>spark.comet.sparkToColumnar.supportedOperatorList</p></td>
<td><p>A comma-separated list of operators that will be converted to Arrow
columnar format when ‘spark.comet.sparkToColumnar.enabled’ is true</p></td>
<td><p>Range,InMemoryTableScan,RDDScan</p></td>
</tr>
-<tr class="row-odd"><td><p>spark.hadoop.fs.comet.libhdfs.schemes</p></td>
+<tr class="row-even"><td><p>spark.hadoop.fs.comet.libhdfs.schemes</p></td>
<td><p>Defines filesystem schemes (e.g., hdfs, webhdfs) that the native side
accesses via libhdfs, separated by commas. Valid only when built with hdfs
feature enabled.</p></td>
<td><p></p></td>
</tr>
diff --git a/user-guide/latest/tuning.html b/user-guide/latest/tuning.html
index aa47b10fb..216263b74 100644
--- a/user-guide/latest/tuning.html
+++ b/user-guide/latest/tuning.html
@@ -552,13 +552,8 @@ under the License.
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#configuring-comet-memory-in-off-heap-mode">
- Configuring Comet Memory in Off-Heap Mode
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#configuring-comet-memory-in-on-heap-mode">
- Configuring Comet Memory in On-Heap Mode
+ <a class="reference internal nav-link" href="#configuring-comet-memory">
+ Configuring Comet Memory
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
@@ -579,18 +574,8 @@ under the License.
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#configuring-off-heap-memory-pools">
- Configuring Off-Heap Memory Pools
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#configuring-on-heap-memory-pools">
- Configuring On-Heap Memory Pools
- </a>
- </li>
- <li class="toc-h3 nav-item toc-entry">
- <a class="reference internal nav-link"
href="#configuring-spark-executor-memoryoverhead-in-on-heap-mode">
- Configuring spark.executor.memoryOverhead in On-Heap Mode
+ <a class="reference internal nav-link"
href="#configuring-comet-memory-pools">
+ Configuring Comet Memory Pools
</a>
</li>
</ul>
@@ -702,36 +687,13 @@ and <code class="docutils literal notranslate"><span
class="pre">COMET_MAX_BLOCK
cases, it may be possible to reduce the amount of memory allocated to Spark so
that overall memory allocation is
the same or lower than the original configuration. In other cases, enabling
Comet may require allocating more memory
than before. See the <a class="reference internal"
href="#determining-how-much-memory-to-allocate">Determining How Much Memory to
Allocate</a> section for more details.</p>
-<p>Comet supports Spark’s on-heap (the default) and off-heap mode for
allocating memory. However, we strongly recommend
-using off-heap mode. Comet has some limitations when running in on-heap mode,
such as requiring more memory overall,
-and requiring shuffle memory to be separately configured.</p>
-<section id="configuring-comet-memory-in-off-heap-mode">
-<h3>Configuring Comet Memory in Off-Heap Mode<a class="headerlink"
href="#configuring-comet-memory-in-off-heap-mode" title="Link to this
heading">¶</a></h3>
-<p>The recommended way to allocate memory for Comet is to set <code
class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.enabled=true</span></code>. This allows
-Comet to share an off-heap memory pool with Spark, reducing the overall memory
overhead. The size of the pool is
+<section id="configuring-comet-memory">
+<h3>Configuring Comet Memory<a class="headerlink"
href="#configuring-comet-memory" title="Link to this heading">¶</a></h3>
+<p>Comet shares an off-heap memory pool with Spark. The size of the pool is
specified by <code class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code>. For more details about
Spark off-heap memory mode, please refer to
<a class="reference external"
href="https://spark.apache.org/docs/latest/configuration.html";>Spark
documentation</a>. For full details on configuring Comet memory in off-heap
mode, see the <a class="reference internal"
href="#advanced-memory-tuning">Advanced Memory Tuning</a>
section of this guide.</p>
</section>
-<section id="configuring-comet-memory-in-on-heap-mode">
-<h3>Configuring Comet Memory in On-Heap Mode<a class="headerlink"
href="#configuring-comet-memory-in-on-heap-mode" title="Link to this
heading">¶</a></h3>
-<div class="admonition warning">
-<p class="admonition-title">Warning</p>
-<p>Support for on-heap memory pools is deprecated and will be removed from a
future release.</p>
-</div>
-<p>Comet is disabled by default in on-heap mode, but can be enabled by setting
<code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.onHeap.enabled=true</span></code>.</p>
-<p>When running in on-heap mode, Comet memory can be allocated by setting
<code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code>. If this setting
-is not provided, it will be calculated by multiplying the current Spark
executor memory by
-<code class="docutils literal notranslate"><span
class="pre">spark.comet.memory.overhead.factor</span></code> (default value is
<code class="docutils literal notranslate"><span class="pre">0.2</span></code>)
which may or may not result in enough memory for
-Comet to operate. It is not recommended to rely on this behavior. It is better
to specify <code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code>
-explicitly.</p>
-<p>Comet supports native shuffle and columnar shuffle (these terms are
explained in the <a class="reference internal" href="#shuffle">shuffle</a>
section below).
-In on-heap mode, columnar shuffle memory must be separately allocated using
<code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.memorySize</span></code>.
-If this setting is not provided, it will be calculated by multiplying <code
class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code> by
-<code class="docutils literal notranslate"><span
class="pre">spark.comet.columnar.shuffle.memory.factor</span></code> (default
value is <code class="docutils literal notranslate"><span
class="pre">1.0</span></code>). If a shuffle exceeds this amount of memory
-then the query will fail. For full details on configuring Comet memory in
on-heap mode, see the <a class="reference internal"
href="#advanced-memory-tuning">Advanced Memory Tuning</a>
-section of this guide.</p>
-</section>
<section id="determining-how-much-memory-to-allocate">
<h3>Determining How Much Memory to Allocate<a class="headerlink"
href="#determining-how-much-memory-to-allocate" title="Link to this
heading">¶</a></h3>
<p>Generally, increasing the amount of memory allocated to Comet will improve
query performance by reducing the
@@ -748,7 +710,7 @@ local Parquet files using the 100 GB data set.</p>
</ul>
<p>Comet Performance</p>
<ul class="simple">
-<li><p>Comet requires at least 5 GB of RAM in off-heap mode and 6 GB RAM in
on-heap mode, but performance at this level
+<li><p>Comet requires at least 5 GB of RAM, but performance at this level
is around 340 seconds, which is significantly faster than Spark with any
amount of RAM</p></li>
<li><p>Comet running in off-heap with 8 cores completes the benchmark in 295
seconds, more than 2x faster than Spark</p></li>
<li><p>It is worth noting that running Comet with only 4 cores and 4 GB RAM
completes the benchmark in 520 seconds,
@@ -770,64 +732,26 @@ https://github.com/apache/datafusion/issues/14692 for
more information.</p>
</section>
<section id="advanced-memory-tuning">
<h2>Advanced Memory Tuning<a class="headerlink" href="#advanced-memory-tuning"
title="Link to this heading">¶</a></h2>
-<section id="configuring-off-heap-memory-pools">
-<h3>Configuring Off-Heap Memory Pools<a class="headerlink"
href="#configuring-off-heap-memory-pools" title="Link to this
heading">¶</a></h3>
+<section id="configuring-comet-memory-pools">
+<h3>Configuring Comet Memory Pools<a class="headerlink"
href="#configuring-comet-memory-pools" title="Link to this heading">¶</a></h3>
<p>Comet implements multiple memory pool implementations. The type of pool can
be specified with <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool</span></code>.</p>
-<p>The valid pool types for off-heap mode are:</p>
+<p>The valid pool types are:</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span
class="pre">fair_unified</span></code> (default when <code class="docutils
literal notranslate"><span
class="pre">spark.memory.offHeap.enabled=true</span></code> is set)</p></li>
<li><p><code class="docutils literal notranslate"><span
class="pre">greedy_unified</span></code></p></li>
</ul>
<p>Both of these pools share off-heap memory between Spark and Comet. This
approach is referred to as
unified memory management. The size of the pool is specified by <code
class="docutils literal notranslate"><span
class="pre">spark.memory.offHeap.size</span></code>.</p>
-<p>The <code class="docutils literal notranslate"><span
class="pre">greedy_unified</span></code> pool type implements a greedy
first-come first-serve limit. This pool works well for queries that do not
-need to spill or have a single spillable operator.</p>
-<p>The <code class="docutils literal notranslate"><span
class="pre">fair_unified</span></code> pool type prevents operators from using
more than an even fraction of the available memory
+<p>Comet’s memory accounting isn’t 100% accurate and this can result in Comet
using more memory than it reserves,
+leading to out-of-memory exceptions. To work around this issue, it is possible
to
+set <code class="docutils literal notranslate"><span
class="pre">spark.comet.exec.memoryPool.fraction</span></code> to a value less
than <code class="docutils literal notranslate"><span
class="pre">1.0</span></code> to restrict the amount of memory that can be
+reserved by Comet.</p>
+<p>The <code class="docutils literal notranslate"><span
class="pre">fair_unified</span></code> pool types prevents operators from using
more than an even fraction of the available memory
(i.e. <code class="docutils literal notranslate"><span
class="pre">pool_size</span> <span class="pre">/</span> <span
class="pre">num_reservations</span></code>). This pool works best when you know
beforehand
the query has multiple operators that will likely all need to spill. Sometimes
it will cause spills even
when there is sufficient memory in order to leave enough memory for other
operators.</p>
-</section>
-<section id="configuring-on-heap-memory-pools">
-<h3>Configuring On-Heap Memory Pools<a class="headerlink"
href="#configuring-on-heap-memory-pools" title="Link to this heading">¶</a></h3>
-<div class="admonition warning">
-<p class="admonition-title">Warning</p>
-<p>Support for on-heap memory pools is deprecated and will be removed from a
future release.</p>
-</div>
-<p>When running in on-heap mode, Comet will use its own dedicated memory pools
that are not shared with Spark.</p>
-<p>The type of pool can be specified with <code class="docutils literal
notranslate"><span class="pre">spark.comet.exec.memoryPool</span></code>. The
default setting is <code class="docutils literal notranslate"><span
class="pre">greedy_task_shared</span></code>.</p>
-<p>The valid pool types for on-heap mode are:</p>
-<ul class="simple">
-<li><p><code class="docutils literal notranslate"><span
class="pre">greedy</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">greedy_global</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">greedy_task_shared</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">fair_spill</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">fair_spill_global</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">fair_spill_task_shared</span></code></p></li>
-<li><p><code class="docutils literal notranslate"><span
class="pre">unbounded</span></code></p></li>
-</ul>
-<p>Pool types ending with <code class="docutils literal notranslate"><span
class="pre">_global</span></code> use a single global memory pool between all
tasks on same executor.</p>
-<p>Pool types ending with <code class="docutils literal notranslate"><span
class="pre">_task_shared</span></code> share a single memory pool across all
attempts for a single task.</p>
-<p>Other pool types create a dedicated pool per native query plan using a
fraction of the available pool size based on number of cores
-and cores per task.</p>
-<p>The <code class="docutils literal notranslate"><span
class="pre">greedy*</span></code> pool types use DataFusion’s <a
class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.GreedyMemoryPool.html";>GreedyMemoryPool</a>,
which implements a greedy first-come first-serve limit. This
-pool works well for queries that do not need to spill or have a single
spillable operator.</p>
-<p>The <code class="docutils literal notranslate"><span
class="pre">fair_spill*</span></code> pool types use DataFusion’s <a
class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.FairSpillPool.html";>FairSpillPool</a>,
which prevents spillable reservations from using more
-than an even fraction of the available memory sans any unspillable reservations
-(i.e. <code class="docutils literal notranslate"><span
class="pre">(pool_size</span> <span class="pre">-</span> <span
class="pre">unspillable_memory)</span> <span class="pre">/</span> <span
class="pre">num_spillable_reservations</span></code>). This pool works best
when you know beforehand
-the query has multiple spillable operators that will likely all need to spill.
Sometimes it will cause spills even
-when there was sufficient memory (reserved for other operators) to avoid doing
so. Unspillable memory is allocated in
-a first-come, first-serve fashion</p>
-<p>The <code class="docutils literal notranslate"><span
class="pre">unbounded</span></code> pool type uses DataFusion’s <a
class="reference external"
href="https://docs.rs/datafusion/latest/datafusion/execution/memory_pool/struct.UnboundedMemoryPool.html";>UnboundedMemoryPool</a>,
which enforces no limit. This option is useful for
-development/testing purposes, where there is no room to allow spilling and
rather choose to fail the job.
-Spilling significantly slows down the job and this option is one way to
measure the best performance scenario without
-adjusting how much memory to allocate.</p>
-</section>
-<section id="configuring-spark-executor-memoryoverhead-in-on-heap-mode">
-<h3>Configuring spark.executor.memoryOverhead in On-Heap Mode<a
class="headerlink"
href="#configuring-spark-executor-memoryoverhead-in-on-heap-mode" title="Link
to this heading">¶</a></h3>
-<p>In some environments, such as Kubernetes and YARN, it is important to
correctly set <code class="docutils literal notranslate"><span
class="pre">spark.executor.memoryOverhead</span></code> so
-that it is possible to allocate off-heap memory when running in on-heap
mode.</p>
-<p>Comet will automatically set <code class="docutils literal
notranslate"><span class="pre">spark.executor.memoryOverhead</span></code>
based on the <code class="docutils literal notranslate"><span
class="pre">spark.comet.memory*</span></code> settings so that
-resource managers respect Apache Spark memory configuration before starting
the containers.</p>
+<p>The <code class="docutils literal notranslate"><span
class="pre">greedy_unified</span></code> pool type implements a greedy
first-come first-serve limit. This pool works well for queries that do not
+need to spill or have a single spillable operator.</p>
</section>
</section>
<section id="optimizing-joins">
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
