This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new dd2c84b8 Publish built docs triggered by
1f77953e60ca5cdee972590402d4f05144098249
dd2c84b8 is described below
commit dd2c84b8bbde78d602135bf0dd41dcc5c90cd95e
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Sat Jul 20 03:11:28 2024 +0000
Publish built docs triggered by 1f77953e60ca5cdee972590402d4f05144098249
---
_sources/user-guide/tuning.md.txt | 15 +++++++++++++++
searchindex.js | 2 +-
user-guide/tuning.html | 18 ++++++++++++++++++
3 files changed, 34 insertions(+), 1 deletion(-)
diff --git a/_sources/user-guide/tuning.md.txt
b/_sources/user-guide/tuning.md.txt
index f46ab9e0..876f498d 100644
--- a/_sources/user-guide/tuning.md.txt
+++ b/_sources/user-guide/tuning.md.txt
@@ -21,6 +21,21 @@ under the License.
Comet provides some tuning options to help you get the best performance from
your queries.
+## Memory Tuning
+
+Comet currently doesn't share the memory allocation from Spark but owns its
own memory allocation.
+That's said, Comet requires additional memory to be allocated. Comet provides
some memory related configurations to help you tune the memory usage.
+
+By default, the amount of memory is `spark.comet.memory.overhead.factor` *
`spark.executor.memory`.
+The default value for `spark.comet.memory.overhead.factor` is 0.2. You can
increase the factor to require more
+memory for Comet to use, if you see OOM error. Generally, increasing memory
overhead will improve the performance of your queries.
+For example, some operators like `SortMergeJoin` and `HashAggregate` may
require more memory to run.
+Once the memory is not enough, the operator will spill to disk, which will
slow down the query.
+
+Besides, you can also set the memory explicitly by setting
`spark.comet.memoryOverhead` to the desired value.
+Comet will allocate at least `spark.comet.memory.overhead.min` memory.
+
+If both `spark.comet.memoryOverhead` and `spark.comet.memory.overhead.factor`
are set, the former will be used.
## Shuffle
diff --git a/searchindex.js b/searchindex.js
index a5839b7c..2e5f19d9 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[7, "install-comet"]], "2.
Clone Spark and Apply Diff": [[7, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[7, "run-spark-sql-tests"]], "ANSI mode": [[9, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[8, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, nu [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[7, "install-comet"]], "2.
Clone Spark and Apply Diff": [[7, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[7, "run-spark-sql-tests"]], "ANSI mode": [[9, "ansi-mode"]], "API
Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[8, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, nu [...]
\ No newline at end of file
diff --git a/user-guide/tuning.html b/user-guide/tuning.html
index 90837c85..48f6aac6 100644
--- a/user-guide/tuning.html
+++ b/user-guide/tuning.html
@@ -266,6 +266,11 @@ under the License.
<nav id="bd-toc-nav">
<ul class="visible nav section-nav flex-column">
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#memory-tuning">
+ Memory Tuning
+ </a>
+ </li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#shuffle">
Shuffle
@@ -344,6 +349,19 @@ under the License.
<section id="tuning-guide">
<h1>Tuning Guide<a class="headerlink" href="#tuning-guide" title="Link to this
heading">¶</a></h1>
<p>Comet provides some tuning options to help you get the best performance
from your queries.</p>
+<section id="memory-tuning">
+<h2>Memory Tuning<a class="headerlink" href="#memory-tuning" title="Link to
this heading">¶</a></h2>
+<p>Comet currently doesn’t share the memory allocation from Spark but owns its
own memory allocation.
+That’s said, Comet requires additional memory to be allocated. Comet provides
some memory related configurations to help you tune the memory usage.</p>
+<p>By default, the amount of memory is <code class="docutils literal
notranslate"><span class="pre">spark.comet.memory.overhead.factor</span></code>
* <code class="docutils literal notranslate"><span
class="pre">spark.executor.memory</span></code>.
+The default value for <code class="docutils literal notranslate"><span
class="pre">spark.comet.memory.overhead.factor</span></code> is 0.2. You can
increase the factor to require more
+memory for Comet to use, if you see OOM error. Generally, increasing memory
overhead will improve the performance of your queries.
+For example, some operators like <code class="docutils literal
notranslate"><span class="pre">SortMergeJoin</span></code> and <code
class="docutils literal notranslate"><span
class="pre">HashAggregate</span></code> may require more memory to run.
+Once the memory is not enough, the operator will spill to disk, which will
slow down the query.</p>
+<p>Besides, you can also set the memory explicitly by setting <code
class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code> to the desired value.
+Comet will allocate at least <code class="docutils literal notranslate"><span
class="pre">spark.comet.memory.overhead.min</span></code> memory.</p>
+<p>If both <code class="docutils literal notranslate"><span
class="pre">spark.comet.memoryOverhead</span></code> and <code class="docutils
literal notranslate"><span
class="pre">spark.comet.memory.overhead.factor</span></code> are set, the
former will be used.</p>
+</section>
<section id="shuffle">
<h2>Shuffle<a class="headerlink" href="#shuffle" title="Link to this
heading">¶</a></h2>
<p>Comet provides Comet shuffle features that can be used to improve the
performance of your queries.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
