This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 1f77953e doc: Add memory tuning section to user guide (#684)
1f77953e is described below
commit 1f77953e60ca5cdee972590402d4f05144098249
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Fri Jul 19 20:11:03 2024 -0700
doc: Add memory tuning section to user guide (#684)
* doc: Add memory tuning section to user guide
* Update docs/source/user-guide/tuning.md
Co-authored-by: Andy Grove <[email protected]>
* More
---------
Co-authored-by: Andy Grove <[email protected]>
---
docs/source/user-guide/tuning.md | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/docs/source/user-guide/tuning.md b/docs/source/user-guide/tuning.md
index f46ab9e0..876f498d 100644
--- a/docs/source/user-guide/tuning.md
+++ b/docs/source/user-guide/tuning.md
@@ -21,6 +21,21 @@ under the License.
Comet provides some tuning options to help you get the best performance from
your queries.
+## Memory Tuning
+
+Comet currently doesn't share the memory allocation from Spark but owns its
own memory allocation.
+That's said, Comet requires additional memory to be allocated. Comet provides
some memory related configurations to help you tune the memory usage.
+
+By default, the amount of memory is `spark.comet.memory.overhead.factor` *
`spark.executor.memory`.
+The default value for `spark.comet.memory.overhead.factor` is 0.2. You can
increase the factor to require more
+memory for Comet to use, if you see OOM error. Generally, increasing memory
overhead will improve the performance of your queries.
+For example, some operators like `SortMergeJoin` and `HashAggregate` may
require more memory to run.
+Once the memory is not enough, the operator will spill to disk, which will
slow down the query.
+
+Besides, you can also set the memory explicitly by setting
`spark.comet.memoryOverhead` to the desired value.
+Comet will allocate at least `spark.comet.memory.overhead.min` memory.
+
+If both `spark.comet.memoryOverhead` and `spark.comet.memory.overhead.factor`
are set, the former will be used.
## Shuffle
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
