This is an automated email from the ASF dual-hosted git repository.
comphead pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 27ab86be doc: Document CometPlugin to start Comet in cluster mode
(#836)
27ab86be is described below
commit 27ab86bee87af76db075508b85c6630c0636607c
Author: Oleks V <[email protected]>
AuthorDate: Mon Aug 19 12:32:02 2024 -0700
doc: Document CometPlugin to start Comet in cluster mode (#836)
* Documenting CometPlugin
Co-authored-by: Liang-Chi Hsieh <[email protected]>
---
docs/source/user-guide/installation.md | 3 +++
docs/source/user-guide/tuning.md | 11 +++++++++++
2 files changed, 14 insertions(+)
diff --git a/docs/source/user-guide/installation.md
b/docs/source/user-guide/installation.md
index 45f988e2..c6900d6d 100644
--- a/docs/source/user-guide/installation.md
+++ b/docs/source/user-guide/installation.md
@@ -152,3 +152,6 @@ To enable columnar shuffle which supports all partitioning
and basic complex typ
```
--conf spark.comet.exec.shuffle.mode=jvm
```
+
+### Memory tuning
+In addition to Apache Spark memory configuration parameters the Comet
introduces own parameters to configure memory allocation for native execution.
More [Comet Memory Tuning](./tuning.md)
\ No newline at end of file
diff --git a/docs/source/user-guide/tuning.md b/docs/source/user-guide/tuning.md
index 876f498d..9704bade 100644
--- a/docs/source/user-guide/tuning.md
+++ b/docs/source/user-guide/tuning.md
@@ -37,6 +37,17 @@ Comet will allocate at least
`spark.comet.memory.overhead.min` memory.
If both `spark.comet.memoryOverhead` and `spark.comet.memory.overhead.factor`
are set, the former will be used.
+## Memory Tuning using CometPlugin
+Configuring memory for Spark and Comet might be a tedious task as it requires
to tune Spark executor overhead memory and Comet memory overhead configs. Comet
provides a Spark plugin `CometPlugin` which can be set up to your Spark
application to help memory settings.
+
+For users running the Comet in clusters like Kubernetes or YARN, `CometPlugin`
can also make the resource manager respect correctly Comet memory parameters
`spark.comet.memory*`.
+it is needed to pass to the starting command line additional Spark
configuration parameter `--conf spark.plugins=org.apache.spark.CometPlugin`
+
+The resource managers respects Apache Spark memory configuration before
starting the containers.
+
+The `CometPlugin` plugin overrides `spark.executor.memoryOverhead` adding up
the Comet memory configuration.
+
+
## Shuffle
Comet provides Comet shuffle features that can be used to improve the
performance of your queries.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
