This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4d21583b Publish built docs triggered by
03164fcf4810c52fffad437d6504cd20bced939e
4d21583b is described below
commit 4d21583b6472e35a56dfbcde15f85af8b7d961ff
Author: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
AuthorDate: Tue Sep 3 15:36:28 2024 +0000
Publish built docs triggered by 03164fcf4810c52fffad437d6504cd20bced939e
---
_sources/user-guide/kubernetes.md.txt | 74 ++++++++++++++++++++++++++++++++++
searchindex.js | 2 +-
user-guide/kubernetes.html | 76 +++++++++++++++++++++++++++++++++++
3 files changed, 151 insertions(+), 1 deletion(-)
diff --git a/_sources/user-guide/kubernetes.md.txt
b/_sources/user-guide/kubernetes.md.txt
index 01483593..c69c1707 100644
--- a/_sources/user-guide/kubernetes.md.txt
+++ b/_sources/user-guide/kubernetes.md.txt
@@ -33,3 +33,77 @@ docker build -t apache/datafusion-comet -f kube/Dockerfile .
The exact syntax will vary depending on the Kubernetes distribution, but an
example `spark-submit` command can be
found [here](https://github.com/apache/datafusion-comet/tree/main/benchmarks).
+## Helm chart
+
+Install helm Spark operator for Kubernetes
+```bash
+helm repo add spark-operator https://kubeflow.github.io/spark-operator
+
+helm repo update
+
+helm install my-release spark-operator/spark-operator --namespace
spark-operator --create-namespace --set webhook.enable=true
+````
+
+Check the operator is deployed
+```bash
+helm status --namespace spark-operator my-release
+
+NAME: my-release
+NAMESPACE: spark-operator
+STATUS: deployed
+REVISION: 1
+TEST SUITE: None
+```
+
+Create example Spark application file `spark-pi.yaml`
+```bash
+apiVersion: sparkoperator.k8s.io/v1beta2
+kind: SparkApplication
+metadata:
+ name: spark-pi
+ namespace: default
+spec:
+ type: Scala
+ mode: cluster
+ image: ghcr.io/apache/datafusion-comet:spark-3.4-scala-2.12-0.2.0
+ imagePullPolicy: IfNotPresent
+ mainClass: org.apache.spark.examples.SparkPi
+ mainApplicationFile:
local:///opt/spark/examples/jars/spark-examples_2.12-3.4.2.jar
+ sparkConf:
+ "spark.executor.extraClassPath":
"/opt/spark/jars/comet-spark-spark3.4_2.12-0.2.0.jar"
+ "spark.driver.extraClassPath":
"/opt/spark/jars/comet-spark-spark3.4_2.12-0.2.0.jar"
+ "spark.plugins": "org.apache.spark.CometPlugin"
+ "spark.comet.enabled": "true"
+ "spark.comet.exec.enabled": "true"
+ "spark.comet.cast.allowIncompatible": "true"
+ "spark.comet.exec.shuffle.enabled": "true"
+ "spark.comet.exec.shuffle.mode": "auto"
+ "conf spark.shuffle.manager":
"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager"
+ sparkVersion: 3.4.3
+ driver:
+ labels:
+ version: 3.4.3
+ cores: 1
+ coreLimit: 1200m
+ memory: 512m
+ serviceAccount: spark-operator-spark
+ executor:
+ labels:
+ version: 3.4.3
+ instances: 1
+ cores: 2
+ coreLimit: 1200m
+ memory: 512m
+```
+Refer to [Comet builds](#comet-docker-images)
+
+Run Apache Spark application with Comet enabled
+```bash
+kubectl apply -f spark-pi.yaml
+```
+
+Check application status
+```bash
+kubectl describe sparkapplication --namespace=spark-operator
+```
+More info on Kube Spark operator
https://www.kubeflow.org/docs/components/spark-operator/getting-started/
\ No newline at end of file
diff --git a/searchindex.js b/searchindex.js
index 5c0ff19d..6148a5ed 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"1. Install Comet": [[9, "install-comet"]], "2.
Clone Spark and Apply Diff": [[9, "clone-spark-and-apply-diff"]], "3. Run Spark
SQL Tests": [[9, "run-spark-sql-tests"]], "ANSI mode": [[11, "ansi-mode"]],
"API Differences Between Spark Versions": [[0,
"api-differences-between-spark-versions"]], "ASF Links": [[10, null]], "Adding
Spark-side Tests for the New Expression": [[0,
"adding-spark-side-tests-for-the-new-expression"]], "Adding a New Expression":
[[0, [...]
\ No newline at end of file
diff --git a/user-guide/kubernetes.html b/user-guide/kubernetes.html
index ea6ff2fb..b1240e1a 100644
--- a/user-guide/kubernetes.html
+++ b/user-guide/kubernetes.html
@@ -279,6 +279,11 @@ under the License.
Example Spark Submit
</a>
</li>
+ <li class="toc-h2 nav-item toc-entry">
+ <a class="reference internal nav-link" href="#helm-chart">
+ Helm chart
+ </a>
+ </li>
</ul>
</nav>
@@ -340,6 +345,77 @@ Docker image from
https://github.com/orgs/apache/packages?repo_name=datafusion-c
<p>The exact syntax will vary depending on the Kubernetes distribution, but an
example <code class="docutils literal notranslate"><span
class="pre">spark-submit</span></code> command can be
found <a class="reference external"
href="https://github.com/apache/datafusion-comet/tree/main/benchmarks";>here</a>.</p>
</section>
+<section id="helm-chart">
+<h2>Helm chart<a class="headerlink" href="#helm-chart" title="Link to this
heading">ΒΆ</a></h2>
+<p>Install helm Spark operator for Kubernetes</p>
+<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>helm<span class="w"> </span>repo<span
class="w"> </span>add<span class="w"> </span>spark-operator<span class="w">
</span>https://kubeflow.github.io/spark-operator
+
+helm<span class="w"> </span>repo<span class="w"> </span>update
+
+helm<span class="w"> </span>install<span class="w"> </span>my-release<span
class="w"> </span>spark-operator/spark-operator<span class="w">
</span>--namespace<span class="w"> </span>spark-operator<span class="w">
</span>--create-namespace<span class="w"> </span>--set<span class="w">
</span>webhook.enable<span class="o">=</span><span class="nb">true</span>
+</pre></div>
+</div>
+<p>Check the operator is deployed</p>
+<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>helm<span class="w"> </span>status<span
class="w"> </span>--namespace<span class="w"> </span>spark-operator<span
class="w"> </span>my-release
+
+NAME:<span class="w"> </span>my-release
+NAMESPACE:<span class="w"> </span>spark-operator
+STATUS:<span class="w"> </span>deployed
+REVISION:<span class="w"> </span><span class="m">1</span>
+TEST<span class="w"> </span>SUITE:<span class="w"> </span>None
+</pre></div>
+</div>
+<p>Create example Spark application file <code class="docutils literal
notranslate"><span class="pre">spark-pi.yaml</span></code></p>
+<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>apiVersion:<span class="w">
</span>sparkoperator.k8s.io/v1beta2
+kind:<span class="w"> </span>SparkApplication
+metadata:
+<span class="w"> </span>name:<span class="w"> </span>spark-pi
+<span class="w"> </span>namespace:<span class="w"> </span>default
+spec:
+<span class="w"> </span>type:<span class="w"> </span>Scala
+<span class="w"> </span>mode:<span class="w"> </span>cluster
+<span class="w"> </span>image:<span class="w">
</span>ghcr.io/apache/datafusion-comet:spark-3.4-scala-2.12-0.2.0
+<span class="w"> </span>imagePullPolicy:<span class="w"> </span>IfNotPresent
+<span class="w"> </span>mainClass:<span class="w">
</span>org.apache.spark.examples.SparkPi
+<span class="w"> </span>mainApplicationFile:<span class="w">
</span>local:///opt/spark/examples/jars/spark-examples_2.12-3.4.2.jar
+<span class="w"> </span>sparkConf:
+<span class="w"> </span><span
class="s2">"spark.executor.extraClassPath"</span>:<span class="w">
</span><span
class="s2">"/opt/spark/jars/comet-spark-spark3.4_2.12-0.2.0.jar"</span>
+<span class="w"> </span><span
class="s2">"spark.driver.extraClassPath"</span>:<span class="w">
</span><span
class="s2">"/opt/spark/jars/comet-spark-spark3.4_2.12-0.2.0.jar"</span>
+<span class="w"> </span><span
class="s2">"spark.plugins"</span>:<span class="w"> </span><span
class="s2">"org.apache.spark.CometPlugin"</span>
+<span class="w"> </span><span
class="s2">"spark.comet.enabled"</span>:<span class="w"> </span><span
class="s2">"true"</span>
+<span class="w"> </span><span
class="s2">"spark.comet.exec.enabled"</span>:<span class="w">
</span><span class="s2">"true"</span>
+<span class="w"> </span><span
class="s2">"spark.comet.cast.allowIncompatible"</span>:<span
class="w"> </span><span class="s2">"true"</span>
+<span class="w"> </span><span
class="s2">"spark.comet.exec.shuffle.enabled"</span>:<span class="w">
</span><span class="s2">"true"</span>
+<span class="w"> </span><span
class="s2">"spark.comet.exec.shuffle.mode"</span>:<span class="w">
</span><span class="s2">"auto"</span>
+<span class="w"> </span><span class="s2">"conf
spark.shuffle.manager"</span>:<span class="w"> </span><span
class="s2">"org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager"</span>
+<span class="w"> </span>sparkVersion:<span class="w"> </span><span
class="m">3</span>.4.3
+<span class="w"> </span>driver:
+<span class="w"> </span>labels:
+<span class="w"> </span>version:<span class="w"> </span><span
class="m">3</span>.4.3
+<span class="w"> </span>cores:<span class="w"> </span><span
class="m">1</span>
+<span class="w"> </span>coreLimit:<span class="w"> </span>1200m
+<span class="w"> </span>memory:<span class="w"> </span>512m
+<span class="w"> </span>serviceAccount:<span class="w">
</span>spark-operator-spark
+<span class="w"> </span>executor:
+<span class="w"> </span>labels:
+<span class="w"> </span>version:<span class="w"> </span><span
class="m">3</span>.4.3
+<span class="w"> </span>instances:<span class="w"> </span><span
class="m">1</span>
+<span class="w"> </span>cores:<span class="w"> </span><span
class="m">2</span>
+<span class="w"> </span>coreLimit:<span class="w"> </span>1200m
+<span class="w"> </span>memory:<span class="w"> </span>512m
+</pre></div>
+</div>
+<p>Refer to <a class="reference internal" href="#comet-docker-images">Comet
builds</a></p>
+<p>Run Apache Spark application with Comet enabled</p>
+<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>kubectl<span class="w"> </span>apply<span
class="w"> </span>-f<span class="w"> </span>spark-pi.yaml
+</pre></div>
+</div>
+<p>Check application status</p>
+<div class="highlight-bash notranslate"><div
class="highlight"><pre><span></span>kubectl<span class="w">
</span>describe<span class="w"> </span>sparkapplication<span class="w">
</span>--namespace<span class="o">=</span>spark-operator
+</pre></div>
+</div>
+<p>More info on Kube Spark operator
https://www.kubeflow.org/docs/components/spark-operator/getting-started/</p>
+</section>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
