This is an automated email from the ASF dual-hosted git repository.
agrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new b4100018 chore: Add documentation on running benchmarks with Microk8s
(#848)
b4100018 is described below
commit b4100018d37cb273a2245cbdbd6aea988446fbab
Author: Andy Grove <[email protected]>
AuthorDate: Tue Aug 27 08:37:21 2024 -0600
chore: Add documentation on running benchmarks with Microk8s (#848)
* Add initial documentation for k8s
* update docs
* save
* update
* update
* update
* add k8s docs to user guide:
* Update benchmarks/Dockerfile
Co-authored-by: Oleks V <[email protected]>
---------
Co-authored-by: Oleks V <[email protected]>
---
benchmarks/Dockerfile | 23 ++++++++
benchmarks/README.md | 105 +++++++++++++++++++++++++++++++++
docs/source/user-guide/installation.md | 6 +-
docs/source/user-guide/kubernetes.md | 35 +++++++++++
4 files changed, 166 insertions(+), 3 deletions(-)
diff --git a/benchmarks/Dockerfile b/benchmarks/Dockerfile
new file mode 100644
index 00000000..bd05f96d
--- /dev/null
+++ b/benchmarks/Dockerfile
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM apache/datafusion-comet:latest
+
+RUN apt update \
+ && apt install -y git python3 python3-pip \
+ && apt clean
+
+RUN cd /opt \
+ && git clone https://github.com/apache/datafusion-benchmarks.git
diff --git a/benchmarks/README.md b/benchmarks/README.md
new file mode 100644
index 00000000..1042762d
--- /dev/null
+++ b/benchmarks/README.md
@@ -0,0 +1,105 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Running Comet Benchmarks in Microk8s
+
+This guide explains how to run benchmarks derived from TPC-H and TPC-DS in
Apache DataFusion Comet deployed in a
+local Microk8s cluster.
+
+## Use Microk8s locally
+
+Install Micro8s following the instructions at
https://microk8s.io/docs/getting-started and then perform these
+additional steps, ensuring that any existing kube config is backed up first.
+
+```shell
+mkdir -p ~/.kube
+microk8s config > ~/.kube/config
+
+microk8s enable dns
+microk8s enable registry
+
+microk8s kubectl create serviceaccount spark
+```
+
+## Build Comet Docker Image
+
+Run the following command from the root of this repository to build the Comet
Docker image, or use a published
+Docker image from
https://github.com/orgs/apache/packages?repo_name=datafusion-comet
+
+```shell
+docker build -t apache/datafusion-comet -f kube/Dockerfile .
+```
+
+## Build Comet Benchmark Docker Image
+
+Build the benchmark Docker image and push to the Microk8s Docker registry.
+
+```shell
+docker build -t apache/datafusion-comet-tpcbench .
+docker tag apache/datafusion-comet-tpcbench
localhost:32000/apache/datafusion-comet-tpcbench:latest
+docker push localhost:32000/apache/datafusion-comet-tpcbench:latest
+```
+
+## Run benchmarks
+
+```shell
+export SPARK_MASTER=k8s://https://127.0.0.1:16443
+export
COMET_DOCKER_IMAGE=localhost:32000/apache/datafusion-comet-tpcbench:latest
+# Location of Comet JAR within the Docker image
+export COMET_JAR=/opt/spark/jars/comet-spark-spark3.4_2.12-0.2.0-SNAPSHOT.jar
+
+$SPARK_HOME/bin/spark-submit \
+ --master $SPARK_MASTER \
+ --deploy-mode cluster \
+ --name comet-tpcbench \
+ --driver-memory 8G \
+ --conf spark.driver.memory=8G \
+ --conf spark.executor.instances=1 \
+ --conf spark.executor.memory=32G \
+ --conf spark.executor.cores=8 \
+ --conf spark.cores.max=8 \
+ --conf spark.task.cpus=1 \
+ --conf spark.executor.memoryOverhead=3G \
+ --jars local://$COMET_JAR \
+ --conf spark.executor.extraClassPath=$COMET_JAR \
+ --conf spark.driver.extraClassPath=$COMET_JAR \
+ --conf spark.plugins=org.apache.spark.CometPlugin \
+ --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions \
+ --conf spark.comet.enabled=true \
+ --conf spark.comet.exec.enabled=true \
+ --conf spark.comet.exec.all.enabled=true \
+ --conf spark.comet.cast.allowIncompatible=true \
+ --conf spark.comet.exec.shuffle.enabled=true \
+ --conf spark.comet.exec.shuffle.mode=auto \
+ --conf
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
\
+ --conf spark.kubernetes.namespace=default \
+ --conf spark.kubernetes.driver.pod.name=tpcbench \
+ --conf spark.kubernetes.container.image=$COMET_DOCKER_IMAGE \
+ --conf
spark.kubernetes.driver.volumes.hostPath.tpcdata.mount.path=/mnt/bigdata/tpcds/sf100/
\
+ --conf
spark.kubernetes.driver.volumes.hostPath.tpcdata.options.path=/mnt/bigdata/tpcds/sf100/
\
+ --conf
spark.kubernetes.executor.volumes.hostPath.tpcdata.mount.path=/mnt/bigdata/tpcds/sf100/
\
+ --conf
spark.kubernetes.executor.volumes.hostPath.tpcdata.options.path=/mnt/bigdata/tpcds/sf100/
\
+ --conf
spark.kubernetes.authenticate.caCertFile=/var/snap/microk8s/current/certs/ca.crt
\
+ local:///opt/datafusion-benchmarks/runners/datafusion-comet/tpcbench.py \
+ --benchmark tpcds \
+ --data /mnt/bigdata/tpcds/sf100/ \
+ --queries /opt/datafusion-benchmarks/tpcds/queries-spark \
+ --iterations 1
+```
+
diff --git a/docs/source/user-guide/installation.md
b/docs/source/user-guide/installation.md
index c095c213..3c0e4b6d 100644
--- a/docs/source/user-guide/installation.md
+++ b/docs/source/user-guide/installation.md
@@ -32,13 +32,13 @@ Make sure the following requirements are met and software
installed on your mach
- JDK 8 and up
- GLIBC 2.17 (Centos 7) and up
-## Using a Published Docker Image
+## Deploying to Kubernetes
-Docker images are available at
https://github.com/orgs/apache/packages?repo_name=datafusion-comet
+See the [Comet Kubernetes Guide](kubernetes.md) guide.
## Using a Published JAR File
-There are no published binary releases yet.
+There are no published JAR files yet.
## Using a Published Source Release
diff --git a/docs/source/user-guide/kubernetes.md
b/docs/source/user-guide/kubernetes.md
new file mode 100644
index 00000000..01483593
--- /dev/null
+++ b/docs/source/user-guide/kubernetes.md
@@ -0,0 +1,35 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Comet Kubernetes Support
+
+## Comet Docker Images
+
+Run the following command from the root of this repository to build the Comet
Docker image, or use a published
+Docker image from
https://github.com/orgs/apache/packages?repo_name=datafusion-comet
+
+```shell
+docker build -t apache/datafusion-comet -f kube/Dockerfile .
+```
+
+## Example Spark Submit
+
+The exact syntax will vary depending on the Kubernetes distribution, but an
example `spark-submit` command can be
+found [here](https://github.com/apache/datafusion-comet/tree/main/benchmarks).
+
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
