This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-docker.git
The following commit(s) were added to refs/heads/master by this push:
new b69d21d [SPARK-45497] Add a symbolic link file `spark-examples.jar`
in K8s Docker images
b69d21d is described below
commit b69d21da5da1f5d35ea0125188a700c4ca9897bb
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Sep 18 16:09:30 2024 -0700
[SPARK-45497] Add a symbolic link file `spark-examples.jar` in K8s Docker
images
### What changes were proposed in this pull request?
This PR aims to add a symbolic link file, `spark-examples.jar`, in the
example jar directory.
Apache Spark repository is updated already via
- https://github.com/apache/spark/pull/43324
```
$ docker run -it --rm spark:latest ls -al /opt/spark/examples/jars | tail
-n6
total 1620
drwxr-xr-x 1 root root 4096 Oct 11 04:37 .
drwxr-xr-x 1 root root 4096 Sep 9 02:08 ..
-rw-r--r-- 1 root root 78803 Sep 9 02:08 scopt_2.12-3.7.1.jar
-rw-r--r-- 1 root root 1564255 Sep 9 02:08 spark-examples_2.12-3.5.0.jar
lrwxrwxrwx 1 root root 29 Oct 11 04:37 spark-examples.jar ->
spark-examples_2.12-3.5.0.jar
```
### Why are the changes needed?
Like PySpark example (`pi.py`), we can submit the examples without
considering the version numbers which was painful before.
```
bin/spark-submit \
--master k8s://$K8S_MASTER \
--deploy-mode cluster \
...
--class org.apache.spark.examples.SparkPi \
local:///opt/spark/examples/jars/spark-examples.jar 10000
```
The following is the driver pod log.
```
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit ...
--deploy-mode client
--properties-file /opt/spark/conf/spark.properties
--class org.apache.spark.examples.SparkPi
local:///opt/spark/examples/jars/spark-examples.jar 10000
Files local:///opt/spark/examples/jars/spark-examples.jar from
/opt/spark/examples/jars/spark-examples.jar to
/opt/spark/work-dir/./spark-examples.jar
```
### Does this PR introduce _any_ user-facing change?
No, this is an additional file.
### How was this patch tested?
Manually build the docker image and do `ls`.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #67 from dongjoon-hyun/SPARK-45497.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
4.0.0-preview1/scala2.13-java17-ubuntu/Dockerfile | 1 +
Dockerfile.template | 1 +
2 files changed, 2 insertions(+)
diff --git a/4.0.0-preview1/scala2.13-java17-ubuntu/Dockerfile
b/4.0.0-preview1/scala2.13-java17-ubuntu/Dockerfile
index 0a487dd..2685a87 100644
--- a/4.0.0-preview1/scala2.13-java17-ubuntu/Dockerfile
+++ b/4.0.0-preview1/scala2.13-java17-ubuntu/Dockerfile
@@ -59,6 +59,7 @@ RUN set -ex; \
mv sbin /opt/spark/; \
mv kubernetes/dockerfiles/spark/decom.sh /opt/; \
mv examples /opt/spark/; \
+ ln -s "$(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar))"
/opt/spark/examples/jars/spark-examples.jar; \
mv kubernetes/tests /opt/spark/; \
mv data /opt/spark/; \
mv python/pyspark /opt/spark/python/pyspark/; \
diff --git a/Dockerfile.template b/Dockerfile.template
index 3d0aacf..c19e961 100644
--- a/Dockerfile.template
+++ b/Dockerfile.template
@@ -59,6 +59,7 @@ RUN set -ex; \
mv sbin /opt/spark/; \
mv kubernetes/dockerfiles/spark/decom.sh /opt/; \
mv examples /opt/spark/; \
+ ln -s "$(basename $(ls /opt/spark/examples/jars/spark-examples_*.jar))"
/opt/spark/examples/jars/spark-examples.jar; \
mv kubernetes/tests /opt/spark/; \
mv data /opt/spark/; \
mv python/pyspark /opt/spark/python/pyspark/; \
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
