rickcheng created SPARK-36088:
---------------------------------
Summary: 'spark.archives' does not extract the archive file into
the driver under client mode
Key: SPARK-36088
URL: https://issues.apache.org/jira/browse/SPARK-36088
Project: Spark
Issue Type: Improvement
Components: Kubernetes, Spark Submit
Affects Versions: 3.1.2
Reporter: rickcheng
When running spark in the k8s cluster, there are 2 deploy modes: cluster and
client. After my test, in the cluster mode, *spark.archives* can extract the
archive file to the working directory of the executors and driver. But in
client mode, *spark.archives* can only extract the archive file to the working
directory of the executors.
However, I need *spark.archives* to send the virtual environment tar file
packaged by conda to both the driver and executors under client mode (So that
the executor and the driver have the same python environment).
Why *spark.archives* does not extract the archive file into the working
directory of the driver under client mode?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
