Sandy Ryza created SPARK-3560:
---------------------------------
Summary: In yarn-cluster mode, jars are distributed through
multiple mechanisms.
Key: SPARK-3560
URL: https://issues.apache.org/jira/browse/SPARK-3560
Project: Spark
Issue Type: Bug
Reporter: Sandy Ryza
Priority: Critical
In yarn-cluster mode, jars given to spark-submit's --jars argument should be
distributed to executors through the distributed cache, not through fetching.
Currently, Spark tries to distribute the jars both ways, which can cause
executor errors related to trying to overwrite symlinks without write
permissions.
It looks like this was introduced by SPARK-2260, which sets spark.jars in
yarn-cluster mode. Setting spark.jars is necessary for standalone cluster
deploy mode, but harmful for yarn cluster deploy mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
