steveloughran commented on issue #23938: [MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it URL: https://github.com/apache/spark/pull/23938#issuecomment-469236386 you should compile with hadoop-cloud and add those JARs it pulls in to the spark tarball placed on the shared cluster FS for YARN to pick up. Don't know about other deployment engines I'm afraid. The build also adds it to the SPARK_HOME/lib, which gives it to you for spark-standalone during spark submit, either for anything related to JAR upload, or for any store which implements delegation tokens (HADOOP-14456, HADOOP-16068, etc), so it collects the tokens for all stores listed in spark.yarn.hadoopFilesystems.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
