Shardul Mahadik created SPARK-35072:
---------------------------------------
Summary: spark.jars.ivysettings should support local:// and
hdfs:// schemes
Key: SPARK-35072
URL: https://issues.apache.org/jira/browse/SPARK-35072
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.2.0
Reporter: Shardul Mahadik
During reviews of SPARK-34472, there was a desire to support local:// and
hdfs:// schemes and potentially other schemes with spark.jars.ivysettings. See
https://github.com/apache/spark/pull/31591#discussion_r598850998 and
https://github.com/apache/spark/pull/31591#issuecomment-817951152. Currently
this fails with the following error:
{code:java}
./bin/spark-shell --conf
spark.jars.packages="org.apache.commons:commons-lang:3.4" --conf
spark.jars.ivySettings="local:///Use
rs/test/temp/ivysettings.xml"
Exception in thread "main" java.lang.IllegalArgumentException: requirement
failed: Ivy settings file local:/Users/test/temp/ivysettings.xml does not exist
at scala.Predef$.require(Predef.scala:281)
at
org.apache.spark.deploy.SparkSubmitUtils$.loadIvySettings(SparkSubmit.scala:1288)
at
org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:176)
at
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:895)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1031)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1040)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
We should make sure that this also works with {{SparkContext#addJar}} when
running in cluster mode. At this point, only YARN resource manager supports
managing {{ivysettings}} file in cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
