Nathan McCarthy created SPARK-8475:
--------------------------------------
Summary: SparkSubmit with Ivy jars is very slow to load when no
internet access
Key: SPARK-8475
URL: https://issues.apache.org/jira/browse/SPARK-8475
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 1.4.0
Reporter: Nathan McCarthy
Priority: Minor
Spark Submit adds maven central & spark bintray to the ChainResolver before it
adds any external resolvers.
When running on a cluster without internet access, this means the spark shell
takes forever to launch as it tries these two remote repos before the ones
specified in the --repositories list.
This is also a problem for users who proxy maven/ivy repos with something like
Nexus/Artifactory.
I see two options for a fix;
* Change the order repos are added to the ChainResolver, making the
--repositories supplied repos come before anything else.
* Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default
true) which when false wont add the maven central & bintry to the
ChainResolver.
Happy to do a PR for this fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
