[
https://issues.apache.org/jira/browse/SPARK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan McCarthy updated SPARK-8475:
-----------------------------------
Description:
Spark Submit adds maven central & spark bintray to the ChainResolver before it
adds any external resolvers.
https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L821
When running on a cluster without internet access, this means the spark shell
takes forever to launch as it tries these two remote repos before the ones
specified in the --repositories list. In our case we have a proxy which the
cluster can access it and supply it via --repositories.
This is also a problem for users who maintain a proxy for maven/ivy repos with
something like Nexus/Artifactory. Having a repo proxy is popular at many
organisations so I'd say this would be a useful change for these users as well.
In the current state even if a maven central proxy is supplied, it will still
try and hit central.
I see two options for a fix;
* Change the order repos are added to the ChainResolver, making the
--repositories supplied repos come before anything else.
https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L843
* Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default
true) which when false wont add the maven central & bintry to the
ChainResolver.
Happy to do a PR for this fix.
was:
Spark Submit adds maven central & spark bintray to the ChainResolver before it
adds any external resolvers.
https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L821
When running on a cluster without internet access, this means the spark shell
takes forever to launch as it tries these two remote repos before the ones
specified in the --repositories list.
This is also a problem for users who proxy maven/ivy repos with something like
Nexus/Artifactory.
I see two options for a fix;
* Change the order repos are added to the ChainResolver, making the
--repositories supplied repos come before anything else.
https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L843
* Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default
true) which when false wont add the maven central & bintry to the
ChainResolver.
Happy to do a PR for this fix.
> SparkSubmit with Ivy jars is very slow to load with no internet access
> ----------------------------------------------------------------------
>
> Key: SPARK-8475
> URL: https://issues.apache.org/jira/browse/SPARK-8475
> Project: Spark
> Issue Type: Bug
> Components: Spark Submit
> Affects Versions: 1.4.0
> Reporter: Nathan McCarthy
> Priority: Minor
>
> Spark Submit adds maven central & spark bintray to the ChainResolver before
> it adds any external resolvers.
> https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L821
> When running on a cluster without internet access, this means the spark shell
> takes forever to launch as it tries these two remote repos before the ones
> specified in the --repositories list. In our case we have a proxy which the
> cluster can access it and supply it via --repositories.
> This is also a problem for users who maintain a proxy for maven/ivy repos
> with something like Nexus/Artifactory. Having a repo proxy is popular at many
> organisations so I'd say this would be a useful change for these users as
> well. In the current state even if a maven central proxy is supplied, it will
> still try and hit central.
> I see two options for a fix;
> * Change the order repos are added to the ChainResolver, making the
> --repositories supplied repos come before anything else.
> https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L843
>
> * Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default
> true) which when false wont add the maven central & bintry to the
> ChainResolver.
> Happy to do a PR for this fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
