Qianlan Chen created SPARK-59966:
------------------------------------
Summary: Do not append the default remote repository when custom
Ivy settings are used for Hive metastore jars
Key: SPARK-59966
URL: https://issues.apache.org/jira/browse/SPARK-59966
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.4.0
Reporter: Qianlan Chen
After SPARK-56867, `IsolatedClientLoader` respects `spark.jars.ivySettings`
when resolving Hive metastore jars.
However, it reads `spark.sql.maven.additionalRemoteRepositories` using
`SparkConf.get(ConfigEntry)`. This applies the default Google-hosted Maven
Central mirror even when the user did not explicitly configure additional
repositories.
`MavenUtils.loadIvySettings` then appends that repository to the resolver chain
loaded from `ivysettings.xml`.
As a result, `ivysettings.xml` does not fully control which repositories are
contacted. This can cause unexpected network access in environments that intend
to use only private or authenticated repositories.
Expected behavior:
* If `spark.jars.ivySettings` is set and
`spark.sql.maven.additionalRemoteRepositories` is not explicitly set, use only
the resolvers from `ivysettings.xml`.
* If both configurations are explicitly set, append the configured additional
repositories.
* If `spark.jars.ivySettings` is not set, preserve the existing default
repository behavior for compatibility.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
