uros-b commented on code in PR #59220:
URL: https://github.com/apache/spark/pull/59220#discussion_r4187202327
##########
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala:
##########
@@ -122,12 +122,22 @@ private[hive] object IsolatedClientLoader extends Logging
{
}
}
+ private[hive] def getRemoteRepos(
+ sparkConf: SparkConf,
+ ivySettingsPath: Option[String]): Option[String] = {
+ if (ivySettingsPath.isDefined) {
+ sparkConf.getOption(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES.key)
+ } else {
Review Comment:
The two branches now read the same conf in different ways:
- The default branch (and the pre-PR code on both branches) uses
`sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)`. That goes through
`ConfigReader`, which expands `${env:...}`, `${system:...}` and
`${spark.other.key}` references (`ConfigEntryWithDefaultString.readFrom` ->
`ConfigReader.get` = `conf.get(key).map(substitute)`).
- The custom-settings branch uses `sparkConf.getOption(key)`, which returns
the raw value (`Option(settings.get(key))`) with no substitution.
So, with `spark.jars.ivySettings` set, a value such as
`${env:MAVEN_MIRROR_URL}` that resolved correctly before this PR now reaches
Ivy as a literal repository root. Fix: use presence only as the gate and keep
the typed read:
```
if (ivySettingsPath.isEmpty ||
sparkConf.contains(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))
Some(sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)) else None
```
`SparkConf.contains(entry)` is public and this conf has no alternative keys,
so the presence check behaves the same and all four new assertions still pass.
A fifth test case with a `${spark.some.key}` value would lock it in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
