xkrogen commented on a change in pull request #29874:
URL: https://github.com/apache/spark/pull/29874#discussion_r508095208
##########
File path:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/IsolatedClientLoader.scala
##########
@@ -61,7 +61,8 @@ private[hive] object IsolatedClientLoader extends Logging {
val files = if (resolvedVersions.contains((resolvedVersion,
hadoopVersion))) {
resolvedVersions((resolvedVersion, hadoopVersion))
} else {
- val remoteRepos = sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES)
+ val remoteRepos = sys.env.getOrElse(
+ "DEFAULT_ARTIFACT_REPOSITORY",
sparkConf.get(SQLConf.ADDITIONAL_REMOTE_REPOSITORIES))
Review comment:
@mridulm So it is a bit more complicated than this. We need the options
on the Surefire JVM, not the Maven JVM, so it's actually proper to use a flag
like `-DargLine="-Dspark.sql.maven.additionalRemoteRepositories=..."`. In my
tests this had some negative interaction with the Hive module, which overrides
the default `argLine` with a custom one to disable assertions.
Even if that problem can be solved, the Hive test suite launches _entirely
separate processes_ via `spark-submit` which are unknown to Maven/Surefire, so
even `argLine` is insufficient to pass along configs.
I was able to get it picked up by setting `JAVA_TOOL_OPTIONS="-Dspark...."`,
but this affects every single JVM and is a bit messy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
