xkrogen commented on pull request #31694:
URL: https://github.com/apache/spark/pull/31694#issuecomment-789149154
The `CliSuite` test passed in the new run, but the modified `SQLQuerySuite`
test failed. Not sure why that wasn't showing up in the first test execution.
I dug a bit deeper and it turns out what I said in the description wasn't
quite right (will update now). The issue is that the behavior of
`IvySettings#getDefaultIvyUserDir` is different from that of `IvySettings#load`
when it comes to `ivy.home` and `ivy.default.ivy.user.dir`. For
`getDefaultIvyUserDir`, only `ivy.home` is used, even though the logs claim it
is using `ivy.default.ivy.user.dir`:
```
public synchronized File getDefaultIvyUserDir() {
if (defaultUserDir == null) {
if (getVariable("ivy.home") != null) {
setDefaultIvyUserDir(Checks.checkAbsolute(getVariable("ivy.home"), "ivy.home"));
Message.verbose("using ivy.default.ivy.user.dir variable for
default ivy user dir: "
+ defaultUserDir);
} else {
setDefaultIvyUserDir(new
File(System.getProperty("user.home"), ".ivy2"));
Message.verbose("no default ivy user dir defined: set to " +
defaultUserDir);
}
}
return defaultUserDir;
}
```
But for `load`, `ivy.default.ivy.user.dir` is checked, and if present, it is
used to set the value `ivy.home` (within `setDefaultIvyUserDir`):
```
public synchronized void load(File settingsFile) throws ParseException,
IOException {
...
if (getVariable("ivy.default.ivy.user.dir") != null) {
setDefaultIvyUserDir(Checks.checkAbsolute(getVariable("ivy.default.ivy.user.dir"),
"ivy.default.ivy.user.dir"));
} else {
getDefaultIvyUserDir();
}
...
}
```
So for the `new IvySettings` case, `ivy.home` takes precedence, but for the
`(new IvySettings).load(...)` case, `ivy.default.ivy.user.dir` takes
precedence. In our environment we configure a custom Ivy settings file so I was
only testing the latter case initially.
I have updated the patch to leave the system property as `ivy.home`. The
test will still break if `spark.jars.ivySettings` and
`ivy.default.ivy.user.dir` are used in tandem, but this should be a corner case
and I don't think there is a way to resolve it further without modifying Ivy
itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
