GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/2232
[SPARK-3319 / 3338] Resolve Spark submit config paths
**SPARK-3319.** There is currently a divergence in behavior when the user
passes in additional jars through `--jars` and through setting `spark.jars` in
the default properties file. The former will happily resolve the paths (e.g.
convert `my.jar` to `file:/absolute/path/to/my.jar`), while the latter does
not. We should resolve paths consistently in both cases. This also applies to
the following pairs of command line arguments and Spark configs:
- `--jars` ~ `spark.jars`
- `--files` ~ `spark.files` / `spark.yarn.dist.files`
- `--archives` ~ `spark.yarn.dist.archives`
- `--py-files` ~ `spark.submit.pyFiles`
**SPARK-3338.** This PR also fixes the following bug: if the user sets
`spark.submit.pyFiles` in his/her properties file, it does not actually get
picked up, even if `--py-files` is not set. This is because it is overridden by
an empty string.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark resolve-config-paths
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2232.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2232
----
commit fe039d35855b4af1635ff7046adccbe639884ed1
Author: Andrew Or <[email protected]>
Date: 2014-08-30T01:58:07Z
Beef up tests to test fixed-pointed-ness of Utils.resolveURI(s)
commit 460117e9744b173f4209f28a6c7dde2fcc519efd
Author: Andrew Or <[email protected]>
Date: 2014-09-01T22:58:16Z
Resolve config paths properly
commit 05e03d649980d5025a1ba53c4c9c27b51e5224cc
Author: Andrew Or <[email protected]>
Date: 2014-09-01T22:58:51Z
Add tests for resolving both command line and config paths
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
