GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/21476
[SPARK-24446][yarn] Properly quote library path for YARN.
Because the way YARN executes commands via bash -c, everything needs
to be quoted so that the whole command is fully contained inside a
bash string and is interpreted correctly when the string is read by
bash. This is a bit different than the quoting done when executing
things as if typing in a bash shell.
Tweaked unit tests to exercise the bad behavior, which would cause
existing tests to time out without the fix. Also tested on a real
cluster, verifying the shell script created by YARN to run the
container.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark SPARK-24446
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21476.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21476
----
commit 0a3b71f9085d74c9ced427534a8792ec633b0539
Author: Marcelo Vanzin <vanzin@...>
Date: 2018-05-30T23:50:10Z
[SPARK-24446][yarn] Properly quote library path for YARN.
Because the way YARN executes commands via bash -c, everything needs
to be quoted so that the whole command is fully contained inside a
bash string and is interpreted correctly when the string is read by
bash. This is a bit different than the quoting done when executing
things as if typing in a bash shell.
Tweaked unit tests to exercise the bad behavior, which would cause
existing tests to time out without the fix. Also tested on a real
cluster, verifying the shell script created by YARN to run the
container.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
