GitHub user cbvoxel opened a pull request:
https://github.com/apache/spark/pull/8053
[SPARK-5754] [yarn] Spark Driver/Executor cmd exits with error code 1 due
to escaping cmd/jvm
This is my retry to suggest a fix for using Spark on Yarn on Windows. The
former request lacked coding style which I hope to have learned to do better,
and wasn't a true solution as I didn't really understand where the problem came
from. Albeit being still a bit obscure, I can name the "players" and have come
up with a better explaination of why I am suggesting this fix.
I also used @vanzin and @srowen input to *try* to give a more elegant
solution. I am not so sure if that worked out though.
I still hope that this PR is a lot more useful than the last. Also do I
hope that this is a _solution_ to the problem that Spark doesn't work on Yarn
on Windows. With these changes it works (and I can also explain why!).
I still believe that a Unit Test should be included, kind of like the one I
committed the last time. But that was premature, as I want to get the principal
'Go' from @vanzin and @srowen.
Thanks for your time both of you.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cbvoxel/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8053.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8053
----
commit ee3173836b6a7ffc22829eeac745fada79ea85a7
Author: Carsten Blank <[email protected]>
Date: 2015-08-08T22:28:17Z
[SPARK-5754] A reference to an absolute path fails to be in a URI
compatible format on Windows.
commit 431bf188ecdd82ef9d9c32f4e54fee3ddbf83cc6
Author: Carsten Blank <[email protected]>
Date: 2015-08-08T22:29:00Z
[SPARK-5754] The problem was that driver and executors failed in
yarn-client mode using Spark on Yarn on Windows because of obscure escaping
issues. We solve this by exposing org.apache.spark.launcher.CommandBuilderUtils
for proper escaping of arguments in a Windows environment. Main changes are in
YarnSparkHadoopUtil. We differentiate escapeForShell for Windows / Unix usage,
employing CommandBuilderUtils. That is not enough, as we must handle the OOM
JVM handler. Therefore we must also change ExecutorRunnable to use
YarnSparkHadoopUtil.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
