Github user cbvoxel commented on the pull request:
https://github.com/apache/spark/pull/8053#issuecomment-130836117
@vanzin: no it doesn't seem to be needed. It seems that
`quoteForBatchScript` works correctly with arguments entered on the command
line. Problems arise with called cmd files though.
What I see is that in cmd files every `%` needs to be escaped by another
`%`. This really doesn't happen on the command line! I have an example (use of
grep: I have MinGW msys installed):
### Using the command line ###
> java "-XX:OnOutOfMemoryError=taskkill /F /PID %%%%p" -XX:+PrintFlagsFinal
-version | grep OnOutOfMemoryError
> bool HeapDumpOnOutOfMemoryError = false
{manageable}
> ccstrlist OnOutOfMemoryError := taskkill /F /PID
%%%%p{product}
> java version "1.7.0_79"
> Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
> Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
### Using the a .cmd file ###
> sparkCmdTest.cmd
> bool HeapDumpOnOutOfMemoryError = false
{manageable}
> ccstrlist OnOutOfMemoryError := taskkill /F /PID
%%p{product}
> java version "1.7.0_79"
> Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
> Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
where `sparkCmdTest.cmd` has this one line:
> java "-XX:OnOutOfMemoryError=taskkill /F /PID %%%%p" -XX:+PrintFlagsFinal
-version | grep OnOutOfMemoryError
Now what made me wonder is, that for the executor to work we somehow need
four (!) `%`. I would 'guess' that somewhere along the line there is another
cmd file being called, and that with the first pass `%%%%` is reduced to `%%`
and then to `%`. Acutally I have no solid idea, all I can say that if I
executed the cmd file spark passes to Yarn, it *must* be four `%`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
