[ 
https://issues.apache.org/jira/browse/YARN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14243590#comment-14243590
 ] 

Tsuyoshi OZAWA commented on YARN-2929:
--------------------------------------

Sorry for the shortage of explanations and thanks for your clarification.

{quote}
I am assuming here that a path originating from a shell script would get passed 
into a Spark Java process, where Spark code would have an opportunity to use 
the Path class like I described.
{quote}

In fact, this problem happens when launching AM - before launching JVM of the 
AM.
For instance, AM need classpath to launch. Sometimes classpath includes 
subdirectory with path separators. This is a case. The path separators cannot 
be parsed on the OS and fails to find jar.

{code}
// JVM option in launch-container.sh
-classpath=local/lib/hadoop # cannot be parsed on Windows
{code}

For the case, following path is converted into platform depend paths:
{code}
// JVM option in launch-container.sh
-classpath=local<FPS>lib<FPS>hadoop # can be converted into platform depend 
paths by expandEnvironments
{code}

{quote}
By minimizing token replacement rules like this, we'd reduce the number of 
special cases that YARN application writers would need to consider.
{quote}

Yes, I agree with you. If we don't need it, we shouldn't include the 
modifications like this.

> Adding separator ApplicationConstants.FILE_PATH_SEPARATOR for better Windows 
> support
> ------------------------------------------------------------------------------------
>
>                 Key: YARN-2929
>                 URL: https://issues.apache.org/jira/browse/YARN-2929
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-2929.001.patch
>
>
> Some frameworks like Spark is tackling to run jobs on Windows(SPARK-1825). 
> For better multiple platform support, we should introduce 
> ApplicationConstants.FILE_PATH_SEPARATOR for making filepath 
> platform-independent.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to