Tsuyoshi OZAWA commented on YARN-2929:

Sorry for the shortage of explanations and thanks for your clarification.

I am assuming here that a path originating from a shell script would get passed 
into a Spark Java process, where Spark code would have an opportunity to use 
the Path class like I described.

In fact, this problem happens when launching AM - before launching JVM of the 
For instance, AM need classpath to launch. Sometimes classpath includes 
subdirectory with path separators. This is a case. The path separators cannot 
be parsed on the OS and fails to find jar.

// JVM option in launch-container.sh
-classpath=local/lib/hadoop # cannot be parsed on Windows

For the case, following path is converted into platform depend paths:
// JVM option in launch-container.sh
-classpath=local<FPS>lib<FPS>hadoop # can be converted into platform depend 
paths by expandEnvironments

By minimizing token replacement rules like this, we'd reduce the number of 
special cases that YARN application writers would need to consider.

Yes, I agree with you. If we don't need it, we shouldn't include the 
modifications like this.

> Adding separator ApplicationConstants.FILE_PATH_SEPARATOR for better Windows 
> support
> ------------------------------------------------------------------------------------
>                 Key: YARN-2929
>                 URL: https://issues.apache.org/jira/browse/YARN-2929
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-2929.001.patch
> Some frameworks like Spark is tackling to run jobs on Windows(SPARK-1825). 
> For better multiple platform support, we should introduce 
> ApplicationConstants.FILE_PATH_SEPARATOR for making filepath 
> platform-independent.

This message was sent by Atlassian JIRA

Reply via email to