Chris Nauroth commented on YARN-2929:

Hi, [~ozawa].

I would not expect this change to be necessary.  To achieve cross-platform 
application submissions, I believe you would just need to pass a given file 
system path through the {{org.apache.hadoop.fs.Path}} class.  In that class, we 
have implemented logic for handling Windows paths.  Part of that logic replaces 
all back slashes with forward slashes when running on Windows.  For example, 
{{new Path("C:\foo\bar").toString()}} yields the string {{"C:/foo/bar"}} when 
running on Windows.  The forward slash format works fine on both Linux and 

This is different from the classpath separator.  We didn't have any similar 
special handling for that, which is why we needed to implement YARN-1824.

I looked at SPARK-1825, and it seems there is a question about handling of 
SPARK_HOME.  I'd expect this path could differ between client and server.  
Regardless of the issue of path separator, the actual path is likely to be 
different, so it seems like the server side really needs to be responsible for 
injecting this.

I don't have any experience with Spark though, so please let me know if I'm 
missing something.  Thanks!

> Adding separator ApplicationConstants.FILE_PATH_SEPARATOR for better Windows 
> support
> ------------------------------------------------------------------------------------
>                 Key: YARN-2929
>                 URL: https://issues.apache.org/jira/browse/YARN-2929
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-2929.001.patch
> Some frameworks like Spark is tackling to run jobs on Windows(SPARK-1825). 
> For better multiple platform support, we should introduce 
> ApplicationConstants.FILE_PATH_SEPARATOR for making filepath 
> platform-independent.

This message was sent by Atlassian JIRA

Reply via email to