[ 
https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585365#comment-14585365
 ] 

zhihai xu commented on YARN-3549:
---------------------------------

thanks [~vinodkv] and [~cnauroth]! it makes sense to do the optimization at 
{{loadPermissionInfo}}, so all users can benefit from it.
I did some profiling and I find It can speed up 3-4x using NativeIO.POSIX. 
{{checkLocalDir}} takes about 15ms with shell-based implementation currently, 
it only takes about 4 ms using NativeIO.POSIX. I will make this issue duplicate 
as HADOOP-11935.

> use JNI-based FileStatus implementation from 
> io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation 
> from RawLocalFileSystem in checkLocalDir.
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3549
>                 URL: https://issues.apache.org/jira/browse/YARN-3549
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>
> Use JNI-based FileStatus implementation from 
> io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation 
> from RawLocalFileSystem in checkLocalDir.
> As discussed in YARN-3491, shell-based implementation getPermission runs 
> shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow).
> We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in 
> RawLocalFileSystem to get rid of shell-based implementation for FileStatus.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to