[
https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532965#comment-14532965
]
Chris Nauroth commented on YARN-3549:
-------------------------------------
bq. Can we not get RawLocalFileSystem to automatically use native fstat if the
native library is available? That way all users can simply benefit from this
seamlessly.
That's an interesting idea. Looking at this more closely,
{{ResourceLocalizationService#checkLocalDir}} really can't use the existing
{{NativeIO.POSIX#getFstat}} method anyway. That one is a passthrough to
{{fstat}}, which operates on an open file descriptor. Instead,
{{ResourceLocalizationService#checkLocalDir}} really wants plain {{stat}} or
maybe {{lstat}}, which operates on a path string. Forcing this code path to
open the file just for the sake of passing an fd to {{fstat}} isn't ideal.
Let's try it! I filed HADOOP-11935 as a pre-requisite to do the Hadoop Common
work.
> use JNI-based FileStatus implementation from
> io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation
> from RawLocalFileSystem in checkLocalDir.
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-3549
> URL: https://issues.apache.org/jira/browse/YARN-3549
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: zhihai xu
> Assignee: zhihai xu
>
> Use JNI-based FileStatus implementation from
> io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation
> from RawLocalFileSystem in checkLocalDir.
> As discussed in YARN-3491, shell-based implementation getPermission runs
> shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow).
> We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in
> RawLocalFileSystem to get rid of shell-based implementation for FileStatus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
