[jira] [Commented] (YARN-3549) use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
[
https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585365#comment-14585365
]
zhihai xu commented on YARN-3549:
-
thanks [~vinodkv] and [~cnauroth]! it makes sense to do the optimization at
{{loadPermissionInfo}}, so all users can benefit from it.
I did some profiling and I find It can speed up 3-4x using NativeIO.POSIX.
{{checkLocalDir}} takes about 15ms with shell-based implementation currently,
it only takes about 4 ms using NativeIO.POSIX. I will make this issue duplicate
as HADOOP-11935.
> use JNI-based FileStatus implementation from
> io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation
> from RawLocalFileSystem in checkLocalDir.
>
>
> Key: YARN-3549
> URL: https://issues.apache.org/jira/browse/YARN-3549
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
>
> Use JNI-based FileStatus implementation from
> io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation
> from RawLocalFileSystem in checkLocalDir.
> As discussed in YARN-3491, shell-based implementation getPermission runs
> shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow).
> We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in
> RawLocalFileSystem to get rid of shell-based implementation for FileStatus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3549) use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
[
https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532965#comment-14532965
]
Chris Nauroth commented on YARN-3549:
-
bq. Can we not get RawLocalFileSystem to automatically use native fstat if the
native library is available? That way all users can simply benefit from this
seamlessly.
That's an interesting idea. Looking at this more closely,
{{ResourceLocalizationService#checkLocalDir}} really can't use the existing
{{NativeIO.POSIX#getFstat}} method anyway. That one is a passthrough to
{{fstat}}, which operates on an open file descriptor. Instead,
{{ResourceLocalizationService#checkLocalDir}} really wants plain {{stat}} or
maybe {{lstat}}, which operates on a path string. Forcing this code path to
open the file just for the sake of passing an fd to {{fstat}} isn't ideal.
Let's try it! I filed HADOOP-11935 as a pre-requisite to do the Hadoop Common
work.
> use JNI-based FileStatus implementation from
> io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation
> from RawLocalFileSystem in checkLocalDir.
>
>
> Key: YARN-3549
> URL: https://issues.apache.org/jira/browse/YARN-3549
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: zhihai xu
>Assignee: zhihai xu
>
> Use JNI-based FileStatus implementation from
> io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation
> from RawLocalFileSystem in checkLocalDir.
> As discussed in YARN-3491, shell-based implementation getPermission runs
> shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow).
> We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in
> RawLocalFileSystem to get rid of shell-based implementation for FileStatus.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3549) use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
[ https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531590#comment-14531590 ] Vinod Kumar Vavilapalli commented on YARN-3549: --- Can we not get RawLocalFileSystem to automatically use native fstat if the native library is available? That way all users can simply benefit from this seamlessly. > use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > > > Key: YARN-3549 > URL: https://issues.apache.org/jira/browse/YARN-3549 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: zhihai xu >Assignee: zhihai xu > > Use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > As discussed in YARN-3491, shell-based implementation getPermission runs > shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow). > We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in > RawLocalFileSystem to get rid of shell-based implementation for FileStatus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3549) use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
[ https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528750#comment-14528750 ] Junping Du commented on YARN-3549: -- bq. but that code already exists. I see. Thanks [~cnauroth] for clarifying on this. > use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > > > Key: YARN-3549 > URL: https://issues.apache.org/jira/browse/YARN-3549 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: zhihai xu >Assignee: zhihai xu > > Use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > As discussed in YARN-3491, shell-based implementation getPermission runs > shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow). > We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in > RawLocalFileSystem to get rid of shell-based implementation for FileStatus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3549) use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
[ https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528725#comment-14528725 ] Chris Nauroth commented on YARN-3549: - Hi [~djp]. This will be a YARN code change in the localizer, so YARN is the appropriate project to track it. The code change will involve calling a native fstat method provided in hadoop-common, but that code already exists, and I don't expect it will need any changes to support this. > use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > > > Key: YARN-3549 > URL: https://issues.apache.org/jira/browse/YARN-3549 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: zhihai xu >Assignee: zhihai xu > > Use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > As discussed in YARN-3491, shell-based implementation getPermission runs > shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow). > We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in > RawLocalFileSystem to get rid of shell-based implementation for FileStatus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3549) use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
[ https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528392#comment-14528392 ] Junping Du commented on YARN-3549: -- Hi [~zxu] and [~cnauroth], this sounds like a change need to happen in hadoop-common project instead of YARN. Shall we move this JIRA from YARN to HADOOP? > use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > > > Key: YARN-3549 > URL: https://issues.apache.org/jira/browse/YARN-3549 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: zhihai xu >Assignee: zhihai xu > > Use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > As discussed in YARN-3491, shell-based implementation getPermission runs > shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow). > We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in > RawLocalFileSystem to get rid of shell-based implementation for FileStatus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3549) use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
[ https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514878#comment-14514878 ] zhihai xu commented on YARN-3549: - Thanks [~cnauroth], It is very nice of you to help test on windows environment. > use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > > > Key: YARN-3549 > URL: https://issues.apache.org/jira/browse/YARN-3549 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: zhihai xu >Assignee: zhihai xu > > Use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > As discussed in YARN-3491, shell-based implementation getPermission runs > shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow). > We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in > RawLocalFileSystem to get rid of shell-based implementation for FileStatus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3549) use JNI-based FileStatus implementation from io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation from RawLocalFileSystem in checkLocalDir.
[ https://issues.apache.org/jira/browse/YARN-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514842#comment-14514842 ] Chris Nauroth commented on YARN-3549: - Hi [~zxu]. Since this is a proposal to call native code, I'd like to make sure test suites are passing on both Linux and Windows when it's ready. The method is implemented for both Linux and Windows, so I do expect it would work fine, but I'd like to make sure. If you don't have access to a Windows VM for testing, I'd be happy to volunteer to test on Windows for you when a patch is ready. Thanks! > use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > > > Key: YARN-3549 > URL: https://issues.apache.org/jira/browse/YARN-3549 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: zhihai xu >Assignee: zhihai xu > > Use JNI-based FileStatus implementation from > io.nativeio.NativeIO.POSIX#getFstat instead of shell-based implementation > from RawLocalFileSystem in checkLocalDir. > As discussed in YARN-3491, shell-based implementation getPermission runs > shell command "ls -ld" to get permission, which take 4 or 5 ms(very slow). > We should switch to io.nativeio.NativeIO.POSIX#getFstat as implementation in > RawLocalFileSystem to get rid of shell-based implementation for FileStatus. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
