[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411719#comment-16411719 ] Chris Douglas commented on HADOOP-14600: bq. Without realizing you had already fixed this issue, I put a patch up in HADOOP-15337 that shows the alternative implementation approach. Sorry again for being so late to the conversation, not intending to step on toes. No worries. If the approach in HADOOP-15337 is more portable, performs comparably, and is equivalent, then let's take the simpler approach. Where applicable, do the unit tests on this issue pass in HADOOP-15337, on Linux/MacOS/Windows? > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java, > command_line_test_result__linux.txt, command_line_test_result__windows.txt > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16411586#comment-16411586 ] Jonathan Eagles commented on HADOOP-14600: -- [~myapachejira] Was taking a look at this exact bug I found this week while running 2.8. Filed HADOOP-15337 to fix, but this is likely the same exactly. It would be great to have this feature fix in 2.8, but I see this JIRA took a vastly different approach that seems more work to backport. Did you compare the posix file system approach implementation approach for speeds and support? Without realizing you had already fixed this issue, I put a patch up in HADOOP-15337 that shows the alternative implementation approach. Sorry again for being so late to the conversation, not intending to step on toes. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu >Priority: Major > Fix For: 3.1.0 > > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java, > command_line_test_result__linux.txt, command_line_test_result__windows.txt > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16275613#comment-16275613 ] Ping Liu commented on HADOOP-14600: --- This is great to hear! Finally, this gets in. Thanks [~chris.douglas]! > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Fix For: 3.1.0 > > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java, > command_line_test_result__linux.txt, command_line_test_result__windows.txt > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16275443#comment-16275443 ] Hudson commented on HADOOP-14600: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13312 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13312/]) HADOOP-14600. LocatedFileStatus constructor forces RawLocalFS to exec a (cdouglas: rev f9d195dfe9cc2c3e4659c3475319ac7c937b5c44) * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java * (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/StatUtils.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/permission/TestFsPermission.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/RawLocalFileSystem.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/permission/FsPermission.java * (edit) hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/io/nativeio/NativeIO.c * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestRawLocalFileSystemContract.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/nativeio/TestNativeIO.java > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Fix For: 3.1.0 > > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java, > command_line_test_result__linux.txt, command_line_test_result__windows.txt > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274962#comment-16274962 ] Chris Douglas commented on HADOOP-14600: Thanks, [~myapachejira]. Glad to have that cleared up. +1 on the latest patch. [~ste...@apache.org], unless you have other feedback, let's leave further refinements to followup JIRAs and commit this. One question: this speeds up calls through {{DeprecatedRawLocalFileStatus}}. Did you look at refactoring the deprecation logic, to see if this class is still necessary? There are multiple checks for the platform and whether the native library is loaded, and not only for {{FileStatus}} operations. This is likely due to accumulated layers of ad hoc improvements and optimizations in {{RawLocalFileSystem}}. At a glance, it looks feasible to cut the number of inline checks substantially. What do you think? > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java, > command_line_test_result__linux.txt, command_line_test_result__windows.txt > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273970#comment-16273970 ] Ping Liu commented on HADOOP-14600: --- Just verified. There is no error! I missed {{-Pnative}} in Maven build that is required profile to generate JNI native code. Now things look good. I tried the patch on IntelliJ in both Windows and Linux and made sure seeing the code flow into the test cases. Also tested command line console. I am attaching the command line test results from both Windows and Linux (see attachments: {{command_line_test_result__linux.txt}}, {{command_line_test_result__windows.txt}}). cc: [~chris.douglas], [~steve_l] > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java, > command_line_test_result__linux.txt, command_line_test_result__windows.txt > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268098#comment-16268098 ] Ping Liu commented on HADOOP-14600: --- Yes, Chris. I am verifying the patch. There is an issue just found tonight in my Linux environment. In TestRawLocalFileSystemContract.testPermission(), the native call failed with {{java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat;}}. I'll look into it further. I guess it is due to my last change. I'll come back with update. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16267246#comment-16267246 ] Chris Douglas commented on HADOOP-14600: bq. replace `listLocatedStatus` call with `listStatusIterator` because it returns FileStatus rather than LocatedFileStatus and that doesn't trigger all the getPermission() mess at all Good point. Unfortunately, for everything accepted by the filter (which defaults to accepting everything, IIRC), we double the RPCs if the client subsequently asks for locations. That's bad for HDFS, but irrelevant to the local FS and object stores that don't report locality information. [~myapachejira], have you had a chance to verify the patch, yet? > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16265476#comment-16265476 ] Ohad Raviv commented on HADOOP-14600: - I'm not sure I was clear enough in the above comment. I totaly agree with your current patch, but I'm saying that the initiator of all this mess from SPARK-21137 is org.apache.hadoop.mapred.FileInputFormat with the following code: {code} for (FileStatus globStat: matches) { if (globStat.isDirectory()) { RemoteIterator iter = fs.listLocatedStatus(globStat.getPath()); while (iter.hasNext()) { LocatedFileStatus stat = iter.next(); if (inputFilter.accept(stat.getPath())) { if (recursive && stat.isDirectory()) { addInputPathRecursively(result, fs, stat.getPath(), inputFilter); } else { result.add(stat); } } } } else { result.add(globStat); } } {code} All I'm suggesting is to replace `listLocatedStatus` call with `listStatusIterator` because it returns FileStatus rather than LocatedFileStatus and that doesn't trigger all the getPermission() mess at all. testing that localy (on Mac) caused the test to run in less than a second instead of about 30 seconds for about 1 files. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264461#comment-16264461 ] Steve Loughran commented on HADOOP-14600: - doing a recursive walk like that will kill performance against object stores though; the API calls which recurse in the FS client allow the connectors to move from a mock-treewalk to a flat list operation. It's best to fix the FileSystem client the way this patch does. If spark does -> that new new loop then know it will be pathological on s3 & bring back SPARK-17593. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16264406#comment-16264406 ] Ohad Raviv commented on HADOOP-14600: - Hi, We also encountered a similar problem as SPARK-21137, when we debugged it we saw that all the time got consumed in the initilazation of LocatedFileStatus when in FileInputFormat tries to get: {code} RemoteIterator iter = fs.listLocatedStatus(globStat.getPath()); {code} However, non of the "special" methods of LocatedFileStatus are used in the loop, so changing it to: {code} for (FileStatus globStat: matches) { if (globStat.isDirectory()) { RemoteIterator iter = fs.listStatusIterator(globStat.getPath()); while (iter.hasNext()) { FileStatus stat = iter.next(); if (inputFilter.accept(stat.getPath())) { if (recursive && stat.isDirectory()) { addInputPathRecursively(result, fs, stat.getPath(), inputFilter); } else { result.add(stat); } } } } else { result.add(globStat); } } {code} actually solved all the problem. I know that it is not the only place that needs to change, but I would like to know what do you think about that? thanks, Ohad. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263924#comment-16263924 ] Chris Douglas commented on HADOOP-14600: lgtm, but if you have cycles to verify the patch, then let's commit it as soon as you +1 it. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263821#comment-16263821 ] Ping Liu commented on HADOOP-14600: --- [~chris.douglas] Finally, this round is green. That's great! Do you still need me verify it? If so, I need learn how to use "git apply " :) > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > HADOOP-14600.009.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263136#comment-16263136 ] Hadoop QA commented on HADOOP-14600: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 34s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 56s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 41s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 19 new + 227 unchanged - 1 fixed = 246 total (was 228) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 50s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 15s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 89m 49s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-14600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12898893/HADOOP-14600.009.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux ece29766f12c 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / de8b6ca | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13737/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13737/testReport/ | | Max. process+thread count | 1570 (vs. ulimit of 5000) | | modules | C: hadoop-common-project/hadoop-common U:
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16260386#comment-16260386 ] Chris Douglas commented on HADOOP-14600: bq. The only question I have is the number of spaces for indentation [...] Oh, as I just read the Oracle/Sun code convention, it says indentation should be four spaces That exception is buried in the contribution [docs|https://wiki.apache.org/hadoop/HowToContribute#line-67]. Hadoop uses 2 spaces instead of 4. Something went horribly awry with Jenkins. Have you been able to test v008 locally? > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257912#comment-16257912 ] Ping Liu commented on HADOOP-14600: --- [~chris.douglas] excellent catch! Your correction is perfect. Yes, {{recursive}} is boolean and will instruct {{Shell.getSetPermissionCommand()}} to get "set permission command" either recursively or not based on the flag. Currently, only non-recursive mode is used. But in the future, recursive mode can be used when needed. Other changes look good. Thanks for detailed changes! The only question I have is the number of spaces for indentation. I notice you are using two spaces in {{StatUtils}}. I was using two-space before as I think this saves space but was told four-space should be used for readability as two-space looks busy. Oh, as I just read the Oracle/Sun code convention, it says indentation should be four spaces. Other than indentation, all else look good. Thanks Chris! > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, HADOOP-14600.007.patch, HADOOP-14600.008.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256301#comment-16256301 ] Hadoop QA commented on HADOOP-14600: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 22s{color} | {color:red} root in trunk failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 23s{color} | {color:red} root in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 22s{color} | {color:red} hadoop-common in trunk failed. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 1m 3s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 22s{color} | {color:red} hadoop-common in trunk failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s{color} | {color:red} hadoop-common in trunk failed. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 21s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 21s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 21s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s{color} | {color:red} root in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 17s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 246 new + 0 unchanged - 0 fixed = 246 total (was 0) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 44s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m 39s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 21s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 23s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-14600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12898099/HADOOP-14600.008.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux f829e2ac21dc 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / e182e77 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | mvninstall | https://builds.apache.org/job/PreCommit-HADOOP-Build/13704/artifact/out/branch-mvninstall-root.txt | | compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/13704/artifact/out/branch-compile-root.txt | | mvnsite | https://builds.apache.org/job/PreCommit-HADOOP-Build/13704/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt | |
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16256234#comment-16256234 ] Hadoop QA commented on HADOOP-14600: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 10s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m 27s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 27s{color} | {color:red} root generated 1 new + 1234 unchanged - 0 fixed = 1235 total (was 1234) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 43s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 19 new + 228 unchanged - 1 fixed = 247 total (was 229) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 49s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 10m 20s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 92m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ha.TestZKFailoverController | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-14600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12898077/HADOOP-14600.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux 774586e826d4 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 0987a7b | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_151 | | findbugs | v3.1.0-RC1 | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/13703/artifact/out/diff-compile-javac-root.txt | | checkstyle |
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239865#comment-16239865 ] Ping Liu commented on HADOOP-14600: --- This time unit test fails on different test case (TestZKFailoverController.testGracefulFailover). But again it is irrelevant to the patch. cc: [~chris.douglas], [~ste...@apache.org] > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239376#comment-16239376 ] Hadoop QA commented on HADOOP-14600: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 5s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 48s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 13m 34s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 13m 34s{color} | {color:red} root generated 1 new + 1238 unchanged - 0 fixed = 1239 total (was 1238) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 44s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 126 new + 227 unchanged - 1 fixed = 353 total (was 228) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 50s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 22s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 99m 24s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ha.TestZKFailoverController | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-14600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12896102/HADOOP-14600.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux c04b08ae0511 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / b6d8f87 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/13629/artifact/out/diff-compile-javac-root.txt | | checkstyle |
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239375#comment-16239375 ] Hadoop QA commented on HADOOP-14600: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m 35s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 35s{color} | {color:red} root generated 1 new + 1238 unchanged - 0 fixed = 1239 total (was 1238) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 40s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 126 new + 228 unchanged - 1 fixed = 354 total (was 229) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 52s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 16s{color} | {color:green} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 89m 12s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | HADOOP-14600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12896102/HADOOP-14600.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux f9dbcd6c0c70 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / b6d8f87 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/13630/artifact/out/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13630/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | Test Results |
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16239347#comment-16239347 ] Ping Liu commented on HADOOP-14600: --- [~chris.douglas] You are right. {{path}} doesn't connect to return value. It should be released regardless the value of {{ret}}. I just added updated new patch. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > HADOOP-14600.006.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236837#comment-16236837 ] Chris Douglas commented on HADOOP-14600: Just skimmed the patch, but this line jumped out: {{noformat}} +cleanup: + if (ret == NULL) { +if (path) + (*env)->ReleaseStringChars(env, j_path, (const jchar*) path); {{noformat}} Shouldn't {{path}} be released if not null, even if {{ret != NULL}} ? Checkstyle output is gone, but it looks like {{Helper.java}} is not indented correctly. Other than that, this looks good. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu >Priority: Major > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235082#comment-16235082 ] Ping Liu commented on HADOOP-14600: --- Can someone have a look at this? As I said before, the unit test failure is irrelevant to this fix. [~ste...@apache.org] > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu >Priority: Major > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, HADOOP-14600.004.patch, HADOOP-14600.005.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16192445#comment-16192445 ] Hadoop QA commented on HADOOP-14600: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 31s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m 52s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 52s{color} | {color:red} root generated 1 new + 1271 unchanged - 0 fixed = 1272 total (was 1271) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 41s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 126 new + 227 unchanged - 1 fixed = 353 total (was 228) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 8m 41s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 3s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 74m 58s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestRaceWhenRelogin | | | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HADOOP-14600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890472/HADOOP-14600.005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux b2d29a3938e2 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / e6e614e | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/13455/artifact/patchprocess/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13455/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt | |
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189239#comment-16189239 ] Hadoop QA commented on HADOOP-14600: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 53s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 34m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 37m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 44s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 28m 17s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 28m 17s{color} | {color:red} root generated 1 new + 1272 unchanged - 0 fixed = 1273 total (was 1272) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 126 new + 227 unchanged - 1 fixed = 353 total (was 228) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 47s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 20s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 40s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}156m 52s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestRaceWhenRelogin | | | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HADOOP-14600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890083/HADOOP-14600.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux e305e8aa1f0c 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d4d2fd1 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/13442/artifact/patchprocess/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13442/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt | |
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186891#comment-16186891 ] Hadoop QA commented on HADOOP-14600: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 36s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m 51s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 51s{color} | {color:red} root generated 5 new + 1289 unchanged - 0 fixed = 1294 total (was 1289) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 41s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 126 new + 227 unchanged - 1 fixed = 353 total (was 228) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 14s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 6s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 30s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 76m 35s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestKDiag | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HADOOP-14600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12889823/HADOOP-14600.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc | | uname | Linux a86b131ab6c1 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 06df6ab | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/13438/artifact/patchprocess/diff-compile-javac-root.txt | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13438/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | unit |
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186875#comment-16186875 ] Ping Liu commented on HADOOP-14600: --- [~ste...@apache.org] Thanks for your detailed code review! Really appreciate it. I just finished making recommended changes and attached *HADOOP-14600.003.patch*. Following are details. {{Helper.java (new class):}} For better supporting unit test, I added some more testing mechanism and moved the logic into a new class called Helper.java. Now I can not only check permission but also change permission. I didn't find a place where we can put the utilities. So just add this one. In case, if one want to added other common utility method. Helper class can be the place. {{TestRawLocalFileSystemContract.java:}} With this addition, I can improve test by adding testPermission() into TestRawLocalFileSystemContract where now both loadPermissionInfoByNativeIO() and loadPermissionInfoByNonNativeIO() can be directly tested as you suggested. I can test it on Linux. But on Windows, sticky bit change doesn't take effect. I guess Windows probably doesn't have sticky bit feature. {{TestNativeIO.java:}} Similarly, doStatTest() was simplified by calling the Helper method. Also improved testStatOnError() by using LambdaTestUtils, improved testMultiThreadedStat() by using ExecutorService and Future, also adding testMultiThreadedStatOnError(). {{RawLocalFileSystem.java:}} A minor issue found (loadPermissionInfo()) is that domain returned with group in Windows. So we need remove domain. This is the same as removing domain from domain/user in existing code. Lastly, for {{NativeIO.c}}, FindFileOwnerAndPermission is not a MSDN function. I found it defined at Line 811 in hadoop-common/src/main/winutils/libwinutils.c. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > HADOOP-14600.003.patch, TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16171960#comment-16171960 ] Steve Loughran commented on HADOOP-14600: - I like where this is going, done another review. Without looking at my first review...sorry if I'm now inconsistent. h3. production code * I'm hoping other people with native build setups can play with this. Any volunteers watching? * {{FsPermission}}: wrap the new octal mask table with tags for formatting in javadocs; document {{mode}} param. h3. {{NativeIO}} * L224, L434, anywhere else (TestFsPermission) add a "." at the end of the first javadoc sentence in the javadocs. Java 8 javadoc expects that. * L455, Raise a {{PathIOException}} with the path, exception string to include the value of nioe.toString. e.g {{throw new PathIOException(path, nioe.toString()), nioe)}} h3. {{NativeIO.c}} * Where is the windows API call {{FindFileOwnerAndPermission}} coming from? I couldn't find it on MSDN. h3. {{RawLocalFileSystem}} * L700: don't bother with the HADOOP- ref in the comments; we use the git logs for that. * but do use javadocs instead. * L707 use {{NativeIO.isAvailable()}} as the check for it being present. * {{loadPermissionByNativeIO}}. Have it rethrow as an IOE & catch that in loadPermissionInfo() at L771. One thing to consider is: if the native IO fails for one instance, is it going to stay broken for everything? That is: after 1 failure, set some static variable so there are no further attempts? I think this is probably over-complex for now, and so just leave as is. h3. TestNativeIO * in the finally clauses, {{fileSystem != null && path != null}} can never be false. Use {{ContractTestUtils.cleanup()}} for a more resilient cleanup. * {{doStatTest}} for all the tests which are assertTrue/False/Not null, include a text message which is meanginful. Maybe include the stat.toString() value * {{testStatOnError}} Use {{LambdaTestUtils.intercept}} for better reporting when an exception isn't raised, checking of the response, etc. {code} LambdaTestUtils.intercept(IOException.class, testFilePath, () -> NativeIO.POSIX.getStat(testFilePath)); {code} {{testMultiThreadedStat()}}. It's good to see some work here, we (I!) undertest the concurrency. * could you use Futures & Callable? That has exception raising built in. Or, if you must use Thread, give it a java 8 closure rather than a subclassing. One thing to consdier is could {{DeprecatedRawLocalFileStatus.loadPermissionInfoByNonNativeIO()}} could be made package scoped, tagged @VisibleForTesting and then used explicitly in {{TestNativeIO.doStatTest()}}? That'd remove the stat command buildup there, and verify that the output of the native and non-native stat calls match exactly,. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160168#comment-16160168 ] Ping Liu commented on HADOOP-14600: --- Changes has been made. *HADOOP-14600.002.patch* is attached. Also added unit tests. CC: [~ste...@apache.org] > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16157223#comment-16157223 ] Ping Liu commented on HADOOP-14600: --- Excellent comments. I'm going to make the suggested changes. For all of those tests that timed out, they don't use the file permission. They test something else. I should have clarified it. Thanks Steve! > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16156841#comment-16156841 ] Steve Loughran commented on HADOOP-14600: - bq. I got lots of cuts and blood with getting mvn test -Dtest=foo running on Window I like that phrase, I'll have to use it. If tests are timing out, that's something a (separate) patch should address. h3. RawLocalFileSystem L678, 687, 696, 751: need to fallback to non-native code if not present; a lot of client uses of the hadoop libs don't use the native libs (including all our tests) we may want to think about actually falling back to the original mechanism if the native IO Fails, e.g {code} if (!isPermissionLoaded() && nativeLibPresent) { try { loadPermissionsByNativeIO() } catch (NativeIOException ex) { LOG.debug("Native call failed", ex) } } if (!isPermissionLoaded()) { // invoked if loadPermissionInfo(); } {code} Probably best to pull this out into some common method "maybeLoadPermissionInfo". Looking at the code, I see we don't appear to be doing any synchronization. Really, if a call is made to get the perm info, it should block other attempts. Making the proposed {{maybeLoadPermissionInfo()}} synchronized would be the easy way to do this L751 don't bother with StringifyException. I know a lot of old code uses it, but its best just to go throw new RuntimeException(" failure in ..." + e, e) this adds e.toString() to the exception text, and includes the nested exception there for code and logs to print. h3. NativeIO.java L451: include the caught exception in the new one L456. given this adds nothing, just rethrow the caught exception h3. NativeIO.c L435. Prefer {{if (path == NULL)}} for consistency. h3. proposed tests: # pass in a null path # query a nonexistent file > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154192#comment-16154192 ] Ping Liu commented on HADOOP-14600: --- I couldn't successfully set up a local environment to run test-patch. So I went to the test result at https://builds.apache.org/job/PreCommit-HADOOP-Build/13153/testReport/ in the above table from [~hadoopqa]. I did manual test on all of five tests as follows. * TestSFTPFileSystem.testStatFile * TestDNS.testDefaultDnsServer * TestRaceWhenRelogin.test * TestKDiag.testKeytabAndPrincipal * TestKDiag.testFileOutput * TestKDiag.testLoadResource But none of the tests hits on the new method *loadPermissionInfoByNativeIO()* in *RawLocalFileSystem* -- *loadPermissionInfoByNativeIO()* is the new code that swaps the original *_loadPermissionInfo()_* and is the only change to the previous version. Additionally, I ran "mvn test -Pnative -Dtest=allNative" on my local environment and found 3 failures and 5 errors. But they are mainly timed out. After giving more time, majority of the tests passed. For TestRPCWaitForProxy.testInterruptedWaitForProxy, it's the only one still generating error after timeout time has been increased. However, manual test on it didn't hit the break point in *loadPermissionInfoByNativeIO()* too. In summary, I didn't find any failed test case for the target new method, *loadPermissionInfoByNativeIO()*. Please let me know if this is enough for the verification or there are more tests to run and how. CC: [~hadoopqa] > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151289#comment-16151289 ] Ping Liu commented on HADOOP-14600: --- Yeah, it must be automatically included with MingW, Visual Studio, or some other installation. Thanks for telling me that! It's good to know all of these especially when using command line. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151275#comment-16151275 ] Allen Wittenauer commented on HADOOP-14600: --- bq. As I checked Program Files, I found there is another Git installation just as you mentioned! It's good to know my spyware is still working. ;) In reality, MingW (which is what that is) comes with the KitWare cmake binary or maybe the git binary package. Most people who go down this path have it installed already so it was a safe hunch. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151255#comment-16151255 ] Ping Liu commented on HADOOP-14600: --- Thanks [~aw]! I have both Cygwin and Git. But Neither has /usr/bin. As I checked Program Files, I found there is another Git installation just as you mentioned! I can try this one. But you are right. I got lots of cuts and blood with getting mvn test -Dtest=foo running on Windows. I'll try to run test-patch on Linux and come back test the same scenario on Windows probably with mvn test -Dtest=foo one by one as a workaround. Thanks! > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151203#comment-16151203 ] Allen Wittenauer commented on HADOOP-14600: --- You actually can run test-patch on Windows. I've got a daily job that runs on the ASF infra now. \Program Files\Git\usr\bin has almost everything you need, outside of cmake, protoc, maven, and whatever features you want. But there are a lot of gotchas. Beyond that, there are a lot of problems with just using mvn test on Windows. It's probably better just to specifically use mvn test -Dtest=foo and test your specific foo at the moment. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151090#comment-16151090 ] Ping Liu commented on HADOOP-14600: --- Now I am trying to do the patch test on my Windows. It looks like dev-support/bin/test-patch is a BASH script and cannot be run on Windows. Is there any guide on how to run it on Windows or the patch test is not expected to be run on Windows? CC: [~aw] > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150948#comment-16150948 ] Hadoop QA commented on HADOOP-14600: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 20s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 39s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch generated 5 new + 156 unchanged - 0 fixed = 161 total (was 156) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 29s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 64m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestRaceWhenRelogin | | | hadoop.fs.sftp.TestSFTPFileSystem | | | hadoop.security.TestKDiag | | | hadoop.net.TestDNS | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | HADOOP-14600 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12884977/HADOOP-14600.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc | | uname | Linux 568b8f8ee110 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 99a7f5d | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/13153/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt | | unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/13153/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/13153/testReport/ | | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common | | Console output |
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150822#comment-16150822 ] Ping Liu commented on HADOOP-14600: --- Hi [~jzhuge], thanks for your help! The patch and the test file are now attached. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > Attachments: HADOOP-14600.001.patch, > TestRawLocalFileSystemContract.java > > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150811#comment-16150811 ] John Zhuge commented on HADOOP-14600: - Please follow the guidelines in https://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch. The patch file name should be: *HADOOP-14600.001.patch*. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150810#comment-16150810 ] John Zhuge commented on HADOOP-14600: - Hi [~myapachejira], added you to the contributor list and assigned the jira to you. Thank you for working on the issue. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran >Assignee: Ping Liu > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150254#comment-16150254 ] Ping Liu commented on HADOOP-14600: --- Oops, looks like I don't have permission to attach files. I'll see if I can request the permission from the mailing list. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150245#comment-16150245 ] Ping Liu commented on HADOOP-14600: --- I just followed [~steve_l]'s idea to add stat() native implementation. Yes, it is similar to fstat() but doesn't need open file as it doesn't require a file descriptor. Now there is no need to spawn extra thread to gather process info any more. I did some manual test on both Windows 10 and Linux (Ubuntu on VirtualBox). It looks like it has dramatic improvement on both systems. {noformat} Windows number of files time (ms) time (ms) with native IO 100 14274 1234 150 19002 1782 200 21865 2250 500 timed out 5125 1000timed out 9735 2000timed out 18875 Linux number of files time (ms) time (ms) with native IO 100 45391632 150 61372031 200 71392764 500 15566 5292 1000timed out 7490 2000timed out 14040 {noformat} The test is primitive but sufficiently shows the improvement. Attached is the patch file: *HADOOP-14600__Patch__20170901.txt*. When doing the test, I added testListStatusForPerformance() to TestRawLocalFileSystem.java. Also attached above. > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066510#comment-16066510 ] Steve Loughran commented on HADOOP-14600: - no, fstat() doesn't work, as it needs an open FD, and we don't want to call `open()` here: that could require permissions which the caller may not actually have. We need the `stat(path)` call, which isn't in the package yet. Any volunteers to add it? > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
[ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065441#comment-16065441 ] Steve Loughran commented on HADOOP-14600: - {{NativeIO.POSIX.getFstat}} should handle all of this, cross platform. Testable > LocatedFileStatus constructor forces RawLocalFS to exec a process to get the > permissions > > > Key: HADOOP-14600 > URL: https://issues.apache.org/jira/browse/HADOOP-14600 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.7.3 > Environment: file:// in a dir with many files >Reporter: Steve Loughran > > Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws > against the local FS, because {{FileStatus.getPemissions}} call forces > {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI > values. > That is: for every other FS, what's a field lookup or even a no-op, on the > local FS it's a process exec/spawn, with all the costs. This gets expensive > if you have many files. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org