[
https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154192#comment-16154192
]
Ping Liu commented on HADOOP-14600:
-----------------------------------
I couldn't successfully set up a local environment to run test-patch. So I
went to the test result at
https://builds.apache.org/job/PreCommit-HADOOP-Build/13153/testReport/ in the
above table from [~hadoopqa].
I did manual test on all of five tests as follows.
* TestSFTPFileSystem.testStatFile
* TestDNS.testDefaultDnsServer
* TestRaceWhenRelogin.test
* TestKDiag.testKeytabAndPrincipal
* TestKDiag.testFileOutput
* TestKDiag.testLoadResource
But none of the tests hits on the new method *loadPermissionInfoByNativeIO()*
in *RawLocalFileSystem* -- *loadPermissionInfoByNativeIO()* is the new code
that swaps the original *_loadPermissionInfo()_* and is the only change to the
previous version.
Additionally, I ran "mvn test -Pnative -Dtest=allNative" on my local
environment and found 3 failures and 5 errors.
But they are mainly timed out. After giving more time, majority of the tests
passed. For TestRPCWaitForProxy.testInterruptedWaitForProxy, it's the only
one still generating error after timeout time has been increased. However,
manual test on it didn't hit the break point in
*loadPermissionInfoByNativeIO()* too.
In summary, I didn't find any failed test case for the target new method,
*loadPermissionInfoByNativeIO()*. Please let me know if this is enough for the
verification or there are more tests to run and how.
CC: [~hadoopqa]
> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the
> permissions
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-14600
> URL: https://issues.apache.org/jira/browse/HADOOP-14600
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
> Affects Versions: 2.7.3
> Environment: file:// in a dir with many files
> Reporter: Steve Loughran
> Assignee: Ping Liu
> Attachments: HADOOP-14600.001.patch,
> TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws
> against the local FS, because {{FileStatus.getPemissions}} call forces
> {{DeprecatedRawLocalFileStatus}} tp spawn a process to read the real UGI
> values.
> That is: for every other FS, what's a field lookup or even a no-op, on the
> local FS it's a process exec/spawn, with all the costs. This gets expensive
> if you have many files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
