[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[
https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13435110#comment-13435110
]
Daryn Sharp commented on HADOOP-8649:
-
I'm just generally concerned about the implications of stacking filesystems.
Ie. a {{FilterFileSystem}} over a {{ChRootedFileSystem}} over a
{{FilterFileSystem}}, etc. I'm not sure it's a problem, but you should make
sure there are tests that prove the stacking works.
I conceptually like the approach suggested. Throw something up and let's see
how it looks!
ChecksumFileSystem should have an overriding implementation of
listStatus(Path, PathFilter) for improved performance
Key: HADOOP-8649
URL: https://issues.apache.org/jira/browse/HADOOP-8649
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch,
HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch,
HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3,
TestChecksumFileSystemOnDFS.java, trunk-HADOOP-8649.patch,
trunk-HADOOP-8649.patch
Currently, ChecksumFileSystem implements only listStatus(Path).
The other form of listStatus(Path, customFilter) results in parsing the list
twice to apply each of the filters - custom and checksum filter.
By using a composite filter instead, we limit the parsing to once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[
https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433220#comment-13433220
]
Daryn Sharp commented on HADOOP-8649:
-
You may want to test if there's any incompatibilities with the chrooted
filesystem. If so, I wonder if it would be better as in more generalized, to
push the change down into {{FilterFileSystem}} or {{FileSystem}} itself.
Haven't thought it all the way through, but a compound filter may use an array
and each filesystem is given the opportunity to add additional filters.
If there's no problem with chroot, and you feel that's too much work, perhaps
it could be something for another jira.
ChecksumFileSystem should have an overriding implementation of
listStatus(Path, PathFilter) for improved performance
Key: HADOOP-8649
URL: https://issues.apache.org/jira/browse/HADOOP-8649
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch,
HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch,
HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3,
TestChecksumFileSystemOnDFS.java, trunk-HADOOP-8649.patch,
trunk-HADOOP-8649.patch
Currently, ChecksumFileSystem implements only listStatus(Path).
The other form of listStatus(Path, customFilter) results in parsing the list
twice to apply each of the filters - custom and checksum filter.
By using a composite filter instead, we limit the parsing to once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[
https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433541#comment-13433541
]
Karthik Kambatla commented on HADOOP-8649:
--
Thanks for the review, Daryn.
- I don't think it is incompatible with ChRootedFileSystem as it does not
filter out any files.
- +1 on generalizing and pushing the change down to FileSystem itself.
-- We can add {{protected/public FileSystem#listStatus(Path f, ListPathFilter
filters)}} and use {{MultiPathFilter}} as in {{o.a.h.m.FileInputFormat}}
-- All FileSystems can use this to build a list of {{PathFilter}}s to be
evaluated.
-- {{o.a.h.m.FileInputFormat}} can use the common version of {{MultiPathFilter}}
If we decide on this, I can go ahead and make the required changes.
ChecksumFileSystem should have an overriding implementation of
listStatus(Path, PathFilter) for improved performance
Key: HADOOP-8649
URL: https://issues.apache.org/jira/browse/HADOOP-8649
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch,
HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch,
HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3,
TestChecksumFileSystemOnDFS.java, trunk-HADOOP-8649.patch,
trunk-HADOOP-8649.patch
Currently, ChecksumFileSystem implements only listStatus(Path).
The other form of listStatus(Path, customFilter) results in parsing the list
twice to apply each of the filters - custom and checksum filter.
By using a composite filter instead, we limit the parsing to once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13431992#comment-13431992 ] Hadoop QA commented on HADOOP-8649: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540032/trunk-HADOOP-8649.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1272//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1272//console This message is automatically generated. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13432021#comment-13432021 ] Karthik Kambatla commented on HADOOP-8649: -- I don't think the patch has anything to do with the two failing tests, these tests fail on the latest trunk as well. Casual code examination shows no intersection between the patch and failing tests. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13431337#comment-13431337 ] Hadoop QA commented on HADOOP-8649: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539893/trunk-HADOOP-8649.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.fs.TestLocalFSFileContextMainOperations org.apache.hadoop.fs.TestFileContextDeleteOnExit org.apache.hadoop.fs.TestFSMainOperationsLocalFileSystem org.apache.hadoop.hdfs.web.TestWebHDFS org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1266//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1266//console This message is automatically generated. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13431389#comment-13431389 ] Karthik Kambatla commented on HADOOP-8649: -- Found the javadoc warning. The noticed test failures seem to be due to one of the patch's tests creating a file and not deleting it. Running all the tests locally to make sure these issues are fixed. Will upload updated patch soon. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
