[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Status: Open (was: Patch Available) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.0-alpha, 1.0.3 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, trunk-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: trunk-HADOOP-8649.patch branch1-HADOOP-8649.patch Uploading updated patches for branch1 and trunk. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: trunk-HADOOP-8649.patch branch1-HADOOP-8649.patch Uploading patches for trunk and branch-1 addressing Daryn's comments. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Affects Version/s: 1.0.3 2.0.0-alpha Status: Patch Available (was: Open) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.0-alpha, 1.0.3 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Status: Open (was: Patch Available) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: TestChecksumFileSystemOnDFS.java Sorry for the false alarm. As per Daryn's suggestion, I wrote a test to check the same that I am uploading here. Daryn's description of the flow is right, and there is no bug. Sorry again. Also, as Daryn commented, using a composite fiter would improve the performance. I ll update the description of the JIRA to reflect the same and upload patches for branch-1 and test including this test. Thanks again for your thorough review, Daryn. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Description: Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. was: Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. Issue Type: Improvement (was: Bug) Summary: ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance (was: ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Priority: Major (was: Blocker) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch Uploading patch from branch-1. The patch: - implements ChecksumFileSystem#listStatus(Path, PathFilter) - adds test for listStatus in TestChecksumFileSystem - cleans up Test file to use junit4. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch_v2 Updated patch: - Fixes invocation order in joinFilter - FileSystem#listStatus() checks for null PathFilter - TestFileSystem has a new test for the same. New test passes, but another test - TestFileSystem#testFS - fails. Unable to find why. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2 Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: (was: HADOOP-8649_branch1.patch_v2) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch_v2 Updated patch to fix wrong placement of null check in FileSystem. TestFileSystem#writeTest still fails. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2 Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch_v3 Turns out ChecksumFileSystem#listStatus() was buggy and was causing TestFileSystem to fail. Updated the patch accordingly. - Fixed ChecksumFileSystem#listStatus() - TestFileSystem and TestChecksumFileSystem pass just fine. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3 Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
