[jira] [Created] (HADOOP-8552) Conflict: Same security.log.file for multiple users.
Karthik Kambatla created HADOOP-8552: Summary: Conflict: Same security.log.file for multiple users. Key: HADOOP-8552 URL: https://issues.apache.org/jira/browse/HADOOP-8552 Project: Hadoop Common Issue Type: Bug Components: conf, security Affects Versions: 2.0.0-alpha, 1.0.3 Reporter: Karthik Kambatla In log4j.properties, hadoop.security.log.file is set to SecurityAuth.audit. In the presence of multiple users, this can lead to a potential conflict. Adding username to the log file would avoid this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8552) Conflict: Same security.log.file for multiple users.
[ https://issues.apache.org/jira/browse/HADOOP-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13406131#comment-13406131 ] Karthik Kambatla commented on HADOOP-8552: -- Hi Suresh, Thanks for looking into this. The problem we came across was -- at times, multiple (2 in our case) users might write to the same file (at least attempt to open the file) simultaneously, because the hadoop.security.log.file is set to the same value. Please let me know if I am missing something here. Thanks Conflict: Same security.log.file for multiple users. - Key: HADOOP-8552 URL: https://issues.apache.org/jira/browse/HADOOP-8552 Project: Hadoop Common Issue Type: Bug Components: conf, security Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Attachments: HADOOP-8552_branch1.patch, HADOOP-8552_branch2.patch In log4j.properties, hadoop.security.log.file is set to SecurityAuth.audit. In the presence of multiple users, this can lead to a potential conflict. Adding username to the log file would avoid this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8552) Conflict: Same security.log.file for multiple users.
[ https://issues.apache.org/jira/browse/HADOOP-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8552: - Attachment: HADOOP-8552_branch2.patch HADOOP-8552_branch1.patch I am uploading patches for branch-1 and branch-2. Conflict: Same security.log.file for multiple users. - Key: HADOOP-8552 URL: https://issues.apache.org/jira/browse/HADOOP-8552 Project: Hadoop Common Issue Type: Bug Components: conf, security Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Attachments: HADOOP-8552_branch1.patch, HADOOP-8552_branch2.patch In log4j.properties, hadoop.security.log.file is set to SecurityAuth.audit. In the presence of multiple users, this can lead to a potential conflict. Adding username to the log file would avoid this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8552) Conflict: Same security.log.file for multiple users.
[ https://issues.apache.org/jira/browse/HADOOP-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned HADOOP-8552: Assignee: Karthik Kambatla Conflict: Same security.log.file for multiple users. - Key: HADOOP-8552 URL: https://issues.apache.org/jira/browse/HADOOP-8552 Project: Hadoop Common Issue Type: Bug Components: conf, security Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8552_branch1.patch, HADOOP-8552_branch2.patch In log4j.properties, hadoop.security.log.file is set to SecurityAuth.audit. In the presence of multiple users, this can lead to a potential conflict. Adding username to the log file would avoid this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8552) Conflict: Same security.log.file for multiple users.
[ https://issues.apache.org/jira/browse/HADOOP-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8552: - Status: Patch Available (was: Open) Conflict: Same security.log.file for multiple users. - Key: HADOOP-8552 URL: https://issues.apache.org/jira/browse/HADOOP-8552 Project: Hadoop Common Issue Type: Bug Components: conf, security Affects Versions: 2.0.0-alpha, 1.0.3 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8552_branch1.patch, HADOOP-8552_branch2.patch In log4j.properties, hadoop.security.log.file is set to SecurityAuth.audit. In the presence of multiple users, this can lead to a potential conflict. Adding username to the log file would avoid this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8552) Conflict: Same security.log.file for multiple users.
[ https://issues.apache.org/jira/browse/HADOOP-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8552: - Attachment: HADOOP-8552_branch1.patch Updating the patch after testing. Tested it on a secure cluster, and the appropriate log file is created. Conflict: Same security.log.file for multiple users. - Key: HADOOP-8552 URL: https://issues.apache.org/jira/browse/HADOOP-8552 Project: Hadoop Common Issue Type: Bug Components: conf, security Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8552_branch1.patch, HADOOP-8552_branch2.patch In log4j.properties, hadoop.security.log.file is set to SecurityAuth.audit. In the presence of multiple users, this can lead to a potential conflict. Adding username to the log file would avoid this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8552) Conflict: Same security.log.file for multiple users.
[ https://issues.apache.org/jira/browse/HADOOP-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8552: - Attachment: (was: HADOOP-8552_branch1.patch) Conflict: Same security.log.file for multiple users. - Key: HADOOP-8552 URL: https://issues.apache.org/jira/browse/HADOOP-8552 Project: Hadoop Common Issue Type: Bug Components: conf, security Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8552_branch1.patch, HADOOP-8552_branch2.patch In log4j.properties, hadoop.security.log.file is set to SecurityAuth.audit. In the presence of multiple users, this can lead to a potential conflict. Adding username to the log file would avoid this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8552) Conflict: Same security.log.file for multiple users.
[ https://issues.apache.org/jira/browse/HADOOP-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413317#comment-13413317 ] Karthik Kambatla commented on HADOOP-8552: -- Devaraj, thanks for the feedback. It is both on the client/server side. By server side, I mean for the jobtracker/namenode. Thanks for pointing the potential compatibility issue, I agree we need to note the incompatibility in log file change. Conflict: Same security.log.file for multiple users. - Key: HADOOP-8552 URL: https://issues.apache.org/jira/browse/HADOOP-8552 Project: Hadoop Common Issue Type: Bug Components: conf, security Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8552_branch1.patch, HADOOP-8552_branch2.patch In log4j.properties, hadoop.security.log.file is set to SecurityAuth.audit. In the presence of multiple users, this can lead to a potential conflict. Adding username to the log file would avoid this scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8568) DNS#reverseDns fails on IPv6 addresses
[
https://issues.apache.org/jira/browse/HADOOP-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla reassigned HADOOP-8568:
Assignee: Karthik Kambatla
DNS#reverseDns fails on IPv6 addresses
--
Key: HADOOP-8568
URL: https://issues.apache.org/jira/browse/HADOOP-8568
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Karthik Kambatla
Labels: newbie
DNS#reverseDns assumes hostIp is a v4 address (4 parts separated by dots),
blows up if given a v6 address:
{noformat}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
at org.apache.hadoop.net.DNS.reverseDns(DNS.java:79)
at org.apache.hadoop.net.DNS.getHosts(DNS.java:237)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:340)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:358)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:337)
at org.apache.hadoop.hbase.master.HMaster.init(HMaster.java:235)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1649)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8638) TestUlimit fails locally (on some machines)
Karthik Kambatla created HADOOP-8638: Summary: TestUlimit fails locally (on some machines) Key: HADOOP-8638 URL: https://issues.apache.org/jira/browse/HADOOP-8638 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.3 Environment: Linux 2.6.32-71.14.1.el6.x86_64 java version 1.6.0_26 Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) Reporter: Karthik Kambatla Attachments: test-ulimit-out ant clean test -Dtestcase=TestUlimit -Dtest.output=yes fails locally Attaching the dump. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8638) TestUlimit fails locally (on some machines)
[ https://issues.apache.org/jira/browse/HADOOP-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8638: - Attachment: test-ulimit-out TestUlimit fails locally (on some machines) --- Key: HADOOP-8638 URL: https://issues.apache.org/jira/browse/HADOOP-8638 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.3 Environment: Linux 2.6.32-71.14.1.el6.x86_64 java version 1.6.0_26 Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) Reporter: Karthik Kambatla Attachments: test-ulimit-out ant clean test -Dtestcase=TestUlimit -Dtest.output=yes fails locally Attaching the dump. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8638) TestUlimit fails locally (on some machines)
[ https://issues.apache.org/jira/browse/HADOOP-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13426255#comment-13426255 ] Karthik Kambatla commented on HADOOP-8638: -- Interestingly, the exit code from the default task controller is different across machines, I have seen 1 or 134 so far. TestUlimit fails locally (on some machines) --- Key: HADOOP-8638 URL: https://issues.apache.org/jira/browse/HADOOP-8638 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.3 Environment: Linux 2.6.32-71.14.1.el6.x86_64 java version 1.6.0_26 Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) Reporter: Karthik Kambatla Attachments: test-ulimit-out ant clean test -Dtestcase=TestUlimit -Dtest.output=yes fails locally Attaching the dump. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8638) TestUlimit fails locally (on some machines)
[ https://issues.apache.org/jira/browse/HADOOP-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned HADOOP-8638: Assignee: Karthik Kambatla TestUlimit fails locally (on some machines) --- Key: HADOOP-8638 URL: https://issues.apache.org/jira/browse/HADOOP-8638 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.3 Environment: Linux 2.6.32-71.14.1.el6.x86_64 java version 1.6.0_26 Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: test-ulimit-out ant clean test -Dtestcase=TestUlimit -Dtest.output=yes fails locally Attaching the dump. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8638) TestUlimit fails locally (on some machines)
[ https://issues.apache.org/jira/browse/HADOOP-8638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved HADOOP-8638. -- Resolution: Fixed MAPREDUCE-4036 addresses the same issue - the corresponding patch there seems to solve the issue on the local machine. We might have to re-open this should anyone experience the same in other (new) environments. TestUlimit fails locally (on some machines) --- Key: HADOOP-8638 URL: https://issues.apache.org/jira/browse/HADOOP-8638 Project: Hadoop Common Issue Type: Bug Affects Versions: 1.0.3 Environment: Linux 2.6.32-71.14.1.el6.x86_64 java version 1.6.0_26 Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: test-ulimit-out ant clean test -Dtestcase=TestUlimit -Dtest.output=yes fails locally Attaching the dump. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
Karthik Kambatla created HADOOP-8649: Summary: ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned HADOOP-8649: Assignee: Karthik Kambatla ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Priority: Major (was: Blocker) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch Uploading patch from branch-1. The patch: - implements ChecksumFileSystem#listStatus(Path, PathFilter) - adds test for listStatus in TestChecksumFileSystem - cleans up Test file to use junit4. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429259#comment-13429259 ] Karthik Kambatla commented on HADOOP-8649: -- Thanks for the review, Daryn. Great points, I have missed them. bq. I question the null check, although useful, because FileSystems doesn't allow a null filter. This would make a FilterFileSystem behave differently. FileSystem itself should probably be changed to ignore a null since it appears to cause a NPE. Agree. FileSystem should check for null. However, I believe ChecksumFileSystem should also check for a null, otherwise the joinFilter would be non-null and have a constituent null filter. Fixed the invocation order - will upload the new patch soon. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch_v2 Updated patch: - Fixes invocation order in joinFilter - FileSystem#listStatus() checks for null PathFilter - TestFileSystem has a new test for the same. New test passes, but another test - TestFileSystem#testFS - fails. Unable to find why. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2 Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429293#comment-13429293 ] Karthik Kambatla commented on HADOOP-8649: -- Wrong placement of null check in FileSystem. Will fix it, the test and update the patch. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2 Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: (was: HADOOP-8649_branch1.patch_v2) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch_v2 Updated patch to fix wrong placement of null check in FileSystem. TestFileSystem#writeTest still fails. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2 Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch_v3 Turns out ChecksumFileSystem#listStatus() was buggy and was causing TestFileSystem to fail. Updated the patch accordingly. - Fixed ChecksumFileSystem#listStatus() - TestFileSystem and TestChecksumFileSystem pass just fine. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3 Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[
https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13429459#comment-13429459
]
Karthik Kambatla commented on HADOOP-8649:
--
Hi Daryn,
The trunk code seems to be the roughly the same as branch-1. So, I believe both
have the bug.
Let me illustrate the issue that I see through some psuedo-code. Let me know if
I am missing something here.
{code}
DistributedFileSystem dfs = ... // some initialization of distributed file
system.
ChecksumFileSystem cfs = new ChecksumFileSystem(dfs); // cfs.fs is set to dfs.
Path randomPath = new Path(random-path); // some path with 'random' in it.
PathFilter randomFilter = new PathFilter() {
boolean accept(Path file) {return !file.toString().contains(random);}
};
FileStatus[] listWithoutFilter = cfs.listStatus(randomPath); // in turn calls
dfs.listStatus(randomPath, ChecksumFileSystem.DEFAULT_FILTER)
FileStatus[] listWithFilter = cfs.listStatus(randomPath, randomFilter); // in
turn calls dfs.listStatus(randomPath, randomFilter)
{code}
dfs.listStatus(Path, PathFilter) calls FileSystem.listStatus(Path, PathFilter),
which first calls dfs.listStatus(path) and then applies PathFilter. Hence,
while checksum filter is used in the first cfs.listStatus, it is not used in
the second call to listStatus().
ChecksumFileSystem should have an overriding implementation of
listStatus(Path, PathFilter)
---
Key: HADOOP-8649
URL: https://issues.apache.org/jira/browse/HADOOP-8649
Project: Hadoop Common
Issue Type: Bug
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2,
HADOOP-8649_branch1.patch_v3
Currently, ChecksumFileSystem implements only listStatus(Path). The other
form of listStatus(Path, PathFilter) is implemented by parent class
FileSystem, and hence doesn't filter out check-sum files.
The implementation should use a composite filter of passed Filter and the
Checksum filter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13430626#comment-13430626 ] Karthik Kambatla commented on HADOOP-8649: -- Hi Daryn, thanks for your comments. I noticed it through casual observation. Let me put together a test case to test/explain this perceived bug better. Will update my patch soon with the new test case. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3 Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Status: Open (was: Patch Available) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: TestChecksumFileSystemOnDFS.java Sorry for the false alarm. As per Daryn's suggestion, I wrote a test to check the same that I am uploading here. Daryn's description of the flow is right, and there is no bug. Sorry again. Also, as Daryn commented, using a composite fiter would improve the performance. I ll update the description of the JIRA to reflect the same and upload patches for branch-1 and test including this test. Thanks again for your thorough review, Daryn. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) --- Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Description: Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. was: Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, PathFilter) is implemented by parent class FileSystem, and hence doesn't filter out check-sum files. The implementation should use a composite filter of passed Filter and the Checksum filter. Issue Type: Improvement (was: Bug) Summary: ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance (was: ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter)) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: HADOOP-8649_branch1.patch ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: trunk-HADOOP-8649.patch branch1-HADOOP-8649.patch Uploading patches for trunk and branch-1 addressing Daryn's comments. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Affects Version/s: 1.0.3 2.0.0-alpha Status: Patch Available (was: Open) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.0-alpha, 1.0.3 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13431389#comment-13431389 ] Karthik Kambatla commented on HADOOP-8649: -- Found the javadoc warning. The noticed test failures seem to be due to one of the patch's tests creating a file and not deleting it. Running all the tests locally to make sure these issues are fixed. Will upload updated patch soon. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Attachment: trunk-HADOOP-8649.patch branch1-HADOOP-8649.patch Uploading updated patches for branch1 and trunk. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13432021#comment-13432021 ] Karthik Kambatla commented on HADOOP-8649: -- I don't think the patch has anything to do with the two failing tests, these tests fail on the latest trunk as well. Casual code examination shows no intersection between the patch and failing tests. ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch, trunk-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[
https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13433541#comment-13433541
]
Karthik Kambatla commented on HADOOP-8649:
--
Thanks for the review, Daryn.
- I don't think it is incompatible with ChRootedFileSystem as it does not
filter out any files.
- +1 on generalizing and pushing the change down to FileSystem itself.
-- We can add {{protected/public FileSystem#listStatus(Path f, ListPathFilter
filters)}} and use {{MultiPathFilter}} as in {{o.a.h.m.FileInputFormat}}
-- All FileSystems can use this to build a list of {{PathFilter}}s to be
evaluated.
-- {{o.a.h.m.FileInputFormat}} can use the common version of {{MultiPathFilter}}
If we decide on this, I can go ahead and make the required changes.
ChecksumFileSystem should have an overriding implementation of
listStatus(Path, PathFilter) for improved performance
Key: HADOOP-8649
URL: https://issues.apache.org/jira/browse/HADOOP-8649
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch,
HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch,
HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3,
TestChecksumFileSystemOnDFS.java, trunk-HADOOP-8649.patch,
trunk-HADOOP-8649.patch
Currently, ChecksumFileSystem implements only listStatus(Path).
The other form of listStatus(Path, customFilter) results in parsing the list
twice to apply each of the filters - custom and checksum filter.
By using a composite filter instead, we limit the parsing to once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8649) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance
[ https://issues.apache.org/jira/browse/HADOOP-8649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8649: - Status: Open (was: Patch Available) ChecksumFileSystem should have an overriding implementation of listStatus(Path, PathFilter) for improved performance Key: HADOOP-8649 URL: https://issues.apache.org/jira/browse/HADOOP-8649 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.0-alpha, 1.0.3 Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: branch1-HADOOP-8649.patch, branch1-HADOOP-8649.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch, HADOOP-8649_branch1.patch_v2, HADOOP-8649_branch1.patch_v3, TestChecksumFileSystemOnDFS.java, trunk-HADOOP-8649.patch, trunk-HADOOP-8649.patch Currently, ChecksumFileSystem implements only listStatus(Path). The other form of listStatus(Path, customFilter) results in parsing the list twice to apply each of the filters - custom and checksum filter. By using a composite filter instead, we limit the parsing to once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8568) DNS#reverseDns fails on IPv6 addresses
[
https://issues.apache.org/jira/browse/HADOOP-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8568:
-
Assignee: (was: Karthik Kambatla)
Thanks for the patch, Tony. At first glance, the patch looks like it should
solve the problem.
I am marking the JIRA Unassigned. Please feel free to assign it to yourself.
DNS#reverseDns fails on IPv6 addresses
--
Key: HADOOP-8568
URL: https://issues.apache.org/jira/browse/HADOOP-8568
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Labels: newbie
Attachments: HADOOP-8568.patch
DNS#reverseDns assumes hostIp is a v4 address (4 parts separated by dots),
blows up if given a v6 address:
{noformat}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
at org.apache.hadoop.net.DNS.reverseDns(DNS.java:79)
at org.apache.hadoop.net.DNS.getHosts(DNS.java:237)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:340)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:358)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:337)
at org.apache.hadoop.hbase.master.HMaster.init(HMaster.java:235)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1649)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8801) ExitUtil#terminate should capture the exception stack trace
[ https://issues.apache.org/jira/browse/HADOOP-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13454972#comment-13454972 ] Karthik Kambatla commented on HADOOP-8801: -- +1 ExitUtil#terminate should capture the exception stack trace --- Key: HADOOP-8801 URL: https://issues.apache.org/jira/browse/HADOOP-8801 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Eli Collins Attachments: hadoop-8801.txt ExitUtil#terminate(status,Throwable) should capture and log the stack trace of the given throwable. This will help debug issues like HDFS-3933. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call inputstream.seek(0) before using input stream
[
https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8833:
-
Status: Patch Available (was: Open)
fs -text should make sure to call inputstream.seek(0) before using input
stream
---
Key: HADOOP-8833
URL: https://issues.apache.org/jira/browse/HADOOP-8833
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Harsh J
Assignee: Harsh J
Attachments: HADOOP-8833.patch, HADOOP-8833.patch
From Muddy Dixon on HADOOP-8449:
Hi
We found the changes in order of switch and guard block in
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
{code}
Because of this change, return value of
{code}
codec.createInputStream(i)
{code}
is changed if codec exists.
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
// check codecs
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
switch(i.readShort()) {
// cases
}
{code}
New:
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
switch(i.readShort()) { // === index (or pointer) processes!!
// cases
default: {
// Check the type of compression instead, depending on Codec class's
// own detection methods, based on the provided path.
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
break;
}
}
// File is non-compressed, or not a file container we know.
i.seek(0);
return i;
}
{code}
Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call inputstream.seek(0) before using input stream
[
https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8833:
-
Status: Open (was: Patch Available)
Cancelling patch to re-submit and kick Jenkins
fs -text should make sure to call inputstream.seek(0) before using input
stream
---
Key: HADOOP-8833
URL: https://issues.apache.org/jira/browse/HADOOP-8833
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Harsh J
Assignee: Harsh J
Attachments: HADOOP-8833.patch, HADOOP-8833.patch
From Muddy Dixon on HADOOP-8449:
Hi
We found the changes in order of switch and guard block in
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
{code}
Because of this change, return value of
{code}
codec.createInputStream(i)
{code}
is changed if codec exists.
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
// check codecs
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
switch(i.readShort()) {
// cases
}
{code}
New:
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
switch(i.readShort()) { // === index (or pointer) processes!!
// cases
default: {
// Check the type of compression instead, depending on Codec class's
// own detection methods, based on the provided path.
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
break;
}
}
// File is non-compressed, or not a file container we know.
i.seek(0);
return i;
}
{code}
Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call inputstream.seek(0) before using input stream
[
https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8833:
-
Status: Open (was: Patch Available)
fs -text should make sure to call inputstream.seek(0) before using input
stream
---
Key: HADOOP-8833
URL: https://issues.apache.org/jira/browse/HADOOP-8833
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Harsh J
Assignee: Harsh J
Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
From Muddy Dixon on HADOOP-8449:
Hi
We found the changes in order of switch and guard block in
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
{code}
Because of this change, return value of
{code}
codec.createInputStream(i)
{code}
is changed if codec exists.
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
// check codecs
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
switch(i.readShort()) {
// cases
}
{code}
New:
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
switch(i.readShort()) { // === index (or pointer) processes!!
// cases
default: {
// Check the type of compression instead, depending on Codec class's
// own detection methods, based on the provided path.
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
break;
}
}
// File is non-compressed, or not a file container we know.
i.seek(0);
return i;
}
{code}
Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call inputstream.seek(0) before using input stream
[
https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8833:
-
Attachment: HADOOP-8833.patch
Uploading the same patch again.
fs -text should make sure to call inputstream.seek(0) before using input
stream
---
Key: HADOOP-8833
URL: https://issues.apache.org/jira/browse/HADOOP-8833
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Harsh J
Assignee: Harsh J
Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
From Muddy Dixon on HADOOP-8449:
Hi
We found the changes in order of switch and guard block in
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
{code}
Because of this change, return value of
{code}
codec.createInputStream(i)
{code}
is changed if codec exists.
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
// check codecs
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
switch(i.readShort()) {
// cases
}
{code}
New:
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
switch(i.readShort()) { // === index (or pointer) processes!!
// cases
default: {
// Check the type of compression instead, depending on Codec class's
// own detection methods, based on the provided path.
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
break;
}
}
// File is non-compressed, or not a file container we know.
i.seek(0);
return i;
}
{code}
Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8833) fs -text should make sure to call inputstream.seek(0) before using input stream
[
https://issues.apache.org/jira/browse/HADOOP-8833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8833:
-
Status: Patch Available (was: Open)
fs -text should make sure to call inputstream.seek(0) before using input
stream
---
Key: HADOOP-8833
URL: https://issues.apache.org/jira/browse/HADOOP-8833
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 2.0.2-alpha
Reporter: Harsh J
Assignee: Harsh J
Attachments: HADOOP-8833.patch, HADOOP-8833.patch, HADOOP-8833.patch
From Muddy Dixon on HADOOP-8449:
Hi
We found the changes in order of switch and guard block in
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException
{code}
Because of this change, return value of
{code}
codec.createInputStream(i)
{code}
is changed if codec exists.
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
// check codecs
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
switch(i.readShort()) {
// cases
}
{code}
New:
{code}
private InputStream forMagic(Path p, FileSystem srcFs) throws IOException {
FSDataInputStream i = srcFs.open(p);
switch(i.readShort()) { // === index (or pointer) processes!!
// cases
default: {
// Check the type of compression instead, depending on Codec class's
// own detection methods, based on the provided path.
CompressionCodecFactory cf = new CompressionCodecFactory(getConf());
CompressionCodec codec = cf.getCodec(p);
if (codec != null) {
return codec.createInputStream(i);
}
break;
}
}
// File is non-compressed, or not a file container we know.
i.seek(0);
return i;
}
{code}
Fix is to use i.seek(0) before we use i anywhere. I missed that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned HADOOP-8852: Assignee: Karthik Kambatla DelegationTokenRenewer thread is not stopped when its filesystem is closed -- Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Reporter: Tom White Assignee: Karthik Kambatla HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[
https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464120#comment-13464120
]
Karthik Kambatla commented on HADOOP-8852:
--
{{DelegationTokenRenewer}} fields in both filesystems are static fields. As
stopping the thread corresponding to static field in close() does not seem like
good practice, we should make them non-static.
DelegationTokenRenewer thread is not stopped when its filesystem is closed
--
Key: HADOOP-8852
URL: https://issues.apache.org/jira/browse/HADOOP-8852
Project: Hadoop Common
Issue Type: Bug
Reporter: Tom White
Assignee: Karthik Kambatla
HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer
thread when they are closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8852: - Status: Patch Available (was: Open) DelegationTokenRenewer thread is not stopped when its filesystem is closed -- Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8852: - Affects Version/s: 2.0.0-alpha DelegationTokenRenewer thread is not stopped when its filesystem is closed -- Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[
https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464235#comment-13464235
]
Karthik Kambatla commented on HADOOP-8852:
--
By the way, in the little time I worked on Hadoop so far, I noticed several
cases like this where {{close()}} does not stop _all_ the threads it spawns. I
feel we should add more structure to ensure we do not miss any of them.
Thoughts?
DelegationTokenRenewer thread is not stopped when its filesystem is closed
--
Key: HADOOP-8852
URL: https://issues.apache.org/jira/browse/HADOOP-8852
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Tom White
Assignee: Karthik Kambatla
Attachments: hadoop-8852.patch
HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer
thread when they are closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464319#comment-13464319 ] Karthik Kambatla commented on HADOOP-8852: -- When I ran test-patch.sh locally, it didn't complain of any findbugs warnings. DelegationTokenRenewer thread is not stopped when its filesystem is closed -- Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[
https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8852:
-
Attachment: hadoop-8852.patch
Moved {{DelegationTokenRenewer#start()}} to constructor.
Ran test-patch.sh locally - findbugs doesn't show any warnings.
{color:green}+1 overall{color}.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version ) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
DelegationTokenRenewer thread is not stopped when its filesystem is closed
--
Key: HADOOP-8852
URL: https://issues.apache.org/jira/browse/HADOOP-8852
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Tom White
Assignee: Karthik Kambatla
Attachments: hadoop-8852.patch, hadoop-8852.patch
HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer
thread when they are closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[
https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13464846#comment-13464846
]
Karthik Kambatla commented on HADOOP-8852:
--
Thanks Tom and Daryn for your comments.
Daryn, your suggestions make absolute sense. Will work on the patch. While
working on this earlier, I thought of making it a Singleton, but thought there
was a reason why it was not Singleton :)
I will add {{cancelToken()}} and {{shutdown()}} to {{DelegationTokenRenewer}},
and use them in the FileSystems.
DelegationTokenRenewer thread is not stopped when its filesystem is closed
--
Key: HADOOP-8852
URL: https://issues.apache.org/jira/browse/HADOOP-8852
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Tom White
Assignee: Karthik Kambatla
Attachments: hadoop-8852.patch, hadoop-8852.patch
HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer
thread when they are closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8852: - Status: Open (was: Patch Available) DelegationTokenRenewer thread is not stopped when its filesystem is closed -- Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch, hadoop-8852.patch HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8852) DelegationTokenRenewer thread is not stopped when its filesystem is closed
[
https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8852:
-
Attachment: hadoop-8852-v1.patch
Here is a preliminary patch that makes {{DelegationTokenRenewer}} a singleton:
# Dropped the generic part of it
# capture the requirement to extend {{FileSystem}} by augmenting the API -
{{public T extends FileSystem Renewable void addRenewAction(final T fs)}}
# {{WebHdfsFileSystem}} and {{HftpFileSystem}} de-register from
{{DelegationTokenRenewer}} in {{close()}}
Removing a FileSystem from the queue currently is linear in queue-size. I
thought it might be okay, given that the queue should be small. Alternatively,
we can maintain a list of removed FileSystems; {{run()}} could remove
{{RenewAction}} (from {{queue.take()}}) if it were present in the remove-list.
This patch doesn't shutdown {{DelegationTokenRenewer}} when it can't renew
tokens. Shutting down the thread alone can lead to interesting scenarios when
new {{RenewAction}}s are added. Nullifying the {{INSTANCE}} after interrupting
the thread will lead to {{FileSystem}}'s trying to use the stale reference? Not
sure what the best way to handle this would be. Thoughts?
DelegationTokenRenewer thread is not stopped when its filesystem is closed
--
Key: HADOOP-8852
URL: https://issues.apache.org/jira/browse/HADOOP-8852
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Tom White
Assignee: Karthik Kambatla
Attachments: hadoop-8852.patch, hadoop-8852.patch,
hadoop-8852-v1.patch
HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer
thread when they are closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8852) DelegationTokenRenewer should be Singleton
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8852: - Description: Updated description: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Original issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. was:HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. Summary: DelegationTokenRenewer should be Singleton (was: DelegationTokenRenewer thread is not stopped when its filesystem is closed) DelegationTokenRenewer should be Singleton -- Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch, hadoop-8852.patch, hadoop-8852-v1.patch Updated description: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Original issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8852) DelegationTokenRenewer should be Singleton
[
https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13468968#comment-13468968
]
Karthik Kambatla commented on HADOOP-8852:
--
After spending some time on this, I noticed a couple of other related issues:
- WebHdfsFileSystem and HftpFileSystem have their own implemenations of
TokenRenewer that are not used at all.
- In DelegationTokenRenewer#renew() (see snippet below), we fetch new tokens in
case the renew fails. However, we use only the first token and ignore the rest.
Later, if we can't renew this token, we re-fetch new tokens instead of using
the previously fetched ones. Also, we might not be able to fetch new ones, in
which case we give up. Is this valid behavior? Or, should we be first using the
initially fetched tokens, and re-fetch only when we run out of them.
{code}
Token?[] tokens = fs.addDelegationTokens(null, null);
if (tokens.length == 0) {
throw new IOException(addDelegationTokens returned no tokens);
}
fs.setDelegationToken(tokens[0]);
{code}
Taking the above into consideration along with the goal of further decoupling
DelegationTokenRenewer from actual FS-specific details of
renewal/refetching/cancellation, I propose the following. Please advise
appropriately.
# TokenRenewer should be an interface (it is currently a fully abstract class).
# Create a new abstract class TokenManager that holds a list (queue) of tokens,
their kind, renewal period, a TokenRenewer implementation, and an abstract
method #manage() to manage (renew/re-fetch/cancel) the tokens as appropriate.
# Each FileSystem (WebHdfs and Hftp) has its own implemenation (local extended
class) of the TokenManager.
# DelegationTokenRenewer allows registering and de-registering TokenManagers.
FileSystems call register() at init(), and de-register() at close().
# DelegationTokenRenewer stores the registered TokenManagers in its DelayQueue.
On a TokenManager's turn, the manage() method is invoked. If the manage fails,
the TokenManager is automatically de-registered.
DelegationTokenRenewer should be Singleton
--
Key: HADOOP-8852
URL: https://issues.apache.org/jira/browse/HADOOP-8852
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Tom White
Assignee: Karthik Kambatla
Attachments: hadoop-8852.patch, hadoop-8852.patch,
hadoop-8852-v1.patch
Updated description:
DelegationTokenRenewer should be Singleton - the instance and renewer threads
should be created/started lazily. The filesystems using the renewer shouldn't
need to explicity start/stop the renewer, and only register/de-register for
token renewal.
Original issue:
HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer
thread when they are closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (HADOOP-8877) Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity
[ https://issues.apache.org/jira/browse/HADOOP-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla moved MAPREDUCE-4702 to HADOOP-8877: - Key: HADOOP-8877 (was: MAPREDUCE-4702) Project: Hadoop Common (was: Hadoop Map/Reduce) Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity Key: HADOOP-8877 URL: https://issues.apache.org/jira/browse/HADOOP-8877 Project: Hadoop Common Issue Type: Wish Reporter: Karthik Kambatla Priority: Trivial While browsing through the code, I came across the TrivialRenewer. It would definitely be easy to comprehend if we rename it to UnmanagedRenewer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8852) DelegationTokenRenewer should be Singleton
[
https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469569#comment-13469569
]
Karthik Kambatla commented on HADOOP-8852:
--
Isn't it more efficient to renew a token than to get a new token every time? I
agree we can get the first token lazily, instead of implicitly obtaining a
token during {{FileSystem#init()}}
For FileSystems, is there a penalty for not renewing the tokens within the
renewal period? If there is no penalty, it might actually be detrimental to
periodically renew even in the absence of any activity.
On the other hand, if there is a penalty, I see some merit to this class. In a
world with several filesystems, MR jobs, and other external entities, this
class can enable renewing tokens sequentially in the order the systems come up
for renewal.
DelegationTokenRenewer should be Singleton
--
Key: HADOOP-8852
URL: https://issues.apache.org/jira/browse/HADOOP-8852
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Tom White
Assignee: Karthik Kambatla
Attachments: hadoop-8852.patch, hadoop-8852.patch,
hadoop-8852-v1.patch
Updated description:
DelegationTokenRenewer should be Singleton - the instance and renewer threads
should be created/started lazily. The filesystems using the renewer shouldn't
need to explicity start/stop the renewer, and only register/de-register for
token renewal.
Original issue:
HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer
thread when they are closed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8877) Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity
[ https://issues.apache.org/jira/browse/HADOOP-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned HADOOP-8877: Assignee: Karthik Kambatla Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity Key: HADOOP-8877 URL: https://issues.apache.org/jira/browse/HADOOP-8877 Project: Hadoop Common Issue Type: Wish Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Trivial While browsing through the code, I came across the TrivialRenewer. It would definitely be easy to comprehend if we rename it to UnmanagedRenewer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8877) Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity
[ https://issues.apache.org/jira/browse/HADOOP-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8877: - Attachment: hadoop-8877.patch Trivial patch that renames TrivialRenewer to UnmanagedRenewer. No tests, because the logic hasn't changed in anyway. Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity Key: HADOOP-8877 URL: https://issues.apache.org/jira/browse/HADOOP-8877 Project: Hadoop Common Issue Type: Wish Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Trivial Attachments: hadoop-8877.patch While browsing through the code, I came across the TrivialRenewer. It would definitely be easy to comprehend if we rename it to UnmanagedRenewer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8877) Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity
[ https://issues.apache.org/jira/browse/HADOOP-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8877: - Affects Version/s: 2.0.1-alpha Status: Patch Available (was: Open) Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity Key: HADOOP-8877 URL: https://issues.apache.org/jira/browse/HADOOP-8877 Project: Hadoop Common Issue Type: Wish Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Trivial Attachments: hadoop-8877.patch While browsing through the code, I came across the TrivialRenewer. It would definitely be easy to comprehend if we rename it to UnmanagedRenewer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8852) DelegationTokenRenewer should be Singleton
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13469929#comment-13469929 ] Karthik Kambatla commented on HADOOP-8852: -- Offline discussion with ATM has helped me understand this better. Gist for others in my position (without a clear understanding of Kerberos): # Initial authentication is indeed expensive and involves communicating with KDC, one gets Krb ticket upon authentication. # Subsequent authentications with this Krb ticket are quick and don't need KDC communication. # Hadoop tokens will not make these subsequent authentications any more efficient, because functionally this is same as the above authentication. Tokens are useful when one doesn't have tickets - e.g., MR tasks that execute on different nodes. As per Daryn's suggestion, I ll go ahead and remove DelegationTokenRenewer and the relevant logic from these filesystems. Thanks Daryn and ATM. DelegationTokenRenewer should be Singleton -- Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch, hadoop-8852.patch, hadoop-8852-v1.patch Updated description: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Original issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8852) Remove DelegationTokenRenewer
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8852: - Summary: Remove DelegationTokenRenewer (was: DelegationTokenRenewer should be Singleton) Remove DelegationTokenRenewer - Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch, hadoop-8852.patch, hadoop-8852-v1.patch Updated description: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Original issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8852) Remove DelegationTokenRenewer
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8852: - Description: Update 2: DelegationTokenRenewer is not required. The filesystems that are using it already have Krb tickets and do not need tokens. Remove DelegationTokenRenewer and all the related logic from WebHdfs and Hftp filesystems. Update1: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Original issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. was: Updated description: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Original issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. Remove DelegationTokenRenewer - Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch, hadoop-8852.patch, hadoop-8852-v1.patch Update 2: DelegationTokenRenewer is not required. The filesystems that are using it already have Krb tickets and do not need tokens. Remove DelegationTokenRenewer and all the related logic from WebHdfs and Hftp filesystems. Update1: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Original issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8852) WebHdfsFileSystem and HftpFileSystem don't need delegation tokens
[ https://issues.apache.org/jira/browse/HADOOP-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8852: - Description: Parent JIRA to track the work of removing delegation tokens from these filesystems. This JIRA has evolved from the initial issue of these filesystems not stopping the DelegationTokenRenewer thread they were creating. After further investigation, Daryn pointed out - If you can get a token, you don't need a token! Hence, these filesystems shouldn't use delegation tokens. Evolution of the JIRA is listed below: Update 2: DelegationTokenRenewer is not required. The filesystems that are using it already have Krb tickets and do not need tokens. Remove DelegationTokenRenewer and all the related logic from WebHdfs and Hftp filesystems. Update1: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Initial issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. was: Update 2: DelegationTokenRenewer is not required. The filesystems that are using it already have Krb tickets and do not need tokens. Remove DelegationTokenRenewer and all the related logic from WebHdfs and Hftp filesystems. Update1: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Original issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. Issue Type: Improvement (was: Bug) Summary: WebHdfsFileSystem and HftpFileSystem don't need delegation tokens (was: Remove DelegationTokenRenewer) WebHdfsFileSystem and HftpFileSystem don't need delegation tokens - Key: HADOOP-8852 URL: https://issues.apache.org/jira/browse/HADOOP-8852 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Karthik Kambatla Attachments: hadoop-8852.patch, hadoop-8852.patch, hadoop-8852-v1.patch Parent JIRA to track the work of removing delegation tokens from these filesystems. This JIRA has evolved from the initial issue of these filesystems not stopping the DelegationTokenRenewer thread they were creating. After further investigation, Daryn pointed out - If you can get a token, you don't need a token! Hence, these filesystems shouldn't use delegation tokens. Evolution of the JIRA is listed below: Update 2: DelegationTokenRenewer is not required. The filesystems that are using it already have Krb tickets and do not need tokens. Remove DelegationTokenRenewer and all the related logic from WebHdfs and Hftp filesystems. Update1: DelegationTokenRenewer should be Singleton - the instance and renewer threads should be created/started lazily. The filesystems using the renewer shouldn't need to explicity start/stop the renewer, and only register/de-register for token renewal. Initial issue: HftpFileSystem and WebHdfsFileSystem should stop the DelegationTokenRenewer thread when they are closed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8890) Remove unused TokenRenewer implementation from WebHdfsFileSystem and HftpFileSystem
Karthik Kambatla created HADOOP-8890: Summary: Remove unused TokenRenewer implementation from WebHdfsFileSystem and HftpFileSystem Key: HADOOP-8890 URL: https://issues.apache.org/jira/browse/HADOOP-8890 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla WebHdfsFileSystem and HftpFileSystem implement TokenRenewer without using anywhere. As we are in the process of migrating them to not use tokens, this code should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
Karthik Kambatla created HADOOP-8891: Summary: Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Sub-task Reporter: Karthik Kambatla -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8892) WebHdfsFileSystem shouldn't use delegation tokens
Karthik Kambatla created HADOOP-8892: Summary: WebHdfsFileSystem shouldn't use delegation tokens Key: HADOOP-8892 URL: https://issues.apache.org/jira/browse/HADOOP-8892 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8890) Remove unused TokenRenewer implementation from WebHdfsFileSystem and HftpFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned HADOOP-8890: Assignee: Karthik Kambatla Remove unused TokenRenewer implementation from WebHdfsFileSystem and HftpFileSystem --- Key: HADOOP-8890 URL: https://issues.apache.org/jira/browse/HADOOP-8890 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla WebHdfsFileSystem and HftpFileSystem implement TokenRenewer without using anywhere. As we are in the process of migrating them to not use tokens, this code should be removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned HADOOP-8891: Assignee: Karthik Kambatla Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Sub-task Reporter: Karthik Kambatla Assignee: Karthik Kambatla -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Issue Type: Improvement (was: Sub-task) Parent: (was: HADOOP-8852) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Reporter: Karthik Kambatla Assignee: Karthik Kambatla -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Description: Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. Affects Version/s: 2.0.1-alpha Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Status: Patch Available (was: Open) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Attachment: HADOOP-8891.patch Uploading a patch that removes DelegationTokenRenewer and its use in WebHdfs, Hftp, HttpFS filesystems. Ran all *WebHdfs*, *Hftp*, and *HttpFS* tests - the patch doesn't introduce any new test failures. Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class
Karthik Kambatla created HADOOP-8895: Summary: TokenRenewer should be an interface, it is currently a fully abstract class Key: HADOOP-8895 URL: https://issues.apache.org/jira/browse/HADOOP-8895 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Minor TokenRenewer is a fully abstract class. Making it an interface will allow classes extending other classes to implement the interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class
[
https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8895:
-
Attachment: HADOOP-8895.patch
Daryn,
Thanks for your input.
Though it is not absolutely necessary for {{TokenRenewer}} to be an interface,
I felt it would make things simpler when working on HDFS-4009.
For instance, if a {{WebHdfsFileSystem}} could implement {{TokenRenewer}}, we
might not need {{DelegationTokenRenewer.Renewable}}. Now that we are removing
{{DelegationTokenRenewer}} completely - {{HADOOP-8891}} - it won't be
immediately applicable. However, I was not sure why it should be an abstract
class.
Do you think we should leave it as is? Is there an advantage of abstract class
over interface?
I am uploading a patch with my proposed changes.
TokenRenewer should be an interface, it is currently a fully abstract class
---
Key: HADOOP-8895
URL: https://issues.apache.org/jira/browse/HADOOP-8895
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Priority: Minor
Attachments: HADOOP-8895.patch
TokenRenewer is a fully abstract class. Making it an interface will allow
classes extending other classes to implement the interface.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class
[ https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8895: - Status: Patch Available (was: Open) TokenRenewer should be an interface, it is currently a fully abstract class --- Key: HADOOP-8895 URL: https://issues.apache.org/jira/browse/HADOOP-8895 Project: Hadoop Common Issue Type: Bug Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Minor Attachments: HADOOP-8895.patch TokenRenewer is a fully abstract class. Making it an interface will allow classes extending other classes to implement the interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class
[ https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8895: - Affects Version/s: 2.0.1-alpha TokenRenewer should be an interface, it is currently a fully abstract class --- Key: HADOOP-8895 URL: https://issues.apache.org/jira/browse/HADOOP-8895 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Minor Attachments: HADOOP-8895.patch TokenRenewer is a fully abstract class. Making it an interface will allow classes extending other classes to implement the interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-8891:
-
Attachment: HADOOP-8891.patch
Updated the patch to remove {{setDelegationToken()}}. Also, fixed the findbugs
warnings.
Verified the patch doesn't introduce any new test failures for {{*WebHdfs*,
*Hftp*, *HttpFS*}} tests.
Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and
HftpFileSystem
-
Key: HADOOP-8891
URL: https://issues.apache.org/jira/browse/HADOOP-8891
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: HADOOP-8891.patch, HADOOP-8891.patch
Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks.
Created this to track the removal of DelegationTokenRenewer alone.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Attachment: (was: HADOOP-8891.patch) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Attachment: HADOOP-8891.patch Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch, HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Status: Open (was: Patch Available) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch, HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472246#comment-13472246
]
Karthik Kambatla commented on HADOOP-8891:
--
I am a little confused with {{HttpFSFileSystem}}. {{#initialize()}} doesn't
check for an existing delegation token, nor does it sign up for token renewal.
However, it still implements {{getDelegationToken()}} and
{{setDelegationToken()}}, I am not sure where these methods are actually
invoked from.
Will investigate this further. Meanwhile, I have uploaded a patch that removes
{{DelegationTokenRenewer}} and updates {{WebHdfsFileSystem}} and
{{HftpFileSystem}} accordingly. It would really help if someone could take a
look at the available patch and any thoughts on how to proceed with
{{HttpFSFileSystem}}.
Thanks
Karthik
Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and
HftpFileSystem
-
Key: HADOOP-8891
URL: https://issues.apache.org/jira/browse/HADOOP-8891
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: HADOOP-8891.patch, HADOOP-8891.patch
Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks.
Created this to track the removal of DelegationTokenRenewer alone.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class
[ https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472615#comment-13472615 ] Karthik Kambatla commented on HADOOP-8895: -- Fair enough. Thanks Tom and Daryn. I ll close this JIRA as won't fix. We can may be re-open should a need arise. Thanks again. TokenRenewer should be an interface, it is currently a fully abstract class --- Key: HADOOP-8895 URL: https://issues.apache.org/jira/browse/HADOOP-8895 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Minor Attachments: HADOOP-8895.patch TokenRenewer is a fully abstract class. Making it an interface will allow classes extending other classes to implement the interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8895) TokenRenewer should be an interface, it is currently a fully abstract class
[ https://issues.apache.org/jira/browse/HADOOP-8895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8895: - Resolution: Won't Fix Status: Resolved (was: Patch Available) TokenRenewer should be an interface, it is currently a fully abstract class --- Key: HADOOP-8895 URL: https://issues.apache.org/jira/browse/HADOOP-8895 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Minor Attachments: HADOOP-8895.patch TokenRenewer is a fully abstract class. Making it an interface will allow classes extending other classes to implement the interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13472631#comment-13472631
]
Karthik Kambatla commented on HADOOP-8891:
--
bq. Are some of these changes (like removal of the renewer) duplicated in your
other jiras? If so, that will cause merge conflicts.
Yes. I created the other JIRAs thinking it would make it easier, but now
realize that there could be conflicts. I was thinking of finishing this one,
and see if anything else needs to be done for the other JIRAs. Otherwise, can
close them out as duplicates.
bq. setDelegationToken is only used by the renewer you are removing, so it
should be removed.
{{TestHttpFSWithKerberos}} calls {{setDelegationToken()}} on all these
filesystems. I am trying to figure out if there is any way around it, short of
removing the tests.
Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and
HftpFileSystem
-
Key: HADOOP-8891
URL: https://issues.apache.org/jira/browse/HADOOP-8891
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.1-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: HADOOP-8891.patch, HADOOP-8891.patch
Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks.
Created this to track the removal of DelegationTokenRenewer alone.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from FileSystems using it
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Summary: Remove DelegationTokenRenewer and its logic from FileSystems using it (was: Remove DelegationTokenRenewer and its logic from WebHdfsFileSystem and HftpFileSystem) Remove DelegationTokenRenewer and its logic from FileSystems using it - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch, HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from FileSystems using it
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Attachment: HADOOP-8891.patch Updated patch addressing Daryn's comments. TestHttpFSWithKerberos fails both before and after the patch. Unlike WebHdfs and Hftp, HttpFSFileSystem doesn't check UGI for tokens at init. But, that is out of the scope of this JIRA - can create another for it. Remove DelegationTokenRenewer and its logic from FileSystems using it - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch, HADOOP-8891.patch, HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from FileSystems using it
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Status: Patch Available (was: Open) Remove DelegationTokenRenewer and its logic from FileSystems using it - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch, HADOOP-8891.patch, HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8877) Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity
[ https://issues.apache.org/jira/browse/HADOOP-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8877: - Resolution: Won't Fix Status: Resolved (was: Patch Available) Trivial change, closing as won't fix for now. Can re-open and commit changes later if need be. Rename o.a.h.security.token.Token.TrivialRenewer to UnmanagedRenewer for clarity Key: HADOOP-8877 URL: https://issues.apache.org/jira/browse/HADOOP-8877 Project: Hadoop Common Issue Type: Wish Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Priority: Trivial Attachments: hadoop-8877.patch While browsing through the code, I came across the TrivialRenewer. It would definitely be easy to comprehend if we rename it to UnmanagedRenewer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from FileSystems using it
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-8891: - Status: Open (was: Patch Available) Remove DelegationTokenRenewer and its logic from FileSystems using it - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch, HADOOP-8891.patch, HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9049) DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from.
Karthik Kambatla created HADOOP-9049: Summary: DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from. Key: HADOOP-9049 URL: https://issues.apache.org/jira/browse/HADOOP-9049 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.0.2-alpha Reporter: Karthik Kambatla Currently, DelegationTokenRenewer is not singleton. Each filesystem using it spawns its own DelegationTokenRenewer. Also, they don't stop the Renewer leading to other problems. A single DelegationTokenRenewer should be sufficient for all FileSystems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9049) DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from.
[ https://issues.apache.org/jira/browse/HADOOP-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned HADOOP-9049: Assignee: Karthik Kambatla DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from. Key: HADOOP-9049 URL: https://issues.apache.org/jira/browse/HADOOP-9049 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.0.2-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: hadoop-9049.patch Currently, DelegationTokenRenewer is not singleton. Each filesystem using it spawns its own DelegationTokenRenewer. Also, they don't stop the Renewer leading to other problems. A single DelegationTokenRenewer should be sufficient for all FileSystems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9049) DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from.
[ https://issues.apache.org/jira/browse/HADOOP-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-9049: - Attachment: hadoop-9049.patch Attaching the latest patch from HDFS-4009.patch DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from. Key: HADOOP-9049 URL: https://issues.apache.org/jira/browse/HADOOP-9049 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.0.2-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: hadoop-9049.patch Currently, DelegationTokenRenewer is not singleton. Each filesystem using it spawns its own DelegationTokenRenewer. Also, they don't stop the Renewer leading to other problems. A single DelegationTokenRenewer should be sufficient for all FileSystems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9049) DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from.
[ https://issues.apache.org/jira/browse/HADOOP-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-9049: - Status: Patch Available (was: Open) DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from. Key: HADOOP-9049 URL: https://issues.apache.org/jira/browse/HADOOP-9049 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.0.2-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: hadoop-9049.patch Currently, DelegationTokenRenewer is not singleton. Each filesystem using it spawns its own DelegationTokenRenewer. Also, they don't stop the Renewer leading to other problems. A single DelegationTokenRenewer should be sufficient for all FileSystems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8891) Remove DelegationTokenRenewer and its logic from FileSystems using it
[ https://issues.apache.org/jira/browse/HADOOP-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla resolved HADOOP-8891. -- Resolution: Invalid Per discussion on HDFS-4009, this is invalid anymore. The issue at hand is being addressed in HADOOP-9049. Remove DelegationTokenRenewer and its logic from FileSystems using it - Key: HADOOP-8891 URL: https://issues.apache.org/jira/browse/HADOOP-8891 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.1-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: HADOOP-8891.patch, HADOOP-8891.patch, HADOOP-8891.patch Moved the HDFS part of HADOOP-8852 to HDFS-4009 along with other sub-tasks. Created this to track the removal of DelegationTokenRenewer alone. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9049) DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from.
[ https://issues.apache.org/jira/browse/HADOOP-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated HADOOP-9049: - Attachment: hadoop-9049.patch Thanks Daryn. Definitely looking forward for your full review. Meanwhile, I am uploading a new patch to address Daryn's comments: # make getInstance() synchronized and make INSTANCE non-volatile # call super.close() in close() of WebHdfs and Hftp FS DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from. Key: HADOOP-9049 URL: https://issues.apache.org/jira/browse/HADOOP-9049 Project: Hadoop Common Issue Type: Bug Components: security Affects Versions: 2.0.2-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Attachments: hadoop-9049.patch, hadoop-9049.patch Currently, DelegationTokenRenewer is not singleton. Each filesystem using it spawns its own DelegationTokenRenewer. Also, they don't stop the Renewer leading to other problems. A single DelegationTokenRenewer should be sufficient for all FileSystems. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9064) Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction
Karthik Kambatla created HADOOP-9064:
Summary: Augment DelegationTokenRenewer API to cancel the tokens
on calls to removeRenewAction
Key: HADOOP-9064
URL: https://issues.apache.org/jira/browse/HADOOP-9064
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Post HADOOP-9049, FileSystems register with DelegationTokenRenewer (a
singleton), to renew tokens.
To avoid a bunch of defunct tokens clog the NN, we should augment the API to
{{#removeRenewAction(boolean cancel)}} and cancel the token appropriately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9049) DelegationTokenRenewer needs to be Singleton and FileSystems should register/deregister to/from.
[
https://issues.apache.org/jira/browse/HADOOP-9049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500425#comment-13500425
]
Karthik Kambatla commented on HADOOP-9049:
--
Hi Daryn,
Thanks for the review.
I have filed HADOOP-9064 to add canceling the token. I feel we should pass a
boolean to {{#removeRenewAction(boolean cancel)}} and cancel the token
accordingly. The filesystems can call removeRenewAction with cancel set to true.
DelegationTokenRenewer needs to be Singleton and FileSystems should
register/deregister to/from.
Key: HADOOP-9049
URL: https://issues.apache.org/jira/browse/HADOOP-9049
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: hadoop-9049.patch, hadoop-9049.patch
Currently, DelegationTokenRenewer is not singleton.
Each filesystem using it spawns its own DelegationTokenRenewer. Also, they
don't stop the Renewer leading to other problems.
A single DelegationTokenRenewer should be sufficient for all FileSystems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9064) Augment DelegationTokenRenewer API to cancel the tokens on calls to removeRenewAction
[
https://issues.apache.org/jira/browse/HADOOP-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated HADOOP-9064:
-
Attachment: hadoop-9064.patch
Posting the patch that augments the API as in the description.
Augment DelegationTokenRenewer API to cancel the tokens on calls to
removeRenewAction
-
Key: HADOOP-9064
URL: https://issues.apache.org/jira/browse/HADOOP-9064
Project: Hadoop Common
Issue Type: Bug
Components: security
Affects Versions: 2.0.2-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Attachments: hadoop-9064.patch
Post HADOOP-9049, FileSystems register with DelegationTokenRenewer (a
singleton), to renew tokens.
To avoid a bunch of defunct tokens clog the NN, we should augment the API to
{{#removeRenewAction(boolean cancel)}} and cancel the token appropriately.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
