[jira] [Reopened] (HDFS-3373) FileContext HDFS implementation can leak socket caches
[ https://issues.apache.org/jira/browse/HDFS-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George reopened HDFS-3373: --- Reopening for branch 23 FileContext HDFS implementation can leak socket caches -- Key: HDFS-3373 URL: https://issues.apache.org/jira/browse/HDFS-3373 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0 Reporter: Todd Lipcon Assignee: John George Fix For: 2.0.3-alpha Attachments: HDFS-3373.branch-23.patch, HDFS-3373.branch23.patch, HDFS-3373.trunk.patch, HDFS-3373.trunk.patch.1, HDFS-3373.trunk.patch.2, HDFS-3373.trunk.patch.3, HDFS-3373.trunk.patch.3, HDFS-3373.trunk.patch.4 As noted by Nicholas in HDFS-3359, FileContext doesn't have a close() method, and thus never calls DFSClient.close(). This means that, until finalizers run, DFSClient will hold on to its SocketCache object and potentially have a lot of outstanding sockets/fds held on to. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3001) dfsadmin -refreshServiceAcl fails Kerb authentication with valid Kerb ticket, other subcommands succeed
[ https://issues.apache.org/jira/browse/HDFS-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George resolved HDFS-3001. --- Resolution: Cannot Reproduce Pat, I am going to close this as not-reproducible anymore. If you disagree, please feel free to re-open. dfsadmin -refreshServiceAcl fails Kerb authentication with valid Kerb ticket, other subcommands succeed --- Key: HDFS-3001 URL: https://issues.apache.org/jira/browse/HDFS-3001 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.1 Reporter: patrick white Assignee: John George With a valid hdfs kerberos ticket, the dfsadmin subcommand '-refreshServiceAcl' still fails on Kerb authentication. Please see the comment for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2033) JIRA to redesign client side read path
[ https://issues.apache.org/jira/browse/HDFS-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George resolved HDFS-2033. --- Resolution: Invalid I do not have any new plans or ideas about this. Closing this for now. JIRA to redesign client side read path -- Key: HDFS-2033 URL: https://issues.apache.org/jira/browse/HDFS-2033 Project: Hadoop HDFS Issue Type: Bug Reporter: John George Assignee: John George Priority: Minor This is a JIRA that came up based on comments on HADOOP-7316. This is to study the current design of the client side read path and redesign if necessary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3379) move setBlock() to INodeFileUC
John George created HDFS-3379: - Summary: move setBlock() to INodeFileUC Key: HDFS-3379 URL: https://issues.apache.org/jira/browse/HDFS-3379 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0, 3.0.0 Reporter: John George Assignee: John George Priority: Minor Move BlockCollection.setBlock(..) to MutableBlockCollection and change the parameter in BlockManger.forceCompleteBlock(..) and completeBlock(..) to MutableBlockCollection. We should also move setBlock(..) from INodeFile to INodeFileUnderConstruction. Based on discussion in HDFS-3363 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3369) change variable names referring to inode in blockmanagement to more appropriate
John George created HDFS-3369: - Summary: change variable names referring to inode in blockmanagement to more appropriate Key: HDFS-3369 URL: https://issues.apache.org/jira/browse/HDFS-3369 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.0.0, 3.0.0 Reporter: John George Assignee: John George Priority: Minor We should rename BlocksMap.getINode(..) and, in addition, the local variable names such as fileInode to match 'block collection' -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3363) blockmanagement should stop using INodeFile INodeFileUC
John George created HDFS-3363: - Summary: blockmanagement should stop using INodeFile INodeFileUC Key: HDFS-3363 URL: https://issues.apache.org/jira/browse/HDFS-3363 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.0.0, 3.0.0 Reporter: John George Assignee: John George Blockmanagement should stop using INodeFile and INodeFileUnderConstruction. One way would be to create an interface, like BlockColletion, that is passed along to the blockmanagement. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3338) change INode to package private
[ https://issues.apache.org/jira/browse/HDFS-3338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George resolved HDFS-3338. --- Resolution: Duplicate Target Version/s: 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0) Accidentally filed this - dupe of HDFS-3339 change INode to package private --- Key: HDFS-3338 URL: https://issues.apache.org/jira/browse/HDFS-3338 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 2.0.0, 3.0.0 Reporter: John George Assignee: John George Priority: Minor Only a few place in blockmanagement is using INode and all of them can be replaced by INodeFile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3339) change INode to package private
John George created HDFS-3339: - Summary: change INode to package private Key: HDFS-3339 URL: https://issues.apache.org/jira/browse/HDFS-3339 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.0.0, 3.0.0 Reporter: John George Assignee: John George Priority: Minor Only a few place in blockmanagement is using INode and all of them can be replaced by INodeFile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3338) change INode to package private
John George created HDFS-3338: - Summary: change INode to package private Key: HDFS-3338 URL: https://issues.apache.org/jira/browse/HDFS-3338 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.0.0, 3.0.0 Reporter: John George Assignee: John George Priority: Minor Only a few place in blockmanagement is using INode and all of them can be replaced by INodeFile. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2201) combine sequence read and position read
[ https://issues.apache.org/jira/browse/HDFS-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George resolved HDFS-2201. --- Resolution: Invalid On further analysis, I realize that it is better that the paths are left separate. This is because sequential read/seek and position reads are supposed to be run in parallel and produce separate results. This means that those variables have to be maintained separately inorder to have a common path. It looks like it might be better to leave that alone. Hence marking it invalid and closing the bug. combine sequence read and position read --- Key: HDFS-2201 URL: https://issues.apache.org/jira/browse/HDFS-2201 Project: Hadoop HDFS Issue Type: Sub-task Reporter: John George Assignee: John George Priority: Minor Fix For: 0.23.0 Attachments: HDFS-2201.patch It might be beneficial (especially from maintenance pov) to combine the logic of sequence and position read calls in DFSInputStream.java. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2114) re-commission of a decommissioned node does not delete excess replica
re-commission of a decommissioned node does not delete excess replica - Key: HDFS-2114 URL: https://issues.apache.org/jira/browse/HDFS-2114 Project: Hadoop HDFS Issue Type: Bug Reporter: John George Assignee: John George If a decommissioned node is removed from the decommissioned list, namenode does not delete the excess replicas it created while the node was decommissioned. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2033) JIRA to redesign client side read path
JIRA to redesign client side read path -- Key: HDFS-2033 URL: https://issues.apache.org/jira/browse/HDFS-2033 Project: Hadoop HDFS Issue Type: Bug Reporter: John George Assignee: John George Priority: Minor This is a JIRA that came up based on comments on HDFS-7316. This is to study the current design of the client side read path and redesign if necessary. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1927) audit logs could ignore certain xsactions and also could contain ip=null
audit logs could ignore certain xsactions and also could contain ip=null -- Key: HDFS-1927 URL: https://issues.apache.org/jira/browse/HDFS-1927 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.2, 0.23.0 Reporter: John George Assignee: John George Namenode audit logs could be ignoring certain transactions that are successfully completed. This is because it check if the RemoteIP is null to decide if a transaction is remote or not. In certain cases, RemoteIP could return null but the xsaction could still be remote. An example is a case where a client gets killed while in the middle of the transaction. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1889) incorrect path in start/stop dfs script
incorrect path in start/stop dfs script --- Key: HDFS-1889 URL: https://issues.apache.org/jira/browse/HDFS-1889 Project: Hadoop HDFS Issue Type: Bug Reporter: John George Assignee: John George HADOOP_HOME in start-dfs.sh and stop-dfs.sh should be changed to HADOOP_HDFS_HOME because hdfs script is in the hdfs directory and not common directory -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1845) symlink comes up as directory after namenode restart
symlink comes up as directory after namenode restart Key: HDFS-1845 URL: https://issues.apache.org/jira/browse/HDFS-1845 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0 Reporter: John George Assignee: John George When a symlink is first created, it get added to EditLogs. When namenode is restarted, it reads from this editlog and represents a symlink correctly and saves this information to its image. If the namenode is restarted again, it reads its from this FSImage, but thinks that a symlink is a directory. This is because it uses Block[] blocks to determine if an INode is a directory, a file, or symlink. Since both a directory and a symlink has blocks as null, it thinks that a symlink is a directory. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1821) FileContext.createSymlink with kerberos enabled sets wrong owner
FileContext.createSymlink with kerberos enabled sets wrong owner Key: HDFS-1821 URL: https://issues.apache.org/jira/browse/HDFS-1821 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0 Environment: Kerberos enabled on cluster Reporter: John George Assignee: John George TEST SETUP Using attached sample hdfs java program that illustrates the issue. Using cluster with Kerberos enabled on cluster # Compile instructions $ javac Symlink.java -cp `hadoop classpath` $ jar -cfe Symlink.jar Symlink Symlink.class # create test file for symlink to use 1. hadoop fs -touchz /user/username/filetest # Create symlink using file context 2. hadoop jar Symlink.jar ln /user/username/filetest /user/username/testsymlink # Verify owner of test file 3. hadoop jar Symlink.jar ls /user/username/ -rw-r--r-- jeagles hdfs /user/jeagles/filetest -rwxrwxrwx jeag...@xx..x.xxx hdfs /user/username/testsymlink RESULTS 1. Owner shows 'usern...@xx..x.xxx' for symlink, expecting 'username'. 2. Symlink is corrupted and can't removed, since it was created with an invalid user Sample program to create Symlink FileContext fc = FileContext.getFileContext(getConf()); fc.createSymlink(target, symlink, false); --- -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1189) Quota counts missed between clear quota and set quota
[ https://issues.apache.org/jira/browse/HDFS-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George reopened HDFS-1189: --- Need to submit patch for .20 release Quota counts missed between clear quota and set quota - Key: HDFS-1189 URL: https://issues.apache.org/jira/browse/HDFS-1189 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20.2, 0.22.0 Reporter: Kang Xiao Assignee: John George Labels: hdfs, quota Fix For: 0.21.1, Federation Branch, 0.22.0, 0.23.0 Attachments: HDFS-1189.patch, HDFS-1189.patch, HDFS-1189.patch, hdfs-1189-1.patch HDFS Quota counts will be missed between a clear quota operation and a set quota. When setting quota for a dir, the INodeDirectory will be replaced by INodeDirectoryWithQuota and dir.isQuotaSet() becomes true. When INodeDirectoryWithQuota is newly created, quota counting will be performed. However, when clearing quota, the quota conf is set to -1 and dir.isQuotaSet() becomes false while INodeDirectoryWithQuota will NOT be replaced back to INodeDirectory. FSDirectory.updateCount just update the quota count for inodes that isQuotaSet() is true. So after clear quota for a dir, its quota counts will not be updated and it's reasonable. But when re seting quota for this dir, quota counting will not be performed and some counts will be missed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1782) FSNamesystem.startFileInternal(..) throws NullPointerException
FSNamesystem.startFileInternal(..) throws NullPointerException -- Key: HDFS-1782 URL: https://issues.apache.org/jira/browse/HDFS-1782 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0 Reporter: John George Assignee: John George Fix For: 0.22.0 I'm observing when there is one balancer running trying to run another one results in Java.lang.NullPointerException error. I was hoping to see message Another balancer is running. Exiting Exiting This is a reproducible issue. Details 1) Cluster -elrond [hdfs@gsbl90568 smilli]$ hadoop version Hadoop 0.22.0.1102280202 Subversion git://hadoopre5.corp.sk1.yahoo.com/home/y/var/builds/thread2/workspace/Cloud-HadoopCOMMON-0.22-Secondary -r c7c9a21d7289e29f0133452acf8b761e455a84b5 Compiled by hadoopqa on Mon Feb 28 02:12:38 PST 2011 From source with checksum 9ecbc6f17e8847a1cddca2282dbd9b31 [hdfs@gsbl90568 smilli]$ 2) Run first balancer [hdfs@gsbl90565 smilli]$ hdfs balancer 11/03/09 16:33:56 INFO balancer.Balancer: namenodes = [gsbl90565.blue.ygrid.yahoo.com/98.137.97.57:8020, gsbl90569.blue.ygrid.yahoo.com/98.137.97.53:8020] 11/03/09 16:33:56 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved 11/03/09 16:33:57 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 11/03/09 16:33:57 INFO balancer.Balancer: Block token params received from NN: keyUpdateInterval=600 min(s), tokenLifetime=600 min(s) 11/03/09 16:33:57 INFO block.BlockTokenSecretManager: Setting block keys 11/03/09 16:33:57 INFO balancer.Balancer: Balancer will update its block keys every 150 minute(s) 11/03/09 16:33:57 INFO block.BlockTokenSecretManager: Setting block keys 11/03/09 16:33:57 INFO balancer.Balancer: Block token params received from NN: keyUpdateInterval=600 min(s), tokenLifetime=600 min(s) 11/03/09 16:33:57 INFO block.BlockTokenSecretManager: Setting block keys 11/03/09 16:33:57 INFO balancer.Balancer: Balancer will update its block keys every 150 minute(s) 11/03/09 16:33:57 INFO block.BlockTokenSecretManager: Setting block keys 11/03/09 16:33:57 INFO net.NetworkTopology: Adding a new node: /98.137.97.0/98.137.97.62:1004 11/03/09 16:33:57 INFO net.NetworkTopology: Adding a new node: /98.137.97.0/98.137.97.58:1004 11/03/09 16:33:57 INFO net.NetworkTopology: Adding a new node: /98.137.97.0/98.137.97.60:1004 11/03/09 16:33:57 INFO net.NetworkTopology: Adding a new node: /98.137.97.0/98.137.97.59:1004 11/03/09 16:33:57 INFO balancer.Balancer: 1 over-utilized: [Source[98.137.97.62:1004, utilization=24.152507825759344]] 11/03/09 16:33:57 INFO balancer.Balancer: 0 underutilized: [] 11/03/09 16:33:57 INFO balancer.Balancer: Need to move 207.98 GB to make the cluster balanced. 11/03/09 16:33:57 INFO balancer.Balancer: Decided to move 10 GB bytes from 98.137.97.62:1004 to 98.137.97.58:1004 11/03/09 16:33:57 INFO balancer.Balancer: Will move 10 GB in this iteration Mar 9, 2011 4:33:57 PM0 0 KB 207.98 GB 10 GB . . . 11/03/09 16:34:36 INFO balancer.Balancer: Moving block -63570336576981940 from 98.137.97.62:1004 to 98.137.97.59:1004 through 98.137.97.62:1004 is succeeded. 11/03/09 16:34:39 INFO balancer.Balancer: Moving block 2379736326585824737 from 98.137.97.62:1004 to 98.137.97.59:1004 through 98.137.97.62:1004 is succeeded. 11/03/09 16:35:21 INFO balancer.Balancer: Moving block 8884583953927078028 from 98.137.97.62:1004 to 98.137.97.59:1004 through 98.137.97.62:1004 is succeeded. 11/03/09 16:35:24 INFO balancer.Balancer: Moving block -135758138424743964 from 98.137.97.62:1004 to 98.137.97.59:1004 through 98.137.97.62:1004 is succeeded. 11/03/09 16:35:27 INFO balancer.Balancer: Moving block -4598153351946352185 from 98.137.97.62:1004 to 98.137.97.59:1004 through 98.137.97.62:1004 is succeeded. 11/03/09 16:35:33 INFO balancer.Balancer: Moving block 2966087210491094643 from 98.137.97.62:1004 to 98.137.97.59:1004 through 98.137.97.62:1004 is succeeded. 11/03/09 16:35:42 INFO balancer.Balancer: Moving block -5573983508500804184 from 98.137.97.62:1004 to 98.137.97.59:1004 through 98.137.97.62:1004 is succeeded. 11/03/09 16:35:58 INFO balancer.Balancer: Moving block -6222779741597113957 from 98.137.97.62:1004 to 98.137.97.59:1004 through 98.137.97.62:1004 is succeeded. 3) Run another balancer observe [hdfs@gsbl90568 smilli]$ hdfs balancer 11/03/09 16:34:32 INFO balancer.Balancer: namenodes = [gsbl90565.blue.ygrid.yahoo.com/98.137.97.57:8020, gsbl90569.blue.ygrid.yahoo.com/98.137.97.53:8020] 11/03/09 16:34:32 INFO balancer.Balancer: p = Balancer.Parameters[BalancingPolicy.Node, threshold=10.0] Time Stamp Iteration# Bytes Already
