[jira] [Commented] (HDFS-10850) getEZForPath should NOT throw FNF
[ https://issues.apache.org/jira/browse/HDFS-10850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483282#comment-15483282 ] Andrew Wang commented on HDFS-10850: [~spena] could you help shed some light on how Hive uses this API? Wondering if this is intentional, or if it could be worked around somehow. > getEZForPath should NOT throw FNF > - > > Key: HDFS-10850 > URL: https://issues.apache.org/jira/browse/HDFS-10850 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Assignee: Rakesh R >Priority: Blocker > > HDFS-9433 made an incompatible change to the semantics of getEZForPath. It > used to return the EZ of the closest ancestor path. It never threw FNF. A > common use of getEZForPath to determining if a file can be renamed, or must > be copied due to mismatched EZs. Notably, this has broken hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[ https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15483279#comment-15483279 ] Vinayakumar B commented on HDFS-10838: -- {quote}Was having one doubt about the compatibility of dfsadmin -report output. Allen Wittenauer, Can you confirm addition of new line "Last Block Report: ").append(new Date(lastBlockReportTime)) for each DN , whether this will be incompatible change? Asking this doubt, because its a new addition in separate line. No modifications to existing output lines.{quote} Related to compatibility of 'dfsadmin -report', we can mark this jira as incompatible as per [docs| http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoop-project-dist/hadoop-common/Compatibility.html#Command_Line_Interface_CLI] or Can separate only 'dfsadmin -report' related changes to separate Jira which can be present in only trunk and push remaining changes to branch-2* as well. > Last full block report received time for each DN should be easily discoverable > -- > > Key: HDFS-10838 > URL: https://issues.apache.org/jira/browse/HDFS-10838 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Arpit Agarwal >Assignee: Surendra Singh Lilhore > Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch, > HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch, NN_UI.png, > NN_UI_relative_time.png, NN_UI_relative_time_minutes.png > > > It should be easy for administrators to discover the time of last full block > report from each DataNode. > We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin > -report}}, or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10855) Fix typos in HDFS documents
[ https://issues.apache.org/jira/browse/HDFS-10855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482731#comment-15482731 ] Yiqun Lin commented on HDFS-10855: -- Thanks [~xiaochen] for the review and commit! > Fix typos in HDFS documents > > > Key: HDFS-10855 > URL: https://issues.apache.org/jira/browse/HDFS-10855 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10855.001.patch > > > Now there many typos in many HDFS documents. The detail typos info: > * {{HDFSHighAvailabilityWithNFS.md}} > Beacuse->Because > processs->process > * {{ArchivalStorage.md}} > specificed->specified > * {{ViewFs.md}} > Futher->Further > * {{HdfsNfsGateway.md}} > differnt->different > regrulation->regulation > * {{HdfsMultihoming.md}}, {{hdfs-default.xml}} > adress->address -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9333) Some tests using MiniDFSCluster errored complaining port in use
[ https://issues.apache.org/jira/browse/HDFS-9333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482672#comment-15482672 ] Masatake Iwasaki commented on HDFS-9333: Thanks for the comment, [~andrew.wang]. I'm looking for the way to make the test work when restarting datanodes on random ports by {{MiniDFSCluster#restartDataNodes(false)}}. Since the test retrieves LocatedBlocks and use it even after DN restart, it might not be trivial fix. > Some tests using MiniDFSCluster errored complaining port in use > --- > > Key: HDFS-9333 > URL: https://issues.apache.org/jira/browse/HDFS-9333 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Kai Zheng >Assignee: Masatake Iwasaki >Priority: Minor > Attachments: HDFS-9333.001.patch, HDFS-9333.002.patch > > > Ref. the following: > {noformat} > Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 30.483 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped > testRead(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped) > Time elapsed: 11.021 sec <<< ERROR! > java.net.BindException: Port in use: localhost:49333 > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:433) > at sun.nio.ch.Net.bind(Net.java:425) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216) > at > org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:884) > at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:826) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:821) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:675) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555) > at > org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:2015) > at > org.apache.hadoop.hdfs.MiniDFSCluster.restartNameNode(MiniDFSCluster.java:1996) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS.doTestRead(TestBlockTokenWithDFS.java:539) > at > org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped.testRead(TestBlockTokenWithDFSStriped.java:62) > {noformat} > Another one: > {noformat} > Tests run: 5, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 9.859 sec <<< > FAILURE! - in org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController > testFailoverAndBackOnNNShutdown(org.apache.hadoop.hdfs.tools.TestDFSZKFailoverController) > Time elapsed: 0.41 sec <<< ERROR! > java.net.BindException: Problem binding to [localhost:10021] > java.net.BindException: Address already in use; For more details see: > http://wiki.apache.org/hadoop/BindException > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:433) > at sun.nio.ch.Net.bind(Net.java:425) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at org.apache.hadoop.ipc.Server.bind(Server.java:469) > at org.apache.hadoop.ipc.Server$Listener.(Server.java:695) > at org.apache.hadoop.ipc.Server.(Server.java:2464) > at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:535) > at > org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:510) > at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:399) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:742) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:680) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:883) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:862) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1555) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:1245) > at > org.apache.hadoop.hdfs.MiniDFSCluster.confi
[jira] [Commented] (HDFS-10301) BlockReport retransmissions may lead to storages falsely being declared zombie if storage report processing happens out of order
[ https://issues.apache.org/jira/browse/HDFS-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482454#comment-15482454 ] Arpit Agarwal commented on HDFS-10301: -- IIUC we need to fix this logic not just for pruning storages but also deciding when to remove the block report lease. >From BPServiceActor.java, we can assume at line 399 that the storage report >just sent was processed successfully by the NameNode. i.e. DataNode getting >back success is sufficient to conclude the report was successfully processed. {code} 393 for (int r = 0; r < reports.length; r++) { 394 StorageBlockReport singleReport[] = { reports[r] }; 395 DatanodeCommand cmd = bpNamenode.blockReport( 396 bpRegistration, bpos.getBlockPoolId(), singleReport, 397 new BlockReportContext(reports.length, r, reportId, 398 fullBrLeaseId, true)); 399 blockReportSizes.add( 400 calculateBlockReportPBSize(useBlocksBuffer, singleReport)); 401 numReportsSent++; 402 numRPCs++; 403 if (cmd != null) { 404 cmds.add(cmd); 405 } {code} The DN can include a flag in the last RPC message i.e. when {{r == reports.length - 1}} that tells the NameNode it is the last report in this batch and all previous ones were successfully processed. So it's safe to drop the lease and prune zombies. Also +1 for [~daryn]'s idea to ban single-RPC reports, as this approach cannot be used for single-RPC reports. > BlockReport retransmissions may lead to storages falsely being declared > zombie if storage report processing happens out of order > > > Key: HDFS-10301 > URL: https://issues.apache.org/jira/browse/HDFS-10301 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.6.1 >Reporter: Konstantin Shvachko >Assignee: Vinitha Reddy Gankidi >Priority: Critical > Fix For: 2.7.4 > > Attachments: HDFS-10301.002.patch, HDFS-10301.003.patch, > HDFS-10301.004.patch, HDFS-10301.005.patch, HDFS-10301.006.patch, > HDFS-10301.007.patch, HDFS-10301.008.patch, HDFS-10301.009.patch, > HDFS-10301.01.patch, HDFS-10301.010.patch, HDFS-10301.011.patch, > HDFS-10301.012.patch, HDFS-10301.013.patch, HDFS-10301.branch-2.7.patch, > HDFS-10301.branch-2.patch, HDFS-10301.sample.patch, zombieStorageLogs.rtf > > > When NameNode is busy a DataNode can timeout sending a block report. Then it > sends the block report again. Then NameNode while process these two reports > at the same time can interleave processing storages from different reports. > This screws up the blockReportId field, which makes NameNode think that some > storages are zombie. Replicas from zombie storages are immediately removed, > causing missing blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[ https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482336#comment-15482336 ] Arpit Agarwal commented on HDFS-10838: -- By the way there may be some update needed in this logic, per HDFS-10301. Looking at that too. > Last full block report received time for each DN should be easily discoverable > -- > > Key: HDFS-10838 > URL: https://issues.apache.org/jira/browse/HDFS-10838 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Arpit Agarwal >Assignee: Surendra Singh Lilhore > Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch, > HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch, NN_UI.png, > NN_UI_relative_time.png, NN_UI_relative_time_minutes.png > > > It should be easy for administrators to discover the time of last full block > report from each DataNode. > We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin > -report}}, or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[ https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482335#comment-15482335 ] Arpit Agarwal commented on HDFS-10838: -- Hi [~surendrasingh], you may have to reattach the patch with another name. Jenkins seems to be picking up the latest file attached without looking at the extension. I looked at this some more and the {{lastStorageInRpc}} check will be insufficient. We have to look into the BlockReportContext to determine when the last report has been received. e.g. this block of code earlier on in {{processReport}}.. {code} if (context != null) { storageInfo.setLastBlockReportId(context.getReportId()); if (lastStorageInRpc) { int rpcsSeen = node.updateBlockReportContext(context); if (rpcsSeen >= context.getTotalRpcs()) { long leaseId = blockReportLeaseManager.removeLease(node); {code} I'll also take a look at this and help with updating the patch to ensure we only update this timestamp when we are sure we have processed all storage RPCs. > Last full block report received time for each DN should be easily discoverable > -- > > Key: HDFS-10838 > URL: https://issues.apache.org/jira/browse/HDFS-10838 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Arpit Agarwal >Assignee: Surendra Singh Lilhore > Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch, > HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch, NN_UI.png, > NN_UI_relative_time.png, NN_UI_relative_time_minutes.png > > > It should be easy for administrators to discover the time of last full block > report from each DataNode. > We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin > -report}}, or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[ https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-10838: -- Status: Patch Available (was: Open) Resubmitting patch, hope it will take last uploaded patch > Last full block report received time for each DN should be easily discoverable > -- > > Key: HDFS-10838 > URL: https://issues.apache.org/jira/browse/HDFS-10838 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Arpit Agarwal >Assignee: Surendra Singh Lilhore > Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch, > HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch, NN_UI.png, > NN_UI_relative_time.png, NN_UI_relative_time_minutes.png > > > It should be easy for administrators to discover the time of last full block > report from each DataNode. > We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin > -report}}, or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[ https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-10838: -- Status: Open (was: Patch Available) > Last full block report received time for each DN should be easily discoverable > -- > > Key: HDFS-10838 > URL: https://issues.apache.org/jira/browse/HDFS-10838 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Arpit Agarwal >Assignee: Surendra Singh Lilhore > Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch, > HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch, NN_UI.png, > NN_UI_relative_time.png, NN_UI_relative_time_minutes.png > > > It should be easy for administrators to discover the time of last full block > report from each DataNode. > We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin > -report}}, or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[ https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482229#comment-15482229 ] Surendra Singh Lilhore commented on HDFS-10838: --- Thanks [~arpitagarwal]. I think some problem in jenkins build, its treating image file as patch. {code} Processing: HDFS-10838 cut: invalid byte, character or field list Try 'cut --help' for more information. cut: invalid byte, character or field list Try 'cut --help' for more information. HDFS-10838 patch is being downloaded at Sun Sep 11 10:48:13 UTC 2016 from https://issues.apache.org/jira/secure/attachment/12827924/NN_UI_relative_time_minutes.png -> Downloaded ERROR: Unsure how to process HDFS-10838. {code} > Last full block report received time for each DN should be easily discoverable > -- > > Key: HDFS-10838 > URL: https://issues.apache.org/jira/browse/HDFS-10838 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Arpit Agarwal >Assignee: Surendra Singh Lilhore > Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch, > HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch, NN_UI.png, > NN_UI_relative_time.png, NN_UI_relative_time_minutes.png > > > It should be easy for administrators to discover the time of last full block > report from each DataNode. > We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin > -report}}, or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[ https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15482030#comment-15482030 ] Arpit Agarwal commented on HDFS-10838: -- +1 pending Jenkins. Thanks [~surendrasingh]. One remaining point is treating 0 as special case in the web UI also, just like the command-line. It can be done in a followup Jira. > Last full block report received time for each DN should be easily discoverable > -- > > Key: HDFS-10838 > URL: https://issues.apache.org/jira/browse/HDFS-10838 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Arpit Agarwal >Assignee: Surendra Singh Lilhore > Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch, > HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch, NN_UI.png, > NN_UI_relative_time.png, NN_UI_relative_time_minutes.png > > > It should be easy for administrators to discover the time of last full block > report from each DataNode. > We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin > -report}}, or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10838) Last full block report received time for each DN should be easily discoverable
[ https://issues.apache.org/jira/browse/HDFS-10838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-10838: -- Attachment: NN_UI_relative_time_minutes.png HDFS-10838.004.patch Thanks [~arpitagarwal] for clarifying.. Attached updated patch. Please review.. > Last full block report received time for each DN should be easily discoverable > -- > > Key: HDFS-10838 > URL: https://issues.apache.org/jira/browse/HDFS-10838 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Arpit Agarwal >Assignee: Surendra Singh Lilhore > Attachments: DFSAdmin-Report.png, HDFS-10838-001.patch, > HDFS-10838.002.patch, HDFS-10838.003.patch, HDFS-10838.004.patch, NN_UI.png, > NN_UI_relative_time.png, NN_UI_relative_time_minutes.png > > > It should be easy for administrators to discover the time of last full block > report from each DataNode. > We can show it in the NameNode web UI or in the output of {{hdfs dfsadmin > -report}}, or both. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org