[jira] [Commented] (HDFS-12287) Remove a no-longer applicable TODO comment in DatanodeManager
[ https://issues.apache.org/jira/browse/HDFS-12287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122900#comment-16122900 ] Yiqun Lin commented on HDFS-12287: -- +1, will commit this shortly. > Remove a no-longer applicable TODO comment in DatanodeManager > - > > Key: HDFS-12287 > URL: https://issues.apache.org/jira/browse/HDFS-12287 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Trivial > Attachments: HDFS-12287.001.patch > > > {{DatanodeManager}} has this this TODO comment > {code} > // TODO: Enables DFSNetworkTopology by default after more stress > // testings/validations. > {code} > This has been resolved in HDFS-11998, but it missed removing this comment. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122895#comment-16122895 ] Hadoop QA commented on HDFS-12196: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 42s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 23s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 48s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 52s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 13s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 45s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 23s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 48s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}136m 32s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs | | | Thread passed where Runnable expected in org.apache.hadoop.utils.BackgroundService.start() At BackgroundService.java:in org.apache.hadoop.utils.BackgroundService.start() At BackgroundService.java:[line 85] | | Failed junit tests | hadoop.scm.TestArchive | | | hadoop.cblock.TestCBlockReadWrite | | | hadoop.hdfs.TestMaintenanceState | | | hadoop.ozone.container.common.TestDatanodeStateMachine | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.cblock.TestBufferManager | | | hadoop.ozone.web.client.TestKeys | | Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis | | | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12196 |
[jira] [Commented] (HDFS-11738) Hedged pread takes more time when block moved from initial locations
[ https://issues.apache.org/jira/browse/HDFS-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122889#comment-16122889 ] John Zhuge commented on HDFS-11738: --- [~zhangchen] is the original author. [~jojochuang] and I just tried to help move it forward. I am ok with [~vinayrpet]'s plan since [~zhangchen]'s work in DFSInputStream and the new unit test deserves the credit. > Hedged pread takes more time when block moved from initial locations > > > Key: HDFS-11738 > URL: https://issues.apache.org/jira/browse/HDFS-11738 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-11738-01.patch, HDFS-11738-02.patch > > > Scenario : > Same as HDFS-11708. > During Hedge read, > 1. First two locations fails to read the data in hedged mode. > 2. chooseData refetches locations and adds a future to read from DN3. > 3. after adding future to DN3, main thread goes for refetching locations in > loop and stucks there till all 3 retries to fetch locations exhausted, which > consumes ~20 seconds with exponential retry time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11738) Hedged pread takes more time when block moved from initial locations
[ https://issues.apache.org/jira/browse/HDFS-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122878#comment-16122878 ] Vinayakumar B commented on HDFS-11738: -- I see that, Following changes in {{DFSInputStream#hedgedFetchBlockByteRange(..)}} present in both HDFS-11303 and this jira, {code} futures.add(firstRequest); +Future future = null; try { - Future future = hedgedService.poll( + future = hedgedService.poll( conf.getHedgedReadThresholdMillis(), TimeUnit.MILLISECONDS); if (future != null) { ByteBuffer result = future.get(); @@ -1142,16 +1143,18 @@ private void hedgedFetchBlockByteRange(LocatedBlock block, long start, } DFSClient.LOG.debug("Waited {}ms to read from {}; spawning hedged " + "read", conf.getHedgedReadThresholdMillis(), chosenNode.info); - // Ignore this node on next go around. - ignored.add(chosenNode.info); dfsClient.getHedgedReadMetrics().incHedgedReadOps(); // continue; no need to refresh block locations } catch (ExecutionException e) { - // Ignore + futures.remove(future); } catch (InterruptedException e) { throw new InterruptedIOException( "Interrupted while waiting for reading task"); } +// Ignore this node on next go around. +// If poll timeout and the request still ongoing, don't consider it +// again. If read data failed, don't consider it either. +ignored.add(chosenNode.info); } else { // We are starting up a 'hedged' read. We have a read already // ongoing. Call getBestNodeDNAddrPair instead of chooseDataNode. {code} I think its fair to commit HDFS-11303 first and give credit for [~jzhuge]'s efforts, as it has associated test written along with the change. I can update the patch again once HDFS-11303 committed. Anyway test in my patch will fail even after HDFS-11303 is committed. i.e. HDFS-11303 is not exactly the fix for this issue. So lets get HDFS-11303 committed first. > Hedged pread takes more time when block moved from initial locations > > > Key: HDFS-11738 > URL: https://issues.apache.org/jira/browse/HDFS-11738 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-11738-01.patch, HDFS-11738-02.patch > > > Scenario : > Same as HDFS-11708. > During Hedge read, > 1. First two locations fails to read the data in hedged mode. > 2. chooseData refetches locations and adds a future to read from DN3. > 3. after adding future to DN3, main thread goes for refetching locations in > loop and stucks there till all 3 retries to fetch locations exhausted, which > consumes ~20 seconds with exponential retry time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11738) Hedged pread takes more time when block moved from initial locations
[ https://issues.apache.org/jira/browse/HDFS-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122848#comment-16122848 ] Wei-Chiu Chuang commented on HDFS-11738: Thanks [~vinayrpet] for the patch! I haven't reviewed the patch fully, but looking at the patch, this jira is the superset of HDFS-11303. Perhaps we should give up HDFS-11303 and focus on this one. [~jzhuge] how do you think? > Hedged pread takes more time when block moved from initial locations > > > Key: HDFS-11738 > URL: https://issues.apache.org/jira/browse/HDFS-11738 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-11738-01.patch, HDFS-11738-02.patch > > > Scenario : > Same as HDFS-11708. > During Hedge read, > 1. First two locations fails to read the data in hedged mode. > 2. chooseData refetches locations and adds a future to read from DN3. > 3. after adding future to DN3, main thread goes for refetching locations in > loop and stucks there till all 3 retries to fetch locations exhausted, which > consumes ~20 seconds with exponential retry time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11303) Hedged read might hang infinitely if read data from all DN failed
[ https://issues.apache.org/jira/browse/HDFS-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-11303: -- Status: In Progress (was: Patch Available) Looking into TestPread and TestDFSStripedInputStreamWithRandomECPolicy failure. > Hedged read might hang infinitely if read data from all DN failed > -- > > Key: HDFS-11303 > URL: https://issues.apache.org/jira/browse/HDFS-11303 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.0.0-alpha1 >Reporter: Chen Zhang >Assignee: Chen Zhang > Attachments: HDFS-11303-001.patch, HDFS-11303-001.patch, > HDFS-11303-002.patch, HDFS-11303-002.patch, HDFS-11303.003.patch, > HDFS-11303.004.patch > > > Hedged read will read from a DN first, if timeout, then read other DNs > simultaneously. > If read all DN failed, this bug will cause the future-list not empty(the > first timeout request left in list), and hang in the loop infinitely -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122809#comment-16122809 ] Anu Engineer commented on HDFS-12196: - [~cheersyang] Thanks for updating the patch. +1 on the v6 patch, pending Jenkins. > Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks > at background > - > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch, > HDFS-12196-HDFS-7240.006.patch > > > Implement a block deleting service running on datanode to delete stale > blocks. The service scans staled blocks for each container and delete chunks > and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12196) Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12196: --- Description: Implement a block deleting service running on datanode to delete stale blocks. The service scans staled blocks for each container and delete chunks and references periodically. (was: Implement a recycling service running on datanode to delete stale blocks. The recycling service scans staled blocks for each container and delete chunks and references periodically.) > Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks > at background > - > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch, > HDFS-12196-HDFS-7240.006.patch > > > Implement a block deleting service running on datanode to delete stale > blocks. The service scans staled blocks for each container and delete chunks > and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12196) Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12196: --- Summary: Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks at background (was: Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background) > Ozone: DeleteKey-2: Implement block deleting service to delete stale blocks > at background > - > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch, > HDFS-12196-HDFS-7240.006.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12196: --- Attachment: HDFS-12196-HDFS-7240.006.patch Revised the names, java docs etc in v6 patch according to [~anu]'s latest comment. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch, > HDFS-12196-HDFS-7240.006.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12255) Block Storage: Cblock should generated unique trace ID for the ops
[ https://issues.apache.org/jira/browse/HDFS-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122803#comment-16122803 ] Mukul Kumar Singh commented on HDFS-12255: -- [~vagarychen] I have addressed the comments in the v3 patch. Please have a look. [~anu], the error is being logged in the function which is being called as part of the constructor for ContainerCacheFlusher. So this error will be generated only once for each flusher initialization. > Block Storage: Cblock should generated unique trace ID for the ops > -- > > Key: HDFS-12255 > URL: https://issues.apache.org/jira/browse/HDFS-12255 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12255-HDFS-7240.001.patch, > HDFS-12255-HDFS-7240.002.patch, HDFS-12255-HDFS-7240.003.patch > > > Cblock tests fails because cblock does not generate unique trace id for each > op. > {code} > java.lang.AssertionError: expected:<0> but was:<1051> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.cblock.TestBufferManager.testRepeatedBlockWrites(TestBufferManager.java:448) > {code} > This failure is because of following error. > {code} > 017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > scm.XceiverClientHandler (XceiverClientHandler.java:sendCommandAsync(134)) - > Command with Trace already exists. Ignoring this command. . Previous Command: > java.util.concurrent.CompletableFuture@7847fc2d[Not completed, 1 dependents] > 2017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > jscsiHelper.ContainerCacheFlusher (BlockWriterTask.java:run(108)) - Writing > of block:44 failed, We have attempted to write this block 7 tim > es to the container container2483304118.Trace ID: > java.lang.IllegalStateException: Duplicate trace ID. Command with this trace > ID is already executing. Please ensure that trace IDs are not reused. ID: > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommandAsync(XceiverClientHandler.java:139) > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommand(XceiverClientHandler.java:114) > at > org.apache.hadoop.scm.XceiverClient.sendCommand(XceiverClient.java:132) > at > org.apache.hadoop.scm.storage.ContainerProtocolCalls.writeSmallFile(ContainerProtocolCalls.java:225) > at > org.apache.hadoop.cblock.jscsiHelper.BlockWriterTask.run(BlockWriterTask.java:97) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12255) Block Storage: Cblock should generated unique trace ID for the ops
[ https://issues.apache.org/jira/browse/HDFS-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12255: - Attachment: HDFS-12255-HDFS-7240.003.patch > Block Storage: Cblock should generated unique trace ID for the ops > -- > > Key: HDFS-12255 > URL: https://issues.apache.org/jira/browse/HDFS-12255 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12255-HDFS-7240.001.patch, > HDFS-12255-HDFS-7240.002.patch, HDFS-12255-HDFS-7240.003.patch > > > Cblock tests fails because cblock does not generate unique trace id for each > op. > {code} > java.lang.AssertionError: expected:<0> but was:<1051> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.cblock.TestBufferManager.testRepeatedBlockWrites(TestBufferManager.java:448) > {code} > This failure is because of following error. > {code} > 017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > scm.XceiverClientHandler (XceiverClientHandler.java:sendCommandAsync(134)) - > Command with Trace already exists. Ignoring this command. . Previous Command: > java.util.concurrent.CompletableFuture@7847fc2d[Not completed, 1 dependents] > 2017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > jscsiHelper.ContainerCacheFlusher (BlockWriterTask.java:run(108)) - Writing > of block:44 failed, We have attempted to write this block 7 tim > es to the container container2483304118.Trace ID: > java.lang.IllegalStateException: Duplicate trace ID. Command with this trace > ID is already executing. Please ensure that trace IDs are not reused. ID: > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommandAsync(XceiverClientHandler.java:139) > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommand(XceiverClientHandler.java:114) > at > org.apache.hadoop.scm.XceiverClient.sendCommand(XceiverClient.java:132) > at > org.apache.hadoop.scm.storage.ContainerProtocolCalls.writeSmallFile(ContainerProtocolCalls.java:225) > at > org.apache.hadoop.cblock.jscsiHelper.BlockWriterTask.run(BlockWriterTask.java:97) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12289) [READ] HDFS-12091 breaks the tests for provided block reads
[ https://issues.apache.org/jira/browse/HDFS-12289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12289: -- Summary: [READ] HDFS-12091 breaks the tests for provided block reads (was: HDFS-12091 breaks the tests for provided block reads) > [READ] HDFS-12091 breaks the tests for provided block reads > --- > > Key: HDFS-12289 > URL: https://issues.apache.org/jira/browse/HDFS-12289 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12289-HDFS-9806.001.patch > > > In the tests within {{TestNameNodeProvidedImplementation}}, the files that > are supposed to belong to a provided volume are not located under the Storage > directory assigned to the volume in {{MiniDFSCluster}}. With HDFS-12091, this > isn't correct and thus, it breaks the tests. This JIRA is to fix the tests > under {{TestNameNodeProvidedImplementation}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122790#comment-16122790 ] Weiwei Yang commented on HDFS-12196: Hi [~anu] Thanks for your quick response. I will address these comments in this jira with next patch, they are not big changes so lets address them here. Thank you. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122786#comment-16122786 ] Anu Engineer commented on HDFS-12196: - [~cheersyang] +1 for v5 patch, pending Jenkins. I have a small nit, which *we don't need to address* in this patch. My apologies that I did not spot it earlier. Instead of the word *recycling* can we use the word *delete*? I feel that it is easier to understand. Rename ContainerRecyclingService to something like BlockDeletingService. Also, change the comment for this class to something like: {noformat} A per-datanode block deleting service that deletes blocks from active containers. {noformat} You don't have to do this now, please feel free to commit. I know you have 3 more checkins pending on this, so feel free to modify this name is some later patch. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122780#comment-16122780 ] Weiwei Yang commented on HDFS-12196: Thanks [~anu] for the comments. Per offline discussion, since we will reuse background service for multiple components, KSM, SCM as well as datanode, lets use separate thread pools. For rest of your comments bq. Generally we expect the background service to be a deamon thread. Fixed. Now it is constructed like following way {code} ThreadFactory tf = r -> new Thread(threadGroup, r); threadFactory = new ThreadFactoryBuilder() .setThreadFactory(tf) .setDaemon(true) .setNameFormat(serviceName + "#%d") .build(); {code} this is to ensure we have all threads contained in a thread group, to manage them all together. So we can get # of running threads for this service if necessary. bq. Question : Why do we need the testing flag and testing Thread? I have refactor that to a single-test-purpose class {{ContainerRecyclingServicetTestImpl}}, instead of waiting for intervals, the test class run each cycle by a function call, so that I can write more finer-grained UT cases. bq. BackgroundTaskQueue.java is not thread safe, is it by design? This class doesn't have to be thread safe as there is only 1 thread to fetch the tasks. But I use a thread safe implementation in the new patch, in case in future we need to access it in multiple threads. bq. Should we create a new package called background tasks ... Fixed. Thank you! > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12196: --- Attachment: HDFS-12196-HDFS-7240.005.patch > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch, HDFS-12196-HDFS-7240.005.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12289) HDFS-12091 breaks the tests for provided block reads
[ https://issues.apache.org/jira/browse/HDFS-12289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122745#comment-16122745 ] Hadoop QA commented on HDFS-12289: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HDFS-9806 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 9s{color} | {color:green} HDFS-9806 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 24s{color} | {color:green} HDFS-9806 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 8s{color} | {color:green} HDFS-9806 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s{color} | {color:green} HDFS-9806 passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 32s{color} | {color:red} hadoop-tools/hadoop-fs2img in HDFS-9806 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s{color} | {color:green} HDFS-9806 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 20s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 10s{color} | {color:orange} root: The patch generated 3 new + 219 unchanged - 2 fixed = 222 total (was 221) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 18s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s{color} | {color:green} hadoop-fs2img in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}138m 56s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | | | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12289 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881367/HDFS-12289-HDFS-9806.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 7c429296baf9 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-9806 / 5c2a0a1 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20654/artifact/patchprocess/branch-findbugs-hadoop-tools_hadoop-fs2img-warnings.html | | checkstyle |
[jira] [Commented] (HDFS-12285) Better handling of namenode ip address change
[ https://issues.apache.org/jira/browse/HDFS-12285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122746#comment-16122746 ] Ming Ma commented on HDFS-12285: Thanks [~shahrs87]. Yeah indeed related, although the exception and the scenario look different from the other jiras. Even if it is the same, let us keep this jira around for validation when we resolve the issue. > Better handling of namenode ip address change > - > > Key: HDFS-12285 > URL: https://issues.apache.org/jira/browse/HDFS-12285 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Ming Ma > > RPC client layer provides functionality to detect ip address change: > {noformat} > Client.java > private synchronized boolean updateAddress() throws IOException { > // Do a fresh lookup with the old host name. > InetSocketAddress currentAddr = NetUtils.createSocketAddrForHost( >server.getHostName(), server.getPort()); > .. > } > {noformat} > To use this feature, we need to enable retry via > {{dfs.client.retry.policy.enabled}}. Otherwise {{TryOnceThenFail}} > RetryPolicy will be used; which caused {{handleConnectionFailure}} to throw > {{ConnectException}} exception without retrying with the new ip address. > {noformat} > private void handleConnectionFailure(int curRetries, IOException ioe > ) throws IOException { > closeConnection(); > final RetryAction action; > try { > action = connectionRetryPolicy.shouldRetry(ioe, curRetries, 0, true); > } catch(Exception e) { > throw e instanceof IOException? (IOException)e: new IOException(e); > } > .. > } > {noformat} > However, using such configuration isn't ideal. What happens is DFSClient > still holds onto the cached old ip address created by {{namenode = > proxyInfo.getProxy();}}. Thus when a new rpc connection is created, it starts > with the old ip followed by retry with the new ip. It will be nice if > DFSClient can update namenode proxy automatically upon ip address change. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12066) When Namenode is in safemode,may not allowed to remove an user's erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122714#comment-16122714 ] Hadoop QA commented on HDFS-12066: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 7s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}113m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.tools.TestDFSZKFailoverController | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12066 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881363/HDFS-12066.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux e78e3a83fb84 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a32e013 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20652/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20652/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20652/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20652/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > When Namenode is in safemode,may not allowed to remove
[jira] [Commented] (HDFS-12054) FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode
[ https://issues.apache.org/jira/browse/HDFS-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122706#comment-16122706 ] Hadoop QA commented on HDFS-12054: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 55s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}103m 4s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | | | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12054 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881362/HDFS-12054.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux f454cf15e092 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a32e013 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20653/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20653/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20653/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20653/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to > ensure Namenode
[jira] [Commented] (HDFS-11882) Client fails if acknowledged size is greater than bytes sent
[ https://issues.apache.org/jira/browse/HDFS-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122680#comment-16122680 ] Hadoop QA commented on HDFS-11882: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 28s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 43s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 42s{color} | {color:orange} hadoop-hdfs-project: The patch generated 5 new + 8 unchanged - 0 fixed = 13 total (was 8) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 13s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 10s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}118m 56s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11882 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881353/HDFS-11882.03.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux d4fd13e59a5a 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a32e013 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20651/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html | | findbugs |
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122677#comment-16122677 ] Andrew Wang commented on HDFS-11082: Hi Sammi, thanks for working on this, sorry for the delay reviewing, * I think we should have {{getErasureCodingPolicy}} return the effective EC policy (null) for both files and directories, for ease of use. Yes, let's file a follow-on JIRA for {{getActualErasureCodingPolicy}}. * Grammar/typo in doc: "enfore the direcotry" -> "force the directory" * How hard is it to further hide the special replication policy name, e.g. just call it "REPLICATION" or better, set it with a command like "hdfs ec setPolicy -replicate" and don't expose it as a new policy? I think Rakesh recommended this before, and though worse from a code perspective, it's better from a user perspective. * If we can't do the above, it'd make the suggestion of suing some more fake looking parameters like "REPLICATION-0-0-1K" to make it clear that these parameters are meaningless. > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11082.001.patch, HDFS-11082.002.patch > > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122647#comment-16122647 ] Hadoop QA commented on HDFS-12159: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 46s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 24 new or modified test files. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 57s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 53s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 0s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 54s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 44s{color} | {color:orange} hadoop-hdfs-project: The patch generated 92 new + 60 unchanged - 0 fixed = 152 total (was 60) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 44s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 1s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}144m 13s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client | | | Format-string method String.format(String, Object[]) called with format string "create replication pipeline failed. code :with format string "create replication pipeline failed. code : {} Message: {}" wants 0 arguments but is given 2 in org.apache.hadoop.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.createReplicationPipeline(OzoneProtos$ReplicationType, OzoneProtos$ReplicationFactor, OzoneProtos$NodePool) At StorageContainerLocationProtocolClientSideTranslatorPB.java:[line 237] | | Failed junit tests | hadoop.scm.TestArchive | | | hadoop.ozone.web.client.TestKeys | | | hadoop.cblock.TestCBlockReadWrite | | | hadoop.cblock.TestBufferManager | | |
[jira] [Commented] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122645#comment-16122645 ] Hadoop QA commented on HDFS-12159: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 24 new or modified test files. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 34s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 54s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 55s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 47s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 46s{color} | {color:orange} hadoop-hdfs-project: The patch generated 92 new + 60 unchanged - 0 fixed = 152 total (was 60) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 46s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 5s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}124m 1s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client | | | Format-string method String.format(String, Object[]) called with format string "create replication pipeline failed. code :with format string "create replication pipeline failed. code : {} Message: {}" wants 0 arguments but is given 2 in org.apache.hadoop.scm.protocolPB.StorageContainerLocationProtocolClientSideTranslatorPB.createReplicationPipeline(OzoneProtos$ReplicationType, OzoneProtos$ReplicationFactor, OzoneProtos$NodePool) At StorageContainerLocationProtocolClientSideTranslatorPB.java:[line 237] | | Failed junit tests | hadoop.scm.TestArchive | | | hadoop.ozone.web.client.TestKeys | | | hadoop.cblock.TestCBlockReadWrite | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | | |
[jira] [Commented] (HDFS-12283) Ozone: DeleteKey-5: Implement SCM DeletedBlockLog
[ https://issues.apache.org/jira/browse/HDFS-12283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122631#comment-16122631 ] Anu Engineer commented on HDFS-12283: - +1, I am perfectly fine with doing this using RocksDB.RocksDB internally is an LSM tree, which is based on logs. > Ozone: DeleteKey-5: Implement SCM DeletedBlockLog > - > > Key: HDFS-12283 > URL: https://issues.apache.org/jira/browse/HDFS-12283 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, scm >Reporter: Weiwei Yang >Assignee: Yuanbo Liu > > The DeletedBlockLog is a persisted log in SCM to keep tracking container > blocks which are under deletion. It maintains info about under-deletion > container blocks that notified by KSM, and the state how it is processed. We > can use RocksDB to implement the 1st version of the log, the schema looks like > ||TxID||ContainerName||Block List||ProcessedCount|| > |0|c1|b1,b2,b3|0| > |1|c2|b1|3| > |2|c2|b2, b3|-1| > Some explanations > # TxID is an incremental long value transaction ID for ONE container and > multiple blocks > # Container name is the name of the container > # Block list is a list of block IDs > # ProcessedCount is the number of times SCM has sent this record to datanode, > it represents the "state" of the transaction, it is in range of \[-1, 5\], -1 > means the transaction eventually failed after some retries, 5 is the max > number times of retries. > We need to define {{DeletedBlockLog}} as an interface and implement this with > RocksDB {{MetadataStore}} as the first version. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11303) Hedged read might hang infinitely if read data from all DN failed
[ https://issues.apache.org/jira/browse/HDFS-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122629#comment-16122629 ] Hadoop QA commented on HDFS-11303: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 42s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 32s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 18s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 29s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}136m 20s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics | | | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | | | hadoop.hdfs.TestRollingUpgrade | | | hadoop.hdfs.TestFileAppend | | | hadoop.hdfs.TestPread | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11303 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881344/HDFS-11303.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 6a57954c0fac 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a32e013 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs |
[jira] [Commented] (HDFS-12222) Add EC information to BlockLocation
[ https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122627#comment-16122627 ] Huafeng Wang commented on HDFS-1: - Hi [~drankye], you're right. I think it's a better way and I'll give it a try. > Add EC information to BlockLocation > --- > > Key: HDFS-1 > URL: https://issues.apache.org/jira/browse/HDFS-1 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Huafeng Wang > Labels: hdfs-ec-3.0-nice-to-have > > HDFS applications query block location information to compute splits. One > example of this is FileInputFormat: > https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346 > You see bits of code like this that calculate offsets as follows: > {noformat} > long bytesInThisBlock = blkLocations[startIndex].getOffset() + > blkLocations[startIndex].getLength() - offset; > {noformat} > EC confuses this since the block locations include parity block locations as > well, which are not part of the logical file length. This messes up the > offset calculation and thus topology/caching information too. > Applications can figure out what's a parity block by reading the EC policy > and then parsing the schema, but it'd be a lot better if we exposed this more > generically in BlockLocation instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11882) Client fails if acknowledged size is greater than bytes sent
[ https://issues.apache.org/jira/browse/HDFS-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122621#comment-16122621 ] Kai Zheng commented on HDFS-11882: -- Thanks [~ajisakaa] for the clarifying. I guessed the wrong EC schema :(. bq. If DN9 and 10 are failing, getNumAckedStripes() will return 2, and ackedBytes will become 2 * 10 cells. (The number 10 comes from 10 + 4 schema) The block layouts as follows. So in this case, the returned acked stripes {{2}} is correct, but then the acked bytes calculated as 2 * 10 was wrong, because it didn't reflect the fact that in the 1st stripe actually there were 2 data cells located in D9~10 failed. Am I correct? {noformat} D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D D D D D D D D D D P P P P D D D D D D D D P P P P {noformat} If above sounds good, then {{getNumAckedStripes}} should be replaced with something like {{getNumAckedCells}} because the latter will be more accurate. I haven't looked into the provided patches here yet and just wanted to understand the question first: **why acked bytes size is greater than the bytes sent**, like asked by [~zhz] previously. > Client fails if acknowledged size is greater than bytes sent > > > Key: HDFS-11882 > URL: https://issues.apache.org/jira/browse/HDFS-11882 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11882.01.patch, HDFS-11882.02.patch, > HDFS-11882.03.patch, HDFS-11882.regressiontest.patch > > > Some tests of erasure coding fails by the following exception. The following > test was removed by HDFS-11823, however, this type of error can happen in > real cluster. > {noformat} > Running > org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure > Tests run: 14, Failures: 0, Errors: 1, Skipped: 10, Time elapsed: 89.086 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure > testMultipleDatanodeFailure56(org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure) > Time elapsed: 38.831 sec <<< ERROR! > java.lang.IllegalStateException: null > at > com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at > org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780) > at > org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:664) > at > org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1034) > at > org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at > org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:472) > at > org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTestWithMultipleFailure(TestDFSStripedOutputStreamWithFailure.java:381) > at > org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.testMultipleDatanodeFailure56(TestDFSStripedOutputStreamWithFailure.java:245) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12289) HDFS-12091 breaks the tests for provided block reads
[ https://issues.apache.org/jira/browse/HDFS-12289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12289: -- Issue Type: Sub-task (was: Bug) Parent: HDFS-9806 > HDFS-12091 breaks the tests for provided block reads > > > Key: HDFS-12289 > URL: https://issues.apache.org/jira/browse/HDFS-12289 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Virajith Jalaparti > Attachments: HDFS-12289-HDFS-9806.001.patch > > > In the tests within {{TestNameNodeProvidedImplementation}}, the files that > are supposed to belong to a provided volume are not located under the Storage > directory assigned to the volume in {{MiniDFSCluster}}. With HDFS-12091, this > isn't correct and thus, it breaks the tests. This JIRA is to fix the tests > under {{TestNameNodeProvidedImplementation}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12222) Add EC information to BlockLocation
[ https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122607#comment-16122607 ] Kai Zheng commented on HDFS-1: -- I thought a little bit more about this, Huafeng. Could you help check if it works for you? Thanks! Even we use ECSchema info in hadoop common side codes, it's still tricky to use that info to parse for a erasure coded block locations in hadoop common side since we may need couple with HDFS internals. Could we have a hadoop common class like {{ErasureCodedBlockLocation}} which contain methods to get data/parity block locations plus cell size info and it can be passed into a new {{LocatedFileStatus}} constructor. The object of ErasureCodedBlockLocation can be constructed with parsed info in HDFS side. > Add EC information to BlockLocation > --- > > Key: HDFS-1 > URL: https://issues.apache.org/jira/browse/HDFS-1 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Huafeng Wang > Labels: hdfs-ec-3.0-nice-to-have > > HDFS applications query block location information to compute splits. One > example of this is FileInputFormat: > https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346 > You see bits of code like this that calculate offsets as follows: > {noformat} > long bytesInThisBlock = blkLocations[startIndex].getOffset() + > blkLocations[startIndex].getLength() - offset; > {noformat} > EC confuses this since the block locations include parity block locations as > well, which are not part of the logical file length. This messes up the > offset calculation and thus topology/caching information too. > Applications can figure out what's a parity block by reading the EC policy > and then parsing the schema, but it'd be a lot better if we exposed this more > generically in BlockLocation instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12289) HDFS-12091 breaks the tests for provided block reads
[ https://issues.apache.org/jira/browse/HDFS-12289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12289: -- Attachment: HDFS-12289-HDFS-9806.001.patch Posting a patch that introduces a configuration parameter in {{MiniDFSCluster}} so that the storage directory for {{PROVIDED}} storages can be configured. This ensures that storage directory that will be used by the Datanodes {{MiniDFSCluster}} will be a prefix of the paths used as locations for {{PROVIDED}} files in {{TestNamenodeProvidedImpl}} (and thus pass the check introduced by HDFS-12091). > HDFS-12091 breaks the tests for provided block reads > > > Key: HDFS-12289 > URL: https://issues.apache.org/jira/browse/HDFS-12289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Virajith Jalaparti > Attachments: HDFS-12289-HDFS-9806.001.patch > > > In the tests within {{TestNameNodeProvidedImplementation}}, the files that > are supposed to belong to a provided volume are not located under the Storage > directory assigned to the volume in {{MiniDFSCluster}}. With HDFS-12091, this > isn't correct and thus, it breaks the tests. This JIRA is to fix the tests > under {{TestNameNodeProvidedImplementation}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12289) HDFS-12091 breaks the tests for provided block reads
[ https://issues.apache.org/jira/browse/HDFS-12289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Virajith Jalaparti updated HDFS-12289: -- Status: Patch Available (was: Open) > HDFS-12091 breaks the tests for provided block reads > > > Key: HDFS-12289 > URL: https://issues.apache.org/jira/browse/HDFS-12289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Virajith Jalaparti > Attachments: HDFS-12289-HDFS-9806.001.patch > > > In the tests within {{TestNameNodeProvidedImplementation}}, the files that > are supposed to belong to a provided volume are not located under the Storage > directory assigned to the volume in {{MiniDFSCluster}}. With HDFS-12091, this > isn't correct and thus, it breaks the tests. This JIRA is to fix the tests > under {{TestNameNodeProvidedImplementation}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122593#comment-16122593 ] Hadoop QA commented on HDFS-11576: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 33m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 35s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 45s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 38s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 32s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 7s{color} | {color:orange} root: The patch generated 5 new + 783 unchanged - 0 fixed = 788 total (was 783) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 16s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 6s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}182m 8s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.conf.TestCommonConfigurationFields | | | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | | | hadoop.tools.TestHdfsConfigFields | | | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11576 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881323/HDFS-11576.009.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 2c99cb82e495 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20646/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | checkstyle |
[jira] [Commented] (HDFS-12054) FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode
[ https://issues.apache.org/jira/browse/HDFS-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122588#comment-16122588 ] lufei commented on HDFS-12054: -- [~jojochuang],thanks for your advice. According to your suggestions, I have added the statement in the HDFS-12054.003.patch and HDFS-12066.003.patch . > FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to > ensure Namenode is not in safemode > --- > > Key: HDFS-12054 > URL: https://issues.apache.org/jira/browse/HDFS-12054 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12054.001.patch, HDFS-12054.002.patch, > HDFS-12054.003.patch > > > In the process of FSNamesystem#addErasureCodingPolicies, it would be better > to call checkNameNodeSafeMode() to ensure NN is not in safemode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12066) When Namenode is in safemode,may not allowed to remove an user's erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-12066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-12066: - Attachment: HDFS-12066.003.patch > When Namenode is in safemode,may not allowed to remove an user's erasure > coding policy > -- > > Key: HDFS-12066 > URL: https://issues.apache.org/jira/browse/HDFS-12066 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12066.001.patch, HDFS-12066.002.patch, > HDFS-12066.003.patch > > > FSNamesystem#removeErasureCodingPolicy should call checkNameNodeSafeMode() to > ensure Namenode is not in safemode -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12054) FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode
[ https://issues.apache.org/jira/browse/HDFS-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-12054: - Attachment: HDFS-12054.003.patch > FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to > ensure Namenode is not in safemode > --- > > Key: HDFS-12054 > URL: https://issues.apache.org/jira/browse/HDFS-12054 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12054.001.patch, HDFS-12054.002.patch, > HDFS-12054.003.patch > > > In the process of FSNamesystem#addErasureCodingPolicies, it would be better > to call checkNameNodeSafeMode() to ensure NN is not in safemode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12054) FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to ensure Namenode is not in safemode
[ https://issues.apache.org/jira/browse/HDFS-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lufei updated HDFS-12054: - Attachment: (was: HDFS-12054.003.patch) > FSNamesystem#addErasureCodingPolicies should call checkNameNodeSafeMode() to > ensure Namenode is not in safemode > --- > > Key: HDFS-12054 > URL: https://issues.apache.org/jira/browse/HDFS-12054 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.0.0-alpha3 >Reporter: lufei >Assignee: lufei > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12054.001.patch, HDFS-12054.002.patch, > HDFS-12054.003.patch > > > In the process of FSNamesystem#addErasureCodingPolicies, it would be better > to call checkNameNodeSafeMode() to ensure NN is not in safemode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11738) Hedged pread takes more time when block moved from initial locations
[ https://issues.apache.org/jira/browse/HDFS-11738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-11738: -- Summary: Hedged pread takes more time when block moved from initial locations (was: hedged pread takes more time when block moved from initial locations) > Hedged pread takes more time when block moved from initial locations > > > Key: HDFS-11738 > URL: https://issues.apache.org/jira/browse/HDFS-11738 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS-11738-01.patch, HDFS-11738-02.patch > > > Scenario : > Same as HDFS-11708. > During Hedge read, > 1. First two locations fails to read the data in hedged mode. > 2. chooseData refetches locations and adds a future to read from DN3. > 3. after adding future to DN3, main thread goes for refetching locations in > loop and stucks there till all 3 retries to fetch locations exhausted, which > consumes ~20 seconds with exponential retry time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122564#comment-16122564 ] Hanisha Koneru commented on HDFS-12288: --- Nope. Deleting should also be fine. > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > Attachments: HDFS-12288.001.patch > > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122562#comment-16122562 ] Lukas Majercak commented on HDFS-12288: --- Sure [~hanishakoneru], do you mind if I just delete it? > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > Attachments: HDFS-12288.001.patch > > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12289) HDFS-12091 breaks the tests for provided block reads
Virajith Jalaparti created HDFS-12289: - Summary: HDFS-12091 breaks the tests for provided block reads Key: HDFS-12289 URL: https://issues.apache.org/jira/browse/HDFS-12289 Project: Hadoop HDFS Issue Type: Bug Reporter: Virajith Jalaparti In the tests within {{TestNameNodeProvidedImplementation}}, the files that are supposed to belong to a provided volume are not located under the Storage directory assigned to the volume in {{MiniDFSCluster}}. With HDFS-12091, this isn't correct and thus, it breaks the tests. This JIRA is to fix the tests under {{TestNameNodeProvidedImplementation}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11882) Client fails if acknowledged size is greater than bytes sent
[ https://issues.apache.org/jira/browse/HDFS-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11882: --- Attachment: HDFS-11882.03.patch Here's a new patch. I added a little more testing, and a lot more comments. The EC writing logic is quite complicated. I think we need a test that writes random file lengths and also randomized fault injection. Writing this kind of test would be non-trivial amounts of work, but it'd add a lot of confidence. > Client fails if acknowledged size is greater than bytes sent > > > Key: HDFS-11882 > URL: https://issues.apache.org/jira/browse/HDFS-11882 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11882.01.patch, HDFS-11882.02.patch, > HDFS-11882.03.patch, HDFS-11882.regressiontest.patch > > > Some tests of erasure coding fails by the following exception. The following > test was removed by HDFS-11823, however, this type of error can happen in > real cluster. > {noformat} > Running > org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure > Tests run: 14, Failures: 0, Errors: 1, Skipped: 10, Time elapsed: 89.086 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure > testMultipleDatanodeFailure56(org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure) > Time elapsed: 38.831 sec <<< ERROR! > java.lang.IllegalStateException: null > at > com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at > org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780) > at > org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:664) > at > org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1034) > at > org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at > org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:472) > at > org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTestWithMultipleFailure(TestDFSStripedOutputStreamWithFailure.java:381) > at > org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.testMultipleDatanodeFailure56(TestDFSStripedOutputStreamWithFailure.java:245) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122514#comment-16122514 ] Anu Engineer commented on HDFS-12159: - [~aw] Thanks for the tip, will do so in the future. > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122509#comment-16122509 ] Allen Wittenauer commented on HDFS-12159: - bq. Is there a way to upload PNGs without impacting Jenkins. Yes. Upload it first. > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122479#comment-16122479 ] Anu Engineer edited comment on HDFS-12159 at 8/10/17 11:09 PM: --- the png that I attached was confusing the Jenkins {code} HDFS-12159 patch is being downloaded at Thu Aug 10 22:46:11 UTC 2017 from https://issues.apache.org/jira/secure/attachment/12881104/createFlow.png -> Downloaded ERROR: Unsure how to process HDFS-12159. {code} removed the PNG will resubmit this patch. cc: [~aw] I had posted a PNG right after I posted the patch. This caused the Jenkins to fail since the Jenkins seems to have downloaded PNG for the patch. Is there a way to upload PNGs without impacting Jenkins. was (Author: anu): the png that I attached was confusing the Jenkins {code} HDFS-12159 patch is being downloaded at Thu Aug 10 22:46:11 UTC 2017 from https://issues.apache.org/jira/secure/attachment/12881104/createFlow.png -> Downloaded ERROR: Unsure how to process HDFS-12159. {code} removed the PNG will resubmit this patch. > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122502#comment-16122502 ] Hanisha Koneru commented on HDFS-12288: --- Thanks for the fix, [~lukmajercak]. The patch LGTM. Just one NIT: In _TestNamenodeCapacityReport#testXceiverCountInternal_, can you please update the comment below. {code} // the load for writers is 2 because both the write xceiver & packet // responder threads are counted in the load expectedTotalLoad += fileRepl; expectedInServiceLoad += fileRepl; {code} > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > Attachments: HDFS-12288.001.patch > > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122487#comment-16122487 ] Lukas Majercak commented on HDFS-12288: --- Findbugs/unit test failures are unrelated to the change. > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > Attachments: HDFS-12288.001.patch > > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12221) Replace xcerces in XmlEditsVisitor
[ https://issues.apache.org/jira/browse/HDFS-12221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122486#comment-16122486 ] Lei (Eddy) Xu commented on HDFS-12221: -- Hi, [~ajayydv] Thanks for working this. It LGTM. One nit: * Is {{handler.getTransformer().setOutputProperty(OutputKeys.STANDALONE, "yes");}} necessary? +1 pending. > Replace xcerces in XmlEditsVisitor > --- > > Key: HDFS-12221 > URL: https://issues.apache.org/jira/browse/HDFS-12221 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha4 >Reporter: Lei (Eddy) Xu >Assignee: Ajay Yadav > Attachments: editsStored, fsimage_hdfs-12221.xml, > HDFS-12221.01.patch, HDFS-12221.02.patch, HDFS-12221.03.patch, > HDFS-12221.04.patch, HDFS-12221.05.patch > > > XmlEditsVisitor should use new XML capability in the newer JDK, to make JAR > shading easier (HADOOP-14672) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122480#comment-16122480 ] Anu Engineer commented on HDFS-12159: - just submitted a new job https://builds.apache.org/blue/organizations/jenkins/PreCommit-HDFS-Build/detail/PreCommit-HDFS-Build/20649/pipeline > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122479#comment-16122479 ] Anu Engineer commented on HDFS-12159: - the png that I attached was confusing the Jenkins {code} HDFS-12159 patch is being downloaded at Thu Aug 10 22:46:11 UTC 2017 from https://issues.apache.org/jira/secure/attachment/12881104/createFlow.png -> Downloaded ERROR: Unsure how to process HDFS-12159. {code} removed the PNG will resubmit this patch. > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12159: Attachment: (was: createFlow.png) > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12159) Ozone: SCM: Add create replication pipeline RPC
[ https://issues.apache.org/jira/browse/HDFS-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122453#comment-16122453 ] Anu Engineer commented on HDFS-12159: - started a precommt job by hand. https://builds.apache.org/blue/organizations/jenkins/PreCommit-HDFS-Build/detail/PreCommit-HDFS-Build/20648/pipeline > Ozone: SCM: Add create replication pipeline RPC > --- > > Key: HDFS-12159 > URL: https://issues.apache.org/jira/browse/HDFS-12159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: createFlow.png, HDFS-12159-HDFS-7240.001.patch, > HDFS-12159-HDFS-7240.002.patch, HDFS-12159-HDFS-7240.003.patch > > > Add an API that allows users to create replication pipelines using SCM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122451#comment-16122451 ] Hadoop QA commented on HDFS-12288: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 5s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 46s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}112m 10s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.web.TestWebHdfsFileSystemContract | | | hadoop.hdfs.server.datanode.TestDirectoryScanner | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12288 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881316/HDFS-12288.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux d81228349eea 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20645/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20645/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20645/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20645/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 >
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122449#comment-16122449 ] Anu Engineer commented on HDFS-12196: - [~cheersyang] , Thanks for updating the patch. It looks very good. I am almost a +1 for this change. I had some minor questions/comments. h6. One Design question Should we allow each background task to create its own thread pool or just have a common thread pool? In that case, we will need to have a single service called BackgroundService -- and all other jobs like "ContainerRecyclingService" will become a task, say "ContainerRecyclingTask". In other words, I am saying that we can just queue "ContainerRecyclingTask" directly to a background service without having individual job execution pools. Just one single common pool for all background jobs. Just wondering if that is an interface we want to offer. I do see the down side, we can have a task monopolize the thread pool. If you are free ping me in slack or we can have an offline chat about this. I am also open to checking in this architecture and may be refining it later. h6. some code comments * {{BackgroundService.java}} bq. threadFactory = r -> new Thread(threadGroup, r); Generally we expect the background service to be a deamon thread. Would you like to replace this with something like {code} new ThreadFactoryBuilder().setDaemon(true) .setNameFormat( threadName + "#%d") .build()); {code} Where threadName is an argument to the ctor ? * {{BackgroundService.java}} Question : Why do we need the testing flag and testing Thread? * {{BackgroundTaskQueue.java}} Is not thread safe, is it by design? Just wondering if it is possible for 2 different threads to call into this concurrently. From a quick code reading, I am not able to see it, may we can just add a comment to the class. * {{ContainerRecyclingService}} Should we create a new package called *background* tasks under {{org.apache.hadoop.ozone.container.common.statemachine}} I am presuming we will need much more tasks like this in future. Ps. Sorry for the delay in code review, I was focused on the pluggable pipeline patch. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch, > HDFS-12196-HDFS-7240.002.patch, HDFS-12196-HDFS-7240.003.patch, > HDFS-12196-HDFS-7240.004.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11303) Hedged read might hang infinitely if read data from all DN failed
[ https://issues.apache.org/jira/browse/HDFS-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HDFS-11303: -- Attachment: HDFS-11303.004.patch Patch 004 * Fix checkstyle * Rename futureComplete back to future to avoid gratuitous changes * A few wordings in comments > Hedged read might hang infinitely if read data from all DN failed > -- > > Key: HDFS-11303 > URL: https://issues.apache.org/jira/browse/HDFS-11303 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.0.0-alpha1 >Reporter: Chen Zhang >Assignee: Chen Zhang > Attachments: HDFS-11303-001.patch, HDFS-11303-001.patch, > HDFS-11303-002.patch, HDFS-11303-002.patch, HDFS-11303.003.patch, > HDFS-11303.004.patch > > > Hedged read will read from a DN first, if timeout, then read other DNs > simultaneously. > If read all DN failed, this bug will cause the future-list not empty(the > first timeout request left in list), and hang in the loop infinitely -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122418#comment-16122418 ] Hadoop QA commented on HDFS-12281: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 44s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 53s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 49s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 48s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 24s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}160m 45s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.scm.TestArchive | | | hadoop.cblock.TestBufferManager | | | hadoop.ozone.web.client.TestKeys | | | hadoop.cblock.TestCBlockReadWrite | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | | | hadoop.ozone.container.ozoneimpl.TestRatisManager | | Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis | | | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis | | | org.apache.hadoop.cblock.TestLocalBlockCache | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12281 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881305/HDFS-12281-HDFS-7240.02.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux f124f0e46822 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven
[jira] [Commented] (HDFS-11303) Hedged read might hang infinitely if read data from all DN failed
[ https://issues.apache.org/jira/browse/HDFS-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122411#comment-16122411 ] Hadoop QA commented on HDFS-11303: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 17s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 4s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 18s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 47s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new + 58 unchanged - 0 fixed = 59 total (was 58) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 29s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}138m 26s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs | | | hadoop.hdfs.TestPread | | | hadoop.hdfs.server.namenode.TestNameNodeMXBean | | Timed out junit tests | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11303 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881307/HDFS-11303.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 9924b9e97719 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
[jira] [Comment Edited] (HDFS-11882) Client fails if acknowledged size is greater than bytes sent
[ https://issues.apache.org/jira/browse/HDFS-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122397#comment-16122397 ] Andrew Wang edited comment on HDFS-11882 at 8/10/17 9:48 PM: - I spent some time digging into this, and I think I understand it better. The last stripe can be a partial stripe. If the partial stripe happens to have enough data cells, it counts as an acked stripe (i.e., {{numDataBlock}} streamers at that length). Then it multiplies by the # bytes in a stripe, which can round up the numAckedBytes above the sentBytes. This partial stripe issue only applies to close. IIUC, we pad out the last data cell, and write all the parity cells. Empty cells are assumed to be zero, and count toward the minimum durability threshold of {{numDataBlock}} streamers. Besides close, we're always writing full stripes. To be more concrete, imagine we are doing RS(6,3), and the last stripe looks like this: {noformat} x = full cell |d1|d2|d3|d4|d5|d6|p1|p2|p3| |x |x |x | | | |x |x |x | {noformat} For this partial stripe, 6 cells have data, which satisfies the {{numDataBlocks}} threshold. {noformat} |d1|d2|d3|d4|d5|d6|p1|p2|p3| |x | | | | | |x |x |x | {noformat} For this partial stripe, 4 cells have data, which fails the {{numDataBlocks}} threshold. Also, because there are supposed to be 5 empty cells, we only need one written cell to satisfy the durability requirement. As an example, for a data length of one cell, any of these would be fine: {noformat} |d1|d2|d3|d4|d5|d6|p1|p2|p3| |x | | | | | | | | | | | | | | | | |x | | {noformat} Because this last stripe needs to be handled specially on close, I don't think the current proposed patch fully addresses the issue. We also should try to address this related TODO: {noformat} // TODO we can also succeed if all the failed streamers have not taken // the updated block {noformat} I'm working on a patch to rework this code, but it's pretty complex, and I wanted to post my thinking here first. was (Author: andrew.wang): I spent some time digging into this, and I think I understand it better. The last stripe can be a partial stripe. If the partial stripe happens to have enough data cells, it counts as an acked stripe (i.e., {{numDataBlock}} streamers at that length). Then it multiplies by the # bytes in a stripe, which can round up the numAckedBytes above the sentBytes. This partial stripe issue only applies to close. IIUC, we pad out the last data cell, and write all the parity cells. Empty cells are assumed to be zero, and count toward the minimum durability threshold of {{numDataBlock}} streamers. Besides close, we're always writing full stripes. To be more concrete, imagine we are doing RS(6,3), and the last stripe looks like this: {noformat} x = full cell |d1|d2|d3|d4|d5|d6|p1|p2|p3| |x |x |x | | | |x |x |x | {noformat} For this partial stripe, 6 cells have data, which satisfies the {{numDataBlocks}} threshold. {noformat} |d1|d2|d3|d4|d5|d6|p1|p2|p3| |x | | | | | |x |x |x | {noformat} For this partial stripe, 4 cells have data, which fails the {{numDataBlocks}} threshold. Also, because there are supposed to be 5 empty cells, we only need one written cell to satisfy the durability requirement. As an example, for a data length of one cell, any of these would be fine: {noformat} |d1|d2|d3|d4|d5|d6|p1|p2|p3| |x | | | | | | | | | | | | | | | | |x | | {noformat} Because this last stripe needs to be handled specially on close, I don't think the current proposed patch fully addresses the issue. We also should try to address this related TODO: {noformat} // TODO we can also succeed if all the failed streamers have not taken // the updated block {noformat} I'm working on a patch to rework this code, but it's pretty complex, and I wanted to post my thinking here first. > Client fails if acknowledged size is greater than bytes sent > > > Key: HDFS-11882 > URL: https://issues.apache.org/jira/browse/HDFS-11882 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11882.01.patch, HDFS-11882.02.patch, > HDFS-11882.regressiontest.patch > > > Some tests of erasure coding fails by the following exception. The following > test was removed by HDFS-11823, however, this type of error can happen in > real cluster. > {noformat} > Running > org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure > Tests run: 14, Failures: 0, Errors: 1, Skipped: 10, Time elapsed: 89.086 sec > <<< FAILURE! - in >
[jira] [Commented] (HDFS-11882) Client fails if acknowledged size is greater than bytes sent
[ https://issues.apache.org/jira/browse/HDFS-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122397#comment-16122397 ] Andrew Wang commented on HDFS-11882: I spent some time digging into this, and I think I understand it better. The last stripe can be a partial stripe. If the partial stripe happens to have enough data cells, it counts as an acked stripe (i.e., {{numDataBlock}} streamers at that length). Then it multiplies by the # bytes in a stripe, which can round up the numAckedBytes above the sentBytes. This partial stripe issue only applies to close. IIUC, we pad out the last data cell, and write all the parity cells. Empty cells are assumed to be zero, and count toward the minimum durability threshold of {{numDataBlock}} streamers. Besides close, we're always writing full stripes. To be more concrete, imagine we are doing RS(6,3), and the last stripe looks like this: {noformat} x = full cell |d1|d2|d3|d4|d5|d6|p1|p2|p3| |x |x |x | | | |x |x |x | {noformat} For this partial stripe, 6 cells have data, which satisfies the {{numDataBlocks}} threshold. {noformat} |d1|d2|d3|d4|d5|d6|p1|p2|p3| |x | | | | | |x |x |x | {noformat} For this partial stripe, 4 cells have data, which fails the {{numDataBlocks}} threshold. Also, because there are supposed to be 5 empty cells, we only need one written cell to satisfy the durability requirement. As an example, for a data length of one cell, any of these would be fine: {noformat} |d1|d2|d3|d4|d5|d6|p1|p2|p3| |x | | | | | | | | | | | | | | | | |x | | {noformat} Because this last stripe needs to be handled specially on close, I don't think the current proposed patch fully addresses the issue. We also should try to address this related TODO: {noformat} // TODO we can also succeed if all the failed streamers have not taken // the updated block {noformat} I'm working on a patch to rework this code, but it's pretty complex, and I wanted to post my thinking here first. > Client fails if acknowledged size is greater than bytes sent > > > Key: HDFS-11882 > URL: https://issues.apache.org/jira/browse/HDFS-11882 > Project: Hadoop HDFS > Issue Type: Bug > Components: erasure-coding, test >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-11882.01.patch, HDFS-11882.02.patch, > HDFS-11882.regressiontest.patch > > > Some tests of erasure coding fails by the following exception. The following > test was removed by HDFS-11823, however, this type of error can happen in > real cluster. > {noformat} > Running > org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure > Tests run: 14, Failures: 0, Errors: 1, Skipped: 10, Time elapsed: 89.086 sec > <<< FAILURE! - in > org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure > testMultipleDatanodeFailure56(org.apache.hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure) > Time elapsed: 38.831 sec <<< ERROR! > java.lang.IllegalStateException: null > at > com.google.common.base.Preconditions.checkState(Preconditions.java:129) > at > org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780) > at > org.apache.hadoop.hdfs.DFSStripedOutputStream.checkStreamerFailures(DFSStripedOutputStream.java:664) > at > org.apache.hadoop.hdfs.DFSStripedOutputStream.closeImpl(DFSStripedOutputStream.java:1034) > at > org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:842) > at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72) > at > org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101) > at > org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:472) > at > org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTestWithMultipleFailure(TestDFSStripedOutputStreamWithFailure.java:381) > at > org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.testMultipleDatanodeFailure56(TestDFSStripedOutputStreamWithFailure.java:245) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at >
[jira] [Commented] (HDFS-12287) Remove a no-longer applicable TODO comment in DatanodeManager
[ https://issues.apache.org/jira/browse/HDFS-12287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122380#comment-16122380 ] Hadoop QA commented on HDFS-12287: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 0s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 57s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 95m 32s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | | | hadoop.hdfs.server.namenode.TestDecommissioningStatus | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | | | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.server.namenode.ha.TestPendingCorruptDnMessages | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12287 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881311/HDFS-12287.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux e9cc19f6e92d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20644/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20644/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20644/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output |
[jira] [Commented] (HDFS-12255) Block Storage: Cblock should generated unique trace ID for the ops
[ https://issues.apache.org/jira/browse/HDFS-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122364#comment-16122364 ] Anu Engineer commented on HDFS-12255: - [~msingh] +1, from me, I will commit after [~vagarychen]'s comments are addressed. bq. Also, log a warning when UnknownHostException ex happens? if we decide to log this warning, can we make sure we warn only once? or max a few times. Otherwise, for a client where this lookup fails, the log file will be overrun with this warning. So while it might be a good idea to warn, we might want to restrict the number of times we warn. We use a similar pattern on data-node side, if we are not able to communicate to SCM, we don't warn each try, but only at a selected frequency, we log how many times this call has failed. Another option is to put this as a trace message so that it does not get to the log unless we are debugging. > Block Storage: Cblock should generated unique trace ID for the ops > -- > > Key: HDFS-12255 > URL: https://issues.apache.org/jira/browse/HDFS-12255 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12255-HDFS-7240.001.patch, > HDFS-12255-HDFS-7240.002.patch > > > Cblock tests fails because cblock does not generate unique trace id for each > op. > {code} > java.lang.AssertionError: expected:<0> but was:<1051> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.cblock.TestBufferManager.testRepeatedBlockWrites(TestBufferManager.java:448) > {code} > This failure is because of following error. > {code} > 017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > scm.XceiverClientHandler (XceiverClientHandler.java:sendCommandAsync(134)) - > Command with Trace already exists. Ignoring this command. . Previous Command: > java.util.concurrent.CompletableFuture@7847fc2d[Not completed, 1 dependents] > 2017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > jscsiHelper.ContainerCacheFlusher (BlockWriterTask.java:run(108)) - Writing > of block:44 failed, We have attempted to write this block 7 tim > es to the container container2483304118.Trace ID: > java.lang.IllegalStateException: Duplicate trace ID. Command with this trace > ID is already executing. Please ensure that trace IDs are not reused. ID: > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommandAsync(XceiverClientHandler.java:139) > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommand(XceiverClientHandler.java:114) > at > org.apache.hadoop.scm.XceiverClient.sendCommand(XceiverClient.java:132) > at > org.apache.hadoop.scm.storage.ContainerProtocolCalls.writeSmallFile(ContainerProtocolCalls.java:225) > at > org.apache.hadoop.cblock.jscsiHelper.BlockWriterTask.run(BlockWriterTask.java:97) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12238) Ozone: Add valid trace ID check in sendCommandAsync
[ https://issues.apache.org/jira/browse/HDFS-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122343#comment-16122343 ] Anu Engineer commented on HDFS-12238: - [~ajayydv] Thanks for the contribution, Some minor comments. * There is a bunch of check-style warnings which are mostly more than "80 chars in a line" warning. * I am not sure if the changes in {{Ozonebucket.java}} and {{TestKeys.java}} are part of this change. * Also in the test {{TestOzoneContainer#testInvalidRequest}}, can we be more specific instead of just catching IllegalArgumentException. -- something like catching an exception and verifying that it is indeed what you expect. May be something like {{GenericTestUtils.assertExceptionContains}} or JUnit specific checkers can be used. > Ozone: Add valid trace ID check in sendCommandAsync > --- > > Key: HDFS-12238 > URL: https://issues.apache.org/jira/browse/HDFS-12238 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Ajay Yadav > Labels: newbie > Attachments: HDFS-12238-HDFS-7240.01.patch > > > In the function {{XceiverClientHandler#sendCommandAsync}} we should add a > check > {code} > if(StringUtils.isEmpty(request.getTraceID())) { > throw new IllegalArgumentException("Invalid trace ID"); > } > {code} > To ensure that ozone clients always send a valid trace ID. However, when you > do that a set of current tests that do add a valid trace ID will fail. So we > need to fix these tests too. > {code} > TestContainerMetrics.testContainerMetrics > TestOzoneContainer.testBothGetandPutSmallFile > TestOzoneContainer.testCloseContainer > TestOzoneContainer.testOzoneContainerViaDataNode > TestOzoneContainer.testXcieverClientAsync > TestOzoneContainer.testCreateOzoneContainer > TestOzoneContainer.testDeleteContainer > TestContainerServer.testClientServer > TestContainerServer.testClientServerWithContainerDispatcher > TestKeys.testPutAndGetKeyWithDnRestart > {code} > This is based on a comment from [~vagarychen] in HDFS-11580. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-11576: -- Attachment: HDFS-11576.009.patch > Block recovery will fail indefinitely if recovery time > heartbeat interval > --- > > Key: HDFS-11576 > URL: https://issues.apache.org/jira/browse/HDFS-11576 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs, namenode >Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Critical > Attachments: HDFS-11576.001.patch, HDFS-11576.002.patch, > HDFS-11576.003.patch, HDFS-11576.004.patch, HDFS-11576.005.patch, > HDFS-11576.006.patch, HDFS-11576.007.patch, HDFS-11576.008.patch, > HDFS-11576.009.patch, HDFS-11576.repro.patch > > > Block recovery will fail indefinitely if the time to recover a block is > always longer than the heartbeat interval. Scenario: > 1. DN sends heartbeat > 2. NN sends a recovery command to DN, recoveryID=X > 3. DN starts recovery > 4. DN sends another heartbeat > 5. NN sends a recovery command to DN, recoveryID=X+1 > 6. DN calls commitBlockSyncronization after succeeding with first recovery to > NN, which fails because X < X+1 > ... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122306#comment-16122306 ] Lukas Majercak commented on HDFS-11576: --- Patch 008 was missing some changes, updating new one in a sec. > Block recovery will fail indefinitely if recovery time > heartbeat interval > --- > > Key: HDFS-11576 > URL: https://issues.apache.org/jira/browse/HDFS-11576 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs, namenode >Affects Versions: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha1, 3.0.0-alpha2 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Critical > Attachments: HDFS-11576.001.patch, HDFS-11576.002.patch, > HDFS-11576.003.patch, HDFS-11576.004.patch, HDFS-11576.005.patch, > HDFS-11576.006.patch, HDFS-11576.007.patch, HDFS-11576.008.patch, > HDFS-11576.repro.patch > > > Block recovery will fail indefinitely if the time to recover a block is > always longer than the heartbeat interval. Scenario: > 1. DN sends heartbeat > 2. NN sends a recovery command to DN, recoveryID=X > 3. DN starts recovery > 4. DN sends another heartbeat > 5. NN sends a recovery command to DN, recoveryID=X+1 > 6. DN calls commitBlockSyncronization after succeeding with first recovery to > NN, which fails because X < X+1 > ... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12117) HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST Interface
[ https://issues.apache.org/jira/browse/HDFS-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122298#comment-16122298 ] Wellington Chevreuil commented on HDFS-12117: - Thanks [~jojochuang]! It seems my commit is depending on commit below, which has not been applied yet on branch-2: ||id||date||author||message|| |12c8fdceaf263425661169cba25402df89d444c1|2017-07-11 19:19| John Zhuge | HDFS-12052. Set SWEBHDFS delegation token kind when ssl is enabled in HttpFS. Contributed by Zoran Dimitrijevic.| Should I cherry pick this one, together with mine into branch-2? > HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST > Interface > --- > > Key: HDFS-12117 > URL: https://issues.apache.org/jira/browse/HDFS-12117 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 3.0.0-alpha3 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil > Fix For: 3.0.0-beta1 > > Attachments: HDFS-12117.003.patch, HDFS-12117.004.patch, > HDFS-12117.005.patch, HDFS-12117.006.patch, HDFS-12117.patch.01, > HDFS-12117.patch.02 > > > Currently, HttpFS is lacking implementation for SNAPSHOT related methods from > WebHDFS REST interface as defined by WebHDFS documentation [WebHDFS > documentation|https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Snapshot_Operations] > I would like to work on this implementation, following the existing design > approach already implemented by other WebHDFS methods on current HttpFS > project, so I'll be proposing an initial patch soon for reviews. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122281#comment-16122281 ] Anu Engineer edited comment on HDFS-7240 at 8/10/17 8:37 PM: - [~johnament] Would you please care to comment on the ASF slack usage for people without Apache email ID? was (Author: anu): [~johnament] Would please care to comment on the ASF slack usage for people without Apache email ID? > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, > ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7240) Object store in HDFS
[ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122281#comment-16122281 ] Anu Engineer commented on HDFS-7240: [~johnament] Would please care to comment on the ASF slack usage for people without Apache email ID? > Object store in HDFS > > > Key: HDFS-7240 > URL: https://issues.apache.org/jira/browse/HDFS-7240 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, > ozone_user_v0.pdf > > > This jira proposes to add object store capabilities into HDFS. > As part of the federation work (HDFS-1052) we separated block storage as a > generic storage layer. Using the Block Pool abstraction, new kinds of > namespaces can be built on top of the storage layer i.e. datanodes. > In this jira I will explore building an object store using the datanode > storage, but independent of namespace metadata. > I will soon update with a detailed design document. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-12288: -- Status: Patch Available (was: In Progress) > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > Attachments: HDFS-12288.001.patch > > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-12288: -- Attachment: HDFS-12288.001.patch > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > Attachments: HDFS-12288.001.patch > > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12255) Block Storage: Cblock should generated unique trace ID for the ops
[ https://issues.apache.org/jira/browse/HDFS-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122260#comment-16122260 ] Hadoop QA commented on HDFS-12255: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 32s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 33s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 43s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 33s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 8 unchanged - 0 fixed = 12 total (was 8) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 90m 54s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 | | | hadoop.ozone.web.client.TestKeys | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration | | Timed out junit tests | org.apache.hadoop.hdfs.TestFileChecksum | | | org.apache.hadoop.ozone.web.client.TestKeysRatis | | | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 | | | org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | | | org.apache.hadoop.hdfs.TestDFSFinalize | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12255 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881297/HDFS-12255-HDFS-7240.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux ea3a9224a802 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-7240 / 0e32bf1 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle |
[jira] [Commented] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122254#comment-16122254 ] Hadoop QA commented on HDFS-11576: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 27s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 19s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 59s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 31s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m 22s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 22s{color} | {color:red} root in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 29s{color} | {color:orange} root: The patch generated 3 new + 784 unchanged - 0 fixed = 787 total (was 784) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 35s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 35s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 48s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 1 new + 9 unchanged - 0 fixed = 10 total (was 9) {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 24s{color} | {color:red} hadoop-common in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 28s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 73m 58s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.security.TestRaceWhenRelogin | | | hadoop.conf.TestCommonConfigurationFields | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11576 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879929/HDFS-11576.008.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 8428a060d4f3 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20640/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | mvninstall | https://builds.apache.org/job/PreCommit-HDFS-Build/20640/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
[jira] [Commented] (HDFS-5040) Audit log for admin commands/ logging output of all DFS admin commands
[ https://issues.apache.org/jira/browse/HDFS-5040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122231#comment-16122231 ] Hadoop QA commented on HDFS-5040: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 9 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 277 unchanged - 8 fixed = 277 total (was 285) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 18s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}104m 37s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.TestFileTruncate | | | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-5040 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881285/HDFS-5040.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 3ff42984b63c 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20638/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20638/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20638/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20638/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Audit log for admin commands/ logging output of
[jira] [Commented] (HDFS-11900) Hedged reads thread pool creation not synchronized
[ https://issues.apache.org/jira/browse/HDFS-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122217#comment-16122217 ] Hadoop QA commented on HDFS-11900: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 15s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs-client: The patch generated 0 new + 49 unchanged - 1 fixed = 49 total (was 50) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 13s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 21m 37s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11900 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12870502/HDFS-11900.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 5a2140d60c18 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20642/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client-warnings.html | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20642/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-client U: hadoop-hdfs-project/hadoop-hdfs-client | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20642/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Hedged reads thread pool creation not synchronized > -- > > Key: HDFS-11900 > URL: https://issues.apache.org/jira/browse/HDFS-11900 >
[jira] [Work started] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-12288 started by Lukas Majercak. - > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-12288: -- Summary: Fix DataNode's xceiver count calculation (was: Fix DN xceiver count calculation) > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12288) Fix DataNode's xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122203#comment-16122203 ] Lukas Majercak commented on HDFS-12288: --- I'll send a patch soon. > Fix DataNode's xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12288) Fix DN xceiver count calculation
[ https://issues.apache.org/jira/browse/HDFS-12288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-12288: -- Summary: Fix DN xceiver count calculation (was: Change DN xceiver count calculation) > Fix DN xceiver count calculation > > > Key: HDFS-12288 > URL: https://issues.apache.org/jira/browse/HDFS-12288 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs >Reporter: Lukas Majercak >Assignee: Lukas Majercak > > The problem with the ThreadGroup.activeCount() method is that the method is > only a very rough estimate, and in reality returns the total number of > threads in the thread group as opposed to the threads actually running. > In some DNs, we saw this to return 50~ for a long time, even though the > actual number of DataXceiver threads was next to none. > This is a big issue as we use the xceiverCount to make decisions on the NN > for choosing replication source DN or returning DNs to clients for R/W. > The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value > which only accounts for actual number of DataXcevier threads currently > running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12255) Block Storage: Cblock should generated unique trace ID for the ops
[ https://issues.apache.org/jira/browse/HDFS-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122198#comment-16122198 ] Hadoop QA commented on HDFS-12255: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 46s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 57s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 8 unchanged - 0 fixed = 12 total (was 8) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}107m 39s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 | | | hadoop.ozone.web.client.TestKeys | | | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery | | Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis | | | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12255 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881283/HDFS-12255-HDFS-7240.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 4511f87887e1 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-7240 / 0e32bf1 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20636/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results |
[jira] [Created] (HDFS-12288) Change DN xceiver count calculation
Lukas Majercak created HDFS-12288: - Summary: Change DN xceiver count calculation Key: HDFS-12288 URL: https://issues.apache.org/jira/browse/HDFS-12288 Project: Hadoop HDFS Issue Type: Bug Components: datanode, hdfs Reporter: Lukas Majercak Assignee: Lukas Majercak The problem with the ThreadGroup.activeCount() method is that the method is only a very rough estimate, and in reality returns the total number of threads in the thread group as opposed to the threads actually running. In some DNs, we saw this to return 50~ for a long time, even though the actual number of DataXceiver threads was next to none. This is a big issue as we use the xceiverCount to make decisions on the NN for choosing replication source DN or returning DNs to clients for R/W. The plan is to reuse the DataNodeMetrics.dataNodeActiveXceiversCount value which only accounts for actual number of DataXcevier threads currently running and thus represents the load on the DN much better. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12287) Remove a no-longer applicable TODO comment in DatanodeManager
[ https://issues.apache.org/jira/browse/HDFS-12287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12287: -- Attachment: HDFS-12287.001.patch > Remove a no-longer applicable TODO comment in DatanodeManager > - > > Key: HDFS-12287 > URL: https://issues.apache.org/jira/browse/HDFS-12287 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Trivial > Attachments: HDFS-12287.001.patch > > > {{DatanodeManager}} has this this TODO comment > {code} > // TODO: Enables DFSNetworkTopology by default after more stress > // testings/validations. > {code} > This has been resolved in HDFS-11998, but it missed removing this comment. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12287) Remove a no-longer applicable TODO comment in DatanodeManager
[ https://issues.apache.org/jira/browse/HDFS-12287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12287: -- Status: Patch Available (was: Open) > Remove a no-longer applicable TODO comment in DatanodeManager > - > > Key: HDFS-12287 > URL: https://issues.apache.org/jira/browse/HDFS-12287 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Trivial > Attachments: HDFS-12287.001.patch > > > {{DatanodeManager}} has this this TODO comment > {code} > // TODO: Enables DFSNetworkTopology by default after more stress > // testings/validations. > {code} > This has been resolved in HDFS-11998, but it missed removing this comment. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12255) Block Storage: Cblock should generated unique trace ID for the ops
[ https://issues.apache.org/jira/browse/HDFS-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122176#comment-16122176 ] Chen Liang edited comment on HDFS-12255 at 8/10/17 7:35 PM: Thanks [~msingh] for taking care of this! {{TestBufferManager}} did pass in my local run after applying the patch. One comment though, can we change this function {{getHostIP}} to {{getTraceIDPrefix}} or something? Because it does not always an ip address, can be UUID instead. Also, log a warning when {{UnknownHostException ex}} happens? was (Author: vagarychen): Thanks [~msingh] for taking care of this! {{TestBufferManager}} did pass in my local run. One comment though, can we change this function {{getHostIP}} to {{getTraceIDPrefix}} or something? Because it does not always an ip address, can be UUID instead. Also, log a warning when {{UnknownHostException ex}} happens? > Block Storage: Cblock should generated unique trace ID for the ops > -- > > Key: HDFS-12255 > URL: https://issues.apache.org/jira/browse/HDFS-12255 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12255-HDFS-7240.001.patch, > HDFS-12255-HDFS-7240.002.patch > > > Cblock tests fails because cblock does not generate unique trace id for each > op. > {code} > java.lang.AssertionError: expected:<0> but was:<1051> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.cblock.TestBufferManager.testRepeatedBlockWrites(TestBufferManager.java:448) > {code} > This failure is because of following error. > {code} > 017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > scm.XceiverClientHandler (XceiverClientHandler.java:sendCommandAsync(134)) - > Command with Trace already exists. Ignoring this command. . Previous Command: > java.util.concurrent.CompletableFuture@7847fc2d[Not completed, 1 dependents] > 2017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > jscsiHelper.ContainerCacheFlusher (BlockWriterTask.java:run(108)) - Writing > of block:44 failed, We have attempted to write this block 7 tim > es to the container container2483304118.Trace ID: > java.lang.IllegalStateException: Duplicate trace ID. Command with this trace > ID is already executing. Please ensure that trace IDs are not reused. ID: > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommandAsync(XceiverClientHandler.java:139) > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommand(XceiverClientHandler.java:114) > at > org.apache.hadoop.scm.XceiverClient.sendCommand(XceiverClient.java:132) > at > org.apache.hadoop.scm.storage.ContainerProtocolCalls.writeSmallFile(ContainerProtocolCalls.java:225) > at > org.apache.hadoop.cblock.jscsiHelper.BlockWriterTask.run(BlockWriterTask.java:97) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12255) Block Storage: Cblock should generated unique trace ID for the ops
[ https://issues.apache.org/jira/browse/HDFS-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122176#comment-16122176 ] Chen Liang commented on HDFS-12255: --- Thanks [~msingh] for taking care of this! {{TestBufferManager}} did pass in my local run. One comment though, can we change this function {{getHostIP}} to {{getTraceIDPrefix}} or something? Because it does not always an ip address, can be UUID instead. Also, log a warning when {{UnknownHostException ex}} happens? > Block Storage: Cblock should generated unique trace ID for the ops > -- > > Key: HDFS-12255 > URL: https://issues.apache.org/jira/browse/HDFS-12255 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12255-HDFS-7240.001.patch, > HDFS-12255-HDFS-7240.002.patch > > > Cblock tests fails because cblock does not generate unique trace id for each > op. > {code} > java.lang.AssertionError: expected:<0> but was:<1051> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.cblock.TestBufferManager.testRepeatedBlockWrites(TestBufferManager.java:448) > {code} > This failure is because of following error. > {code} > 017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > scm.XceiverClientHandler (XceiverClientHandler.java:sendCommandAsync(134)) - > Command with Trace already exists. Ignoring this command. . Previous Command: > java.util.concurrent.CompletableFuture@7847fc2d[Not completed, 1 dependents] > 2017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > jscsiHelper.ContainerCacheFlusher (BlockWriterTask.java:run(108)) - Writing > of block:44 failed, We have attempted to write this block 7 tim > es to the container container2483304118.Trace ID: > java.lang.IllegalStateException: Duplicate trace ID. Command with this trace > ID is already executing. Please ensure that trace IDs are not reused. ID: > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommandAsync(XceiverClientHandler.java:139) > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommand(XceiverClientHandler.java:114) > at > org.apache.hadoop.scm.XceiverClient.sendCommand(XceiverClient.java:132) > at > org.apache.hadoop.scm.storage.ContainerProtocolCalls.writeSmallFile(ContainerProtocolCalls.java:225) > at > org.apache.hadoop.cblock.jscsiHelper.BlockWriterTask.run(BlockWriterTask.java:97) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11303) Hedged read might hang infinitely if read data from all DN failed
[ https://issues.apache.org/jira/browse/HDFS-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122161#comment-16122161 ] Wei-Chiu Chuang commented on HDFS-11303: Hi Chen Zhang. Your 002 patch is mostly good. I updated your latest patch to address some comments I had. Hope you don't mind. In addition, * Added a fail() to ensure DFSInputStream#read() fails as expected. * Because DFSInputStream#read() throws exception, the asserts are not reachable. Moved them to catch block. * Replaced the deprecated method {{IOUtils.cleanup}} with {{IOUtils.cleanupWithLogger}}. Regarding my first comment, I verified the test fails without the fix, and passes with the fix. > Hedged read might hang infinitely if read data from all DN failed > -- > > Key: HDFS-11303 > URL: https://issues.apache.org/jira/browse/HDFS-11303 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.0.0-alpha1 >Reporter: Chen Zhang >Assignee: Chen Zhang > Attachments: HDFS-11303-001.patch, HDFS-11303-001.patch, > HDFS-11303-002.patch, HDFS-11303-002.patch, HDFS-11303.003.patch > > > Hedged read will read from a DN first, if timeout, then read other DNs > simultaneously. > If read all DN failed, this bug will cause the future-list not empty(the > first timeout request left in list), and hang in the loop infinitely -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11900) Hedged reads thread pool creation not synchronized
[ https://issues.apache.org/jira/browse/HDFS-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122172#comment-16122172 ] Wei-Chiu Chuang commented on HDFS-11900: Thanks for the patch John, I kicked off a precommit build for you. > Hedged reads thread pool creation not synchronized > -- > > Key: HDFS-11900 > URL: https://issues.apache.org/jira/browse/HDFS-11900 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 2.8.0 >Reporter: John Zhuge >Assignee: John Zhuge > Attachments: HDFS-11900.001.patch > > > *Non-static* synchronized method initThreadsNumForHedgedReads can't > synchronize the access to the *static* class variable HEDGED_READ_THREAD_POOL. > {code} > private static ThreadPoolExecutor HEDGED_READ_THREAD_POOL; > ... > private synchronized void initThreadsNumForHedgedReads(int num) { > {code} > 2 DFS clients may update the same static variable in a race because the lock > is on each DFS client object, not on the shared DFSClient class object. > There are 2 possible fixes: > 1. "Global thread pool": Change initThreadsNumForHedgedReads to static > 2. "Per-client thread pool": Change HEDGED_READ_THREAD_POOL to non-static > From the description for property {{dfs.client.hedged.read.threadpool.size}}: > {quote} > to a positive number. The threadpool size is how many threads to dedicate > to the running of these 'hedged', concurrent reads in your client. > {quote} > it seems to indicate the thread pool is per DFS client. > Let's assume we go with #1 "Global thread pool". One DFS client has the > property set to 10 in its config, while the other client has the property set > to 5 in its config, what is supposed to the size of the global thread pool? > 5? 10? Or 15? > The 2nd fix seems more reasonable to me. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11303) Hedged read might hang infinitely if read data from all DN failed
[ https://issues.apache.org/jira/browse/HDFS-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11303: --- Attachment: HDFS-11303.003.patch > Hedged read might hang infinitely if read data from all DN failed > -- > > Key: HDFS-11303 > URL: https://issues.apache.org/jira/browse/HDFS-11303 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.0.0-alpha1 >Reporter: Chen Zhang >Assignee: Chen Zhang > Attachments: HDFS-11303-001.patch, HDFS-11303-001.patch, > HDFS-11303-002.patch, HDFS-11303-002.patch, HDFS-11303.003.patch > > > Hedged read will read from a DN first, if timeout, then read other DNs > simultaneously. > If read all DN failed, this bug will cause the future-list not empty(the > first timeout request left in list), and hang in the loop infinitely -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12281: -- Description: # XML Property: ozone.handler.type Value - local ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed # XML Property: ozone.scm.handler.count.key - 20 ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 # XML Property: ozone.metastore.impl- RocksDB ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- OZONE_METADATA_STORE_IMPL_ROCKSDB was: # XML Property: ozone.handler.type Value - local ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed # XML Property: ozone.scm.handler.count.key - 20 ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 # XML Property: ozone.metastore.impl- RocksDB ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > Attachments: HDFS-12281-HDFS-7240.01.patch, > HDFS-12281-HDFS-7240.02.patch > > > # XML Property: ozone.handler.type Value - local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed > # XML Property: ozone.scm.handler.count.key - 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 > # XML Property: ozone.metastore.impl- RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- > OZONE_METADATA_STORE_IMPL_ROCKSDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122149#comment-16122149 ] Ajay Yadav commented on HDFS-12281: --- Hi [~cheersyang], updated the patch with RocksDB as default implementation for metastore. > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > Attachments: HDFS-12281-HDFS-7240.01.patch, > HDFS-12281-HDFS-7240.02.patch > > > # XML Property: ozone.handler.type Value - local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed > # XML Property: ozone.scm.handler.count.key - 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 > # XML Property: ozone.metastore.impl- RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12273) Federation UI
[ https://issues.apache.org/jira/browse/HDFS-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122146#comment-16122146 ] Íñigo Goiri commented on HDFS-12273: [~raviprak], we would appreciate feedback on the pure UI side that tries to mimic the Namenode UI. > Federation UI > - > > Key: HDFS-12273 > URL: https://issues.apache.org/jira/browse/HDFS-12273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > Attachments: HDFS-12273-HDFS-10467-000.patch, > HDFS-12273-HDFS-10467-001.patch > > > Add the Web UI to the Router to expose the status of the federated cluster. > It includes the federation metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12281) Ozone: Ozone-default.xml has 3 properties that do not match the default Config value
[ https://issues.apache.org/jira/browse/HDFS-12281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav updated HDFS-12281: -- Attachment: HDFS-12281-HDFS-7240.02.patch > Ozone: Ozone-default.xml has 3 properties that do not match the default > Config value > > > Key: HDFS-12281 > URL: https://issues.apache.org/jira/browse/HDFS-12281 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Yadav >Assignee: Ajay Yadav > Fix For: HDFS-7240 > > Attachments: HDFS-12281-HDFS-7240.01.patch, > HDFS-12281-HDFS-7240.02.patch > > > # XML Property: ozone.handler.type Value - local > ## Config Name: OZONE_HANDLER_TYPE_DEFAULT - distributed > # XML Property: ozone.scm.handler.count.key - 20 > ## Config Name: OZONE_SCM_HANDLER_COUNT_DEFAULT - 10 > # XML Property: ozone.metastore.impl- RocksDB > ## Config Name: OZONE_METADATA_STORE_IMPL_DEFAULT- LevelDB -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Majercak updated HDFS-11576: -- Status: Patch Available (was: In Progress) > Block recovery will fail indefinitely if recovery time > heartbeat interval > --- > > Key: HDFS-11576 > URL: https://issues.apache.org/jira/browse/HDFS-11576 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs, namenode >Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.7.3, 2.7.2, 2.7.1 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Critical > Attachments: HDFS-11576.001.patch, HDFS-11576.002.patch, > HDFS-11576.003.patch, HDFS-11576.004.patch, HDFS-11576.005.patch, > HDFS-11576.006.patch, HDFS-11576.007.patch, HDFS-11576.008.patch, > HDFS-11576.repro.patch > > > Block recovery will fail indefinitely if the time to recover a block is > always longer than the heartbeat interval. Scenario: > 1. DN sends heartbeat > 2. NN sends a recovery command to DN, recoveryID=X > 3. DN starts recovery > 4. DN sends another heartbeat > 5. NN sends a recovery command to DN, recoveryID=X+1 > 6. DN calls commitBlockSyncronization after succeeding with first recovery to > NN, which fails because X < X+1 > ... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12287) Remove a no-longer applicable TODO comment in DatanodeManager
Chen Liang created HDFS-12287: - Summary: Remove a no-longer applicable TODO comment in DatanodeManager Key: HDFS-12287 URL: https://issues.apache.org/jira/browse/HDFS-12287 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Chen Liang Assignee: Chen Liang Priority: Trivial {{DatanodeManager}} has this this TODO comment {code} // TODO: Enables DFSNetworkTopology by default after more stress // testings/validations. {code} This has been resolved in HDFS-11998, but it missed removing this comment. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11576) Block recovery will fail indefinitely if recovery time > heartbeat interval
[ https://issues.apache.org/jira/browse/HDFS-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-11576: --- Status: In Progress (was: Patch Available) > Block recovery will fail indefinitely if recovery time > heartbeat interval > --- > > Key: HDFS-11576 > URL: https://issues.apache.org/jira/browse/HDFS-11576 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode, hdfs, namenode >Affects Versions: 3.0.0-alpha2, 3.0.0-alpha1, 2.7.3, 2.7.2, 2.7.1 >Reporter: Lukas Majercak >Assignee: Lukas Majercak >Priority: Critical > Attachments: HDFS-11576.001.patch, HDFS-11576.002.patch, > HDFS-11576.003.patch, HDFS-11576.004.patch, HDFS-11576.005.patch, > HDFS-11576.006.patch, HDFS-11576.007.patch, HDFS-11576.008.patch, > HDFS-11576.repro.patch > > > Block recovery will fail indefinitely if the time to recover a block is > always longer than the heartbeat interval. Scenario: > 1. DN sends heartbeat > 2. NN sends a recovery command to DN, recoveryID=X > 3. DN starts recovery > 4. DN sends another heartbeat > 5. NN sends a recovery command to DN, recoveryID=X+1 > 6. DN calls commitBlockSyncronization after succeeding with first recovery to > NN, which fails because X < X+1 > ... -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12238) Ozone: Add valid trace ID check in sendCommandAsync
[ https://issues.apache.org/jira/browse/HDFS-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122092#comment-16122092 ] Mukul Kumar Singh commented on HDFS-12238: -- [~anu] Yes, We can commit this patch, HDFS-12255 will fix Cblock test failures. > Ozone: Add valid trace ID check in sendCommandAsync > --- > > Key: HDFS-12238 > URL: https://issues.apache.org/jira/browse/HDFS-12238 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Ajay Yadav > Labels: newbie > Attachments: HDFS-12238-HDFS-7240.01.patch > > > In the function {{XceiverClientHandler#sendCommandAsync}} we should add a > check > {code} > if(StringUtils.isEmpty(request.getTraceID())) { > throw new IllegalArgumentException("Invalid trace ID"); > } > {code} > To ensure that ozone clients always send a valid trace ID. However, when you > do that a set of current tests that do add a valid trace ID will fail. So we > need to fix these tests too. > {code} > TestContainerMetrics.testContainerMetrics > TestOzoneContainer.testBothGetandPutSmallFile > TestOzoneContainer.testCloseContainer > TestOzoneContainer.testOzoneContainerViaDataNode > TestOzoneContainer.testXcieverClientAsync > TestOzoneContainer.testCreateOzoneContainer > TestOzoneContainer.testDeleteContainer > TestContainerServer.testClientServer > TestContainerServer.testClientServerWithContainerDispatcher > TestKeys.testPutAndGetKeyWithDnRestart > {code} > This is based on a comment from [~vagarychen] in HDFS-11580. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12255) Block Storage: Cblock should generated unique trace ID for the ops
[ https://issues.apache.org/jira/browse/HDFS-12255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12255: - Attachment: HDFS-12255-HDFS-7240.002.patch > Block Storage: Cblock should generated unique trace ID for the ops > -- > > Key: HDFS-12255 > URL: https://issues.apache.org/jira/browse/HDFS-12255 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12255-HDFS-7240.001.patch, > HDFS-12255-HDFS-7240.002.patch > > > Cblock tests fails because cblock does not generate unique trace id for each > op. > {code} > java.lang.AssertionError: expected:<0> but was:<1051> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:743) > at org.junit.Assert.assertEquals(Assert.java:118) > at org.junit.Assert.assertEquals(Assert.java:555) > at org.junit.Assert.assertEquals(Assert.java:542) > at > org.apache.hadoop.cblock.TestBufferManager.testRepeatedBlockWrites(TestBufferManager.java:448) > {code} > This failure is because of following error. > {code} > 017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > scm.XceiverClientHandler (XceiverClientHandler.java:sendCommandAsync(134)) - > Command with Trace already exists. Ignoring this command. . Previous Command: > java.util.concurrent.CompletableFuture@7847fc2d[Not completed, 1 dependents] > 2017-08-02 17:50:34,569 [Cache Block Writer Thread #4] ERROR > jscsiHelper.ContainerCacheFlusher (BlockWriterTask.java:run(108)) - Writing > of block:44 failed, We have attempted to write this block 7 tim > es to the container container2483304118.Trace ID: > java.lang.IllegalStateException: Duplicate trace ID. Command with this trace > ID is already executing. Please ensure that trace IDs are not reused. ID: > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommandAsync(XceiverClientHandler.java:139) > at > org.apache.hadoop.scm.XceiverClientHandler.sendCommand(XceiverClientHandler.java:114) > at > org.apache.hadoop.scm.XceiverClient.sendCommand(XceiverClient.java:132) > at > org.apache.hadoop.scm.storage.ContainerProtocolCalls.writeSmallFile(ContainerProtocolCalls.java:225) > at > org.apache.hadoop.cblock.jscsiHelper.BlockWriterTask.run(BlockWriterTask.java:97) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12238) Ozone: Add valid trace ID check in sendCommandAsync
[ https://issues.apache.org/jira/browse/HDFS-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122086#comment-16122086 ] Chen Liang commented on HDFS-12238: --- Thanks [~msingh] [~anu] for the follow-up! Looks good to me. > Ozone: Add valid trace ID check in sendCommandAsync > --- > > Key: HDFS-12238 > URL: https://issues.apache.org/jira/browse/HDFS-12238 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Ajay Yadav > Labels: newbie > Attachments: HDFS-12238-HDFS-7240.01.patch > > > In the function {{XceiverClientHandler#sendCommandAsync}} we should add a > check > {code} > if(StringUtils.isEmpty(request.getTraceID())) { > throw new IllegalArgumentException("Invalid trace ID"); > } > {code} > To ensure that ozone clients always send a valid trace ID. However, when you > do that a set of current tests that do add a valid trace ID will fail. So we > need to fix these tests too. > {code} > TestContainerMetrics.testContainerMetrics > TestOzoneContainer.testBothGetandPutSmallFile > TestOzoneContainer.testCloseContainer > TestOzoneContainer.testOzoneContainerViaDataNode > TestOzoneContainer.testXcieverClientAsync > TestOzoneContainer.testCreateOzoneContainer > TestOzoneContainer.testDeleteContainer > TestContainerServer.testClientServer > TestContainerServer.testClientServerWithContainerDispatcher > TestKeys.testPutAndGetKeyWithDnRestart > {code} > This is based on a comment from [~vagarychen] in HDFS-11580. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-9153) Pretty-format the output for DFSIO
[ https://issues.apache.org/jira/browse/HDFS-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122049#comment-16122049 ] Konstantin Shvachko edited comment on HDFS-9153 at 8/10/17 6:35 PM: Hey guys I think the metrics you introduced is absolutely deceiving, and has nothing to do with the throughput the benchmark is intended to measure. "Test exec time" is the running time of the job, which includes the compute overhead: scheduling, cleanup, and retries if there were failed maps. While we want to benchmark the average throughput of the actual data transfers on HDFS. You should see the implementation measures time of transfers only. The formatting changes are fine. But I think "Total Throughput" should be removed. The bug reported in MAPREDUCE-6931 makes it invalid, but even if fixed it is still deceiving. -Also, DFSIO issues should be filed on HDFS jira. Then you should expect more prompt response.- _Sorry last part was for the other jira. Please ignore._ was (Author: shv): Hey guys I think the metrics you introduced is absolutely deceiving, and has nothing to do with the throughput the benchmark is intended to measure. "Test exec time" is the running time of the job, which includes the compute overhead: scheduling, cleanup, and retries if there were failed maps. While we want to benchmark the average throughput of the actual data transfers on HDFS. You should see the implementation measures time of transfers only. The formatting changes are fine. But I think "Total Throughput" should be removed. The bug reported in MAPREDUCE-6931 makes it invalid, but even if fixed it is still deceiving. Also, DFSIO issues should be filed on HDFS jira. Then you should expect more prompt response. > Pretty-format the output for DFSIO > -- > > Key: HDFS-9153 > URL: https://issues.apache.org/jira/browse/HDFS-9153 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-9153-v1.patch > > > Ref. the following DFSIO output, I was surprised the test throughput was only > {{17}} MB/s, which doesn't make sense for a real cluster. Maybe it's used for > other purpose? For users, it may make more sense to give the throughput 1610 > MB/s (1228800/763), calculated by *Total MBytes processed / Test exec time*. > {noformat} > 15/09/28 11:42:23 INFO fs.TestDFSIO: - TestDFSIO - : write > 15/09/28 11:42:23 INFO fs.TestDFSIO:Date & time: Mon Sep 28 > 11:42:23 CST 2015 > 15/09/28 11:42:23 INFO fs.TestDFSIO:Number of files: 100 > 15/09/28 11:42:23 INFO fs.TestDFSIO: Total MBytes processed: 1228800.0 > 15/09/28 11:42:23 INFO fs.TestDFSIO: Throughput mb/sec: > 17.457387239456878 > 15/09/28 11:42:23 INFO fs.TestDFSIO: Average IO rate mb/sec: 17.57563018798828 > 15/09/28 11:42:23 INFO fs.TestDFSIO: IO rate std deviation: > 1.7076328985378455 > 15/09/28 11:42:23 INFO fs.TestDFSIO: Test exec time sec: 762.697 > 15/09/28 11:42:23 INFO fs.TestDFSIO: > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12273) Federation UI
[ https://issues.apache.org/jira/browse/HDFS-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122059#comment-16122059 ] Ravi Prakash commented on HDFS-12273: - I didn't realize this was in the Federation branch. This is way out of my field. Sorry. Could some branch committers please review it? > Federation UI > - > > Key: HDFS-12273 > URL: https://issues.apache.org/jira/browse/HDFS-12273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > Attachments: HDFS-12273-HDFS-10467-000.patch, > HDFS-12273-HDFS-10467-001.patch > > > Add the Web UI to the Router to expose the status of the federated cluster. > It includes the federation metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9153) Pretty-format the output for DFSIO
[ https://issues.apache.org/jira/browse/HDFS-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122049#comment-16122049 ] Konstantin Shvachko commented on HDFS-9153: --- Hey guys I think the metrics you introduced is absolutely deceiving, and has nothing to do with the throughput the benchmark is intended to measure. "Test exec time" is the running time of the job, which includes the compute overhead: scheduling, cleanup, and retries if there were failed maps. While we want to benchmark the average throughput of the actual data transfers on HDFS. You should see the implementation measures time of transfers only. The formatting changes are fine. But I think "Total Throughput" should be removed. The bug reported in MAPREDUCE-6931 makes it invalid, but even if fixed it is still deceiving. Also, DFSIO issues should be filed on HDFS jira. Then you should expect more prompt response. > Pretty-format the output for DFSIO > -- > > Key: HDFS-9153 > URL: https://issues.apache.org/jira/browse/HDFS-9153 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Kai Zheng >Assignee: Kai Zheng > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: HDFS-9153-v1.patch > > > Ref. the following DFSIO output, I was surprised the test throughput was only > {{17}} MB/s, which doesn't make sense for a real cluster. Maybe it's used for > other purpose? For users, it may make more sense to give the throughput 1610 > MB/s (1228800/763), calculated by *Total MBytes processed / Test exec time*. > {noformat} > 15/09/28 11:42:23 INFO fs.TestDFSIO: - TestDFSIO - : write > 15/09/28 11:42:23 INFO fs.TestDFSIO:Date & time: Mon Sep 28 > 11:42:23 CST 2015 > 15/09/28 11:42:23 INFO fs.TestDFSIO:Number of files: 100 > 15/09/28 11:42:23 INFO fs.TestDFSIO: Total MBytes processed: 1228800.0 > 15/09/28 11:42:23 INFO fs.TestDFSIO: Throughput mb/sec: > 17.457387239456878 > 15/09/28 11:42:23 INFO fs.TestDFSIO: Average IO rate mb/sec: 17.57563018798828 > 15/09/28 11:42:23 INFO fs.TestDFSIO: IO rate std deviation: > 1.7076328985378455 > 15/09/28 11:42:23 INFO fs.TestDFSIO: Test exec time sec: 762.697 > 15/09/28 11:42:23 INFO fs.TestDFSIO: > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org