[jira] [Commented] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17373232#comment-17373232 ] Ayush Saxena commented on HDFS-16100: - I am also not very confident about this part of code. In general looks ok, but might miss some corner case or so, not very sure > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Assignee: wudeyu >Priority: Major > Attachments: HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16106) Fix flaky unit test TestDFSShell
[
https://issues.apache.org/jira/browse/HDFS-16106?focusedWorklogId=617975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617975
]
ASF GitHub Bot logged work on HDFS-16106:
-
Author: ASF GitHub Bot
Created on: 02/Jul/21 04:57
Start Date: 02/Jul/21 04:57
Worklog Time Spent: 10m
Work Description: tomscut commented on pull request #3168:
URL: https://github.com/apache/hadoop/pull/3168#issuecomment-872713688
Thanks @ayushtkn for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 617975)
Time Spent: 50m (was: 40m)
> Fix flaky unit test TestDFSShell
>
>
> Key: HDFS-16106
> URL: https://issues.apache.org/jira/browse/HDFS-16106
> Project: Hadoop HDFS
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> This unit test occasionally fails.
> The value set for dfs.namenode.accesstime.precision is too low, result in the
> execution of the method, accesstime could be set many times, eventually
> leading to failed assert.
> IMO, dfs.namenode.accesstime.precision should be greater than or equal to the
> timeout(120s) of TestDFSShell#testCopyCommandsWithPreserveOption(), or
> directly set to 0 to disable this feature.
>
> {code:java}
> [ERROR] Tests run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 106.778 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell[ERROR] Tests
> run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 106.778 s <<<
> FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.353 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095098319> but was:<1625095099374> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.467 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095192527> but was:<1625095193950> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2323)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
>
[jira] [Work logged] (HDFS-16106) Fix flaky unit test TestDFSShell
[
https://issues.apache.org/jira/browse/HDFS-16106?focusedWorklogId=617946=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617946
]
ASF GitHub Bot logged work on HDFS-16106:
-
Author: ASF GitHub Bot
Created on: 02/Jul/21 02:00
Start Date: 02/Jul/21 02:00
Worklog Time Spent: 10m
Work Description: tomscut commented on pull request #3168:
URL: https://github.com/apache/hadoop/pull/3168#issuecomment-872657263
> Thank you @tomscut
Thanks @aajisaka for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 617946)
Time Spent: 40m (was: 0.5h)
> Fix flaky unit test TestDFSShell
>
>
> Key: HDFS-16106
> URL: https://issues.apache.org/jira/browse/HDFS-16106
> Project: Hadoop HDFS
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This unit test occasionally fails.
> The value set for dfs.namenode.accesstime.precision is too low, result in the
> execution of the method, accesstime could be set many times, eventually
> leading to failed assert.
> IMO, dfs.namenode.accesstime.precision should be greater than or equal to the
> timeout(120s) of TestDFSShell#testCopyCommandsWithPreserveOption(), or
> directly set to 0 to disable this feature.
>
> {code:java}
> [ERROR] Tests run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 106.778 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell[ERROR] Tests
> run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 106.778 s <<<
> FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.353 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095098319> but was:<1625095099374> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.467 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095192527> but was:<1625095193950> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2323)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
>
[jira] [Work logged] (HDFS-16106) Fix flaky unit test TestDFSShell
[
https://issues.apache.org/jira/browse/HDFS-16106?focusedWorklogId=617940=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617940
]
ASF GitHub Bot logged work on HDFS-16106:
-
Author: ASF GitHub Bot
Created on: 02/Jul/21 01:33
Start Date: 02/Jul/21 01:33
Worklog Time Spent: 10m
Work Description: tomscut edited a comment on pull request #3168:
URL: https://github.com/apache/hadoop/pull/3168#issuecomment-872649177
Hi @aajisaka @tasanuma @jojochuang, could you please help to look at this.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 617940)
Time Spent: 0.5h (was: 20m)
> Fix flaky unit test TestDFSShell
>
>
> Key: HDFS-16106
> URL: https://issues.apache.org/jira/browse/HDFS-16106
> Project: Hadoop HDFS
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This unit test occasionally fails.
> The value set for dfs.namenode.accesstime.precision is too low, result in the
> execution of the method, accesstime could be set many times, eventually
> leading to failed assert.
> IMO, dfs.namenode.accesstime.precision should be greater than or equal to the
> timeout(120s) of TestDFSShell#testCopyCommandsWithPreserveOption(), or
> directly set to 0 to disable this feature.
>
> {code:java}
> [ERROR] Tests run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 106.778 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell[ERROR] Tests
> run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 106.778 s <<<
> FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.353 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095098319> but was:<1625095099374> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.467 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095192527> but was:<1625095193950> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2323)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
>
[jira] [Work logged] (HDFS-16106) Fix flaky unit test TestDFSShell
[
https://issues.apache.org/jira/browse/HDFS-16106?focusedWorklogId=617939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617939
]
ASF GitHub Bot logged work on HDFS-16106:
-
Author: ASF GitHub Bot
Created on: 02/Jul/21 01:31
Start Date: 02/Jul/21 01:31
Worklog Time Spent: 10m
Work Description: tomscut commented on pull request #3168:
URL: https://github.com/apache/hadoop/pull/3168#issuecomment-872649177
Hi @aajisaka @tamaashu @jojochuang, could you please help to look at this.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 617939)
Time Spent: 20m (was: 10m)
> Fix flaky unit test TestDFSShell
>
>
> Key: HDFS-16106
> URL: https://issues.apache.org/jira/browse/HDFS-16106
> Project: Hadoop HDFS
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This unit test occasionally fails.
> The value set for dfs.namenode.accesstime.precision is too low, result in the
> execution of the method, accesstime could be set many times, eventually
> leading to failed assert.
> IMO, dfs.namenode.accesstime.precision should be greater than or equal to the
> timeout(120s) of TestDFSShell#testCopyCommandsWithPreserveOption(), or
> directly set to 0 to disable this feature.
>
> {code:java}
> [ERROR] Tests run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 106.778 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell[ERROR] Tests
> run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 106.778 s <<<
> FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.353 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095098319> but was:<1625095099374> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.467 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095192527> but was:<1625095193950> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2323)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
>
[jira] [Updated] (HDFS-16106) Fix flaky unit test TestDFSShell
[
https://issues.apache.org/jira/browse/HDFS-16106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-16106:
---
Issue Type: Wish (was: Bug)
> Fix flaky unit test TestDFSShell
>
>
> Key: HDFS-16106
> URL: https://issues.apache.org/jira/browse/HDFS-16106
> Project: Hadoop HDFS
> Issue Type: Wish
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This unit test occasionally fails.
> The value set for dfs.namenode.accesstime.precision is too low, result in the
> execution of the method, accesstime could be set many times, eventually
> leading to failed assert.
> IMO, dfs.namenode.accesstime.precision should be greater than or equal to the
> timeout(120s) of TestDFSShell#testCopyCommandsWithPreserveOption(), or
> directly set to 0 to disable this feature.
>
> {code:java}
> [ERROR] Tests run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 106.778 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell[ERROR] Tests
> run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 106.778 s <<<
> FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.353 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095098319> but was:<1625095099374> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.467 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095192527> but was:<1625095193950> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2323)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.173 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095196756> but was:<1625095197975> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2303)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>
[jira] [Updated] (HDFS-16106) Fix flaky unit test TestDFSShell
[
https://issues.apache.org/jira/browse/HDFS-16106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-16106:
--
Labels: pull-request-available (was: )
> Fix flaky unit test TestDFSShell
>
>
> Key: HDFS-16106
> URL: https://issues.apache.org/jira/browse/HDFS-16106
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This unit test occasionally fails.
> The value set for dfs.namenode.accesstime.precision is too low, result in the
> execution of the method, accesstime could be set many times, eventually
> leading to failed assert.
> IMO, dfs.namenode.accesstime.precision should be greater than or equal to the
> timeout(120s) of TestDFSShell#testCopyCommandsWithPreserveOption(), or
> directly set to 0 to disable this feature.
>
> {code:java}
> [ERROR] Tests run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
> 106.778 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell[ERROR] Tests
> run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 106.778 s <<<
> FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.353 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095098319> but was:<1625095099374> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.467 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095192527> but was:<1625095193950> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2323)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.lang.Thread.run(Thread.java:748)
> [ERROR]
> testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
> elapsed: 2.173 s <<< FAILURE! java.lang.AssertionError:
> expected:<1625095196756> but was:<1625095197975> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> org.junit.Assert.assertEquals(Assert.java:633) at
> org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2303)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>
[jira] [Work logged] (HDFS-16106) Fix flaky unit test TestDFSShell
[ https://issues.apache.org/jira/browse/HDFS-16106?focusedWorklogId=617938=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617938 ] ASF GitHub Bot logged work on HDFS-16106: - Author: ASF GitHub Bot Created on: 02/Jul/21 01:27 Start Date: 02/Jul/21 01:27 Worklog Time Spent: 10m Work Description: tomscut opened a new pull request #3168: URL: https://github.com/apache/hadoop/pull/3168 JIRA: [HDFS-16106](https://issues.apache.org/jira/browse/HDFS-16106) This unit test occasionally fails. The value set for dfs.namenode.accesstime.precision is too low, result in the execution of the method, accesstime could be set many times, eventually leading to failed assert. IMO, dfs.namenode.accesstime.precision should be greater than or equal to the timeout(120s) of TestDFSShell#testCopyCommandsWithPreserveOption(), or directly set to 0 to disable this feature. ```[ERROR] Tests run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 106.778 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell [ERROR] testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time elapsed: 2.353 s <<< FAILURE! java.lang.AssertionError: expected:<1625095098319> but was:<1625095099374> at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:633) at org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2282) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time elapsed: 2.467 s <<< FAILURE! java.lang.AssertionError: expected:<1625095192527> but was:<1625095193950> at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:633) at org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2323) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) [ERROR] testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time elapsed: 2.173 s <<< FAILURE! java.lang.AssertionError: expected:<1625095196756> but was:<1625095197975> at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:633) at org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2303) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
[jira] [Created] (HDFS-16106) Fix flaky unit test TestDFSShell
tomscut created HDFS-16106:
--
Summary: Fix flaky unit test TestDFSShell
Key: HDFS-16106
URL: https://issues.apache.org/jira/browse/HDFS-16106
Project: Hadoop HDFS
Issue Type: Bug
Reporter: tomscut
Assignee: tomscut
This unit test occasionally fails.
The value set for dfs.namenode.accesstime.precision is too low, result in the
execution of the method, accesstime could be set many times, eventually leading
to failed assert.
IMO, dfs.namenode.accesstime.precision should be greater than or equal to the
timeout(120s) of TestDFSShell#testCopyCommandsWithPreserveOption(), or directly
set to 0 to disable this feature.
{code:java}
[ERROR] Tests run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed:
106.778 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell[ERROR] Tests
run: 52, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 106.778 s <<<
FAILURE! - in org.apache.hadoop.hdfs.TestDFSShell [ERROR]
testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell) Time
elapsed: 2.353 s <<< FAILURE! java.lang.AssertionError:
expected:<1625095098319> but was:<1625095099374> at
org.junit.Assert.fail(Assert.java:89) at
org.junit.Assert.failNotEquals(Assert.java:835) at
org.junit.Assert.assertEquals(Assert.java:647) at
org.junit.Assert.assertEquals(Assert.java:633) at
org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2282)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.lang.Thread.run(Thread.java:748)
[ERROR] testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell)
Time elapsed: 2.467 s <<< FAILURE! java.lang.AssertionError:
expected:<1625095192527> but was:<1625095193950> at
org.junit.Assert.fail(Assert.java:89) at
org.junit.Assert.failNotEquals(Assert.java:835) at
org.junit.Assert.assertEquals(Assert.java:647) at
org.junit.Assert.assertEquals(Assert.java:633) at
org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2323)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.lang.Thread.run(Thread.java:748)
[ERROR] testCopyCommandsWithPreserveOption(org.apache.hadoop.hdfs.TestDFSShell)
Time elapsed: 2.173 s <<< FAILURE! java.lang.AssertionError:
expected:<1625095196756> but was:<1625095197975> at
org.junit.Assert.fail(Assert.java:89) at
org.junit.Assert.failNotEquals(Assert.java:835) at
org.junit.Assert.assertEquals(Assert.java:647) at
org.junit.Assert.assertEquals(Assert.java:633) at
org.apache.hadoop.hdfs.TestDFSShell.testCopyCommandsWithPreserveOption(TestDFSShell.java:2303)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
[jira] [Work logged] (HDFS-16105) Edit log corruption due to mismatch between fileId and path
[
https://issues.apache.org/jira/browse/HDFS-16105?focusedWorklogId=617935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617935
]
ASF GitHub Bot logged work on HDFS-16105:
-
Author: ASF GitHub Bot
Created on: 02/Jul/21 01:06
Start Date: 02/Jul/21 01:06
Worklog Time Spent: 10m
Work Description: ferhui closed pull request #3161:
URL: https://github.com/apache/hadoop/pull/3161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 617935)
Time Spent: 0.5h (was: 20m)
> Edit log corruption due to mismatch between fileId and path
> ---
>
> Key: HDFS-16105
> URL: https://issues.apache.org/jira/browse/HDFS-16105
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.2.2, 3.3.1
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> We do stress testing in FUSE with HDFS, Standby Namenode crashes.
> The log is following
> {quote}
> 2021-06-25 17:13:02,953 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
> on operation AddBlockOp [path=/xxx/fiotest_write.354.46,
> penultimateBlock=xxx, lastBlock=xxx, numOfBytes=0}, RpcClientId=,
> RpcCallId=-2]
> java.io.FileNotFoundException: File /xxx/fiotest_write.354.46 does not exist.
> {quote}
> The following steps can reproduce it (Illegal writes).
> 1 create file A (fileId X) (1st block is being written)
> 2 rename file A to file B(stile fileId X)
> 3 continue to write file A(using above the same outputstream, 2nd block need
> to be written)
> 4 standby namenode load the above edits and would crash
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16088) Standby NameNode process getLiveDatanodeStorageReport request to reduce Active load
[ https://issues.apache.org/jira/browse/HDFS-16088?focusedWorklogId=617934=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617934 ] ASF GitHub Bot logged work on HDFS-16088: - Author: ASF GitHub Bot Created on: 02/Jul/21 00:59 Start Date: 02/Jul/21 00:59 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3140: URL: https://github.com/apache/hadoop/pull/3140#issuecomment-872639218 Failed junit tests | hadoop.hdfs.TestRollingUpgrade Hi @Hexiaoqiao , This failed unit test work fine locally. And I added a separate unit test. Could you please take a quick look? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617934) Time Spent: 50m (was: 40m) > Standby NameNode process getLiveDatanodeStorageReport request to reduce > Active load > --- > > Key: HDFS-16088 > URL: https://issues.apache.org/jira/browse/HDFS-16088 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > As with HDFS-13183, NameNodeConnector#getLiveDatanodeStorageReport() can also > request to SNN to reduce the ANN load. > There are two points that need to be mentioned: > 1. FSNamesystem#getLiveDatanodeStorageReport() is > OperationCategory.UNCHECKED, so we can access SNN directly. > 2. We can share the same UT(testBalancerRequestSBNWithHA) with > NameNodeConnector#getBlocks(). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16088) Standby NameNode process getLiveDatanodeStorageReport request to reduce Active load
[
https://issues.apache.org/jira/browse/HDFS-16088?focusedWorklogId=617715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617715
]
ASF GitHub Bot logged work on HDFS-16088:
-
Author: ASF GitHub Bot
Created on: 01/Jul/21 17:13
Start Date: 01/Jul/21 17:13
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3140:
URL: https://github.com/apache/hadoop/pull/3140#issuecomment-872416289
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 49s | | trunk passed |
| +1 :green_heart: | compile | 1m 40s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 34s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 243m 35s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3140/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 336m 38s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3140/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3140 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 29cf82750999 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 218fafc781df4e88022a6077eb3da5bc8c876f5d |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3140/2/testReport/ |
| Max. process+thread count | 3606 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-16100) HA: Improve performance of Standby node transition to Active
[ https://issues.apache.org/jira/browse/HDFS-16100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372912#comment-17372912 ] Xiaoqiao He commented on HDFS-16100: I am not familiar to this logic, but it sounds reasonable at first glance. cc [~weichiu],[~ayushtkn] what do you think about? > HA: Improve performance of Standby node transition to Active > - > > Key: HDFS-16100 > URL: https://issues.apache.org/jira/browse/HDFS-16100 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.3.1 >Reporter: wudeyu >Assignee: wudeyu >Priority: Major > Attachments: HDFS-16100.patch > > > pendingDNMessages in Standby is used to support process postponed block > reports. Block reports in pendingDNMessages would be processed: > # If GS of replica is in the future, Standby Node will process it when > corresponding edit log(e.g add_block) is loaded. > # If replica is corrupted, Standby Node will process it while it transfer to > Active. > # If DataNode is removed, corresponding of block reports will be removed in > pendingDNMessages. > Obviously, if num of corrupted replica grows, more time cost during > transferring. In out situation, there're 60 millions block reports in > pendingDNMessages before transfer. Processing block reports cost almost 7mins > and it's killed by zkfc. The replica state of the most block reports is RBW > with wrong GS(less than storedblock in Standby Node). > In my opinion, Standby Node could ignore the block reports that replica state > is RBW with wrong GS. Because Active node/DataNode will remove it later. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16099) Make bpServiceToActive to be volatile
[ https://issues.apache.org/jira/browse/HDFS-16099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16099. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to trunk. Thanks [~zhangshuyan] for your report and contribution! > Make bpServiceToActive to be volatile > - > > Key: HDFS-16099 > URL: https://issues.apache.org/jira/browse/HDFS-16099 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 40m > Remaining Estimate: 0h > > BPOfferService#bpServiceToActive is not volatile, which may cause > _CommandProcessingThread_ to get the out-of-date active namenode. > When a failover occurs, the old ANN's _CommandProcessingThread_ may read the > outdated BPOfferService#bpServiceToActive and execute the NN's command. At > this time, if the new ANN's _CommandProcessingThread_ reads the new value of > bpServiceToActive, split brain will occur; otherwise, the new ANN's commands > cannot be executed normally, which is also unacceptable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16099) Make bpServiceToActive to be volatile
[ https://issues.apache.org/jira/browse/HDFS-16099?focusedWorklogId=617664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617664 ] ASF GitHub Bot logged work on HDFS-16099: - Author: ASF GitHub Bot Created on: 01/Jul/21 15:53 Start Date: 01/Jul/21 15:53 Worklog Time Spent: 10m Work Description: Hexiaoqiao commented on pull request #3159: URL: https://github.com/apache/hadoop/pull/3159#issuecomment-872358780 Committed to trunk. Thanks @zhangshuyan0 for your contribution! Thanks @jojochuang and @ayushtkn for your reviews! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617664) Time Spent: 40m (was: 0.5h) > Make bpServiceToActive to be volatile > - > > Key: HDFS-16099 > URL: https://issues.apache.org/jira/browse/HDFS-16099 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > BPOfferService#bpServiceToActive is not volatile, which may cause > _CommandProcessingThread_ to get the out-of-date active namenode. > When a failover occurs, the old ANN's _CommandProcessingThread_ may read the > outdated BPOfferService#bpServiceToActive and execute the NN's command. At > this time, if the new ANN's _CommandProcessingThread_ reads the new value of > bpServiceToActive, split brain will occur; otherwise, the new ANN's commands > cannot be executed normally, which is also unacceptable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16099) Make bpServiceToActive to be volatile
[ https://issues.apache.org/jira/browse/HDFS-16099?focusedWorklogId=617661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617661 ] ASF GitHub Bot logged work on HDFS-16099: - Author: ASF GitHub Bot Created on: 01/Jul/21 15:52 Start Date: 01/Jul/21 15:52 Worklog Time Spent: 10m Work Description: Hexiaoqiao merged pull request #3159: URL: https://github.com/apache/hadoop/pull/3159 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617661) Time Spent: 0.5h (was: 20m) > Make bpServiceToActive to be volatile > - > > Key: HDFS-16099 > URL: https://issues.apache.org/jira/browse/HDFS-16099 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Shuyan Zhang >Assignee: Shuyan Zhang >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > BPOfferService#bpServiceToActive is not volatile, which may cause > _CommandProcessingThread_ to get the out-of-date active namenode. > When a failover occurs, the old ANN's _CommandProcessingThread_ may read the > outdated BPOfferService#bpServiceToActive and execute the NN's command. At > this time, if the new ANN's _CommandProcessingThread_ reads the new value of > bpServiceToActive, split brain will occur; otherwise, the new ANN's commands > cannot be executed normally, which is also unacceptable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[ https://issues.apache.org/jira/browse/HDFS-16104?focusedWorklogId=617631=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617631 ] ASF GitHub Bot logged work on HDFS-16104: - Author: ASF GitHub Bot Created on: 01/Jul/21 14:46 Start Date: 01/Jul/21 14:46 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3160: URL: https://github.com/apache/hadoop/pull/3160#issuecomment-872309002 > Thanx @tomscut for the contribution!!! Thanks @ayushtkn for the merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617631) Time Spent: 1h 20m (was: 1h 10m) > Remove unused parameter and fix java doc for DiskBalancerCLI > > > Key: HDFS-16104 > URL: https://issues.apache.org/jira/browse/HDFS-16104 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16105) Edit log corruption due to mismatch between fileId and path
[
https://issues.apache.org/jira/browse/HDFS-16105?focusedWorklogId=617606=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617606
]
ASF GitHub Bot logged work on HDFS-16105:
-
Author: ASF GitHub Bot
Created on: 01/Jul/21 13:26
Start Date: 01/Jul/21 13:26
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3161:
URL: https://github.com/apache/hadoop/pull/3161#issuecomment-872246146
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 21s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 18s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 356m 16s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3161/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 450m 35s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.TestINodeFile |
| | hadoop.hdfs.TestFileCreation |
| | hadoop.hdfs.server.namenode.TestDeleteRace |
| | hadoop.fs.contract.hdfs.TestHDFSContractAppend |
| | hadoop.hdfs.TestRenameWhileOpen |
| | hadoop.hdfs.TestLease |
| | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
| | hadoop.hdfs.server.namenode.TestNameNodeXAttr |
| | hadoop.hdfs.TestReservedRawPaths |
| | hadoop.hdfs.server.namenode.TestHDFSConcat |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.server.namenode.TestFileContextXAttr |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsVolumeList |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.web.TestWebHDFSXAttr |
| |
hadoop.hdfs.server.namenode.TestDecommissioningStatusWithBackoffMonitor |
| | hadoop.hdfs.TestFileAppend3 |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
[jira] [Work logged] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[ https://issues.apache.org/jira/browse/HDFS-16104?focusedWorklogId=617600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617600 ] ASF GitHub Bot logged work on HDFS-16104: - Author: ASF GitHub Bot Created on: 01/Jul/21 13:14 Start Date: 01/Jul/21 13:14 Worklog Time Spent: 10m Work Description: ayushtkn merged pull request #3160: URL: https://github.com/apache/hadoop/pull/3160 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617600) Time Spent: 1h (was: 50m) > Remove unused parameter and fix java doc for DiskBalancerCLI > > > Key: HDFS-16104 > URL: https://issues.apache.org/jira/browse/HDFS-16104 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[ https://issues.apache.org/jira/browse/HDFS-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372783#comment-17372783 ] Ayush Saxena commented on HDFS-16104: - Committed to trunk. > Remove unused parameter and fix java doc for DiskBalancerCLI > > > Key: HDFS-16104 > URL: https://issues.apache.org/jira/browse/HDFS-16104 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[ https://issues.apache.org/jira/browse/HDFS-16104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HDFS-16104. - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Remove unused parameter and fix java doc for DiskBalancerCLI > > > Key: HDFS-16104 > URL: https://issues.apache.org/jira/browse/HDFS-16104 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[ https://issues.apache.org/jira/browse/HDFS-16104?focusedWorklogId=617601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617601 ] ASF GitHub Bot logged work on HDFS-16104: - Author: ASF GitHub Bot Created on: 01/Jul/21 13:14 Start Date: 01/Jul/21 13:14 Worklog Time Spent: 10m Work Description: ayushtkn commented on pull request #3160: URL: https://github.com/apache/hadoop/pull/3160#issuecomment-872237581 Thanx @tomscut for the contribution!!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617601) Time Spent: 1h 10m (was: 1h) > Remove unused parameter and fix java doc for DiskBalancerCLI > > > Key: HDFS-16104 > URL: https://issues.apache.org/jira/browse/HDFS-16104 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14788) Use dynamic regex filter to ignore copy of source files in Distcp
[ https://issues.apache.org/jira/browse/HDFS-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372761#comment-17372761 ] Hongbing Wang commented on HDFS-14788: -- Is there a plan to filter files by modtime? In the scenario of incremental data synchronization, if files in certain time windows can be specified, efficiency can be greatly improved. > Use dynamic regex filter to ignore copy of source files in Distcp > - > > Key: HDFS-14788 > URL: https://issues.apache.org/jira/browse/HDFS-14788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 3.2.1 >Reporter: Mukund Thakur >Assignee: Mukund Thakur >Priority: Major > Fix For: 3.3.0 > > > There is a feature in Distcp where we can ignore specific files to get copied > to the destination. This is currently based on a filter regex which is read > from a specific file. The process of creating different regex file for > different distcp jobs seems like a tedious task. What we are proposing is to > expose a regex_filter parameter which can be set during Distcp job creation > and use this filter in a new implementation CopyFilter class. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16104) Remove unused parameter and fix java doc for DiskBalancerCLI
[ https://issues.apache.org/jira/browse/HDFS-16104?focusedWorklogId=617571=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617571 ] ASF GitHub Bot logged work on HDFS-16104: - Author: ASF GitHub Bot Created on: 01/Jul/21 12:09 Start Date: 01/Jul/21 12:09 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3160: URL: https://github.com/apache/hadoop/pull/3160#issuecomment-872191677 Hi @ayushtkn , these UTs are related to the change and work fine locally. Could you please take a look? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617571) Time Spent: 50m (was: 40m) > Remove unused parameter and fix java doc for DiskBalancerCLI > > > Key: HDFS-16104 > URL: https://issues.apache.org/jira/browse/HDFS-16104 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Remove unused parameter and fix java doc for DiskBalancerCLI. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16095) Add lsQuotaList command and getQuotaListing api for hdfs quota
[ https://issues.apache.org/jira/browse/HDFS-16095?focusedWorklogId=617548=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617548 ] ASF GitHub Bot logged work on HDFS-16095: - Author: ASF GitHub Bot Created on: 01/Jul/21 11:35 Start Date: 01/Jul/21 11:35 Worklog Time Spent: 10m Work Description: zhuxiangyi commented on pull request #3155: URL: https://github.com/apache/hadoop/pull/3155#issuecomment-872169594 > It has a potential to hold the fsn/fsd lock for a long time and cause service outage or delays. > hold the fsn/fsd lock @kihwal Thanks for your comment. I tested to obtain quota information for 10w directories, and its holding lock time was about 300ms. Under normal circumstances, our quota list is limited, I think it will not be particularly high, this is just my guess. In addition, I did a test to get only the Quota directory path, and the 10w directory also took 20ms. We first get the path of the quota directory, and then get the quota information through getQuotaUsage, is this better? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617548) Time Spent: 1h (was: 50m) > Add lsQuotaList command and getQuotaListing api for hdfs quota > -- > > Key: HDFS-16095 > URL: https://issues.apache.org/jira/browse/HDFS-16095 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.4.0 >Reporter: Xiangyi Zhu >Assignee: Xiangyi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Currently hdfs does not support obtaining all quota information. The > administrator may need to check which quotas have been added to a certain > directory, or the quotas of the entire cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16098) ERROR tools.DiskBalancerCLI: java.lang.IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDFS-16098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangyanfu updated HDFS-16098: - Attachment: image-2021-07-01-18-34-54-905.png > ERROR tools.DiskBalancerCLI: java.lang.IllegalArgumentException > --- > > Key: HDFS-16098 > URL: https://issues.apache.org/jira/browse/HDFS-16098 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Affects Versions: 2.6.0 > Environment: VERSION info: > Hadoop 2.6.0-cdh5.14.4 >Reporter: wangyanfu >Priority: Blocker > Labels: diskbalancer > Fix For: 2.6.0 > > Attachments: image-2021-07-01-18-34-54-905.png, on-branch-3.1.jpg > > Original Estimate: 504h > Remaining Estimate: 504h > > when i tried to run > hdfs diskbalancer -plan $(hostname -f) > > > > i get this notice: > 21/06/30 11:30:41 ERROR tools.DiskBalancerCLI: > java.lang.IllegalArgumentException > > then i tried write the real hostname into my command , not work and same > error notice > i also tried use --plan instead of -plan , not work and same error notice > i found this > [link|https://community.cloudera.com/t5/Support-Questions/Error-trying-to-balance-disks-on-node/m-p/59989#M54850] > but there's no resolve solution , can somebody help me? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16098) ERROR tools.DiskBalancerCLI: java.lang.IllegalArgumentException
[ https://issues.apache.org/jira/browse/HDFS-16098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372618#comment-17372618 ] wangyanfu commented on HDFS-16098: -- this is the full log !image-2021-07-01-18-34-54-905.png! > ERROR tools.DiskBalancerCLI: java.lang.IllegalArgumentException > --- > > Key: HDFS-16098 > URL: https://issues.apache.org/jira/browse/HDFS-16098 > Project: Hadoop HDFS > Issue Type: Bug > Components: diskbalancer >Affects Versions: 2.6.0 > Environment: VERSION info: > Hadoop 2.6.0-cdh5.14.4 >Reporter: wangyanfu >Priority: Blocker > Labels: diskbalancer > Fix For: 2.6.0 > > Attachments: image-2021-07-01-18-34-54-905.png, on-branch-3.1.jpg > > Original Estimate: 504h > Remaining Estimate: 504h > > when i tried to run > hdfs diskbalancer -plan $(hostname -f) > > > > i get this notice: > 21/06/30 11:30:41 ERROR tools.DiskBalancerCLI: > java.lang.IllegalArgumentException > > then i tried write the real hostname into my command , not work and same > error notice > i also tried use --plan instead of -plan , not work and same error notice > i found this > [link|https://community.cloudera.com/t5/Support-Questions/Error-trying-to-balance-disks-on-node/m-p/59989#M54850] > but there's no resolve solution , can somebody help me? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16103) mvn test failed about hadoop@3.2.1
[
https://issues.apache.org/jira/browse/HDFS-16103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372611#comment-17372611
]
Steve Loughran commented on HDFS-16103:
---
looks like something is very odd with your local filesystem. All tests which
lock down local FS permissions and then try to read/write/delete expecting an
error are failing because the FS operation *Worked*
# what OS are you running on?
# what filesystem contains the hadoop source?
# how much space is there?
# you running these tests as a superuser? that could explain the problem
Unless you can find similar error messages in other JIRAs, you are going to
have to debug it yourself. Add the project to an IDE, set a breakpoint on the
at test case which fails, and step through to see where things diverged
> mvn test failed about hadoop@3.2.1
> --
>
> Key: HDFS-16103
> URL: https://issues.apache.org/jira/browse/HDFS-16103
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.2.1
>Reporter: shixijun
>Priority: Major
>
> {panel:title=mvn test failed about hadoop@3.2.1}
> mvn test failed
> {panel}
> [root@localhost spack-src]# mvn -version
> Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
> Maven home:
> /home/all_spack_env/spack/opt/spack/linux-centos8-aarch64/gcc-8.4.1/maven-3.6.3-fpgpwvz7es5yiaz2tez2pnlilrcatuvg
> Java version: 1.8.0_191, vendor: AdoptOpenJdk, runtime:
> /home/all_spack_env/spack/opt/spack/linux-centos8-aarch64/gcc-8.4.1/openjdk-1.8.0_191-b12-fidptihybskgklbjoo4lagkacm6n6lod/jre
> Default locale: en_US, platform encoding: ANSI_X3.4-1968
> OS name: "linux", version: "4.18.0-80.el8.aarch64", arch: "aarch64", family:
> "unix"
> [root@localhost spack-src]# java -version
> openjdk version "1.8.0_191"
> OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_191-b12)
> OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.191-b12, mixed mode)
> [root@localhost spack-src]# mvn test
> ……
> [INFO] Running org.apache.hadoop.tools.TestCommandShell
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.111
> s - in org.apache.hadoop.tools.TestCommandShell
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR]
> TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testGlobStatusThrowsExceptionForUnreadableDir:643
> Should throw IOException
> [ERROR]
> TestFSMainOperationsLocalFileSystem>FSMainOperationsBaseTest.testListStatusThrowsExceptionForUnreadableDir:288
> Should throw IOException
> [ERROR]
> TestFileUtil.testFailFullyDelete:446->validateAndSetWritablePermissions:422
> The directory xSubDir *should* not have been deleted. expected: but
> was:
> [ERROR]
> TestFileUtil.testFailFullyDeleteContents:525->validateAndSetWritablePermissions:422
> The directory xSubDir *should* not have been deleted. expected: but
> was:
> [ERROR] TestFileUtil.testGetDU:571
> [ERROR] TestFsShellCopy.testPutSrcDirNoPerm:627->shellRun:80 expected:<1>
> but was:<0>
> [ERROR] TestFsShellCopy.testPutSrcFileNoPerm:652->shellRun:80 expected:<1>
> but was:<0>
> [ERROR] TestLocalDirAllocator.test0:140->validateTempDirCreation:109
> Checking for build/test/temp/RELATIVE1 in
> build/test/temp/RELATIVE0/block995011826146306285.tmp - FAILED!
> [ERROR] TestLocalDirAllocator.test0:140->validateTempDirCreation:109
> Checking for
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE1
> in
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE0/block792666236482175348.tmp
> - FAILED!
> [ERROR] TestLocalDirAllocator.test0:141->validateTempDirCreation:109
> Checking for
> file:/home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED1
> in
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/QUALIFIED0/block5124616846677903649.tmp
> - FAILED!
> [ERROR]
> TestLocalDirAllocator.testROBufferDirAndRWBufferDir:162->validateTempDirCreation:109
> Checking for build/test/temp/RELATIVE2 in
> build/test/temp/RELATIVE1/block1176062344115776027.tmp - FAILED!
> [ERROR]
> TestLocalDirAllocator.testROBufferDirAndRWBufferDir:163->validateTempDirCreation:109
> Checking for
> /home/all_spack_env/spack_stage/root/spack-stage-hadoop-3.2.1-xvpobktnlicqhfzwbkriy4cick5tpsab/spack-src/hadoop-common-project/hadoop-common/build/test/temp/ABSOLUTE2
> in
>
[jira] [Commented] (HDFS-16056) Can't start by resouceManager
[ https://issues.apache.org/jira/browse/HDFS-16056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372508#comment-17372508 ] chencf commented on HDFS-16056: --- 直接不通过start-all.sh 通过 yarn-daemon.sh start resourcemanager 是同样的报错吗？ > Can't start by resouceManager > - > > Key: HDFS-16056 > URL: https://issues.apache.org/jira/browse/HDFS-16056 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.3.0 > Environment: windows 10 >Reporter: JYXL >Priority: Major > > When I use start-all.cmd, it can start namenode, datanode, nodemanager > successfully, but cannot start resoucemanager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15650) Make the socket timeout for computing checksum of striped blocks configurable
[ https://issues.apache.org/jira/browse/HDFS-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372509#comment-17372509 ] Wei-Chiu Chuang commented on HDFS-15650: Ok I honestly don't remember what held me back but I am okay to use a ec specific socket timeout given the current EC architecture. > Make the socket timeout for computing checksum of striped blocks configurable > - > > Key: HDFS-15650 > URL: https://issues.apache.org/jira/browse/HDFS-15650 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ec, erasure-coding >Reporter: Yushi Hayasaka >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Regarding the DataNode tries to get the checksum of EC internal blocks from > another DataNode for computing the checksum of striped blocks, the timeout is > hard-coded now, but it should be configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16101) Remove unuse variable and IoException in ProvidedStorageMap
[ https://issues.apache.org/jira/browse/HDFS-16101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372478#comment-17372478 ] Ayush Saxena commented on HDFS-16101: - Something has happened to the build. I tried to fix one, but it again failed. [~aajisaka] any pointers, did we change anything recently? > Remove unuse variable and IoException in ProvidedStorageMap > --- > > Key: HDFS-16101 > URL: https://issues.apache.org/jira/browse/HDFS-16101 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: lei w >Assignee: lei w >Priority: Minor > Attachments: HDFS-16101.001.patch > > > Remove unuse variable and IoException in ProvidedStorageMap -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16105) Edit log corruption due to mismatch between fileId and path
[
https://issues.apache.org/jira/browse/HDFS-16105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei resolved HDFS-16105.
Resolution: Invalid
> Edit log corruption due to mismatch between fileId and path
> ---
>
> Key: HDFS-16105
> URL: https://issues.apache.org/jira/browse/HDFS-16105
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.2.2, 3.3.1
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We do stress testing in FUSE with HDFS, Standby Namenode crashes.
> The log is following
> {quote}
> 2021-06-25 17:13:02,953 ERROR
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception
> on operation AddBlockOp [path=/xxx/fiotest_write.354.46,
> penultimateBlock=xxx, lastBlock=xxx, numOfBytes=0}, RpcClientId=,
> RpcCallId=-2]
> java.io.FileNotFoundException: File /xxx/fiotest_write.354.46 does not exist.
> {quote}
> The following steps can reproduce it (Illegal writes).
> 1 create file A (fileId X) (1st block is being written)
> 2 rename file A to file B(stile fileId X)
> 3 continue to write file A(using above the same outputstream, 2nd block need
> to be written)
> 4 standby namenode load the above edits and would crash
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16086) Add volume information to datanode log for tracing
[ https://issues.apache.org/jira/browse/HDFS-16086?focusedWorklogId=617425=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-617425 ] ASF GitHub Bot logged work on HDFS-16086: - Author: ASF GitHub Bot Created on: 01/Jul/21 06:23 Start Date: 01/Jul/21 06:23 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3136: URL: https://github.com/apache/hadoop/pull/3136#issuecomment-871958420 > The checkstyle warnings are old, unrelated. Mergeing the PR. Thanks @jojochuang for the merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 617425) Time Spent: 3h 20m (was: 3h 10m) > Add volume information to datanode log for tracing > -- > > Key: HDFS-16086 > URL: https://issues.apache.org/jira/browse/HDFS-16086 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: CreatingRbw.jpg, Received.jpg > > Time Spent: 3h 20m > Remaining Estimate: 0h > > To keep track of the block in volume, we can add the volume information to > the datanode log. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
