[jira] [Resolved] (HDFS-16519) Add throttler to EC reconstruction
[ https://issues.apache.org/jira/browse/HDFS-16519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma resolved HDFS-16519. - Fix Version/s: 3.4.0 3.3.4 Resolution: Fixed > Add throttler to EC reconstruction > -- > > Key: HDFS-16519 > URL: https://issues.apache.org/jira/browse/HDFS-16519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.4 > > Time Spent: 3h > Remaining Estimate: 0h > > HDFS already have throttlers for data transfer(replication) and balancer, the > throttlers reduce the impact of these background procedures to user > read/write. > We should add a throttler to EC background reconstruction too. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16519) Add throttler to EC reconstruction
[ https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=761225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761225 ] ASF GitHub Bot logged work on HDFS-16519: - Author: ASF GitHub Bot Created on: 23/Apr/22 03:45 Start Date: 23/Apr/22 03:45 Worklog Time Spent: 10m Work Description: tasanuma commented on PR #4101: URL: https://github.com/apache/hadoop/pull/4101#issuecomment-1107352301 Merged. Thanks for your contribution, @cndaimin. Thanks for your review, @tomscut. Issue Time Tracking --- Worklog Id: (was: 761225) Time Spent: 3h (was: 2h 50m) > Add throttler to EC reconstruction > -- > > Key: HDFS-16519 > URL: https://issues.apache.org/jira/browse/HDFS-16519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > HDFS already have throttlers for data transfer(replication) and balancer, the > throttlers reduce the impact of these background procedures to user > read/write. > We should add a throttler to EC background reconstruction too. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16519) Add throttler to EC reconstruction
[ https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=761224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761224 ] ASF GitHub Bot logged work on HDFS-16519: - Author: ASF GitHub Bot Created on: 23/Apr/22 03:45 Start Date: 23/Apr/22 03:45 Worklog Time Spent: 10m Work Description: tasanuma merged PR #4101: URL: https://github.com/apache/hadoop/pull/4101 Issue Time Tracking --- Worklog Id: (was: 761224) Time Spent: 2h 50m (was: 2h 40m) > Add throttler to EC reconstruction > -- > > Key: HDFS-16519 > URL: https://issues.apache.org/jira/browse/HDFS-16519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > HDFS already have throttlers for data transfer(replication) and balancer, the > throttlers reduce the impact of these background procedures to user > read/write. > We should add a throttler to EC background reconstruction too. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16519) Add throttler to EC reconstruction
[ https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=761221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761221 ] ASF GitHub Bot logged work on HDFS-16519: - Author: ASF GitHub Bot Created on: 23/Apr/22 03:26 Start Date: 23/Apr/22 03:26 Worklog Time Spent: 10m Work Description: tasanuma commented on PR #4101: URL: https://github.com/apache/hadoop/pull/4101#issuecomment-1107346064 @cndaimin Thanks for sharing the information about your practice, and updating PR. @tomscut Thanks for your review. Now I'm +1 on this PR. Issue Time Tracking --- Worklog Id: (was: 761221) Time Spent: 2h 40m (was: 2.5h) > Add throttler to EC reconstruction > -- > > Key: HDFS-16519 > URL: https://issues.apache.org/jira/browse/HDFS-16519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > HDFS already have throttlers for data transfer(replication) and balancer, the > throttlers reduce the impact of these background procedures to user > read/write. > We should add a throttler to EC background reconstruction too. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16552) Fix NPE for TestBlockManager
[
https://issues.apache.org/jira/browse/HDFS-16552?focusedWorklogId=761220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761220
]
ASF GitHub Bot logged work on HDFS-16552:
-
Author: ASF GitHub Bot
Created on: 23/Apr/22 03:20
Start Date: 23/Apr/22 03:20
Worklog Time Spent: 10m
Work Description: tomscut commented on PR #4210:
URL: https://github.com/apache/hadoop/pull/4210#issuecomment-110737
Thank you @tasanuma .
Issue Time Tracking
---
Worklog Id: (was: 761220)
Time Spent: 1h 40m (was: 1.5h)
> Fix NPE for TestBlockManager
>
>
> Key: HDFS-16552
> URL: https://issues.apache.org/jira/browse/HDFS-16552
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> There is a NPE in BlockManager when run
> TestBlockManager#testSkipReconstructionWithManyBusyNodes2. Because
> NameNodeMetrics is not initialized in this unit test.
>
> Related ci link, see
> [this|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4209/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt].
> {code:java}
> [ERROR] Tests run: 34, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 30.088 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
> [ERROR]
> testSkipReconstructionWithManyBusyNodes2(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager)
> Time elapsed: 2.783 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.scheduleReconstruction(BlockManager.java:2171)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.testSkipReconstructionWithManyBusyNodes2(TestBlockManager.java:947)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16552) Fix NPE for TestBlockManager
[
https://issues.apache.org/jira/browse/HDFS-16552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma resolved HDFS-16552.
-
Fix Version/s: 3.4.0
3.2.4
3.3.4
Resolution: Fixed
> Fix NPE for TestBlockManager
>
>
> Key: HDFS-16552
> URL: https://issues.apache.org/jira/browse/HDFS-16552
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> There is a NPE in BlockManager when run
> TestBlockManager#testSkipReconstructionWithManyBusyNodes2. Because
> NameNodeMetrics is not initialized in this unit test.
>
> Related ci link, see
> [this|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4209/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt].
> {code:java}
> [ERROR] Tests run: 34, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 30.088 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
> [ERROR]
> testSkipReconstructionWithManyBusyNodes2(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager)
> Time elapsed: 2.783 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.scheduleReconstruction(BlockManager.java:2171)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.testSkipReconstructionWithManyBusyNodes2(TestBlockManager.java:947)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16552) Fix NPE for TestBlockManager
[
https://issues.apache.org/jira/browse/HDFS-16552?focusedWorklogId=761218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761218
]
ASF GitHub Bot logged work on HDFS-16552:
-
Author: ASF GitHub Bot
Created on: 23/Apr/22 03:14
Start Date: 23/Apr/22 03:14
Worklog Time Spent: 10m
Work Description: tasanuma merged PR #4210:
URL: https://github.com/apache/hadoop/pull/4210
Issue Time Tracking
---
Worklog Id: (was: 761218)
Time Spent: 1h 20m (was: 1h 10m)
> Fix NPE for TestBlockManager
>
>
> Key: HDFS-16552
> URL: https://issues.apache.org/jira/browse/HDFS-16552
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> There is a NPE in BlockManager when run
> TestBlockManager#testSkipReconstructionWithManyBusyNodes2. Because
> NameNodeMetrics is not initialized in this unit test.
>
> Related ci link, see
> [this|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4209/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt].
> {code:java}
> [ERROR] Tests run: 34, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 30.088 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
> [ERROR]
> testSkipReconstructionWithManyBusyNodes2(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager)
> Time elapsed: 2.783 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.scheduleReconstruction(BlockManager.java:2171)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.testSkipReconstructionWithManyBusyNodes2(TestBlockManager.java:947)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16552) Fix NPE for TestBlockManager
[
https://issues.apache.org/jira/browse/HDFS-16552?focusedWorklogId=761219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761219
]
ASF GitHub Bot logged work on HDFS-16552:
-
Author: ASF GitHub Bot
Created on: 23/Apr/22 03:14
Start Date: 23/Apr/22 03:14
Worklog Time Spent: 10m
Work Description: tasanuma commented on PR #4210:
URL: https://github.com/apache/hadoop/pull/4210#issuecomment-1107342992
Thanks for fixing it, @tomscut!
Issue Time Tracking
---
Worklog Id: (was: 761219)
Time Spent: 1.5h (was: 1h 20m)
> Fix NPE for TestBlockManager
>
>
> Key: HDFS-16552
> URL: https://issues.apache.org/jira/browse/HDFS-16552
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> There is a NPE in BlockManager when run
> TestBlockManager#testSkipReconstructionWithManyBusyNodes2. Because
> NameNodeMetrics is not initialized in this unit test.
>
> Related ci link, see
> [this|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4209/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt].
> {code:java}
> [ERROR] Tests run: 34, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 30.088 s <<< FAILURE! - in
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
> [ERROR]
> testSkipReconstructionWithManyBusyNodes2(org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager)
> Time elapsed: 2.783 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.scheduleReconstruction(BlockManager.java:2171)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager.testSkipReconstructionWithManyBusyNodes2(TestBlockManager.java:947)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes
[
https://issues.apache.org/jira/browse/HDFS-16540?focusedWorklogId=761028=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761028
]
ASF GitHub Bot logged work on HDFS-16540:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 19:04
Start Date: 22/Apr/22 19:04
Worklog Time Spent: 10m
Work Description: huaxiangsun commented on code in PR #4170:
URL: https://github.com/apache/hadoop/pull/4170#discussion_r856498429
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeManager.java:
##
@@ -138,6 +138,35 @@ public void testNumVersionsCorrectAfterReregister()
mapToCheck.get("version1").intValue(), 1);
}
+ /**
+ * This test checks that if a node is re-registered with a different ip, its
+ * host2DatanodeMap is correctly updated with the new ip.
+ */
+ @Test
Review Comment:
Yeah, the NetworkTopology map is another map. It is quite complicated to add
a unittest for it. I am going to leave to hdfs experts to decide if a unittest
is needed.
Issue Time Tracking
---
Worklog Id: (was: 761028)
Time Spent: 2h 40m (was: 2.5h)
> Data locality is lost when DataNode pod restarts in kubernetes
> ---
>
> Key: HDFS-16540
> URL: https://issues.apache.org/jira/browse/HDFS-16540
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.2
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod
> restarts, we found that data locality is lost after we do a major compaction
> of hbase regions. After some debugging, we found that upon pod restarts, its
> ip changes. In DatanodeManager, maps like networktopology are updated with
> the new info. host2DatanodeMap is not updated accordingly. When hdfs client
> with the new ip tries to find a local DataNode, it fails.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes
[
https://issues.apache.org/jira/browse/HDFS-16540?focusedWorklogId=761024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761024
]
ASF GitHub Bot logged work on HDFS-16540:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 18:56
Start Date: 22/Apr/22 18:56
Worklog Time Spent: 10m
Work Description: huaxiangsun commented on code in PR #4170:
URL: https://github.com/apache/hadoop/pull/4170#discussion_r856493693
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##
@@ -1189,16 +1190,25 @@ public void registerDatanode(DatanodeRegistration
nodeReg)
nodes with its data cleared (or user can just remove the StorageID
value in "VERSION" file under the data directory of the datanode,
but this is might not work if VERSION file format has changed
- */
+ */
+ // Check if nodeS's host information is same as nodeReg's, if not,
+ // it needs to update host2DatanodeMap accordringly.
+ updateHost2DatanodeMap =
!nodeS.getXferAddr().equals(nodeReg.getXferAddr());
+
NameNode.stateChangeLog.info("BLOCK* registerDatanode: " + nodeS
+ " is replaced by " + nodeReg + " with the same storageID "
- + nodeReg.getDatanodeUuid());
+ + nodeReg.getDatanodeUuid() + ", updateHost2DatanodeMap: " +
updateHost2DatanodeMap);
Review Comment:
I am going to upload the patch which does not log updateHost2DatanodeMap.
Issue Time Tracking
---
Worklog Id: (was: 761024)
Time Spent: 2.5h (was: 2h 20m)
> Data locality is lost when DataNode pod restarts in kubernetes
> ---
>
> Key: HDFS-16540
> URL: https://issues.apache.org/jira/browse/HDFS-16540
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.2
>Reporter: Huaxiang Sun
>Assignee: Huaxiang Sun
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod
> restarts, we found that data locality is lost after we do a major compaction
> of hbase regions. After some debugging, we found that upon pod restarts, its
> ip changes. In DatanodeManager, maps like networktopology are updated with
> the new info. host2DatanodeMap is not updated accordingly. When hdfs client
> with the new ip tries to find a local DataNode, it fails.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes
[ https://issues.apache.org/jira/browse/HDFS-16540?focusedWorklogId=761022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761022 ] ASF GitHub Bot logged work on HDFS-16540: - Author: ASF GitHub Bot Created on: 22/Apr/22 18:55 Start Date: 22/Apr/22 18:55 Worklog Time Spent: 10m Work Description: huaxiangsun commented on code in PR #4170: URL: https://github.com/apache/hadoop/pull/4170#discussion_r856493317 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java: ## @@ -1189,16 +1190,25 @@ public void registerDatanode(DatanodeRegistration nodeReg) nodes with its data cleared (or user can just remove the StorageID value in "VERSION" file under the data directory of the datanode, but this is might not work if VERSION file format has changed - */ + */ + // Check if nodeS's host information is same as nodeReg's, if not, + // it needs to update host2DatanodeMap accordringly. + updateHost2DatanodeMap = !nodeS.getXferAddr().equals(nodeReg.getXferAddr()); Review Comment: Yeah, since nodeS is not null, nodeS.getXferAddr() cannot be null. Issue Time Tracking --- Worklog Id: (was: 761022) Time Spent: 2h 20m (was: 2h 10m) > Data locality is lost when DataNode pod restarts in kubernetes > --- > > Key: HDFS-16540 > URL: https://issues.apache.org/jira/browse/HDFS-16540 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.2 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod > restarts, we found that data locality is lost after we do a major compaction > of hbase regions. After some debugging, we found that upon pod restarts, its > ip changes. In DatanodeManager, maps like networktopology are updated with > the new info. host2DatanodeMap is not updated accordingly. When hdfs client > with the new ip tries to find a local DataNode, it fails. > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16519) Add throttler to EC reconstruction
[
https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=761012=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-761012
]
ASF GitHub Bot logged work on HDFS-16519:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 18:31
Start Date: 22/Apr/22 18:31
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4101:
URL: https://github.com/apache/hadoop/pull/4101#issuecomment-1106760921
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 56s | | trunk passed |
| +1 :green_heart: | compile | 1m 50s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 1m 34s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 15s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 36s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 245m 36s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 12s | | The patch does not
generate ASF License warnings. |
| | | 366m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4101/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4101 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux d92eb4d3996f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e2c3cb0347eb8de268a8f25f7779e12ba5f5f0c1 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4101/7/testReport/ |
| Max. process+thread count | 3164 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-16556) Fix typos in distcp
[ https://issues.apache.org/jira/browse/HDFS-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17526608#comment-17526608 ] Dinesh Chitlangia commented on HDFS-16556: -- Thanks [~philipse] for the improvment. > Fix typos in distcp > --- > > Key: HDFS-16556 > URL: https://issues.apache.org/jira/browse/HDFS-16556 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.2 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.3 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Fix typos in distcp -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16556) Fix typos in distcp
[ https://issues.apache.org/jira/browse/HDFS-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia resolved HDFS-16556. -- Fix Version/s: 3.3.3 Resolution: Fixed > Fix typos in distcp > --- > > Key: HDFS-16556 > URL: https://issues.apache.org/jira/browse/HDFS-16556 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.2 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Fix For: 3.3.3 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Fix typos in distcp -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16556) Fix typos in distcp
[ https://issues.apache.org/jira/browse/HDFS-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDFS-16556: - Summary: Fix typos in distcp (was: Fixtypos in distcp) > Fix typos in distcp > --- > > Key: HDFS-16556 > URL: https://issues.apache.org/jira/browse/HDFS-16556 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.2 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Fix typos in distcp -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16556) Fixtypos in distcp
[ https://issues.apache.org/jira/browse/HDFS-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HDFS-16556: - Description: Fix typos in distcp (was: Fixtypos in distcp) > Fixtypos in distcp > -- > > Key: HDFS-16556 > URL: https://issues.apache.org/jira/browse/HDFS-16556 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.2 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Fix typos in distcp -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16556) Fix typos in distcp
[ https://issues.apache.org/jira/browse/HDFS-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia reassigned HDFS-16556: Assignee: guophilipse > Fix typos in distcp > --- > > Key: HDFS-16556 > URL: https://issues.apache.org/jira/browse/HDFS-16556 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.2 >Reporter: guophilipse >Assignee: guophilipse >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Fix typos in distcp -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16556) Fixtypos in distcp
[ https://issues.apache.org/jira/browse/HDFS-16556?focusedWorklogId=760999=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760999 ] ASF GitHub Bot logged work on HDFS-16556: - Author: ASF GitHub Bot Created on: 22/Apr/22 18:01 Start Date: 22/Apr/22 18:01 Worklog Time Spent: 10m Work Description: dineshchitlangia merged PR #4217: URL: https://github.com/apache/hadoop/pull/4217 Issue Time Tracking --- Worklog Id: (was: 760999) Time Spent: 0.5h (was: 20m) > Fixtypos in distcp > -- > > Key: HDFS-16556 > URL: https://issues.apache.org/jira/browse/HDFS-16556 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 3.3.2 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Fixtypos in distcp -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-1179) Implement a file change log
[ https://issues.apache.org/jira/browse/HDFS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17526579#comment-17526579 ] HAO ZIQI commented on HDFS-1179: Hello everyone, how can I find the use case of HDFS notify function in Flink on this page? emmm. I'm still a rookie, I hope to get your help, thank you all > Implement a file change log > --- > > Key: HDFS-1179 > URL: https://issues.apache.org/jira/browse/HDFS-1179 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Dhruba Borthakur >Assignee: Dhruba Borthakur >Priority: Major > > The File Change Log (FCL) is a construct that records all metadata changes in > a file system (http://en.wikipedia.org/wiki/File_change_log). This JIRA is to > discuss various high-level approaches to implementing such a FCL for HDFS. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16557) BootstrapStandby failed because of checking gap for inprogress EditLogInputStream
[
https://issues.apache.org/jira/browse/HDFS-16557?focusedWorklogId=760879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760879
]
ASF GitHub Bot logged work on HDFS-16557:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 15:33
Start Date: 22/Apr/22 15:33
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4219:
URL: https://github.com/apache/hadoop/pull/4219#issuecomment-1106630776
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 52s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 248m 17s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 371m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4219/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4219 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 762330a41f0c 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / abd27315b22d09607203cbe3ff1fbb9ed8b47ca2 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4219/1/testReport/ |
| Max. process+thread count | 3553 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4219/1/console |
| versions |
[jira] [Work logged] (HDFS-16519) Add throttler to EC reconstruction
[
https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=760855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760855
]
ASF GitHub Bot logged work on HDFS-16519:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 14:27
Start Date: 22/Apr/22 14:27
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4101:
URL: https://github.com/apache/hadoop/pull/4101#issuecomment-1106572107
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 247m 21s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4101/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not
generate ASF License warnings. |
| | | 369m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4101/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4101 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 37ca91675a7d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bfb190f4a52bf7ea893821abfd388e99d9f0da5a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
[jira] [Work logged] (HDFS-16552) Fix NPE for TestBlockManager
[
https://issues.apache.org/jira/browse/HDFS-16552?focusedWorklogId=760815=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760815
]
ASF GitHub Bot logged work on HDFS-16552:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 13:07
Start Date: 22/Apr/22 13:07
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4210:
URL: https://github.com/apache/hadoop/pull/4210#issuecomment-1106497235
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 20s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 54s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 359m 6s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 477m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4210/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4210 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux e4f25f7f85a6 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c721b5c626423e584ea529c24b20b372b7c74289 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4210/2/testReport/ |
| Max. process+thread count | 1998 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4210/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This
[jira] [Work logged] (HDFS-16554) Remove unused configuration dfs.namenode.block.deletion.increment.
[
https://issues.apache.org/jira/browse/HDFS-16554?focusedWorklogId=760809=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760809
]
ASF GitHub Bot logged work on HDFS-16554:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 13:02
Start Date: 22/Apr/22 13:02
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4213:
URL: https://github.com/apache/hadoop/pull/4213#issuecomment-1106492118
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 34s | | trunk passed |
| +1 :green_heart: | compile | 1m 50s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 54s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 30s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 456m 58s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4213/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 16s | | The patch does not
generate ASF License warnings. |
| | | 575m 50s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4213/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4213 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux bf0a5d99b861 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2cc6d0d4e3765aaf9f4115e169e6544a7ef28609 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
[jira] [Work logged] (HDFS-16519) Add throttler to EC reconstruction
[ https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=760787=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760787 ] ASF GitHub Bot logged work on HDFS-16519: - Author: ASF GitHub Bot Created on: 22/Apr/22 12:25 Start Date: 22/Apr/22 12:25 Worklog Time Spent: 10m Work Description: cndaimin commented on PR #4101: URL: https://github.com/apache/hadoop/pull/4101#issuecomment-1106462870 @tomscut hdfs-default.xml updated. Thanks for your reminding! Issue Time Tracking --- Worklog Id: (was: 760787) Time Spent: 2h 10m (was: 2h) > Add throttler to EC reconstruction > -- > > Key: HDFS-16519 > URL: https://issues.apache.org/jira/browse/HDFS-16519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > HDFS already have throttlers for data transfer(replication) and balancer, the > throttlers reduce the impact of these background procedures to user > read/write. > We should add a throttler to EC background reconstruction too. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16545) Provide option to balance rack level in Balancer
[
https://issues.apache.org/jira/browse/HDFS-16545?focusedWorklogId=760761=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760761
]
ASF GitHub Bot logged work on HDFS-16545:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 11:48
Start Date: 22/Apr/22 11:48
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4207:
URL: https://github.com/apache/hadoop/pull/4207#issuecomment-1106435853
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 9s | | trunk passed |
| +1 :green_heart: | compile | 1m 45s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4207/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 164 unchanged
- 0 fixed = 167 total (was 164) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 35s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 463m 59s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4207/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 11s | | The patch does not
generate ASF License warnings. |
| | | 592m 41s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4207/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4207 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux 4d8de28815f1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d4509eca2343f584637ec27bf97c6e6d6c229d2e |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
|
[jira] [Work logged] (HDFS-16545) Provide option to balance rack level in Balancer
[ https://issues.apache.org/jira/browse/HDFS-16545?focusedWorklogId=760745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760745 ] ASF GitHub Bot logged work on HDFS-16545: - Author: ASF GitHub Bot Created on: 22/Apr/22 11:24 Start Date: 22/Apr/22 11:24 Worklog Time Spent: 10m Work Description: prasad-acit commented on PR #4207: URL: https://github.com/apache/hadoop/pull/4207#issuecomment-1106418525 Failed UT is not related. Checkstyle 1 not related & 2 related to param count which already exceeded. @hemanthboyina @surendralilhore Can you please review the PR? Issue Time Tracking --- Worklog Id: (was: 760745) Time Spent: 40m (was: 0.5h) > Provide option to balance rack level in Balancer > > > Key: HDFS-16545 > URL: https://issues.apache.org/jira/browse/HDFS-16545 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Minor > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Currently Balancer tool run on entire cluster and balance across the racks. > In we need to balance within a rack, then need to provide an option to > support the rack level balancing. > [~surendralilhore] [~hemanthboyina] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16553) Fix checkstyle for the length of BlockManager construction method over limit.
[
https://issues.apache.org/jira/browse/HDFS-16553?focusedWorklogId=760719=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760719
]
ASF GitHub Bot logged work on HDFS-16553:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 10:03
Start Date: 22/Apr/22 10:03
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4211:
URL: https://github.com/apache/hadoop/pull/4211#issuecomment-1106320862
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 29s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 1s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 80 unchanged - 4
fixed = 80 total (was 84) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 350m 4s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4211/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 484m 49s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4211/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4211 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 562e9c09ef3b 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 68182bf713e1ba5f73d6e673bd5e2ec65cb1c8cc |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Updated] (HDFS-16453) okhttp vulnerable library update
[
https://issues.apache.org/jira/browse/HDFS-16453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-16453:
--
Issue Type: Bug (was: Wish)
> okhttp vulnerable library update
>
>
> Key: HDFS-16453
> URL: https://issues.apache.org/jira/browse/HDFS-16453
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.3.1
>Reporter: Ivan Viaznikov
>Priority: Major
>
> {{org.apache.hadoop:hadoop-hdfs-client}} comes with
> {{com.squareup.okhttp:okhttp:2.7.5}} as a dependency, which is vulnerable to
> an information disclosure issue due to how the contents of sensitive headers,
> such as the {{Authorization}} header, can be logged when an
> {{IllegalArgumentException}} is thrown.
> This issue could allow an attacker or malicious user who has access to the
> logs to obtain the sensitive contents of the affected headers which could
> facilitate further attacks.
> Fixed in {{5.0.0-alpha3}} by
> [this|https://github.com/square/okhttp/commit/dcc6483b7dc6d9c0b8e03ff7c30c13f3c75264a5]
> commit. The fix was cherry-picked and backported into {{4.9.2}} with
> [this|https://github.com/square/okhttp/commit/1fd7c0afdc2cee9ba982b07d49662af7f60e1518]
> commit.
> Requesting you to clarify if this dependency will be updated to a fixed
> version in the following releases
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16555) rename mixlead method name in DistCpOptions
[ https://issues.apache.org/jira/browse/HDFS-16555?focusedWorklogId=760713=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760713 ] ASF GitHub Bot logged work on HDFS-16555: - Author: ASF GitHub Bot Created on: 22/Apr/22 09:43 Start Date: 22/Apr/22 09:43 Worklog Time Spent: 10m Work Description: steveloughran commented on PR #4216: URL: https://github.com/apache/hadoop/pull/4216#issuecomment-1106271801 api is tagged as @InterfaceAudience.Public;; we can't remove the existing method. you could add a new one with the better name and mark the old one as deprecated Issue Time Tracking --- Worklog Id: (was: 760713) Time Spent: 0.5h (was: 20m) > rename mixlead method name in DistCpOptions > --- > > Key: HDFS-16555 > URL: https://issues.apache.org/jira/browse/HDFS-16555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: distcp >Affects Versions: 3.3.2 >Reporter: guophilipse >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently method in DistCpOptions withCRC was used as the following > ``` > withCRC(true) means check without crc > withCRC(false) means check with crc > ``` > which mislead the developer when we pass the paramter, we can rename the > method to clear that.after that it should be: > ``` > withSkipCRC(true) means check without crc > withSkipCRC(false) means check with crc > ``` > so it will be more understandable. > > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking gap for inprogress EditLogInputStream
[
https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-16557:
---
Description:
The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily
HdfsServerConstants.INVALID_TXID. We can determine its status directly by
EditLogInputStream#isInProgress.
For example, set {color:#ff}{{dfs.ha.tail-edits.in-progress=true}}{color}.
Then bootstrapStandby, the EditLogInputStream of inProgress is misjudged,
resulting in a gap check failure, which causes bootstrapStandby to fail.
hdfs namenode -bootstrapStandby
!image-2022-04-22-17-17-32-487.png|width=766,height=161!
!image-2022-04-22-17-17-14-577.png|width=598,height=187!
was:
The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily
HdfsServerConstants.INVALID_TXID. We can determine its status directly by
EditLogInputStream#isInProgress.
For example, set {color:#FF}{{dfs.ha.tail-edits.in-progress=true}}{color}.
Then bootstrapStandby, the EditLogInputStream of inProgress is misjudged,
resulting in a gap check failure, which causes bootstrapStandby to fail.
!image-2022-04-22-17-17-32-487.png|width=766,height=161!
!image-2022-04-22-17-17-14-577.png|width=598,height=187!
> BootstrapStandby failed because of checking gap for inprogress
> EditLogInputStream
> -
>
> Key: HDFS-16557
> URL: https://issues.apache.org/jira/browse/HDFS-16557
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-04-22-17-17-14-577.png,
> image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png,
> image-2022-04-22-17-17-32-487.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily
> HdfsServerConstants.INVALID_TXID. We can determine its status directly by
> EditLogInputStream#isInProgress.
> For example, set
> {color:#ff}{{dfs.ha.tail-edits.in-progress=true}}{color}. Then
> bootstrapStandby, the EditLogInputStream of inProgress is misjudged,
> resulting in a gap check failure, which causes bootstrapStandby to fail.
> hdfs namenode -bootstrapStandby
> !image-2022-04-22-17-17-32-487.png|width=766,height=161!
> !image-2022-04-22-17-17-14-577.png|width=598,height=187!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking gap for inprogress EditLogInputStream
[
https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-16557:
---
Description:
The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily
HdfsServerConstants.INVALID_TXID. We can determine its status directly by
EditLogInputStream#isInProgress.
For example, set {color:#FF}{{dfs.ha.tail-edits.in-progress=true}}{color}.
Then bootstrapStandby, the EditLogInputStream of inProgress is misjudged,
resulting in a gap check failure, which causes bootstrapStandby to fail.
!image-2022-04-22-17-17-32-487.png|width=766,height=161!
!image-2022-04-22-17-17-14-577.png|width=598,height=187!
was:
The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily
HdfsServerConstants.INVALID_TXID. We can determine its status directly by
EditLogInputStream#isInProgress.
For example, when bootstrapStandby, the EditLogInputStream of inProgress is
misjudged, resulting in a gap check failure, which causes bootstrapStandby to
fail.
!image-2022-04-22-17-17-32-487.png|width=766,height=161!
!image-2022-04-22-17-17-14-577.png|width=598,height=187!
> BootstrapStandby failed because of checking gap for inprogress
> EditLogInputStream
> -
>
> Key: HDFS-16557
> URL: https://issues.apache.org/jira/browse/HDFS-16557
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-04-22-17-17-14-577.png,
> image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png,
> image-2022-04-22-17-17-32-487.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily
> HdfsServerConstants.INVALID_TXID. We can determine its status directly by
> EditLogInputStream#isInProgress.
> For example, set
> {color:#FF}{{dfs.ha.tail-edits.in-progress=true}}{color}. Then
> bootstrapStandby, the EditLogInputStream of inProgress is misjudged,
> resulting in a gap check failure, which causes bootstrapStandby to fail.
> !image-2022-04-22-17-17-32-487.png|width=766,height=161!
> !image-2022-04-22-17-17-14-577.png|width=598,height=187!
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking gap for inprogress EditLogInputStream
[ https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16557: --- Summary: BootstrapStandby failed because of checking gap for inprogress EditLogInputStream (was: BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream) > BootstrapStandby failed because of checking gap for inprogress > EditLogInputStream > - > > Key: HDFS-16557 > URL: https://issues.apache.org/jira/browse/HDFS-16557 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2022-04-22-17-17-14-577.png, > image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png, > image-2022-04-22-17-17-32-487.png > > Time Spent: 10m > Remaining Estimate: 0h > > The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily > HdfsServerConstants.INVALID_TXID. We can determine its status directly by > EditLogInputStream#isInProgress. > For example, when bootstrapStandby, the EditLogInputStream of inProgress is > misjudged, resulting in a gap check failure, which causes bootstrapStandby to > fail. > !image-2022-04-22-17-17-32-487.png|width=766,height=161! > !image-2022-04-22-17-17-14-577.png|width=598,height=187! -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream
[ https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16557: -- Labels: pull-request-available (was: ) > BootstrapStandby failed because of checking Gap for inprogress > EditLogInputStream > - > > Key: HDFS-16557 > URL: https://issues.apache.org/jira/browse/HDFS-16557 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2022-04-22-17-17-14-577.png, > image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png, > image-2022-04-22-17-17-32-487.png > > Time Spent: 10m > Remaining Estimate: 0h > > The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily > HdfsServerConstants.INVALID_TXID. We can determine its status directly by > EditLogInputStream#isInProgress. > For example, when bootstrapStandby, the EditLogInputStream of inProgress is > misjudged, resulting in a gap check failure, which causes bootstrapStandby to > fail. > !image-2022-04-22-17-17-32-487.png|width=766,height=161! > !image-2022-04-22-17-17-14-577.png|width=598,height=187! -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16557) BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream
[ https://issues.apache.org/jira/browse/HDFS-16557?focusedWorklogId=760706=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760706 ] ASF GitHub Bot logged work on HDFS-16557: - Author: ASF GitHub Bot Created on: 22/Apr/22 09:21 Start Date: 22/Apr/22 09:21 Worklog Time Spent: 10m Work Description: tomscut opened a new pull request, #4219: URL: https://github.com/apache/hadoop/pull/4219 JIRA: HDFS-16557. The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily HdfsServerConstants.INVALID_TXID. We can determine its status directly by EditLogInputStream#isInProgress. For example, when bootstrapStandby, the EditLogInputStream of inProgress is misjudged, resulting in a gap check failure, which causes bootstrapStandby to fail.   Issue Time Tracking --- Worklog Id: (was: 760706) Remaining Estimate: 0h Time Spent: 10m > BootstrapStandby failed because of checking Gap for inprogress > EditLogInputStream > - > > Key: HDFS-16557 > URL: https://issues.apache.org/jira/browse/HDFS-16557 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Attachments: image-2022-04-22-17-17-14-577.png, > image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png, > image-2022-04-22-17-17-32-487.png > > Time Spent: 10m > Remaining Estimate: 0h > > The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily > HdfsServerConstants.INVALID_TXID. We can determine its status directly by > EditLogInputStream#isInProgress. > For example, when bootstrapStandby, the EditLogInputStream of inProgress is > misjudged, resulting in a gap check failure, which causes bootstrapStandby to > fail. > !image-2022-04-22-17-17-32-487.png|width=766,height=161! > !image-2022-04-22-17-17-14-577.png|width=598,height=187! -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream
[ https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16557: --- Attachment: image-2022-04-22-17-17-32-487.png > BootstrapStandby failed because of checking Gap for inprogress > EditLogInputStream > - > > Key: HDFS-16557 > URL: https://issues.apache.org/jira/browse/HDFS-16557 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Attachments: image-2022-04-22-17-17-14-577.png, > image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png, > image-2022-04-22-17-17-32-487.png > > > The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily > HdfsServerConstants.INVALID_TXID. We can determine its status directly by > EditLogInputStream#isInProgress. > For example, when bootstrapStandby, the EditLogInputStream of inProgress is > misjudged, resulting in a gap check failure, which causes bootstrapStandby to > fail. > !image-2022-04-22-17-17-23-113.png! > !image-2022-04-22-17-17-14-577.png! -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream
[ https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16557: --- Description: The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily HdfsServerConstants.INVALID_TXID. We can determine its status directly by EditLogInputStream#isInProgress. For example, when bootstrapStandby, the EditLogInputStream of inProgress is misjudged, resulting in a gap check failure, which causes bootstrapStandby to fail. !image-2022-04-22-17-17-23-113.png! !image-2022-04-22-17-17-14-577.png! was: The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily HdfsServerConstants.INVALID_TXID. We can determine its status directly by EditLogInputStream#isInProgress. For example, when bootstrapStandby, the EditLogInputStream of inProgress is misjudged, resulting in a gap check failure, which causes bootstrapStandby to fail. > BootstrapStandby failed because of checking Gap for inprogress > EditLogInputStream > - > > Key: HDFS-16557 > URL: https://issues.apache.org/jira/browse/HDFS-16557 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Attachments: image-2022-04-22-17-17-14-577.png, > image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png, > image-2022-04-22-17-17-32-487.png > > > The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily > HdfsServerConstants.INVALID_TXID. We can determine its status directly by > EditLogInputStream#isInProgress. > For example, when bootstrapStandby, the EditLogInputStream of inProgress is > misjudged, resulting in a gap check failure, which causes bootstrapStandby to > fail. > !image-2022-04-22-17-17-23-113.png! > !image-2022-04-22-17-17-14-577.png! -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream
[ https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16557: --- Attachment: image-2022-04-22-17-17-14-618.png > BootstrapStandby failed because of checking Gap for inprogress > EditLogInputStream > - > > Key: HDFS-16557 > URL: https://issues.apache.org/jira/browse/HDFS-16557 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Attachments: image-2022-04-22-17-17-14-577.png, > image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png, > image-2022-04-22-17-17-32-487.png > > > The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily > HdfsServerConstants.INVALID_TXID. We can determine its status directly by > EditLogInputStream#isInProgress. > For example, when bootstrapStandby, the EditLogInputStream of inProgress is > misjudged, resulting in a gap check failure, which causes bootstrapStandby to > fail. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream
[ https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16557: --- Attachment: image-2022-04-22-17-17-23-113.png > BootstrapStandby failed because of checking Gap for inprogress > EditLogInputStream > - > > Key: HDFS-16557 > URL: https://issues.apache.org/jira/browse/HDFS-16557 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Attachments: image-2022-04-22-17-17-14-577.png, > image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png, > image-2022-04-22-17-17-32-487.png > > > The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily > HdfsServerConstants.INVALID_TXID. We can determine its status directly by > EditLogInputStream#isInProgress. > For example, when bootstrapStandby, the EditLogInputStream of inProgress is > misjudged, resulting in a gap check failure, which causes bootstrapStandby to > fail. > !image-2022-04-22-17-17-23-113.png! > !image-2022-04-22-17-17-14-577.png! -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream
[ https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16557: --- Description: The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily HdfsServerConstants.INVALID_TXID. We can determine its status directly by EditLogInputStream#isInProgress. For example, when bootstrapStandby, the EditLogInputStream of inProgress is misjudged, resulting in a gap check failure, which causes bootstrapStandby to fail. !image-2022-04-22-17-17-32-487.png|width=766,height=161! !image-2022-04-22-17-17-14-577.png|width=598,height=187! was: The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily HdfsServerConstants.INVALID_TXID. We can determine its status directly by EditLogInputStream#isInProgress. For example, when bootstrapStandby, the EditLogInputStream of inProgress is misjudged, resulting in a gap check failure, which causes bootstrapStandby to fail. !image-2022-04-22-17-17-23-113.png! !image-2022-04-22-17-17-14-577.png! > BootstrapStandby failed because of checking Gap for inprogress > EditLogInputStream > - > > Key: HDFS-16557 > URL: https://issues.apache.org/jira/browse/HDFS-16557 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Attachments: image-2022-04-22-17-17-14-577.png, > image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png, > image-2022-04-22-17-17-32-487.png > > > The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily > HdfsServerConstants.INVALID_TXID. We can determine its status directly by > EditLogInputStream#isInProgress. > For example, when bootstrapStandby, the EditLogInputStream of inProgress is > misjudged, resulting in a gap check failure, which causes bootstrapStandby to > fail. > !image-2022-04-22-17-17-32-487.png|width=766,height=161! > !image-2022-04-22-17-17-14-577.png|width=598,height=187! -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16557) BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream
[ https://issues.apache.org/jira/browse/HDFS-16557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tomscut updated HDFS-16557: --- Attachment: image-2022-04-22-17-17-14-577.png > BootstrapStandby failed because of checking Gap for inprogress > EditLogInputStream > - > > Key: HDFS-16557 > URL: https://issues.apache.org/jira/browse/HDFS-16557 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: tomscut >Assignee: tomscut >Priority: Major > Attachments: image-2022-04-22-17-17-14-577.png, > image-2022-04-22-17-17-14-618.png, image-2022-04-22-17-17-23-113.png, > image-2022-04-22-17-17-32-487.png > > > The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily > HdfsServerConstants.INVALID_TXID. We can determine its status directly by > EditLogInputStream#isInProgress. > For example, when bootstrapStandby, the EditLogInputStream of inProgress is > misjudged, resulting in a gap check failure, which causes bootstrapStandby to > fail. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16557) BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream
tomscut created HDFS-16557: -- Summary: BootstrapStandby failed because of checking Gap for inprogress EditLogInputStream Key: HDFS-16557 URL: https://issues.apache.org/jira/browse/HDFS-16557 Project: Hadoop HDFS Issue Type: Bug Reporter: tomscut Assignee: tomscut The lastTxId of an inprogress EditLogInputStream lastTxId isn't necessarily HdfsServerConstants.INVALID_TXID. We can determine its status directly by EditLogInputStream#isInProgress. For example, when bootstrapStandby, the EditLogInputStream of inProgress is misjudged, resulting in a gap check failure, which causes bootstrapStandby to fail. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16510) Fix EC decommission when rack is not enough
[ https://issues.apache.org/jira/browse/HDFS-16510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma resolved HDFS-16510. - Resolution: Duplicate I'm closing this jira as it is superceded by HDFS-16456. Thanks for trying to solve this issue, [~cndaimin]. > Fix EC decommission when rack is not enough > --- > > Key: HDFS-16510 > URL: https://issues.apache.org/jira/browse/HDFS-16510 > Project: Hadoop HDFS > Issue Type: Bug > Components: block placement, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > The decommission always fail when we start decommission multiple nodes on a > cluster whose racks is not enough, a cluster with 6 racks to deploy RS-6-3, > for example. > We find that those decommission nodes cover at least a rack, it's actulaly > like we are decommission one or more racks. And rack decommission is not well > supported currently, especially for cluster whose racks is not enough already. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16519) Add throttler to EC reconstruction
[ https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=760693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760693 ] ASF GitHub Bot logged work on HDFS-16519: - Author: ASF GitHub Bot Created on: 22/Apr/22 08:28 Start Date: 22/Apr/22 08:28 Worklog Time Spent: 10m Work Description: tomscut commented on PR #4101: URL: https://github.com/apache/hadoop/pull/4101#issuecomment-1106186456 Thanks @tasanuma and @cndaimin for your explanation. This makes sense to me. @cndaimin Could you please add this configurations to hdfs-default.xml? Thanks. Issue Time Tracking --- Worklog Id: (was: 760693) Time Spent: 2h (was: 1h 50m) > Add throttler to EC reconstruction > -- > > Key: HDFS-16519 > URL: https://issues.apache.org/jira/browse/HDFS-16519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > HDFS already have throttlers for data transfer(replication) and balancer, the > throttlers reduce the impact of these background procedures to user > read/write. > We should add a throttler to EC background reconstruction too. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16519) Add throttler to EC reconstruction
[ https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=760683=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760683 ] ASF GitHub Bot logged work on HDFS-16519: - Author: ASF GitHub Bot Created on: 22/Apr/22 08:13 Start Date: 22/Apr/22 08:13 Worklog Time Spent: 10m Work Description: cndaimin commented on PR #4101: URL: https://github.com/apache/hadoop/pull/4101#issuecomment-1106154139 @tasanuma Thanks for your review and comments! Yes, we have used the feature in our production cluster for nearly 1 year and it looks good. - `dfs.datanode.data.write.bandwidthPerSec`: Currently we have no limitation on user writes, and when user write bandwidth exceeds the bottleneck of hardware (NIC generally), the write will automatically slow down. - `dfs.datanode.ec.reconstruction.xmits.weight`: We keep it to default value of 0.5 - `dfs.datanode.data.transfer.bandwidthPerSec`: Same as the new added `dfs.datanode.ec.reconstruct.read.bandwidthPerSec`, we set it to half of available network traffic to be sure it will not affect user read/write. Issue Time Tracking --- Worklog Id: (was: 760683) Time Spent: 1h 50m (was: 1h 40m) > Add throttler to EC reconstruction > -- > > Key: HDFS-16519 > URL: https://issues.apache.org/jira/browse/HDFS-16519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > HDFS already have throttlers for data transfer(replication) and balancer, the > throttlers reduce the impact of these background procedures to user > read/write. > We should add a throttler to EC background reconstruction too. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16545) Provide option to balance rack level in Balancer
[
https://issues.apache.org/jira/browse/HDFS-16545?focusedWorklogId=760679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760679
]
ASF GitHub Bot logged work on HDFS-16545:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 07:55
Start Date: 22/Apr/22 07:55
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4207:
URL: https://github.com/apache/hadoop/pull/4207#issuecomment-1106125775
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 28s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4207/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 163 unchanged
- 0 fixed = 166 total (was 163) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 29s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 250m 36s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4207/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 8s | | The patch does not
generate ASF License warnings. |
| | | 373m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4207/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4207 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
| uname | Linux cd6a553a0e43 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d4509eca2343f584637ec27bf97c6e6d6c229d2e |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK
[jira] [Work logged] (HDFS-16519) Add throttler to EC reconstruction
[
https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=760671=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760671
]
ASF GitHub Bot logged work on HDFS-16519:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 07:38
Start Date: 22/Apr/22 07:38
Worklog Time Spent: 10m
Work Description: cndaimin commented on code in PR #4101:
URL: https://github.com/apache/hadoop/pull/4101#discussion_r855857036
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##
@@ -119,6 +119,14 @@ public class DFSConfigKeys extends CommonConfigurationKeys
{
"dfs.datanode.data.transfer.bandwidthPerSec";
public static final long DFS_DATANODE_DATA_TRANSFER_BANDWIDTHPERSEC_DEFAULT =
0; // A value of zero indicates no limit
+ public static final String
DFS_DATANODE_EC_RECONSTRUCT_READ_BANDWIDTHPERSEC_KEY =
+ "dfs.datanode.ec.reconstruct.read.bandwidthPerSec";
+ public static final long
DFS_DATANODE_EC_RECONSTRUCT_READ_BANDWIDTHPERSEC_DEFAULT =
+ 0; // A value of zero indicates no limit
+ public static final String
DFS_DATANODE_EC_RECONSTRUCT_WRITE_BANDWIDTHPERSEC_KEY =
+ "dfs.datanode.ec.reconstruct.write.bandwidthPerSec";
+ public static final long
DFS_DATANODE_EC_RECONSTRUCT_WRITE_BANDWIDTHPERSEC_DEFAULT =
+ 0; // A value of zero indicates no limit
Review Comment:
Thanks. I will update it in a minute.
Issue Time Tracking
---
Worklog Id: (was: 760671)
Time Spent: 1h 40m (was: 1.5h)
> Add throttler to EC reconstruction
> --
>
> Key: HDFS-16519
> URL: https://issues.apache.org/jira/browse/HDFS-16519
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, ec
>Affects Versions: 3.3.1, 3.3.2
>Reporter: daimin
>Assignee: daimin
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> HDFS already have throttlers for data transfer(replication) and balancer, the
> throttlers reduce the impact of these background procedures to user
> read/write.
> We should add a throttler to EC background reconstruction too.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16520) Improve EC pread: avoid potential reading whole block
[ https://issues.apache.org/jira/browse/HDFS-16520?focusedWorklogId=760665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760665 ] ASF GitHub Bot logged work on HDFS-16520: - Author: ASF GitHub Bot Created on: 22/Apr/22 07:30 Start Date: 22/Apr/22 07:30 Worklog Time Spent: 10m Work Description: cndaimin commented on PR #4104: URL: https://github.com/apache/hadoop/pull/4104#issuecomment-1106106812 @tasanuma Thanks for your review. Issue Time Tracking --- Worklog Id: (was: 760665) Time Spent: 2h 10m (was: 2h) > Improve EC pread: avoid potential reading whole block > - > > Key: HDFS-16520 > URL: https://issues.apache.org/jira/browse/HDFS-16520 > Project: Hadoop HDFS > Issue Type: Improvement > Components: dfsclient, ec >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > HDFS client 'pread' represents 'position read', this kind of read just need a > range of data instead of reading the whole file/block. By using > BlockReaderFactory#setLength, client tells datanode the block length to be > read from disk and sent to client. > To EC file, the block length to read is not well set, by default using > 'block.getBlockSize() - offsetInBlock' to both pread and sread. Thus datanode > read much more data and send to client, and abort when client closes > connection. There is a lot waste of resource to this situation. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16542) Fix failed unit tests in branch branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16542?focusedWorklogId=760654=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760654 ] ASF GitHub Bot logged work on HDFS-16542: - Author: ASF GitHub Bot Created on: 22/Apr/22 07:03 Start Date: 22/Apr/22 07:03 Worklog Time Spent: 10m Work Description: aajisaka commented on PR #4177: URL: https://github.com/apache/hadoop/pull/4177#issuecomment-1106077830 > Please help backporting [HDFS-13983](https://issues.apache.org/jira/browse/HDFS-13983). Thanks. Done. Thanks! Issue Time Tracking --- Worklog Id: (was: 760654) Time Spent: 1h 40m (was: 1.5h) > Fix failed unit tests in branch branch-3.2 > -- > > Key: HDFS-16542 > URL: https://issues.apache.org/jira/browse/HDFS-16542 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Affects Versions: 3.2.3 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > Tests fail in branch branch-3.2: > hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer > hadoop.hdfs.TestReconstructStripedFileWithValidator > hadoop.hdfs.server.namenode.TestFsck > hadoop.hdfs.server.blockmanagement.TestBlockManager -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13983) TestOfflineImageViewer crashes in windows
[ https://issues.apache.org/jira/browse/HDFS-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-13983: - Fix Version/s: 3.2.4 Cherry-picked to branch-3.2. > TestOfflineImageViewer crashes in windows > - > > Key: HDFS-13983 > URL: https://issues.apache.org/jira/browse/HDFS-13983 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Major > Fix For: 3.3.0, 3.2.4 > > Attachments: HDFS-13893-with-patch-intellij-idea.JPG, > HDFS-13893-with-patch-mvn.JPG, > HDFS-13893-with-patch-without-sysout-close-intellij-idea.JPG, > HDFS-13893-without-patch-intellij-idea.JPG, HDFS-13893-without-patch-mvn.JPG, > HDFS-13983-01.patch, HDFS-13983-02.patch, HDFS-13983-03.patch > > > TestOfflineImageViewer crashes in windows because, OfflineImageViewer > REVERSEXML tries to delete the outputfile and re-create the same stream which > is already created. > Also there are unclosed RAF for input files which blocks from files being > deleted. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16546) Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to branch branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-16546: - Affects Version/s: 3.2.3 (was: 3.2.0) > Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to > branch branch-3.2 > --- > > Key: HDFS-16546 > URL: https://issues.apache.org/jira/browse/HDFS-16546 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.2.3 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.4 > > Time Spent: 20m > Remaining Estimate: 0h > > The test fails due to incorrect layoutVersion. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16542) Fix failed unit tests in branch branch-3.2
[
https://issues.apache.org/jira/browse/HDFS-16542?focusedWorklogId=760646=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760646
]
ASF GitHub Bot logged work on HDFS-16542:
-
Author: ASF GitHub Bot
Created on: 22/Apr/22 06:45
Start Date: 22/Apr/22 06:45
Worklog Time Spent: 10m
Work Description: aajisaka commented on code in PR #4177:
URL: https://github.com/apache/hadoop/pull/4177#discussion_r855814689
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java:
##
@@ -838,7 +838,7 @@ public void testReverseXmlWithoutSnapshotDiffSection()
throws Throwable {
writer.println("");
writer.println("");
writer.println("");
- writer.println("-67");
+ writer.println("-65");
Review Comment:
Fixed by https://issues.apache.org/jira/browse/HDFS-16546. Thanks
Issue Time Tracking
---
Worklog Id: (was: 760646)
Time Spent: 1.5h (was: 1h 20m)
> Fix failed unit tests in branch branch-3.2
> --
>
> Key: HDFS-16542
> URL: https://issues.apache.org/jira/browse/HDFS-16542
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 3.2.3
>Reporter: daimin
>Assignee: daimin
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Tests fail in branch branch-3.2:
> hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
> hadoop.hdfs.TestReconstructStripedFileWithValidator
> hadoop.hdfs.server.namenode.TestFsck
> hadoop.hdfs.server.blockmanagement.TestBlockManager
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16546) Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to branch branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka resolved HDFS-16546. -- Fix Version/s: 3.2.4 Resolution: Fixed Committed to branch-3.2. Thank you [~cndaimin] for your contribution! > Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to > branch branch-3.2 > --- > > Key: HDFS-16546 > URL: https://issues.apache.org/jira/browse/HDFS-16546 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.2.0 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Fix For: 3.2.4 > > Time Spent: 20m > Remaining Estimate: 0h > > The test fails due to incorrect layoutVersion. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16546) Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to branch branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16546?focusedWorklogId=760638=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-760638 ] ASF GitHub Bot logged work on HDFS-16546: - Author: ASF GitHub Bot Created on: 22/Apr/22 06:34 Start Date: 22/Apr/22 06:34 Worklog Time Spent: 10m Work Description: aajisaka merged PR #4187: URL: https://github.com/apache/hadoop/pull/4187 Issue Time Tracking --- Worklog Id: (was: 760638) Time Spent: 20m (was: 10m) > Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to > branch branch-3.2 > --- > > Key: HDFS-16546 > URL: https://issues.apache.org/jira/browse/HDFS-16546 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 3.2.0 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The test fails due to incorrect layoutVersion. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16546) Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to branch branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-16546: - Priority: Major (was: Minor) > Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to > branch branch-3.2 > --- > > Key: HDFS-16546 > URL: https://issues.apache.org/jira/browse/HDFS-16546 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.2.0 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The test fails due to incorrect layoutVersion. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16546) Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to branch branch-3.2
[ https://issues.apache.org/jira/browse/HDFS-16546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-16546: - Issue Type: Bug (was: Test) > Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to > branch branch-3.2 > --- > > Key: HDFS-16546 > URL: https://issues.apache.org/jira/browse/HDFS-16546 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 3.2.0 >Reporter: daimin >Assignee: daimin >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The test fails due to incorrect layoutVersion. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
