[jira] [Work logged] (HDFS-16351) add path exception information in FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-16351?focusedWorklogId=685669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685669
]
ASF GitHub Bot logged work on HDFS-16351:
-
Author: ASF GitHub Bot
Created on: 24/Nov/21 07:56
Start Date: 24/Nov/21 07:56
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3713:
URL: https://github.com/apache/hadoop/pull/3713#issuecomment-977619424
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 24s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 224m 59s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3713/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 331m 19s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3713/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3713 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 98f10ebf805a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 84386f92ad9e917f2e30760c436db855eb97226f |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3713/2/testReport/ |
| Max. process+thread count | 3754 (vs. ulimit of 5500) |
| modules | C:
[jira] [Work started] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-16350 started by Viraj Jasani. --- > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HDFS-16350: Status: Patch Available (was: In Progress) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16336) De-flake TestRollingUpgrade#testRollback
[
https://issues.apache.org/jira/browse/HDFS-16336?focusedWorklogId=685642=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685642
]
ASF GitHub Bot logged work on HDFS-16336:
-
Author: ASF GitHub Bot
Created on: 24/Nov/21 07:10
Start Date: 24/Nov/21 07:10
Worklog Time Spent: 10m
Work Description: virajjasani commented on pull request #3686:
URL: https://github.com/apache/hadoop/pull/3686#issuecomment-977592414
FYI @ferhui if you would also like to take a look. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 685642)
Time Spent: 1h 50m (was: 1h 40m)
> De-flake TestRollingUpgrade#testRollback
>
>
> Key: HDFS-16336
> URL: https://issues.apache.org/jira/browse/HDFS-16336
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Kevin Wikant
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This pull request: [https://github.com/apache/hadoop/pull/3675]
> Failed Jenkins pre-commit job due to an unrelated unit test failure:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3675/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt]
> {code:java}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.TestRollingUpgrade.testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade)
> [ERROR] Run 1: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 2: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 3: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})> {code}
> Seems that perhaps "TestRollingUpgrade.testRollback" is a flaky unit test
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685636 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 24/Nov/21 06:30 Start Date: 24/Nov/21 06:30 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-977572315 @ferhui could you please also take a look? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685636) Time Spent: 2h 10m (was: 2h) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[ https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=685633=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685633 ] ASF GitHub Bot logged work on HDFS-16331: - Author: ASF GitHub Bot Created on: 24/Nov/21 05:57 Start Date: 24/Nov/21 05:57 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3676: URL: https://github.com/apache/hadoop/pull/3676#issuecomment-977557224 @jojochuang @tasanuma @ferhui @ayushtkn , PTAL. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685633) Time Spent: 1h 20m (was: 1h 10m) > Make dfs.blockreport.intervalMsec reconfigurable > > > Key: HDFS-16331 > URL: https://issues.apache.org/jira/browse/HDFS-16331 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-18-09-33-24-236.png, > image-2021-11-18-09-35-35-400.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We have a cold data cluster, which stores as EC policy. There are 24 fast > disks on each node and each disk is 7 TB. > Recently, many nodes have more than 10 million blocks, and the interval of > FBR is 6h as default. Frequent FBR caused great pressure on NN. > !image-2021-11-18-09-35-35-400.png|width=334,height=229! > !image-2021-11-18-09-33-24-236.png|width=566,height=159! > We want to increase the interval of FBR, but have to rolling restart the DNs, > this operation is very heavy. In this scenario, it is necessary to make > _dfs.blockreport.intervalMsec_ reconfigurable. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16331) Make dfs.blockreport.intervalMsec reconfigurable
[
https://issues.apache.org/jira/browse/HDFS-16331?focusedWorklogId=685630=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685630
]
ASF GitHub Bot logged work on HDFS-16331:
-
Author: ASF GitHub Bot
Created on: 24/Nov/21 05:43
Start Date: 24/Nov/21 05:43
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3676:
URL: https://github.com/apache/hadoop/pull/3676#issuecomment-977551764
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 1s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 151 unchanged - 2
fixed = 151 total (was 153) |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 224m 37s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 322m 41s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3676/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3676 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 9ede66a9065b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / df0448d85a6676e58c0f6b5e2918546259e20c4e |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3676/7/testReport/ |
| Max. process+thread count | 3196 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3676/7/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
|
[jira] [Assigned] (HDFS-16352) return the real datanode numBlocks in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren reassigned HDFS-16352: --- Assignee: qinyuren > return the real datanode numBlocks in #getDatanodeStorageReport > --- > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Assignee: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-23-22-04-06-131.png > > Time Spent: 20m > Remaining Estimate: 0h > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16345) Fix test cases fail in TestBlockStoragePolicy
[ https://issues.apache.org/jira/browse/HDFS-16345?focusedWorklogId=685586=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685586 ] ASF GitHub Bot logged work on HDFS-16345: - Author: ASF GitHub Bot Created on: 24/Nov/21 02:33 Start Date: 24/Nov/21 02:33 Worklog Time Spent: 10m Work Description: GuoPhilipse edited a comment on pull request #3696: URL: https://github.com/apache/hadoop/pull/3696#issuecomment-976220112 cc @aajisaka @ayushtkn here are several PR tests fail for BindException of `TestBlockStoragePolicy` class. https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3705/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3684/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3661/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3714/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685586) Time Spent: 50m (was: 40m) > Fix test cases fail in TestBlockStoragePolicy > - > > Key: HDFS-16345 > URL: https://issues.apache.org/jira/browse/HDFS-16345 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build >Affects Versions: 3.3.1 >Reporter: guo >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > test class ``TestBlockStoragePolicy` ` fail frequently for the > `BindException`, it block all normal source code build. we can improve it. > [ERROR] Tests run: 26, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 49.295 s <<< FAILURE! - in org.apache.hadoop.hdfs.TestBlockStoragePolicy > [ERROR] > testChooseTargetWithTopology(org.apache.hadoop.hdfs.TestBlockStoragePolicy) > Time elapsed: 0.551 s <<< ERROR! java.net.BindException: Problem binding to > [localhost:43947] java.net.BindException: Address already in use; For more > details see: http://wiki.apache.org/hadoop/BindException at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at > org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:931) at > org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:827) at > org.apache.hadoop.ipc.Server.bind(Server.java:657) at > org.apache.hadoop.ipc.Server$Listener.(Server.java:1352) at > org.apache.hadoop.ipc.Server.(Server.java:3252) at > org.apache.hadoop.ipc.RPC$Server.(RPC.java:1062) at > org.apache.hadoop.ipc.ProtobufRpcEngine2$Server.(ProtobufRpcEngine2.java:468) > at > org.apache.hadoop.ipc.ProtobufRpcEngine2.getServer(ProtobufRpcEngine2.java:371) > at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:853) at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.(NameNodeRpcServer.java:466) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:860) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:766) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:1017) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) > at > org.apache.hadoop.hdfs.TestBlockStoragePolicy.testChooseTargetWithTopology(TestBlockStoragePolicy.java:1275) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at >
[jira] [Work logged] (HDFS-16351) add path exception information in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-16351?focusedWorklogId=685581=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685581 ] ASF GitHub Bot logged work on HDFS-16351: - Author: ASF GitHub Bot Created on: 24/Nov/21 02:24 Start Date: 24/Nov/21 02:24 Worklog Time Spent: 10m Work Description: GuoPhilipse commented on pull request #3713: URL: https://github.com/apache/hadoop/pull/3713#issuecomment-977415121 Thanks @virajjasani for your review .test case have just updated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685581) Time Spent: 0.5h (was: 20m) > add path exception information in FSNamesystem > -- > > Key: HDFS-16351 > URL: https://issues.apache.org/jira/browse/HDFS-16351 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > add path information in exception message to make message more clear in > FSNamesystem -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16333) fix balancer bug when transfer an EC block
[
https://issues.apache.org/jira/browse/HDFS-16333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hui Fei reassigned HDFS-16333:
--
Assignee: qinyuren
> fix balancer bug when transfer an EC block
> --
>
> Key: HDFS-16333
> URL: https://issues.apache.org/jira/browse/HDFS-16333
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Reporter: qinyuren
>Assignee: qinyuren
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-11-18-17-25-13-089.png,
> image-2021-11-18-17-25-50-556.png, image-2021-11-18-17-28-03-155.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We set the EC policy to (6+3) and we also have nodes that were
> decommissioning when we executed balancer.
> With the balancer running, we find many error logs as follow.
> !image-2021-11-18-17-25-13-089.png|width=858,height=135!
> Node A wants to transfer an EC block to node B, but we found that the block
> is not on node A. The FSCK command to show the block status as follow
> !image-2021-11-18-17-25-50-556.png|width=607,height=189!
> In the dispatcher. getBlockList function
> !image-2021-11-18-17-28-03-155.png!
>
> Assume that the location of the an EC block in storageGroupMap look like this
> indices:[0, 1, 2, 3, 4, 5, 6, 7, 8]
> node:[a, b, c, d, e, f, g, h, i]
> after decommission operation, the internal block on indices[1] were
> decommission to another node.
> indices:[0, 1, 2, 3, 4, 5, 6, 7, 8]
> node:[a, {color:#FF}j{color}, c, d, e, f, g, h, i]
> the location of indices[1] change from node {color:#FF}b{color} to node
> {color:#FF}j{color}.
>
> When the balancer get the block location and check it with the location in
> storageGroupMap.
> If a node is not found in storageGroupMap, it will not be add to block
> locations.
> In this case, node {color:#FF}j {color}will not be added to the block
> locations, while the indices is not updated.
> Finally, the block location may look like this,
> indices:[0, 1, 2, 3, 4, 5, 6, 7, 8]
> {color:#FF}block.location:[a, c, d, e, f, g, h, i]{color}
> the location of the nodes does not match their indices
>
> Solution:
> we should update the indices and match with the nodes
> {color:#FF}indices:[0, 2, 3, 4, 5, 6, 7, 8]{color}
> {color:#FF}block.location:[a, c, d, e, f, g, h, i]{color}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16333) fix balancer bug when transfer an EC block
[
https://issues.apache.org/jira/browse/HDFS-16333?focusedWorklogId=685565=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685565
]
ASF GitHub Bot logged work on HDFS-16333:
-
Author: ASF GitHub Bot
Created on: 24/Nov/21 01:16
Start Date: 24/Nov/21 01:16
Worklog Time Spent: 10m
Work Description: liubingxing commented on pull request #3679:
URL: https://github.com/apache/hadoop/pull/3679#issuecomment-977356044
> @liubingxing can you extend an UT for your scenario
@hemanthboyina sorry for the late reply, I add a UT to simulate the balancer
with EC file, the excluded node is to simulate the decommissioning node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 685565)
Time Spent: 1h 10m (was: 1h)
> fix balancer bug when transfer an EC block
> --
>
> Key: HDFS-16333
> URL: https://issues.apache.org/jira/browse/HDFS-16333
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer mover
>Reporter: qinyuren
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2021-11-18-17-25-13-089.png,
> image-2021-11-18-17-25-50-556.png, image-2021-11-18-17-28-03-155.png
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> We set the EC policy to (6+3) and we also have nodes that were
> decommissioning when we executed balancer.
> With the balancer running, we find many error logs as follow.
> !image-2021-11-18-17-25-13-089.png|width=858,height=135!
> Node A wants to transfer an EC block to node B, but we found that the block
> is not on node A. The FSCK command to show the block status as follow
> !image-2021-11-18-17-25-50-556.png|width=607,height=189!
> In the dispatcher. getBlockList function
> !image-2021-11-18-17-28-03-155.png!
>
> Assume that the location of the an EC block in storageGroupMap look like this
> indices:[0, 1, 2, 3, 4, 5, 6, 7, 8]
> node:[a, b, c, d, e, f, g, h, i]
> after decommission operation, the internal block on indices[1] were
> decommission to another node.
> indices:[0, 1, 2, 3, 4, 5, 6, 7, 8]
> node:[a, {color:#FF}j{color}, c, d, e, f, g, h, i]
> the location of indices[1] change from node {color:#FF}b{color} to node
> {color:#FF}j{color}.
>
> When the balancer get the block location and check it with the location in
> storageGroupMap.
> If a node is not found in storageGroupMap, it will not be add to block
> locations.
> In this case, node {color:#FF}j {color}will not be added to the block
> locations, while the indices is not updated.
> Finally, the block location may look like this,
> indices:[0, 1, 2, 3, 4, 5, 6, 7, 8]
> {color:#FF}block.location:[a, c, d, e, f, g, h, i]{color}
> the location of the nodes does not match their indices
>
> Solution:
> we should update the indices and match with the nodes
> {color:#FF}indices:[0, 2, 3, 4, 5, 6, 7, 8]{color}
> {color:#FF}block.location:[a, c, d, e, f, g, h, i]{color}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16344) Improve DirectoryScanner.Stats#toString
[ https://issues.apache.org/jira/browse/HDFS-16344?focusedWorklogId=685557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685557 ] ASF GitHub Bot logged work on HDFS-16344: - Author: ASF GitHub Bot Created on: 24/Nov/21 00:33 Start Date: 24/Nov/21 00:33 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3695: URL: https://github.com/apache/hadoop/pull/3695#issuecomment-977317596 Hi @tasanuma @ayushtkn , PTAL. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685557) Time Spent: 1h (was: 50m) > Improve DirectoryScanner.Stats#toString > --- > > Key: HDFS-16344 > URL: https://issues.apache.org/jira/browse/HDFS-16344 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-21-19-35-16-838.png > > Time Spent: 1h > Remaining Estimate: 0h > > Improve DirectoryScanner.Stats#toString. > !image-2021-11-21-19-35-16-838.png|width=1019,height=71! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16339) Show the threshold when mover threads quota is exceeded
[ https://issues.apache.org/jira/browse/HDFS-16339?focusedWorklogId=685556=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685556 ] ASF GitHub Bot logged work on HDFS-16339: - Author: ASF GitHub Bot Created on: 24/Nov/21 00:32 Start Date: 24/Nov/21 00:32 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3689: URL: https://github.com/apache/hadoop/pull/3689#issuecomment-977317248 Hi @tasanuma @ferhui , PTAL. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685556) Time Spent: 1h (was: 50m) > Show the threshold when mover threads quota is exceeded > --- > > Key: HDFS-16339 > URL: https://issues.apache.org/jira/browse/HDFS-16339 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Attachments: image-2021-11-20-17-23-04-924.png > > Time Spent: 1h > Remaining Estimate: 0h > > Show the threshold when mover threads quota is exceeded in > DataXceiver#replaceBlock and DataXceiver#copyBlock. > !image-2021-11-20-17-23-04-924.png|width=1233,height=124! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16351) add path exception information in FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-16351?focusedWorklogId=685518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685518
]
ASF GitHub Bot logged work on HDFS-16351:
-
Author: ASF GitHub Bot
Created on: 23/Nov/21 21:59
Start Date: 23/Nov/21 21:59
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3713:
URL: https://github.com/apache/hadoop/pull/3713#issuecomment-977204613
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 8s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 37m 33s | | trunk passed |
| +1 :green_heart: | compile | 2m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 10s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 30m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 36s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 36s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 371m 25s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 493m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3713/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3713 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 9394cd30049c 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e4fb5ce5b18447840e99529bba9e0947082ee5ae |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3713/1/testReport/ |
| Max. process+thread count | 2217 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3713/1/console |
| versions |
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[
https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685516=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685516
]
ASF GitHub Bot logged work on HDFS-16350:
-
Author: ASF GitHub Bot
Created on: 23/Nov/21 21:46
Start Date: 23/Nov/21 21:46
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3711:
URL: https://github.com/apache/hadoop/pull/3711#issuecomment-977195641
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 1s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 313m 46s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 421m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3711/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3711 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux aec6689d0c15 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 776b6548240a21d8fc5c08dea40adef542e98852 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3711/5/testReport/ |
| Max. process+thread count | 2267 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3711/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This
[jira] [Work logged] (HDFS-16352) return the real datanode numBlocks in #getDatanodeStorageReport
[
https://issues.apache.org/jira/browse/HDFS-16352?focusedWorklogId=685480=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685480
]
ASF GitHub Bot logged work on HDFS-16352:
-
Author: ASF GitHub Bot
Created on: 23/Nov/21 20:01
Start Date: 23/Nov/21 20:01
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3714:
URL: https://github.com/apache/hadoop/pull/3714#issuecomment-977104836
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 16s | | trunk passed |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 241m 41s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3714/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 346m 0s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3714/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3714 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux e071f1581257 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1299cc2ccc155f15572f9ef6ebdfd73ffd92e555 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
[jira] [Work logged] (HDFS-16333) fix balancer bug when transfer an EC block
[
https://issues.apache.org/jira/browse/HDFS-16333?focusedWorklogId=685468=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685468
]
ASF GitHub Bot logged work on HDFS-16333:
-
Author: ASF GitHub Bot
Created on: 23/Nov/21 19:25
Start Date: 23/Nov/21 19:25
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3679:
URL: https://github.com/apache/hadoop/pull/3679#issuecomment-977056058
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | |
hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 150 unchanged - 1
fixed = 150 total (was 151) |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 245m 42s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3679/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 347m 58s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3679/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3679 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 0e6e56cf6333 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e999e8708ade3fec186bd54f1705ba91e6add2eb |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3679/4/testReport/ |
| Max. process+thread count |
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685302=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685302 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 14:34 Start Date: 23/Nov/21 14:34 Worklog Time Spent: 10m Work Description: virajjasani edited a comment on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976586442 Starttime calculation for Namenode and Router seem better than Datanode case, because it's not the main classes that have starttime defined (i.e. NameNode and DFSRouter) but rather other classes that are internally initialized (e.g. FSNamesystem and Router) as part of the main process initialization, have the starttime defined. In the case of Datanode, the main class itself (i.e. DataNode) has starttime defined but it is initialized soon as the Datanode is instantiated. Hence, I think we should only improve Datanode's starttime to match with the actual process start time. I understand that for a healthy process initialization, there should not be much difference in starttime if we keep it at instance init level (without this patch) vs if we keep it after RPC server is started (with this patch), but my concern is that if for some reason, RPC handler start time is significantly delayed, then we would have incorrect starttime reported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685302) Time Spent: 1h 50m (was: 1h 40m) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685300=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685300 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 14:33 Start Date: 23/Nov/21 14:33 Worklog Time Spent: 10m Work Description: virajjasani edited a comment on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976586442 Starttime calculation for Namenode and Router seem better than Datanode case, because it's not the main classes that have starttime defined (i.e. NameNode and DFSRouter) but rather other classes that are internally initialized (e.g. FSNamesystem and Router) as part of the main process initialization, have the starttime defined. In the case of Datanode, the main class itself (i.e. DataNode) has starttime defined but it is initialized soon as the Datanode is instantiated. Hence, I think we can improve only Datanode's starttime to match with the actual process start time. I understand that for a healthy process initialization, there should not be much difference in starttime if we keep it at instance init level (without this patch) vs if we keep it soon after RPC server is started (with this patch), but my concern is that if for some reason, RPC handler start time is significantly delayed, then we would have incorrect starttime reported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685300) Time Spent: 1h 40m (was: 1.5h) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16352) return the real datanode numBlocks in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16352: -- Labels: pull-request-available (was: ) > return the real datanode numBlocks in #getDatanodeStorageReport > --- > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-23-22-04-06-131.png > > Time Spent: 10m > Remaining Estimate: 0h > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16352) return the real datanode numBlocks in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?focusedWorklogId=685293=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685293 ] ASF GitHub Bot logged work on HDFS-16352: - Author: ASF GitHub Bot Created on: 23/Nov/21 14:13 Start Date: 23/Nov/21 14:13 Worklog Time Spent: 10m Work Description: liubingxing opened a new pull request #3714: URL: https://github.com/apache/hadoop/pull/3714 JIRA: [HDFS-16352](https://issues.apache.org/jira/browse/HDFS-16352) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685293) Remaining Estimate: 0h Time Spent: 10m > return the real datanode numBlocks in #getDatanodeStorageReport > --- > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-23-22-04-06-131.png > > Time Spent: 10m > Remaining Estimate: 0h > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16352) return the real datanode numBlocks in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16352: Summary: return the real datanode numBlocks in #getDatanodeStorageReport (was: return the real datanode blocknums in #getDatanodeStorageReport) > return the real datanode numBlocks in #getDatanodeStorageReport > --- > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2021-11-23-22-04-06-131.png > > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685289 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 14:07 Start Date: 23/Nov/21 14:07 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976586442 Starttime calculation for Namenode and Router seem better than Datanode case, because it's not the main classes that have starttime defined (i.e. NameNode and DFSRouter) but rather other classes that are internally initialized (e.g. FSNamesystem and Router) as part of the main process initialization, have the starttime defined. In the case of Datanode, the main class itself (i.e. DataNode) has starttime defined but it is initialized before soon as the Datanode is instantiated. Hence, I think we can improve only Datanode's starttime to match with the actual process start time. I understand that for a healthy process initialization, there should not be much difference in starttime if we keep it at instance init level (without this patch) vs if we keep it soon after RPC server is started (with this patch), but my concern is that if for some reason, RPC handler start time is significantly delayed, then we would have incorrect starttime reported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685289) Time Spent: 1.5h (was: 1h 20m) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16352) return the real datanode blocknums in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16352: Summary: return the real datanode blocknums in #getDatanodeStorageReport (was: fill the real datanode blocknums in #getDatanodeStorageReport) > return the real datanode blocknums in #getDatanodeStorageReport > --- > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2021-11-23-22-04-06-131.png > > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16352) fill the real datanode blocknums in #getDatanodeStorageReport
[ https://issues.apache.org/jira/browse/HDFS-16352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qinyuren updated HDFS-16352: Description: #getDatanodeStorageReport will return the array of DatanodeStorageReport which contains the DatanodeInfo in each DatanodeStorageReport, but the numBlocks in DatanodeInfo is always zero, which is confusing !image-2021-11-23-22-04-06-131.png|width=683,height=338! Or we can return the real numBlocks in DatanodeInfo when we call #getDatanodeStorageReport was: #getDatanodeStorageReport will return the array of DatanodeStorageReport which contains the DatanodeInfo in each DatanodeStorageReport, but the numBlocks in DatanodeInfo is always zero, which is confusing !image-2021-11-23-22-04-06-131.png! Or we can return the real numBlocks in DatanodeInfo when we call #getDatanodeStorageReport > fill the real datanode blocknums in #getDatanodeStorageReport > - > > Key: HDFS-16352 > URL: https://issues.apache.org/jira/browse/HDFS-16352 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: qinyuren >Priority: Major > Attachments: image-2021-11-23-22-04-06-131.png > > > #getDatanodeStorageReport will return the array of DatanodeStorageReport > which contains the DatanodeInfo in each DatanodeStorageReport, but the > numBlocks in DatanodeInfo is always zero, which is confusing > !image-2021-11-23-22-04-06-131.png|width=683,height=338! > Or we can return the real numBlocks in DatanodeInfo when we call > #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16343) Add some debug logs when the dfsUsed are not used during Datanode startup
[ https://issues.apache.org/jira/browse/HDFS-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell resolved HDFS-16343. -- Resolution: Fixed > Add some debug logs when the dfsUsed are not used during Datanode startup > - > > Key: HDFS-16343 > URL: https://issues.apache.org/jira/browse/HDFS-16343 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.3 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16352) fill the real datanode blocknums in #getDatanodeStorageReport
qinyuren created HDFS-16352: --- Summary: fill the real datanode blocknums in #getDatanodeStorageReport Key: HDFS-16352 URL: https://issues.apache.org/jira/browse/HDFS-16352 Project: Hadoop HDFS Issue Type: Improvement Reporter: qinyuren Attachments: image-2021-11-23-22-04-06-131.png #getDatanodeStorageReport will return the array of DatanodeStorageReport which contains the DatanodeInfo in each DatanodeStorageReport, but the numBlocks in DatanodeInfo is always zero, which is confusing !image-2021-11-23-22-04-06-131.png! Or we can return the real numBlocks in DatanodeInfo when we call #getDatanodeStorageReport -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16343) Add some debug logs when the dfsUsed are not used during Datanode startup
[ https://issues.apache.org/jira/browse/HDFS-16343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDFS-16343: - Fix Version/s: 3.4.0 3.2.4 3.3.3 > Add some debug logs when the dfsUsed are not used during Datanode startup > - > > Key: HDFS-16343 > URL: https://issues.apache.org/jira/browse/HDFS-16343 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.3 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16351) add path exception information in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-16351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16351: -- Labels: pull-request-available (was: ) > add path exception information in FSNamesystem > -- > > Key: HDFS-16351 > URL: https://issues.apache.org/jira/browse/HDFS-16351 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > add path information in exception message to make message more clear in > FSNamesystem -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16351) add path exception information in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-16351?focusedWorklogId=685275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685275 ] ASF GitHub Bot logged work on HDFS-16351: - Author: ASF GitHub Bot Created on: 23/Nov/21 13:44 Start Date: 23/Nov/21 13:44 Worklog Time Spent: 10m Work Description: GuoPhilipse opened a new pull request #3713: URL: https://github.com/apache/hadoop/pull/3713 add path information in exception message to make message more clear in FSNamesystem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685275) Remaining Estimate: 0h Time Spent: 10m > add path exception information in FSNamesystem > -- > > Key: HDFS-16351 > URL: https://issues.apache.org/jira/browse/HDFS-16351 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Affects Versions: 3.3.1 >Reporter: guo >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > add path information in exception message to make message more clear in > FSNamesystem -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16351) add path exception information in FSNamesystem
guo created HDFS-16351: -- Summary: add path exception information in FSNamesystem Key: HDFS-16351 URL: https://issues.apache.org/jira/browse/HDFS-16351 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs Affects Versions: 3.3.1 Reporter: guo add path information in exception message to make message more clear in FSNamesystem -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685272=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685272 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 13:41 Start Date: 23/Nov/21 13:41 Worklog Time Spent: 10m Work Description: virajjasani edited a comment on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976548773 Thanks @tomscut. Yes this is a minor improvement in the behaviour. Basically, we should set the time according to the actual server startup time. The point is by the time RPC servers are started, Datanode is not considered started so we should keep the start time accordingly, but as I mentioned, it's minor improvement but the timing should be more accurate I believe. > At present, both `Router` and `Namenode` obtain `startTime` in this way. In that case, they might also require change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685272) Time Spent: 1h 20m (was: 1h 10m) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685269=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685269 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 13:39 Start Date: 23/Nov/21 13:39 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976548773 Thanks @tomscut. Yes this is a minor improvement in the behaviour. Basically, we should set the time according to the actual server startup time. The point is by the time RPC servers are started, Datanode is not considered started so we should keep the start time accordingly, but as I mentioned, it's minor improvement. > At present, both `Router` and `Namenode` obtain `startTime` in this way. In that case, they might also require change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685269) Time Spent: 1h 10m (was: 1h) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16343) Add some debug logs when the dfsUsed are not used during Datanode startup
[ https://issues.apache.org/jira/browse/HDFS-16343?focusedWorklogId=685267=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685267 ] ASF GitHub Bot logged work on HDFS-16343: - Author: ASF GitHub Bot Created on: 23/Nov/21 13:36 Start Date: 23/Nov/21 13:36 Worklog Time Spent: 10m Work Description: sodonnel merged pull request #3694: URL: https://github.com/apache/hadoop/pull/3694 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685267) Time Spent: 1h 10m (was: 1h) > Add some debug logs when the dfsUsed are not used during Datanode startup > - > > Key: HDFS-16343 > URL: https://issues.apache.org/jira/browse/HDFS-16343 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16343) Add some debug logs when the dfsUsed are not used during Datanode startup
[
https://issues.apache.org/jira/browse/HDFS-16343?focusedWorklogId=685235=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685235
]
ASF GitHub Bot logged work on HDFS-16343:
-
Author: ASF GitHub Bot
Created on: 23/Nov/21 13:00
Start Date: 23/Nov/21 13:00
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3694:
URL: https://github.com/apache/hadoop/pull/3694#issuecomment-976496184
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 6s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 27m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 1m 37s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 37s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 173m 1s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3694/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 42s | | The patch does not
generate ASF License warnings. |
| | | 292m 33s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.TestNameNodeHttpServerXFrame |
| | hadoop.hdfs.server.datanode.TestDataNodeFaultInjector |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithShortCircuitRead |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3694/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3694 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 3801af974af6 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision |
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685227=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685227 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 12:49 Start Date: 23/Nov/21 12:49 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976471066 > @tomscut @goiri could you please take a look? Thanks Thanks @virajjasani for reminding me here. At present, both `Router` and `Namenode` obtain `startTime` in this way. Do you mean there will be a small gap? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685227) Time Spent: 1h (was: 50m) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 1h > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685221 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 12:40 Start Date: 23/Nov/21 12:40 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976471066 > @tomscut @goiri could you please take a look? Thanks Thanks @virajjasani for reminding me here. At present, both `Router` and `Namenode` obtain `startTime` in this way. Do you mean there will be a small error? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685221) Time Spent: 50m (was: 40m) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 50m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685220 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 12:39 Start Date: 23/Nov/21 12:39 Worklog Time Spent: 10m Work Description: tomscut edited a comment on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976471066 > @tomscut @goiri could you please take a look? Thanks Thanks @virajjasani for reminding me here. At present, both router and Namenode obtain startTime in this way. Do you mean there will be a small error? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685220) Time Spent: 40m (was: 0.5h) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 40m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685219=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685219 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 12:39 Start Date: 23/Nov/21 12:39 Worklog Time Spent: 10m Work Description: tomscut commented on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976471066 > @tomscut @goiri could you please take a look? Thanks > @tomscut @goiri could you please take a look? Thanks Thanks @virajjasani for reminding me here. At present, both router and Namenode obtain startTime in this way. Do you mean there will be a small error? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685219) Time Spent: 0.5h (was: 20m) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-16336) De-flake TestRollingUpgrade#testRollback
[
https://issues.apache.org/jira/browse/HDFS-16336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-16336 started by Viraj Jasani.
---
> De-flake TestRollingUpgrade#testRollback
>
>
> Key: HDFS-16336
> URL: https://issues.apache.org/jira/browse/HDFS-16336
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Kevin Wikant
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This pull request: [https://github.com/apache/hadoop/pull/3675]
> Failed Jenkins pre-commit job due to an unrelated unit test failure:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3675/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt]
> {code:java}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.TestRollingUpgrade.testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade)
> [ERROR] Run 1: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 2: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 3: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})> {code}
> Seems that perhaps "TestRollingUpgrade.testRollback" is a flaky unit test
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16336) De-flake TestRollingUpgrade#testRollback
[
https://issues.apache.org/jira/browse/HDFS-16336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated HDFS-16336:
Status: Patch Available (was: In Progress)
> De-flake TestRollingUpgrade#testRollback
>
>
> Key: HDFS-16336
> URL: https://issues.apache.org/jira/browse/HDFS-16336
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Kevin Wikant
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This pull request: [https://github.com/apache/hadoop/pull/3675]
> Failed Jenkins pre-commit job due to an unrelated unit test failure:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3675/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt]
> {code:java}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.TestRollingUpgrade.testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade)
> [ERROR] Run 1: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 2: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 3: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})> {code}
> Seems that perhaps "TestRollingUpgrade.testRollback" is a flaky unit test
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16350: -- Labels: pull-request-available (was: ) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 10m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685191=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685191 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 11:16 Start Date: 23/Nov/21 11:16 Worklog Time Spent: 10m Work Description: virajjasani commented on pull request #3711: URL: https://github.com/apache/hadoop/pull/3711#issuecomment-976414780 @tomscut @goiri could you please take a look? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685191) Time Spent: 20m (was: 10m) > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 20m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?focusedWorklogId=685190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685190 ] ASF GitHub Bot logged work on HDFS-16350: - Author: ASF GitHub Bot Created on: 23/Nov/21 11:16 Start Date: 23/Nov/21 11:16 Worklog Time Spent: 10m Work Description: virajjasani opened a new pull request #3711: URL: https://github.com/apache/hadoop/pull/3711 ### Description of PR We set start time of Datanode when the class is instantiated but it should be ideally set only after RPC server starts and RPC handlers are initialized to serve client requests. ### How was this patch tested? Tested locally. UI screenshot: https://user-images.githubusercontent.com/34790606/143014810-4b80aee5-aa10-4bd0-8899-5a40485e5e5a.png;> ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685190) Remaining Estimate: 0h Time Spent: 10m > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Labels: pull-request-available > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > Time Spent: 10m > Remaining Estimate: 0h > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16336) De-flake TestRollingUpgrade#testRollback
[
https://issues.apache.org/jira/browse/HDFS-16336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated HDFS-16336:
Target Version/s: 3.4.0, 3.2.3, 3.3.3 (was: 3.4.0, 3.2.3, 3.3.2)
> De-flake TestRollingUpgrade#testRollback
>
>
> Key: HDFS-16336
> URL: https://issues.apache.org/jira/browse/HDFS-16336
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Kevin Wikant
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This pull request: [https://github.com/apache/hadoop/pull/3675]
> Failed Jenkins pre-commit job due to an unrelated unit test failure:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3675/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt]
> {code:java}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.TestRollingUpgrade.testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade)
> [ERROR] Run 1: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 2: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 3: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})> {code}
> Seems that perhaps "TestRollingUpgrade.testRollback" is a flaky unit test
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
[ https://issues.apache.org/jira/browse/HDFS-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani updated HDFS-16350: Attachment: Screenshot 2021-11-23 at 4.32.04 PM.png > Datanode start time should be set after RPC server starts successfully > -- > > Key: HDFS-16350 > URL: https://issues.apache.org/jira/browse/HDFS-16350 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Minor > Attachments: Screenshot 2021-11-23 at 4.32.04 PM.png > > > We set start time of Datanode when the class is instantiated but it should be > ideally set only after RPC server starts and RPC handlers are initialized to > serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16350) Datanode start time should be set after RPC server starts successfully
Viraj Jasani created HDFS-16350: --- Summary: Datanode start time should be set after RPC server starts successfully Key: HDFS-16350 URL: https://issues.apache.org/jira/browse/HDFS-16350 Project: Hadoop HDFS Issue Type: Bug Reporter: Viraj Jasani Assignee: Viraj Jasani We set start time of Datanode when the class is instantiated but it should be ideally set only after RPC server starts and RPC handlers are initialized to serve client requests. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16336) De-flake TestRollingUpgrade#testRollback
[
https://issues.apache.org/jira/browse/HDFS-16336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated HDFS-16336:
Target Version/s: 3.4.0, 3.2.3, 3.3.2
> De-flake TestRollingUpgrade#testRollback
>
>
> Key: HDFS-16336
> URL: https://issues.apache.org/jira/browse/HDFS-16336
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Kevin Wikant
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This pull request: [https://github.com/apache/hadoop/pull/3675]
> Failed Jenkins pre-commit job due to an unrelated unit test failure:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3675/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt]
> {code:java}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.TestRollingUpgrade.testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade)
> [ERROR] Run 1: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 2: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 3: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})> {code}
> Seems that perhaps "TestRollingUpgrade.testRollback" is a flaky unit test
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16347) Fix directory scan throttle default value
[
https://issues.apache.org/jira/browse/HDFS-16347?focusedWorklogId=685175=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685175
]
ASF GitHub Bot logged work on HDFS-16347:
-
Author: ASF GitHub Bot
Created on: 23/Nov/21 10:36
Start Date: 23/Nov/21 10:36
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3703:
URL: https://github.com/apache/hadoop/pull/3703#issuecomment-976384445
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 22s | | trunk passed |
| +1 :green_heart: | compile | 1m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 19s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 29m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 48s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 3s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3703/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 7 unchanged -
0 fixed = 9 total (was 7) |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 223m 42s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 340m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3703/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3703 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell xml spotbugs checkstyle |
| uname | Linux ef76374584d5 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ffb97680ba8bd324411dfccca34f54906c972eaf |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3703/2/testReport/ |
| Max. process+thread count
[jira] [Commented] (HDFS-16349) FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
[
https://issues.apache.org/jira/browse/HDFS-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17447862#comment-17447862
]
chuanjie.duan commented on HDFS-16349:
--
when I change to -2
if (rollingRollback) {
// note that the first image in imageFiles is the special checkpoint
// for the rolling upgrade
toAtLeastTxId = imageFiles.get(0).getCheckpointTxId() - 2;
}
checkForGaps would go into
if (txId > toAtLeastTxId) return;
At last, it worked
> FSEditLog checkForGaps break HDFS RollingUpgrade Rollback
> -
>
> Key: HDFS-16349
> URL: https://issues.apache.org/jira/browse/HDFS-16349
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.2
>Reporter: chuanjie.duan
>Priority: Blocker
>
> 2021-11-22 20:36:44,440 INFO
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Using longest
> log: 10.65.57.133:8485=segmentState {
> startTxId: 3906965
> endTxId: 3906965
> isInProgress: false
> }
> lastWriterEpoch: 5
> lastCommittedTxId: 3906964
> 2021-11-22 20:36:44,457 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in /data12/data/flashHadoopU/namenode/current
> 2021-11-22 20:36:44,495 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits
> file
> /data12/data/flashHadoopU/namenode/current/edits_inprogress_3898378
> ->
> /data12/data/flashHadoopU/namenode/current/edits_3898378-3898412
> 2021-11-22 20:36:44,657 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> java.io.IOException: Gap in transactions. Expected to be able to read up
> until at least txid 2510934 but unable to find any edit logs containing txid
> 2510933
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.checkForGaps(FSEditLog.java:1578)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1536)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:652)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:976)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:585)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:645)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:812)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:796)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
> 2021-11-22 20:36:44,660 INFO org.mortbay.log: Stopped
> HttpServer2$selectchannelconnectorwithsafestar...@pro-hadoop-dc01-057133.vm.dc01.hellocloud.tech:50070
> 2021-11-22 20:36:44,760 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
> system...
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2021-11-22 20:36:44,761 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2021-11-22 20:36:44,761 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> Old version: 2.7.3
> New version: 3.2.2
> Steps to Reproduce
> Step 1: Start NN1 as active , NN2 as standby .
> Step 2: Perform "hdfs dfsadmin -rollingUpgrade prepare"
> Step 3: Start NN2 active and NN1 as standby with rolling upgrade started
> option.
> Step 4: DN also restarted in upgrade mode.
> Step 5: Restart journalnode with new hadoop version
> Step 6: a few days later
> Step 7: bring down both NN, journalnode, DN
> Step 8: Start JN with old version
> Step 9: Start NN1 with rolling upgrade rollback option. nn started failed
> with above ERROR(Above mentioned txid version 2510933 has been deleted
> because of checkpoint mechanism)
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16337) Show start time of Datanode on Web
[ https://issues.apache.org/jira/browse/HDFS-16337?focusedWorklogId=685139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685139 ] ASF GitHub Bot logged work on HDFS-16337: - Author: ASF GitHub Bot Created on: 23/Nov/21 08:55 Start Date: 23/Nov/21 08:55 Worklog Time Spent: 10m Work Description: jojochuang commented on pull request #3682: URL: https://github.com/apache/hadoop/pull/3682#issuecomment-976295031 Unless it's a critical or complex change, a single +1 is sufficient to commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 685139) Time Spent: 1h 10m (was: 1h) > Show start time of Datanode on Web > -- > > Key: HDFS-16337 > URL: https://issues.apache.org/jira/browse/HDFS-16337 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.3 > > Attachments: image-2021-11-19-08-55-58-343.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Show _start time_ of Datanode on Web. > !image-2021-11-19-08-55-58-343.png|width=540,height=155! > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16336) De-flake TestRollingUpgrade#testRollback
[
https://issues.apache.org/jira/browse/HDFS-16336?focusedWorklogId=685121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-685121
]
ASF GitHub Bot logged work on HDFS-16336:
-
Author: ASF GitHub Bot
Created on: 23/Nov/21 08:31
Start Date: 23/Nov/21 08:31
Worklog Time Spent: 10m
Work Description: virajjasani commented on pull request #3686:
URL: https://github.com/apache/hadoop/pull/3686#issuecomment-976265531
Does this introduction of wait look good @aajisaka @tasanuma?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 685121)
Time Spent: 1h 40m (was: 1.5h)
> De-flake TestRollingUpgrade#testRollback
>
>
> Key: HDFS-16336
> URL: https://issues.apache.org/jira/browse/HDFS-16336
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.4.0
>Reporter: Kevin Wikant
>Assignee: Viraj Jasani
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This pull request: [https://github.com/apache/hadoop/pull/3675]
> Failed Jenkins pre-commit job due to an unrelated unit test failure:
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3675/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt]
> {code:java}
> [ERROR] Failures:
> [ERROR]
> org.apache.hadoop.hdfs.TestRollingUpgrade.testRollback(org.apache.hadoop.hdfs.TestRollingUpgrade)
> [ERROR] Run 1: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 2: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})>
> [ERROR] Run 3: TestRollingUpgrade.testRollback:328->checkMxBeanIsNull:299
> expected null, but
> was: createdRollbackImages=true, finalizeTime=0, startTime=1637204448659})> {code}
> Seems that perhaps "TestRollingUpgrade.testRollback" is a flaky unit test
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16337) Show start time of Datanode on Web
[ https://issues.apache.org/jira/browse/HDFS-16337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-16337: --- Fix Version/s: 3.3.3 (was: 3.3.2) > Show start time of Datanode on Web > -- > > Key: HDFS-16337 > URL: https://issues.apache.org/jira/browse/HDFS-16337 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.3 > > Attachments: image-2021-11-19-08-55-58-343.png > > Time Spent: 1h > Remaining Estimate: 0h > > Show _start time_ of Datanode on Web. > !image-2021-11-19-08-55-58-343.png|width=540,height=155! > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16337) Show start time of Datanode on Web
[ https://issues.apache.org/jira/browse/HDFS-16337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-16337: --- Fix Version/s: 3.2.3 3.3.2 > Show start time of Datanode on Web > -- > > Key: HDFS-16337 > URL: https://issues.apache.org/jira/browse/HDFS-16337 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.2.3, 3.3.2 > > Attachments: image-2021-11-19-08-55-58-343.png > > Time Spent: 1h > Remaining Estimate: 0h > > Show _start time_ of Datanode on Web. > !image-2021-11-19-08-55-58-343.png|width=540,height=155! > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
