[jira] [Updated] (HDFS-16173) Improve CopyCommands#Put#executor queue configurability
[ https://issues.apache.org/jira/browse/HDFS-16173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-16173: --- Fix Version/s: 3.2.4 3.3.2 3.4.0 > Improve CopyCommands#Put#executor queue configurability > --- > > Key: HDFS-16173 > URL: https://issues.apache.org/jira/browse/HDFS-16173 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2, 3.2.4 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > In CopyCommands#Put, the number of executor queues is a fixed value, 1024. > We should make him configurable, because there are different usage > environments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16173) Improve CopyCommands#Put#executor queue configurability
[ https://issues.apache.org/jira/browse/HDFS-16173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HDFS-16173. Resolution: Fixed > Improve CopyCommands#Put#executor queue configurability > --- > > Key: HDFS-16173 > URL: https://issues.apache.org/jira/browse/HDFS-16173 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2, 3.2.4 > > Time Spent: 5h 20m > Remaining Estimate: 0h > > In CopyCommands#Put, the number of executor queues is a fixed value, 1024. > We should make him configurable, because there are different usage > environments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16173) Improve CopyCommands#Put#executor queue configurability
[ https://issues.apache.org/jira/browse/HDFS-16173?focusedWorklogId=642686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642686 ] ASF GitHub Bot logged work on HDFS-16173: - Author: ASF GitHub Bot Created on: 27/Aug/21 03:42 Start Date: 27/Aug/21 03:42 Worklog Time Spent: 10m Work Description: jojochuang merged pull request #3302: URL: https://github.com/apache/hadoop/pull/3302 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 642686) Time Spent: 5h 20m (was: 5h 10m) > Improve CopyCommands#Put#executor queue configurability > --- > > Key: HDFS-16173 > URL: https://issues.apache.org/jira/browse/HDFS-16173 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 5h 20m > Remaining Estimate: 0h > > In CopyCommands#Put, the number of executor queues is a fixed value, 1024. > We should make him configurable, because there are different usage > environments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16173) Improve CopyCommands#Put#executor queue configurability
[ https://issues.apache.org/jira/browse/HDFS-16173?focusedWorklogId=642684=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642684 ] ASF GitHub Bot logged work on HDFS-16173: - Author: ASF GitHub Bot Created on: 27/Aug/21 03:39 Start Date: 27/Aug/21 03:39 Worklog Time Spent: 10m Work Description: jojochuang commented on pull request #3302: URL: https://github.com/apache/hadoop/pull/3302#issuecomment-906900261 +1 we're good to go. Thanks @jianghuazhu ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 642684) Time Spent: 5h 10m (was: 5h) > Improve CopyCommands#Put#executor queue configurability > --- > > Key: HDFS-16173 > URL: https://issues.apache.org/jira/browse/HDFS-16173 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 5h 10m > Remaining Estimate: 0h > > In CopyCommands#Put, the number of executor queues is a fixed value, 1024. > We should make him configurable, because there are different usage > environments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16188) Router to support resolving monitored namenodes with DNS
[ https://issues.apache.org/jira/browse/HDFS-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leon Gao updated HDFS-16188: Parent: (was: HDFS-15831) Issue Type: Improvement (was: Sub-task) > Router to support resolving monitored namenodes with DNS > > > Key: HDFS-16188 > URL: https://issues.apache.org/jira/browse/HDFS-16188 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > > We can use a DNS round-robin record to configure list of monitored namenodes, > so we don't have to reconfigure everything namenode hostname is changed. For > example, in containerized environment the hostname of namenode/observers can > change pretty often. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16188) Router to support resolving monitored namenodes with DNS
[ https://issues.apache.org/jira/browse/HDFS-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leon Gao updated HDFS-16188: Parent: HDFS-15831 Issue Type: Sub-task (was: Improvement) > Router to support resolving monitored namenodes with DNS > > > Key: HDFS-16188 > URL: https://issues.apache.org/jira/browse/HDFS-16188 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > > We can use a DNS round-robin record to configure list of monitored namenodes, > so we don't have to reconfigure everything namenode hostname is changed. For > example, in containerized environment the hostname of namenode/observers can > change pretty often. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16188) Router to support resolving monitored namenodes with DNS
[ https://issues.apache.org/jira/browse/HDFS-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leon Gao updated HDFS-16188: Environment: (was: We can use a DNS round-robin record to configure list of monitored namenodes, so we don't have to reconfigure everything namenode hostname is changed. For example, in containerized environment the hostname of namenode/observers can change pretty often.) > Router to support resolving monitored namenodes with DNS > > > Key: HDFS-16188 > URL: https://issues.apache.org/jira/browse/HDFS-16188 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16188) Router to support resolving monitored namenodes with DNS
[ https://issues.apache.org/jira/browse/HDFS-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leon Gao updated HDFS-16188: Description: We can use a DNS round-robin record to configure list of monitored namenodes, so we don't have to reconfigure everything namenode hostname is changed. For example, in containerized environment the hostname of namenode/observers can change pretty often. > Router to support resolving monitored namenodes with DNS > > > Key: HDFS-16188 > URL: https://issues.apache.org/jira/browse/HDFS-16188 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: Leon Gao >Assignee: Leon Gao >Priority: Minor > > We can use a DNS round-robin record to configure list of monitored namenodes, > so we don't have to reconfigure everything namenode hostname is changed. For > example, in containerized environment the hostname of namenode/observers can > change pretty often. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16188) Router to support resolving monitored namenodes with DNS
Leon Gao created HDFS-16188: --- Summary: Router to support resolving monitored namenodes with DNS Key: HDFS-16188 URL: https://issues.apache.org/jira/browse/HDFS-16188 Project: Hadoop HDFS Issue Type: Improvement Components: rbf Environment: We can use a DNS round-robin record to configure list of monitored namenodes, so we don't have to reconfigure everything namenode hostname is changed. For example, in containerized environment the hostname of namenode/observers can change pretty often. Reporter: Leon Gao Assignee: Leon Gao -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16155) Allow configurable exponential backoff in DFSInputStream refetchLocations
[
https://issues.apache.org/jira/browse/HDFS-16155?focusedWorklogId=642544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642544
]
ASF GitHub Bot logged work on HDFS-16155:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 19:30
Start Date: 26/Aug/21 19:30
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3271:
URL: https://github.com/apache/hadoop/pull/3271#issuecomment-906684045
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 30s | | trunk passed |
| +1 :green_heart: | compile | 5m 52s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 5m 34s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 27s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 2m 13s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 29s | | the patch passed |
| +1 :green_heart: | compile | 5m 41s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 5m 41s | | the patch passed |
| +1 :green_heart: | compile | 5m 7s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 5m 7s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 10s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 14s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 56s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 6m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 28s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 245m 3s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 372m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3271/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3271 |
| JIRA Issue | HDFS-16155 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 37c7e964f56c 4.15.0-151-generic #157-Ubuntu SMP Fri Jul 9
23:07:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4e1300a8d082ded6433c90f1a8f907f355920e92 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3271/7/testReport/ |
| Max.
[jira] [Commented] (HDFS-16155) Allow configurable exponential backoff in DFSInputStream refetchLocations
[
https://issues.apache.org/jira/browse/HDFS-16155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405436#comment-17405436
]
Hadoop QA commented on HDFS-16155:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
50s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 2 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 12m
46s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
30s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
52s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
34s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
27s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
48s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 6m
26s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 38s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
41s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
41s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
7s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
7s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
14s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 6m
12s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} |
[jira] [Work logged] (HDFS-16017) RBF: The getListing method should not overwrite the Listings returned by the NameNode
[ https://issues.apache.org/jira/browse/HDFS-16017?focusedWorklogId=642490=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642490 ] ASF GitHub Bot logged work on HDFS-16017: - Author: ASF GitHub Bot Created on: 26/Aug/21 17:53 Start Date: 26/Aug/21 17:53 Worklog Time Spent: 10m Work Description: goiri commented on pull request #2993: URL: https://github.com/apache/hadoop/pull/2993#issuecomment-906618573 @ayushtkn point is correct; resetting my review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 642490) Time Spent: 50m (was: 40m) > RBF: The getListing method should not overwrite the Listings returned by the > NameNode > - > > Key: HDFS-16017 > URL: https://issues.apache.org/jira/browse/HDFS-16017 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.3.1 >Reporter: Xiangyi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Regarding the getListing method, when the directory returned by NameNode is a > mount point in MountTable, the list returned by Namenode shall prevail. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=642432=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642432
]
ASF GitHub Bot logged work on HDFS-16187:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 16:32
Start Date: 26/Aug/21 16:32
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340#issuecomment-906564419
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 30m 58s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 1m 3s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 1m 13s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 1m 13s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 1m 5s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 1m 5s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 52s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 175 unchanged
- 1 fixed = 177 total (was 176) |
| -1 :x: | mvnsite | 1m 9s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3340/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private
[jira] [Work logged] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?focusedWorklogId=642396=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642396
]
ASF GitHub Bot logged work on HDFS-16187:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 15:17
Start Date: 26/Aug/21 15:17
Worklog Time Spent: 10m
Work Description: bshashikant opened a new pull request #3340:
URL: https://github.com/apache/hadoop/pull/3340
Please check https://issues.apache.org/jira/browse/HDFS-16187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 642396)
Remaining Estimate: 0h
Time Spent: 10m
> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN
> restarts with checkpointing
> ---
>
> Key: HDFS-16187
> URL: https://issues.apache.org/jira/browse/HDFS-16187
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Srinivasu Majeti
>Assignee: Shashikant Banerjee
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not
> consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
> checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
> final Path snapshottable = new Path("/zones");
> fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
> true);
> dfsAdmin.allowSnapshot(snapshottable);
> dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
> fs.createSnapshot(snapshottable, "snap1");
> SnapshotDiffReport report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> System.out.println(report);
> Assert.assertEquals(0, report.getDiffList().size());
> fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
> fs.saveNamespace();
> fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
> cluster.restartNameNode(true);
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> M .{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
[
https://issues.apache.org/jira/browse/HDFS-16187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-16187:
--
Labels: pull-request-available (was: )
> SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN
> restarts with checkpointing
> ---
>
> Key: HDFS-16187
> URL: https://issues.apache.org/jira/browse/HDFS-16187
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: snapshots
>Reporter: Srinivasu Majeti
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The below test shows the snapshot diff between across snapshots is not
> consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
> checkpointed FsImage.
> {code:java}
> @Test
> public void testEncryptionZonesWithSnapshots() throws Exception {
> final Path snapshottable = new Path("/zones");
> fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
> true);
> dfsAdmin.allowSnapshot(snapshottable);
> dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
> fs.createSnapshot(snapshottable, "snap1");
> SnapshotDiffReport report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> System.out.println(report);
> Assert.assertEquals(0, report.getDiffList().size());
> fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
> fs.saveNamespace();
> fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
> cluster.restartNameNode(true);
> report =
> fs.getSnapshotDiffReport(snapshottable, "snap1", "");
> Assert.assertEquals(0, report.getDiffList().size());
> }{code}
> {code:java}
> Pre Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> Post Restart:
> Difference between snapshot snap1 and current directory under directory
> /zones:
> M .{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16187) SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing
Shashikant Banerjee created HDFS-16187:
--
Summary: SnapshotDiff behaviour with Xattrs and Acls is not
consistent across NN restarts with checkpointing
Key: HDFS-16187
URL: https://issues.apache.org/jira/browse/HDFS-16187
Project: Hadoop HDFS
Issue Type: Bug
Components: snapshots
Reporter: Srinivasu Majeti
Assignee: Shashikant Banerjee
The below test shows the snapshot diff between across snapshots is not
consistent with Xattr(EZ here settinh the Xattr) across NN restarts with
checkpointed FsImage.
{code:java}
@Test
public void testEncryptionZonesWithSnapshots() throws Exception {
final Path snapshottable = new Path("/zones");
fsWrapper.mkdir(snapshottable, FsPermission.getDirDefault(),
true);
dfsAdmin.allowSnapshot(snapshottable);
dfsAdmin.createEncryptionZone(snapshottable, TEST_KEY, NO_TRASH);
fs.createSnapshot(snapshottable, "snap1");
SnapshotDiffReport report =
fs.getSnapshotDiffReport(snapshottable, "snap1", "");
Assert.assertEquals(0, report.getDiffList().size());
report =
fs.getSnapshotDiffReport(snapshottable, "snap1", "");
System.out.println(report);
Assert.assertEquals(0, report.getDiffList().size());
fs.setSafeMode(SafeModeAction.SAFEMODE_ENTER);
fs.saveNamespace();
fs.setSafeMode(SafeModeAction.SAFEMODE_LEAVE);
cluster.restartNameNode(true);
report =
fs.getSnapshotDiffReport(snapshottable, "snap1", "");
Assert.assertEquals(0, report.getDiffList().size());
}{code}
{code:java}
Pre Restart:
Difference between snapshot snap1 and current directory under directory /zones:
Post Restart:
Difference between snapshot snap1 and current directory under directory /zones:
M .{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?focusedWorklogId=642360=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642360
]
ASF GitHub Bot logged work on HDFS-16182:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 14:12
Start Date: 26/Aug/21 14:12
Worklog Time Spent: 10m
Work Description: Neilxzn commented on pull request #3320:
URL: https://github.com/apache/hadoop/pull/3320#issuecomment-906450340
> @Neilxzn Please fix checkstyle and check failed unit tests first. Thanks
@Hexiaoqiao Thank you for your review.
1. About BlockPlacementPolicyDefault$chooseTarget, ParameterNumber
checkstyle warning is not generated by this patch and it is hard to fix. Maybe
we ignore it this patch?
2. I checked these failed tests agian and these tests run pass locally. And
these tests seem unrelated. Please review these tests again. Thanks.
> hadoop.hdfs.server.namenode.ha.TestEditLogTailer
> hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped
> hadoop.hdfs.TestHDFSFileSystemContract
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 642360)
Time Spent: 3h (was: 2h 50m)
> numOfReplicas is given the wrong value in
> BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with
> Heterogeneous Storage
> ---
>
> Key: HDFS-16182
> URL: https://issues.apache.org/jira/browse/HDFS-16182
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.4.0
>Reporter: Max Xie
>Assignee: Max Xie
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-16182.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> In our hdfs cluster, we use heterogeneous storage to store data in SSD for a
> better performance. Sometimes hdfs client transfer data in pipline, it will
> throw IOException and exit. Exception logs are below:
> ```
> java.io.IOException: Failed to replace a bad datanode on the existing
> pipeline due to no more good datanodes being available to try. (Nodes:
> current=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK],
>
> DatanodeInfoWithStorage[dn03_ip:5004,DS-a388c067-76a4-4014-a16c-ccc49c8da77b,SSD],
>
> DatanodeInfoWithStorage[dn04_ip:5004,DS-b81da262-0dd9-4567-a498-c516fab84fe0,SSD],
>
> DatanodeInfoWithStorage[dn05_ip:5004,DS-34e3af2e-da80-46ac-938c-6a3218a646b9,SSD]],
>
> original=[DatanodeInfoWithStorage[dn01_ip:5004,DS-ef7882e0-427d-4c1e-b9ba-a929fac44fb4,DISK],
>
> DatanodeInfoWithStorage[dn02_ip:5004,DS-3871282a-ad45-4332-866a-f000f9361ecb,DISK]]).
> The current failed datanode replacement policy is DEFAULT, and a client may
> configure this via
> 'dfs.client.block.write.replace-datanode-on-failure.policy' in its
> configuration.
> ```
> After search it, I found when existing pipline need replace new dn to
> transfer data, the client will get one additional dn from namenode and check
> that the number of dn is the original number + 1.
> ```
> ## DataStreamer$findNewDatanode
> if (nodes.length != original.length + 1) {
> throw new IOException(
> "Failed to replace a bad datanode on the existing pipeline "
> + "due to no more good datanodes being available to try. "
> + "(Nodes: current=" + Arrays.asList(nodes)
> + ", original=" + Arrays.asList(original) + "). "
> + "The current failed datanode replacement policy is "
> + dfsClient.dtpReplaceDatanodeOnFailure
> + ", and a client may configure this via '"
> + BlockWrite.ReplaceDatanodeOnFailure.POLICY_KEY
> + "' in its configuration.");
> }
> ```
> The root cause is that Namenode$getAdditionalDatanode returns multi datanodes
> , not one in DataStreamer.addDatanode2ExistingPipeline.
>
> Maybe we can fix it in BlockPlacementPolicyDefault$chooseTarget. I think
> numOfReplicas should not be assigned by requiredStorageTypes.
>
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16183) RBF: Delete unnecessary metric of proxyOpComplete in routerRpcClient
[
https://issues.apache.org/jira/browse/HDFS-16183?focusedWorklogId=642335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642335
]
ASF GitHub Bot logged work on HDFS-16183:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 13:36
Start Date: 26/Aug/21 13:36
Worklog Time Spent: 10m
Work Description: wzhallright commented on a change in pull request #3328:
URL: https://github.com/apache/hadoop/pull/3328#discussion_r696637202
##
File path:
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java
##
@@ -517,7 +517,6 @@ private Object invokeMethod(
// Communication retries are handled by the retry policy
if (this.rpcMonitor != null) {
this.rpcMonitor.proxyOpFailureCommunicate();
-this.rpcMonitor.proxyOpComplete(false);
Review comment:
sorry, I didn't understand what you mean.
In my thought, if catch the exception, at least will be +1 by
proxyOpFailureCommunicate.
If namenodes is null, will throw exception.
so maybe proxyOpComplete(false) doesn't make sense.
Or I got it wrong, please correct me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 642335)
Time Spent: 1h (was: 50m)
> RBF: Delete unnecessary metric of proxyOpComplete in routerRpcClient
> -
>
> Key: HDFS-16183
> URL: https://issues.apache.org/jira/browse/HDFS-16183
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: wangzhaohui
>Assignee: wangzhaohui
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> In routerRpcClient, this.rpcMonitor.proxyOpComplete(false) do nothing, so
> maybe it doesn't need to exist.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16155) Allow configurable exponential backoff in DFSInputStream refetchLocations
[ https://issues.apache.org/jira/browse/HDFS-16155?focusedWorklogId=642327=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642327 ] ASF GitHub Bot logged work on HDFS-16155: - Author: ASF GitHub Bot Created on: 26/Aug/21 13:17 Start Date: 26/Aug/21 13:17 Worklog Time Spent: 10m Work Description: bbeaudreault commented on a change in pull request #3271: URL: https://github.com/apache/hadoop/pull/3271#discussion_r696619390 ## File path: hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml ## @@ -4440,8 +4440,29 @@ 3000 Base time window in ms for DFSClient retries. For each retry attempt, -this value is extended linearly (e.g. 3000 ms for first attempt and -first retry, 6000 ms for second retry, 9000 ms for third retry, etc.). +this value is extended exponentially based on dfs.client.retry.window.multiplier. + + + + + dfs.client.retry.window.multiplier + 1 + +Multiplier for extending the retry time window. For each retry attempt, +the retry time window is extended by multiplying dfs.client.retry.window.base +by this multiplier raised to the power of the current failure count. The default +value of 1 means the window will expand linearly (e.g. 3000 ms for first attempt +and first retry, 6000 ms for second retry, 9000 ms for third retry, etc.). + + + + + dfs.client.retry.window.max + 2147483647 Review comment: @Hexiaoqiao I've pushed a commit which lowers the window max to 30s. As mentioned above, this may cap some custom backoffs people have configured. But that may be beneficial. It should not affect the default case, given the default of 3 retries does not reach 30s. Let me know if you'd prefer a different default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 642327) Time Spent: 2h 10m (was: 2h) > Allow configurable exponential backoff in DFSInputStream refetchLocations > - > > Key: HDFS-16155 > URL: https://issues.apache.org/jira/browse/HDFS-16155 > Project: Hadoop HDFS > Issue Type: Improvement > Components: dfsclient >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > The retry policy in > [DFSInputStream#refetchLocations|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java#L1018-L1040] > was first written many years ago. It allows configuration of the base time > window, but subsequent retries double in an un-configurable way. This retry > strategy makes sense in some clusters as it's very conservative and will > avoid DDOSing the namenode in certain systemic failure modes – for example, > if a file is being read by a large hadoop job and the underlying blocks are > moved by the balancer. In this case, enough datanodes would be added to the > deadNodes list and all hadoop tasks would simultaneously try to refetch the > blocks. The 3s doubling with random factor helps break up that stampeding > herd. > However, not all cluster use-cases are created equal, so there are other > cases where a more aggressive initial backoff is preferred. For example in a > low-latency single reader scenario. In this case, if the balancer moves > enough blocks, the reader hits this 3s backoff which is way too long for a > low latency use-case. > One could configure the the window very low (10ms), but then you can hit > other systemic failure modes which would result in readers DDOSing the > namenode again. For example, if blocks went missing due to truly dead > datanodes. In this case, many readers might be refetching locations for > different files with retry backoffs like 10ms, 20ms, 40ms, etc. It takes a > while to backoff enough to avoid impacting the namenode with that strategy. > I suggest adding a configurable multiplier to the backoff strategy so that > operators can tune this as they see fit for their use-case. In the above low > latency case, one could set the base very low (say 2ms) and the multiplier > very high (say 50). This gives an aggressive first retry that very quickly > backs off. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To
[jira] [Work logged] (HDFS-6874) Add GETFILEBLOCKLOCATIONS operation to HttpFS
[
https://issues.apache.org/jira/browse/HDFS-6874?focusedWorklogId=642313=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642313
]
ASF GitHub Bot logged work on HDFS-6874:
Author: ASF GitHub Bot
Created on: 26/Aug/21 12:59
Start Date: 26/Aug/21 12:59
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3322:
URL: https://github.com/apache/hadoop/pull/3322#issuecomment-906385099
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 12m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 55s | | trunk passed |
| +1 :green_heart: | compile | 22m 49s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 31s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 3m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 20s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 34s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 16m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 38s | | the patch passed |
| +1 :green_heart: | compile | 22m 12s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 22m 12s | | the patch passed |
| +1 :green_heart: | compile | 19m 25s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 53s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 9 new + 477 unchanged - 1 fixed = 486 total (was
478) |
| +1 :green_heart: | mvnsite | 3m 45s | | the patch passed |
| +1 :green_heart: | xml | 0m 2s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 2m 54s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 3m 18s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +0 :ok: | spotbugs | 0m 30s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 17m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 27s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 2m 27s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 326m 15s | | hadoop-hdfs in the patch
passed. |
| -1 :x: | unit | 14m 54s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt)
| hadoop-hdfs-httpfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 550m 53s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem |
| | hadoop.fs.http.client.TestHttpFSFileSystemLocalFileSystem |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3322/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3322 |
|
[jira] [Work logged] (HDFS-16182) numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage
[
https://issues.apache.org/jira/browse/HDFS-16182?focusedWorklogId=642291=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642291
]
ASF GitHub Bot logged work on HDFS-16182:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 12:00
Start Date: 26/Aug/21 12:00
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3320:
URL: https://github.com/apache/hadoop/pull/3320#issuecomment-906344882
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 18m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 53s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 153 unchanged
- 2 fixed = 154 total (was 155) |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 360m 45s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 452m 43s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3320/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3320 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux b22575c707a1 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9bb5daa16c3aef93b81499ec512bd7eca21b5868 |
| Default Java | Private
[jira] [Work logged] (HDFS-16143) TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
[
https://issues.apache.org/jira/browse/HDFS-16143?focusedWorklogId=642266=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642266
]
ASF GitHub Bot logged work on HDFS-16143:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 10:42
Start Date: 26/Aug/21 10:42
Worklog Time Spent: 10m
Work Description: virajjasani commented on pull request #3235:
URL: https://github.com/apache/hadoop/pull/3235#issuecomment-906292769
Thanks everyone for the in-depth reviews!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 642266)
Time Spent: 10h 50m (was: 10h 40m)
> TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
> -
>
> Key: HDFS-16143
> URL: https://issues.apache.org/jira/browse/HDFS-16143
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3229/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
> {quote}
> [ERROR]
> testStandbyTriggersLogRollsWhenTailInProgressEdits[0](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 6.862 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRollsWhenTailInProgressEdits(TestEditLogTailer.java:444)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16186) Datanode kicks out hard disk logic optimization
[ https://issues.apache.org/jira/browse/HDFS-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17405111#comment-17405111 ] yanbin.zhang commented on HDFS-16186: - It took two days from the disk error to the datanode kicking out the hard disk, because org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker.ResultHandler#markHealthy was successfully bypassed during the check, and the datanode became a slow node > Datanode kicks out hard disk logic optimization > --- > > Key: HDFS-16186 > URL: https://issues.apache.org/jira/browse/HDFS-16186 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.1.2 > Environment: In the hadoop cluster, a certain hard disk in a certain > Datanode has a problem, but the datanode of hdfs did not kick out the hard > disk in time, causing the datanode to become a slow node >Reporter: yanbin.zhang >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > 2021-08-24 08:56:10,456 WARN datanode.DataNode > (BlockSender.java:readChecksum(681)) - Could not read or failed to verify > checksum for data at offset 113115136 for block > BP-1801371083-x.x.x.x-1603704063698:blk_5635828768_4563943709 > java.io.IOException: Input/output error > at java.io.FileInputStream.readBytes(Native Method) > at java.io.FileInputStream.read(FileInputStream.java:255) > at > org.apache.hadoop.hdfs.server.datanode.FileIoProvider$WrappedFileInputStream.read(FileIoProvider.java:876) > at java.io.FilterInputStream.read(FilterInputStream.java:133) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at java.io.DataInputStream.read(DataInputStream.java:149) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaInputStreams.readChecksumFully(ReplicaInputStreams.java:90) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.readChecksum(BlockSender.java:679) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:588) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:803) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:750) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:448) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:558) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:633) > 2021-08-24 08:56:11,121 WARN datanode.VolumeScanner > (VolumeScanner.java:handle(292)) - Reporting bad > BP-1801371083-x.x.x.x-1603704063698:blk_5635828768_4563943709 on > /data11/hdfs/data -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16173) Improve CopyCommands#Put#executor queue configurability
[ https://issues.apache.org/jira/browse/HDFS-16173?focusedWorklogId=642247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642247 ] ASF GitHub Bot logged work on HDFS-16173: - Author: ASF GitHub Bot Created on: 26/Aug/21 09:47 Start Date: 26/Aug/21 09:47 Worklog Time Spent: 10m Work Description: jianghuazhu edited a comment on pull request #3302: URL: https://github.com/apache/hadoop/pull/3302#issuecomment-906256247 @jojochuang , I'm very sorry, I previously overlooked a hidden detail in testConf.xml. I have solved it now. Can you help review again. thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 642247) Time Spent: 5h (was: 4h 50m) > Improve CopyCommands#Put#executor queue configurability > --- > > Key: HDFS-16173 > URL: https://issues.apache.org/jira/browse/HDFS-16173 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 5h > Remaining Estimate: 0h > > In CopyCommands#Put, the number of executor queues is a fixed value, 1024. > We should make him configurable, because there are different usage > environments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16173) Improve CopyCommands#Put#executor queue configurability
[ https://issues.apache.org/jira/browse/HDFS-16173?focusedWorklogId=642246=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642246 ] ASF GitHub Bot logged work on HDFS-16173: - Author: ASF GitHub Bot Created on: 26/Aug/21 09:45 Start Date: 26/Aug/21 09:45 Worklog Time Spent: 10m Work Description: jianghuazhu commented on pull request #3302: URL: https://github.com/apache/hadoop/pull/3302#issuecomment-906256247 @jojochuang , I'm very sorry, I previously overlooked a hidden detail in testConf.xml. Now I have solved him. Can you help review again. thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 642246) Time Spent: 4h 50m (was: 4h 40m) > Improve CopyCommands#Put#executor queue configurability > --- > > Key: HDFS-16173 > URL: https://issues.apache.org/jira/browse/HDFS-16173 > Project: Hadoop HDFS > Issue Type: Improvement > Components: fs >Reporter: JiangHua Zhu >Assignee: JiangHua Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 4h 50m > Remaining Estimate: 0h > > In CopyCommands#Put, the number of executor queues is a fixed value, 1024. > We should make him configurable, because there are different usage > environments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16173) Improve CopyCommands#Put#executor queue configurability
[
https://issues.apache.org/jira/browse/HDFS-16173?focusedWorklogId=642236=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642236
]
ASF GitHub Bot logged work on HDFS-16173:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 09:29
Start Date: 26/Aug/21 09:29
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3302:
URL: https://github.com/apache/hadoop/pull/3302#issuecomment-906245391
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 9s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 58s | | trunk passed |
| +1 :green_heart: | compile | 22m 14s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 19m 18s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 21m 28s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javac | 21m 28s | | the patch passed |
| +1 :green_heart: | compile | 19m 0s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | javac | 19m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 46s | | the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 2m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 16m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 16s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 56s | | The patch does not
generate ASF License warnings. |
| | | 181m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3302 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint xml |
| uname | Linux 790bd14567a1 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 80bb9d8160cea33005fd38aef75d4bb80eb52bc1 |
| Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3302/7/testReport/ |
| Max. process+thread count | 1282 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Work logged] (HDFS-16186) Datanode kicks out hard disk logic optimization
[
https://issues.apache.org/jira/browse/HDFS-16186?focusedWorklogId=642235=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642235
]
ASF GitHub Bot logged work on HDFS-16186:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 09:24
Start Date: 26/Aug/21 09:24
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3334:
URL: https://github.com/apache/hadoop/pull/3334#issuecomment-906242748
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 18s | | trunk passed |
| +1 :green_heart: | compile | 1m 45s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 4m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 1m 18s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/2/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 1m 33s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 1m 33s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 1m 22s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 1m 22s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 4s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 34 new + 13 unchanged
- 0 fixed = 47 total (was 13) |
| -1 :x: | mvnsite | 1m 20s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/2/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
[jira] [Updated] (HDFS-16186) Datanode kicks out hard disk logic optimization
[ https://issues.apache.org/jira/browse/HDFS-16186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16186: -- Labels: pull-request-available (was: ) > Datanode kicks out hard disk logic optimization > --- > > Key: HDFS-16186 > URL: https://issues.apache.org/jira/browse/HDFS-16186 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Affects Versions: 3.1.2 > Environment: In the hadoop cluster, a certain hard disk in a certain > Datanode has a problem, but the datanode of hdfs did not kick out the hard > disk in time, causing the datanode to become a slow node >Reporter: yanbin.zhang >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > 2021-08-24 08:56:10,456 WARN datanode.DataNode > (BlockSender.java:readChecksum(681)) - Could not read or failed to verify > checksum for data at offset 113115136 for block > BP-1801371083-x.x.x.x-1603704063698:blk_5635828768_4563943709 > java.io.IOException: Input/output error > at java.io.FileInputStream.readBytes(Native Method) > at java.io.FileInputStream.read(FileInputStream.java:255) > at > org.apache.hadoop.hdfs.server.datanode.FileIoProvider$WrappedFileInputStream.read(FileIoProvider.java:876) > at java.io.FilterInputStream.read(FilterInputStream.java:133) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at java.io.DataInputStream.read(DataInputStream.java:149) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaInputStreams.readChecksumFully(ReplicaInputStreams.java:90) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.readChecksum(BlockSender.java:679) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:588) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:803) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:750) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:448) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:558) > at > org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:633) > 2021-08-24 08:56:11,121 WARN datanode.VolumeScanner > (VolumeScanner.java:handle(292)) - Reporting bad > BP-1801371083-x.x.x.x-1603704063698:blk_5635828768_4563943709 on > /data11/hdfs/data -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16186) Datanode kicks out hard disk logic optimization
[
https://issues.apache.org/jira/browse/HDFS-16186?focusedWorklogId=642227=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642227
]
ASF GitHub Bot logged work on HDFS-16186:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 08:59
Start Date: 26/Aug/21 08:59
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3334:
URL: https://github.com/apache/hadoop/pull/3334#issuecomment-906225049
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 13m 2s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/3/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 12s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 0m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 1m 3s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/3/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 1m 11s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 1m 11s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 1m 10s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 1m 10s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 50s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 34 new + 14 unchanged
- 0 fixed = 48 total (was 14) |
| -1 :x: | mvnsite | 1m 6s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/3/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
|
[jira] [Work logged] (HDFS-16116) Fix Hadoop federationBanance markdown bug.
[
https://issues.apache.org/jira/browse/HDFS-16116?focusedWorklogId=642190=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642190
]
ASF GitHub Bot logged work on HDFS-16116:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 08:01
Start Date: 26/Aug/21 08:01
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3181:
URL: https://github.com/apache/hadoop/pull/3181#issuecomment-906183789
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 11m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 1m 35s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 11s | | No new issues. |
| +1 :green_heart: | shadedclient | 16m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 45s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 0m 23s | | hadoop-federation-balance in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 77m 10s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3181/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/3181 |
| Optional Tests | dupname asflicense mvnsite unit codespell shellcheck
shelldocs markdownlint |
| uname | Linux c0a44614325e 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13
01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c6b7bd264170390bd12a3d9c195af785d5d26b7b |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3181/10/testReport/ |
| Max. process+thread count | 522 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-tools/hadoop-federation-balance U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3181/10/console |
| versions | git=2.25.1 maven=3.6.3 shellcheck=0.7.0 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 642190)
Time Spent: 1h 40m (was: 1.5h)
> Fix Hadoop federationBanance markdown bug.
> --
>
> Key: HDFS-16116
> URL: https://issues.apache.org/jira/browse/HDFS-16116
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, rbf
>Reporter: panlijie
>Assignee: panlijie
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Fix Hadoop FedBalance shell and federationBanance markdown bug.
--
This message was sent by
[jira] [Created] (HDFS-16186) Datanode kicks out hard disk logic optimization
yanbin.zhang created HDFS-16186: --- Summary: Datanode kicks out hard disk logic optimization Key: HDFS-16186 URL: https://issues.apache.org/jira/browse/HDFS-16186 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.1.2 Environment: In the hadoop cluster, a certain hard disk in a certain Datanode has a problem, but the datanode of hdfs did not kick out the hard disk in time, causing the datanode to become a slow node Reporter: yanbin.zhang 2021-08-24 08:56:10,456 WARN datanode.DataNode (BlockSender.java:readChecksum(681)) - Could not read or failed to verify checksum for data at offset 113115136 for block BP-1801371083-x.x.x.x-1603704063698:blk_5635828768_4563943709 java.io.IOException: Input/output error at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at org.apache.hadoop.hdfs.server.datanode.FileIoProvider$WrappedFileInputStream.read(FileIoProvider.java:876) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at java.io.DataInputStream.read(DataInputStream.java:149) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) at org.apache.hadoop.hdfs.server.datanode.fsdataset.ReplicaInputStreams.readChecksumFully(ReplicaInputStreams.java:90) at org.apache.hadoop.hdfs.server.datanode.BlockSender.readChecksum(BlockSender.java:679) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:588) at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:803) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:750) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:448) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:558) at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:633) 2021-08-24 08:56:11,121 WARN datanode.VolumeScanner (VolumeScanner.java:handle(292)) - Reporting bad BP-1801371083-x.x.x.x-1603704063698:blk_5635828768_4563943709 on /data11/hdfs/data -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16184) De-flake TestBlockScanner#testSkipRecentAccessFile
[
https://issues.apache.org/jira/browse/HDFS-16184?focusedWorklogId=642178=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642178
]
ASF GitHub Bot logged work on HDFS-16184:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 07:45
Start Date: 26/Aug/21 07:45
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on pull request #3334:
URL: https://github.com/apache/hadoop/pull/3334#issuecomment-906174449
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 29m 28s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 16m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 1m 4s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 1m 12s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | javac | 1m 12s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. |
| -1 :x: | compile | 1m 8s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| -1 :x: | javac | 1m 8s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 52s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 34 new + 13 unchanged
- 0 fixed = 47 total (was 13) |
| -1 :x: | mvnsite | 1m 6s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3334/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| +1 :green_heart: | javadoc | 0m 46s | | the patch passed with JDK
[jira] [Updated] (HDFS-16143) TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
[
https://issues.apache.org/jira/browse/HDFS-16143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-16143:
-
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
> -
>
> Key: HDFS-16143
> URL: https://issues.apache.org/jira/browse/HDFS-16143
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
> Time Spent: 10h 40m
> Remaining Estimate: 0h
>
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3229/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
> {quote}
> [ERROR]
> testStandbyTriggersLogRollsWhenTailInProgressEdits[0](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 6.862 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRollsWhenTailInProgressEdits(TestEditLogTailer.java:444)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16143) TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
[
https://issues.apache.org/jira/browse/HDFS-16143?focusedWorklogId=642171=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642171
]
ASF GitHub Bot logged work on HDFS-16143:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 07:38
Start Date: 26/Aug/21 07:38
Worklog Time Spent: 10m
Work Description: liuml07 commented on pull request #3235:
URL: https://github.com/apache/hadoop/pull/3235#issuecomment-906169855
Thank you very much for your contribution. Thank you everyone else for
reviewing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 642171)
Time Spent: 10h 40m (was: 10.5h)
> TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
> -
>
> Key: HDFS-16143
> URL: https://issues.apache.org/jira/browse/HDFS-16143
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Attachments: patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
> Time Spent: 10h 40m
> Remaining Estimate: 0h
>
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3229/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
> {quote}
> [ERROR]
> testStandbyTriggersLogRollsWhenTailInProgressEdits[0](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 6.862 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRollsWhenTailInProgressEdits(TestEditLogTailer.java:444)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16143) TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
[
https://issues.apache.org/jira/browse/HDFS-16143?focusedWorklogId=642170=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642170
]
ASF GitHub Bot logged work on HDFS-16143:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 07:37
Start Date: 26/Aug/21 07:37
Worklog Time Spent: 10m
Work Description: liuml07 merged pull request #3235:
URL: https://github.com/apache/hadoop/pull/3235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 642170)
Time Spent: 10.5h (was: 10h 20m)
> TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
> -
>
> Key: HDFS-16143
> URL: https://issues.apache.org/jira/browse/HDFS-16143
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Attachments: patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3229/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
> {quote}
> [ERROR]
> testStandbyTriggersLogRollsWhenTailInProgressEdits[0](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 6.862 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRollsWhenTailInProgressEdits(TestEditLogTailer.java:444)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16143) TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
[
https://issues.apache.org/jira/browse/HDFS-16143?focusedWorklogId=642167=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642167
]
ASF GitHub Bot logged work on HDFS-16143:
-
Author: ASF GitHub Bot
Created on: 26/Aug/21 07:26
Start Date: 26/Aug/21 07:26
Worklog Time Spent: 10m
Work Description: tasanuma commented on pull request #3235:
URL: https://github.com/apache/hadoop/pull/3235#issuecomment-906162864
@liuml07 There are three +1s. Can we merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 642167)
Time Spent: 10h 20m (was: 10h 10m)
> TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits is flaky
> -
>
> Key: HDFS-16143
> URL: https://issues.apache.org/jira/browse/HDFS-16143
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
> Attachments: patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
>
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3229/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
> {quote}
> [ERROR]
> testStandbyTriggersLogRollsWhenTailInProgressEdits[0](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
> Time elapsed: 6.862 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:87)
> at org.junit.Assert.assertTrue(Assert.java:42)
> at org.junit.Assert.assertTrue(Assert.java:53)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRollsWhenTailInProgressEdits(TestEditLogTailer.java:444)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16116) Fix Hadoop federationBanance markdown bug.
[ https://issues.apache.org/jira/browse/HDFS-16116?focusedWorklogId=642158=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642158 ] ASF GitHub Bot logged work on HDFS-16116: - Author: ASF GitHub Bot Created on: 26/Aug/21 06:44 Start Date: 26/Aug/21 06:44 Worklog Time Spent: 10m Work Description: xiaoxiaopan118 closed pull request #3181: URL: https://github.com/apache/hadoop/pull/3181 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 642158) Time Spent: 1h 20m (was: 1h 10m) > Fix Hadoop federationBanance markdown bug. > -- > > Key: HDFS-16116 > URL: https://issues.apache.org/jira/browse/HDFS-16116 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, rbf >Reporter: panlijie >Assignee: panlijie >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Fix Hadoop FedBalance shell and federationBanance markdown bug. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16116) Fix Hadoop federationBanance markdown bug.
[ https://issues.apache.org/jira/browse/HDFS-16116?focusedWorklogId=642159=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-642159 ] ASF GitHub Bot logged work on HDFS-16116: - Author: ASF GitHub Bot Created on: 26/Aug/21 06:44 Start Date: 26/Aug/21 06:44 Worklog Time Spent: 10m Work Description: xiaoxiaopan118 commented on pull request #3181: URL: https://github.com/apache/hadoop/pull/3181#issuecomment-906139287 close the pr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 642159) Time Spent: 1.5h (was: 1h 20m) > Fix Hadoop federationBanance markdown bug. > -- > > Key: HDFS-16116 > URL: https://issues.apache.org/jira/browse/HDFS-16116 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, rbf >Reporter: panlijie >Assignee: panlijie >Priority: Minor > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > Fix Hadoop FedBalance shell and federationBanance markdown bug. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
