[jira] [Updated] (HDFS-15872) Add the failed reason to Metrics duiring choosing Datanode.
[ https://issues.apache.org/jira/browse/HDFS-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15872: Attachment: (was: HDFS-15872.001.patch) > Add the failed reason to Metrics duiring choosing Datanode. > --- > > Key: HDFS-15872 > URL: https://issues.apache.org/jira/browse/HDFS-15872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode > Environment: Add the failed reason to Metrics duiring choosing > Datanode. >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15872.001.patch > > > Add the failed reason to metrics duiring choosing Datanode. So we can > troubleshoot or add storage related monitoring. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15872) Add the failed reason to Metrics duiring choosing Datanode.
[ https://issues.apache.org/jira/browse/HDFS-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15872: Attachment: HDFS-15872.001.patch Status: Patch Available (was: Open) > Add the failed reason to Metrics duiring choosing Datanode. > --- > > Key: HDFS-15872 > URL: https://issues.apache.org/jira/browse/HDFS-15872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode > Environment: Add the failed reason to Metrics duiring choosing > Datanode. >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15872.001.patch, HDFS-15872.001.patch > > > Add the failed reason to metrics duiring choosing Datanode. So we can > troubleshoot or add storage related monitoring. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15872) Add the failed reason to Metrics duiring choosing Datanode.
[ https://issues.apache.org/jira/browse/HDFS-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15872: Status: Open (was: Patch Available) > Add the failed reason to Metrics duiring choosing Datanode. > --- > > Key: HDFS-15872 > URL: https://issues.apache.org/jira/browse/HDFS-15872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode > Environment: Add the failed reason to Metrics duiring choosing > Datanode. >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15872.001.patch, HDFS-15872.001.patch > > > Add the failed reason to metrics duiring choosing Datanode. So we can > troubleshoot or add storage related monitoring. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15875) Check whether file is being truncated before truncate
[
https://issues.apache.org/jira/browse/HDFS-15875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-15875:
--
Labels: pull-request-available (was: )
> Check whether file is being truncated before truncate
> -
>
> Key: HDFS-15875
> URL: https://issues.apache.org/jira/browse/HDFS-15875
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0, 3.1.4, 3.2.2
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We have got this problem.
> * A job sends truncate to namenode, and the block recovery goes.
> * DataNode D is timeout while it connects another datanode (60s), so block
> recovery costs 60+s
> * A job tails, and B job starts and it sends truncate to namenode. New
> recoveryId generates during recovery lease.
> * DataNode D commitBlockSynchronization and get errors "does not match
> current recovery id"
> So truncate will not complete forever. Datanode D has replica with new length
> and two other datanodes have replica old length.
> DN has the error messages "Inconsistent size of finalized replicas"
> the related code is in BlockRecoveryWorker.java
> {code}
> for (BlockRecord r : syncList) {
> assert r.rInfo.getNumBytes() > 0 : "zero length replica";
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if (rState.getValue() < bestState.getValue()) {
> bestState = rState;
> }
> if(rState == ReplicaState.FINALIZED) {
> if (finalizedLength > 0 && finalizedLength != r.rInfo.getNumBytes()) {
> throw new IOException("Inconsistent size of finalized replicas. " +
> "Replica " + r.rInfo + " expected size: " + finalizedLength);
> }
> finalizedLength = r.rInfo.getNumBytes();
> }
> }
> {code}
>
>
> {code:java}
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15875) Check whether file is being truncated before truncate
[
https://issues.apache.org/jira/browse/HDFS-15875?focusedWorklogId=561297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-561297
]
ASF GitHub Bot logged work on HDFS-15875:
-
Author: ASF GitHub Bot
Created on: 05/Mar/21 05:49
Start Date: 05/Mar/21 05:49
Worklog Time Spent: 10m
Work Description: ferhui opened a new pull request #2746:
URL: https://github.com/apache/hadoop/pull/2746
## NOTICE
Please create an issue in ASF JIRA before opening a pull request,
and you need to set the title of the pull request which starts with
the corresponding JIRA issue number. (e.g. HADOOP-X. Fix a typo in YYY.)
For more details, please see
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 561297)
Remaining Estimate: 0h
Time Spent: 10m
> Check whether file is being truncated before truncate
> -
>
> Key: HDFS-15875
> URL: https://issues.apache.org/jira/browse/HDFS-15875
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0, 3.1.4, 3.2.2
>Reporter: Hui Fei
>Assignee: Hui Fei
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We have got this problem.
> * A job sends truncate to namenode, and the block recovery goes.
> * DataNode D is timeout while it connects another datanode (60s), so block
> recovery costs 60+s
> * A job tails, and B job starts and it sends truncate to namenode. New
> recoveryId generates during recovery lease.
> * DataNode D commitBlockSynchronization and get errors "does not match
> current recovery id"
> So truncate will not complete forever. Datanode D has replica with new length
> and two other datanodes have replica old length.
> DN has the error messages "Inconsistent size of finalized replicas"
> the related code is in BlockRecoveryWorker.java
> {code}
> for (BlockRecord r : syncList) {
> assert r.rInfo.getNumBytes() > 0 : "zero length replica";
> ReplicaState rState = r.rInfo.getOriginalReplicaState();
> if (rState.getValue() < bestState.getValue()) {
> bestState = rState;
> }
> if(rState == ReplicaState.FINALIZED) {
> if (finalizedLength > 0 && finalizedLength != r.rInfo.getNumBytes()) {
> throw new IOException("Inconsistent size of finalized replicas. " +
> "Replica " + r.rInfo + " expected size: " + finalizedLength);
> }
> finalizedLength = r.rInfo.getNumBytes();
> }
> }
> {code}
>
>
> {code:java}
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15875) Check whether file is being truncated before truncate
Hui Fei created HDFS-15875:
--
Summary: Check whether file is being truncated before truncate
Key: HDFS-15875
URL: https://issues.apache.org/jira/browse/HDFS-15875
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.2.2, 3.1.4, 3.3.0
Reporter: Hui Fei
Assignee: Hui Fei
We have got this problem.
* A job sends truncate to namenode, and the block recovery goes.
* DataNode D is timeout while it connects another datanode (60s), so block
recovery costs 60+s
* A job tails, and B job starts and it sends truncate to namenode. New
recoveryId generates during recovery lease.
* DataNode D commitBlockSynchronization and get errors "does not match current
recovery id"
So truncate will not complete forever. Datanode D has replica with new length
and two other datanodes have replica old length.
DN has the error messages "Inconsistent size of finalized replicas"
the related code is in BlockRecoveryWorker.java
{code}
for (BlockRecord r : syncList) {
assert r.rInfo.getNumBytes() > 0 : "zero length replica";
ReplicaState rState = r.rInfo.getOriginalReplicaState();
if (rState.getValue() < bestState.getValue()) {
bestState = rState;
}
if(rState == ReplicaState.FINALIZED) {
if (finalizedLength > 0 && finalizedLength != r.rInfo.getNumBytes()) {
throw new IOException("Inconsistent size of finalized replicas. " +
"Replica " + r.rInfo + " expected size: " + finalizedLength);
}
finalizedLength = r.rInfo.getNumBytes();
}
}
{code}
{code:java}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.
[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15874: -- Description: Now top metrics for namenode op is only supported aggregation for user, the top user op is useful. But actually, the most useful is the callerContext aggregation, we can use this to aggregate the running apps in yarn or the scheduling jobs by extend the callerContext in such as (oozie or airflow), so that we can get the real time top pressure to namenode. cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] [~ferhui] was: Now top metrics for namenode op is only supported aggregation for user, the top user op is useful. But actually, the most useful is the callerContext aggregation, we can use this to aggregate the running apps in yarn or the scheduling jobs by extend the callerContext in such as (oozie or airflow), so that we can get the real time top pressure to namenode. cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] > Extend TopMetrics to support callerContext aggregation. > --- > > Key: HDFS-15874 > URL: https://issues.apache.org/jira/browse/HDFS-15874 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Attachments: image-2021-03-05-12-01-16-852.png > > Time Spent: 10m > Remaining Estimate: 0h > > Now top metrics for namenode op is only supported aggregation for user, the > top user op is useful. > But actually, the most useful is the callerContext aggregation, we can use > this to aggregate the running apps in yarn or the scheduling jobs by extend > the callerContext in such as (oozie or airflow), so that we can get the real > time top pressure to namenode. > cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] [~ferhui] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.
[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295744#comment-17295744 ] Qi Zhu commented on HDFS-15874: --- Tested the poc pull request in my local cluster. !image-2021-03-05-12-01-16-852.png|width=1151,height=306! > Extend TopMetrics to support callerContext aggregation. > --- > > Key: HDFS-15874 > URL: https://issues.apache.org/jira/browse/HDFS-15874 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Attachments: image-2021-03-05-12-01-16-852.png > > Time Spent: 10m > Remaining Estimate: 0h > > Now top metrics for namenode op is only supported aggregation for user, the > top user op is useful. > But actually, the most useful is the callerContext aggregation, we can use > this to aggregate the running apps in yarn or the scheduling jobs by extend > the callerContext in such as (oozie or airflow), so that we can get the real > time top pressure to namenode. > cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.
[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15874: -- Attachment: image-2021-03-05-12-01-16-852.png > Extend TopMetrics to support callerContext aggregation. > --- > > Key: HDFS-15874 > URL: https://issues.apache.org/jira/browse/HDFS-15874 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Attachments: image-2021-03-05-12-01-16-852.png > > Time Spent: 10m > Remaining Estimate: 0h > > Now top metrics for namenode op is only supported aggregation for user, the > top user op is useful. > But actually, the most useful is the callerContext aggregation, we can use > this to aggregate the running apps in yarn or the scheduling jobs by extend > the callerContext in such as (oozie or airflow), so that we can get the real > time top pressure to namenode. > cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.
[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15874: -- Labels: pull-request-available (was: ) > Extend TopMetrics to support callerContext aggregation. > --- > > Key: HDFS-15874 > URL: https://issues.apache.org/jira/browse/HDFS-15874 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Now top metrics for namenode op is only supported aggregation for user, the > top user op is useful. > But actually, the most useful is the callerContext aggregation, we can use > this to aggregate the running apps in yarn or the scheduling jobs by extend > the callerContext in such as (oozie or airflow), so that we can get the real > time top pressure to namenode. > cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.
[ https://issues.apache.org/jira/browse/HDFS-15874?focusedWorklogId=561275=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-561275 ] ASF GitHub Bot logged work on HDFS-15874: - Author: ASF GitHub Bot Created on: 05/Mar/21 04:00 Start Date: 05/Mar/21 04:00 Worklog Time Spent: 10m Work Description: qizhu-lucas opened a new pull request #2744: URL: https://github.com/apache/hadoop/pull/2744 jira: [HDFS-15874](https://issues.apache.org/jira/browse/HDFS-15874) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 561275) Remaining Estimate: 0h Time Spent: 10m > Extend TopMetrics to support callerContext aggregation. > --- > > Key: HDFS-15874 > URL: https://issues.apache.org/jira/browse/HDFS-15874 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Now top metrics for namenode op is only supported aggregation for user, the > top user op is useful. > But actually, the most useful is the callerContext aggregation, we can use > this to aggregate the running apps in yarn or the scheduling jobs by extend > the callerContext in such as (oozie or airflow), so that we can get the real > time top pressure to namenode. > cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.
[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15874 started by Qi Zhu. - > Extend TopMetrics to support callerContext aggregation. > --- > > Key: HDFS-15874 > URL: https://issues.apache.org/jira/browse/HDFS-15874 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > > Now top metrics for namenode op is only supported aggregation for user, the > top user op is useful. > But actually, the most useful is the callerContext aggregation, we can use > this to aggregate the running apps in yarn or the scheduling jobs by extend > the callerContext in such as (oozie or airflow), so that we can get the real > time top pressure to namenode. > cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.
Qi Zhu created HDFS-15874: - Summary: Extend TopMetrics to support callerContext aggregation. Key: HDFS-15874 URL: https://issues.apache.org/jira/browse/HDFS-15874 Project: Hadoop HDFS Issue Type: New Feature Reporter: Qi Zhu Assignee: Qi Zhu Now top metrics for namenode op is only supported aggregation for user, the top user op is useful. But actually, the most useful is the callerContext aggregation, we can use this to aggregate the running apps in yarn or the scheduling jobs by extend the callerContext in such as (oozie or airflow), so that we can get the real time top pressure to namenode. cc [~weichiu] [~hexiaoqiao] [~ayushtkn] [~shv] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15863) RBF: Validation message to be corrected in FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295729#comment-17295729
]
Hadoop QA commented on HDFS-15863:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
32s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 6s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
14s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 47s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch
[jira] [Commented] (HDFS-15486) Costly sendResponse operation slows down async editlog handling
[ https://issues.apache.org/jira/browse/HDFS-15486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295706#comment-17295706 ] Haoze Wu commented on HDFS-15486: - [~linyiqun], [~yuanbo] , [~John Smith] I've reported a similar issue in [HDFS-15869|https://issues.apache.org/jira/browse/HDFS-15869] in the latest release version (3.2.2) and there're more analyses. I also have a pull request, which probably requires further improvement. Should we fix it together? > Costly sendResponse operation slows down async editlog handling > --- > > Key: HDFS-15486 > URL: https://issues.apache.org/jira/browse/HDFS-15486 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Yiqun Lin >Priority: Major > Attachments: Async-profile-(2).jpg, async-profile-(1).jpg > > > When our cluster NameNode in a very high load, we find it often stuck in > Async-editlog handling. > We use async-profile tool to get the flamegraph. > !Async-profile-(2).jpg! > This happened in that async editlog thread consumes Edit from the queue and > triggers the sendResponse call. > But here the sendResponse call is a little expensive since our cluster > enabled the security env and will do some encode operations when doing the > return response operation. > We often catch some moments of costly sendResponse operation when rpc call > queue is fulled. > !async-profile-(1).jpg! > Slowness on consuming Edit in async editlog will make Edit pending Queue > easily become the fulled state, then block its enqueue operation that is > invoked in writeLock type methods in FSNamesystem class. > Here the enhancement is that we can use multiple thread to parallel execute > sendResponse call. sendResponse doesn't need use the write lock to do > protection, so this change is safe. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15863) RBF: Validation message to be corrected in FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renukaprasad C updated HDFS-15863:
--
Attachment: HDFS-15863.005.patch
> RBF: Validation message to be corrected in FairnessPolicyController
> ---
>
> Key: HDFS-15863
> URL: https://issues.apache.org/jira/browse/HDFS-15863
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Minor
> Attachments: HDFS-15863.001.patch, HDFS-15863.002.patch,
> HDFS-15863.003.patch, HDFS-15863.004.patch, HDFS-15863.005.patch
>
>
> org.apache.hadoop.hdfs.server.federation.fairness.StaticRouterRpcFairnessPolicyController#validateCount
> When dfs.federation.router.handler.count is lessthan the total dedicated
> handlers for all NS, then error message shows 0 & -ve values in error
> message, instead can show the actual configured values.
> Current message is : "Available handlers -5 lower than min 0 for nsId nn1"
> This can be changed to: "Configured handlers
> ${DFS_ROUTER_HANDLER_COUNT_KEY}=10 lower than min 15 for nsId nn1", where 10
> is hander count & 15 is sum of dedicated handler count.
> Related to: HDFS-14090
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15873) Add namenode address in logs for block report
[ https://issues.apache.org/jira/browse/HDFS-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-15873: -- Labels: pull-request-available (was: ) > Add namenode address in logs for block report > - > > Key: HDFS-15873 > URL: https://issues.apache.org/jira/browse/HDFS-15873 > Project: Hadoop HDFS > Issue Type: Wish > Components: datanode, hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add namenode address in logs for block report. It's easier to track when the > block report was sent to ANN or SNN. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-15873) Add namenode address in logs for block report
[ https://issues.apache.org/jira/browse/HDFS-15873?focusedWorklogId=561254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-561254 ] ASF GitHub Bot logged work on HDFS-15873: - Author: ASF GitHub Bot Created on: 05/Mar/21 01:50 Start Date: 05/Mar/21 01:50 Worklog Time Spent: 10m Work Description: tomscut opened a new pull request #2743: URL: https://github.com/apache/hadoop/pull/2743 JIRA: https://issues.apache.org/jira/browse/HDFS-15873 Add namenode address in logs for block report. It's easier to track when the block report was sent to ANN or SNN. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 561254) Remaining Estimate: 0h Time Spent: 10m > Add namenode address in logs for block report > - > > Key: HDFS-15873 > URL: https://issues.apache.org/jira/browse/HDFS-15873 > Project: Hadoop HDFS > Issue Type: Wish > Components: datanode, hdfs >Reporter: tomscut >Assignee: tomscut >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Add namenode address in logs for block report. It's easier to track when the > block report was sent to ANN or SNN. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15873) Add namenode address in logs for block report
tomscut created HDFS-15873: -- Summary: Add namenode address in logs for block report Key: HDFS-15873 URL: https://issues.apache.org/jira/browse/HDFS-15873 Project: Hadoop HDFS Issue Type: Wish Components: datanode, hdfs Reporter: tomscut Assignee: tomscut Add namenode address in logs for block report. It's easier to track when the block report was sent to ANN or SNN. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15863) RBF: Validation message to be corrected in FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295544#comment-17295544
]
Hadoop QA commented on HDFS-15863:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
21s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 35m
2s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 5s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
14s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/512/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt{color}
| {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1
new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 58s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green}{color} |
[jira] [Updated] (HDFS-15863) RBF: Validation message to be corrected in FairnessPolicyController
[
https://issues.apache.org/jira/browse/HDFS-15863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Renukaprasad C updated HDFS-15863:
--
Attachment: HDFS-15863.004.patch
> RBF: Validation message to be corrected in FairnessPolicyController
> ---
>
> Key: HDFS-15863
> URL: https://issues.apache.org/jira/browse/HDFS-15863
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: Renukaprasad C
>Assignee: Renukaprasad C
>Priority: Minor
> Attachments: HDFS-15863.001.patch, HDFS-15863.002.patch,
> HDFS-15863.003.patch, HDFS-15863.004.patch
>
>
> org.apache.hadoop.hdfs.server.federation.fairness.StaticRouterRpcFairnessPolicyController#validateCount
> When dfs.federation.router.handler.count is lessthan the total dedicated
> handlers for all NS, then error message shows 0 & -ve values in error
> message, instead can show the actual configured values.
> Current message is : "Available handlers -5 lower than min 0 for nsId nn1"
> This can be changed to: "Configured handlers
> ${DFS_ROUTER_HANDLER_COUNT_KEY}=10 lower than min 15 for nsId nn1", where 10
> is hander count & 15 is sum of dedicated handler count.
> Related to: HDFS-14090
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15872) Add the failed reason to Metrics duiring choosing Datanode.
[
https://issues.apache.org/jira/browse/HDFS-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295259#comment-17295259
]
Hadoop QA commented on HDFS-15872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 23m
7s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 29s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
14s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
11s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 57s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/511/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt{color}
| {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 26 new +
121 unchanged - 0 fixed = 147 total (was 121) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 44s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green}{color} |
[jira] [Assigned] (HDFS-15872) Add the failed reason to Metrics duiring choosing Datanode.
[ https://issues.apache.org/jira/browse/HDFS-15872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun reassigned HDFS-15872: --- Assignee: Yang Yun > Add the failed reason to Metrics duiring choosing Datanode. > --- > > Key: HDFS-15872 > URL: https://issues.apache.org/jira/browse/HDFS-15872 > Project: Hadoop HDFS > Issue Type: Improvement > Components: block placement, namenode > Environment: Add the failed reason to Metrics duiring choosing > Datanode. >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15872.001.patch > > > Add the failed reason to metrics duiring choosing Datanode. So we can > troubleshoot or add storage related monitoring. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
