[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=672316=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672316
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 30/Oct/21 05:22
Start Date: 30/Oct/21 05:22
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-955150991


   Fix code style and remove useless code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672316)
Time Spent: 1h  (was: 50m)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16091) WebHDFS should support getSnapshotDiffReportListing

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16091?focusedWorklogId=672311=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672311
 ]

ASF GitHub Bot logged work on HDFS-16091:
-

Author: ASF GitHub Bot
Created on: 30/Oct/21 04:10
Start Date: 30/Oct/21 04:10
Worklog Time Spent: 10m 
  Work Description: iwasakims commented on pull request #3374:
URL: https://github.com/apache/hadoop/pull/3374#issuecomment-955142177


   Thanks, @jojochuang. I rebased this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672311)
Time Spent: 2.5h  (was: 2h 20m)

> WebHDFS should support getSnapshotDiffReportListing
> ---
>
> Key: HDFS-16091
> URL: https://issues.apache.org/jira/browse/HDFS-16091
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> When there are millions of diffs between two snapshots, the old 
> getSnapshotDiffReport() isn't scalable.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?focusedWorklogId=672289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672289
 ]

ASF GitHub Bot logged work on HDFS-16290:
-

Author: ASF GitHub Bot
Created on: 30/Oct/21 01:16
Start Date: 30/Oct/21 01:16
Worklog Time Spent: 10m 
  Work Description: jianghuazhu commented on a change in pull request #3600:
URL: https://github.com/apache/hadoop/pull/3600#discussion_r739590581



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
##
@@ -378,7 +378,7 @@ void verifyAndSetNamespaceInfo(BPServiceActor actor, 
NamespaceInfo nsInfo)
 
 if(nsInfo.getState() == HAServiceState.ACTIVE
 && bpServiceToActive == null) {
-  LOG.info("Acknowledging ACTIVE Namenode during handshake" + actor);
+  LOG.info("Acknowledging ACTIVE Namenode during handshake " + actor);

Review comment:
   Thanks for your comments and reviews, @ayushtkn  @tomscut.
   I will update it later.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672289)
Time Spent: 1h 20m  (was: 1h 10m)

> Make log more standardized when executing verifyAndSetNamespaceInfo()
> -
>
> Key: HDFS-16290
> URL: https://issues.apache.org/jira/browse/HDFS-16290
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>  Labels: pull-request-available
> Attachments: screenshot-1.png
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> When verifyAndSetNamespaceInfo() is executed, the log will record some 
> information. E.g:
> '
> 2021-10-27 18:08:36,242 [50867]-INFO 
> [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
> handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
> 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
> .xxx.xxx.org/.xxx.xxx.xxx:8021
> '
> Here, the connection between the'handshake' and the'Block pool' is too tight, 
> and the readability is not good.
>  !screenshot-1.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?focusedWorklogId=672264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672264
 ]

ASF GitHub Bot logged work on HDFS-16290:
-

Author: ASF GitHub Bot
Created on: 30/Oct/21 00:11
Start Date: 30/Oct/21 00:11
Worklog Time Spent: 10m 
  Work Description: tomscut commented on a change in pull request #3600:
URL: https://github.com/apache/hadoop/pull/3600#discussion_r739582404



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
##
@@ -378,7 +378,7 @@ void verifyAndSetNamespaceInfo(BPServiceActor actor, 
NamespaceInfo nsInfo)
 
 if(nsInfo.getState() == HAServiceState.ACTIVE
 && bpServiceToActive == null) {
-  LOG.info("Acknowledging ACTIVE Namenode during handshake" + actor);
+  LOG.info("Acknowledging ACTIVE Namenode during handshake " + actor);

Review comment:
   I agree with @ayushtkn .




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672264)
Time Spent: 1h 10m  (was: 1h)

> Make log more standardized when executing verifyAndSetNamespaceInfo()
> -
>
> Key: HDFS-16290
> URL: https://issues.apache.org/jira/browse/HDFS-16290
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>  Labels: pull-request-available
> Attachments: screenshot-1.png
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> When verifyAndSetNamespaceInfo() is executed, the log will record some 
> information. E.g:
> '
> 2021-10-27 18:08:36,242 [50867]-INFO 
> [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
> handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
> 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
> .xxx.xxx.org/.xxx.xxx.xxx:8021
> '
> Here, the connection between the'handshake' and the'Block pool' is too tight, 
> and the readability is not good.
>  !screenshot-1.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16266) Add remote port information to HDFS audit log

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16266?focusedWorklogId=672260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672260
 ]

ASF GitHub Bot logged work on HDFS-16266:
-

Author: ASF GitHub Bot
Created on: 30/Oct/21 00:08
Start Date: 30/Oct/21 00:08
Worklog Time Spent: 10m 
  Work Description: tomscut commented on pull request #3538:
URL: https://github.com/apache/hadoop/pull/3538#issuecomment-955108462


   > How about using the word of "port" instead of "clientPort" here, and 
adding "clientPort" as the actual client server port for RBF in another JIRA?
   
   Thank you @tasanuma for your advice.
   
   Based on your suggestion, I have an idea like this:
   
   We leave the ClientPort field for now.
   
   1. When a client sends a request directly to Namenode, the ```clientPort``` 
records the port of the real client.
   
   2. When the client sends a request to Router and then forwards to NameNode, 
we set the ```clientPort``` (the real clientPort) in the ```CallerContext``` of 
the Router. Before namenode prints the audit log, if the ```CallerContext``` 
already contains the ```clientPort``` field, we will not set port again. (I 
plan to do this in another PR)
   
   In both cases, only one ```clientPort``` field is left in the CallerContext, 
which holds the actual clientPort. What do you think of this? Looking forward 
to your comments. Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672260)
Time Spent: 6h 20m  (was: 6h 10m)

> Add remote port information to HDFS audit log
> -
>
> Key: HDFS-16266
> URL: https://issues.apache.org/jira/browse/HDFS-16266
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6h 20m
>  Remaining Estimate: 0h
>
> In our production environment, we occasionally encounter a problem where a 
> user submits an abnormal computation task, causing a sudden flood of 
> requests, which causes the queueTime and processingTime of the Namenode to 
> rise very high, causing a large backlog of tasks.
> We usually locate and kill specific Spark, Flink, or MapReduce tasks based on 
> metrics and audit logs. Currently, IP and UGI are recorded in audit logs, but 
> there is no port information, so it is difficult to locate specific processes 
> sometimes. Therefore, I propose that we add the port information to the audit 
> log, so that we can easily track the upstream process.
> Currently, some projects contain port information in audit logs, such as 
> Hbase and Alluxio. I think it is also necessary to add port information for 
> HDFS audit logs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16266) Add remote port information to HDFS audit log

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16266?focusedWorklogId=672261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672261
 ]

ASF GitHub Bot logged work on HDFS-16266:
-

Author: ASF GitHub Bot
Created on: 30/Oct/21 00:08
Start Date: 30/Oct/21 00:08
Worklog Time Spent: 10m 
  Work Description: tomscut edited a comment on pull request #3538:
URL: https://github.com/apache/hadoop/pull/3538#issuecomment-955108462


   > How about using the word of "port" instead of "clientPort" here, and 
adding "clientPort" as the actual client server port for RBF in another JIRA?
   
   Thank you @tasanuma for your advice.
   
   Based on your suggestion, I have an idea like this:
   
   We leave the ClientPort field for now.
   
   1. When a client sends a request directly to Namenode, the ```clientPort``` 
records the port of the real client.
   
   2. When the client sends a request to Router and then forwards to NameNode, 
we set the ```clientPort``` (the real clientPort) in the ```CallerContext``` of 
the Router. Before NameNode prints the audit log, if the ```CallerContext``` 
already contains the ```clientPort``` field, we will not set port again. (I 
plan to do this in another PR)
   
   In both cases, only one ```clientPort``` field is left in the CallerContext, 
which holds the actual clientPort. What do you think of this? Looking forward 
to your comments. Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672261)
Time Spent: 6.5h  (was: 6h 20m)

> Add remote port information to HDFS audit log
> -
>
> Key: HDFS-16266
> URL: https://issues.apache.org/jira/browse/HDFS-16266
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6.5h
>  Remaining Estimate: 0h
>
> In our production environment, we occasionally encounter a problem where a 
> user submits an abnormal computation task, causing a sudden flood of 
> requests, which causes the queueTime and processingTime of the Namenode to 
> rise very high, causing a large backlog of tasks.
> We usually locate and kill specific Spark, Flink, or MapReduce tasks based on 
> metrics and audit logs. Currently, IP and UGI are recorded in audit logs, but 
> there is no port information, so it is difficult to locate specific processes 
> sometimes. Therefore, I propose that we add the port information to the audit 
> log, so that we can easily track the upstream process.
> Currently, some projects contain port information in audit logs, such as 
> Hbase and Alluxio. I think it is also necessary to add port information for 
> HDFS audit logs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16266) Add remote port information to HDFS audit log

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16266?focusedWorklogId=672247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672247
 ]

ASF GitHub Bot logged work on HDFS-16266:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 23:16
Start Date: 29/Oct/21 23:16
Worklog Time Spent: 10m 
  Work Description: tomscut commented on pull request #3538:
URL: https://github.com/apache/hadoop/pull/3538#issuecomment-955096647


   > Thanks for updating it, @tomscut. I tried it with my RBF cluster. There is 
a client server (1.1.1.1), a DFS Router (2.2.2.2), and NameNode. When a client 
sends a request to the Router, NameNode logs the following.
   > 
   > ```
   > INFO FSNamesystem.audit: allowed=true   ugi=tasanuma ip=/2.2.2.2   
cmd=listStatus  src=/user/tasanuma  dst=nullperm=null   
proto=rpc   callerContext=CLI,clientIp:1.1.1.1,clientPort:33070
   > ```
   > 
   > In this case, `clientIp:1.1.1.1` is the IP of the client server, but 
`clientPort:33070` is the port of the DFS Router (2.2.2.2), not the one of the 
client server. It would be confusing for the users.
   
   Thank you @tasanuma  very much for your test. Because the router acts as a 
client of the Namenode, in this case the port of router is indeed regarded as a 
```clientport```.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672247)
Time Spent: 6h 10m  (was: 6h)

> Add remote port information to HDFS audit log
> -
>
> Key: HDFS-16266
> URL: https://issues.apache.org/jira/browse/HDFS-16266
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> In our production environment, we occasionally encounter a problem where a 
> user submits an abnormal computation task, causing a sudden flood of 
> requests, which causes the queueTime and processingTime of the Namenode to 
> rise very high, causing a large backlog of tasks.
> We usually locate and kill specific Spark, Flink, or MapReduce tasks based on 
> metrics and audit logs. Currently, IP and UGI are recorded in audit logs, but 
> there is no port information, so it is difficult to locate specific processes 
> sometimes. Therefore, I propose that we add the port information to the audit 
> log, so that we can easily track the upstream process.
> Currently, some projects contain port information in audit logs, such as 
> Hbase and Alluxio. I think it is also necessary to add port information for 
> HDFS audit logs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-16283) RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls

[Jing Zhao (Jira)]

[ 
https://issues.apache.org/jira/browse/HDFS-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17436168#comment-17436168
 ] 

Jing Zhao commented on HDFS-16283:
--

Checking into the code, looks like besides the router side performance issue, 
the whole lease mechanism may need to get updated to support hdfs router.

Currently the DFSClient uses a map(INodeID, DFSOutputStream) to track all the 
being-written files. The assumption is that all the being-written files are in 
the same nameservice thus there is no INodeID conflict. Now with the support of 
router, we may have two files belonging to two different nameservices sharing 
the same INodeID (though the possibility is very low in production). So 
theoretically we should update the being-written-file map to ((nameservice, 
INodeID) – DFSOutputStream).

I understand the concern that with router we do not want client to know 
individual nameservices. It would be better if we can still hide nameservices. 
We can discuss different solutions in this ticket. In summary I guess we need 
to have a new mechanism to align the current INode ID based lease renewal 
approach to the new router architecture.

> RBF: improve renewLease() to call only a specific NameNode rather than make 
> fan-out calls
> -
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: rbf
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the 
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes 
> are slow, then eventually the router queues are blocked by all renewLease() 
> and cause router degradation. 
> We will make a change in the client side to keep track of NameNode Id in 
> additional to current fileId so routers understand which NameNodes the client 
> is renewing lease against.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=672141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672141
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 19:35
Start Date: 29/Oct/21 19:35
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-955001002


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m  6s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  35m 37s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 35s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   1m 23s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   1m  2s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 31s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m  2s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 30s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 38s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 34s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 23s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 31s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   1m 31s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   1m 18s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   0m 55s | 
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 122 unchanged 
- 0 fixed = 127 total (was 122)  |
   | +1 :green_heart: |  mvnsite  |   1m 24s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 54s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 26s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 41s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  25m 50s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 385m 45s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   1m  4s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 495m 49s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestHDFSFileSystemContract |
   |   | hadoop.hdfs.TestViewDistributedFileSystemContract |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3596/2/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3596 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 297d371d729d 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 91aaaea72037e88b2a3af970ab81380a990307e8 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | 

[jira] [Work logged] (HDFS-16283) RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=672091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672091
 ]

ASF GitHub Bot logged work on HDFS-16283:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 17:52
Start Date: 29/Oct/21 17:52
Worklog Time Spent: 10m 
  Work Description: aihuaxu commented on pull request #3595:
URL: https://github.com/apache/hadoop/pull/3595#issuecomment-954934697


   Make sense. I will put up a design for discussion. Thanks Ayush.
   
   On Fri, Oct 29, 2021 at 10:41 AM Ayush Saxena ***@***.***>
   wrote:
   
   > A quick look on the PR & the discussions so far. I would also prefer we
   > discuss this somewhere first and have a proper design explaining the
   > details, pros and cons of other alternatives, if considered and then move
   > on to the coding bit. A bunch of questions floating in my mind as I went
   > through the code, So, should be better if we have a proposal before hand.
   >
   > @aihuaxu  not denying anything proposed here,
   > may be as you said this might be the best possible way to do things, but if
   > you put up a design for discussion, it would be easier for others to
   > understand as well.
   >
   > —
   > You are receiving this because you were mentioned.
   > Reply to this email directly, view it on GitHub
   > , or
   > unsubscribe
   > 

   > .
   > Triage notifications on the go with GitHub Mobile for iOS
   > 

   > or Android
   > 
.
   >
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672091)
Time Spent: 1.5h  (was: 1h 20m)

> RBF: improve renewLease() to call only a specific NameNode rather than make 
> fan-out calls
> -
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: rbf
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the 
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes 
> are slow, then eventually the router queues are blocked by all renewLease() 
> and cause router degradation. 
> We will make a change in the client side to keep track of NameNode Id in 
> additional to current fileId so routers understand which NameNodes the client 
> is renewing lease against.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16283) RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=672087=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672087
 ]

ASF GitHub Bot logged work on HDFS-16283:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 17:41
Start Date: 29/Oct/21 17:41
Worklog Time Spent: 10m 
  Work Description: ayushtkn commented on pull request #3595:
URL: https://github.com/apache/hadoop/pull/3595#issuecomment-954926793


   A quick look on the PR & the discussions so far. I would also prefer we 
discuss this somewhere first and have a proper design explaining the details, 
pros and cons of other alternatives, if considered and then move on to the 
coding bit. A bunch of questions floating in my mind as I went through the 
code, So, should be better if we have a proposal before hand.
   
   @aihuaxu not denying anything proposed here, may be as you said this might 
be the best possible way to do things, but if you put up a design for 
discussion, it would be easier for others to understand as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672087)
Time Spent: 1h 20m  (was: 1h 10m)

> RBF: improve renewLease() to call only a specific NameNode rather than make 
> fan-out calls
> -
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: rbf
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the 
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes 
> are slow, then eventually the router queues are blocked by all renewLease() 
> and cause router degradation. 
> We will make a change in the client side to keep track of NameNode Id in 
> additional to current fileId so routers understand which NameNodes the client 
> is renewing lease against.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?focusedWorklogId=672079=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672079
 ]

ASF GitHub Bot logged work on HDFS-16290:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 17:26
Start Date: 29/Oct/21 17:26
Worklog Time Spent: 10m 
  Work Description: ayushtkn commented on a change in pull request #3600:
URL: https://github.com/apache/hadoop/pull/3600#discussion_r739415540



##
File path: 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPOfferService.java
##
@@ -378,7 +378,7 @@ void verifyAndSetNamespaceInfo(BPServiceActor actor, 
NamespaceInfo nsInfo)
 
 if(nsInfo.getState() == HAServiceState.ACTIVE
 && bpServiceToActive == null) {
-  LOG.info("Acknowledging ACTIVE Namenode during handshake" + actor);
+  LOG.info("Acknowledging ACTIVE Namenode during handshake " + actor);

Review comment:
   Can use logger format:
   ```
 LOG.info("Acknowledging ACTIVE Namenode during handshake {}", actor);
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672079)
Time Spent: 1h  (was: 50m)

> Make log more standardized when executing verifyAndSetNamespaceInfo()
> -
>
> Key: HDFS-16290
> URL: https://issues.apache.org/jira/browse/HDFS-16290
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>  Labels: pull-request-available
> Attachments: screenshot-1.png
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> When verifyAndSetNamespaceInfo() is executed, the log will record some 
> information. E.g:
> '
> 2021-10-27 18:08:36,242 [50867]-INFO 
> [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
> handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
> 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
> .xxx.xxx.org/.xxx.xxx.xxx:8021
> '
> Here, the connection between the'handshake' and the'Block pool' is too tight, 
> and the readability is not good.
>  !screenshot-1.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16283) RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=672069=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672069
 ]

ASF GitHub Bot logged work on HDFS-16283:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 17:12
Start Date: 29/Oct/21 17:12
Worklog Time Spent: 10m 
  Work Description: aihuaxu edited a comment on pull request #3595:
URL: https://github.com/apache/hadoop/pull/3595#issuecomment-954900267


   Thanks for the comments. @goiri, @symious and @Hexiaoqiao. We have thought 
of the solution completely on the router side to avoid the additional info 
exposed in the DFSClient. But the routers need to keep track of the mapping of 
the client->files and such mapping needs to be shared across multiple routers. 
That brings much complexity and also introduces the high latency on the router 
side. 
   
   renewLease(String clientName, String path) is a good solution from interface 
perspective, but it could cause NameNode overload as @Hexiaoqiao mentioned 
since instead of renewing per client, now we are renewing lease per file. This 
solution and current implementation will run into scale issue.
   
   Since adding renewLease(String clientName, String nsId) achieves the best 
performance, I feel we should still pursue similar idea rather than sacrificing 
the performance? 
   
   Another thought: maybe we can add an interface of renewLease(String 
clientName, List files) to renew lease for all the files of that client 
and then the router can resolve to name services and make one call against one 
name service.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672069)
Time Spent: 1h 10m  (was: 1h)

> RBF: improve renewLease() to call only a specific NameNode rather than make 
> fan-out calls
> -
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: rbf
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the 
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes 
> are slow, then eventually the router queues are blocked by all renewLease() 
> and cause router degradation. 
> We will make a change in the client side to keep track of NameNode Id in 
> additional to current fileId so routers understand which NameNodes the client 
> is renewing lease against.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16283) RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=672062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-672062
 ]

ASF GitHub Bot logged work on HDFS-16283:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 16:59
Start Date: 29/Oct/21 16:59
Worklog Time Spent: 10m 
  Work Description: aihuaxu commented on pull request #3595:
URL: https://github.com/apache/hadoop/pull/3595#issuecomment-954900267


   Thanks for the comments. @goiri, @symious and @Hexiaoqiao. We have thought 
of the solution completely on the router side to avoid the additional info 
exposed in the DFSClient. But the routers need to keep track of the mapping of 
the client->files and such mapping needs to be shared across multiple routers. 
That brings much complexity and also introduces the high latency on the router 
side. 
   
   renewLease(String clientName, String path) is a good solution from interface 
perspective, but it could cause NameNode overload as @Hexiaoqiao mentioned 
since instead of renewing per client, now we are renewing lease per file. This 
solution and current implementation will run into scale issue.
   
   Since adding renewLease(String clientName, String nsId) achieves the best 
performance, I feel we should still pursue similar idea rather than sacrificing 
the performance? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 672062)
Time Spent: 1h  (was: 50m)

> RBF: improve renewLease() to call only a specific NameNode rather than make 
> fan-out calls
> -
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: rbf
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the 
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes 
> are slow, then eventually the router queues are blocked by all renewLease() 
> and cause router degradation. 
> We will make a change in the client side to keep track of NameNode Id in 
> additional to current fileId so routers understand which NameNodes the client 
> is renewing lease against.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16269) [Fix] Improve NNThroughputBenchmark#blockReport operation

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16269?focusedWorklogId=671966=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671966
 ]

ASF GitHub Bot logged work on HDFS-16269:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 13:29
Start Date: 29/Oct/21 13:29
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3544:
URL: https://github.com/apache/hadoop/pull/3544#issuecomment-954745070


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 51s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 2 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  33m 17s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 22s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   1m 13s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   1m  0s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 23s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 56s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 24s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 20s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  24m 52s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 13s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 18s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   1m 18s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m  8s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   1m  8s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 52s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 14s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 47s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 19s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 22s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  25m 18s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | +1 :green_heart: |  unit  | 319m  0s |  |  hadoop-hdfs in the patch 
passed.  |
   | +1 :green_heart: |  asflicense  |   0m 39s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 422m 57s |  |  |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/4/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3544 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 9d661a554853 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 
19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 6979d3bce5df2e8504afae2def9212ae83fe529f |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/4/testReport/ |
   | Max. process+thread count | 1909 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
   | Console output | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3544/4/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This 

[jira] [Work logged] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?focusedWorklogId=671915=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671915
 ]

ASF GitHub Bot logged work on HDFS-16290:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 11:39
Start Date: 29/Oct/21 11:39
Worklog Time Spent: 10m 
  Work Description: jianghuazhu edited a comment on pull request #3600:
URL: https://github.com/apache/hadoop/pull/3600#issuecomment-954668716


   Some exceptions occurred here, during the operation of jenkins.
   E.g:
   TestHDFSFileSystemContract
   TestDecommission
   TestLeaseRecovery
   TestHdfsCryptoStreams
   TestSeveralNameNodes
   
   It seems that these exceptions have little to do with the code I submitted.
   @ayushtkn @tomscut, would you like to spend some time reviewing this pr.
   thank you very much.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 671915)
Time Spent: 50m  (was: 40m)

> Make log more standardized when executing verifyAndSetNamespaceInfo()
> -
>
> Key: HDFS-16290
> URL: https://issues.apache.org/jira/browse/HDFS-16290
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>  Labels: pull-request-available
> Attachments: screenshot-1.png
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> When verifyAndSetNamespaceInfo() is executed, the log will record some 
> information. E.g:
> '
> 2021-10-27 18:08:36,242 [50867]-INFO 
> [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
> handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
> 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
> .xxx.xxx.org/.xxx.xxx.xxx:8021
> '
> Here, the connection between the'handshake' and the'Block pool' is too tight, 
> and the readability is not good.
>  !screenshot-1.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?focusedWorklogId=671914=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671914
 ]

ASF GitHub Bot logged work on HDFS-16290:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 11:36
Start Date: 29/Oct/21 11:36
Worklog Time Spent: 10m 
  Work Description: jianghuazhu edited a comment on pull request #3600:
URL: https://github.com/apache/hadoop/pull/3600#issuecomment-954668716


   Some exceptions occurred here, during the operation of jenkins.
   E.g:
   TestHDFSFileSystemContract
   TestDecommission
   TestLeaseRecovery
   TestHdfsCryptoStreams
   
   It seems that these exceptions have little to do with the code I submitted.
   @ayushtkn @tomscut, would you like to spend some time reviewing this pr.
   thank you very much.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 671914)
Time Spent: 40m  (was: 0.5h)

> Make log more standardized when executing verifyAndSetNamespaceInfo()
> -
>
> Key: HDFS-16290
> URL: https://issues.apache.org/jira/browse/HDFS-16290
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>  Labels: pull-request-available
> Attachments: screenshot-1.png
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> When verifyAndSetNamespaceInfo() is executed, the log will record some 
> information. E.g:
> '
> 2021-10-27 18:08:36,242 [50867]-INFO 
> [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
> handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
> 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
> .xxx.xxx.org/.xxx.xxx.xxx:8021
> '
> Here, the connection between the'handshake' and the'Block pool' is too tight, 
> and the readability is not good.
>  !screenshot-1.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?focusedWorklogId=671913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671913
 ]

ASF GitHub Bot logged work on HDFS-16290:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 11:30
Start Date: 29/Oct/21 11:30
Worklog Time Spent: 10m 
  Work Description: jianghuazhu commented on pull request #3600:
URL: https://github.com/apache/hadoop/pull/3600#issuecomment-954668716


   Some exceptions occurred here, during the operation of jenkins.
   E.g:
   TestHDFSFileSystemContract
   
   It seems that these exceptions have little to do with the code I submitted.
   @ayushtkn @tomscut, would you like to spend some time reviewing this pr.
   thank you very much.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 671913)
Time Spent: 0.5h  (was: 20m)

> Make log more standardized when executing verifyAndSetNamespaceInfo()
> -
>
> Key: HDFS-16290
> URL: https://issues.apache.org/jira/browse/HDFS-16290
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>  Labels: pull-request-available
> Attachments: screenshot-1.png
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When verifyAndSetNamespaceInfo() is executed, the log will record some 
> information. E.g:
> '
> 2021-10-27 18:08:36,242 [50867]-INFO 
> [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
> handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
> 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
> .xxx.xxx.org/.xxx.xxx.xxx:8021
> '
> Here, the connection between the'handshake' and the'Block pool' is too tight, 
> and the readability is not good.
>  !screenshot-1.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16287?focusedWorklogId=671912=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671912
 ]

ASF GitHub Bot logged work on HDFS-16287:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 11:28
Start Date: 29/Oct/21 11:28
Worklog Time Spent: 10m 
  Work Description: haiyang1987 commented on pull request #3596:
URL: https://github.com/apache/hadoop/pull/3596#issuecomment-954667726


   Update PR, Make the following changes:
   1. Consider for  make dfs.namenode.avoid.read.slow.datanode reconfigurable
   2. HDFS-15120 already support 
BlockPlacementPolicyDefault#dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled
 reconfigurable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 671912)
Time Spent: 40m  (was: 0.5h)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

[Haiyang Hu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haiyang Hu updated HDFS-16287:
--
Description: 
1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable and 
rapid rollback in case this feature 
[HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected things 
happen in production environment  

2.  DatanodeManager#startSlowPeerCollector by parameter 
'dfs.datanode.peer.stats.enabled' to control




  was:
1. Consider that make dfs.namenode.avoid.read.slow.datanode and 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable 
and rapid rollback in case this feature 
[HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] and 
[HDFS-15879|https://issues.apache.org/jira/browse/HDFS-15879] unexpected things 
happen in production environment  

2. Consider In DatanodeManager dealing with choosing targets for blocks  filter 
out slow nodes logic 




> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode  reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] unexpected 
> things happen in production environment  
> 2.  DatanodeManager#startSlowPeerCollector by parameter 
> 'dfs.datanode.peer.stats.enabled' to control



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?focusedWorklogId=671911=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671911
 ]

ASF GitHub Bot logged work on HDFS-16290:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 11:15
Start Date: 29/Oct/21 11:15
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #3600:
URL: https://github.com/apache/hadoop/pull/3600#issuecomment-954660123


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   1m  9s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include 
any new or modified tests. Please justify why no new tests are needed for this 
patch. Also please list what manual steps were performed to verify this patch.  
|
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  35m 22s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   1m 37s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   1m 21s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   1m  3s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 32s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   1m  7s |  |  trunk passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 29s |  |  trunk passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 37s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  25m 55s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   1m 20s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 29s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   1m 29s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   1m 16s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   1m 16s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 54s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   1m 23s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 53s |  |  the patch passed with JDK 
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m 26s |  |  the patch passed with JDK 
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   3m 43s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  25m 35s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  | 354m 48s | 
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3600/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
 |  hadoop-hdfs in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 42s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 464m 38s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.hdfs.TestHDFSFileSystemContract |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3600/1/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/3600 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux f85bd3e36fc2 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 
16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 2ec74aa91dc58bedae20f5e1d39d2ebb9fdec17f |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | 

[jira] [Updated] (HDFS-16287) Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable

[Haiyang Hu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haiyang Hu updated HDFS-16287:
--
Summary: Support to make dfs.namenode.avoid.read.slow.datanode  
reconfigurable  (was: Support to make dfs.namenode.avoid.read.slow.datanode and 
dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable)

> Support to make dfs.namenode.avoid.read.slow.datanode  reconfigurable
> -
>
> Key: HDFS-16287
> URL: https://issues.apache.org/jira/browse/HDFS-16287
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> 1. Consider that make dfs.namenode.avoid.read.slow.datanode and 
> dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable 
> and rapid rollback in case this feature 
> [HDFS-16076|https://issues.apache.org/jira/browse/HDFS-16076] and 
> [HDFS-15879|https://issues.apache.org/jira/browse/HDFS-15879] unexpected 
> things happen in production environment  
> 2. Consider In DatanodeManager dealing with choosing targets for blocks  
> filter out slow nodes logic 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[JiangHua Zhu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

JiangHua Zhu updated HDFS-16290:

Attachment: screenshot-1.png

> Make log more standardized when executing verifyAndSetNamespaceInfo()
> -
>
> Key: HDFS-16290
> URL: https://issues.apache.org/jira/browse/HDFS-16290
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>  Labels: pull-request-available
> Attachments: screenshot-1.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When verifyAndSetNamespaceInfo() is executed, the log will record some 
> information. E.g:
> '
> 2021-10-27 18:08:36,242 [50867]-INFO 
> [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
> handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
> 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
> .xxx.xxx.org/.xxx.xxx.xxx:8021
> '
> Here, the connection between the'handshake' and the'Block pool' is too tight, 
> and the readability is not good.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[JiangHua Zhu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

JiangHua Zhu updated HDFS-16290:

Description: 
When verifyAndSetNamespaceInfo() is executed, the log will record some 
information. E.g:
'
2021-10-27 18:08:36,242 [50867]-INFO 
[Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
.xxx.xxx.org/.xxx.xxx.xxx:8021
'
Here, the connection between the'handshake' and the'Block pool' is too tight, 
and the readability is not good.

 !screenshot-1.png! 

  was:
When verifyAndSetNamespaceInfo() is executed, the log will record some 
information. E.g:
'
2021-10-27 18:08:36,242 [50867]-INFO 
[Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
.xxx.xxx.org/.xxx.xxx.xxx:8021
'
Here, the connection between the'handshake' and the'Block pool' is too tight, 
and the readability is not good.


> Make log more standardized when executing verifyAndSetNamespaceInfo()
> -
>
> Key: HDFS-16290
> URL: https://issues.apache.org/jira/browse/HDFS-16290
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>  Labels: pull-request-available
> Attachments: screenshot-1.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When verifyAndSetNamespaceInfo() is executed, the log will record some 
> information. E.g:
> '
> 2021-10-27 18:08:36,242 [50867]-INFO 
> [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
> handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
> 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
> .xxx.xxx.org/.xxx.xxx.xxx:8021
> '
> Here, the connection between the'handshake' and the'Block pool' is too tight, 
> and the readability is not good.
>  !screenshot-1.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16266) Add remote port information to HDFS audit log

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16266?focusedWorklogId=671871=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671871
 ]

ASF GitHub Bot logged work on HDFS-16266:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 08:38
Start Date: 29/Oct/21 08:38
Worklog Time Spent: 10m 
  Work Description: tasanuma commented on pull request #3538:
URL: https://github.com/apache/hadoop/pull/3538#issuecomment-954558413


   How about using the word of "port" instead of "clientPort" here, and adding 
"clientPort" as the actual client server port for RBF in another JIRA?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 671871)
Time Spent: 6h  (was: 5h 50m)

> Add remote port information to HDFS audit log
> -
>
> Key: HDFS-16266
> URL: https://issues.apache.org/jira/browse/HDFS-16266
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> In our production environment, we occasionally encounter a problem where a 
> user submits an abnormal computation task, causing a sudden flood of 
> requests, which causes the queueTime and processingTime of the Namenode to 
> rise very high, causing a large backlog of tasks.
> We usually locate and kill specific Spark, Flink, or MapReduce tasks based on 
> metrics and audit logs. Currently, IP and UGI are recorded in audit logs, but 
> there is no port information, so it is difficult to locate specific processes 
> sometimes. Therefore, I propose that we add the port information to the audit 
> log, so that we can easily track the upstream process.
> Currently, some projects contain port information in audit logs, such as 
> Hbase and Alluxio. I think it is also necessary to add port information for 
> HDFS audit logs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16266) Add remote port information to HDFS audit log

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16266?focusedWorklogId=671868=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671868
 ]

ASF GitHub Bot logged work on HDFS-16266:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 08:19
Start Date: 29/Oct/21 08:19
Worklog Time Spent: 10m 
  Work Description: tasanuma commented on pull request #3538:
URL: https://github.com/apache/hadoop/pull/3538#issuecomment-954546028


   Thanks for updating it, @tomscut.
   I tried it with my RBF cluster. There is a client server (1.1.1.1), a DFS 
Router (2.2.2.2), and NameNode. When a client sends a request to the Router, 
NameNode logs the following.
   ```
   INFO FSNamesystem.audit: allowed=true   ugi=tasanuma ip=/2.2.2.2   
cmd=listStatus  src=/user/tasanuma  dst=nullperm=null   
proto=rpc   callerContext=CLI,clientIp:1.1.1.1,clientPort:33070
   ```
   In this case, `clientIp:1.1.1.1` is the IP of the client server, but 
`clientPort:33070` is the port of the DFS Router (2.2.2.2), not the one of the 
client server. It would be confusing for the users.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 671868)
Time Spent: 5h 50m  (was: 5h 40m)

> Add remote port information to HDFS audit log
> -
>
> Key: HDFS-16266
> URL: https://issues.apache.org/jira/browse/HDFS-16266
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> In our production environment, we occasionally encounter a problem where a 
> user submits an abnormal computation task, causing a sudden flood of 
> requests, which causes the queueTime and processingTime of the Namenode to 
> rise very high, causing a large backlog of tasks.
> We usually locate and kill specific Spark, Flink, or MapReduce tasks based on 
> metrics and audit logs. Currently, IP and UGI are recorded in audit logs, but 
> there is no port information, so it is difficult to locate specific processes 
> sometimes. Therefore, I propose that we add the port information to the audit 
> log, so that we can easily track the upstream process.
> Currently, some projects contain port information in audit logs, such as 
> Hbase and Alluxio. I think it is also necessary to add port information for 
> HDFS audit logs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-16259) Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger)

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HDFS-16259:
--
Labels: pull-request-available  (was: )

> Catch and re-throw sub-classes of AccessControlException thrown by any 
> permission provider plugins (eg Ranger)
> --
>
> Key: HDFS-16259
> URL: https://issues.apache.org/jira/browse/HDFS-16259
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When a permission provider plugin is enabled (eg Ranger) there are some 
> scenarios where it can throw a sub-class of an AccessControlException (eg 
> RangerAccessControlException). If this exception is allowed to propagate up 
> the stack, it can give problems in the HDFS Client, when it unwraps the 
> remote exception containing the AccessControlException sub-class.
> Ideally, we should make AccessControlException final so it cannot be 
> sub-classed, but that would be a breaking change at this point. Therefore I 
> believe the safest thing to do, is to catch any AccessControlException that 
> comes out of the permission enforcer plugin, and re-throw an 
> AccessControlException instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16259) Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger)

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16259?focusedWorklogId=671860=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671860
 ]

ASF GitHub Bot logged work on HDFS-16259:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 08:06
Start Date: 29/Oct/21 08:06
Worklog Time Spent: 10m 
  Work Description: sodonnel commented on pull request #3598:
URL: https://github.com/apache/hadoop/pull/3598#issuecomment-954527321


   There is a related test failure in TestPermissionSymlink.testAccess, as it 
actually throws a TraverseAccessControlException, which is being caught and 
then re-thrown as an AccessControlException.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 671860)
Remaining Estimate: 0h
Time Spent: 10m

> Catch and re-throw sub-classes of AccessControlException thrown by any 
> permission provider plugins (eg Ranger)
> --
>
> Key: HDFS-16259
> URL: https://issues.apache.org/jira/browse/HDFS-16259
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Reporter: Stephen O'Donnell
>Assignee: Stephen O'Donnell
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When a permission provider plugin is enabled (eg Ranger) there are some 
> scenarios where it can throw a sub-class of an AccessControlException (eg 
> RangerAccessControlException). If this exception is allowed to propagate up 
> the stack, it can give problems in the HDFS Client, when it unwraps the 
> remote exception containing the AccessControlException sub-class.
> Ideally, we should make AccessControlException final so it cannot be 
> sub-classed, but that would be a breaking change at this point. Therefore I 
> believe the safest thing to do, is to catch any AccessControlException that 
> comes out of the permission enforcer plugin, and re-throw an 
> AccessControlException instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-16290) Make log more standardized when executing verifyAndSetNamespaceInfo()

[JiangHua Zhu (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

JiangHua Zhu updated HDFS-16290:

Affects Version/s: 2.9.2

> Make log more standardized when executing verifyAndSetNamespaceInfo()
> -
>
> Key: HDFS-16290
> URL: https://issues.apache.org/jira/browse/HDFS-16290
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 2.9.2
>Reporter: JiangHua Zhu
>Assignee: JiangHua Zhu
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> When verifyAndSetNamespaceInfo() is executed, the log will record some 
> information. E.g:
> '
> 2021-10-27 18:08:36,242 [50867]-INFO 
> [Thread-33:BPOfferService@376]-Acknowledging ACTIVE Namenode during 
> handshakeBlock pool BP-597961518-xxx.xxx.xxx.xxx-1534758275943 (Datanode Uuid 
> 9b2aedc9-f8b2 -4ee2-b99f-877bc6e42c87) service to 
> .xxx.xxx.org/.xxx.xxx.xxx:8021
> '
> Here, the connection between the'handshake' and the'Block pool' is too tight, 
> and the readability is not good.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Work logged] (HDFS-16283) RBF: improve renewLease() to call only a specific NameNode rather than make fan-out calls

[ASF GitHub Bot (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16283?focusedWorklogId=671838=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-671838
 ]

ASF GitHub Bot logged work on HDFS-16283:
-

Author: ASF GitHub Bot
Created on: 29/Oct/21 06:31
Start Date: 29/Oct/21 06:31
Worklog Time Spent: 10m 
  Work Description: Hexiaoqiao commented on pull request #3595:
URL: https://github.com/apache/hadoop/pull/3595#issuecomment-954466025


   Thanks for bringing this improvement. Sorry I do not receive discussion in 
mail-list and reply here directly.
   I am not sure if it is the best solution to expose namespace id to client 
which will confuse the end-user in my opinion.
   `If we are adding new protocols, maybe we can add "renewLease(String 
clientName, String path)" and let Router do the resolve from path to namespace.`
   In my practice, I think `path` will be the good choice here. Strong +1. And 
router could resolve it which keep the same logic for other interface. But the 
bad news is that it will increase the invoke interface times. Anyway, I think 
this solution will not break anything.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 671838)
Time Spent: 50m  (was: 40m)

> RBF: improve renewLease() to call only a specific NameNode rather than make 
> fan-out calls
> -
>
> Key: HDFS-16283
> URL: https://issues.apache.org/jira/browse/HDFS-16283
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: rbf
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Currently renewLease() against a router will make fan-out to all the 
> NameNodes. Since renewLease() call is so frequent and if one of the NameNodes 
> are slow, then eventually the router queues are blocked by all renewLease() 
> and cause router degradation. 
> We will make a change in the client side to keep track of NameNode Id in 
> additional to current fileId so routers understand which NameNodes the client 
> is renewing lease against.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org