[jira] [Comment Edited] (HDFS-16028) Add a configuration item for special trash dir

[ https://issues.apache.org/jira/browse/HDFS-16028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17347406#comment-17347406 ] Qi Zhu edited comment on HDFS-16028 at 5/19/21, 8:24 AM: - Thanks

[jira] [Commented] (HDFS-16028) Add a configuration item for special trash dir

[ https://issues.apache.org/jira/browse/HDFS-16028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17347406#comment-17347406 ] Qi Zhu commented on HDFS-16028: --- Thanks [~zhengzhuobinzzb] for patch. We should also add the new conf in

[jira] [Commented] (HDFS-8708) DFSClient should ignore dfs.client.retry.policy.enabled for HA proxies

[ https://issues.apache.org/jira/browse/HDFS-8708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17346631#comment-17346631 ] Qi Zhu commented on HDFS-8708: -- cc [~hexiaoqiao] [~ayushtkn]  I agree with [~chengbing.liu], we should

[jira] [Commented] (HDFS-16003) ProcessReport print invalidatedBlocks should judge debug level at first

[ https://issues.apache.org/jira/browse/HDFS-16003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17341776#comment-17341776 ] Qi Zhu commented on HDFS-16003: --- LGTM +1.  > ProcessReport print invalidatedBlocks should judge debug

[jira] [Commented] (HDFS-15968) Improve the log for The DecayRpcScheduler

[ https://issues.apache.org/jira/browse/HDFS-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17341675#comment-17341675 ] Qi Zhu commented on HDFS-15968: --- Thanks [~bpatel] for cotribution. The path LGTM +1. > Improve the log

[jira] [Comment Edited] (HDFS-15994) Deletion should sleep some time, when there are too many pending deletion blocks.

[ https://issues.apache.org/jira/browse/HDFS-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340857#comment-17340857 ] Qi Zhu edited comment on HDFS-15994 at 5/7/21, 2:08 PM: Thanks a lot

[jira] [Commented] (HDFS-15994) Deletion should sleep some time, when there are too many pending deletion blocks.

[ https://issues.apache.org/jira/browse/HDFS-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340857#comment-17340857 ] Qi Zhu commented on HDFS-15994: --- Thanks a lot [~hexiaoqiao] for reply. The deletion process: 1）Namespace

[jira] [Commented] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.

[ https://issues.apache.org/jira/browse/HDFS-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339963#comment-17339963 ] Qi Zhu commented on HDFS-15934: --- Thanks [~ayushtkn] for commit. > Make DirectoryScanner reconcile blocks

[jira] [Commented] (HDFS-13904) ContentSummary does not always respect processing limit, resulting in long lock acquisitions

[ https://issues.apache.org/jira/browse/HDFS-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331878#comment-17331878 ] Qi Zhu commented on HDFS-13904: --- Is this going on? I met the problem also.-- Thanks. > ContentSummary

[jira] [Comment Edited] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331414#comment-17331414 ] Qi Zhu edited comment on HDFS-14617 at 4/25/21, 5:49 AM: - cc [~weichiu]

[jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index

[ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17331414#comment-17331414 ] Qi Zhu commented on HDFS-14617: --- cc [~sodonnell] [~hexiaoqiao]  Could you help backport to 3.2.2 and 3.2.1

[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable

[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327227#comment-17327227 ] Qi Zhu commented on HDFS-13831: --- [~weichiu] [~linyiqun] [~gaofeng6]  I created HDFS-15994 to improve this

[jira] [Comment Edited] (HDFS-15994) Deletion should sleep some time, when there are too many pending deletion blocks.

[ https://issues.apache.org/jira/browse/HDFS-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327218#comment-17327218 ] Qi Zhu edited comment on HDFS-15994 at 4/22/21, 9:50 AM: - cc [~weichiu] 

[jira] [Commented] (HDFS-15994) Deletion should sleep some time, when there are too many pending deletion blocks.

[ https://issues.apache.org/jira/browse/HDFS-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327218#comment-17327218 ] Qi Zhu commented on HDFS-15994: --- cc [~weichiu] [~hexiaoqiao] [~sodonnell] [~ayushtkn]  [~linyiqun]

[jira] [Updated] (HDFS-15994) Deletion should sleep some time, when there are too many pending deletion blocks.

[ https://issues.apache.org/jira/browse/HDFS-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15994: -- Attachment: HDFS-15994.001.patch Status: Patch Available (was: Open) > Deletion should sleep some

[jira] [Updated] (HDFS-15994) Deletion should sleep some time, when there are too many pending deletion blocks.

[ https://issues.apache.org/jira/browse/HDFS-15994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15994: -- Description: HDFS-13831 realize that we can control the frequency of other waiters to get the lock chance.

[jira] [Created] (HDFS-15994) Deletion should sleep some time, when there are too many pending deletion blocks.

Qi Zhu created HDFS-15994: - Summary: Deletion should sleep some time, when there are too many pending deletion blocks. Key: HDFS-15994 URL: https://issues.apache.org/jira/browse/HDFS-15994 Project: Hadoop

[jira] [Comment Edited] (HDFS-14525) JspHelper ignores hadoop.http.authentication.type

[ https://issues.apache.org/jira/browse/HDFS-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327104#comment-17327104 ] Qi Zhu edited comment on HDFS-14525 at 4/22/21, 6:52 AM: - [~prabhujoseph] 

[jira] [Commented] (HDFS-14525) JspHelper ignores hadoop.http.authentication.type

[ https://issues.apache.org/jira/browse/HDFS-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327104#comment-17327104 ] Qi Zhu commented on HDFS-14525: --- [~prabhujoseph]  I also think this is needed, if we can add an option to

[jira] [Commented] (HDFS-15601) Batch listing: gracefully fallback to use non-batched listing when NameNode doesn't support the feature

[ https://issues.apache.org/jira/browse/HDFS-15601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314657#comment-17314657 ] Qi Zhu commented on HDFS-15601: --- [~ayushtkn] [~hexiaoqiao] Could you help review this when you are free?

[jira] [Commented] (HDFS-15601) Batch listing: gracefully fallback to use non-batched listing when NameNode doesn't support the feature

[ https://issues.apache.org/jira/browse/HDFS-15601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17314516#comment-17314516 ] Qi Zhu commented on HDFS-15601: --- I will take it,  since still no one take. > Batch listing: gracefully

[jira] [Updated] (HDFS-15601) Batch listing: gracefully fallback to use non-batched listing when NameNode doesn't support the feature

[ https://issues.apache.org/jira/browse/HDFS-15601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15601: -- Attachment: HDFS-15601.001.patch Status: Patch Available (was: Open) > Batch listing: gracefully

[jira] [Assigned] (HDFS-15601) Batch listing: gracefully fallback to use non-batched listing when NameNode doesn't support the feature

[ https://issues.apache.org/jira/browse/HDFS-15601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu reassigned HDFS-15601: - Assignee: Qi Zhu > Batch listing: gracefully fallback to use non-batched listing when NameNode >

[jira] [Commented] (HDFS-15924) Log4j will cause Server handler blocked when audit log boom.

[ https://issues.apache.org/jira/browse/HDFS-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311354#comment-17311354 ] Qi Zhu commented on HDFS-15924: --- Thanks a lot [~weichiu] for reply, i think  HDFS-15720 may give me a

[jira] [Updated] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.

[ https://issues.apache.org/jira/browse/HDFS-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15934: -- Issue Type: Improvement (was: Bug) > Make DirectoryScanner reconcile blocks batch size and interval between

[jira] [Updated] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.

[ https://issues.apache.org/jira/browse/HDFS-15934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15934: -- Status: Patch Available (was: Open) > Make DirectoryScanner reconcile blocks batch size and interval between

[jira] [Created] (HDFS-15934) Make DirectoryScanner reconcile blocks batch size and interval between batch configurable.

Qi Zhu created HDFS-15934: - Summary: Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Key: HDFS-15934 URL: https://issues.apache.org/jira/browse/HDFS-15934 Project:

[jira] [Updated] (HDFS-15930) Fix some @param errors in DirectoryScanner.

[ https://issues.apache.org/jira/browse/HDFS-15930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15930: -- Status: Patch Available (was: Open) > Fix some @param errors in DirectoryScanner. >

[jira] [Created] (HDFS-15930) Fix some @param errors in DirectoryScanner.

Qi Zhu created HDFS-15930: - Summary: Fix some @param errors in DirectoryScanner. Key: HDFS-15930 URL: https://issues.apache.org/jira/browse/HDFS-15930 Project: Hadoop HDFS Issue Type: Bug

[jira] [Commented] (HDFS-15924) Log4j will cause Server handler blocked when audit log boom.

[ https://issues.apache.org/jira/browse/HDFS-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309442#comment-17309442 ] Qi Zhu commented on HDFS-15924: --- [~marvelrock] The version of this cluster is 2.6.0-cdh5.11.0. > Log4j

[jira] [Commented] (HDFS-15924) Log4j will cause Server handler blocked when audit log boom.

[ https://issues.apache.org/jira/browse/HDFS-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309403#comment-17309403 ] Qi Zhu commented on HDFS-15924: --- Thanks [~sodonnell] for reply. Actually the async audit logging is

[jira] [Updated] (HDFS-15924) Log4j will cause Server handler blocked when audit log boom.

[ https://issues.apache.org/jira/browse/HDFS-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15924: -- Description: !image-2021-03-26-16-18-03-341.png|width=707,height=234!

[jira] [Created] (HDFS-15924) Log4j will cause Server handler blocked when audit log boom.

Qi Zhu created HDFS-15924: - Summary: Log4j will cause Server handler blocked when audit log boom. Key: HDFS-15924 URL: https://issues.apache.org/jira/browse/HDFS-15924 Project: Hadoop HDFS Issue

[jira] [Comment Edited] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.

[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297235#comment-17297235 ] Qi Zhu edited comment on HDFS-15874 at 3/10/21, 2:37 PM: - [~leosun08] [~linyiqun] 

[jira] [Commented] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.

[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297235#comment-17297235 ] Qi Zhu commented on HDFS-15874: --- [~weichiu] [~hexiaoqiao] [~ayushtkn]  [~shv] [~ferhui] Could you help

[jira] [Commented] (HDFS-15879) Exclude slow nodes when choose targets for blocks

[ https://issues.apache.org/jira/browse/HDFS-15879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17297228#comment-17297228 ] Qi Zhu commented on HDFS-15879: --- Thanks [~tomscut] for proposal. It seems duplicate with HDFS-14789.   >

[jira] [Updated] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.

[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15874: -- Status: Patch Available (was: In Progress) > Extend TopMetrics to support callerContext aggregation. >

[jira] [Updated] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.

[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15874: -- Issue Type: Improvement (was: New Feature) > Extend TopMetrics to support callerContext aggregation. >

[jira] [Updated] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.

[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15874: -- Description: Now top metrics for namenode op is only supported aggregation for user,  the top user op is

[jira] [Commented] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.

[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295744#comment-17295744 ] Qi Zhu commented on HDFS-15874: --- Tested the poc pull request in my local cluster. 

[jira] [Updated] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.

[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15874: -- Attachment: image-2021-03-05-12-01-16-852.png > Extend TopMetrics to support callerContext aggregation. >

[jira] [Work started] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.

[ https://issues.apache.org/jira/browse/HDFS-15874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-15874 started by Qi Zhu. - > Extend TopMetrics to support callerContext aggregation. >

[jira] [Created] (HDFS-15874) Extend TopMetrics to support callerContext aggregation.

Qi Zhu created HDFS-15874: - Summary: Extend TopMetrics to support callerContext aggregation. Key: HDFS-15874 URL: https://issues.apache.org/jira/browse/HDFS-15874 Project: Hadoop HDFS Issue Type:

[jira] [Commented] (HDFS-15856) Make recover the pipeline in same packet exceed times for stream closed configurable.

[ https://issues.apache.org/jira/browse/HDFS-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17292964#comment-17292964 ] Qi Zhu commented on HDFS-15856: --- [~hexiaoqiao] [~ayushtkn] Could you help review this? Thanks. > Make

[jira] [Comment Edited] (HDFS-15856) Make recover the pipeline in same packet exceed times for stream closed configurable.

[ https://issues.apache.org/jira/browse/HDFS-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291417#comment-17291417 ] Qi Zhu edited comment on HDFS-15856 at 2/26/21, 6:33 AM: - [~ayushtkn]

[jira] [Commented] (HDFS-15856) Make recover the pipeline in same packet exceed times for stream closed configurable.

[ https://issues.apache.org/jira/browse/HDFS-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17291417#comment-17291417 ] Qi Zhu commented on HDFS-15856: --- [~ayushtkn] [~hexiaoqiao]  If you any other advice? Thanks. > Make

[jira] [Commented] (HDFS-15180) DataNode FsDatasetImpl Fine-Grained Locking via BlockPool.

[ https://issues.apache.org/jira/browse/HDFS-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290808#comment-17290808 ] Qi Zhu commented on HDFS-15180: --- Thanks [~hexiaoqiao] for your reply, great work as expected, i will help

[jira] [Created] (HDFS-15856) Make recover the pipeline in same packet exceed times for stream closed configurable.

Qi Zhu created HDFS-15856: - Summary: Make recover the pipeline in same packet exceed times for stream closed configurable. Key: HDFS-15856 URL: https://issues.apache.org/jira/browse/HDFS-15856 Project:

[jira] [Commented] (HDFS-15180) DataNode FsDatasetImpl Fine-Grained Locking via BlockPool.

[ https://issues.apache.org/jira/browse/HDFS-15180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289853#comment-17289853 ] Qi Zhu commented on HDFS-15180: --- [~hexiaoqiao] [~sodonnell] [~Aiphag0] Is it going on, i think we can push

[jira] [Commented] (HDFS-15584) Improve HDFS large deletion cause namenode lockqueue boom and pending deletion boom.

[ https://issues.apache.org/jira/browse/HDFS-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289721#comment-17289721 ] Qi Zhu commented on HDFS-15584: --- [~sodonnell]  [~hexiaoqiao]  If you any other advice about this? It will

[jira] [Commented] (HDFS-15849) ExpiredHeartbeats metric should be of Type.COUNTER

[ https://issues.apache.org/jira/browse/HDFS-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288998#comment-17288998 ] Qi Zhu commented on HDFS-15849: --- [~shv] [~tomscut] Fixed the metric related test in latest patch. >

[jira] [Updated] (HDFS-15849) ExpiredHeartbeats metric should be of Type.COUNTER

[ https://issues.apache.org/jira/browse/HDFS-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15849: -- Attachment: HDFS-15849.002.patch > ExpiredHeartbeats metric should be of Type.COUNTER >

[jira] [Commented] (HDFS-15808) Add metrics for FSNamesystem read/write lock hold long time

[ https://issues.apache.org/jira/browse/HDFS-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288982#comment-17288982 ] Qi Zhu commented on HDFS-15808: --- Thanks [~tomscut] for contribution, patch LGTM. +1  > Add metrics for

[jira] [Comment Edited] (HDFS-15849) ExpiredHeartbeats metric should be of Type.COUNTER

[ https://issues.apache.org/jira/browse/HDFS-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288849#comment-17288849 ] Qi Zhu edited comment on HDFS-15849 at 2/23/21, 5:27 AM: - [~shv] [~xkrogen]

[jira] [Updated] (HDFS-15849) ExpiredHeartbeats metric should be of Type.COUNTER

[ https://issues.apache.org/jira/browse/HDFS-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu updated HDFS-15849: -- Attachment: HDFS-15849.001.patch Status: Patch Available (was: Open) > ExpiredHeartbeats metric

[jira] [Commented] (HDFS-15849) ExpiredHeartbeats metric should be of Type.COUNTER

[ https://issues.apache.org/jira/browse/HDFS-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288849#comment-17288849 ] Qi Zhu commented on HDFS-15849: --- [~shv]  In our production cluster, we also want to delta it to show in

[jira] [Assigned] (HDFS-15849) ExpiredHeartbeats metric should be of Type.COUNTER

[ https://issues.apache.org/jira/browse/HDFS-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qi Zhu reassigned HDFS-15849: - Assignee: Qi Zhu > ExpiredHeartbeats metric should be of Type.COUNTER >