[jira] [Updated] (HADOOP-16695) Make LogThrottlingHelper thread-safe
[
https://issues.apache.org/jira/browse/HADOOP-16695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Zhang updated HADOOP-16695:
Description: HADOOP-15726 introduced the \{{LogThrottlingHelper}}, but this
class is not thread-safe, which limits its usage scenarios, this Jira will try
to improve it.
> Make LogThrottlingHelper thread-safe
>
>
> Key: HADOOP-16695
> URL: https://issues.apache.org/jira/browse/HADOOP-16695
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
>
> HADOOP-15726 introduced the \{{LogThrottlingHelper}}, but this class is not
> thread-safe, which limits its usage scenarios, this Jira will try to improve
> it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-16695) Make LogThrottlingHelper thread-safe
[ https://issues.apache.org/jira/browse/HADOOP-16695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Zhang reassigned HADOOP-16695: --- Assignee: Chen Zhang > Make LogThrottlingHelper thread-safe > > > Key: HADOOP-16695 > URL: https://issues.apache.org/jira/browse/HADOOP-16695 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Reporter: Chen Zhang >Assignee: Chen Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16695) Make LogThrottlingHelper thread-safe
Chen Zhang created HADOOP-16695: --- Summary: Make LogThrottlingHelper thread-safe Key: HADOOP-16695 URL: https://issues.apache.org/jira/browse/HADOOP-16695 Project: Hadoop Common Issue Type: Improvement Components: common Reporter: Chen Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16531) Log more detail for slow RPC
[
https://issues.apache.org/jira/browse/HADOOP-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943289#comment-16943289
]
Chen Zhang commented on HADOOP-16531:
-
Thanks [~weichiu] for the commit.
> Log more detail for slow RPC
>
>
> Key: HADOOP-16531
> URL: https://issues.apache.org/jira/browse/HADOOP-16531
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HADOOP-16531.001.patch
>
>
> Current implementation only log process time
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
> LOG.warn("Slow RPC : {} took {} {} to process from client {}",
> methodName, processingTime, RpcMetrics.TIMEUNIT, call);
> rpcMetrics.incrSlowRpc();
> }
> {code}
> We need to log more details to help us locate the problem (eg. how long it
> take to request lock, holding lock, or do other things)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16531) Log more detail for slow RPC
[
https://issues.apache.org/jira/browse/HADOOP-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924716#comment-16924716
]
Chen Zhang commented on HADOOP-16531:
-
Thanks [~xkrogen] for the review and the commit
> Log more detail for slow RPC
>
>
> Key: HADOOP-16531
> URL: https://issues.apache.org/jira/browse/HADOOP-16531
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-16531.001.patch
>
>
> Current implementation only log process time
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
> LOG.warn("Slow RPC : {} took {} {} to process from client {}",
> methodName, processingTime, RpcMetrics.TIMEUNIT, call);
> rpcMetrics.incrSlowRpc();
> }
> {code}
> We need to log more details to help us locate the problem (eg. how long it
> take to request lock, holding lock, or do other things)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16531) Log more detail for slow RPC
[
https://issues.apache.org/jira/browse/HADOOP-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922999#comment-16922999
]
Chen Zhang commented on HADOOP-16531:
-
[~ayushtkn] [~xkrogen], do you have time to help review this patch? Thanks
> Log more detail for slow RPC
>
>
> Key: HADOOP-16531
> URL: https://issues.apache.org/jira/browse/HADOOP-16531
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HADOOP-16531.001.patch
>
>
> Current implementation only log process time
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
> LOG.warn("Slow RPC : {} took {} {} to process from client {}",
> methodName, processingTime, RpcMetrics.TIMEUNIT, call);
> rpcMetrics.incrSlowRpc();
> }
> {code}
> We need to log more details to help us locate the problem (eg. how long it
> take to request lock, holding lock, or do other things)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15726) Create utility to limit frequency of log statements
[
https://issues.apache.org/jira/browse/HADOOP-15726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922982#comment-16922982
]
Chen Zhang commented on HADOOP-15726:
-
Thanks [~xkrogen] for your response, I'd like to try to follow the enhancement
of log throttle on read-lock, will file a Jira later.
> Create utility to limit frequency of log statements
> ---
>
> Key: HADOOP-15726
> URL: https://issues.apache.org/jira/browse/HADOOP-15726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, util
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15726.000.patch, HADOOP-15726.001.patch,
> HADOOP-15726.002.patch, HADOOP-15726.003.patch,
> HDFS-15726-branch-3.0.003.patch
>
>
> There is a common pattern of logging a behavior that is normally extraneous.
> Under some circumstances, such a behavior becomes common, flooding the logs
> and making it difficult to see what else is going on in the system. Under
> such situations it is beneficial to limit how frequently the extraneous
> behavior is logged, while capturing some summary information about the
> suppressed log statements.
> This is currently implemented in {{FSNamesystemLock}} (in HDFS-10713). We
> have additional use cases for this in HDFS-13791, so this is a good time to
> create a common utility for different sites to share this logic.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16531) Log more detail for slow RPC
[
https://issues.apache.org/jira/browse/HADOOP-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Zhang updated HADOOP-16531:
Status: Patch Available (was: Open)
upload patch v1.
> Log more detail for slow RPC
>
>
> Key: HADOOP-16531
> URL: https://issues.apache.org/jira/browse/HADOOP-16531
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HADOOP-16531.001.patch
>
>
> Current implementation only log process time
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
> LOG.warn("Slow RPC : {} took {} {} to process from client {}",
> methodName, processingTime, RpcMetrics.TIMEUNIT, call);
> rpcMetrics.incrSlowRpc();
> }
> {code}
> We need to log more details to help us locate the problem (eg. how long it
> take to request lock, holding lock, or do other things)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16531) Log more detail for slow RPC
[
https://issues.apache.org/jira/browse/HADOOP-16531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Zhang updated HADOOP-16531:
Attachment: HADOOP-16531.001.patch
> Log more detail for slow RPC
>
>
> Key: HADOOP-16531
> URL: https://issues.apache.org/jira/browse/HADOOP-16531
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Chen Zhang
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HADOOP-16531.001.patch
>
>
> Current implementation only log process time
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
> LOG.warn("Slow RPC : {} took {} {} to process from client {}",
> methodName, processingTime, RpcMetrics.TIMEUNIT, call);
> rpcMetrics.incrSlowRpc();
> }
> {code}
> We need to log more details to help us locate the problem (eg. how long it
> take to request lock, holding lock, or do other things)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15726) Create utility to limit frequency of log statements
[
https://issues.apache.org/jira/browse/HADOOP-15726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915487#comment-16915487
]
Chen Zhang commented on HADOOP-15726:
-
Hi [~xkrogen], the patch of this Jira only throttle log of write-lock, don't
consider log of read-lock, any consideration here?
> Create utility to limit frequency of log statements
> ---
>
> Key: HADOOP-15726
> URL: https://issues.apache.org/jira/browse/HADOOP-15726
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, util
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-15726.000.patch, HADOOP-15726.001.patch,
> HADOOP-15726.002.patch, HADOOP-15726.003.patch,
> HDFS-15726-branch-3.0.003.patch
>
>
> There is a common pattern of logging a behavior that is normally extraneous.
> Under some circumstances, such a behavior becomes common, flooding the logs
> and making it difficult to see what else is going on in the system. Under
> such situations it is beneficial to limit how frequently the extraneous
> behavior is logged, while capturing some summary information about the
> suppressed log statements.
> This is currently implemented in {{FSNamesystemLock}} (in HDFS-10713). We
> have additional use cases for this in HDFS-13791, so this is a good time to
> create a common utility for different sites to share this logic.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16531) Log more detail for slow RPC
Chen Zhang created HADOOP-16531:
---
Summary: Log more detail for slow RPC
Key: HADOOP-16531
URL: https://issues.apache.org/jira/browse/HADOOP-16531
Project: Hadoop Common
Issue Type: Improvement
Reporter: Chen Zhang
Assignee: Chen Zhang
Current implementation only log process time
{code:java}
if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
(processingTime > threeSigma)) {
LOG.warn("Slow RPC : {} took {} {} to process from client {}",
methodName, processingTime, RpcMetrics.TIMEUNIT, call);
rpcMetrics.incrSlowRpc();
}
{code}
We need to log more details to help us locate the problem (eg. how long it take
to request lock, holding lock, or do other things)
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-16528) Update document for web authentication kerberos principal
[
https://issues.apache.org/jira/browse/HADOOP-16528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913260#comment-16913260
]
Chen Zhang edited comment on HADOOP-16528 at 8/22/19 12:05 PM:
---
Hi [~jojochuang], I can't assign the Jira to myself, could you help? Thanks!
was (Author: zhangchen):
[~jojochuang], I can't assign the Jira to myself, could help? Thanks!
> Update document for web authentication kerberos principal
> -
>
> Key: HADOOP-16528
> URL: https://issues.apache.org/jira/browse/HADOOP-16528
> Project: Hadoop Common
> Issue Type: Bug
> Components: auth
>Reporter: Chen Zhang
>Priority: Major
>
> The config \{{dfs.web.authentication.kerberos.principal}} is not used anymore
> after HADOOP-16354, but the document for WebHDFS is not updated, the
> hdfs-default.xml should be updated as well.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-16528) Update document for web authentication kerberos principal configuration
[
https://issues.apache.org/jira/browse/HADOOP-16528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Zhang updated HADOOP-16528:
Summary: Update document for web authentication kerberos principal
configuration (was: Update document for web authentication kerberos principal)
> Update document for web authentication kerberos principal configuration
> ---
>
> Key: HADOOP-16528
> URL: https://issues.apache.org/jira/browse/HADOOP-16528
> Project: Hadoop Common
> Issue Type: Bug
> Components: auth
>Reporter: Chen Zhang
>Priority: Major
>
> The config \{{dfs.web.authentication.kerberos.principal}} is not used anymore
> after HADOOP-16354, but the document for WebHDFS is not updated, the
> hdfs-default.xml should be updated as well.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16528) Update document for web authentication kerberos principal
[
https://issues.apache.org/jira/browse/HADOOP-16528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913260#comment-16913260
]
Chen Zhang commented on HADOOP-16528:
-
[~jojochuang], I can't assign the Jira to myself, could help? Thanks!
> Update document for web authentication kerberos principal
> -
>
> Key: HADOOP-16528
> URL: https://issues.apache.org/jira/browse/HADOOP-16528
> Project: Hadoop Common
> Issue Type: Bug
> Components: auth
>Reporter: Chen Zhang
>Priority: Major
>
> The config \{{dfs.web.authentication.kerberos.principal}} is not used anymore
> after HADOOP-16354, but the document for WebHDFS is not updated, the
> hdfs-default.xml should be updated as well.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16528) Update document for web authentication kerberos principal
Chen Zhang created HADOOP-16528:
---
Summary: Update document for web authentication kerberos principal
Key: HADOOP-16528
URL: https://issues.apache.org/jira/browse/HADOOP-16528
Project: Hadoop Common
Issue Type: Bug
Components: auth
Reporter: Chen Zhang
The config \{{dfs.web.authentication.kerberos.principal}} is not used anymore
after HADOOP-16354, but the document for WebHDFS is not updated, the
hdfs-default.xml should be updated as well.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
