[
https://issues.apache.org/jira/browse/HDFS-13613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969719#comment-16969719
]
Duo Zhang commented on HDFS-13613:
----------------------------------
+1 on removing the log. I believe if this happens, it will always flood the log
file, as this means we are overloaded, and flood the log file with this message
will make things even worse...
A better way is to implement a counter for this event, and log periodically
about this event with some numbers. Can do this in a follow on issue?
> RegionServer log is flooded with "Execution rejected, Executing in current
> thread"
> ----------------------------------------------------------------------------------
>
> Key: HDFS-13613
> URL: https://issues.apache.org/jira/browse/HDFS-13613
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Environment: CDH 5.13, HBase RegionServer, Kerberized, hedged read
> Reporter: Wei-Chiu Chuang
> Priority: Major
> Attachments:
> 0001-HDFS-13613-RegionServer-log-is-flooded-with-Executio.patch
>
>
> In the log of a HBase RegionServer with hedged read, we saw the following
> message flooding the log file.
> {noformat}
> 2018-05-19 17:22:55,691 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,692 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,695 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,696 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,696 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> ....
> {noformat}
> Sometimes the RS spits tens of thousands of lines of this message in a
> minute. We should do something to stop this message flooding the log file.
> Also, we should make this message more actionable. Discussed with
> [~huaxiang], this message can appear if there are stale DataNodes.
> I believe this issue existed since HDFS-5776.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
