[
https://issues.apache.org/jira/browse/HDFS-13613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969596#comment-16969596
]
Michael Stack commented on HDFS-13613:
--------------------------------------
Thanks [~ndimiduk] and [~inigoiri] for taking a look.
Thanks for your experience disabling hedged reads. Will try that too.
I could add a check for DEBUG but just doing check in this old logging system
of ours -- last release was more than 5 years ago -- requires our passing
across a synchronized block. When this was log was spewing in a running
process, as it will tend to do when HDFS is struggling and all threads are
waiting on HDFS syncs to return, I changed the log level but then saw access to
HDFS blocking on the log level-check (thread dumping so rough take only).
Limiting the number of emissions would require a system to count and it'd have
to be configurable, and so on. Seems a bit OTT.
I was thinking that if you are interested in thread count for hedged reads,
you'd study the metrics incremented on the line that follows; it'd give you
better notion than what this bare log does.
Thanks again for taking a look.
> RegionServer log is flooded with "Execution rejected, Executing in current
> thread"
> ----------------------------------------------------------------------------------
>
> Key: HDFS-13613
> URL: https://issues.apache.org/jira/browse/HDFS-13613
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Environment: CDH 5.13, HBase RegionServer, Kerberized, hedged read
> Reporter: Wei-Chiu Chuang
> Priority: Major
> Attachments:
> 0001-HDFS-13613-RegionServer-log-is-flooded-with-Executio.patch
>
>
> In the log of a HBase RegionServer with hedged read, we saw the following
> message flooding the log file.
> {noformat}
> 2018-05-19 17:22:55,691 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,692 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,695 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,696 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,696 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> ....
> {noformat}
> Sometimes the RS spits tens of thousands of lines of this message in a
> minute. We should do something to stop this message flooding the log file.
> Also, we should make this message more actionable. Discussed with
> [~huaxiang], this message can appear if there are stale DataNodes.
> I believe this issue existed since HDFS-5776.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
