[
https://issues.apache.org/jira/browse/HDFS-13613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16967021#comment-16967021
]
Michael Stack commented on HDFS-13613:
--------------------------------------
Just ran into this one. Thread dump showded loads of threads BLOCKED here:
{code}
"RpcServer.default.FPBQ.Fifo.handler=85,queue=25,port=16020" #137 daemon prio=5
os_prio=0 cpu=85786.24ms elapsed=157927.35s tid=0x00007f3dddad6000 nid=0xf390
waiting for monitor entry [0x00007f3dd21a9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.log4j.Category.callAppenders(Category.java:204)
- waiting to lock <0x000000008080c258> (a
org.apache.log4j.spi.RootLogger)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:305)
at
org.apache.hadoop.hdfs.DFSClient$2.rejectedExecution(DFSClient.java:2904)
{code}
i.e. trying to log the above useless message.
I turned off logging but we still go into the logging system and hit the
BLOCKED section.
The RS backs up, fills all call queues. Nothing can come in the front door. We
start to burn all CPUs.
> RegionServer log is flooded with "Execution rejected, Executing in current
> thread"
> ----------------------------------------------------------------------------------
>
> Key: HDFS-13613
> URL: https://issues.apache.org/jira/browse/HDFS-13613
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.4.0
> Environment: CDH 5.13, HBase RegionServer, Kerberized, hedged read
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> In the log of a HBase RegionServer with hedged read, we saw the following
> message flooding the log file.
> {noformat}
> 2018-05-19 17:22:55,691 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,692 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,695 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,696 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> 2018-05-19 17:22:55,696 INFO org.apache.hadoop.hdfs.DFSClient: Execution
> rejected, Executing in current thread
> ....
> {noformat}
> Sometimes the RS spits tens of thousands of lines of this message in a
> minute. We should do something to stop this message flooding the log file.
> Also, we should make this message more actionable. Discussed with
> [~huaxiang], this message can appear if there are stale DataNodes.
> I believe this issue existed since HDFS-5776.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
