[ https://issues.apache.org/jira/browse/HADOOP-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860304#comment-15860304 ]
Kihwal Lee commented on HADOOP-14033: ------------------------------------- +1 for the current patch. Please file a new jira for adding metrics if you have ideas. > Reduce fair call queue lock contention > -------------------------------------- > > Key: HADOOP-14033 > URL: https://issues.apache.org/jira/browse/HADOOP-14033 > Project: Hadoop Common > Issue Type: Sub-task > Components: ipc > Affects Versions: 2.7.0 > Reporter: Daryn Sharp > Assignee: Daryn Sharp > Attachments: HADOOP-14033.patch > > > Under heavy load the call queue may run dry yet clients experience high > latency. > The fcq requires producers and consumers to sync via a shared lock. Polling > consumers hold the lock while scanning all sub-queues. Consumers are > serialized despite the sub-queues being thread-safe blocking queues. The > effect is to cause other producers/consumers to frequently park. > The lock is unfair, so producers/consumers attempt to barge in on the lock. > The outnumbered producers tend to remain blocked for an extended time. As > load increases and the queues fill, the barging consumers drain the queues > faster than the producers can fill it. > Server metrics provide an illusion of healthy throughput, response time, and > call queue length due to starvation on the ingress. Often as the load gets > worse, the server looks better. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org