[
https://issues.apache.org/jira/browse/HADOOP-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587434#action_12587434
]
Raghu Angadi commented on HADOOP-2910:
--------------------------------------
Doug, what do you conclude from the above?
I modified the test to connect in a loop rather than in new threads. What I saw
is that 128 or so connected immediately.. Then new connections are tricking
slowly (one every a few seconds). These connections are in 'SYN_RECV' on sever
side and 'ESTABLISHED' on client side. So the kernel is probably accepting
extra connections beyond 128 very slowly. This is on Linux. Not sure if this
consistent with your impression. Looks like kernel keeps 40 or so connections
in SYN_RECV state and slowly moves them to ESTABLISHED state.
I will modify the threaded version to connect using a thread pool of 100
threads and wait for 10min once an exception occurs.
We had a discussion with Sameer and Sanjay. I will write up a small proposal
that will be an extension of earlier (reverted) patch in the next comment.
> Throttle IPC Client/Server during bursts of requests or server slowdown
> -----------------------------------------------------------------------
>
> Key: HADOOP-2910
> URL: https://issues.apache.org/jira/browse/HADOOP-2910
> Project: Hadoop Core
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.16.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.0
>
> Attachments: callQueue.patch, callQueue1.patch, callQueue2.patch,
> callQueue3.patch, TestBacklog.java, throttleClient.patch
>
>
> I propose the following to avoid an IPC server being swarmed by too many
> requests and connections
> 1. Limit call queue length or limit the amount of memory used in the call
> queue. This can be done by including the size of a request in the header and
> storing unmarshaled requests in the call queue.
> 2. If the call queue is full or queue buffer is full, stop reading requests
> from sockets. So requests stay at the server's system buffer or at the client
> side and thus eventually throttle the client.
> 3. Limit the total number of connections. Do not accept new connections if
> the connection limit is exceeded. (Note: this solution is unfair to new
> connections.)
> 4. If receive out of memory exception, close the current connection.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
