[
https://issues.apache.org/jira/browse/HADOOP-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587320#action_12587320
]
Doug Cutting commented on HADOOP-2910:
--------------------------------------
> It does not impose a limitation on the number of connections.
Yes, it does, the limit is just defined elsewhere, e.g., in the number of file
handles, or in the buffer size (according to Raghu's analysis). The number of
accepted connections will always be limited somewhere.
There's an existing mechanism to queue connections: the socket backlog.
Shouldn't we use that rather than implement our own queue of accepted
connections? Then, with queuing handled by the OS, the application can be
simpler. We don't need to accept connections that we're going to ignore, but
rather can wait until we're ready to service them before we accept them. We
then won't have to worry about running out of file handles, blocking on
buffers, etc.
Perhaps the socket backlog won't work for this, but I don't see that anyone has
tested that yet. I'll write some simple tests & attach them to this issue.
> Throttle IPC Client/Server during bursts of requests or server slowdown
> -----------------------------------------------------------------------
>
> Key: HADOOP-2910
> URL: https://issues.apache.org/jira/browse/HADOOP-2910
> Project: Hadoop Core
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.16.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.0
>
> Attachments: callQueue.patch, callQueue1.patch, callQueue2.patch,
> callQueue3.patch, throttleClient.patch
>
>
> I propose the following to avoid an IPC server being swarmed by too many
> requests and connections
> 1. Limit call queue length or limit the amount of memory used in the call
> queue. This can be done by including the size of a request in the header and
> storing unmarshaled requests in the call queue.
> 2. If the call queue is full or queue buffer is full, stop reading requests
> from sockets. So requests stay at the server's system buffer or at the client
> side and thus eventually throttle the client.
> 3. Limit the total number of connections. Do not accept new connections if
> the connection limit is exceeded. (Note: this solution is unfair to new
> connections.)
> 4. If receive out of memory exception, close the current connection.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
