[
https://issues.apache.org/jira/browse/HADOOP-2910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Doug Cutting updated HADOOP-2910:
---------------------------------
Attachment: TestBacklog.java
Here's a slightly modified version. I can get around the thread limit by
specifying '-Xss48k' and '-Xms128M'. I also introduced a 5ms pause in the
loop, to give the threads more of a fighting chance. It still fails around
16-24k connections, but now with "Connection refused" for the most recent
connection attempts. So I changed it to only make 15k connections, then sleep.
Now the connect attempts timeout after around four minutes.
To me this indicates that the backlog is effective up to around 16k
connections, but that connections queued this way will timeout after four
minutes.
I also tried Windows and also found that it failed after a few hundred
connections and could find no way to tweak OS settings to improve this.
> Throttle IPC Client/Server during bursts of requests or server slowdown
> -----------------------------------------------------------------------
>
> Key: HADOOP-2910
> URL: https://issues.apache.org/jira/browse/HADOOP-2910
> Project: Hadoop Core
> Issue Type: Improvement
> Components: ipc
> Affects Versions: 0.16.0
> Reporter: Hairong Kuang
> Assignee: Hairong Kuang
> Fix For: 0.18.0
>
> Attachments: callQueue.patch, callQueue1.patch, callQueue2.patch,
> callQueue3.patch, TestBacklog.java, TestBacklog.java,
> TestBacklogWithPool.java, throttleClient.patch
>
>
> I propose the following to avoid an IPC server being swarmed by too many
> requests and connections
> 1. Limit call queue length or limit the amount of memory used in the call
> queue. This can be done by including the size of a request in the header and
> storing unmarshaled requests in the call queue.
> 2. If the call queue is full or queue buffer is full, stop reading requests
> from sockets. So requests stay at the server's system buffer or at the client
> side and thus eventually throttle the client.
> 3. Limit the total number of connections. Do not accept new connections if
> the connection limit is exceeded. (Note: this solution is unfair to new
> connections.)
> 4. If receive out of memory exception, close the current connection.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
