[
https://issues.apache.org/jira/browse/HADOOP-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12504197
]
Devaraj Das commented on HADOOP-1432:
-------------------------------------
Summarizing some points (most of them are from the email exchanges I had with
the Grizzly inventor Jeanfrancois):
1) We should set the keepAliveTimeout to 0. Since we are not using the
Persistent Connections HTTP feature, we can do that, and, that will improve
performance since we won't hit the keepalive subsystem.
2) JDK-1.6.0 has a bug to do with NIO -
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6481709 and we were hitting
that bug. I checked that the JDK-1.6.0u1 solves the problem.
3) Grizzly 1.5 version has a bug in that we cannot have multiple selectors.
Ideally, we should be using multiple selectors for accepting client connections
if we have multiple CPUs. There is now a bug open on that -
https://grizzly.dev.java.net/issues/show_bug.cgi?id=4. Also, one point to note
here is that the previous version of Grizzly, ver 1.0, supports this feature.
4) Persistent connections HTTP feature should help us (significantly?). This is
from the two papers -
http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html &
http://www.ibiblio.org/mdma-release/http-prob.html . I think the problems
discussed in the second paper could be alleviated to a certain extent if we
reuse connections for multiple fetches.
5) Another point worth investigating is whether DirectByteBuffers gives us
additional improvement. This is because our sort benchmark is data intensive.
> Use Grizzly for Fetching Map Output in Shuffle
> ----------------------------------------------
>
> Key: HADOOP-1432
> URL: https://issues.apache.org/jira/browse/HADOOP-1432
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Tahir Hashmi
> Assignee: Tahir Hashmi
> Attachments: 1432.patch, grizzly.tgz
>
>
> As mentioned in HADOOP-1273 and references therefrom, Jetty 6 still doesn't
> seem to be stable enough for use in Hadoop. Instead, we've decided to
> consider the usage of Grizzly Framework [https://grizzly.dev.java.net/] for
> NIO based communication.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
