[
https://issues.apache.org/jira/browse/HADOOP-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12503168
]
Devaraj Das commented on HADOOP-1432:
-------------------------------------
There are a few issues with the patch:
1) The port rolling for grizzly won't work as implemented in the patch. This is
because the port is bound to within a thread (mapOut.start()) and in the patch,
you don't wait for the bind to happen and so you won't discover that you
couldn't bind to the port. After some experiments (although not conclusive), I
figured that it is not straightforward to implement that.
I think the best way to handle this issue is to pass "0" as the port and let
the OS decide. Grizzly might need to be patched to return the port it bound to
for the case where 0 is passed.
2) To get better perfomance, we need to have multiple selectors. This can be
achieved by doing mapOut.setSelectorReadThreadsCount(<number>);
3) Also, we need to have queues for the selectors -
mapOut.setMaxQueueSizeInBytes(<number>);
4) The mapOut needs to be a daemon thread - mapOut.setDaemon(true).
Some more experimentation needs to be done to make sure we are using the most
appropriate Grizzly APIs (in terms of performance) before this patch can go in.
> Use Grizzly for Fetching Map Output in Shuffle
> ----------------------------------------------
>
> Key: HADOOP-1432
> URL: https://issues.apache.org/jira/browse/HADOOP-1432
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Tahir Hashmi
> Assignee: Tahir Hashmi
> Attachments: 1432.patch, grizzly.tgz
>
>
> As mentioned in HADOOP-1273 and references therefrom, Jetty 6 still doesn't
> seem to be stable enough for use in Hadoop. Instead, we've decided to
> consider the usage of Grizzly Framework [https://grizzly.dev.java.net/] for
> NIO based communication.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
