[
https://issues.apache.org/jira/browse/HADOOP-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated HADOOP-1650:
----------------------------------
Attachment: shuffleData.zip
To test performance against Jetty5, I ran a 400 node shuffle test. 40 maps
wrote 5GB each, feeding 400 reducers running on 385 nodes. Jetty6 finished in
684s, Jetty5 in 711; a negligible difference. In the attached graphs, each of
the two runs is visible (two nodes in the cluster (tt1, tt2) that ran map jobs
from both runs). The run on the left is Jetty6, the run on the right is Jetty5.
|| Filename || Tracking || Notes ||
| \*net | Network | Nearly identical; included to show when the map task ended
and when the shuffle began |
| \*mem | Memory | Neither Jetty5 nor Jetty6 had substantial footprints during
the shuffle. |
| \*conn | TCP connections | The rate of failed connections in Jetty5,
contrasted with Jetty6, is encouraging |
| \*cpu | CPU | Comparable profiles from both Jetty5 and Jetty6 on both nodes. |
| \*load | Load | Slightly higher load running Jetty5. The spike on tt1 running
Jetty5 is worth mentioning. |
> Upgrade Jetty to 6.x
> --------------------
>
> Key: HADOOP-1650
> URL: https://issues.apache.org/jira/browse/HADOOP-1650
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Attachments: 1650-0.patch, 1650-1_core.patch, 1650-1_merge.patch,
> 1650-1_proxy_delta.patch, 1650-2.patch, 1650-20080120.patch,
> 1650-jetty6.1.9.patch, 1650-jetty6.1.9.v1.patch,
> hadoop-1650-jetty6.1.5.patch, hadoop-jetty6.1.4-lib.tar.gz,
> hadoop-jetty6.1.6-lib.tar.gz, hadoop-jetty6.1.9-lib.tar.gz,
> jetty-hadoop-6.1.6.patch, jetty-hbase.patch, jetty6.1.4.patch,
> jetty6.1.6.patch, shuffleData.zip
>
>
> This is the third attempt at moving to jetty6. Apparently, the jetty-6.1.4
> has fixed some of the issues we discovered in jetty during HADOOP-736 and
> HADOOP-1273. I'd like to keep this issue open for sometime so that we have
> enough time to test out things.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
