[
https://issues.apache.org/jira/browse/MAPREDUCE-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikola Vujic updated MAPREDUCE-5791:
------------------------------------
Status: Patch Available (was: Open)
Patch contains implementation of a new function to do a data transfer. Existing
implementation relies on nio transferTo method which is slow on Windows. New
function does simple transfer by using an intermediate buffer in memory to
transfer data from a disk and send to a socket. Size of the intermediate buffer
determines the size of the IO requests. This way it is possible to manage size
of the IO request in the shuffle phase. Managing the size of the IO requests
turns to be important from the performance point of view on the Windows
machines.
I observed that the new code improves AVG Shuffle Time on windows for 1.8x. End
to end improvement in 100 GB Terasort is 1.3x when the new code is used (tested
on a cluster with 4 datanodes).
> Shuffle phase is slow in Windows - FadviseFileRegion::transferTo does not
> read disks efficiently
> ------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5791
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5791
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Nikola Vujic
> Assignee: Nikola Vujic
> Attachments: MAPREDUCE-5791.patch
>
>
> transferTo method in org.apache.hadoop.mapred.FadvisedFileRegion is using
> transferTo method from a FileChannel to transfer data from a disk to socket.
> This is performing slow in Windows, slower than in Linux. The reason is that
> transferTo method for the java.nio is issuing 32K IO requests all the time.
> In Windows, these 32K transfers are not optimal and we don't get the best
> performance form the underlying IO subsystem. In order to achieve better
> performance when reading from the drives, we need to read data in bigger
> chunks, 512K for example.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
