Re: Shuffle buffer size in presence of small partitions

Hi all, fyi this is the ticket I opened up: https://issues.apache.org/jira/browse/MAPREDUCE-6923 Thanks in advance! Robert On Mon, Jul 31, 2017 at 10:21 PM, Ravi Prakash wrote: > Hi Robert! > > I'm sorry I do not have a Windows box and probably don't understand the >

Re: Shuffle buffer size in presence of small partitions

Hi Robert! I'm sorry I do not have a Windows box and probably don't understand the shuffle process well enough. Could you please create a JIRA in the mapreduce proect if you would like this fixed upstream? https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=116=MAPREDUCE Thanks Ravi

Shuffle buffer size in presence of small partitions

Hi all, I just ran into an issue, which likely resulted from my not very intelligent configuration, but nonetheless I'd like to share this with the community. This is all on Hadoop 2.7.3. In my setup, each reducer roughly fetched 65K from each mapper's spill file. I disabled transferTo during