Hi Ed, thanks for the reply. The art of evaluating performance is a black one indeed:)
I'm basing my interest on the advice from Todd Lipcon, referenced here: http://www.mentby.com/todd-lipcon/iosortmb-configuration.html Specifically: "you're unlikely to see a big difference in performance unless you cut down the number of spills from >1 to 1" So, given an io.sort.mb of 2047MB and a io.sort.record.percent of 0.57 I get: 2010-10-19 13:59:45,596 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2010-10-19 13:59:45,900 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=MAP, sessionId= 2010-10-19 13:59:46,161 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 2047 2010-10-19 13:59:48,134 INFO org.apache.hadoop.mapred.MapTask: data buffer = 738373632/922967040 2010-10-19 13:59:48,134 INFO org.apache.hadoop.mapred.MapTask: record buffer = 61173404/76466752 2010-10-19 13:59:48,160 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 2010-10-19 14:03:18,803 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true 2010-10-19 14:03:18,803 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 734080848; bufvoid = 922967040 2010-10-19 14:03:18,803 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 61173404; length = 76466752 2010-10-19 14:07:00,799 INFO org.apache.hadoop.mapred.MapTask: Finished spill 0 2010-10-19 14:09:35,992 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true 2010-10-19 14:09:35,992 INFO org.apache.hadoop.mapred.MapTask: bufstart = 734080848; bufend = 545194644; bufvoid = 922967040 2010-10-19 14:09:35,992 INFO org.apache.hadoop.mapred.MapTask: kvstart = 61173404; kvend = 45880055; length = 76466752 2010-10-19 14:14:29,105 INFO org.apache.hadoop.mapred.MapTask: Finished spill 1 2010-10-19 14:14:59,017 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output 2010-10-19 14:15:54,328 INFO org.apache.hadoop.mapred.MapTask: Finished spill 2 2010-10-19 14:15:54,365 INFO org.apache.hadoop.mapred.Merger: Merging 3 sorted segments 2010-10-19 14:15:54,387 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 2024217502 bytes 2010-10-19 14:18:03,284 INFO org.apache.hadoop.mapred.Task: Task:attempt_201010180321_0028_m_000000_0 is done. And is in the process of commiting 2010-10-19 14:18:03,363 INFO org.apache.hadoop.mapred.Task: Task 'attempt_201010180321_0028_m_000000_0' done. The two minutes 9 seconds spent doing the merge could presumably be saved if there was no spill? The trouble is I can't test this given the 2047MB cap. I'm not set up for building from source yet ... Cheers, Donovan. On 19 October 2010 14:44, ed <[email protected]> wrote: > HI Donovan, > > This is sort of tangential to your question but we tried upping our > io.sort.mb to a really high value and it actually resulted in slower > performance (I think we bumped it up to 1400MB and this was slower than > leaving it at 256 on a machine with 32GB of RAM). I'm not entirely sure why > this was the case. It could have been a garbage collection issue or some > other secondary effect that was slowing things down. > > Keep in mind that Hadoop will always spill map outputs to disk no matter how > large your sort buffer is in case the reducer crashes, the data needs to > exist on disk somewhere for the next reducer making the attempt so it might > be counterproductive to try and "eliminate" spills. > > ~Ed > > > On Tue, Oct 19, 2010 at 8:02 AM, Donovan Hide <[email protected]> wrote: > >> Hi, >> is there a reason why the io.sort.mb setting is hard-coded to the >> maximum of 2047MB? >> >> MapTask.java 789-791 >> >> if ((sortmb & 0x7FF) != sortmb) { >> throw new IOException("Invalid \"io.sort.mb\": " + sortmb); >> } >> >> Given that the EC2 High-Memory Quadruple Extra Large Instance has >> 68.4GB of memory and 8 cores, it would make sense to be able to set >> the io.sort.mb to close to 8GB. I have map task that outputs >> 144,586,867 records of average size 12 bytes, and a greater than >> 2047MB sort buffer would allow me to prevent the inevitable spills. I >> know I can reduce the size of the map inputs to solve the problem, but >> 2047MB seems a bit arbitrary given the spec of EC2 instances. >> >> Cheers, >> Donovan. >> >
