The length printed there is the # of elements in an internal array, not a data size length. If you wish to raise that you can raise up io.sort.spill.percent (default 0.80, or 80%).
On Tue, Aug 27, 2013 at 4:03 AM, kaveh minooie <[email protected]> wrote: > Thaks Harsh, but my io.sort.mb is 320 while, as you can see, length is > only 1mb (1048576) > > > On 08/26/2013 03:14 PM, Harsh J wrote: >> >> You need to raise io.sort.mb to increase the sort buffer capacity. >> >> On Tue, Aug 27, 2013 at 12:56 AM, kaveh minooie <[email protected]> wrote: >>> >>> I am getting these in job output in the 1.2.1 >>> >>> Spilling map output: record full = true >>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: bufstart = >>> 111149676; bufend = 122264640; bufvoid = 318767104 >>> 2013-08-26 11:40:33,413 INFO org.apache.hadoop.mapred.MapTask: kvstart = >>> 1048561; kvend = 838844; length = 1048576 >>> 2013-08-26 11:40:35,879 INFO org.apache.hadoop.mapred.MapTask: Finished >>> spill 10 >>> >>> >>> my question is what is bufvoid? and how can I increase the amount of the >>> length? >>> >>> thanks, >>> -- >>> Kaveh Minooie >> >> >> >> > > -- > Kaveh Minooie -- Harsh J
