Allen, Say I were to bring the property back to the default of -Xmx200m, which buffers do you think I should adjust? io.sort.mb? io.sort.factor? How would you adjust them?
Thanks for your help! Pony On Mon, Jul 11, 2011 at 4:41 PM, Allen Wittenauer <[email protected]> wrote: > > On Jul 11, 2011, at 9:28 AM, Juan P. wrote: > > > > * <property>* > > * <name>mapred.child.java.opts</name>* > > * <value>-Xmx400m</value>* > > * </property>* > > "Single core machines with 600MB of RAM." > > 2x400m = 800m just for the heap of the map and reduce > phases, not counting the other memory that the jvm will need. io buffer > sizes aren't adjusted downward either, so you're likely looking at a > swapping + spills = death scenario. slowstart set to 1 is going to be > pretty much required.
