sorry to resurrect a old thread. But I am having the same problem here for Pig 0.7
011-10-19 13:03:22,895 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage threshold exceeded) init = 263323648(257152K) used = 502624976(490844K) committed = 617086976(602624K) max = 715849728(699072K) 2011-10-19 13:03:58,776 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 494572568(482981K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:04:01,248 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage threshold exceeded) init = 263323648(257152K) used = 501339160(489589K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:05:01,250 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 589147376(575339K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:05:42,181 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 487209296(475790K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:06:07,300 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Usage threshold exceeded) init = 263323648(257152K) used = 501963568(490198K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:07:11,850 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 575026232(561549K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:07:53,782 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 505687176(493835K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:09:02,451 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 571977464(558571K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:09:44,027 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 509238248(497302K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:10:55,596 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 561933048(548762K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:11:40,931 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 517425112(505297K) committed = 715849728(699072K) max = 715849728(699072K) 2011-10-19 13:12:48,891 INFO org.apache.pig.impl.util.SpillableMemoryManager: low memory handler called (Collection threshold exceeded) init = 263323648(257152K) used = 558586544(545494K) committed = 715849728(699072K) max = 715849728(699072K) The job took a while to finish and not sure what causes it, it only happens in the reduce phase and i am wondering how to increase the max memory to be used by pig. In my mapreduce.reduce.java.opts, I have set the value to 1GB but it seems the max is only 700MB for whatever reason. Thanks, Felix
