Which version of pig are you using ? Does your query have just a load and store statement ? What load and store functions are you using ? What would be the largest size of a single tuple in your data ?
Thanks, Thejas On 4/20/12 1:18 PM, shan s wrote:
I am simply reading 1.5 gb file with 2458220 records, and storing it back. I am getting Java Heap space error The current setting is mapred.child.java.opts -Xmx1073741824 Below is the error from log. 2012-04-20 15:03:15,769 INFO org.apache.pig.impl.util.SpillableMemoryManager: first memory handler call- Usage threshold init = 350617600(342400K) used = 522736064(510484K) committed = 596443136(582464K) max = 715849728(699072K) Could you please help me interpret this.How do I set space for Java heap. I am looking at archive on this topic, but didn't see anything similar. (most of those talk about UDF, but I am simply doing a read/write). Thanks, Prashant.
