Hi Nayeem,
What version of Parquet are you using? Parquet recently moved to using
FileSplit instead of writing its own ParquetInputSplit class. Updating
to that version may fix your problem, assuming that this is a version
that is creating ParquetInputSplits. I don't see what else would cause
the getSplits method to serialize the schema.
rb
On 10/11/2015 11:50 PM, Sanjamala Nayeem wrote:
Hi Experts,
When submitting job I am getting Out of Memory, when writing split I get
this error,
Can i get inputs how to resolve/or calculate memory to be allocated for
avoiding the the OOM.
Following is the stack trace.
Failing Oozie Launcher, Main
class[org.apache.oozie.action.hadoop.JavaMain], main() threw exception,
java.lang.OutOfMemoryError: Java heap space
org.apache.oozie.action.hadoop.JavaMainException:
java.lang.OutOfMemoryError: Java heap space
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:58)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:40)
at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2367)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
at java.lang.StringBuilder.append(StringBuilder.java:132)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:167)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.MessageType.writeToStringBuilder(MessageType.java:69)
at parquet.schema.Type.toString(Type.java:205)
at parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:374)
at parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:344)
at
org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:115)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
Thanks for the help.
Regards,
Nayeem,
MapR Technologies.
--
Ryan Blue
Software Engineer
Cloudera, Inc.
