Sorry Ryan, I missed to mention Parquet version, 1.8.1 Thanks for the help. if you can point which version of hadoop is compatible with which versin of Parquet will be very help full.
Thanks once again for the help. Regards, Nayeem, MapR Technologies. On Mon, Oct 12, 2015 at 10:42 PM, Sanjamala Nayeem <[email protected]> wrote: > Thanks Ryan, > > Can you please let me which version of Parquet has this feature. > > Regards, > Nayeem, > MapR Technologies. > > On Mon, Oct 12, 2015 at 10:36 PM, Ryan Blue <[email protected]> wrote: > >> Hi Nayeem, >> >> What version of Parquet are you using? Parquet recently moved to using >> FileSplit instead of writing its own ParquetInputSplit class. Updating to >> that version may fix your problem, assuming that this is a version that is >> creating ParquetInputSplits. I don't see what else would cause the >> getSplits method to serialize the schema. >> >> rb >> >> >> On 10/11/2015 11:50 PM, Sanjamala Nayeem wrote: >> >>> Hi Experts, >>> When submitting job I am getting Out of Memory, when writing split I get >>> this error, >>> >>> Can i get inputs how to resolve/or calculate memory to be allocated for >>> avoiding the the OOM. >>> >>> Following is the stack trace. >>> >>> Failing Oozie Launcher, Main >>> class[org.apache.oozie.action.hadoop.JavaMain], main() threw exception, >>> java.lang.OutOfMemoryError: Java heap space >>> org.apache.oozie.action.hadoop.JavaMainException: >>> java.lang.OutOfMemoryError: Java heap space >>> at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:58) >>> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:40) >>> at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> >>> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228) >>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) >>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435) >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) >>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at >>> >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566) >>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) >>> Caused by: java.lang.OutOfMemoryError: Java heap space >>> at java.util.Arrays.copyOf(Arrays.java:2367) >>> at >>> >>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130) >>> at >>> >>> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114) >>> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415) >>> at java.lang.StringBuilder.append(StringBuilder.java:132) >>> at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:167) >>> at parquet.schema.GroupType.membersDisplayString(GroupType.java:154) >>> at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173) >>> at parquet.schema.GroupType.membersDisplayString(GroupType.java:154) >>> at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173) >>> at parquet.schema.GroupType.membersDisplayString(GroupType.java:154) >>> at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173) >>> at parquet.schema.GroupType.membersDisplayString(GroupType.java:154) >>> at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173) >>> at parquet.schema.GroupType.membersDisplayString(GroupType.java:154) >>> at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173) >>> at parquet.schema.GroupType.membersDisplayString(GroupType.java:154) >>> at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173) >>> at parquet.schema.GroupType.membersDisplayString(GroupType.java:154) >>> at parquet.schema.MessageType.writeToStringBuilder(MessageType.java:69) >>> at parquet.schema.Type.toString(Type.java:205) >>> at >>> parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:374) >>> at >>> parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:344) >>> at >>> >>> org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:115) >>> at >>> >>> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493) >>> at >>> >>> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510) >>> at >>> >>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394) >>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285) >>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at >>> >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566) >>> Thanks for the help. >>> >>> Regards, >>> Nayeem, >>> MapR Technologies. >>> >>> >> >> -- >> Ryan Blue >> Software Engineer >> Cloudera, Inc. >> > >
