Nayeem,
The fix went into 1.6.0. I don't think it is possible for you to be
using 1.8.1 because the stack trace you posted used the old package
names, parquet.* instead of org.apache.parquet.*. That means you are
using <= 1.7.0 and probably <= 1.6.0.
Parquet is (or should be) compatible with all Hadoop versions.
rb
On 10/12/2015 10:23 AM, Sanjamala Nayeem wrote:
Sorry Ryan, I missed to mention Parquet version, 1.8.1
Thanks for the help. if you can point which version of hadoop is compatible
with which versin of Parquet will be very help full.
Thanks once again for the help.
Regards,
Nayeem,
MapR Technologies.
On Mon, Oct 12, 2015 at 10:42 PM, Sanjamala Nayeem <[email protected]>
wrote:
Thanks Ryan,
Can you please let me which version of Parquet has this feature.
Regards,
Nayeem,
MapR Technologies.
On Mon, Oct 12, 2015 at 10:36 PM, Ryan Blue <[email protected]> wrote:
Hi Nayeem,
What version of Parquet are you using? Parquet recently moved to using
FileSplit instead of writing its own ParquetInputSplit class. Updating to
that version may fix your problem, assuming that this is a version that is
creating ParquetInputSplits. I don't see what else would cause the
getSplits method to serialize the schema.
rb
On 10/11/2015 11:50 PM, Sanjamala Nayeem wrote:
Hi Experts,
When submitting job I am getting Out of Memory, when writing split I get
this error,
Can i get inputs how to resolve/or calculate memory to be allocated for
avoiding the the OOM.
Following is the stack trace.
Failing Oozie Launcher, Main
class[org.apache.oozie.action.hadoop.JavaMain], main() threw exception,
java.lang.OutOfMemoryError: Java heap space
org.apache.oozie.action.hadoop.JavaMainException:
java.lang.OutOfMemoryError: Java heap space
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:58)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:40)
at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:228)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2367)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
at java.lang.StringBuilder.append(StringBuilder.java:132)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:167)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.GroupType.writeToStringBuilder(GroupType.java:173)
at parquet.schema.GroupType.membersDisplayString(GroupType.java:154)
at parquet.schema.MessageType.writeToStringBuilder(MessageType.java:69)
at parquet.schema.Type.toString(Type.java:205)
at
parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:374)
at
parquet.hadoop.ParquetInputFormat.getSplits(ParquetInputFormat.java:344)
at
org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:115)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1566)
Thanks for the help.
Regards,
Nayeem,
MapR Technologies.
--
Ryan Blue
Software Engineer
Cloudera, Inc.
--
Ryan Blue
Software Engineer
Cloudera, Inc.
