[
https://issues.apache.org/jira/browse/PIG-3455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819567#comment-13819567
]
Aniket Mokashi commented on PIG-3455:
-------------------------------------
It's the logical plan's string representation.
Actually, using a streaming based hash function doesn't help as the
LogicalPlanPrinter's visit method gets the string representation first before
it writes to the stream. We need to change that in order to fix this problem.
{code}
public void visit() throws FrontendException {
try {
if (plan instanceof LogicalPlan) {
mStream.write(depthFirstLP().getBytes());
}
else {
mStream.write(reverseDepthFirstLP().getBytes());
}
} catch (IOException e) {
throw new FrontendException(e);
}
}
{code}
> Pig 0.11.1 OutOfMemory error
> ----------------------------
>
> Key: PIG-3455
> URL: https://issues.apache.org/jira/browse/PIG-3455
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.11.1
> Reporter: Shubham Chopra
> Assignee: Rohini Palaniswamy
> Priority: Critical
> Fix For: 0.12.0, 0.11.2
>
> Attachments: PIG-3455-1.patch
>
>
> When running Pig on a relatively large script (around 1.5k lines, 85
> assignments), Pig fails with the following error even before any jobs are
> fired:
> Pig Stack Trace
> ---------------
> ERROR 2998: Unhandled internal error. Java heap space
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at
> org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.depthFirstLP(LogicalPlanPrinter.java:83)
> at
> org.apache.pig.newplan.logical.optimizer.LogicalPlanPrinter.visit(LogicalPlanPrinter.java:69)
> at
> org.apache.pig.newplan.logical.relational.LogicalPlan.getSignature(LogicalPlan.java:122)
> at org.apache.pig.PigServer.execute(PigServer.java:1237)
> at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
> at org.apache.pig.Main.run(Main.java:604)
> at org.apache.pig.Main.main(Main.java:157)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
> The same script works fine with Pig-0.10.1.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
