Hi all,
With large sql command, job failed with following error. Please give your
suggestion on how to resolve it. Thanks
Sql file size: 676k
Log:
16/04/25 10:55:00 WARN TaskSetManager: Lost task 84.0 in stage 0.0 (TID 6,
BJHC-HADOOP-HERA-17493.jd.local): java.util.concurrent.ExecutionException:
java.lang.Exception: failed to compile:
org.codehaus.janino.JaninoRuntimeException: Code of method
"(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
of class
"org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
grows beyond 64 KB
public Object generate(org.apache.spark.sql.catalyst.expressions.Expression[]
exprs) {
return new SpecificUnsafeProjection(exprs);
}
class SpecificUnsafeProjection extends
org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
......
java.lang.OutOfMemoryError: Java heap space
at com.google.protobuf.ByteString.copyFrom(ByteString.java:192)
at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:324)
at akka.remote.WireFormats$AkkaProtocolMessage.<init>(WireFormats.java:6657)
at akka.remote.WireFormats$AkkaProtocolMessage.<init>(WireFormats.java:6607)
at
akka.remote.WireFormats$AkkaProtocolMessage$1.parsePartialFrom(WireFormats.java:6703)
at
akka.remote.WireFormats$AkkaProtocolMessage$1.parsePartialFrom(WireFormats.java:6698)
at
com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193)
at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
at
akka.remote.WireFormats$AkkaProtocolMessage.parseFrom(WireFormats.java:6821)
at
akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:168)
发自 网易邮箱大师
