Hi all, With large sql command, job failed with following error. Please give your suggestion on how to resolve it. Thanks
Sql file size: 676k Log: 16/04/25 10:55:00 WARN TaskSetManager: Lost task 84.0 in stage 0.0 (TID 6, BJHC-HADOOP-HERA-17493.jd.local): java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.janino.JaninoRuntimeException: Code of method "(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V" of class "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection" grows beyond 64 KB public Object generate(org.apache.spark.sql.catalyst.expressions.Expression[] exprs) { return new SpecificUnsafeProjection(exprs); } class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection { ...... java.lang.OutOfMemoryError: Java heap space at com.google.protobuf.ByteString.copyFrom(ByteString.java:192) at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:324) at akka.remote.WireFormats$AkkaProtocolMessage.<init>(WireFormats.java:6657) at akka.remote.WireFormats$AkkaProtocolMessage.<init>(WireFormats.java:6607) at akka.remote.WireFormats$AkkaProtocolMessage$1.parsePartialFrom(WireFormats.java:6703) at akka.remote.WireFormats$AkkaProtocolMessage$1.parsePartialFrom(WireFormats.java:6698) at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:141) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:176) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:188) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:193) at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49) at akka.remote.WireFormats$AkkaProtocolMessage.parseFrom(WireFormats.java:6821) at akka.remote.transport.AkkaPduProtobufCodec$.decodePdu(AkkaPduCodec.scala:168) 发自 网易邮箱大师