[
https://issues.apache.org/jira/browse/YARN-8691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yicong Cai updated YARN-8691:
-----------------------------
Description:
SparkSQL AM Codegen ERROR，then call unregister AM API and send the error
message to RM, RM receive the AM state and update to RMStateStore. The Codegen
error message maybe is huge, (Our case is about 200MB). If the RMStateStore is
ZKRMStateStore, it causes the same exception as YARN-6125, but YARN-6125
doesn't cover the unregisterApplicationMaster's message cut.
SparkSQL Codegen error message show below:
18/08/18 08:34:54 ERROR codegen.CodeGenerator: failed to compile:
org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM
limit of 0xFFFF
/* 001 */ public java.lang.Object generate(Object[] references)
{ /* 002 */ return new SpecificSafeProjection(references); /* 003 */ }
/* 004 */
/* 005 */ class SpecificSafeProjection extends
org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
..........
about 2 million lines.
..........
was:
SparkSQL AM Codegen ERROR，then call unregister AM API and send the error
message to RM, RM receive the AM state and update to RMStateStore. The Codegen
error message maybe is huge, (Our case is about 200MB). If the RMStateStore is
ZKRMStateStore, it causes the same exception as
[YARN-6125|https://issues.apache.org/jira/browse/YARN-6125], but YARN-6125
doesn't cover the unregisterApplicationMaster's message cut.
SparkSQL Codegen error message show below:
18/08/18 08:34:54 ERROR codegen.CodeGenerator: failed to compile:
org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM
limit of 0xFFFF
/* 001 */ public java.lang.Object generate(Object[] references) {
/* 002 */ return new SpecificSafeProjection(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificSafeProjection extends
org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
..........
about 2 million lines.
..........
> AMRMClient unregisterApplicationMaster Api's appMessage should have a maximum
> size
> ----------------------------------------------------------------------------------
>
> Key: YARN-8691
> URL: https://issues.apache.org/jira/browse/YARN-8691
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.7.3
> Reporter: Yicong Cai
> Priority: Critical
> Fix For: 2.7.7
>
>
> SparkSQL AM Codegen ERROR，then call unregister AM API and send the error
> message to RM, RM receive the AM state and update to RMStateStore. The
> Codegen error message maybe is huge, (Our case is about 200MB). If the
> RMStateStore is ZKRMStateStore, it causes the same exception as YARN-6125,
> but YARN-6125 doesn't cover the unregisterApplicationMaster's message cut.
>
> SparkSQL Codegen error message show below:
> 18/08/18 08:34:54 ERROR codegen.CodeGenerator: failed to compile:
> org.codehaus.janino.JaninoRuntimeException: Constant pool has grown past JVM
> limit of 0xFFFF
> /* 001 */ public java.lang.Object generate(Object[] references)
> { /* 002 */ return new SpecificSafeProjection(references); /* 003 */ }
> /* 004 */
> /* 005 */ class SpecificSafeProjection extends
> org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
> ..........
> about 2 million lines.
> ..........
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
