[
https://issues.apache.org/jira/browse/SPARK-16624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
吴志龙 updated SPARK-16624:
------------------------
Summary: Generated SpecificColumnarIterator code can exceed JVM size limit
for cached DataFrames (was: cd /home/product/bi_etl/dump_data/dp_edw/)
> Generated SpecificColumnarIterator code can exceed JVM size limit for cached
> DataFrames
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-16624
> URL: https://issues.apache.org/jira/browse/SPARK-16624
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.2
> Environment: ./bin/spark-sql --master=yarn --queue queue3
> --driver-memory 4g --driver-java-options -XX:MaxPermSize=1g --num-executors 8
> --executor-memory 4g --conf spark.yarn.am.memory=4096m --hiveconf
> hive.cli.print.header=false -S -e “large sql command”
> Reporter: 吴志龙
> Fix For: 1.6.2
>
>
> java.lang.Exception: failed to compile:
> org.codehaus.janino.JaninoRuntimeException: Code of method
> "(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
> of class
> "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
> grows beyond 64 KB
> SQL: sql size 12K
> data:more tha 300 million the data
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
