[
https://issues.apache.org/jira/browse/HIVE-8373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15769862#comment-15769862
]
Andrew Sears commented on HIVE-8373:
------------------------------------
[~kellyzly] Java 8 removes PermGen memory and replaces with Metaspace. See
above StackOverflow link and some additional ones.
http://karunsubramanian.com/websphere/one-important-change-in-memory-management-in-java-8/
https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent
http://geekswithblogs.net/JoshReuben/archive/2016/04/11/jvm-tuning.aspx
So in JDK8, it appears that all native memory will be used unless capped.
For Local Spark Master:
HADOOP_OPTS="$HADOOP_OPTS -XX:MaxMetaspaceSize=512m"
For Spark History server:
SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -XX:MaxMetaspaceSize=512m"
Let me know if you need anything added to Wiki around this.
> OOM for a simple query with spark.master=local [Spark Branch]
> -------------------------------------------------------------
>
> Key: HIVE-8373
> URL: https://issues.apache.org/jira/browse/HIVE-8373
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: liyunzhang_intel
>
> I have a straigh forward query to run in Spark local mode, but get an OOM
> even though the data volumn is tiny:
> {code}
> Exception in thread "Spark Context Cleaner"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "Spark Context Cleaner"
> Exception in thread "Executor task launch worker-1"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "Executor task launch worker-1"
> Exception in thread "Keep-Alive-Timer"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "Keep-Alive-Timer"
> Exception in thread "Driver Heartbeater"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "Driver Heartbeater"
> {code}
> The query is:
> {code}
> select product_name, avg(item_price) as avg_price from product join item on
> item.product_pk=product.product_pk group by product_name order by avg_price;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
