[
https://issues.apache.org/jira/browse/SPARK-10529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-10529.
-------------------------------
Resolution: Not A Problem
Resolving per the PR discussion: don't make more than one context in a JVM
(i.e. run multiple isolated apps) as that's not the usual mode anyway. If you
must, then increase PermGen.
> When creating multiple HiveContext objects in one jvm, jdbc connections to
> metastore cann't be released and it may cause PermGen OutOfMemoryError.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-10529
> URL: https://issues.apache.org/jira/browse/SPARK-10529
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.1
> Reporter: ZhengYaofeng
>
> Test code as follows:
> object SqlTest {
> def main(args: Array[String]) {
> def createSc = {
> val sparkConf = new SparkConf().setAppName(s"SqlTest")
> .setMaster("spark://zdh221:7077")
> .set("spark.executor.memory", "4g")
> .set("spark.executor.cores", "2")
> .set("spark.cores.max", "6")
> new SparkContext(sparkConf)
> }
> for (index <- 1 to 200) {
> println(s"============Current Index:${index}=============")
> val hc = new HiveContext(createSc)
> hc.sql("show databases").collect().foreach(println)
> hc.sparkContext.stop()
> Thread.sleep(3000)
> }
> Thread.sleep(1000000)
> }
> }
> Testing on spark 1.4.1 with run cmd bellow.
> export
> CLASSPATH="$CLASSPATH:/home/hadoop/spark/conf:/home/hadoop/spark/lib/*:/home/hadoop/zyf/lib/*"
> java -Xmx8096m -Xms1024m -XX:MaxPermSize=1024m -cp $CLASSPATH SqlTest
> Files list:
>
> /home/hadoop/spark/conf:core-site.xml;hdfs-site.xml;hive-site.xml;slaves;spark-defaults.conf;spark-env.sh
>
> /home/hadoop/zyf/lib:hadoop-lzo-0.4.20.jar;mysql-connector-java-5.1.28-bin.jar;sqltest-1.0-SNAPSHOT.jar
>
> MySQL is used as the metastore. You can obviously see that jdbc connections
> to MySQL grow constantly through command 'show status like
> 'Threads_connected';' when my test app is running. Even if you invoke
> 'Hive.closeCurrent()', it cann't release current jdbc connections. Besides I
> can not find another possible way. If you take spark 1.3.1 to test, jdbc
> connections won't grow.
> Meanwhile, it ends with 'java.lang.OutOfMemoryError: PermGen space' when
> cycling 45 times, which means 45 HiveContext objects are created. It's
> interesting that if you set MaxPermSize to '2048m', it can cycle 93 times, if
> you set MaxPermSize to '3072m', it can cycle 141 times. So,it indicates that
> each time creating one HiveContext object, it loads the same amount of new
> classes and they won't be released.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
