[ https://issues.apache.org/jira/browse/SPARK-19255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826041#comment-15826041 ]
Takeshi Yamamuro commented on SPARK-19255: ------------------------------------------ Yea, If #partition increases, the size of `taskIdToMetricUpdates` grows. Then, since `_stageIdToStageMetrics` holds `taskIdToMetricUpdates`, `_stageIdToStageMetrics` seems big as described in your png. Anyway, IIUC, you took 100MB for `_stageIdToStageMetrics` in a driver and hit OOM there, right? If so, you'd be better to make driver memory more bigger, I think. > SQL Listener is causing out of memory, in case of large no of shuffle > partition > ------------------------------------------------------------------------------- > > Key: SPARK-19255 > URL: https://issues.apache.org/jira/browse/SPARK-19255 > Project: Spark > Issue Type: Improvement > Components: SQL > Environment: Linux > Reporter: Ashok Kumar > Priority: Minor > Attachments: spark_sqllistener_oom.png > > > Test steps. > 1.CREATE TABLE sample(imei string,age int,task bigint,num double,level > decimal(10,3),productdate timestamp,name string,point int)USING > com.databricks.spark.csv OPTIONS (path "data.csv", header "false", > inferSchema "false"); > 2. set spark.sql.shuffle.partitions=100000; > 3. select count(*) from (select task,sum(age) from sample group by task) t; > After running above query, number of objects in map variable > _stageIdToStageMetrics has increase to very high number , this increment is > proportional to number of shuffle partition. > Please have a look at attached screenshot -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org