[
https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16368823#comment-16368823
]
Kazuaki Ishizaki edited comment on SPARK-23427 at 2/19/18 7:01 AM:
-------------------------------------------------------------------
I got the OOM with the same stack trace for both configurations when I ran this
program using 256gb heap.
Of course, since we know that to throw OOM is a problem, I will look at this.
We would like to confirm whether this issue depends on the option or not.
was (Author: kiszk):
I got the OOM with the same stack trace for both configurations when I ran this
program using 256gb heap.
> spark.sql.autoBroadcastJoinThreshold causing OOM exception in the driver
> -------------------------------------------------------------------------
>
> Key: SPARK-23427
> URL: https://issues.apache.org/jira/browse/SPARK-23427
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Environment: SPARK 2.0 version
> Reporter: Dhiraj
> Priority: Critical
>
> We are facing issue around value of spark.sql.autoBroadcastJoinThreshold.
> With spark.sql.autoBroadcastJoinThreshold -1 ( disable) we seeing driver
> memory used flat.
> With any other values 10MB, 5MB, 2 MB, 1MB, 10K, 1K we see driver memory used
> goes up with rate depending upon the size of the autoBroadcastThreshold and
> getting OOM exception. The problem is memory used by autoBroadcast is not
> being free up in the driver.
> Application imports oracle tables as master dataframes which are persisted.
> Each job applies filter to these tables and then registered them as
> tempViewTable . Then sql query are using to process data further. At the end
> all the intermediate dataFrame are unpersisted.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
