[
https://issues.apache.org/jira/browse/SPARK-23427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dhiraj updated SPARK-23427:
---------------------------
Description:
We are facing issue around value of spark.sql.autoBroadcastJoinThreshold.
With spark.sql.autoBroadcastJoinThreshold -1 ( disable) we seeing driver memory
used flat.
With any other values 10MB, 5MB, 2 MB, 1MB, 10K, 1K we see driver memory used
goes up with rate depending upon the size of the autoBroadcastThreshold and
getting OOM exception. The problem is memory used by autoBroadcast is not being
free up in the driver.
Application imports oracle tables as master dataframes which are persisted.
Each job applies filter to these tables and then registered them as
tempViewTable . Then sql query are using to process data further. At the end
all the intermediate dataFrame are unpersisted.
was:
We are facing issue around value of spark.sql.autoBroadcastJoinThreshold.
With spark.sql.autoBroadcastJoinThreshold -1 ( disable) we seeing driver memory
used flat.
With any other values 10MB, 5MB, 2 MB, 1MB, 10K, 1K we see driver memory used
goes up with rate depending upon the size of the autoBroadcastThreshold and
getting OOM exception. The problem is memory used by autoBroadcast is not being
free up in the driver.
> spark.sql.autoBroadcastJoinThreshold causing OOM in the driver
> ----------------------------------------------------------------
>
> Key: SPARK-23427
> URL: https://issues.apache.org/jira/browse/SPARK-23427
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Environment: SPARK 2.0 version
> Reporter: Dhiraj
> Priority: Critical
>
> We are facing issue around value of spark.sql.autoBroadcastJoinThreshold.
> With spark.sql.autoBroadcastJoinThreshold -1 ( disable) we seeing driver
> memory used flat.
> With any other values 10MB, 5MB, 2 MB, 1MB, 10K, 1K we see driver memory used
> goes up with rate depending upon the size of the autoBroadcastThreshold and
> getting OOM exception. The problem is memory used by autoBroadcast is not
> being free up in the driver.
> Application imports oracle tables as master dataframes which are persisted.
> Each job applies filter to these tables and then registered them as
> tempViewTable . Then sql query are using to process data further. At the end
> all the intermediate dataFrame are unpersisted.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
