Re: Any way to see the size of the broadcast variable?

Yes each of the executors have 60GB -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ - To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: Any way to see the size of the broadcast variable?

Hi Venkat, do you executors have that much amount of memory? Regards, Gourav Sengupta On Tue, Oct 9, 2018 at 4:44 PM V0lleyBallJunki3 wrote: > Hello, >I have set the value of spark.sql.autoBroadcastJoinThreshold to a very > high value of 20 GB. I am joining a table that I am sure is below

Any way to see the size of the broadcast variable?

Hello, I have set the value of spark.sql.autoBroadcastJoinThreshold to a very high value of 20 GB. I am joining a table that I am sure is below this variable, however spark is doing a SortMergeJoin. If I set a broadcast hint then spark does a broadcast join and job finishes much faster.