Hi,

In Scala is it possible to set a config value to a single query?

I could set/unset the value, but it won't work for multithreading scenarios.

Example:

spark.sql.adaptive.coalescePartitions.enabled = false
                queryA_df.collect
spark.sql.adaptive.coalescePartitions.enabled=original value
                queryB_df.collect
                queryC_df.collect
                queryD_df.collect


If I execute that block of code multiple times using multiple thread, I can end 
up executing Query A with coalescePartitions.enabled=true, and Queries B, C and 
D with the config set to false, because another thread could set it between the 
executions.

Is there any good alternative to this?

Thanks.

Reply via email to