Hi, In Scala is it possible to set a config value to a single query?
I could set/unset the value, but it won't work for multithreading scenarios. Example: spark.sql.adaptive.coalescePartitions.enabled = false queryA_df.collect spark.sql.adaptive.coalescePartitions.enabled=original value queryB_df.collect queryC_df.collect queryD_df.collect If I execute that block of code multiple times using multiple thread, I can end up executing Query A with coalescePartitions.enabled=true, and Queries B, C and D with the config set to false, because another thread could set it between the executions. Is there any good alternative to this? Thanks.