Re: How to set a config for a single query?

[Saurabh Gulati]

Hey Felipe,
Since you are collecting the dataframes, you might as well run them separately 
with desired configs and store them in your storage.

Regards
Saurabh
________________________________
From: Felipe Pessoto <felipepess...@hotmail.com>
Sent: 04 January 2023 01:14
To: user@spark.apache.org <user@spark.apache.org>
Subject: [EXTERNAL] How to set a config for a single query?

Caution! This email originated outside of FedEx. Please do not open attachments 
or click links from an unknown or suspicious origin.


Hi,



In Scala is it possible to set a config value to a single query?



I could set/unset the value, but it won’t work for multithreading scenarios.



Example:



spark.sql.adaptive.coalescePartitions.enabled = false

                queryA_df.collect

spark.sql.adaptive.coalescePartitions.enabled=original value

                queryB_df.collect

                queryC_df.collect

                queryD_df.collect





If I execute that block of code multiple times using multiple thread, I can end 
up executing Query A with coalescePartitions.enabled=true, and Queries B, C and 
D with the config set to false, because another thread could set it between the 
executions.



Is there any good alternative to this?



Thanks.