The Synapse config "spark.yarn.executor.decommission.enabled" is the closest thing to the proposed config "spark.executor.decommission.enabled" we've seen so far, I was only remarking that.
On the other hand, seems like the "decomission" config came out in Spark 3.4.0: https://archive.apache.org/dist/spark/docs/3.3.4/configuration.html https://archive.apache.org/dist/spark/docs/3.4.0/configuration.html El jue, 10 oct 2024 a las 4:44, Jungtaek Lim (<kabhwan.opensou...@gmail.com>) escribió: > Ángel, > > https://spark.apache.org/docs/latest/configuration.html > search through `decommission` in this page. > > The config you may have found from Synapse is > spark."yarn".executor.decommission.enabled. And the question was even "k8s" > and none of the information for the vendor was mentioned from the question. > I don't even think these configs are on the internet - simply saying, > google them (be sure to wrap with double quotes). > > Interestingly, I wouldn't even expect the config for graceful shutdown for > decommission. The functionality Spark provides for "decommission" is > basically a "graceful shutdown" of the executor. It sounds to me as > redundant. > > On Thu, Oct 10, 2024 at 11:11 AM Ángel <angel.alvarez.pas...@gmail.com> > wrote: > >> Looks like it actually exists ... but only for the Spark Synapse >> implementation ... >> >> >> https://learn.microsoft.com/en-us/answers/questions/1496283/purpose-of-spark-yarn-executor-decommission-enable >> >> >> Jay Han was asking for some config on k8s, so .... we shouldn't bring >> this config to the table, should we? >> >> El jue, 10 oct 2024 a las 2:55, Sean Owen (<sro...@gmail.com>) escribió: >> >>> Mich: you can set any key-value pair you want in Spark config. It >>> doesn't mean it is a real flag that code reads. >>> >>> spark.conf.set("ham", "sandwich") >>> print(spark.conf.get("ham")) >>> >>> prints "sandwich" >>> >>> forceKillTimeout is a real config: >>> >>> https://github.com/apache/spark/blob/fed9a8da3d4187794161e0be325aa96be8487783/core/src/main/scala/org/apache/spark/internal/config/package.scala#L2394 >>> >>> The others I cannot find, as in: >>> >>> https://github.com/search?q=repo%3Aapache%2Fspark%20spark.executor.decommission.gracefulShutdown&type=code >>> >>> If you're continuing to suggest these are real configs, where are you >>> finding those two in any docs or source? >>> Or what config were you thinking of if it's a typo? >>> >>> On Wed, Oct 9, 2024 at 5:14 PM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>>> Let us take this for a ride using these so called non-existent >>>> configuration settings >>>> >>>> spark.executor.decommission.enabled=true >>>> spark.executor.decommission.gracefulShutdown=true >>>> >>>> Tested on Spark 3.4 >>>> >>>> from pyspark.sql import SparkSession >>>> # Initialize a Spark session >>>> spark = SparkSession.builder \ >>>> .appName("Verifying Spark Configurations") \ >>>> .config("spark.executor.decommission.enabled", "true") \ >>>> .config("spark.executor.decommission.forceKillTimeout", "100s") \ >>>> .getOrCreate() >>>> >>>> # Access Spark context >>>> sc = spark.sparkContext >>>> # Set the log level to ERROR to reduce verbosity >>>> sc.setLogLevel("ERROR") >>>> print(f"\n\nSpark version: ", sc.version) >>>> >>>> # Verify the configuration for executor decommissioning >>>> decommission_enabled = >>>> sc.getConf().get("spark.executor.decommission.enabled", "false") >>>> force_kill_timeout = >>>> sc.getConf().get("spark.executor.decommission.forceKillTimeout", >>>> "default_value") >>>> >>>> # Print the values >>>> print(f"spark.executor.decommission.enabled: {decommission_enabled}") >>>> print(f"spark.executor.decommission.forceKillTimeout: >>>> {force_kill_timeout}") >>>> >>>> The output >>>> >>>> Spark version: 3.4.0 >>>> spark.executor.decommission.enabled: true >>>> spark.executor.decommission.forceKillTimeout: 100s >>>> >>>> By creating a simple Spark application and verifying the configuration >>>> values, I trust it is shown that these two parameters are valid and are >>>> applied by Spark >>>> >>>> HTH >>>> >>>> Mich Talebzadeh, >>>> >>>> Architect | Data Engineer | Data Science | Financial Crime >>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>> College London <https://en.wikipedia.org/wiki/Imperial_College_London> >>>> London, United Kingdom >>>> >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> >>>> >>>> >>>> *Disclaimer:* The information provided is correct to the best of my >>>> knowledge but of course cannot be guaranteed . It is essential to note >>>> that, as with any advice, quote "one test result is worth one-thousand >>>> expert opinions (Werner >>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>> >>>> >>>> On Wed, 9 Oct 2024 at 16:51, Mich Talebzadeh <mich.talebza...@gmail.com> >>>> wrote: >>>> >>>>> Do you have a better recommendation? >>>>> >>>>> Or trying to waste time as usual. >>>>> >>>>> It is far easier to throw than catch. >>>>> >>>>> Do your homework and stop throwing spanners at work. >>>>> >>>>> Mich Talebzadeh, >>>>> >>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>> College London <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>> >>>>> London, United Kingdom >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> >>>>> >>>>> >>>>> *Disclaimer:* The information provided is correct to the best of my >>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>> expert opinions (Werner >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>> >>>>> >>>>> On Wed, 9 Oct 2024 at 16:43, Nicholas Chammas < >>>>> nicholas.cham...@gmail.com> wrote: >>>>> >>>>>> Mich, >>>>>> >>>>>> Can you please share with the list where *exactly* you are citing >>>>>> these configs from? >>>>>> >>>>>> As far as I can tell, these two configs don’t exist and have never >>>>>> existed in the Spark codebase: >>>>>> >>>>>> spark.executor.decommission.enabled=true >>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>> >>>>>> Where exactly are you getting this information from (and then posting >>>>>> it to the list as advice)? Please be clear and provide specific >>>>>> references. >>>>>> >>>>>> Nick >>>>>> >>>>>> >>>>>> On Oct 9, 2024, at 1:20 PM, Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>> Before responding, what configuration parameters are you using to >>>>>> make this work? >>>>>> >>>>>> spark.executor.decommission.enabled=true >>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>> spark.executor.decommission.forceKillTimeout=100s >>>>>> >>>>>> HTH >>>>>> >>>>>> Mich Talebzadeh, >>>>>> >>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>> College London >>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>> London, United Kingdom >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>> expert opinions (Werner >>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>> >>>>>> >>>>>> On Wed, 9 Oct 2024 at 11:05, Jay Han <tunyu...@gmail.com> wrote: >>>>>> >>>>>>> Hi spark community, >>>>>>> I have such a question: Why driver doesn't shutdown executors >>>>>>> gracefully on k8s. For instance, >>>>>>> kubernetesClient.pods().withGracePeriod(100).delete(). >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best, >>>>>>> Jay >>>>>>> >>>>>> >>>>>>
