to be clear are you referring to these spark.executor.decommission.enabled=true spark.executor.decommission.gracefulShutdown=true
thanks Mich Talebzadeh, Architect | Data Engineer | Data Science | Financial Crime PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College London <https://en.wikipedia.org/wiki/Imperial_College_London> London, United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Thu, 10 Oct 2024 at 09:03, Ángel <angel.alvarez.pas...@gmail.com> wrote: > Do you know by any chance if that config also applies to Databricks? > > El jue, 10 oct 2024 a las 10:02, Ángel (<angel.alvarez.pas...@gmail.com>) > escribió: > >> Thanks a lot for the clarification. Interesting... I've never needed it, >> even though I've been using Spark for over 8 years. >> >> El jue, 10 oct 2024 a las 9:21, Liu Cao (<twcnnj...@gmail.com>) escribió: >> >>> I’m unclear on what the exact issue the OP ran into. >>> >>> But if we are talking about decommission, just one side note: >>> >>> The decommission feature >>> <https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?tab=t.0#heading=h.70pylrl2vdg8> >>> has been in spark for a while, and decommission on K8S and standalone was >>> GA-ed in 3.1 actually. See >>> https://spark.apache.org/releases/spark-release-3-1-1.html >>> >>> The fact that you didn’t see it in the 3.3 site is simply a lack of >>> documentation. The missing documentation was added in 3.4, thanks to >>> https://github.com/apache/spark/pull/38131/files >>> >>> On Wed, Oct 9, 2024 at 10:13 PM Ángel <angel.alvarez.pas...@gmail.com> >>> wrote: >>> >>>> The Synapse config "spark.yarn.executor.decommission.enabled" is the >>>> closest thing to the proposed config "spark.executor.decommission.enabled" >>>> we've seen so far, I was only remarking that. >>>> >>>> On the other hand, seems like the "decomission" config came out in >>>> Spark 3.4.0: >>>> >>>> https://archive.apache.org/dist/spark/docs/3.3.4/configuration.html >>>> https://archive.apache.org/dist/spark/docs/3.4.0/configuration.html >>>> >>>> >>>> >>>> El jue, 10 oct 2024 a las 4:44, Jungtaek Lim (< >>>> kabhwan.opensou...@gmail.com>) escribió: >>>> >>>>> Ángel, >>>>> >>>>> https://spark.apache.org/docs/latest/configuration.html >>>>> search through `decommission` in this page. >>>>> >>>>> The config you may have found from Synapse is >>>>> spark."yarn".executor.decommission.enabled. And the question was even >>>>> "k8s" >>>>> and none of the information for the vendor was mentioned from the >>>>> question. >>>>> I don't even think these configs are on the internet - simply saying, >>>>> google them (be sure to wrap with double quotes). >>>>> >>>>> Interestingly, I wouldn't even expect the config for graceful shutdown >>>>> for decommission. The functionality Spark provides for "decommission" is >>>>> basically a "graceful shutdown" of the executor. It sounds to me as >>>>> redundant. >>>>> >>>>> On Thu, Oct 10, 2024 at 11:11 AM Ángel <angel.alvarez.pas...@gmail.com> >>>>> wrote: >>>>> >>>>>> Looks like it actually exists ... but only for the Spark Synapse >>>>>> implementation ... >>>>>> >>>>>> >>>>>> https://learn.microsoft.com/en-us/answers/questions/1496283/purpose-of-spark-yarn-executor-decommission-enable >>>>>> >>>>>> >>>>>> Jay Han was asking for some config on k8s, so .... we shouldn't bring >>>>>> this config to the table, should we? >>>>>> >>>>>> El jue, 10 oct 2024 a las 2:55, Sean Owen (<sro...@gmail.com>) >>>>>> escribió: >>>>>> >>>>>>> Mich: you can set any key-value pair you want in Spark config. It >>>>>>> doesn't mean it is a real flag that code reads. >>>>>>> >>>>>>> spark.conf.set("ham", "sandwich") >>>>>>> print(spark.conf.get("ham")) >>>>>>> >>>>>>> prints "sandwich" >>>>>>> >>>>>>> forceKillTimeout is a real config: >>>>>>> >>>>>>> https://github.com/apache/spark/blob/fed9a8da3d4187794161e0be325aa96be8487783/core/src/main/scala/org/apache/spark/internal/config/package.scala#L2394 >>>>>>> >>>>>>> The others I cannot find, as in: >>>>>>> >>>>>>> https://github.com/search?q=repo%3Aapache%2Fspark%20spark.executor.decommission.gracefulShutdown&type=code >>>>>>> >>>>>>> If you're continuing to suggest these are real configs, where are >>>>>>> you finding those two in any docs or source? >>>>>>> Or what config were you thinking of if it's a typo? >>>>>>> >>>>>>> On Wed, Oct 9, 2024 at 5:14 PM Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>>> Let us take this for a ride using these so called non-existent >>>>>>>> configuration settings >>>>>>>> >>>>>>>> spark.executor.decommission.enabled=true >>>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>>> >>>>>>>> Tested on Spark 3.4 >>>>>>>> >>>>>>>> from pyspark.sql import SparkSession >>>>>>>> # Initialize a Spark session >>>>>>>> spark = SparkSession.builder \ >>>>>>>> .appName("Verifying Spark Configurations") \ >>>>>>>> .config("spark.executor.decommission.enabled", "true") \ >>>>>>>> .config("spark.executor.decommission.forceKillTimeout", "100s") >>>>>>>> \ >>>>>>>> .getOrCreate() >>>>>>>> >>>>>>>> # Access Spark context >>>>>>>> sc = spark.sparkContext >>>>>>>> # Set the log level to ERROR to reduce verbosity >>>>>>>> sc.setLogLevel("ERROR") >>>>>>>> print(f"\n\nSpark version: ", sc.version) >>>>>>>> >>>>>>>> # Verify the configuration for executor decommissioning >>>>>>>> decommission_enabled = >>>>>>>> sc.getConf().get("spark.executor.decommission.enabled", "false") >>>>>>>> force_kill_timeout = >>>>>>>> sc.getConf().get("spark.executor.decommission.forceKillTimeout", >>>>>>>> "default_value") >>>>>>>> >>>>>>>> # Print the values >>>>>>>> print(f"spark.executor.decommission.enabled: >>>>>>>> {decommission_enabled}") >>>>>>>> print(f"spark.executor.decommission.forceKillTimeout: >>>>>>>> {force_kill_timeout}") >>>>>>>> >>>>>>>> The output >>>>>>>> >>>>>>>> Spark version: 3.4.0 >>>>>>>> spark.executor.decommission.enabled: true >>>>>>>> spark.executor.decommission.forceKillTimeout: 100s >>>>>>>> >>>>>>>> By creating a simple Spark application and verifying the >>>>>>>> configuration values, I trust it is shown that these two parameters are >>>>>>>> valid and are applied by Spark >>>>>>>> >>>>>>>> HTH >>>>>>>> >>>>>>>> Mich Talebzadeh, >>>>>>>> >>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>> College London >>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>> London, United Kingdom >>>>>>>> >>>>>>>> >>>>>>>> view my Linkedin profile >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>> >>>>>>>> >>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>> note >>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>> expert opinions (Werner >>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>> >>>>>>>> >>>>>>>> On Wed, 9 Oct 2024 at 16:51, Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Do you have a better recommendation? >>>>>>>>> >>>>>>>>> Or trying to waste time as usual. >>>>>>>>> >>>>>>>>> It is far easier to throw than catch. >>>>>>>>> >>>>>>>>> Do your homework and stop throwing spanners at work. >>>>>>>>> >>>>>>>>> Mich Talebzadeh, >>>>>>>>> >>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>>> College London >>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>> London, United Kingdom >>>>>>>>> >>>>>>>>> >>>>>>>>> view my Linkedin profile >>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>> note >>>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>>> expert opinions (Werner >>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, 9 Oct 2024 at 16:43, Nicholas Chammas < >>>>>>>>> nicholas.cham...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Mich, >>>>>>>>>> >>>>>>>>>> Can you please share with the list where *exactly* you are >>>>>>>>>> citing these configs from? >>>>>>>>>> >>>>>>>>>> As far as I can tell, these two configs don’t exist and have >>>>>>>>>> never existed in the Spark codebase: >>>>>>>>>> >>>>>>>>>> spark.executor.decommission.enabled=true >>>>>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>>>>> >>>>>>>>>> Where exactly are you getting this information from (and then >>>>>>>>>> posting it to the list as advice)? Please be clear and provide >>>>>>>>>> specific >>>>>>>>>> references. >>>>>>>>>> >>>>>>>>>> Nick >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Oct 9, 2024, at 1:20 PM, Mich Talebzadeh < >>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> Before responding, what configuration parameters are you using to >>>>>>>>>> make this work? >>>>>>>>>> >>>>>>>>>> spark.executor.decommission.enabled=true >>>>>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>>>>> spark.executor.decommission.forceKillTimeout=100s >>>>>>>>>> >>>>>>>>>> HTH >>>>>>>>>> >>>>>>>>>> Mich Talebzadeh, >>>>>>>>>> >>>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>>>> College London >>>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>>> London, United Kingdom >>>>>>>>>> >>>>>>>>>> view my Linkedin profile >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>>> note >>>>>>>>>> that, as with any advice, quote "one test result is worth >>>>>>>>>> one-thousand >>>>>>>>>> expert opinions (Werner >>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, 9 Oct 2024 at 11:05, Jay Han <tunyu...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi spark community, >>>>>>>>>>> I have such a question: Why driver doesn't shutdown >>>>>>>>>>> executors gracefully on k8s. For instance, >>>>>>>>>>> kubernetesClient.pods().withGracePeriod(100).delete(). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Best, >>>>>>>>>>> Jay >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>> >>> -- >>> >>> Liu Cao >>> >>> >>>
