I’m unclear on what the exact issue the OP ran into. But if we are talking about decommission, just one side note:
The decommission feature <https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?tab=t.0#heading=h.70pylrl2vdg8> has been in spark for a while, and decommission on K8S and standalone was GA-ed in 3.1 actually. See https://spark.apache.org/releases/spark-release-3-1-1.html The fact that you didn’t see it in the 3.3 site is simply a lack of documentation. The missing documentation was added in 3.4, thanks to https://github.com/apache/spark/pull/38131/files On Wed, Oct 9, 2024 at 10:13 PM Ángel <angel.alvarez.pas...@gmail.com> wrote: > The Synapse config "spark.yarn.executor.decommission.enabled" is the > closest thing to the proposed config "spark.executor.decommission.enabled" > we've seen so far, I was only remarking that. > > On the other hand, seems like the "decomission" config came out in Spark > 3.4.0: > > https://archive.apache.org/dist/spark/docs/3.3.4/configuration.html > https://archive.apache.org/dist/spark/docs/3.4.0/configuration.html > > > > El jue, 10 oct 2024 a las 4:44, Jungtaek Lim (< > kabhwan.opensou...@gmail.com>) escribió: > >> Ángel, >> >> https://spark.apache.org/docs/latest/configuration.html >> search through `decommission` in this page. >> >> The config you may have found from Synapse is >> spark."yarn".executor.decommission.enabled. And the question was even "k8s" >> and none of the information for the vendor was mentioned from the question. >> I don't even think these configs are on the internet - simply saying, >> google them (be sure to wrap with double quotes). >> >> Interestingly, I wouldn't even expect the config for graceful shutdown >> for decommission. The functionality Spark provides for "decommission" is >> basically a "graceful shutdown" of the executor. It sounds to me as >> redundant. >> >> On Thu, Oct 10, 2024 at 11:11 AM Ángel <angel.alvarez.pas...@gmail.com> >> wrote: >> >>> Looks like it actually exists ... but only for the Spark Synapse >>> implementation ... >>> >>> >>> https://learn.microsoft.com/en-us/answers/questions/1496283/purpose-of-spark-yarn-executor-decommission-enable >>> >>> >>> Jay Han was asking for some config on k8s, so .... we shouldn't bring >>> this config to the table, should we? >>> >>> El jue, 10 oct 2024 a las 2:55, Sean Owen (<sro...@gmail.com>) escribió: >>> >>>> Mich: you can set any key-value pair you want in Spark config. It >>>> doesn't mean it is a real flag that code reads. >>>> >>>> spark.conf.set("ham", "sandwich") >>>> print(spark.conf.get("ham")) >>>> >>>> prints "sandwich" >>>> >>>> forceKillTimeout is a real config: >>>> >>>> https://github.com/apache/spark/blob/fed9a8da3d4187794161e0be325aa96be8487783/core/src/main/scala/org/apache/spark/internal/config/package.scala#L2394 >>>> >>>> The others I cannot find, as in: >>>> >>>> https://github.com/search?q=repo%3Aapache%2Fspark%20spark.executor.decommission.gracefulShutdown&type=code >>>> >>>> If you're continuing to suggest these are real configs, where are you >>>> finding those two in any docs or source? >>>> Or what config were you thinking of if it's a typo? >>>> >>>> On Wed, Oct 9, 2024 at 5:14 PM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> Let us take this for a ride using these so called non-existent >>>>> configuration settings >>>>> >>>>> spark.executor.decommission.enabled=true >>>>> spark.executor.decommission.gracefulShutdown=true >>>>> >>>>> Tested on Spark 3.4 >>>>> >>>>> from pyspark.sql import SparkSession >>>>> # Initialize a Spark session >>>>> spark = SparkSession.builder \ >>>>> .appName("Verifying Spark Configurations") \ >>>>> .config("spark.executor.decommission.enabled", "true") \ >>>>> .config("spark.executor.decommission.forceKillTimeout", "100s") \ >>>>> .getOrCreate() >>>>> >>>>> # Access Spark context >>>>> sc = spark.sparkContext >>>>> # Set the log level to ERROR to reduce verbosity >>>>> sc.setLogLevel("ERROR") >>>>> print(f"\n\nSpark version: ", sc.version) >>>>> >>>>> # Verify the configuration for executor decommissioning >>>>> decommission_enabled = >>>>> sc.getConf().get("spark.executor.decommission.enabled", "false") >>>>> force_kill_timeout = >>>>> sc.getConf().get("spark.executor.decommission.forceKillTimeout", >>>>> "default_value") >>>>> >>>>> # Print the values >>>>> print(f"spark.executor.decommission.enabled: {decommission_enabled}") >>>>> print(f"spark.executor.decommission.forceKillTimeout: >>>>> {force_kill_timeout}") >>>>> >>>>> The output >>>>> >>>>> Spark version: 3.4.0 >>>>> spark.executor.decommission.enabled: true >>>>> spark.executor.decommission.forceKillTimeout: 100s >>>>> >>>>> By creating a simple Spark application and verifying the configuration >>>>> values, I trust it is shown that these two parameters are valid and are >>>>> applied by Spark >>>>> >>>>> HTH >>>>> >>>>> Mich Talebzadeh, >>>>> >>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>> College London <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>> >>>>> London, United Kingdom >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> >>>>> >>>>> >>>>> *Disclaimer:* The information provided is correct to the best of my >>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>> expert opinions (Werner >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>> >>>>> >>>>> On Wed, 9 Oct 2024 at 16:51, Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> Do you have a better recommendation? >>>>>> >>>>>> Or trying to waste time as usual. >>>>>> >>>>>> It is far easier to throw than catch. >>>>>> >>>>>> Do your homework and stop throwing spanners at work. >>>>>> >>>>>> Mich Talebzadeh, >>>>>> >>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>> College London >>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>> London, United Kingdom >>>>>> >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>> expert opinions (Werner >>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>> >>>>>> >>>>>> On Wed, 9 Oct 2024 at 16:43, Nicholas Chammas < >>>>>> nicholas.cham...@gmail.com> wrote: >>>>>> >>>>>>> Mich, >>>>>>> >>>>>>> Can you please share with the list where *exactly* you are citing >>>>>>> these configs from? >>>>>>> >>>>>>> As far as I can tell, these two configs don’t exist and have never >>>>>>> existed in the Spark codebase: >>>>>>> >>>>>>> spark.executor.decommission.enabled=true >>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>> >>>>>>> Where exactly are you getting this information from (and then >>>>>>> posting it to the list as advice)? Please be clear and provide specific >>>>>>> references. >>>>>>> >>>>>>> Nick >>>>>>> >>>>>>> >>>>>>> On Oct 9, 2024, at 1:20 PM, Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>> Before responding, what configuration parameters are you using to >>>>>>> make this work? >>>>>>> >>>>>>> spark.executor.decommission.enabled=true >>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>> spark.executor.decommission.forceKillTimeout=100s >>>>>>> >>>>>>> HTH >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> >>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>> College London >>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>> London, United Kingdom >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>> >>>>>>> >>>>>>> On Wed, 9 Oct 2024 at 11:05, Jay Han <tunyu...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi spark community, >>>>>>>> I have such a question: Why driver doesn't shutdown executors >>>>>>>> gracefully on k8s. For instance, >>>>>>>> kubernetesClient.pods().withGracePeriod(100).delete(). >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best, >>>>>>>> Jay >>>>>>>> >>>>>>> >>>>>>> -- Liu Cao
