Thank you for your prompt reply. By the way, I am utilizing Spark 2.4.4. My question is as the following: When the Spark driver tries to remove executors from Kubernetes, it invokes pods().delete() without specifying a grace period, regardless of whether it's due to job failure or success. If this is an intentional design by the Spark developers, is it possible for me to alter this default behavior, like pods().gracePeriod(30).delete()?
Mich Talebzadeh <mich.talebza...@gmail.com> 于2024年10月10日周四 16:22写道： > to be clear are you referring to these > > spark.executor.decommission.enabled=true > spark.executor.decommission.gracefulShutdown=true > > thanks > > Mich Talebzadeh, > > Architect | Data Engineer | Data Science | Financial Crime > PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial College > London <https://en.wikipedia.org/wiki/Imperial_College_London> > London, United Kingdom > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von > Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". > > > On Thu, 10 Oct 2024 at 09:03, Ángel <angel.alvarez.pas...@gmail.com> > wrote: > >> Do you know by any chance if that config also applies to Databricks? >> >> El jue, 10 oct 2024 a las 10:02, Ángel (<angel.alvarez.pas...@gmail.com>) >> escribió: >> >>> Thanks a lot for the clarification. Interesting... I've never needed it, >>> even though I've been using Spark for over 8 years. >>> >>> El jue, 10 oct 2024 a las 9:21, Liu Cao (<twcnnj...@gmail.com>) >>> escribió: >>> >>>> I’m unclear on what the exact issue the OP ran into. >>>> >>>> But if we are talking about decommission, just one side note: >>>> >>>> The decommission feature >>>> <https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?tab=t.0#heading=h.70pylrl2vdg8> >>>> has been in spark for a while, and decommission on K8S and standalone was >>>> GA-ed in 3.1 actually. See >>>> https://spark.apache.org/releases/spark-release-3-1-1.html >>>> >>>> The fact that you didn’t see it in the 3.3 site is simply a lack of >>>> documentation. The missing documentation was added in 3.4, thanks to >>>> https://github.com/apache/spark/pull/38131/files >>>> >>>> On Wed, Oct 9, 2024 at 10:13 PM Ángel <angel.alvarez.pas...@gmail.com> >>>> wrote: >>>> >>>>> The Synapse config "spark.yarn.executor.decommission.enabled" is the >>>>> closest thing to the proposed config "spark.executor.decommission.enabled" >>>>> we've seen so far, I was only remarking that. >>>>> >>>>> On the other hand, seems like the "decomission" config came out in >>>>> Spark 3.4.0: >>>>> >>>>> https://archive.apache.org/dist/spark/docs/3.3.4/configuration.html >>>>> https://archive.apache.org/dist/spark/docs/3.4.0/configuration.html >>>>> >>>>> >>>>> >>>>> El jue, 10 oct 2024 a las 4:44, Jungtaek Lim (< >>>>> kabhwan.opensou...@gmail.com>) escribió: >>>>> >>>>>> Ángel, >>>>>> >>>>>> https://spark.apache.org/docs/latest/configuration.html >>>>>> search through `decommission` in this page. >>>>>> >>>>>> The config you may have found from Synapse is >>>>>> spark."yarn".executor.decommission.enabled. And the question was even >>>>>> "k8s" >>>>>> and none of the information for the vendor was mentioned from the >>>>>> question. >>>>>> I don't even think these configs are on the internet - simply saying, >>>>>> google them (be sure to wrap with double quotes). >>>>>> >>>>>> Interestingly, I wouldn't even expect the config for graceful >>>>>> shutdown for decommission. The functionality Spark provides for >>>>>> "decommission" is basically a "graceful shutdown" of the executor. It >>>>>> sounds to me as redundant. >>>>>> >>>>>> On Thu, Oct 10, 2024 at 11:11 AM Ángel < >>>>>> angel.alvarez.pas...@gmail.com> wrote: >>>>>> >>>>>>> Looks like it actually exists ... but only for the Spark Synapse >>>>>>> implementation ... >>>>>>> >>>>>>> >>>>>>> https://learn.microsoft.com/en-us/answers/questions/1496283/purpose-of-spark-yarn-executor-decommission-enable >>>>>>> >>>>>>> >>>>>>> Jay Han was asking for some config on k8s, so .... we shouldn't >>>>>>> bring this config to the table, should we? >>>>>>> >>>>>>> El jue, 10 oct 2024 a las 2:55, Sean Owen (<sro...@gmail.com>) >>>>>>> escribió: >>>>>>> >>>>>>>> Mich: you can set any key-value pair you want in Spark config. It >>>>>>>> doesn't mean it is a real flag that code reads. >>>>>>>> >>>>>>>> spark.conf.set("ham", "sandwich") >>>>>>>> print(spark.conf.get("ham")) >>>>>>>> >>>>>>>> prints "sandwich" >>>>>>>> >>>>>>>> forceKillTimeout is a real config: >>>>>>>> >>>>>>>> https://github.com/apache/spark/blob/fed9a8da3d4187794161e0be325aa96be8487783/core/src/main/scala/org/apache/spark/internal/config/package.scala#L2394 >>>>>>>> >>>>>>>> The others I cannot find, as in: >>>>>>>> >>>>>>>> https://github.com/search?q=repo%3Aapache%2Fspark%20spark.executor.decommission.gracefulShutdown&type=code >>>>>>>> >>>>>>>> If you're continuing to suggest these are real configs, where are >>>>>>>> you finding those two in any docs or source? >>>>>>>> Or what config were you thinking of if it's a typo? >>>>>>>> >>>>>>>> On Wed, Oct 9, 2024 at 5:14 PM Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Let us take this for a ride using these so called non-existent >>>>>>>>> configuration settings >>>>>>>>> >>>>>>>>> spark.executor.decommission.enabled=true >>>>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>>>> >>>>>>>>> Tested on Spark 3.4 >>>>>>>>> >>>>>>>>> from pyspark.sql import SparkSession >>>>>>>>> # Initialize a Spark session >>>>>>>>> spark = SparkSession.builder \ >>>>>>>>> .appName("Verifying Spark Configurations") \ >>>>>>>>> .config("spark.executor.decommission.enabled", "true") \ >>>>>>>>> .config("spark.executor.decommission.forceKillTimeout", >>>>>>>>> "100s") \ >>>>>>>>> .getOrCreate() >>>>>>>>> >>>>>>>>> # Access Spark context >>>>>>>>> sc = spark.sparkContext >>>>>>>>> # Set the log level to ERROR to reduce verbosity >>>>>>>>> sc.setLogLevel("ERROR") >>>>>>>>> print(f"\n\nSpark version: ", sc.version) >>>>>>>>> >>>>>>>>> # Verify the configuration for executor decommissioning >>>>>>>>> decommission_enabled = >>>>>>>>> sc.getConf().get("spark.executor.decommission.enabled", "false") >>>>>>>>> force_kill_timeout = >>>>>>>>> sc.getConf().get("spark.executor.decommission.forceKillTimeout", >>>>>>>>> "default_value") >>>>>>>>> >>>>>>>>> # Print the values >>>>>>>>> print(f"spark.executor.decommission.enabled: >>>>>>>>> {decommission_enabled}") >>>>>>>>> print(f"spark.executor.decommission.forceKillTimeout: >>>>>>>>> {force_kill_timeout}") >>>>>>>>> >>>>>>>>> The output >>>>>>>>> >>>>>>>>> Spark version: 3.4.0 >>>>>>>>> spark.executor.decommission.enabled: true >>>>>>>>> spark.executor.decommission.forceKillTimeout: 100s >>>>>>>>> >>>>>>>>> By creating a simple Spark application and verifying the >>>>>>>>> configuration values, I trust it is shown that these two parameters >>>>>>>>> are >>>>>>>>> valid and are applied by Spark >>>>>>>>> >>>>>>>>> HTH >>>>>>>>> >>>>>>>>> Mich Talebzadeh, >>>>>>>>> >>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>>> College London >>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>> London, United Kingdom >>>>>>>>> >>>>>>>>> >>>>>>>>> view my Linkedin profile >>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>> note >>>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>>> expert opinions (Werner >>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, 9 Oct 2024 at 16:51, Mich Talebzadeh < >>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Do you have a better recommendation? >>>>>>>>>> >>>>>>>>>> Or trying to waste time as usual. >>>>>>>>>> >>>>>>>>>> It is far easier to throw than catch. >>>>>>>>>> >>>>>>>>>> Do your homework and stop throwing spanners at work. >>>>>>>>>> >>>>>>>>>> Mich Talebzadeh, >>>>>>>>>> >>>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>>>> College London >>>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>>> London, United Kingdom >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> view my Linkedin profile >>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>>> note >>>>>>>>>> that, as with any advice, quote "one test result is worth >>>>>>>>>> one-thousand >>>>>>>>>> expert opinions (Werner >>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, 9 Oct 2024 at 16:43, Nicholas Chammas < >>>>>>>>>> nicholas.cham...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Mich, >>>>>>>>>>> >>>>>>>>>>> Can you please share with the list where *exactly* you are >>>>>>>>>>> citing these configs from? >>>>>>>>>>> >>>>>>>>>>> As far as I can tell, these two configs don’t exist and have >>>>>>>>>>> never existed in the Spark codebase: >>>>>>>>>>> >>>>>>>>>>> spark.executor.decommission.enabled=true >>>>>>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>>>>>> >>>>>>>>>>> Where exactly are you getting this information from (and then >>>>>>>>>>> posting it to the list as advice)? Please be clear and provide >>>>>>>>>>> specific >>>>>>>>>>> references. >>>>>>>>>>> >>>>>>>>>>> Nick >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Oct 9, 2024, at 1:20 PM, Mich Talebzadeh < >>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> Before responding, what configuration parameters are you using >>>>>>>>>>> to make this work? >>>>>>>>>>> >>>>>>>>>>> spark.executor.decommission.enabled=true >>>>>>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>>>>>> spark.executor.decommission.forceKillTimeout=100s >>>>>>>>>>> >>>>>>>>>>> HTH >>>>>>>>>>> >>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>> >>>>>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>>>>> College London >>>>>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>>>>> London, United Kingdom >>>>>>>>>>> >>>>>>>>>>> view my Linkedin profile >>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>> essential to >>>>>>>>>>> note that, as with any advice, quote "one test result is worth >>>>>>>>>>> one-thousand expert opinions (Werner >>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, 9 Oct 2024 at 11:05, Jay Han <tunyu...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi spark community, >>>>>>>>>>>> I have such a question: Why driver doesn't shutdown >>>>>>>>>>>> executors gracefully on k8s. For instance, >>>>>>>>>>>> kubernetesClient.pods().withGracePeriod(100).delete(). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Best, >>>>>>>>>>>> Jay >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>> >>>> -- >>>> >>>> Liu Cao >>>> >>>> >>>> -- Best, Jay
