Thanks a lot for the clarification. Interesting... I've never needed it, even though I've been using Spark for over 8 years.
El jue, 10 oct 2024 a las 9:21, Liu Cao (<twcnnj...@gmail.com>) escribió: > I’m unclear on what the exact issue the OP ran into. > > But if we are talking about decommission, just one side note: > > The decommission feature > <https://docs.google.com/document/d/1xVO1b6KAwdUhjEJBolVPl9C6sLj7oOveErwDSYdT-pE/edit?tab=t.0#heading=h.70pylrl2vdg8> > has been in spark for a while, and decommission on K8S and standalone was > GA-ed in 3.1 actually. See > https://spark.apache.org/releases/spark-release-3-1-1.html > > The fact that you didn’t see it in the 3.3 site is simply a lack of > documentation. The missing documentation was added in 3.4, thanks to > https://github.com/apache/spark/pull/38131/files > > On Wed, Oct 9, 2024 at 10:13 PM Ángel <angel.alvarez.pas...@gmail.com> > wrote: > >> The Synapse config "spark.yarn.executor.decommission.enabled" is the >> closest thing to the proposed config "spark.executor.decommission.enabled" >> we've seen so far, I was only remarking that. >> >> On the other hand, seems like the "decomission" config came out in Spark >> 3.4.0: >> >> https://archive.apache.org/dist/spark/docs/3.3.4/configuration.html >> https://archive.apache.org/dist/spark/docs/3.4.0/configuration.html >> >> >> >> El jue, 10 oct 2024 a las 4:44, Jungtaek Lim (< >> kabhwan.opensou...@gmail.com>) escribió: >> >>> Ángel, >>> >>> https://spark.apache.org/docs/latest/configuration.html >>> search through `decommission` in this page. >>> >>> The config you may have found from Synapse is >>> spark."yarn".executor.decommission.enabled. And the question was even "k8s" >>> and none of the information for the vendor was mentioned from the question. >>> I don't even think these configs are on the internet - simply saying, >>> google them (be sure to wrap with double quotes). >>> >>> Interestingly, I wouldn't even expect the config for graceful shutdown >>> for decommission. The functionality Spark provides for "decommission" is >>> basically a "graceful shutdown" of the executor. It sounds to me as >>> redundant. >>> >>> On Thu, Oct 10, 2024 at 11:11 AM Ángel <angel.alvarez.pas...@gmail.com> >>> wrote: >>> >>>> Looks like it actually exists ... but only for the Spark Synapse >>>> implementation ... >>>> >>>> >>>> https://learn.microsoft.com/en-us/answers/questions/1496283/purpose-of-spark-yarn-executor-decommission-enable >>>> >>>> >>>> Jay Han was asking for some config on k8s, so .... we shouldn't bring >>>> this config to the table, should we? >>>> >>>> El jue, 10 oct 2024 a las 2:55, Sean Owen (<sro...@gmail.com>) >>>> escribió: >>>> >>>>> Mich: you can set any key-value pair you want in Spark config. It >>>>> doesn't mean it is a real flag that code reads. >>>>> >>>>> spark.conf.set("ham", "sandwich") >>>>> print(spark.conf.get("ham")) >>>>> >>>>> prints "sandwich" >>>>> >>>>> forceKillTimeout is a real config: >>>>> >>>>> https://github.com/apache/spark/blob/fed9a8da3d4187794161e0be325aa96be8487783/core/src/main/scala/org/apache/spark/internal/config/package.scala#L2394 >>>>> >>>>> The others I cannot find, as in: >>>>> >>>>> https://github.com/search?q=repo%3Aapache%2Fspark%20spark.executor.decommission.gracefulShutdown&type=code >>>>> >>>>> If you're continuing to suggest these are real configs, where are you >>>>> finding those two in any docs or source? >>>>> Or what config were you thinking of if it's a typo? >>>>> >>>>> On Wed, Oct 9, 2024 at 5:14 PM Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> Let us take this for a ride using these so called non-existent >>>>>> configuration settings >>>>>> >>>>>> spark.executor.decommission.enabled=true >>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>> >>>>>> Tested on Spark 3.4 >>>>>> >>>>>> from pyspark.sql import SparkSession >>>>>> # Initialize a Spark session >>>>>> spark = SparkSession.builder \ >>>>>> .appName("Verifying Spark Configurations") \ >>>>>> .config("spark.executor.decommission.enabled", "true") \ >>>>>> .config("spark.executor.decommission.forceKillTimeout", "100s") \ >>>>>> .getOrCreate() >>>>>> >>>>>> # Access Spark context >>>>>> sc = spark.sparkContext >>>>>> # Set the log level to ERROR to reduce verbosity >>>>>> sc.setLogLevel("ERROR") >>>>>> print(f"\n\nSpark version: ", sc.version) >>>>>> >>>>>> # Verify the configuration for executor decommissioning >>>>>> decommission_enabled = >>>>>> sc.getConf().get("spark.executor.decommission.enabled", "false") >>>>>> force_kill_timeout = >>>>>> sc.getConf().get("spark.executor.decommission.forceKillTimeout", >>>>>> "default_value") >>>>>> >>>>>> # Print the values >>>>>> print(f"spark.executor.decommission.enabled: {decommission_enabled}") >>>>>> print(f"spark.executor.decommission.forceKillTimeout: >>>>>> {force_kill_timeout}") >>>>>> >>>>>> The output >>>>>> >>>>>> Spark version: 3.4.0 >>>>>> spark.executor.decommission.enabled: true >>>>>> spark.executor.decommission.forceKillTimeout: 100s >>>>>> >>>>>> By creating a simple Spark application and verifying the >>>>>> configuration values, I trust it is shown that these two parameters are >>>>>> valid and are applied by Spark >>>>>> >>>>>> HTH >>>>>> >>>>>> Mich Talebzadeh, >>>>>> >>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>> College London >>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>> London, United Kingdom >>>>>> >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>> expert opinions (Werner >>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>> >>>>>> >>>>>> On Wed, 9 Oct 2024 at 16:51, Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> Do you have a better recommendation? >>>>>>> >>>>>>> Or trying to waste time as usual. >>>>>>> >>>>>>> It is far easier to throw than catch. >>>>>>> >>>>>>> Do your homework and stop throwing spanners at work. >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> >>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>> College London >>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>> London, United Kingdom >>>>>>> >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>> >>>>>>> >>>>>>> On Wed, 9 Oct 2024 at 16:43, Nicholas Chammas < >>>>>>> nicholas.cham...@gmail.com> wrote: >>>>>>> >>>>>>>> Mich, >>>>>>>> >>>>>>>> Can you please share with the list where *exactly* you are citing >>>>>>>> these configs from? >>>>>>>> >>>>>>>> As far as I can tell, these two configs don’t exist and have never >>>>>>>> existed in the Spark codebase: >>>>>>>> >>>>>>>> spark.executor.decommission.enabled=true >>>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>>> >>>>>>>> Where exactly are you getting this information from (and then >>>>>>>> posting it to the list as advice)? Please be clear and provide specific >>>>>>>> references. >>>>>>>> >>>>>>>> Nick >>>>>>>> >>>>>>>> >>>>>>>> On Oct 9, 2024, at 1:20 PM, Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>> Before responding, what configuration parameters are you using to >>>>>>>> make this work? >>>>>>>> >>>>>>>> spark.executor.decommission.enabled=true >>>>>>>> spark.executor.decommission.gracefulShutdown=true >>>>>>>> spark.executor.decommission.forceKillTimeout=100s >>>>>>>> >>>>>>>> HTH >>>>>>>> >>>>>>>> Mich Talebzadeh, >>>>>>>> >>>>>>>> Architect | Data Engineer | Data Science | Financial Crime >>>>>>>> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >>>>>>>> College London >>>>>>>> <https://en.wikipedia.org/wiki/Imperial_College_London> >>>>>>>> London, United Kingdom >>>>>>>> >>>>>>>> view my Linkedin profile >>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>> >>>>>>>> >>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>> note >>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>> expert opinions (Werner >>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>> >>>>>>>> >>>>>>>> On Wed, 9 Oct 2024 at 11:05, Jay Han <tunyu...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi spark community, >>>>>>>>> I have such a question: Why driver doesn't shutdown executors >>>>>>>>> gracefully on k8s. For instance, >>>>>>>>> kubernetesClient.pods().withGracePeriod(100).delete(). >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Best, >>>>>>>>> Jay >>>>>>>>> >>>>>>>> >>>>>>>> > > -- > > Liu Cao > > >
