PMC members, can someone please push this thing forward? Thanks! Nimrod
בתאריך יום ג׳, 9 ביולי 2024, 01:57, מאת Pavan Kotikalapudi < pkotikalap...@twilio.com>: > Definitely!. We internally use it extensively in all our apps and would > love to get community feedback. > > I think we have enough work done to move this feature forward. > We had discussion and vote threads already published in the past, but we > need enough backing/votes of the PMC members to take it to completion. > > > cc: @Jungtaek Lim <kabhwan.opensou...@gmail.com>, @Mich Talebzadeh > <mich.talebza...@gmail.com> mentors of this effort. > > Cheers, > > Pavan > > > > > On Mon, Jul 8, 2024 at 10:33 AM Nimrod Ofek <ofek.nim...@gmail.com> wrote: > >> Hi, >> >> Thanks Pavan. >> >> I think that the change is very important due to the amount of Spark >> structured streaming apps running today out there... >> IMHO this should be introduced in the upcoming Spark 4.0.0 version as an >> experimental feature for evaluation by the community... >> >> What should be the next steps to make sure the community gets this >> important feature, at least to evaluate? >> How can the community experiment with it to decide if it's good enough >> for production use? >> >> Thanks! >> Nimrod >> >> On Mon, Jul 8, 2024 at 4:10 PM Pavan Kotikalapudi < >> pkotikalap...@twilio.com> wrote: >> >>> Hi, >>> >>> I have taken up the responsibility for the development of that feature >>> right now. >>> >>> Here is the current work https://github.com/apache/spark/pull/42352 >>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!epwFOKDnUs1tKQwFN1ZD0m0aVwHaHbrrZTjf32J7XWyimbwdEN8gp9lxpAKtEnhurALB3fHf_BDNI6PVIGKj5coZ$> >>> >>> last active email thread (maybe you want to reply to this): Re: Vote on >>> Dynamic resource allocation for structured streaming [SPARK-24815] >>> <https://urldefense.com/v3/__https://lists.apache.org/thread/wpvtvf4w3zygtkfgq4sthbf00y5pqxvr__;!!NCc8flgU!epwFOKDnUs1tKQwFN1ZD0m0aVwHaHbrrZTjf32J7XWyimbwdEN8gp9lxpAKtEnhurALB3fHf_BDNI6PVIPycqhCt$> >>> >>> doc: Dynamic resource allocation for structured streaming >>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit__;!!NCc8flgU!epwFOKDnUs1tKQwFN1ZD0m0aVwHaHbrrZTjf32J7XWyimbwdEN8gp9lxpAKtEnhurALB3fHf_BDNI6PVIN-NBSBO$> >>> . >>> >>> This still needs to be reviewed/approved by PMC members, so not sure >>> about the timeline at this point. >>> >>> Thanks, >>> >>> Pavan >>> >>> >>> >>> On Thu, Jul 4, 2024 at 10:46 AM Nimrod Ofek <ofek.nim...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I remember there was a discussion about better supporting auto scaling >>>> for structured streaming. >>>> Is there anything happening with that for the upcoming Spark 4.0 >>>> release? >>>> Will there be support for auto scaling (at least on K8s) spark >>>> structured streaming apps? >>>> >>>> Thanks, >>>> Nimrod >>>> >>>
