Thanks a lot for creating the risk table Pavan. My apologies. I was tied up with high priority items for the last couple weeks and could not respond. I will review the PR by tomorrow's end, and get back to you.
Appreciate your patience. Thanks Bhuwan Sahni On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi <pkotikalap...@twilio.com> wrote: > Hi Bhuwan, > > I hope the team got a chance to review the draft PR, looking for some > comments to see if the plan looks alright?. I have updated the document > about the risks > <https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit#heading=h.577aawlyiedf>.(also > mentioned below). Please confirm if it looks alright? > > *Spark application type* > > *auto-scaling capability* > > *with New auto-scaling capability* > > Spark Batch job > > Works with current DRA > > No - change > > Streaming query without trigger interval > > No implementation > > Can work with this implementation - (have to set certain scale back > configs based on previous usage pattern) - maybe automate with future work? > > Spark Streaming query with Trigger interval > > No implementation > > With this implementation > > Spark Streaming query with one-time micro batch > > Works with current DRA > > No - change > > Spark Streaming query with > > Availablenow micro batch > > Works with current DRA > > No - change > > Batch + Streaming query ( > > default/ > > triggger-interval/ > > once/ > > availablenow modes), other notebook use cases. > > No implementation > > No implementation > > > > We are more than happy to collaborate on a call to make better progress > on this enhancement. Please let us know. > > Thank you, > > Pavan > > On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > >> >> Hi Bhuwan et al, >> >> Thank you for passing on the DataBricks Structured Streaming team's >> review of the SPIP document. FYI, I work closely with Pawan and other >> members to help deliver this piece of work. We appreciate your insights, >> especially regarding the cost savings potential from the PoC. >> >> Pavan already furnished you with some additional info. Your team's point >> about the SPIP currently addressing a specific use case (single streaming >> query with Processing Time trigger) is well-taken. We agree that >> maintaining simplicity is key, particularly as we explore more general >> resource allocation mechanisms in the future. To address the concerns and >> foster open discussion, The DataBricks team are invited to directly add >> their comments and suggestions to the Jira itself >> >> [SPARK-24815] Structured Streaming should support dynamic allocation - >> ASF JIRA (apache.org) >> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >> This will ensure everyone involved can benefit from your team's expertise >> and facilitate further collaboration. >> >> Thanks >> >> Mich Talebzadeh, >> Dad | Technologist | Solutions Architect | Engineer >> London >> United Kingdom >> >> >> view my Linkedin profile >> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCNE366aQ$> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCJndqi8A$> >> >> >> >> *Disclaimer:* The information provided is correct to the best of my >> knowledge but of course cannot be guaranteed . It is essential to note >> that, as with any advice, quote "one test result is worth one-thousand >> expert opinions (Werner >> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$>Von >> Braun >> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$> >> )". >> >> >> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi >> <pkotikalap...@twilio.com.invalid> wrote: >> >>> Thanks Bhuwan and rest of the databricks team for the reviews, >>> >>> I appreciate your reviews, was very helpful in evaluating a few options >>> that were overlooked earlier (especially about mixed spark apps running on >>> notebooks). Regarding the use-cases, It could handle multiple streaming >>> queries provided that they are run on the same trigger interval processing >>> time (very similar to how current batch dra is set up)..but I felt like it >>> would be beneficial if we separate out streaming queries when setting up >>> production pipelines. >>> >>> Regarding the implementation, here is the draft PR >>> https://github.com/apache/spark/pull/42352 >>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UC8iQomlg$>. >>> (already mentioned in ticket SPARK-24815 >>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>> ) >>> >>> I have built it on top of the current Dynamic resource allocation (DRA) >>> algorithm >>> <https://urldefense.com/v3/__https://spark.apache.org/docs/latest/job-scheduling.html*dynamic-resource-allocation__;Iw!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBK8f68fQ$> >>> . >>> While current DRA is catered towards batch jobs. This implementation >>> just makes few changes to that algorithm to >>> - do gradual scale-back. The remove-policy still applies (uses 2 old >>> configs we currently have), but we now remove few executors per round of >>> evaluation ( I have added 2 configs to tune that) >>> - The scale-out process also still uses the same request policy (same >>> uses 2 old configs we currently have). >>> - while we are using the old configs in the both scale-out/back, the >>> difference is that we are now giving configs to them based on the trigger >>> interval as our north star. >>> >>> This implementation is just changes in 2 files to make it work. I have >>> made the changes minimal/limited to just the core module of the spark repo. >>> 1) to make sure it is applied on primitives of task, stage, job which >>> the current dra is already doing. (This will enable us to think about other >>> cases like default and continuous mode can still work provided we have a >>> target processing time range we want to achieve) >>> 2) We are reusing ExecutorAllocationClient, ExecutorMonitor and >>> listeners which are already well tested and working well for batch job use >>> case. >>> >>> We internally (in the company) have also added helpers so that we have >>> less configs to tune. I can contribute that as well, if it makes the dev >>> experience better. >>> >>> Feel free to review the PR, when we decide the direction is alright I >>> will start adding the tests as well. >>> >>> On a side note. Maybe we should consider some future work to have DRA >>> algo per query (batch, streaming queries, mixed etc) rather than per spark >>> context. >>> >>> Thank you, >>> >>> Pavan >>> >>> >>> On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni >>> <bhuwan.sa...@databricks.com.invalid> wrote: >>> >>>> Hi Pavan, >>>> >>>> I am from the DataBricks Structured Streaming team, and we did a review >>>> of the SPIP internally. Wanted to pass on the points discussed in the >>>> meeting. >>>> >>>> Thanks for putting together the SPIP document. It's useful to have >>>> dynamic resource allocation for Streaming queries, and it's exciting to see >>>> the cost saving numbers from your PoC. However, in general we discovered >>>> that the SPIP addresses a very particular use-case (single streaming query >>>> in Spark cluster with Processing time Trigger). Keeping that in mind, it's >>>> useful to make sure that the implementation is simple. Can you please share >>>> your PoC implementation to understand the code complexity. This would help >>>> us to ensure that dynamic resource allocation mechanism for Streaming >>>> queries does not become complicated in the future (if more resource >>>> allocation mechanisms are added to address other use-cases). >>>> >>>> Thanks, >>>> Bhuwan >>>> >>>> On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> Hi Pavan and those who kindly voted for this SPIP >>>>> >>>>> Great to have 6+ votes and no -1 and 0. The so-called mass volume is >>>>> there. The rest is admin matter and how to drive the project forward and >>>>> yes there is more than one way of skinning the cat. I think we need some >>>>> flexibility in the rules given the dwindling (IMO) number of comitters who >>>>> are willing or actively participating. For example, on a similar matter I >>>>> approached Codi Koeninger who was one of the founders of Spark Streaming, >>>>> to shepherd a project almost a year back. Sadly he is no longer active and >>>>> quotes "I haven't been involved lately and would be missing a lot of >>>>> context." So we need to improvise and see how best we can drive this >>>>> and similar ones. We wait a short while for a response otherwise I am >>>>> happy >>>>> to give a hand if needed and work with you guys to drive this. It is >>>>> something worthwhile. >>>>> >>>>> HTH >>>>> >>>>> T >>>>> Mich Talebzadeh, >>>>> Dad | Technologist | Solutions Architect | Engineer >>>>> London >>>>> United Kingdom >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfbGToAE$> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWTA0_mlE$> >>>>> >>>>> >>>>> >>>>> *Disclaimer:* The information provided is correct to the best of my >>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>> expert opinions (Werner >>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$>Von >>>>> Braun >>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$> >>>>> )". >>>>> >>>>> >>>>> On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi >>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>> >>>>>> Thanks for the pointers Mich, will wait for Jungtaek Lee or any other >>>>>> PMC members to respond. >>>>>> >>>>>> aggregating upvotes to this email thread >>>>>> >>>>>> +6 >>>>>> Mich Talebzadeh >>>>>> Adam Hobbs >>>>>> Pavan Kotikalapudi >>>>>> Krystal Mitchell >>>>>> Sona Torosyan >>>>>> Aaron Kern >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Pavan >>>>>> >>>>>> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> please check this doc >>>>>>> >>>>>>> Spark Project Improvement Proposals (SPIP) | Apache Spark >>>>>>> <https://urldefense.com/v3/__https://spark.apache.org/improvement-proposals.html__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5qy0rbHA$> >>>>>>> >>>>>>> and specifically the below extract >>>>>>> >>>>>>> Discussing an SPIP >>>>>>> >>>>>>> All discussion of an SPIP should take place in a public forum, >>>>>>> preferably the discussion attached to the Jira. Any discussions that >>>>>>> happen >>>>>>> offline should be made available online for the public via meeting notes >>>>>>> summarizing the discussions.(done) >>>>>>> >>>>>>> During this discussion, one or more shepherds should be identified >>>>>>> among PMC members. (outstanding) >>>>>>> >>>>>>> Once the discussion settles, the shepherd(s) should call for a vote >>>>>>> on the SPIP moving forward on the dev@ list. The vote should be >>>>>>> open for at least 72 hours and follows the typical Apache vote process >>>>>>> and >>>>>>> passes upon consensus (at least 3 +1 votes from PMC members and no -1 >>>>>>> votes >>>>>>> from PMC members). dev@ should be notified of the vote result. >>>>>>> >>>>>>> If there does not exist at least one PMC member that is committed to >>>>>>> shepherding the change within a month, the SPIP is rejected. >>>>>>> >>>>>>> If a committer does not think a SPIP aligns with long-term project >>>>>>> goals, or is not practical at the point of proposal, the committer >>>>>>> should >>>>>>> -1 the SPIP explicitly and give technical justifications. >>>>>>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee can >>>>>>> kindly help the process >>>>>>> >>>>>>> cheers >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>> London >>>>>>> United Kingdom >>>>>>> >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL6nGmLi3g$> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5rLq6E3w$> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$>Von >>>>>>> Braun >>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$> >>>>>>> )". >>>>>>> >>>>>>> >>>>>>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi >>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>> >>>>>>>> Hi Mich, >>>>>>>> >>>>>>>> We have >>>>>>>> >>>>>>>> five +1s till now. >>>>>>>> >>>>>>>> Mich Talebzadeh >>>>>>>> Adam Hobbs >>>>>>>> Pavan Kotikalapudi >>>>>>>> Krystal Mitchell >>>>>>>> Sona Torosyan >>>>>>>> (few more in github pr) >>>>>>>> +0: None >>>>>>>> >>>>>>>> -1: None >>>>>>>> >>>>>>>> Does it pass the required condition as approved? >>>>>>>> >>>>>>>> >>>>>>>> Not sure of that though, nothing about minimum required is >>>>>>>> mentioned in the past emails. >>>>>>>> >>>>>>>> I would request spark PMC members or any others who have done this >>>>>>>> in the past to understand the process better. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Pavan >>>>>>>> >>>>>>>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Pavan, >>>>>>>>> >>>>>>>>> Do you have a list of votes for this feature by any chance? Does >>>>>>>>> it pass the required condition as approved? >>>>>>>>> >>>>>>>>> HTH >>>>>>>>> >>>>>>>>> Mich Talebzadeh, >>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>> London >>>>>>>>> United Kingdom >>>>>>>>> >>>>>>>>> >>>>>>>>> view my Linkedin profile >>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A1aYWcDAg$> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0gQVKWXw$> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>> note >>>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>>> expert opinions (Werner >>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$>Von >>>>>>>>> Braun >>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$> >>>>>>>>> )". >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi >>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>> >>>>>>>>>> Yes. The PR was closed due to inactivity by github actions.. >>>>>>>>>> >>>>>>>>>> The msg >>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1865306284__;Iw!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A113artKQ$> >>>>>>>>>> also >>>>>>>>>> says >>>>>>>>>> >>>>>>>>>> > If you'd like to revive this PR, please reopen it and ask a >>>>>>>>>> committer to remove the Stale tag! >>>>>>>>>> >>>>>>>>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh < >>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> I can see it was closed. Was it because of inactivity? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>> London >>>>>>>>>>> United Kingdom >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> view my Linkedin profile >>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lCFDAOXA$> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7kBRUgBOQ$> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>> essential to >>>>>>>>>>> note that, as with any advice, quote "one test result is worth >>>>>>>>>>> one-thousand expert opinions (Werner >>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$>Von >>>>>>>>>>> Braun >>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> >>>>>>>>>>> )". >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi >>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Spark PMC members, >>>>>>>>>>>> >>>>>>>>>>>> I think we have few upvotes for this effort here and more >>>>>>>>>>>> people are showing interest (see PR comments >>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1955238640__;Iw!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>>>>>> .) >>>>>>>>>>>> >>>>>>>>>>>> Is anyone interested in mentoring and reviewing this effort? >>>>>>>>>>>> >>>>>>>>>>>> Also can the repository admin/owner re-open the PR? ( I guess >>>>>>>>>>>> people only with admin access to the repository can do that). >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> >>>>>>>>>>>> Pavan >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>>>>>> <kryst...@adobe.com.invalid> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> +1 >>>>>>>>>>>>> >>>>>>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>>>>>> > >>>>>>>>>>>>> > adding some references to the thread. >>>>>>>>>>>>> > >>>>>>>>>>>>> > - Jira ticket - SPARK-24815 >>>>>>>>>>>>> > <https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>>>>>> > - Design Doc >>>>>>>>>>>>> > < >>>>>>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>>>>>> > >>>>>>>>>>>>> > - discussion thread >>>>>>>>>>>>> > < >>>>>>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>>>>>> > - PR with initial implementation - >>>>>>>>>>>>> > https://github.com/apache/spark/pull/42352 >>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>>>>>> > >>>>>>>>>>>>> > Please vote with: >>>>>>>>>>>>> > >>>>>>>>>>>>> > [ ] +1: Accept the proposal and start with the development. >>>>>>>>>>>>> > [ ] +0 >>>>>>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>>>>>> > >>>>>>>>>>>>> > Thank you, >>>>>>>>>>>>> > >>>>>>>>>>>>> > Pavan >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh < >>>>>>>>>>>>> mi...@gmail.com> >>>>>>>>>>>>> > wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > +1 for me (non binding) >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>>>>> responsibility for any >>>>>>>>>>>>> > > loss, damage or destruction of data or any other property >>>>>>>>>>>>> which may arise >>>>>>>>>>>>> > > from relying on this email's technical content is >>>>>>>>>>>>> explicitly disclaimed. >>>>>>>>>>>>> > > The author will in no case be liable for any monetary >>>>>>>>>>>>> damages arising from >>>>>>>>>>>>> > > such loss, damage or destruction. >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > >>>>>>>>>>>>> >>>>>>>>>>>> >>>> >>>> -- >>>> >>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfJvj8wi$> >>>> *Bhuwan Sahni* >>>> Staff Software Engineer >>>> >>>> bhuwan.sa...@databricks.com >>>> 500 108th Ave. NE >>>> Bellevue, WA 98004 >>>> USA >>>> >>> -- <http://www.databricks.com> *Bhuwan Sahni* Staff Software Engineer bhuwan.sa...@databricks.com 500 108th Ave. NE Bellevue, WA 98004 USA
