Hi Bhuwan, I hope the team got a chance to review the draft PR, looking for some comments to see if the plan looks alright?. I have updated the document about the risks <https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit#heading=h.577aawlyiedf>.(also mentioned below). Please confirm if it looks alright?
*Spark application type* *auto-scaling capability* *with New auto-scaling capability* Spark Batch job Works with current DRA No - change Streaming query without trigger interval No implementation Can work with this implementation - (have to set certain scale back configs based on previous usage pattern) - maybe automate with future work? Spark Streaming query with Trigger interval No implementation With this implementation Spark Streaming query with one-time micro batch Works with current DRA No - change Spark Streaming query with Availablenow micro batch Works with current DRA No - change Batch + Streaming query ( default/ triggger-interval/ once/ availablenow modes), other notebook use cases. No implementation No implementation We are more than happy to collaborate on a call to make better progress on this enhancement. Please let us know. Thank you, Pavan On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > > Hi Bhuwan et al, > > Thank you for passing on the DataBricks Structured Streaming team's review > of the SPIP document. FYI, I work closely with Pawan and other members to > help deliver this piece of work. We appreciate your insights, especially > regarding the cost savings potential from the PoC. > > Pavan already furnished you with some additional info. Your team's point > about the SPIP currently addressing a specific use case (single streaming > query with Processing Time trigger) is well-taken. We agree that > maintaining simplicity is key, particularly as we explore more general > resource allocation mechanisms in the future. To address the concerns and > foster open discussion, The DataBricks team are invited to directly add > their comments and suggestions to the Jira itself > > [SPARK-24815] Structured Streaming should support dynamic allocation - ASF > JIRA (apache.org) > <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> > This will ensure everyone involved can benefit from your team's expertise > and facilitate further collaboration. > > Thanks > > Mich Talebzadeh, > Dad | Technologist | Solutions Architect | Engineer > London > United Kingdom > > > view my Linkedin profile > <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCNE366aQ$> > > > https://en.everybodywiki.com/Mich_Talebzadeh > <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCJndqi8A$> > > > > *Disclaimer:* The information provided is correct to the best of my > knowledge but of course cannot be guaranteed . It is essential to note > that, as with any advice, quote "one test result is worth one-thousand > expert opinions (Werner > <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$>Von > Braun > <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$> > )". > > > On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi > <pkotikalap...@twilio.com.invalid> wrote: > >> Thanks Bhuwan and rest of the databricks team for the reviews, >> >> I appreciate your reviews, was very helpful in evaluating a few options >> that were overlooked earlier (especially about mixed spark apps running on >> notebooks). Regarding the use-cases, It could handle multiple streaming >> queries provided that they are run on the same trigger interval processing >> time (very similar to how current batch dra is set up)..but I felt like it >> would be beneficial if we separate out streaming queries when setting up >> production pipelines. >> >> Regarding the implementation, here is the draft PR >> https://github.com/apache/spark/pull/42352 >> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UC8iQomlg$>. >> (already mentioned in ticket SPARK-24815 >> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >> ) >> >> I have built it on top of the current Dynamic resource allocation (DRA) >> algorithm >> <https://urldefense.com/v3/__https://spark.apache.org/docs/latest/job-scheduling.html*dynamic-resource-allocation__;Iw!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBK8f68fQ$> >> . >> While current DRA is catered towards batch jobs. This implementation just >> makes few changes to that algorithm to >> - do gradual scale-back. The remove-policy still applies (uses 2 old >> configs we currently have), but we now remove few executors per round of >> evaluation ( I have added 2 configs to tune that) >> - The scale-out process also still uses the same request policy (same >> uses 2 old configs we currently have). >> - while we are using the old configs in the both scale-out/back, the >> difference is that we are now giving configs to them based on the trigger >> interval as our north star. >> >> This implementation is just changes in 2 files to make it work. I have >> made the changes minimal/limited to just the core module of the spark repo. >> 1) to make sure it is applied on primitives of task, stage, job which the >> current dra is already doing. (This will enable us to think about other >> cases like default and continuous mode can still work provided we have a >> target processing time range we want to achieve) >> 2) We are reusing ExecutorAllocationClient, ExecutorMonitor and listeners >> which are already well tested and working well for batch job use case. >> >> We internally (in the company) have also added helpers so that we have >> less configs to tune. I can contribute that as well, if it makes the dev >> experience better. >> >> Feel free to review the PR, when we decide the direction is alright I >> will start adding the tests as well. >> >> On a side note. Maybe we should consider some future work to have DRA >> algo per query (batch, streaming queries, mixed etc) rather than per spark >> context. >> >> Thank you, >> >> Pavan >> >> >> On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni >> <bhuwan.sa...@databricks.com.invalid> wrote: >> >>> Hi Pavan, >>> >>> I am from the DataBricks Structured Streaming team, and we did a review >>> of the SPIP internally. Wanted to pass on the points discussed in the >>> meeting. >>> >>> Thanks for putting together the SPIP document. It's useful to have >>> dynamic resource allocation for Streaming queries, and it's exciting to see >>> the cost saving numbers from your PoC. However, in general we discovered >>> that the SPIP addresses a very particular use-case (single streaming query >>> in Spark cluster with Processing time Trigger). Keeping that in mind, it's >>> useful to make sure that the implementation is simple. Can you please share >>> your PoC implementation to understand the code complexity. This would help >>> us to ensure that dynamic resource allocation mechanism for Streaming >>> queries does not become complicated in the future (if more resource >>> allocation mechanisms are added to address other use-cases). >>> >>> Thanks, >>> Bhuwan >>> >>> On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>>> Hi Pavan and those who kindly voted for this SPIP >>>> >>>> Great to have 6+ votes and no -1 and 0. The so-called mass volume is >>>> there. The rest is admin matter and how to drive the project forward and >>>> yes there is more than one way of skinning the cat. I think we need some >>>> flexibility in the rules given the dwindling (IMO) number of comitters who >>>> are willing or actively participating. For example, on a similar matter I >>>> approached Codi Koeninger who was one of the founders of Spark Streaming, >>>> to shepherd a project almost a year back. Sadly he is no longer active and >>>> quotes "I haven't been involved lately and would be missing a lot of >>>> context." So we need to improvise and see how best we can drive this >>>> and similar ones. We wait a short while for a response otherwise I am happy >>>> to give a hand if needed and work with you guys to drive this. It is >>>> something worthwhile. >>>> >>>> HTH >>>> >>>> T >>>> Mich Talebzadeh, >>>> Dad | Technologist | Solutions Architect | Engineer >>>> London >>>> United Kingdom >>>> >>>> >>>> view my Linkedin profile >>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfbGToAE$> >>>> >>>> >>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWTA0_mlE$> >>>> >>>> >>>> >>>> *Disclaimer:* The information provided is correct to the best of my >>>> knowledge but of course cannot be guaranteed . It is essential to note >>>> that, as with any advice, quote "one test result is worth one-thousand >>>> expert opinions (Werner >>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$>Von >>>> Braun >>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$> >>>> )". >>>> >>>> >>>> On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi >>>> <pkotikalap...@twilio.com.invalid> wrote: >>>> >>>>> Thanks for the pointers Mich, will wait for Jungtaek Lee or any other >>>>> PMC members to respond. >>>>> >>>>> aggregating upvotes to this email thread >>>>> >>>>> +6 >>>>> Mich Talebzadeh >>>>> Adam Hobbs >>>>> Pavan Kotikalapudi >>>>> Krystal Mitchell >>>>> Sona Torosyan >>>>> Aaron Kern >>>>> >>>>> Thank you, >>>>> >>>>> Pavan >>>>> >>>>> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> please check this doc >>>>>> >>>>>> Spark Project Improvement Proposals (SPIP) | Apache Spark >>>>>> <https://urldefense.com/v3/__https://spark.apache.org/improvement-proposals.html__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5qy0rbHA$> >>>>>> >>>>>> and specifically the below extract >>>>>> >>>>>> Discussing an SPIP >>>>>> >>>>>> All discussion of an SPIP should take place in a public forum, >>>>>> preferably the discussion attached to the Jira. Any discussions that >>>>>> happen >>>>>> offline should be made available online for the public via meeting notes >>>>>> summarizing the discussions.(done) >>>>>> >>>>>> During this discussion, one or more shepherds should be identified >>>>>> among PMC members. (outstanding) >>>>>> >>>>>> Once the discussion settles, the shepherd(s) should call for a vote >>>>>> on the SPIP moving forward on the dev@ list. The vote should be open >>>>>> for at least 72 hours and follows the typical Apache vote process and >>>>>> passes upon consensus (at least 3 +1 votes from PMC members and no -1 >>>>>> votes >>>>>> from PMC members). dev@ should be notified of the vote result. >>>>>> >>>>>> If there does not exist at least one PMC member that is committed to >>>>>> shepherding the change within a month, the SPIP is rejected. >>>>>> >>>>>> If a committer does not think a SPIP aligns with long-term project >>>>>> goals, or is not practical at the point of proposal, the committer should >>>>>> -1 the SPIP explicitly and give technical justifications. >>>>>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee can >>>>>> kindly help the process >>>>>> >>>>>> cheers >>>>>> >>>>>> Mich Talebzadeh, >>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>> London >>>>>> United Kingdom >>>>>> >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL6nGmLi3g$> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5rLq6E3w$> >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>> expert opinions (Werner >>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$>Von >>>>>> Braun >>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$> >>>>>> )". >>>>>> >>>>>> >>>>>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi >>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>> >>>>>>> Hi Mich, >>>>>>> >>>>>>> We have >>>>>>> >>>>>>> five +1s till now. >>>>>>> >>>>>>> Mich Talebzadeh >>>>>>> Adam Hobbs >>>>>>> Pavan Kotikalapudi >>>>>>> Krystal Mitchell >>>>>>> Sona Torosyan >>>>>>> (few more in github pr) >>>>>>> +0: None >>>>>>> >>>>>>> -1: None >>>>>>> >>>>>>> Does it pass the required condition as approved? >>>>>>> >>>>>>> >>>>>>> Not sure of that though, nothing about minimum required is mentioned >>>>>>> in the past emails. >>>>>>> >>>>>>> I would request spark PMC members or any others who have done this >>>>>>> in the past to understand the process better. >>>>>>> >>>>>>> Thank you, >>>>>>> >>>>>>> Pavan >>>>>>> >>>>>>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Pavan, >>>>>>>> >>>>>>>> Do you have a list of votes for this feature by any chance? Does it >>>>>>>> pass the required condition as approved? >>>>>>>> >>>>>>>> HTH >>>>>>>> >>>>>>>> Mich Talebzadeh, >>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>> London >>>>>>>> United Kingdom >>>>>>>> >>>>>>>> >>>>>>>> view my Linkedin profile >>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A1aYWcDAg$> >>>>>>>> >>>>>>>> >>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0gQVKWXw$> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>> note >>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>> expert opinions (Werner >>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$>Von >>>>>>>> Braun >>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$> >>>>>>>> )". >>>>>>>> >>>>>>>> >>>>>>>> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi >>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>> >>>>>>>>> Yes. The PR was closed due to inactivity by github actions.. >>>>>>>>> >>>>>>>>> The msg >>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1865306284__;Iw!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A113artKQ$> >>>>>>>>> also >>>>>>>>> says >>>>>>>>> >>>>>>>>> > If you'd like to revive this PR, please reopen it and ask a >>>>>>>>> committer to remove the Stale tag! >>>>>>>>> >>>>>>>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh < >>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> I can see it was closed. Was it because of inactivity? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Mich Talebzadeh, >>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>> London >>>>>>>>>> United Kingdom >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> view my Linkedin profile >>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lCFDAOXA$> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7kBRUgBOQ$> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>>> note >>>>>>>>>> that, as with any advice, quote "one test result is worth >>>>>>>>>> one-thousand >>>>>>>>>> expert opinions (Werner >>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$>Von >>>>>>>>>> Braun >>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> >>>>>>>>>> )". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi >>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Spark PMC members, >>>>>>>>>>> >>>>>>>>>>> I think we have few upvotes for this effort here and more people >>>>>>>>>>> are showing interest (see PR comments >>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1955238640__;Iw!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>>>>> .) >>>>>>>>>>> >>>>>>>>>>> Is anyone interested in mentoring and reviewing this effort? >>>>>>>>>>> >>>>>>>>>>> Also can the repository admin/owner re-open the PR? ( I guess >>>>>>>>>>> people only with admin access to the repository can do that). >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> >>>>>>>>>>> Pavan >>>>>>>>>>> >>>>>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>>>>> <kryst...@adobe.com.invalid> wrote: >>>>>>>>>>> >>>>>>>>>>>> +1 >>>>>>>>>>>> >>>>>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>>>>> > >>>>>>>>>>>> > adding some references to the thread. >>>>>>>>>>>> > >>>>>>>>>>>> > - Jira ticket - SPARK-24815 >>>>>>>>>>>> > <https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>>>>> > - Design Doc >>>>>>>>>>>> > < >>>>>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>>>>> > >>>>>>>>>>>> > - discussion thread >>>>>>>>>>>> > < >>>>>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>>>>> > - PR with initial implementation - >>>>>>>>>>>> > https://github.com/apache/spark/pull/42352 >>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>>>>> > >>>>>>>>>>>> > Please vote with: >>>>>>>>>>>> > >>>>>>>>>>>> > [ ] +1: Accept the proposal and start with the development. >>>>>>>>>>>> > [ ] +0 >>>>>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>>>>> > >>>>>>>>>>>> > Thank you, >>>>>>>>>>>> > >>>>>>>>>>>> > Pavan >>>>>>>>>>>> > >>>>>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh < >>>>>>>>>>>> mi...@gmail.com> >>>>>>>>>>>> > wrote: >>>>>>>>>>>> > >>>>>>>>>>>> > > >>>>>>>>>>>> > > +1 for me (non binding) >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>>>> responsibility for any >>>>>>>>>>>> > > loss, damage or destruction of data or any other property >>>>>>>>>>>> which may arise >>>>>>>>>>>> > > from relying on this email's technical content is >>>>>>>>>>>> explicitly disclaimed. >>>>>>>>>>>> > > The author will in no case be liable for any monetary >>>>>>>>>>>> damages arising from >>>>>>>>>>>> > > such loss, damage or destruction. >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > >>>>>>>>>>>> >>>>>>>>>>> >>> >>> -- >>> >>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfJvj8wi$> >>> *Bhuwan Sahni* >>> Staff Software Engineer >>> >>> bhuwan.sa...@databricks.com >>> 500 108th Ave. NE >>> Bellevue, WA 98004 >>> USA >>> >>
