Hi Pavan, Thanks for instigating this proposal. Looks like the proposal is ready and has enough votes to be implemented. Having a sheppard will make it more fruitful.
I will leave it to @Jungtaek Lim <kabhwan.opensou...@gmail.com> 's capable hands to drive it forward. Will be there to help if needed. Cheers Mich Talebzadeh, Technologist | Solutions Architect | Data Engineer | Generative AI London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Tue, 26 Mar 2024 at 10:02, Pavan Kotikalapudi <pkotikalap...@twilio.com> wrote: > Hi Bhuwan, > > Glad to hear back from you! Very much appreciate your help on reviewing > the design doc/PR and endorsing this proposal. > > Thank you so much @Jungtaek Lim <kabhwan.opensou...@gmail.com> , @Mich > Talebzadeh <mich.talebza...@gmail.com> for graciously agreeing to > mentor/shepherd this effort. > > Regarding Twilio copyright in Notice binary file: > Twilio Opensource counsel was involved all through the process, I have > placed it in the project file prior to Twilio signing a CCLA for the spark > project contribution( Aug '23). > > Since the CCLA is signed now, I have removed the twilio copyright from > that file. I didn't get a chance to update the PR after github-actions > closed it. > > Please let me know of next steps needed to bring this draft PR/effort to > completion. > > Thank you, > > Pavan > > > On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> I'm happy to, but it looks like I need to check one more thing about the >> license, according to the WIP PR >> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrA6JBQ8S0$> >> . >> >> @Pavan Kotikalapudi <pkotikalap...@twilio.com> >> I see you've added the copyright of Twilio in the NOTICE-binary file, >> which makes me wonder if Twilio had filed CCLA to the Apache Software >> Foundation. >> >> PMC members can correct me if I'm mistaken, but from my understanding >> (and experiences of PMC member in other ASF project), code contribution is >> considered as code donation and copyright belongs to ASF. That's why you >> can't find the copyright of employers for contributors in the codebase. >> What you see copyrights in NOTICE-binary is due to the fact we have binary >> dependency and their licenses may require to explicitly mention about >> copyright. It's not about direct code contribution. >> >> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior, >> could you please engage with a relevant group in the company (could be a >> legal team, or similar with OSS advocate team if there is any) and ensure >> that CCLA is filed? The copyright issue is a legal issue, so we have to be >> conservative and 100% sure that the employer is aware of what is the >> meaning of donating the code to ASF via reviewing CCLA and relevant doc, >> and explicitly express that they are OK with it via filing CCLA. >> >> You can read the description of agreements on contribution and ICLA/CCLA >> form from this page. >> https://www.apache.org/licenses/contributor-agreements.html >> <https://urldefense.com/v3/__https://www.apache.org/licenses/contributor-agreements.html__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAktmm6BY$> >> >> Please let me know if this is resolved. This seems to me as a blocker to >> move on. Please also let me know if the contribution is withdrawn from the >> employer. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >> >> On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni >> <bhuwan.sa...@databricks.com.invalid> wrote: >> >>> Hi Pavan, >>> >>> I looked at the PR, and the changes look simple and contained. It would >>> be useful to add dynamic resource allocation to Spark Structured Streaming. >>> >>> Jungtaek. Would you be able to shepherd this change? >>> >>> >>> On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni < >>> bhuwan.sa...@databricks.com> wrote: >>> >>>> Thanks a lot for creating the risk table Pavan. My apologies. I was >>>> tied up with high priority items for the last couple weeks and could not >>>> respond. I will review the PR by tomorrow's end, and get back to you. >>>> >>>> Appreciate your patience. >>>> >>>> Thanks >>>> Bhuwan Sahni >>>> >>>> On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi < >>>> pkotikalap...@twilio.com> wrote: >>>> >>>>> Hi Bhuwan, >>>>> >>>>> I hope the team got a chance to review the draft PR, looking for some >>>>> comments to see if the plan looks alright?. I have updated the document >>>>> about the risks >>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit*heading=h.577aawlyiedf__;Iw!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAzuRa_bM$>.(also >>>>> mentioned below). Please confirm if it looks alright? >>>>> >>>>> *Spark application type* >>>>> >>>>> *auto-scaling capability* >>>>> >>>>> *with New auto-scaling capability* >>>>> >>>>> Spark Batch job >>>>> >>>>> Works with current DRA >>>>> >>>>> No - change >>>>> >>>>> Streaming query without trigger interval >>>>> >>>>> No implementation >>>>> >>>>> Can work with this implementation - (have to set certain scale back >>>>> configs based on previous usage pattern) - maybe automate with future >>>>> work? >>>>> >>>>> Spark Streaming query with Trigger interval >>>>> >>>>> No implementation >>>>> >>>>> With this implementation >>>>> >>>>> Spark Streaming query with one-time micro batch >>>>> >>>>> Works with current DRA >>>>> >>>>> No - change >>>>> >>>>> Spark Streaming query with >>>>> >>>>> Availablenow micro batch >>>>> >>>>> Works with current DRA >>>>> >>>>> No - change >>>>> >>>>> Batch + Streaming query ( >>>>> >>>>> default/ >>>>> >>>>> triggger-interval/ >>>>> >>>>> once/ >>>>> >>>>> availablenow modes), other notebook use cases. >>>>> >>>>> No implementation >>>>> >>>>> No implementation >>>>> >>>>> >>>>> >>>>> We are more than happy to collaborate on a call to make better >>>>> progress on this enhancement. Please let us know. >>>>> >>>>> Thank you, >>>>> >>>>> Pavan >>>>> >>>>> On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> >>>>>> Hi Bhuwan et al, >>>>>> >>>>>> Thank you for passing on the DataBricks Structured Streaming team's >>>>>> review of the SPIP document. FYI, I work closely with Pawan and other >>>>>> members to help deliver this piece of work. We appreciate your insights, >>>>>> especially regarding the cost savings potential from the PoC. >>>>>> >>>>>> Pavan already furnished you with some additional info. Your team's >>>>>> point about the SPIP currently addressing a specific use case (single >>>>>> streaming query with Processing Time trigger) is well-taken. We agree >>>>>> that >>>>>> maintaining simplicity is key, particularly as we explore more general >>>>>> resource allocation mechanisms in the future. To address the concerns and >>>>>> foster open discussion, The DataBricks team are invited to directly add >>>>>> their comments and suggestions to the Jira itself >>>>>> >>>>>> [SPARK-24815] Structured Streaming should support dynamic allocation >>>>>> - ASF JIRA (apache.org) >>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>>>>> This will ensure everyone involved can benefit from your team's >>>>>> expertise and facilitate further collaboration. >>>>>> >>>>>> Thanks >>>>>> >>>>>> Mich Talebzadeh, >>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>> London >>>>>> United Kingdom >>>>>> >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCNE366aQ$> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCJndqi8A$> >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>> expert opinions (Werner >>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$>Von >>>>>> Braun >>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$> >>>>>> )". >>>>>> >>>>>> >>>>>> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi >>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>> >>>>>>> Thanks Bhuwan and rest of the databricks team for the reviews, >>>>>>> >>>>>>> I appreciate your reviews, was very helpful in evaluating a few >>>>>>> options that were overlooked earlier (especially about mixed spark apps >>>>>>> running on notebooks). Regarding the use-cases, It could handle multiple >>>>>>> streaming queries provided that they are run on the same trigger >>>>>>> interval >>>>>>> processing time (very similar to how current batch dra is set up)..but I >>>>>>> felt like it would be beneficial if we separate out streaming queries >>>>>>> when >>>>>>> setting up production pipelines. >>>>>>> >>>>>>> Regarding the implementation, here is the draft PR >>>>>>> https://github.com/apache/spark/pull/42352 >>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UC8iQomlg$>. >>>>>>> (already mentioned in ticket SPARK-24815 >>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>>>>>> ) >>>>>>> >>>>>>> I have built it on top of the current Dynamic resource allocation >>>>>>> (DRA) algorithm >>>>>>> <https://urldefense.com/v3/__https://spark.apache.org/docs/latest/job-scheduling.html*dynamic-resource-allocation__;Iw!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBK8f68fQ$> >>>>>>> . >>>>>>> While current DRA is catered towards batch jobs. This implementation >>>>>>> just makes few changes to that algorithm to >>>>>>> - do gradual scale-back. The remove-policy still applies (uses 2 old >>>>>>> configs we currently have), but we now remove few executors per round of >>>>>>> evaluation ( I have added 2 configs to tune that) >>>>>>> - The scale-out process also still uses the same request policy >>>>>>> (same uses 2 old configs we currently have). >>>>>>> - while we are using the old configs in the both scale-out/back, the >>>>>>> difference is that we are now giving configs to them based on the >>>>>>> trigger >>>>>>> interval as our north star. >>>>>>> >>>>>>> This implementation is just changes in 2 files to make it work. I >>>>>>> have made the changes minimal/limited to just the core module of the >>>>>>> spark >>>>>>> repo. >>>>>>> 1) to make sure it is applied on primitives of task, stage, job >>>>>>> which the current dra is already doing. (This will enable us to think >>>>>>> about >>>>>>> other cases like default and continuous mode can still work provided we >>>>>>> have a target processing time range we want to achieve) >>>>>>> 2) We are reusing ExecutorAllocationClient, ExecutorMonitor and >>>>>>> listeners which are already well tested and working well for batch job >>>>>>> use >>>>>>> case. >>>>>>> >>>>>>> We internally (in the company) have also added helpers so that we >>>>>>> have less configs to tune. I can contribute that as well, if it makes >>>>>>> the >>>>>>> dev experience better. >>>>>>> >>>>>>> Feel free to review the PR, when we decide the direction is alright >>>>>>> I will start adding the tests as well. >>>>>>> >>>>>>> On a side note. Maybe we should consider some future work to have >>>>>>> DRA algo per query (batch, streaming queries, mixed etc) rather than per >>>>>>> spark context. >>>>>>> >>>>>>> Thank you, >>>>>>> >>>>>>> Pavan >>>>>>> >>>>>>> >>>>>>> On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni >>>>>>> <bhuwan.sa...@databricks.com.invalid> wrote: >>>>>>> >>>>>>>> Hi Pavan, >>>>>>>> >>>>>>>> I am from the DataBricks Structured Streaming team, and we did a >>>>>>>> review of the SPIP internally. Wanted to pass on the points discussed >>>>>>>> in >>>>>>>> the meeting. >>>>>>>> >>>>>>>> Thanks for putting together the SPIP document. It's useful to have >>>>>>>> dynamic resource allocation for Streaming queries, and it's exciting >>>>>>>> to see >>>>>>>> the cost saving numbers from your PoC. However, in general we >>>>>>>> discovered >>>>>>>> that the SPIP addresses a very particular use-case (single streaming >>>>>>>> query >>>>>>>> in Spark cluster with Processing time Trigger). Keeping that in mind, >>>>>>>> it's >>>>>>>> useful to make sure that the implementation is simple. Can you please >>>>>>>> share >>>>>>>> your PoC implementation to understand the code complexity. This would >>>>>>>> help >>>>>>>> us to ensure that dynamic resource allocation mechanism for Streaming >>>>>>>> queries does not become complicated in the future (if more resource >>>>>>>> allocation mechanisms are added to address other use-cases). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Bhuwan >>>>>>>> >>>>>>>> On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Pavan and those who kindly voted for this SPIP >>>>>>>>> >>>>>>>>> Great to have 6+ votes and no -1 and 0. The so-called mass volume >>>>>>>>> is there. The rest is admin matter and how to drive the project >>>>>>>>> forward and >>>>>>>>> yes there is more than one way of skinning the cat. I think we need >>>>>>>>> some >>>>>>>>> flexibility in the rules given the dwindling (IMO) number of >>>>>>>>> comitters who >>>>>>>>> are willing or actively participating. For example, on a similar >>>>>>>>> matter I >>>>>>>>> approached Codi Koeninger who was one of the founders of Spark >>>>>>>>> Streaming, >>>>>>>>> to shepherd a project almost a year back. Sadly he is no longer >>>>>>>>> active and >>>>>>>>> quotes "I haven't been involved lately and would be missing a lot >>>>>>>>> of context." So we need to improvise and see how best we can >>>>>>>>> drive this and similar ones. We wait a short while for a response >>>>>>>>> otherwise >>>>>>>>> I am happy to give a hand if needed and work with you guys to drive >>>>>>>>> this. >>>>>>>>> It is something worthwhile. >>>>>>>>> >>>>>>>>> HTH >>>>>>>>> >>>>>>>>> T >>>>>>>>> Mich Talebzadeh, >>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>> London >>>>>>>>> United Kingdom >>>>>>>>> >>>>>>>>> >>>>>>>>> view my Linkedin profile >>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfbGToAE$> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWTA0_mlE$> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>> note >>>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>>> expert opinions (Werner >>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$>Von >>>>>>>>> Braun >>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$> >>>>>>>>> )". >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi >>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>> >>>>>>>>>> Thanks for the pointers Mich, will wait for Jungtaek Lee or any >>>>>>>>>> other PMC members to respond. >>>>>>>>>> >>>>>>>>>> aggregating upvotes to this email thread >>>>>>>>>> >>>>>>>>>> +6 >>>>>>>>>> Mich Talebzadeh >>>>>>>>>> Adam Hobbs >>>>>>>>>> Pavan Kotikalapudi >>>>>>>>>> Krystal Mitchell >>>>>>>>>> Sona Torosyan >>>>>>>>>> Aaron Kern >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> >>>>>>>>>> Pavan >>>>>>>>>> >>>>>>>>>> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh < >>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> please check this doc >>>>>>>>>>> >>>>>>>>>>> Spark Project Improvement Proposals (SPIP) | Apache Spark >>>>>>>>>>> <https://urldefense.com/v3/__https://spark.apache.org/improvement-proposals.html__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5qy0rbHA$> >>>>>>>>>>> >>>>>>>>>>> and specifically the below extract >>>>>>>>>>> >>>>>>>>>>> Discussing an SPIP >>>>>>>>>>> >>>>>>>>>>> All discussion of an SPIP should take place in a public forum, >>>>>>>>>>> preferably the discussion attached to the Jira. Any discussions >>>>>>>>>>> that happen >>>>>>>>>>> offline should be made available online for the public via meeting >>>>>>>>>>> notes >>>>>>>>>>> summarizing the discussions.(done) >>>>>>>>>>> >>>>>>>>>>> During this discussion, one or more shepherds should be >>>>>>>>>>> identified among PMC members. (outstanding) >>>>>>>>>>> >>>>>>>>>>> Once the discussion settles, the shepherd(s) should call for a >>>>>>>>>>> vote on the SPIP moving forward on the dev@ list. The vote >>>>>>>>>>> should be open for at least 72 hours and follows the typical Apache >>>>>>>>>>> vote >>>>>>>>>>> process and passes upon consensus (at least 3 +1 votes from PMC >>>>>>>>>>> members and >>>>>>>>>>> no -1 votes from PMC members). dev@ should be notified of the >>>>>>>>>>> vote result. >>>>>>>>>>> >>>>>>>>>>> If there does not exist at least one PMC member that is >>>>>>>>>>> committed to shepherding the change within a month, the SPIP is >>>>>>>>>>> rejected. >>>>>>>>>>> >>>>>>>>>>> If a committer does not think a SPIP aligns with long-term >>>>>>>>>>> project goals, or is not practical at the point of proposal, the >>>>>>>>>>> committer >>>>>>>>>>> should -1 the SPIP explicitly and give technical justifications. >>>>>>>>>>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee >>>>>>>>>>> can kindly help the process >>>>>>>>>>> >>>>>>>>>>> cheers >>>>>>>>>>> >>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>> London >>>>>>>>>>> United Kingdom >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> view my Linkedin profile >>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL6nGmLi3g$> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5rLq6E3w$> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>> essential to >>>>>>>>>>> note that, as with any advice, quote "one test result is worth >>>>>>>>>>> one-thousand expert opinions (Werner >>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$>Von >>>>>>>>>>> Braun >>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$> >>>>>>>>>>> )". >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi >>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Mich, >>>>>>>>>>>> >>>>>>>>>>>> We have >>>>>>>>>>>> >>>>>>>>>>>> five +1s till now. >>>>>>>>>>>> >>>>>>>>>>>> Mich Talebzadeh >>>>>>>>>>>> Adam Hobbs >>>>>>>>>>>> Pavan Kotikalapudi >>>>>>>>>>>> Krystal Mitchell >>>>>>>>>>>> Sona Torosyan >>>>>>>>>>>> (few more in github pr) >>>>>>>>>>>> +0: None >>>>>>>>>>>> >>>>>>>>>>>> -1: None >>>>>>>>>>>> >>>>>>>>>>>> Does it pass the required condition as approved? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Not sure of that though, nothing about minimum required is >>>>>>>>>>>> mentioned in the past emails. >>>>>>>>>>>> >>>>>>>>>>>> I would request spark PMC members or any others who have done >>>>>>>>>>>> this in the past to understand the process better. >>>>>>>>>>>> >>>>>>>>>>>> Thank you, >>>>>>>>>>>> >>>>>>>>>>>> Pavan >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh < >>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Pavan, >>>>>>>>>>>>> >>>>>>>>>>>>> Do you have a list of votes for this feature by any >>>>>>>>>>>>> chance? Does it pass the required condition as approved? >>>>>>>>>>>>> >>>>>>>>>>>>> HTH >>>>>>>>>>>>> >>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>>> London >>>>>>>>>>>>> United Kingdom >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A1aYWcDAg$> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0gQVKWXw$> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>>> essential to >>>>>>>>>>>>> note that, as with any advice, quote "one test result is >>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$>Von >>>>>>>>>>>>> Braun >>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$> >>>>>>>>>>>>> )". >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi >>>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Yes. The PR was closed due to inactivity by github actions.. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The msg >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1865306284__;Iw!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A113artKQ$> >>>>>>>>>>>>>> also >>>>>>>>>>>>>> says >>>>>>>>>>>>>> >>>>>>>>>>>>>> > If you'd like to revive this PR, please reopen it and ask a >>>>>>>>>>>>>> committer to remove the Stale tag! >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh < >>>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I can see it was closed. Was it because of inactivity? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>>>>> London >>>>>>>>>>>>>>> United Kingdom >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lCFDAOXA$> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7kBRUgBOQ$> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the >>>>>>>>>>>>>>> best of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>>>>> essential >>>>>>>>>>>>>>> to note that, as with any advice, quote "one test result is >>>>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$>Von >>>>>>>>>>>>>>> Braun >>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> >>>>>>>>>>>>>>> )". >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi >>>>>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Spark PMC members, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think we have few upvotes for this effort here and more >>>>>>>>>>>>>>>> people are showing interest (see PR comments >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1955238640__;Iw!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>>>>>>>>>> .) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Is anyone interested in mentoring and reviewing this >>>>>>>>>>>>>>>> effort? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Also can the repository admin/owner re-open the PR? ( I >>>>>>>>>>>>>>>> guess people only with admin access to the repository can do >>>>>>>>>>>>>>>> that). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Pavan >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>>>>>>>>>> <kryst...@adobe.com.invalid> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>>>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > adding some references to the thread. >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > - Jira ticket - SPARK-24815 >>>>>>>>>>>>>>>>> > <https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>>>>>>>>>> > - Design Doc >>>>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > - discussion thread >>>>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>>>>>>>>>> > - PR with initial implementation - >>>>>>>>>>>>>>>>> > https://github.com/apache/spark/pull/42352 >>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Please vote with: >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > [ ] +1: Accept the proposal and start with the >>>>>>>>>>>>>>>>> development. >>>>>>>>>>>>>>>>> > [ ] +0 >>>>>>>>>>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Thank you, >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > Pavan >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh < >>>>>>>>>>>>>>>>> mi...@gmail.com> >>>>>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > +1 for me (non binding) >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>>>>>>>>> responsibility for any >>>>>>>>>>>>>>>>> > > loss, damage or destruction of data or any other >>>>>>>>>>>>>>>>> property which may arise >>>>>>>>>>>>>>>>> > > from relying on this email's technical content is >>>>>>>>>>>>>>>>> explicitly disclaimed. >>>>>>>>>>>>>>>>> > > The author will in no case be liable for any monetary >>>>>>>>>>>>>>>>> damages arising from >>>>>>>>>>>>>>>>> > > such loss, damage or destruction. >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfJvj8wi$> >>>>>>>> *Bhuwan Sahni* >>>>>>>> Staff Software Engineer >>>>>>>> >>>>>>>> bhuwan.sa...@databricks.com >>>>>>>> 500 108th Ave. NE >>>>>>>> Bellevue, WA 98004 >>>>>>>> USA >>>>>>>> >>>>>>> >>>> >>>> -- >>>> >>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrARPj-LXk$> >>>> *Bhuwan Sahni* >>>> Staff Software Engineer >>>> >>>> bhuwan.sa...@databricks.com >>>> 500 108th Ave. NE >>>> Bellevue, WA 98004 >>>> USA >>>> >>> >>> >>> -- >>> >>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrARPj-LXk$> >>> *Bhuwan Sahni* >>> Staff Software Engineer >>> >>> bhuwan.sa...@databricks.com >>> 500 108th Ave. NE >>> Bellevue, WA 98004 >>> USA >>> >>
