Hi Bhuwan, Glad to hear back from you! Very much appreciate your help on reviewing the design doc/PR and endorsing this proposal.
Thank you so much @Jungtaek Lim <kabhwan.opensou...@gmail.com> , @Mich Talebzadeh <mich.talebza...@gmail.com> for graciously agreeing to mentor/shepherd this effort. Regarding Twilio copyright in Notice binary file: Twilio Opensource counsel was involved all through the process, I have placed it in the project file prior to Twilio signing a CCLA for the spark project contribution( Aug '23). Since the CCLA is signed now, I have removed the twilio copyright from that file. I didn't get a chance to update the PR after github-actions closed it. Please let me know of next steps needed to bring this draft PR/effort to completion. Thank you, Pavan On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > I'm happy to, but it looks like I need to check one more thing about the > license, according to the WIP PR > <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrA6JBQ8S0$> > . > > @Pavan Kotikalapudi <pkotikalap...@twilio.com> > I see you've added the copyright of Twilio in the NOTICE-binary file, > which makes me wonder if Twilio had filed CCLA to the Apache Software > Foundation. > > PMC members can correct me if I'm mistaken, but from my understanding (and > experiences of PMC member in other ASF project), code contribution is > considered as code donation and copyright belongs to ASF. That's why you > can't find the copyright of employers for contributors in the codebase. > What you see copyrights in NOTICE-binary is due to the fact we have binary > dependency and their licenses may require to explicitly mention about > copyright. It's not about direct code contribution. > > Is Twilio aware of this? Also, if Twilio did not file CCLA in prior, could > you please engage with a relevant group in the company (could be a legal > team, or similar with OSS advocate team if there is any) and ensure that > CCLA is filed? The copyright issue is a legal issue, so we have to be > conservative and 100% sure that the employer is aware of what is the > meaning of donating the code to ASF via reviewing CCLA and relevant doc, > and explicitly express that they are OK with it via filing CCLA. > > You can read the description of agreements on contribution and ICLA/CCLA > form from this page. > https://www.apache.org/licenses/contributor-agreements.html > <https://urldefense.com/v3/__https://www.apache.org/licenses/contributor-agreements.html__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAktmm6BY$> > > Please let me know if this is resolved. This seems to me as a blocker to > move on. Please also let me know if the contribution is withdrawn from the > employer. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > > On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni > <bhuwan.sa...@databricks.com.invalid> wrote: > >> Hi Pavan, >> >> I looked at the PR, and the changes look simple and contained. It would >> be useful to add dynamic resource allocation to Spark Structured Streaming. >> >> Jungtaek. Would you be able to shepherd this change? >> >> >> On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni < >> bhuwan.sa...@databricks.com> wrote: >> >>> Thanks a lot for creating the risk table Pavan. My apologies. I was tied >>> up with high priority items for the last couple weeks and could not >>> respond. I will review the PR by tomorrow's end, and get back to you. >>> >>> Appreciate your patience. >>> >>> Thanks >>> Bhuwan Sahni >>> >>> On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi < >>> pkotikalap...@twilio.com> wrote: >>> >>>> Hi Bhuwan, >>>> >>>> I hope the team got a chance to review the draft PR, looking for some >>>> comments to see if the plan looks alright?. I have updated the document >>>> about the risks >>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit*heading=h.577aawlyiedf__;Iw!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAzuRa_bM$>.(also >>>> mentioned below). Please confirm if it looks alright? >>>> >>>> *Spark application type* >>>> >>>> *auto-scaling capability* >>>> >>>> *with New auto-scaling capability* >>>> >>>> Spark Batch job >>>> >>>> Works with current DRA >>>> >>>> No - change >>>> >>>> Streaming query without trigger interval >>>> >>>> No implementation >>>> >>>> Can work with this implementation - (have to set certain scale back >>>> configs based on previous usage pattern) - maybe automate with future work? >>>> >>>> Spark Streaming query with Trigger interval >>>> >>>> No implementation >>>> >>>> With this implementation >>>> >>>> Spark Streaming query with one-time micro batch >>>> >>>> Works with current DRA >>>> >>>> No - change >>>> >>>> Spark Streaming query with >>>> >>>> Availablenow micro batch >>>> >>>> Works with current DRA >>>> >>>> No - change >>>> >>>> Batch + Streaming query ( >>>> >>>> default/ >>>> >>>> triggger-interval/ >>>> >>>> once/ >>>> >>>> availablenow modes), other notebook use cases. >>>> >>>> No implementation >>>> >>>> No implementation >>>> >>>> >>>> >>>> We are more than happy to collaborate on a call to make better progress >>>> on this enhancement. Please let us know. >>>> >>>> Thank you, >>>> >>>> Pavan >>>> >>>> On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh < >>>> mich.talebza...@gmail.com> wrote: >>>> >>>>> >>>>> Hi Bhuwan et al, >>>>> >>>>> Thank you for passing on the DataBricks Structured Streaming team's >>>>> review of the SPIP document. FYI, I work closely with Pawan and other >>>>> members to help deliver this piece of work. We appreciate your insights, >>>>> especially regarding the cost savings potential from the PoC. >>>>> >>>>> Pavan already furnished you with some additional info. Your team's >>>>> point about the SPIP currently addressing a specific use case (single >>>>> streaming query with Processing Time trigger) is well-taken. We agree that >>>>> maintaining simplicity is key, particularly as we explore more general >>>>> resource allocation mechanisms in the future. To address the concerns and >>>>> foster open discussion, The DataBricks team are invited to directly add >>>>> their comments and suggestions to the Jira itself >>>>> >>>>> [SPARK-24815] Structured Streaming should support dynamic allocation - >>>>> ASF JIRA (apache.org) >>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>>>> This will ensure everyone involved can benefit from your team's >>>>> expertise and facilitate further collaboration. >>>>> >>>>> Thanks >>>>> >>>>> Mich Talebzadeh, >>>>> Dad | Technologist | Solutions Architect | Engineer >>>>> London >>>>> United Kingdom >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCNE366aQ$> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCJndqi8A$> >>>>> >>>>> >>>>> >>>>> *Disclaimer:* The information provided is correct to the best of my >>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>> expert opinions (Werner >>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$>Von >>>>> Braun >>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$> >>>>> )". >>>>> >>>>> >>>>> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi >>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>> >>>>>> Thanks Bhuwan and rest of the databricks team for the reviews, >>>>>> >>>>>> I appreciate your reviews, was very helpful in evaluating a few >>>>>> options that were overlooked earlier (especially about mixed spark apps >>>>>> running on notebooks). Regarding the use-cases, It could handle multiple >>>>>> streaming queries provided that they are run on the same trigger interval >>>>>> processing time (very similar to how current batch dra is set up)..but I >>>>>> felt like it would be beneficial if we separate out streaming queries >>>>>> when >>>>>> setting up production pipelines. >>>>>> >>>>>> Regarding the implementation, here is the draft PR >>>>>> https://github.com/apache/spark/pull/42352 >>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UC8iQomlg$>. >>>>>> (already mentioned in ticket SPARK-24815 >>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>>>>> ) >>>>>> >>>>>> I have built it on top of the current Dynamic resource allocation >>>>>> (DRA) algorithm >>>>>> <https://urldefense.com/v3/__https://spark.apache.org/docs/latest/job-scheduling.html*dynamic-resource-allocation__;Iw!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBK8f68fQ$> >>>>>> . >>>>>> While current DRA is catered towards batch jobs. This implementation >>>>>> just makes few changes to that algorithm to >>>>>> - do gradual scale-back. The remove-policy still applies (uses 2 old >>>>>> configs we currently have), but we now remove few executors per round of >>>>>> evaluation ( I have added 2 configs to tune that) >>>>>> - The scale-out process also still uses the same request policy (same >>>>>> uses 2 old configs we currently have). >>>>>> - while we are using the old configs in the both scale-out/back, the >>>>>> difference is that we are now giving configs to them based on the trigger >>>>>> interval as our north star. >>>>>> >>>>>> This implementation is just changes in 2 files to make it work. I >>>>>> have made the changes minimal/limited to just the core module of the >>>>>> spark >>>>>> repo. >>>>>> 1) to make sure it is applied on primitives of task, stage, job which >>>>>> the current dra is already doing. (This will enable us to think about >>>>>> other >>>>>> cases like default and continuous mode can still work provided we have a >>>>>> target processing time range we want to achieve) >>>>>> 2) We are reusing ExecutorAllocationClient, ExecutorMonitor and >>>>>> listeners which are already well tested and working well for batch job >>>>>> use >>>>>> case. >>>>>> >>>>>> We internally (in the company) have also added helpers so that we >>>>>> have less configs to tune. I can contribute that as well, if it makes the >>>>>> dev experience better. >>>>>> >>>>>> Feel free to review the PR, when we decide the direction is alright I >>>>>> will start adding the tests as well. >>>>>> >>>>>> On a side note. Maybe we should consider some future work to have >>>>>> DRA algo per query (batch, streaming queries, mixed etc) rather than per >>>>>> spark context. >>>>>> >>>>>> Thank you, >>>>>> >>>>>> Pavan >>>>>> >>>>>> >>>>>> On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni >>>>>> <bhuwan.sa...@databricks.com.invalid> wrote: >>>>>> >>>>>>> Hi Pavan, >>>>>>> >>>>>>> I am from the DataBricks Structured Streaming team, and we did a >>>>>>> review of the SPIP internally. Wanted to pass on the points discussed in >>>>>>> the meeting. >>>>>>> >>>>>>> Thanks for putting together the SPIP document. It's useful to have >>>>>>> dynamic resource allocation for Streaming queries, and it's exciting to >>>>>>> see >>>>>>> the cost saving numbers from your PoC. However, in general we discovered >>>>>>> that the SPIP addresses a very particular use-case (single streaming >>>>>>> query >>>>>>> in Spark cluster with Processing time Trigger). Keeping that in mind, >>>>>>> it's >>>>>>> useful to make sure that the implementation is simple. Can you please >>>>>>> share >>>>>>> your PoC implementation to understand the code complexity. This would >>>>>>> help >>>>>>> us to ensure that dynamic resource allocation mechanism for Streaming >>>>>>> queries does not become complicated in the future (if more resource >>>>>>> allocation mechanisms are added to address other use-cases). >>>>>>> >>>>>>> Thanks, >>>>>>> Bhuwan >>>>>>> >>>>>>> On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh < >>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Pavan and those who kindly voted for this SPIP >>>>>>>> >>>>>>>> Great to have 6+ votes and no -1 and 0. The so-called mass volume >>>>>>>> is there. The rest is admin matter and how to drive the project >>>>>>>> forward and >>>>>>>> yes there is more than one way of skinning the cat. I think we need >>>>>>>> some >>>>>>>> flexibility in the rules given the dwindling (IMO) number of comitters >>>>>>>> who >>>>>>>> are willing or actively participating. For example, on a similar >>>>>>>> matter I >>>>>>>> approached Codi Koeninger who was one of the founders of Spark >>>>>>>> Streaming, >>>>>>>> to shepherd a project almost a year back. Sadly he is no longer active >>>>>>>> and >>>>>>>> quotes "I haven't been involved lately and would be missing a lot >>>>>>>> of context." So we need to improvise and see how best we can drive >>>>>>>> this and similar ones. We wait a short while for a response otherwise >>>>>>>> I am >>>>>>>> happy to give a hand if needed and work with you guys to drive this. >>>>>>>> It is >>>>>>>> something worthwhile. >>>>>>>> >>>>>>>> HTH >>>>>>>> >>>>>>>> T >>>>>>>> Mich Talebzadeh, >>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>> London >>>>>>>> United Kingdom >>>>>>>> >>>>>>>> >>>>>>>> view my Linkedin profile >>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfbGToAE$> >>>>>>>> >>>>>>>> >>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWTA0_mlE$> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>> note >>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>> expert opinions (Werner >>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$>Von >>>>>>>> Braun >>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$> >>>>>>>> )". >>>>>>>> >>>>>>>> >>>>>>>> On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi >>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>> >>>>>>>>> Thanks for the pointers Mich, will wait for Jungtaek Lee or any >>>>>>>>> other PMC members to respond. >>>>>>>>> >>>>>>>>> aggregating upvotes to this email thread >>>>>>>>> >>>>>>>>> +6 >>>>>>>>> Mich Talebzadeh >>>>>>>>> Adam Hobbs >>>>>>>>> Pavan Kotikalapudi >>>>>>>>> Krystal Mitchell >>>>>>>>> Sona Torosyan >>>>>>>>> Aaron Kern >>>>>>>>> >>>>>>>>> Thank you, >>>>>>>>> >>>>>>>>> Pavan >>>>>>>>> >>>>>>>>> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh < >>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> please check this doc >>>>>>>>>> >>>>>>>>>> Spark Project Improvement Proposals (SPIP) | Apache Spark >>>>>>>>>> <https://urldefense.com/v3/__https://spark.apache.org/improvement-proposals.html__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5qy0rbHA$> >>>>>>>>>> >>>>>>>>>> and specifically the below extract >>>>>>>>>> >>>>>>>>>> Discussing an SPIP >>>>>>>>>> >>>>>>>>>> All discussion of an SPIP should take place in a public forum, >>>>>>>>>> preferably the discussion attached to the Jira. Any discussions that >>>>>>>>>> happen >>>>>>>>>> offline should be made available online for the public via meeting >>>>>>>>>> notes >>>>>>>>>> summarizing the discussions.(done) >>>>>>>>>> >>>>>>>>>> During this discussion, one or more shepherds should be >>>>>>>>>> identified among PMC members. (outstanding) >>>>>>>>>> >>>>>>>>>> Once the discussion settles, the shepherd(s) should call for a >>>>>>>>>> vote on the SPIP moving forward on the dev@ list. The vote >>>>>>>>>> should be open for at least 72 hours and follows the typical Apache >>>>>>>>>> vote >>>>>>>>>> process and passes upon consensus (at least 3 +1 votes from PMC >>>>>>>>>> members and >>>>>>>>>> no -1 votes from PMC members). dev@ should be notified of the >>>>>>>>>> vote result. >>>>>>>>>> >>>>>>>>>> If there does not exist at least one PMC member that is committed >>>>>>>>>> to shepherding the change within a month, the SPIP is rejected. >>>>>>>>>> >>>>>>>>>> If a committer does not think a SPIP aligns with long-term >>>>>>>>>> project goals, or is not practical at the point of proposal, the >>>>>>>>>> committer >>>>>>>>>> should -1 the SPIP explicitly and give technical justifications. >>>>>>>>>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee >>>>>>>>>> can kindly help the process >>>>>>>>>> >>>>>>>>>> cheers >>>>>>>>>> >>>>>>>>>> Mich Talebzadeh, >>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>> London >>>>>>>>>> United Kingdom >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> view my Linkedin profile >>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL6nGmLi3g$> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5rLq6E3w$> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>>> note >>>>>>>>>> that, as with any advice, quote "one test result is worth >>>>>>>>>> one-thousand >>>>>>>>>> expert opinions (Werner >>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$>Von >>>>>>>>>> Braun >>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$> >>>>>>>>>> )". >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi >>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Mich, >>>>>>>>>>> >>>>>>>>>>> We have >>>>>>>>>>> >>>>>>>>>>> five +1s till now. >>>>>>>>>>> >>>>>>>>>>> Mich Talebzadeh >>>>>>>>>>> Adam Hobbs >>>>>>>>>>> Pavan Kotikalapudi >>>>>>>>>>> Krystal Mitchell >>>>>>>>>>> Sona Torosyan >>>>>>>>>>> (few more in github pr) >>>>>>>>>>> +0: None >>>>>>>>>>> >>>>>>>>>>> -1: None >>>>>>>>>>> >>>>>>>>>>> Does it pass the required condition as approved? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Not sure of that though, nothing about minimum required is >>>>>>>>>>> mentioned in the past emails. >>>>>>>>>>> >>>>>>>>>>> I would request spark PMC members or any others who have done >>>>>>>>>>> this in the past to understand the process better. >>>>>>>>>>> >>>>>>>>>>> Thank you, >>>>>>>>>>> >>>>>>>>>>> Pavan >>>>>>>>>>> >>>>>>>>>>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh < >>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Pavan, >>>>>>>>>>>> >>>>>>>>>>>> Do you have a list of votes for this feature by any >>>>>>>>>>>> chance? Does it pass the required condition as approved? >>>>>>>>>>>> >>>>>>>>>>>> HTH >>>>>>>>>>>> >>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>> London >>>>>>>>>>>> United Kingdom >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A1aYWcDAg$> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0gQVKWXw$> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>> essential to >>>>>>>>>>>> note that, as with any advice, quote "one test result is worth >>>>>>>>>>>> one-thousand expert opinions (Werner >>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$>Von >>>>>>>>>>>> Braun >>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$> >>>>>>>>>>>> )". >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi >>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yes. The PR was closed due to inactivity by github actions.. >>>>>>>>>>>>> >>>>>>>>>>>>> The msg >>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1865306284__;Iw!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A113artKQ$> >>>>>>>>>>>>> also >>>>>>>>>>>>> says >>>>>>>>>>>>> >>>>>>>>>>>>> > If you'd like to revive this PR, please reopen it and ask a >>>>>>>>>>>>> committer to remove the Stale tag! >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh < >>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I can see it was closed. Was it because of inactivity? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>>>> London >>>>>>>>>>>>>> United Kingdom >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lCFDAOXA$> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7kBRUgBOQ$> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the >>>>>>>>>>>>>> best of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>>>> essential >>>>>>>>>>>>>> to note that, as with any advice, quote "one test result is >>>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$>Von >>>>>>>>>>>>>> Braun >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> >>>>>>>>>>>>>> )". >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi >>>>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Spark PMC members, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think we have few upvotes for this effort here and more >>>>>>>>>>>>>>> people are showing interest (see PR comments >>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1955238640__;Iw!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>>>>>>>>> .) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is anyone interested in mentoring and reviewing this effort? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also can the repository admin/owner re-open the PR? ( I >>>>>>>>>>>>>>> guess people only with admin access to the repository can do >>>>>>>>>>>>>>> that). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Pavan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>>>>>>>>> <kryst...@adobe.com.invalid> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > adding some references to the thread. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > - Jira ticket - SPARK-24815 >>>>>>>>>>>>>>>> > <https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>>>>>>>>> > - Design Doc >>>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > - discussion thread >>>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>>>>>>>>> > - PR with initial implementation - >>>>>>>>>>>>>>>> > https://github.com/apache/spark/pull/42352 >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Please vote with: >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > [ ] +1: Accept the proposal and start with the >>>>>>>>>>>>>>>> development. >>>>>>>>>>>>>>>> > [ ] +0 >>>>>>>>>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Thank you, >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Pavan >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh < >>>>>>>>>>>>>>>> mi...@gmail.com> >>>>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > +1 for me (non binding) >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>>>>>>>> responsibility for any >>>>>>>>>>>>>>>> > > loss, damage or destruction of data or any other >>>>>>>>>>>>>>>> property which may arise >>>>>>>>>>>>>>>> > > from relying on this email's technical content is >>>>>>>>>>>>>>>> explicitly disclaimed. >>>>>>>>>>>>>>>> > > The author will in no case be liable for any monetary >>>>>>>>>>>>>>>> damages arising from >>>>>>>>>>>>>>>> > > such loss, damage or destruction. >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfJvj8wi$> >>>>>>> *Bhuwan Sahni* >>>>>>> Staff Software Engineer >>>>>>> >>>>>>> bhuwan.sa...@databricks.com >>>>>>> 500 108th Ave. NE >>>>>>> Bellevue, WA 98004 >>>>>>> USA >>>>>>> >>>>>> >>> >>> -- >>> >>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrARPj-LXk$> >>> *Bhuwan Sahni* >>> Staff Software Engineer >>> >>> bhuwan.sa...@databricks.com >>> 500 108th Ave. NE >>> Bellevue, WA 98004 >>> USA >>> >> >> >> -- >> >> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrARPj-LXk$> >> *Bhuwan Sahni* >> Staff Software Engineer >> >> bhuwan.sa...@databricks.com >> 500 108th Ave. NE >> Bellevue, WA 98004 >> USA >> >
