I'm still having a hard time reviewing this. I have been handling a bunch of context right now, and the change is non-trivial to review in parallel. I see people were OK with the algorithm in high-level, but from a code perspective it's uneasy to understand without knowledge of DRA. It would take me some time. I'll see whether I can get help from other folks.
That said, it'd be nice to make iterative changes. Let's go with the first PR with very basic functionality and ensure we document very well so that users would be able to use this. Also we have to document the limitations with enough emphasis - likewise I said, having multiple queries in the same driver (cluster) is a common case and we found the proposal does not consider the case. Let's make a dedicated effort on the documentation. On Sun, Apr 7, 2024 at 8:41 AM Pavan Kotikalapudi <pkotikalap...@twilio.com> wrote: > Hi Jungtaek, > > Status on current SPARK-24815 > <https://issues.apache.org/jira/browse/SPARK-24815>: > Thomas Graves is reviewing the draft PR > <https://github.com/apache/spark/pull/42352>. I need to add documentation > about the configs and usage details, I am planning to do that this week. > He did mention that it would be great if somebody with experience in > structured streaming would take a look at the algorithm. Will you be able > to review it? > > Another point I wanted to discuss is, as you might have already seen in > the design doc > <https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> > we > use traditional DRA configs > spark.dynamicAllocation.enabled, > spark.dynamicAllocation.schedulerBacklogTimeout, > spark.dynamicAllocation.sustainedSchedulerBacklogTimeout, > spark.dynamicAllocation.executorIdleTimeout, > spark.dynamicAllocation.cachedExecutorIdleTimeout, > > and few new configs > > spark.dynamicAllocation.streaming.enabled, > spark.dynamicAllocation.streaming.executorDeallocationRatio, > spark.dynamicAllocation.streaming.executorDeallocationTimeout. > > to make the DRA work for structured streaming. > > While in the design doc I did mention that we have to calculate and set > scale out/back thresholds based on the trigger interval > <https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit#heading=h.15dsvnvbhu2d>. > We (internally in the company) do have helper functions to auto-generate > the above configs based on trigger interval and the threshold configs (we > also got similar feedback in reviews > <https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?disco=AAABIH1y18w> > ). > Here are such configs > > # required - should be greater than 3 seconds as that gives enough > seconds for scaleOut and scaleBack thresholds to work with. > "spark.sql.streaming.triggerInterval.seconds": <Int> > # optional - value should be between 0 and 1 and greater than > scaleBackThreshold : default is 0.9 > "spark.dynamicAllocation.streaming.scaleOutThreshold": <Double> > # optional - value should be between 0 and 1 and less than > scaleOutThreshold : default is 0.5 > "spark.dynamicAllocation.streaming.scaleBackThreshold": <Double> > > The above configs helps us to generate the below configs for app with > different trigger intervals ( or if they change them for some reason) > > spark.dynamicAllocation.schedulerBacklogTimeout, > spark.dynamicAllocation.sustainedSchedulerBacklogTimeout, > spark.dynamicAllocation.executorIdleTimeout, > spark.dynamicAllocation.cachedExecutorIdleTimeout. > > While our additional configs have its own limitations. I would like to get > some feedback if adding such kinds of configs to automate > the process of calculating the thresholds and their respective configs > makes sense? > > Thank you, > > Pavan > > On Thu, Mar 28, 2024 at 3:38 PM Pavan Kotikalapudi < > pkotikalap...@twilio.com> wrote: > >> Hi Jungtaek, >> >> Sorry for the late reply. >> >> I understand the concerns towards finding PMC members, I had similar >> concerns in the past. Do you think we have something to improve in the SPIP >> (certain areas) so that it would get traction from PMC members? Or this >> SPIP might not be a priority to the PMC right now? >> >> I agree this change is small enough that it might not be tagged as an >> SPIP. I started with the template SPIP questions so that it would be easier >> to understand the limitations of the current system, new solution, how it >> works, how to use it, limitations etc....As you might have already >> noticed in the PR, This change is turned off by default, will only work if >> `spark.dynamicAllocation.streaming.enabled` is true. >> >> Regarding the concerns about expertise in DRA, I will find some core >> contributors of this module/DRA and tag them to this email with details, >> Mich has also highlighted the same in the past. Once we get approval from >> them we can further discuss and enhance this to make the user experience >> better. >> >> Thank you, >> >> Pavan >> >> >> On Tue, Mar 26, 2024 at 8:12 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Sounds good. >>> >>> One thing I'd like to clarify before shepherding this SPIP is >>> the process itself. Getting enough traction from PMC members is another >>> issue to pass the SPIP vote. Even a vote from committer is not counted. (I >>> don't have a binding vote.) I only see one PMC member (Thomas Graves, not >>> my team) in the design doc and we still don't get positive feedback. So >>> still a long way to go. We need three supporters from PMC members. >>> >>> Another thing is, I get the proposal at a high level, but I don't have >>> actual expertise in DRA. I could review the code in general, but I feel >>> like I'm not qualified to approve the code. We still need an expert on the >>> CORE area, especially who has expertise with DRA. (Could you please >>> annotate the code and enumerate several people who worked on the codebase?) >>> If they need an expertise of streaming to understand how things will work >>> then either you or I can explain, but I can't just approve and merge the >>> code. >>> >>> That said, if we succeed in finding one and they review the code and >>> LGTM, I'd rather say not to go with taking the process of SPIP unless the >>> expert reviewing your code requires us to do so. The change you proposed is >>> rather small and does not seem to be invasive (experts can also weigh), and >>> there must never be the case that this feature is turned on by default (as >>> we pointed out limitation). It doesn't look like requiring SPIP, if we >>> carefully document the new change and also clearly describe the limitation. >>> (Also a warning in the codebase that this must not be enabled by default.) >>> >>> >>> On Tue, Mar 26, 2024 at 7:02 PM Pavan Kotikalapudi < >>> pkotikalap...@twilio.com> wrote: >>> >>>> Hi Bhuwan, >>>> >>>> Glad to hear back from you! Very much appreciate your help on reviewing >>>> the design doc/PR and endorsing this proposal. >>>> >>>> Thank you so much @Jungtaek Lim <kabhwan.opensou...@gmail.com> , @Mich >>>> Talebzadeh <mich.talebza...@gmail.com> for graciously agreeing to >>>> mentor/shepherd this effort. >>>> >>>> Regarding Twilio copyright in Notice binary file: >>>> Twilio Opensource counsel was involved all through the process, I have >>>> placed it in the project file prior to Twilio signing a CCLA for the spark >>>> project contribution( Aug '23). >>>> >>>> Since the CCLA is signed now, I have removed the twilio copyright from >>>> that file. I didn't get a chance to update the PR after github-actions >>>> closed it. >>>> >>>> Please let me know of next steps needed to bring this draft PR/effort >>>> to completion. >>>> >>>> Thank you, >>>> >>>> Pavan >>>> >>>> >>>> On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim < >>>> kabhwan.opensou...@gmail.com> wrote: >>>> >>>>> I'm happy to, but it looks like I need to check one more thing about >>>>> the license, according to the WIP PR >>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrA6JBQ8S0$> >>>>> . >>>>> >>>>> @Pavan Kotikalapudi <pkotikalap...@twilio.com> >>>>> I see you've added the copyright of Twilio in the NOTICE-binary file, >>>>> which makes me wonder if Twilio had filed CCLA to the Apache Software >>>>> Foundation. >>>>> >>>>> PMC members can correct me if I'm mistaken, but from my understanding >>>>> (and experiences of PMC member in other ASF project), code contribution is >>>>> considered as code donation and copyright belongs to ASF. That's why you >>>>> can't find the copyright of employers for contributors in the codebase. >>>>> What you see copyrights in NOTICE-binary is due to the fact we have binary >>>>> dependency and their licenses may require to explicitly mention about >>>>> copyright. It's not about direct code contribution. >>>>> >>>>> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior, >>>>> could you please engage with a relevant group in the company (could be a >>>>> legal team, or similar with OSS advocate team if there is any) and ensure >>>>> that CCLA is filed? The copyright issue is a legal issue, so we have to be >>>>> conservative and 100% sure that the employer is aware of what is the >>>>> meaning of donating the code to ASF via reviewing CCLA and relevant doc, >>>>> and explicitly express that they are OK with it via filing CCLA. >>>>> >>>>> You can read the description of agreements on contribution and >>>>> ICLA/CCLA form from this page. >>>>> https://www.apache.org/licenses/contributor-agreements.html >>>>> <https://urldefense.com/v3/__https://www.apache.org/licenses/contributor-agreements.html__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAktmm6BY$> >>>>> >>>>> Please let me know if this is resolved. This seems to me as a blocker >>>>> to move on. Please also let me know if the contribution is withdrawn from >>>>> the employer. >>>>> >>>>> Thanks, >>>>> Jungtaek Lim (HeartSaVioR) >>>>> >>>>> >>>>> On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni >>>>> <bhuwan.sa...@databricks.com.invalid> wrote: >>>>> >>>>>> Hi Pavan, >>>>>> >>>>>> I looked at the PR, and the changes look simple and contained. It >>>>>> would be useful to add dynamic resource allocation to Spark Structured >>>>>> Streaming. >>>>>> >>>>>> Jungtaek. Would you be able to shepherd this change? >>>>>> >>>>>> >>>>>> On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni < >>>>>> bhuwan.sa...@databricks.com> wrote: >>>>>> >>>>>>> Thanks a lot for creating the risk table Pavan. My apologies. I was >>>>>>> tied up with high priority items for the last couple weeks and could not >>>>>>> respond. I will review the PR by tomorrow's end, and get back to you. >>>>>>> >>>>>>> Appreciate your patience. >>>>>>> >>>>>>> Thanks >>>>>>> Bhuwan Sahni >>>>>>> >>>>>>> On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi < >>>>>>> pkotikalap...@twilio.com> wrote: >>>>>>> >>>>>>>> Hi Bhuwan, >>>>>>>> >>>>>>>> I hope the team got a chance to review the draft PR, looking for >>>>>>>> some comments to see if the plan looks alright?. I have updated the >>>>>>>> document about the risks >>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit*heading=h.577aawlyiedf__;Iw!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAzuRa_bM$>.(also >>>>>>>> mentioned below). Please confirm if it looks alright? >>>>>>>> >>>>>>>> *Spark application type* >>>>>>>> >>>>>>>> *auto-scaling capability* >>>>>>>> >>>>>>>> *with New auto-scaling capability* >>>>>>>> >>>>>>>> Spark Batch job >>>>>>>> >>>>>>>> Works with current DRA >>>>>>>> >>>>>>>> No - change >>>>>>>> >>>>>>>> Streaming query without trigger interval >>>>>>>> >>>>>>>> No implementation >>>>>>>> >>>>>>>> Can work with this implementation - (have to set certain scale back >>>>>>>> configs based on previous usage pattern) - maybe automate with future >>>>>>>> work? >>>>>>>> >>>>>>>> Spark Streaming query with Trigger interval >>>>>>>> >>>>>>>> No implementation >>>>>>>> >>>>>>>> With this implementation >>>>>>>> >>>>>>>> Spark Streaming query with one-time micro batch >>>>>>>> >>>>>>>> Works with current DRA >>>>>>>> >>>>>>>> No - change >>>>>>>> >>>>>>>> Spark Streaming query with >>>>>>>> >>>>>>>> Availablenow micro batch >>>>>>>> >>>>>>>> Works with current DRA >>>>>>>> >>>>>>>> No - change >>>>>>>> >>>>>>>> Batch + Streaming query ( >>>>>>>> >>>>>>>> default/ >>>>>>>> >>>>>>>> triggger-interval/ >>>>>>>> >>>>>>>> once/ >>>>>>>> >>>>>>>> availablenow modes), other notebook use cases. >>>>>>>> >>>>>>>> No implementation >>>>>>>> >>>>>>>> No implementation >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We are more than happy to collaborate on a call to make better >>>>>>>> progress on this enhancement. Please let us know. >>>>>>>> >>>>>>>> Thank you, >>>>>>>> >>>>>>>> Pavan >>>>>>>> >>>>>>>> On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> Hi Bhuwan et al, >>>>>>>>> >>>>>>>>> Thank you for passing on the DataBricks Structured Streaming >>>>>>>>> team's review of the SPIP document. FYI, I work closely with Pawan and >>>>>>>>> other members to help deliver this piece of work. We appreciate your >>>>>>>>> insights, especially regarding the cost savings potential from the >>>>>>>>> PoC. >>>>>>>>> >>>>>>>>> Pavan already furnished you with some additional info. Your team's >>>>>>>>> point about the SPIP currently addressing a specific use case (single >>>>>>>>> streaming query with Processing Time trigger) is well-taken. We agree >>>>>>>>> that >>>>>>>>> maintaining simplicity is key, particularly as we explore more general >>>>>>>>> resource allocation mechanisms in the future. To address the concerns >>>>>>>>> and >>>>>>>>> foster open discussion, The DataBricks team are invited to directly >>>>>>>>> add >>>>>>>>> their comments and suggestions to the Jira itself >>>>>>>>> >>>>>>>>> [SPARK-24815] Structured Streaming should support dynamic >>>>>>>>> allocation - ASF JIRA (apache.org) >>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>>>>>>>> This will ensure everyone involved can benefit from your team's >>>>>>>>> expertise and facilitate further collaboration. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Mich Talebzadeh, >>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>> London >>>>>>>>> United Kingdom >>>>>>>>> >>>>>>>>> >>>>>>>>> view my Linkedin profile >>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCNE366aQ$> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCJndqi8A$> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>> note >>>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>>> expert opinions (Werner >>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$>Von >>>>>>>>> Braun >>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$> >>>>>>>>> )". >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi >>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>> >>>>>>>>>> Thanks Bhuwan and rest of the databricks team for the reviews, >>>>>>>>>> >>>>>>>>>> I appreciate your reviews, was very helpful in evaluating a few >>>>>>>>>> options that were overlooked earlier (especially about mixed spark >>>>>>>>>> apps >>>>>>>>>> running on notebooks). Regarding the use-cases, It could handle >>>>>>>>>> multiple >>>>>>>>>> streaming queries provided that they are run on the same trigger >>>>>>>>>> interval >>>>>>>>>> processing time (very similar to how current batch dra is set >>>>>>>>>> up)..but I >>>>>>>>>> felt like it would be beneficial if we separate out streaming >>>>>>>>>> queries when >>>>>>>>>> setting up production pipelines. >>>>>>>>>> >>>>>>>>>> Regarding the implementation, here is the draft PR >>>>>>>>>> https://github.com/apache/spark/pull/42352 >>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UC8iQomlg$>. >>>>>>>>>> (already mentioned in ticket SPARK-24815 >>>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>>>>>>>>> ) >>>>>>>>>> >>>>>>>>>> I have built it on top of the current Dynamic resource >>>>>>>>>> allocation (DRA) algorithm >>>>>>>>>> <https://urldefense.com/v3/__https://spark.apache.org/docs/latest/job-scheduling.html*dynamic-resource-allocation__;Iw!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBK8f68fQ$> >>>>>>>>>> . >>>>>>>>>> While current DRA is catered towards batch jobs. This >>>>>>>>>> implementation just makes few changes to that algorithm to >>>>>>>>>> - do gradual scale-back. The remove-policy still applies (uses 2 >>>>>>>>>> old configs we currently have), but we now remove few executors per >>>>>>>>>> round >>>>>>>>>> of evaluation ( I have added 2 configs to tune that) >>>>>>>>>> - The scale-out process also still uses the same request policy >>>>>>>>>> (same uses 2 old configs we currently have). >>>>>>>>>> - while we are using the old configs in the both scale-out/back, >>>>>>>>>> the difference is that we are now giving configs to them based on the >>>>>>>>>> trigger interval as our north star. >>>>>>>>>> >>>>>>>>>> This implementation is just changes in 2 files to make it work. I >>>>>>>>>> have made the changes minimal/limited to just the core module of the >>>>>>>>>> spark >>>>>>>>>> repo. >>>>>>>>>> 1) to make sure it is applied on primitives of task, stage, job >>>>>>>>>> which the current dra is already doing. (This will enable us to >>>>>>>>>> think about >>>>>>>>>> other cases like default and continuous mode can still work >>>>>>>>>> provided we >>>>>>>>>> have a target processing time range we want to achieve) >>>>>>>>>> 2) We are reusing ExecutorAllocationClient, ExecutorMonitor and >>>>>>>>>> listeners which are already well tested and working well for batch >>>>>>>>>> job use >>>>>>>>>> case. >>>>>>>>>> >>>>>>>>>> We internally (in the company) have also added helpers so that we >>>>>>>>>> have less configs to tune. I can contribute that as well, if it >>>>>>>>>> makes the >>>>>>>>>> dev experience better. >>>>>>>>>> >>>>>>>>>> Feel free to review the PR, when we decide the direction is >>>>>>>>>> alright I will start adding the tests as well. >>>>>>>>>> >>>>>>>>>> On a side note. Maybe we should consider some future work to >>>>>>>>>> have DRA algo per query (batch, streaming queries, mixed etc) rather >>>>>>>>>> than >>>>>>>>>> per spark context. >>>>>>>>>> >>>>>>>>>> Thank you, >>>>>>>>>> >>>>>>>>>> Pavan >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni >>>>>>>>>> <bhuwan.sa...@databricks.com.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Pavan, >>>>>>>>>>> >>>>>>>>>>> I am from the DataBricks Structured Streaming team, and we did a >>>>>>>>>>> review of the SPIP internally. Wanted to pass on the points >>>>>>>>>>> discussed in >>>>>>>>>>> the meeting. >>>>>>>>>>> >>>>>>>>>>> Thanks for putting together the SPIP document. It's useful to >>>>>>>>>>> have dynamic resource allocation for Streaming queries, and it's >>>>>>>>>>> exciting >>>>>>>>>>> to see the cost saving numbers from your PoC. However, in general we >>>>>>>>>>> discovered that the SPIP addresses a very particular use-case >>>>>>>>>>> (single >>>>>>>>>>> streaming query in Spark cluster with Processing time Trigger). >>>>>>>>>>> Keeping >>>>>>>>>>> that in mind, it's useful to make sure that the implementation is >>>>>>>>>>> simple. >>>>>>>>>>> Can you please share your PoC implementation to understand the code >>>>>>>>>>> complexity. This would help us to ensure that dynamic resource >>>>>>>>>>> allocation >>>>>>>>>>> mechanism for Streaming queries does not become complicated in the >>>>>>>>>>> future >>>>>>>>>>> (if more resource allocation mechanisms are added to address other >>>>>>>>>>> use-cases). >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Bhuwan >>>>>>>>>>> >>>>>>>>>>> On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh < >>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Pavan and those who kindly voted for this SPIP >>>>>>>>>>>> >>>>>>>>>>>> Great to have 6+ votes and no -1 and 0. The so-called mass >>>>>>>>>>>> volume is there. The rest is admin matter and how to drive the >>>>>>>>>>>> project >>>>>>>>>>>> forward and yes there is more than one way of skinning the cat. I >>>>>>>>>>>> think we >>>>>>>>>>>> need some flexibility in the rules given the dwindling (IMO) >>>>>>>>>>>> number of >>>>>>>>>>>> comitters who are willing or actively participating. For example, >>>>>>>>>>>> on a >>>>>>>>>>>> similar matter I approached Codi Koeninger who was one of the >>>>>>>>>>>> founders of >>>>>>>>>>>> Spark Streaming, to shepherd a project almost a year back. Sadly >>>>>>>>>>>> he is no >>>>>>>>>>>> longer active and quotes "I haven't been involved lately and >>>>>>>>>>>> would be missing a lot of context." So we need to improvise >>>>>>>>>>>> and see how best we can drive this and similar ones. We wait a >>>>>>>>>>>> short while >>>>>>>>>>>> for a response otherwise I am happy to give a hand if needed and >>>>>>>>>>>> work with >>>>>>>>>>>> you guys to drive this. It is something worthwhile. >>>>>>>>>>>> >>>>>>>>>>>> HTH >>>>>>>>>>>> >>>>>>>>>>>> T >>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>> London >>>>>>>>>>>> United Kingdom >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfbGToAE$> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWTA0_mlE$> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>> essential to >>>>>>>>>>>> note that, as with any advice, quote "one test result is worth >>>>>>>>>>>> one-thousand expert opinions (Werner >>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$>Von >>>>>>>>>>>> Braun >>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$> >>>>>>>>>>>> )". >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi >>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks for the pointers Mich, will wait for Jungtaek Lee or >>>>>>>>>>>>> any other PMC members to respond. >>>>>>>>>>>>> >>>>>>>>>>>>> aggregating upvotes to this email thread >>>>>>>>>>>>> >>>>>>>>>>>>> +6 >>>>>>>>>>>>> Mich Talebzadeh >>>>>>>>>>>>> Adam Hobbs >>>>>>>>>>>>> Pavan Kotikalapudi >>>>>>>>>>>>> Krystal Mitchell >>>>>>>>>>>>> Sona Torosyan >>>>>>>>>>>>> Aaron Kern >>>>>>>>>>>>> >>>>>>>>>>>>> Thank you, >>>>>>>>>>>>> >>>>>>>>>>>>> Pavan >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh < >>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> please check this doc >>>>>>>>>>>>>> >>>>>>>>>>>>>> Spark Project Improvement Proposals (SPIP) | Apache Spark >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://spark.apache.org/improvement-proposals.html__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5qy0rbHA$> >>>>>>>>>>>>>> >>>>>>>>>>>>>> and specifically the below extract >>>>>>>>>>>>>> >>>>>>>>>>>>>> Discussing an SPIP >>>>>>>>>>>>>> >>>>>>>>>>>>>> All discussion of an SPIP should take place in a public >>>>>>>>>>>>>> forum, preferably the discussion attached to the Jira. Any >>>>>>>>>>>>>> discussions that >>>>>>>>>>>>>> happen offline should be made available online for the public >>>>>>>>>>>>>> via meeting >>>>>>>>>>>>>> notes summarizing the discussions.(done) >>>>>>>>>>>>>> >>>>>>>>>>>>>> During this discussion, one or more shepherds should be >>>>>>>>>>>>>> identified among PMC members. (outstanding) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Once the discussion settles, the shepherd(s) should call for >>>>>>>>>>>>>> a vote on the SPIP moving forward on the dev@ list. The vote >>>>>>>>>>>>>> should be open for at least 72 hours and follows the typical >>>>>>>>>>>>>> Apache vote >>>>>>>>>>>>>> process and passes upon consensus (at least 3 +1 votes from PMC >>>>>>>>>>>>>> members and >>>>>>>>>>>>>> no -1 votes from PMC members). dev@ should be notified of >>>>>>>>>>>>>> the vote result. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If there does not exist at least one PMC member that is >>>>>>>>>>>>>> committed to shepherding the change within a month, the SPIP is >>>>>>>>>>>>>> rejected. >>>>>>>>>>>>>> >>>>>>>>>>>>>> If a committer does not think a SPIP aligns with long-term >>>>>>>>>>>>>> project goals, or is not practical at the point of proposal, the >>>>>>>>>>>>>> committer >>>>>>>>>>>>>> should -1 the SPIP explicitly and give technical justifications. >>>>>>>>>>>>>> OK a shepherd from PMC members is required. Maybe Jungtaek >>>>>>>>>>>>>> Lee can kindly help the process >>>>>>>>>>>>>> >>>>>>>>>>>>>> cheers >>>>>>>>>>>>>> >>>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>>>> London >>>>>>>>>>>>>> United Kingdom >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL6nGmLi3g$> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5rLq6E3w$> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the >>>>>>>>>>>>>> best of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>>>>> essential >>>>>>>>>>>>>> to note that, as with any advice, quote "one test result is >>>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$>Von >>>>>>>>>>>>>> Braun >>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$> >>>>>>>>>>>>>> )". >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi >>>>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Mich, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We have >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> five +1s till now. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Mich Talebzadeh >>>>>>>>>>>>>>> Adam Hobbs >>>>>>>>>>>>>>> Pavan Kotikalapudi >>>>>>>>>>>>>>> Krystal Mitchell >>>>>>>>>>>>>>> Sona Torosyan >>>>>>>>>>>>>>> (few more in github pr) >>>>>>>>>>>>>>> +0: None >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -1: None >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Does it pass the required condition as approved? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Not sure of that though, nothing about minimum required is >>>>>>>>>>>>>>> mentioned in the past emails. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I would request spark PMC members or any others who have >>>>>>>>>>>>>>> done this in the past to understand the process better. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Pavan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh < >>>>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Pavan, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Do you have a list of votes for this feature by any >>>>>>>>>>>>>>>> chance? Does it pass the required condition as approved? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> HTH >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>>>>>> London >>>>>>>>>>>>>>>> United Kingdom >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A1aYWcDAg$> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0gQVKWXw$> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the >>>>>>>>>>>>>>>> best of my knowledge but of course cannot be guaranteed . It >>>>>>>>>>>>>>>> is essential >>>>>>>>>>>>>>>> to note that, as with any advice, quote "one test result is >>>>>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$>Von >>>>>>>>>>>>>>>> Braun >>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0P4WA5mw$> >>>>>>>>>>>>>>>> )". >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi >>>>>>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes. The PR was closed due to inactivity by github >>>>>>>>>>>>>>>>> actions.. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The msg >>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1865306284__;Iw!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A113artKQ$> >>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>> says >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> > If you'd like to revive this PR, please reopen it and >>>>>>>>>>>>>>>>> ask a committer to remove the Stale tag! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh < >>>>>>>>>>>>>>>>> mich.talebza...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I can see it was closed. Was it because of inactivity? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>>>>>>>>> Dad | Technologist | Solutions Architect | Engineer >>>>>>>>>>>>>>>>>> London >>>>>>>>>>>>>>>>>> United Kingdom >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> view my Linkedin profile >>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lCFDAOXA$> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7kBRUgBOQ$> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> *Disclaimer:* The information provided is correct to the >>>>>>>>>>>>>>>>>> best of my knowledge but of course cannot be guaranteed . It >>>>>>>>>>>>>>>>>> is essential >>>>>>>>>>>>>>>>>> to note that, as with any advice, quote "one test result is >>>>>>>>>>>>>>>>>> worth one-thousand expert opinions (Werner >>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$>Von >>>>>>>>>>>>>>>>>> Braun >>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7lSMcDbbg$> >>>>>>>>>>>>>>>>>> )". >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi >>>>>>>>>>>>>>>>>> <pkotikalap...@twilio.com.invalid> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Spark PMC members, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I think we have few upvotes for this effort here and >>>>>>>>>>>>>>>>>>> more people are showing interest (see PR comments >>>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352*issuecomment-1955238640__;Iw!!NCc8flgU!ay85y5IRZ-bv2v2dR8HP7lChTidWLK_bsLQVbOqng9bwhC30-WY-SKIUNTIJCJaVCLHGgHDJOCmJ11L9pU6yO7k0wc9hCg$> >>>>>>>>>>>>>>>>>>> .) >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Is anyone interested in mentoring and reviewing this >>>>>>>>>>>>>>>>>>> effort? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Also can the repository admin/owner re-open the PR? ( I >>>>>>>>>>>>>>>>>>> guess people only with admin access to the repository can >>>>>>>>>>>>>>>>>>> do that). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Pavan >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell >>>>>>>>>>>>>>>>>>> <kryst...@adobe.com.invalid> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> +1 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: >>>>>>>>>>>>>>>>>>>> > Thanks for proposing and voting for the feature Mich. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > adding some references to the thread. >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > - Jira ticket - SPARK-24815 >>>>>>>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/SPARK-24815> >>>>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r64f0rbw$> >>>>>>>>>>>>>>>>>>>> > - Design Doc >>>>>>>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>>>>>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing> >>>>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-r44a1rO8$> >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > - discussion thread >>>>>>>>>>>>>>>>>>>> > < >>>>>>>>>>>>>>>>>>>> https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo> >>>>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo*3E__;JQ!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rkLpTOYM$> >>>>>>>>>>>>>>>>>>>> > - PR with initial implementation - >>>>>>>>>>>>>>>>>>>> > https://github.com/apache/spark/pull/42352 >>>>>>>>>>>>>>>>>>>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!NCc8flgU!b8v0cnobIeWmrtrGvm7r3lY83cOCZBDfHYW8xGj1tzG-9XYCnzsQoebrCmyMCJBXU52BSm3phgntc1HXve-rZAZFOls$> >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Please vote with: >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > [ ] +1: Accept the proposal and start with the >>>>>>>>>>>>>>>>>>>> development. >>>>>>>>>>>>>>>>>>>> > [ ] +0 >>>>>>>>>>>>>>>>>>>> > [ ] -1: I don’t think this is a good idea because … >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Thank you, >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > Pavan >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh < >>>>>>>>>>>>>>>>>>>> mi...@gmail.com> >>>>>>>>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > +1 for me (non binding) >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > *Disclaimer:* Use it at your own risk. Any and all >>>>>>>>>>>>>>>>>>>> responsibility for any >>>>>>>>>>>>>>>>>>>> > > loss, damage or destruction of data or any other >>>>>>>>>>>>>>>>>>>> property which may arise >>>>>>>>>>>>>>>>>>>> > > from relying on this email's technical content is >>>>>>>>>>>>>>>>>>>> explicitly disclaimed. >>>>>>>>>>>>>>>>>>>> > > The author will in no case be liable for any >>>>>>>>>>>>>>>>>>>> monetary damages arising from >>>>>>>>>>>>>>>>>>>> > > such loss, damage or destruction. >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfJvj8wi$> >>>>>>>>>>> *Bhuwan Sahni* >>>>>>>>>>> Staff Software Engineer >>>>>>>>>>> >>>>>>>>>>> bhuwan.sa...@databricks.com >>>>>>>>>>> 500 108th Ave. NE >>>>>>>>>>> Bellevue, WA 98004 >>>>>>>>>>> USA >>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrARPj-LXk$> >>>>>>> *Bhuwan Sahni* >>>>>>> Staff Software Engineer >>>>>>> >>>>>>> bhuwan.sa...@databricks.com >>>>>>> 500 108th Ave. NE >>>>>>> Bellevue, WA 98004 >>>>>>> USA >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> <https://urldefense.com/v3/__http://www.databricks.com__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrARPj-LXk$> >>>>>> *Bhuwan Sahni* >>>>>> Staff Software Engineer >>>>>> >>>>>> bhuwan.sa...@databricks.com >>>>>> 500 108th Ave. NE >>>>>> Bellevue, WA 98004 >>>>>> USA >>>>>> >>>>>
