Hi Bjørn, In spark it is called dynamic resource allocation: https://spark.apache.org/docs/3.5.1/configuration.html#dynamic-allocation
You can't just use k8s autoscaler since you need the driver to manage the new worker nodes to know the new executors so they will get work from it.. Thanks, Nimrod בתאריך יום א׳, 24 בנוב׳ 2024, 23:32, מאת Bjørn Jørgensen < bjornjorgen...@gmail.com>: > hm... I'm more confused about what this is and what do you mean with > Dynamic resource allocation (DRA) on a closter? > Dynamic resource allocation is a new thing in K8s from version 1.31 > https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/ > > It is for hardware like GPU and so forth. > > Aren't the thing that you will have is called autoscaler like > https://kubernetes.io/docs/concepts/cluster-administration/cluster-autoscaling/ > > > søn. 24. nov. 2024 kl. 22:08 skrev Mich Talebzadeh < > mich.talebza...@gmail.com>: > >> Hi Adam, Pavan, >> >> I agree with your views regarding the prolonged status of this feature. >> *Dynamic >> resource allocation for structured streaming* is indeed a critical >> functionality that many users could benefit from, particularly for >> long-running jobs that need to scale efficiently with changing workloads. >> >> As you rightly pointed out, the potential cost savings and resource >> optimizations this could provide are significant, and it is disappointing >> to see this feature stalled. >> >> If there is a lack of clarity on specific blockers or next steps, it >> would be worth revisiting the discussion with the interested parties and >> contributors associated with the JIRA ticket. We could outline the open >> questions (if any) or clarify the necessary work for the implementation. I >> am happy to contribute where possible to move this forward, provided there >> is a clearer understanding of what remains to be done. This is certainly an >> area that could benefit from more focus. >> >> Looking forward to hearing more thoughts from others. >> >> Mich Talebzadeh, >> >> Architect | Data Science | Financial Crime | GDPR & Compliance Specialist >> PhD <https://en.wikipedia.org/wiki/Doctor_of_Philosophy> Imperial >> College London <https://en.wikipedia.org/wiki/Imperial_College_London> >> London, United Kingdom >> >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> >> *Disclaimer:* The information provided is correct to the best of my >> knowledge but of course cannot be guaranteed . It is essential to note >> that, as with any advice, quote "one test result is worth one-thousand >> expert opinions (Werner >> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >> >> >> On Thu, 21 Nov 2024 at 02:32, Adam Hobbs >> <adam.ho...@bendigoadelaide.com.au.invalid> wrote: >> >>> Hi, >>> >>> >>> >>> I was wanting to enquire about this again. This functionality is really >>> something that I would love to see implemented. It is sad to see it >>> stalled for months. >>> >>> >>> >>> Long running structured streaming jobs really are a use case that could >>> benefit greatly from the ability to scale based on load. The cost savings >>> to users of structured streaming can be significant. >>> >>> >>> >>> Regards, >>> >>> >>> >>> Adam Hobbs >>> >>> >>> >>> >>> C2 - Internal Use >>> From: Pavan Kotikalapudi <pkotikalap...@twilio.com.INVALID> >>> *Sent:* Thursday, 29 August 2024 5:54 AM >>> *To:* Mich Talebzadeh <mich.talebza...@gmail.com> >>> *Cc:* Jungtaek Lim <kabhwan.opensou...@gmail.com>; >>> bhuwan.sa...@databricks.com; jungtaek....@databricks.com; Spark dev >>> list <dev@spark.apache.org>; tgra...@nvidia.com; tgraves...@yahoo.com; >>> tgra...@apache.org >>> *Subject:* Re: Vote on Dynamic resource allocation for structured >>> streaming [SPARK-24815] >>> >>> >>> >>> CAUTION: This email originated from outside of the organisation. Do not >>> click links or open attachments unless you recognise the sender's full >>> email address and know the content is safe. >>> >>> >>> >>> HI Mich, >>> >>> >>> >>> > You had a list of voters if I recall correctly? >>> >>> We had about 5 voters in the mailing list.. and some more in github. >>> >>> >>> >>> Jungtaek, >>> >>> >>> >>> I understand your situation. I have requested one of the earlier >>> reviewers (Jerry) to take another look at it. If at all you need a >>> code-level walkthrough on DRA I can do that. >>> >>> >>> >>> Thanks, >>> >>> >>> >>> Pavan >>> >>> >>> >>> On Wed, Aug 28, 2024 at 12:19 PM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>> Ok, >>> >>> >>> >>> If you are saying that you have not got the bandwidth or your team , >>> then that is understandable and we will look for a workaround. >>> >>> >>> >>> HTH >>> >>> >>> >>> Mich Talebzadeh, >>> >>> >>> >>> Architect | Data Engineer | Data Science | Financial Crime >>> >>> PhD >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Doctor_of_Philosophy__;!!NCc8flgU!dDSdbobNHFw08-cNiw8DTt-Qzm7kJu9WvENu5ew5nKWeDLX4_3s4zV2kxNvNNGKLQLFuxqE86PsLk76D00QRug6x4CytJw$> >>> Imperial College London >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Imperial_College_London__;!!NCc8flgU!dDSdbobNHFw08-cNiw8DTt-Qzm7kJu9WvENu5ew5nKWeDLX4_3s4zV2kxNvNNGKLQLFuxqE86PsLk76D00QRug772pwsSg$> >>> >>> >>> London, United Kingdom >>> >>> >>> >>> [image: Image removed by sender.] view my Linkedin profile >>> <https://urldefense.com/v3/__https:/www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dDSdbobNHFw08-cNiw8DTt-Qzm7kJu9WvENu5ew5nKWeDLX4_3s4zV2kxNvNNGKLQLFuxqE86PsLk76D00QRug5iJNZuIA$> >>> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> <https://urldefense.com/v3/__https:/en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dDSdbobNHFw08-cNiw8DTt-Qzm7kJu9WvENu5ew5nKWeDLX4_3s4zV2kxNvNNGKLQLFuxqE86PsLk76D00QRug4wS2TSrA$> >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of course cannot be guaranteed . It is essential to note >>> that, as with any advice, quote "one test result is worth one-thousand >>> expert opinions (Werner >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dDSdbobNHFw08-cNiw8DTt-Qzm7kJu9WvENu5ew5nKWeDLX4_3s4zV2kxNvNNGKLQLFuxqE86PsLk76D00QRug4kJgdO4A$>Von >>> Braun >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dDSdbobNHFw08-cNiw8DTt-Qzm7kJu9WvENu5ew5nKWeDLX4_3s4zV2kxNvNNGKLQLFuxqE86PsLk76D00QRug4kJgdO4A$> >>> )". >>> >>> >>> >>> >>> >>> On Wed, 28 Aug 2024 at 14:33, Jungtaek Lim <kabhwan.opensou...@gmail.com> >>> wrote: >>> >>> Hey, sorry for the silence from these days. I'm sorry to say, I have >>> zero resources to learn DRA in depth and review the change to sign off. I'm >>> over-booked with my daily work and don't have any margin to put on a >>> volunteering task. (Worth noting that contribution is volunteering, even >>> for committers and PMC members.) >>> >>> >>> >>> I hoped folks being called out could help to make things move forward, >>> but they also seem to be fully booked. Furthermore, the essential >>> precondition of the feature to be adopted is, any committer around the SS >>> area should be willing to maintain the code. I can't add more load from the >>> present. >>> >>> >>> >>> That said, please consider this as a scalability issue rather than a >>> proposal being not accepted. I hope other committers having expertise on >>> DRA could chime in and ever willing to cover SS area, but I assume it's not >>> that easy in practice. For sure, the ideal direction is to have sufficient >>> contributors for the SS area so that I no longer need to struggle to handle >>> most of the loads for this area. It's not just about proposing changes and >>> reporting issues - these are definitely great contributions, but >>> unfortunately increasing my load. What I want to see for contribution is >>> someone proactively reviewing others' contributions in detail. >>> >>> >>> >>> If it doesn't work out, someone may want to build a learning material >>> for Spark DRA implementation in a couple hours (in code-level detail, not a >>> high level) and persuade me to volunteer for one day to catch up and >>> finally be able to review the code. >>> >>> >>> >>> >>> >>> On Tue, Jun 11, 2024 at 4:24 AM Pavan Kotikalapudi < >>> pkotikalap...@twilio.com> wrote: >>> >>> Bumping it up for some reviews/comments/guidance. >>> >>> >>> >>> >>> >>> Thank you, >>> >>> >>> >>> Pavan >>> >>> >>> >>> On Mon, Apr 29, 2024 at 1:44 AM Pavan Kotikalapudi < >>> pkotikalap...@twilio.com> wrote: >>> >>> Hi Jungtaek, >>> >>> >>> >>> I understand your concerns. Feel free to ask more folks who are familiar >>> with DRA and scheduler. Btw, Jerry Peng has started the review for the PR. >>> He has provided some good comments, I am refactoring the PR to cover all >>> cases. >>> >>> >>> >>> He also made an interesting comment >>> <https://urldefense.com/v3/__https:/github.com/apache/spark/pull/42352/files*r1566259719__;Iw!!NCc8flgU!dDSdbobNHFw08-cNiw8DTt-Qzm7kJu9WvENu5ew5nKWeDLX4_3s4zV2kxNvNNGKLQLFuxqE86PsLk76D00QRug5JT1IYsg$> >>> about >>> offering gradual scale down feature in batch queries as well. I agree with >>> him, and would like to have your opinion on this. also cc'ing: Thomas >>> graves (SME on current DRA) >>> >>> >>> >>> >That said, it'd be nice to make iterative changes. Let's go with the >>> first PR with very basic functionality and ensure we document very well..... >>> >>> >>> >>> Sure I will limit the PR with just the changes I proposed in the >>> document. >>> >>> >>> >>> >>> >>> I also wanted to provide some update on *use-case: DRA with multiple >>> queries in the same driver.* >>> >>> >>> >>> I started updating and testing the algorithm so that DRA can even work >>> for the above mentioned use-case. The changes I made ensure that when >>> multiple queries of varied trigger intervals are provided, DRA will scale >>> based one streaming query ( i.e we provide scale configs according one >>> critical streaming query) >>> >>> >>> >>> >>> >>> Till now the results are promising. >>> >>> >>> >>> Test 1: >>> >>> >>> >>> I started with two queries of different trigger intervals (10 mins, 5 >>> mins). Of which I consider the 10 mins query as the critical one. >>> >>> I set the scaling configs to scale according to the 10 mins trigger >>> interval. Here are the results >>> >>> >>> >>> The DRA scaled to make sure that query with 10 mins trigger interval >>> (purple metric) is not over 10 mins. As the peak traffic hours approached, >>> the number of executors (The dotted red metric is no of executors) scaled >>> to handle the traffic. But the second streaming query (with 5 mins >>> trigger interval) didn't exactly stay in the 5 mins query processing time >>> as we are only concerned with optimal resource utilization in regards to >>> the first query. >>> >>> >>> >>> >>> >>> Test 2: >>> >>> >>> >>> Two queries with (1 min, 10mins) trigger intervals. This time I >>> considered the first query of the 1 min trigger interval as the critical >>> one. >>> >>> I set the scaling configs to scale according to the 1 min trigger >>> interval. Here are the results >>> >>> >>> >>> >>> >>> This time the DRA scaled according to the smaller query and used more >>> executors to handle peak traffic. The second query with a 10 mins trigger >>> interval was processing much faster (~3 mins) as a side-effect of having >>> more resources. >>> >>> >>> >>> I will perform another test with 3 streaming queries (1 min, 5 min >>> (critical workload), 10 mins) and will update the proposal with the results. >>> >>> >>> >>> >>> >>> Till now I only covered a limited set of use cases. *I would love to >>> hear from you If there are some important patterns that I need to cover*. >>> I will start testing with them as well. >>> >>> >>> >>> >>> >>> Cheers, >>> >>> >>> >>> pavan >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Apr 11, 2024 at 6:50 PM Jungtaek Lim < >>> kabhwan.opensou...@gmail.com> wrote: >>> >>> I'm still having a hard time reviewing this. I have been handling a >>> bunch of context right now, and the change is non-trivial to review in >>> parallel. I see people were OK with the algorithm in high-level, but from a >>> code perspective it's uneasy to understand without knowledge of DRA. It >>> would take me some time. I'll see whether I can get help from other folks. >>> >>> >>> >>> That said, it'd be nice to make iterative changes. Let's go with the >>> first PR with very basic functionality and ensure we document very well so >>> that users would be able to use this. Also we have to document the >>> limitations with enough emphasis - likewise I said, having multiple queries >>> in the same driver (cluster) is a common case and we found the proposal >>> does not consider the case. Let's make a dedicated effort on the >>> documentation. >>> >>> >>> >>> On Sun, Apr 7, 2024 at 8:41 AM Pavan Kotikalapudi < >>> pkotikalap...@twilio.com> wrote: >>> >>> Hi Jungtaek, >>> >>> >>> >>> Status on current SPARK-24815 >>> <https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!daMCbKw19zjKIh8Qgq7xMwWEqydMwxQwGOau9DPTw8AGetrR4ArLCJNmT_J_9vcyjjrwg8LF-THQVw_e8gIHmoEOfGelrCs$> >>> : >>> >>> Thomas Graves is reviewing the draft PR >>> <https://urldefense.com/v3/__https:/github.com/apache/spark/pull/42352__;!!NCc8flgU!daMCbKw19zjKIh8Qgq7xMwWEqydMwxQwGOau9DPTw8AGetrR4ArLCJNmT_J_9vcyjjrwg8LF-THQVw_e8gIHmoEOj643mQ4$>. >>> I need to add documentation about the configs and usage details, I am >>> planning to do that this week. >>> >>> He did mention that it would be great if somebody with experience in >>> structured streaming would take a look at the algorithm. Will you be able >>> to review it? >>> >>> >>> >>> Another point I wanted to discuss is, as you might have already seen in >>> the design doc >>> <https://urldefense.com/v3/__https:/docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing__;!!NCc8flgU!daMCbKw19zjKIh8Qgq7xMwWEqydMwxQwGOau9DPTw8AGetrR4ArLCJNmT_J_9vcyjjrwg8LF-THQVw_e8gIHmoEOzVm0U8M$> >>> we >>> use traditional DRA configs >>> >>> spark.dynamicAllocation.enabled, >>> >>> spark.dynamicAllocation.schedulerBacklogTimeout, >>> >>> spark.dynamicAllocation.sustainedSchedulerBacklogTimeout, >>> >>> spark.dynamicAllocation.executorIdleTimeout, >>> >>> spark.dynamicAllocation.cachedExecutorIdleTimeout, >>> >>> >>> >>> and few new configs >>> >>> >>> >>> spark.dynamicAllocation.streaming.enabled, >>> >>> spark.dynamicAllocation.streaming.executorDeallocationRatio, >>> >>> spark.dynamicAllocation.streaming.executorDeallocationTimeout. >>> >>> >>> >>> to make the DRA work for structured streaming. >>> >>> >>> >>> While in the design doc I did mention that we have to calculate and >>> set scale out/back thresholds based on the trigger interval >>> <https://urldefense.com/v3/__https:/docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit*heading=h.15dsvnvbhu2d__;Iw!!NCc8flgU!daMCbKw19zjKIh8Qgq7xMwWEqydMwxQwGOau9DPTw8AGetrR4ArLCJNmT_J_9vcyjjrwg8LF-THQVw_e8gIHmoEOweNXdNs$>. >>> We (internally in the company) do have helper functions to auto-generate >>> the above configs based on trigger interval and the threshold configs (we >>> also got similar feedback in reviews >>> <https://urldefense.com/v3/__https:/docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?disco=AAABIH1y18w__;!!NCc8flgU!daMCbKw19zjKIh8Qgq7xMwWEqydMwxQwGOau9DPTw8AGetrR4ArLCJNmT_J_9vcyjjrwg8LF-THQVw_e8gIHmoEOzliZkHM$> >>> ). >>> >>> Here are such configs >>> >>> >>> >>> # required - should be greater than 3 seconds as that gives enough >>> seconds for scaleOut and scaleBack thresholds to work with. >>> "spark.sql.streaming.triggerInterval.seconds": <Int> >>> # optional - value should be between 0 and 1 and greater than >>> scaleBackThreshold : default is 0.9 >>> "spark.dynamicAllocation.streaming.scaleOutThreshold": <Double> >>> # optional - value should be between 0 and 1 and less than >>> scaleOutThreshold : default is 0.5 >>> "spark.dynamicAllocation.streaming.scaleBackThreshold": <Double> >>> >>> >>> >>> The above configs helps us to generate the below configs for app with >>> different trigger intervals ( or if they change them for some reason) >>> >>> >>> >>> spark.dynamicAllocation.schedulerBacklogTimeout, >>> >>> spark.dynamicAllocation.sustainedSchedulerBacklogTimeout, >>> >>> spark.dynamicAllocation.executorIdleTimeout, >>> >>> spark.dynamicAllocation.cachedExecutorIdleTimeout. >>> >>> >>> >>> While our additional configs have its own limitations. I would like to >>> get some feedback if adding such kinds of configs to automate >>> >>> the process of calculating the thresholds and their respective configs >>> makes sense? >>> >>> >>> >>> Thank you, >>> >>> >>> >>> Pavan >>> >>> >>> >>> On Thu, Mar 28, 2024 at 3:38 PM Pavan Kotikalapudi < >>> pkotikalap...@twilio.com> wrote: >>> >>> Hi Jungtaek, >>> >>> >>> >>> Sorry for the late reply. >>> >>> >>> >>> I understand the concerns towards finding PMC members, I had similar >>> concerns in the past. Do you think we have something to improve in the SPIP >>> (certain areas) so that it would get traction from PMC members? Or this >>> SPIP might not be a priority to the PMC right now? >>> >>> >>> >>> I agree this change is small enough that it might not be tagged as an >>> SPIP. I started with the template SPIP questions so that it would be easier >>> to understand the limitations of the current system, new solution, how it >>> works, how to use it, limitations etc....As you might have already >>> noticed in the PR, This change is turned off by default, will only work if >>> `spark.dynamicAllocation.streaming.enabled` is true. >>> >>> >>> >>> Regarding the concerns about expertise in DRA, I will find some core >>> contributors of this module/DRA and tag them to this email with details, >>> Mich has also highlighted the same in the past. Once we get approval from >>> them we can further discuss and enhance this to make the user experience >>> better. >>> >>> >>> >>> Thank you, >>> >>> >>> >>> Pavan >>> >>> >>> >>> >>> >>> On Tue, Mar 26, 2024 at 8:12 PM Jungtaek Lim < >>> kabhwan.opensou...@gmail.com> wrote: >>> >>> Sounds good. >>> >>> >>> >>> One thing I'd like to clarify before shepherding this SPIP is >>> the process itself. Getting enough traction from PMC members is another >>> issue to pass the SPIP vote. Even a vote from committer is not counted. (I >>> don't have a binding vote.) I only see one PMC member (Thomas Graves, not >>> my team) in the design doc and we still don't get positive feedback. So >>> still a long way to go. We need three supporters from PMC members. >>> >>> >>> >>> Another thing is, I get the proposal at a high level, but I don't have >>> actual expertise in DRA. I could review the code in general, but I feel >>> like I'm not qualified to approve the code. We still need an expert on the >>> CORE area, especially who has expertise with DRA. (Could you please >>> annotate the code and enumerate several people who worked on the codebase?) >>> If they need an expertise of streaming to understand how things will work >>> then either you or I can explain, but I can't just approve and merge the >>> code. >>> >>> >>> >>> That said, if we succeed in finding one and they review the code and >>> LGTM, I'd rather say not to go with taking the process of SPIP unless the >>> expert reviewing your code requires us to do so. The change you proposed is >>> rather small and does not seem to be invasive (experts can also weigh), and >>> there must never be the case that this feature is turned on by default (as >>> we pointed out limitation). It doesn't look like requiring SPIP, if we >>> carefully document the new change and also clearly describe the limitation. >>> (Also a warning in the codebase that this must not be enabled by default.) >>> >>> >>> >>> >>> >>> On Tue, Mar 26, 2024 at 7:02 PM Pavan Kotikalapudi < >>> pkotikalap...@twilio.com> wrote: >>> >>> Hi Bhuwan, >>> >>> >>> >>> Glad to hear back from you! Very much appreciate your help on reviewing >>> the design doc/PR and endorsing this proposal. >>> >>> >>> >>> Thank you so much @Jungtaek Lim <kabhwan.opensou...@gmail.com> , @Mich >>> Talebzadeh <mich.talebza...@gmail.com> for graciously agreeing to >>> mentor/shepherd this effort. >>> >>> >>> >>> Regarding Twilio copyright in Notice binary file: >>> >>> Twilio Opensource counsel was involved all through the process, I have >>> placed it in the project file prior to Twilio signing a CCLA for the spark >>> project contribution( Aug '23). >>> >>> >>> >>> Since the CCLA is signed now, I have removed the twilio copyright from >>> that file. I didn't get a chance to update the PR after github-actions >>> closed it. >>> >>> >>> >>> Please let me know of next steps needed to bring this draft PR/effort to >>> completion. >>> >>> >>> >>> Thank you, >>> >>> >>> >>> Pavan >>> >>> >>> >>> >>> >>> On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim < >>> kabhwan.opensou...@gmail.com> wrote: >>> >>> I'm happy to, but it looks like I need to check one more thing about the >>> license, according to the WIP PR >>> <https://urldefense.com/v3/__https:/github.com/apache/spark/pull/42352__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrA6JBQ8S0$>. >>> >>> >>> >>> >>> @Pavan Kotikalapudi <pkotikalap...@twilio.com> >>> >>> I see you've added the copyright of Twilio in the NOTICE-binary file, >>> which makes me wonder if Twilio had filed CCLA to the Apache Software >>> Foundation. >>> >>> >>> >>> PMC members can correct me if I'm mistaken, but from my understanding >>> (and experiences of PMC member in other ASF project), code contribution is >>> considered as code donation and copyright belongs to ASF. That's why you >>> can't find the copyright of employers for contributors in the codebase. >>> What you see copyrights in NOTICE-binary is due to the fact we have binary >>> dependency and their licenses may require to explicitly mention about >>> copyright. It's not about direct code contribution. >>> >>> >>> >>> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior, >>> could you please engage with a relevant group in the company (could be a >>> legal team, or similar with OSS advocate team if there is any) and ensure >>> that CCLA is filed? The copyright issue is a legal issue, so we have to be >>> conservative and 100% sure that the employer is aware of what is the >>> meaning of donating the code to ASF via reviewing CCLA and relevant doc, >>> and explicitly express that they are OK with it via filing CCLA. >>> >>> >>> >>> You can read the description of agreements on contribution and ICLA/CCLA >>> form from this page. >>> >>> https://www.apache.org/licenses/contributor-agreements.html >>> <https://urldefense.com/v3/__https:/www.apache.org/licenses/contributor-agreements.html__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAktmm6BY$> >>> >>> >>> >>> Please let me know if this is resolved. This seems to me as a blocker to >>> move on. Please also let me know if the contribution is withdrawn from the >>> employer. >>> >>> >>> >>> Thanks, >>> >>> Jungtaek Lim (HeartSaVioR) >>> >>> >>> >>> >>> >>> On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni >>> <bhuwan.sa...@databricks.com.invalid> wrote: >>> >>> Hi Pavan, >>> >>> >>> >>> I looked at the PR, and the changes look simple and contained. It would >>> be useful to add dynamic resource allocation to Spark Structured Streaming. >>> >>> >>> >>> Jungtaek. Would you be able to shepherd this change? >>> >>> >>> >>> >>> >>> On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni < >>> bhuwan.sa...@databricks.com> wrote: >>> >>> Thanks a lot for creating the risk table Pavan. My apologies. I was tied >>> up with high priority items for the last couple weeks and could not >>> respond. I will review the PR by tomorrow's end, and get back to you. >>> >>> >>> >>> Appreciate your patience. >>> >>> >>> >>> Thanks >>> >>> Bhuwan Sahni >>> >>> >>> >>> On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi < >>> pkotikalap...@twilio.com> wrote: >>> >>> Hi Bhuwan, >>> >>> >>> >>> I hope the team got a chance to review the draft PR, looking for some >>> comments to see if the plan looks alright?. I have updated the document >>> about the risks >>> <https://urldefense.com/v3/__https:/docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit*heading=h.577aawlyiedf__;Iw!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAzuRa_bM$>.(also >>> mentioned below). Please confirm if it looks alright? >>> >>> >>> >>> *Spark application type* >>> >>> *auto-scaling capability* >>> >>> *with New auto-scaling capability* >>> >>> Spark Batch job >>> >>> Works with current DRA >>> >>> No - change >>> >>> Streaming query without trigger interval >>> >>> No implementation >>> >>> Can work with this implementation - (have to set certain scale back >>> configs based on previous usage pattern) - maybe automate with future work? >>> >>> Spark Streaming query with Trigger interval >>> >>> No implementation >>> >>> With this implementation >>> >>> Spark Streaming query with one-time micro batch >>> >>> Works with current DRA >>> >>> No - change >>> >>> Spark Streaming query with >>> >>> Availablenow micro batch >>> >>> Works with current DRA >>> >>> No - change >>> >>> Batch + Streaming query ( >>> >>> default/ >>> >>> triggger-interval/ >>> >>> once/ >>> >>> availablenow modes), other notebook use cases. >>> >>> No implementation >>> >>> No implementation >>> >>> >>> >>> >>> >>> >>> >>> We are more than happy to collaborate on a call to make better progress >>> on this enhancement. Please let us know. >>> >>> >>> >>> Thank you, >>> >>> >>> >>> Pavan >>> >>> >>> >>> On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>> >>> >>> Hi Bhuwan et al, >>> >>> Thank you for passing on the DataBricks Structured Streaming team's >>> review of the SPIP document. FYI, I work closely with Pawan and other >>> members to help deliver this piece of work. We appreciate your insights, >>> especially regarding the cost savings potential from the PoC. >>> >>> Pavan already furnished you with some additional info. Your team's point >>> about the SPIP currently addressing a specific use case (single streaming >>> query with Processing Time trigger) is well-taken. We agree that >>> maintaining simplicity is key, particularly as we explore more general >>> resource allocation mechanisms in the future. To address the concerns and >>> foster open discussion, The DataBricks team are invited to directly add >>> their comments and suggestions to the Jira itself >>> >>> [SPARK-24815] Structured Streaming should support dynamic allocation - >>> ASF JIRA (apache.org) >>> <https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>> >>> This will ensure everyone involved can benefit from your team's >>> expertise and facilitate further collaboration. >>> >>> >>> >>> Thanks >>> >>> >>> Mich Talebzadeh, >>> >>> Dad | Technologist | Solutions Architect | Engineer >>> >>> London >>> >>> United Kingdom >>> >>> >>> >>> [image: Image removed by sender.] view my Linkedin profile >>> <https://urldefense.com/v3/__https:/www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCNE366aQ$> >>> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> <https://urldefense.com/v3/__https:/en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UCJndqi8A$> >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of course cannot be guaranteed . It is essential to note >>> that, as with any advice, quote "one test result is worth one-thousand >>> expert opinions (Werner >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$>Von >>> Braun >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UDxzB-u4g$> >>> )". >>> >>> >>> >>> >>> >>> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi >>> <pkotikalap...@twilio.com.invalid> wrote: >>> >>> Thanks Bhuwan and rest of the databricks team for the reviews, >>> >>> >>> >>> I appreciate your reviews, was very helpful in evaluating a few options >>> that were overlooked earlier (especially about mixed spark apps running on >>> notebooks). Regarding the use-cases, It could handle multiple streaming >>> queries provided that they are run on the same trigger interval processing >>> time (very similar to how current batch dra is set up)..but I felt like it >>> would be beneficial if we separate out streaming queries when setting up >>> production pipelines. >>> >>> >>> >>> Regarding the implementation, here is the draft PR >>> https://github.com/apache/spark/pull/42352 >>> <https://urldefense.com/v3/__https:/github.com/apache/spark/pull/42352__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UC8iQomlg$>. >>> (already mentioned in ticket SPARK-24815 >>> <https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/SPARK-24815__;!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBUjIj4Iw$> >>> ) >>> >>> >>> >>> I have built it on top of the current Dynamic resource allocation (DRA) >>> algorithm >>> <https://urldefense.com/v3/__https:/spark.apache.org/docs/latest/job-scheduling.html*dynamic-resource-allocation__;Iw!!NCc8flgU!ZBV18VoUoRaD0b9X-yFgk39nnRoGZbGmeye3it4vXjffFIYZXF72EIjYL38AN1F-vPRwKCPGD4-gfiDnr8AS4UBK8f68fQ$> >>> . >>> >>> While current DRA is catered towards batch jobs. This implementation >>> just makes few changes to that algorithm to >>> >>> - do gradual scale-back. The remove-policy still applies (uses 2 old >>> configs we currently have), but we now remove few executors per round of >>> evaluation ( I have added 2 configs to tune that) >>> >>> - The scale-out process also still uses the same request policy (same >>> uses 2 old configs we currently have). >>> >>> - while we are using the old configs in the both scale-out/back, the >>> difference is that we are now giving configs to them based on the trigger >>> interval as our north star. >>> >>> >>> >>> This implementation is just changes in 2 files to make it work. I have >>> made the changes minimal/limited to just the core module of the spark repo. >>> >>> 1) to make sure it is applied on primitives of task, stage, job which >>> the current dra is already doing. (This will enable us to think about other >>> cases like default and continuous mode can still work provided we have a >>> target processing time range we want to achieve) >>> >>> 2) We are reusing ExecutorAllocationClient, ExecutorMonitor and >>> listeners which are already well tested and working well for batch job use >>> case. >>> >>> >>> >>> We internally (in the company) have also added helpers so that we have >>> less configs to tune. I can contribute that as well, if it makes the dev >>> experience better. >>> >>> >>> >>> Feel free to review the PR, when we decide the direction is alright I >>> will start adding the tests as well. >>> >>> >>> >>> On a side note. Maybe we should consider some future work to have DRA >>> algo per query (batch, streaming queries, mixed etc) rather than per spark >>> context. >>> >>> >>> >>> Thank you, >>> >>> >>> >>> Pavan >>> >>> >>> >>> >>> >>> On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni >>> <bhuwan.sa...@databricks.com.invalid> wrote: >>> >>> Hi Pavan, >>> >>> >>> >>> I am from the DataBricks Structured Streaming team, and we did a review >>> of the SPIP internally. Wanted to pass on the points discussed in the >>> meeting. >>> >>> Thanks for putting together the SPIP document. It's useful to have >>> dynamic resource allocation for Streaming queries, and it's exciting to see >>> the cost saving numbers from your PoC. However, in general we discovered >>> that the SPIP addresses a very particular use-case (single streaming query >>> in Spark cluster with Processing time Trigger). Keeping that in mind, it's >>> useful to make sure that the implementation is simple. Can you please share >>> your PoC implementation to understand the code complexity. This would help >>> us to ensure that dynamic resource allocation mechanism for Streaming >>> queries does not become complicated in the future (if more resource >>> allocation mechanisms are added to address other use-cases). >>> >>> >>> >>> Thanks, >>> >>> Bhuwan >>> >>> >>> >>> On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>> Hi Pavan and those who kindly voted for this SPIP >>> >>> >>> >>> Great to have 6+ votes and no -1 and 0. The so-called mass volume is >>> there. The rest is admin matter and how to drive the project forward and >>> yes there is more than one way of skinning the cat. I think we need some >>> flexibility in the rules given the dwindling (IMO) number of comitters who >>> are willing or actively participating. For example, on a similar matter I >>> approached Codi Koeninger who was one of the founders of Spark Streaming, >>> to shepherd a project almost a year back. Sadly he is no longer active and >>> quotes "I haven't been involved lately and would be missing a lot of >>> context." So we need to improvise and see how best we can drive this >>> and similar ones. We wait a short while for a response otherwise I am happy >>> to give a hand if needed and work with you guys to drive this. It is >>> something worthwhile. >>> >>> >>> >>> HTH >>> >>> T >>> >>> Mich Talebzadeh, >>> >>> Dad | Technologist | Solutions Architect | Engineer >>> >>> London >>> >>> United Kingdom >>> >>> >>> >>> [image: Image removed by sender.] view my Linkedin profile >>> <https://urldefense.com/v3/__https:/www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWfbGToAE$> >>> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> <https://urldefense.com/v3/__https:/en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWTA0_mlE$> >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of course cannot be guaranteed . It is essential to note >>> that, as with any advice, quote "one test result is worth one-thousand >>> expert opinions (Werner >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$>Von >>> Braun >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!aSaWrvwsxmouPhWml3DfaL6LSwSmsaX4XQP34pD4nXINAKXtLWeYqtNIUjJnqKdot44IaAexEVjBpcnuKih5d6ZKLWRYWSBVjq6O$> >>> )". >>> >>> >>> >>> >>> >>> On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi >>> <pkotikalap...@twilio.com.invalid> wrote: >>> >>> Thanks for the pointers Mich, will wait for Jungtaek Lee or any other >>> PMC members to respond. >>> >>> >>> >>> aggregating upvotes to this email thread >>> >>> >>> >>> +6 >>> >>> Mich Talebzadeh >>> >>> Adam Hobbs >>> >>> Pavan Kotikalapudi >>> >>> Krystal Mitchell >>> >>> Sona Torosyan >>> >>> Aaron Kern >>> >>> >>> >>> Thank you, >>> >>> >>> >>> Pavan >>> >>> >>> >>> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>> Hi, >>> >>> >>> >>> please check this doc >>> >>> >>> Spark Project Improvement Proposals (SPIP) | Apache Spark >>> <https://urldefense.com/v3/__https:/spark.apache.org/improvement-proposals.html__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5qy0rbHA$> >>> >>> and specifically the below extract >>> >>> >>> Discussing an SPIP >>> >>> All discussion of an SPIP should take place in a public forum, >>> preferably the discussion attached to the Jira. Any discussions that happen >>> offline should be made available online for the public via meeting notes >>> summarizing the discussions.(done) >>> >>> *During this discussion, one or more shepherds should be identified >>> among PMC members. *(outstanding) >>> >>> Once the discussion settles, the shepherd(s) should call for a vote on >>> the SPIP moving forward on the dev@ list. The vote should be open for >>> at least 72 hours and follows the typical Apache vote process and passes >>> upon consensus (at least 3 +1 votes from PMC members and no -1 votes from >>> PMC members). dev@ should be notified of the vote result. >>> >>> If there does not exist at least one PMC member that is committed to >>> shepherding the change within a month, the SPIP is rejected. >>> >>> If a committer does not think a SPIP aligns with long-term project >>> goals, or is not practical at the point of proposal, the committer should >>> -1 the SPIP explicitly and give technical justifications. >>> >>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee can >>> kindly help the process >>> >>> >>> >>> cheers >>> >>> >>> >>> Mich Talebzadeh, >>> >>> Dad | Technologist | Solutions Architect | Engineer >>> >>> London >>> >>> United Kingdom >>> >>> >>> >>> [image: Image removed by sender.] view my Linkedin profile >>> <https://urldefense.com/v3/__https:/www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL6nGmLi3g$> >>> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> <https://urldefense.com/v3/__https:/en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL5rLq6E3w$> >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of course cannot be guaranteed . It is essential to note >>> that, as with any advice, quote "one test result is worth one-thousand >>> expert opinions (Werner >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$>Von >>> Braun >>> <https://urldefense.com/v3/__https:/en.wikipedia.org/wiki/Wernher_von_Braun__;!!NCc8flgU!dJHLBpsBdsmdGt7dGsV2kyUhjpah0Z3g27vaxbmk2IA8gKdE4x_RgGK9V4wFOK7k2sZNMxzBz_9MHb9C5YHtjL4exCs1_Q$> >>> )". >>> >>> >>> >>> >>> >>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi >>> <pkotikalap...@twilio.com.invalid> wrote: >>> >>> Hi Mich, >>> >>> >>> >>> We have >>> >>> >>> >>> five +1s till now. >>> >>> >>> >>> Mich Talebzadeh >>> >>> Adam Hobbs >>> >>> Pavan Kotikalapudi >>> >>> Krystal Mitchell >>> >>> Sona Torosyan >>> >>> (few more in github pr) >>> >>> +0: None >>> >>> -1: None >>> >>> >>> >>> Does it pass the required condition as approved? >>> >>> >>> >>> Not sure of that though, nothing about minimum required is mentioned in >>> the past emails. >>> >>> >>> >>> I would request spark PMC members or any others who have done this in >>> the past to understand the process better. >>> >>> >>> >>> Thank you, >>> >>> >>> >>> Pavan >>> >>> >>> >>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>> Hi Pavan, >>> >>> >>> >>> Do you have a list of votes for this feature by any chance? Does it pass >>> the required condition as approved? >>> >>> >>> >>> HTH >>> >>> >>> Mich Talebzadeh, >>> >>> Dad | Technologist | Solutions Architect | Engineer >>> >>> London >>> >>> United Kingdom >>> >>> >>> >>> [image: Image removed by sender.] view my Linkedin profile >>> <https://urldefense.com/v3/__https:/www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A1aYWcDAg$> >>> >>> >>> >>> >>> <https://urldefense.com/v3/__https:/en.everybodywiki.com/Mich_Talebzadeh__;!!NCc8flgU!d1kZcsoBaeESUOMsb65wLw8dWRZEP3M2DyjVC4M4ie4NbCcMm9jETo-zSzhl3hcGLSFKRzsfReUfos7lbV5t0A0gQVKWXw$> >>> >>>
