Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

uire to explicitly mention about >>>>> copyright. It's not about direct code contribution. >>>>> >>>>> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior, >>>>> could you please engage with a relevant group in the company

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

e that the employer is aware of what is the >>>> meaning of donating the code to ASF via reviewing CCLA and relevant doc, >>>> and explicitly express that they are OK with it via filing CCLA. >>>> >>>> You can read the description of agreements on c

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Andrew, Sandy, Jerry, Thomas, marcelo, Whenchen, YangJie, Shixiong, My apologies. I have tagged soo many of you (on multiple emails), I am in the process of finding the core contributors of the Dynamic resource allocation (DRA) feature in apache/spark <https://github.com/apache/spark>, I

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

es/contributor-agreements.html >>> <https://urldefense.com/v3/__https://www.apache.org/licenses/contributor-agreements.html__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAktmm6BY$> >>> >>> Please let me know if th

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

/contributor-agreements.html >> <https://urldefense.com/v3/__https://www.apache.org/licenses/contributor-agreements.html__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAktmm6BY$> >> >> Please let me know if this is resolved. This

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

, >>> and explicitly express that they are OK with it via filing CCLA. >>> >>> You can read the description of agreements on contribution and ICLA/CCLA >>> form from this page. >>> https://www.apache.org/licenses/contributor-agreements.html >>> <

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

che.org/licenses/contributor-agreements.html__;!!NCc8flgU!a1C5BeYxzO7gVVrGZ56kzunhigqd4SeXMg3dHddtkIdIpO5UwFH3dxzNpK3bc53vuAkFYJ3goLU8Hxev8npLyDrAktmm6BY$> >> >> Please let me know if this is resolved. This seems to me as a blocker to >> move on. Please also let me know if t

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

; Jungtaek Lim (HeartSaVioR) > > > On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni > wrote: > >> Hi Pavan, >> >> I looked at the PR, and the changes look simple and contained. It would >> be useful to add dynamic resource allocation to Spark Structured Streaming.

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

PR, and the changes look simple and contained. It would be > useful to add dynamic resource allocation to Spark Structured Streaming. > > Jungtaek. Would you be able to shepherd this change? > > > On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni > wrote: > >> Thanks a lot

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan, I looked at the PR, and the changes look simple and contained. It would be useful to add dynamic resource allocation to Spark Structured Streaming. Jungtaek. Would you be able to shepherd this change? On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni wrote: > Thanks a lot for creat

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

review of the SPIP document. FYI, I work closely with Pawan and other >> members to help deliver this piece of work. We appreciate your insights, >> especially regarding the cost savings potential from the PoC. >> >> Pavan already furnished you with some additional info.

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

aming > query with Processing Time trigger) is well-taken. We agree that > maintaining simplicity is key, particularly as we explore more general > resource allocation mechanisms in the future. To address the concerns and > foster open discussion, The DataBricks team are invited to direct

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Pavan already furnished you with some additional info. Your team's point about the SPIP currently addressing a specific use case (single streaming query with Processing Time trigger) is well-taken. We agree that maintaining simplicity is key, particularly as we explore more general resource

Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 Nivi

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

/apache/spark/pull/42352. (already mentioned in ticket SPARK-24815 <https://issues.apache.org/jira/browse/SPARK-24815>) I have built it on top of the current Dynamic resource allocation (DRA) algorithm <https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation

RE: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 Nivi

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan, I am from the DataBricks Structured Streaming team, and we did a review of the SPIP internally. Wanted to pass on the points discussed in the meeting. Thanks for putting together the SPIP document. It's useful to have dynamic resource allocation for Streaming queries, and

Re: Please unlock Jira ticket for SPARK-24815, Dynamic resource allocation for structured streaming

Thanks Yuming. On Mon, Feb 26, 2024 at 9:55 PM Yuming Wang wrote: > Unlocked. > > On Tue, Feb 27, 2024 at 11:47 AM Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> >> Hi, >> >> Can a committer please unlock this SPIP? It is for Dynamic re

Re: Please unlock Jira ticket for SPARK-24815, Dynamic resource allocation for structured streaming

Unlocked. On Tue, Feb 27, 2024 at 11:47 AM Mich Talebzadeh wrote: > > Hi, > > Can a committer please unlock this SPIP? It is for Dynamic resource > allocation for structured streaming that has got 6 votes. it was locked > because of inactivity by GitHub actions > >

Please unlock Jira ticket for SPARK-24815, Dynamic resource allocation for structured streaming

Hi, Can a committer please unlock this SPIP? It is for Dynamic resource allocation for structured streaming that has got 6 votes. it was locked because of inactivity by GitHub actions [SPARK-24815] Structured Streaming should support dynamic allocation - ASF JIRA (apache.org) <ht

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan and those who kindly voted for this SPIP Great to have 6+ votes and no -1 and 0. The so-called mass volume is there. The rest is admin matter and how to drive the project forward and yes there is more than one way of skinning the cat. I think we need some flexibility in the rules given th

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for the pointers Mich, will wait for Jungtaek Lee or any other PMC members to respond. aggregating upvotes to this email thread +6 Mich Talebzadeh Adam Hobbs Pavan Kotikalapudi Krystal Mitchell Sona Torosyan Aaron Kern Thank you, Pavan On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh wr

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 for me Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the

Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi, please check this doc Spark Project Improvement Proposals (SPIP) | Apache Spark and specifically the below extract Discussing an SPIP All discussion of an SPIP should take place in a public forum, preferably the discussion attached to t

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Mich, We have five +1s till now. Mich Talebzadeh Adam Hobbs Pavan Kotikalapudi Krystal Mitchell Sona Torosyan (few more in github pr) +0: None -1: None Does it pass the required condition as approved? Not sure of that though, nothing about minimum required is mentioned in the past emails

Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan, Do you have a list of votes for this feature by any chance? Does it pass the required condition as approved? HTH Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Yes. The PR was closed due to inactivity by github actions.. The msg also says > If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh wr

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

I can see it was closed. Was it because of inactivity? Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Spark PMC members, I think we have few upvotes for this effort here and more people are showing interest (see PR comments .) Is anyone interested in mentoring and reviewing this effort? Also can the repository admin/owner re

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: > Thanks for proposing and voting for the feature Mich. > > adding some references to the thread. > >- Jira ticket - SPARK-24815 > >- Design Doc > >

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Here is the link to the voting thread https://lists.apache.org/thread/rlwqrw6ddxdkbvkp78kpd0zgvglgbbp8. Thank you, Pavan On Wed, Jan 17, 2024 at 7:15 PM Pavan Kotikalapudi wrote: > Thanks for the +1, I will propose voting in a new thread now. > > - Pavan > > On Wed, Jan 17, 2024 at 5:28 PM Mic

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

, 2024 at 3:19 AM Adam Hobbs > wrote: > >> +1 >> -- >> *From:* Pavan Kotikalapudi >> *Sent:* Thursday, January 18, 2024 4:19:32 AM >> *To:* Spark dev list >> *Subject:* Re: Vote on Dynamic resource allocation for structured >> streaming [SPARK-24815] >>

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

e: Vote on Dynamic resource allocation for structured > streaming [SPARK-24815] > > > CAUTION: This email originated from outside of the organisation. Do not > click links or open attachments unless you recognise the sender's full > email address and know the content is safe

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 From: Pavan Kotikalapudi Sent: Thursday, January 18, 2024 4:19:32 AM To: Spark dev list Subject: Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815] CAUTION: This email originated from outside of the organisation. Do not click

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for proposing and voting for the feature Mich. adding some references to the thread. - Jira ticket - SPARK-24815 - Design Doc

Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 for me (non binding) *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monet

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for the +1, I will propose voting in a new thread now. - Pavan On Wed, Jan 17, 2024 at 5:28 PM Mich Talebzadeh wrote: > I think we have discussed this enough and I consider it as a useful > feature.. I propose a vote on it. > > + 1 for me > > Mich Talebzadeh, > Dad | Technologist | Solut

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

I think we have discussed this enough and I consider it as a useful feature.. I propose a vote on it. + 1 for me Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile h

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Hi, This is my first time using the dev mailing list so I hope this is the correct way to do it. I would like to lend my support to this proposal and offer my experiences as a consumer of spark, and specifically Spark Structured Streaming (SSS). I am more of an cloud infrastructure devops engi

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

additional feedback and perspectives, especially from those with experience in dynamic resource allocation in Spark. Having said that, the discussion above demonstrates a good understanding of the challenges involved in enhancing Spark Structured Streaming resource management capabilities. HTH Mich

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

and extended it to work for our streaming use-cases. As structured streaming is a micro-batch at the lowest level, I tuned the scaling actions based on micro-batches. Would appreciate it if anybody in the dev community who has worked on dynamic resource allocation (DRA) implementation

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, 20 Aug 2023 at 07:40, Pavan Kotikalapudi < >>>>> pkotikalap...@twi

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

>>>>> algorithm of SSS I believe we should start with some primitives of >>>>> Structured streaming. I would love to get some reviews on the doc and >>>>> opinions on the feasibility of the solution. >>>>> >>

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

ome reviews on the doc and >>>> opinions on the feasibility of the solution. >>>> >>>> We have seen quite some savings using this solution in our team, Would >>>> like to listen to the dev community to see if they are looking >>>>

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

ming. >>> >>> On Mon, Aug 14, 2023 at 9:12 AM Mich Talebzadeh < >>> mich.talebza...@gmail.com> wrote: >>> >>>> Thank you for your comments. >>>> >>>> My vision of integrating machine learning (ML) into Spark Structured >&

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

rating machine learning (ML) into Spark Structured >>> Streaming (SSS) for capacity planning and performance optimization seems to >>> be promising. By leveraging ML techniques, I believe that we can >>> potentially create predictive models that enhance the efficiency and

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

) for capacity planning and performance optimization seems to >> be promising. By leveraging ML techniques, I believe that we can >> potentially create predictive models that enhance the efficiency and >> resource allocation of the data processing pipelines. Here are some >> pot

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

tructured > Streaming (SSS) for capacity planning and performance optimization seems to > be promising. By leveraging ML techniques, I believe that we can > potentially create predictive models that enhance the efficiency and > resource allocation of the data processing pipelines. Here are some

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

efficiency and resource allocation of the data processing pipelines. Here are some potential benefits and considerations for adding ML to SSS for capacity planning. However, I stand corrected 1. *Predictive Capacity Planning:* ML models can analyze historical data (that we discussed already

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

. From: Mich Talebzadeh Sent: Tuesday, August 8, 2023 19:59 To: Pavan Kotikalapudi Cc: dev@spark.apache.org Subject: Re: Dynamic resource allocation for structured streaming [SPARK-24815] EXTERNAL SENDER. Do not click links or open attachments unless you recognize the sender and

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

I am currently contemplating and sharing my thoughts openly. Considering our reliance on previously collected statistics (as mentioned earlier), it raises the question of why we couldn't integrate certain machine learning elements into Spark Structured Streaming? While this might slightly deviate f

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Listeners are the best resources to the allocation manager afaik... It already has SparkListener that it utilizes. We can use it to extract more information (like processing time

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan or anyone else Is there any way one access the matrix displayed on SparkGUI? For example the readings for processing time? Can these be acessed? Thanks For example, Mich Talebzadeh, Solutions Architect/Engineering Lead London United Kingdom view my Linkedin profile

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for the review Mich, Yes, the configuration parameters we end up setting would be based on the trigger interval. > If you are going to have additional indicators why not look at scheduling delay as well Yes. The implementation is based on scheduling delays, not for pending tasks of the cur

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Hi, I glanced over the design doc. You are providing certain configuration parameters plus some settings based on static values. For example: spark.dynamicAllocation.schedulerBacklogTimeout": 54s I cannot see any use of which ought to be at least half of the batch interval to have the correct

Re: Dynamic resource allocation for structured streaming [SPARK-24815]

Oooh fascinating. I’m going on call this week so it will take me awhile but I do want to review this :) On Mon, Aug 7, 2023 at 5:30 PM Pavan Kotikalapudi wrote: > Hi Spark Dev, > > I have extended traditional DRA to work for structured streaming > use-case. > > Here is an initial Implementation

Fwd: Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Spark Dev, I have extended traditional DRA to work for structured streaming use-case. Here is an initial Implementation draft PR https://github.com/apache/spark/pull/42352 and design doc: https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing Please

Re: Re: spark+kafka+dynamic resource allocation

BR > Lingzhe Sun > > > *From:* Mich Talebzadeh > *Date:* 2023-01-30 02:14 > *To:* Lingzhe Sun > *CC:* ashok34...@yahoo.com; User > *Subject:* Re: Re: spark+kafka+dynamic resource allocation > Hi, > > Spark Structured Streaming currently does not support dynamic alloca

Resource allocation in SPARK streaming

I am not much clear about resource allocation (CPU/CORE/Thread level allocation) as per the parallelism by setting number of cores in spark standalone mode . Any guidelines for that . -- Thanks & Regards, Anshu Shukla

Re: Dynamic resource allocation in Standalone mode

pplication during runtime, and it is similar to dynamic > resource allocation, which is only feasible in Spark on Yarn mode. Any > suggestions on how to implement this feature in Standalone mode? > > My current problem is: I want to send a ADD_EXECUTOR comman

Dynamic resource allocation in Standalone mode

Hi all, I am planning to dynamically increase or decrease the number of executors allocated to an application during runtime, and it is similar to dynamic resource allocation, which is only feasible in Spark on Yarn mode. Any suggestions on how to implement this feature in Standalone mode? My

Resource allocation

Hi, I want to incorporate some intelligence while choosing the resources for rdd replication. I thought, if we replicate rdd on specially chosen nodes based on the capabilities, the next application that requires this rdd can be executed more efficiently. But, I found that an rdd creatd by an appp