Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

I'm still having a hard time reviewing this. I have been handling a bunch of context right now, and the change is non-trivial to review in parallel. I see people were OK with the algorithm in high-level, but from a code perspective it's uneasy to understand without knowledge of DRA. It would take

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Jungtaek, Status on current SPARK-24815 : Thomas Graves is reviewing the draft PR . I need to add documentation about the configs and usage details, I am planning to do that this week. He did mention

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Andrew, Sandy, Jerry, Thomas, marcelo, Whenchen, YangJie, Shixiong, My apologies. I have tagged soo many of you (on multiple emails), I am in the process of finding the core contributors of the Dynamic resource allocation (DRA) feature in apache/spark , I could

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Jungtaek, Sorry for the late reply. I understand the concerns towards finding PMC members, I had similar concerns in the past. Do you think we have something to improve in the SPIP (certain areas) so that it would get traction from PMC members? Or this SPIP might not be a priority to the PMC

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Sounds good. One thing I'd like to clarify before shepherding this SPIP is the process itself. Getting enough traction from PMC members is another issue to pass the SPIP vote. Even a vote from committer is not counted. (I don't have a binding vote.) I only see one PMC member (Thomas Graves, not

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Sounds good. Thanks again for your help on guiding the effort from discussion/review through voting phases in the spark dev community. Thank you, Pavan On Tue, Mar 26, 2024 at 4:20 AM Mich Talebzadeh wrote: > Hi Pavan, > > Thanks for instigating this proposal. Looks like the proposal is

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan, Thanks for instigating this proposal. Looks like the proposal is ready and has enough votes to be implemented. Having a sheppard will make it more fruitful. I will leave it to @Jungtaek Lim 's capable hands to drive it forward. Will be there to help if needed. Cheers Mich

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Bhuwan, Glad to hear back from you! Very much appreciate your help on reviewing the design doc/PR and endorsing this proposal. Thank you so much @Jungtaek Lim , @Mich Talebzadeh for graciously agreeing to mentor/shepherd this effort. Regarding Twilio copyright in Notice binary file:

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

I'm happy to, but it looks like I need to check one more thing about the license, according to the WIP PR . @Pavan Kotikalapudi I see you've added the copyright of Twilio in the NOTICE-binary file, which makes me wonder if Twilio had filed CCLA to the

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan, I looked at the PR, and the changes look simple and contained. It would be useful to add dynamic resource allocation to Spark Structured Streaming. Jungtaek. Would you be able to shepherd this change? On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni wrote: > Thanks a lot for creating

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks a lot for creating the risk table Pavan. My apologies. I was tied up with high priority items for the last couple weeks and could not respond. I will review the PR by tomorrow's end, and get back to you. Appreciate your patience. Thanks Bhuwan Sahni On Sun, Mar 17, 2024 at 4:42 PM Pavan

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Bhuwan, I hope the team got a chance to review the draft PR, looking for some comments to see if the plan looks alright?. I have updated the document about the risks .(also mentioned

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Bhuwan et al, Thank you for passing on the DataBricks Structured Streaming team's review of the SPIP document. FYI, I work closely with Pawan and other members to help deliver this piece of work. We appreciate your insights, especially regarding the cost savings potential from the PoC. Pavan

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks Bhuwan and rest of the databricks team for the reviews, I appreciate your reviews, was very helpful in evaluating a few options that were overlooked earlier (especially about mixed spark apps running on notebooks). Regarding the use-cases, It could handle multiple streaming queries

RE: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 Nivi

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan, I am from the DataBricks Structured Streaming team, and we did a review of the SPIP internally. Wanted to pass on the points discussed in the meeting. Thanks for putting together the SPIP document. It's useful to have dynamic resource allocation for Streaming queries, and it's

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan and those who kindly voted for this SPIP Great to have 6+ votes and no -1 and 0. The so-called mass volume is there. The rest is admin matter and how to drive the project forward and yes there is more than one way of skinning the cat. I think we need some flexibility in the rules given

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for the pointers Mich, will wait for Jungtaek Lee or any other PMC members to respond. aggregating upvotes to this email thread +6 Mich Talebzadeh Adam Hobbs Pavan Kotikalapudi Krystal Mitchell Sona Torosyan Aaron Kern Thank you, Pavan On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 for me Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi, please check this doc Spark Project Improvement Proposals (SPIP) | Apache Spark and specifically the below extract Discussing an SPIP All discussion of an SPIP should take place in a public forum, preferably the discussion attached to

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Mich, We have five +1s till now. Mich Talebzadeh Adam Hobbs Pavan Kotikalapudi Krystal Mitchell Sona Torosyan (few more in github pr) +0: None -1: None Does it pass the required condition as approved? Not sure of that though, nothing about minimum required is mentioned in the past

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Pavan, Do you have a list of votes for this feature by any chance? Does it pass the required condition as approved? HTH Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Yes. The PR was closed due to inactivity by github actions.. The msg also says > If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag! On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

I can see it was closed. Was it because of inactivity? Mich Talebzadeh, Dad | Technologist | Solutions Architect | Engineer London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Hi Spark PMC members, I think we have few upvotes for this effort here and more people are showing interest (see PR comments .) Is anyone interested in mentoring and reviewing this effort? Also can the repository admin/owner

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote: > Thanks for proposing and voting for the feature Mich. > > adding some references to the thread. > >- Jira ticket - SPARK-24815 > >- Design Doc > >

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

3:19 AM Adam Hobbs > wrote: > >> +1 >> -- >> *From:* Pavan Kotikalapudi >> *Sent:* Thursday, January 18, 2024 4:19:32 AM >> *To:* Spark dev list >> *Subject:* Re: Vote on Dynamic resource allocation for structured >> streaming [SPARK-24815] >> >> &

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

e: Vote on Dynamic resource allocation for structured > streaming [SPARK-24815] > > > CAUTION: This email originated from outside of the organisation. Do not > click links or open attachments unless you recognise the sender's full > email address and know the content is safe.

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

+1 From: Pavan Kotikalapudi Sent: Thursday, January 18, 2024 4:19:32 AM To: Spark dev list Subject: Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815] CAUTION: This email originated from outside of the organisation. Do not click

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

Thanks for proposing and voting for the feature Mich. adding some references to the thread. - Jira ticket - SPARK-24815 - Design Doc