Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Jungtaek Lim]

I'm still having a hard time reviewing this. I have been handling a bunch
of context right now, and the change is non-trivial to review in parallel.
I see people were OK with the algorithm in high-level, but from a code
perspective it's uneasy to understand without knowledge of DRA. It would
take me some time. I'll see whether I can get help from other folks.

That said, it'd be nice to make iterative changes. Let's go with the first
PR with very basic functionality and ensure we document very well so that
users would be able to use this. Also we have to document the limitations
with enough emphasis - likewise I said, having multiple queries in the same
driver (cluster) is a common case and we found the proposal does not
consider the case. Let's make a dedicated effort on the documentation.

On Sun, Apr 7, 2024 at 8:41 AM Pavan Kotikalapudi 
wrote:

> Hi Jungtaek,
>
> Status on current SPARK-24815
> :
> Thomas Graves is reviewing the draft PR
> . I need to add documentation
> about the configs and usage details, I am planning to do that this week.
> He did mention that it would be great if somebody with experience in
> structured streaming would take a look at the algorithm. Will you be able
> to review it?
>
> Another point I wanted to discuss is, as you might have already seen in
> the design doc
> 
>  we
> use traditional DRA configs
> spark.dynamicAllocation.enabled,
> spark.dynamicAllocation.schedulerBacklogTimeout,
> spark.dynamicAllocation.sustainedSchedulerBacklogTimeout,
> spark.dynamicAllocation.executorIdleTimeout,
> spark.dynamicAllocation.cachedExecutorIdleTimeout,
>
> and few new configs
>
> spark.dynamicAllocation.streaming.enabled,
> spark.dynamicAllocation.streaming.executorDeallocationRatio,
> spark.dynamicAllocation.streaming.executorDeallocationTimeout.
>
> to make the DRA work for structured streaming.
>
> While in the design doc I did mention that we have to calculate  and set
> scale out/back thresholds based on the trigger interval
> .
> We (internally in the company) do have helper functions to auto-generate
> the above configs based on trigger interval and the threshold configs (we
> also got similar feedback in reviews
> 
> ).
> Here are such configs
>
>   # required - should be greater than 3 seconds as that gives enough
> seconds for scaleOut and scaleBack thresholds to work with.
>   "spark.sql.streaming.triggerInterval.seconds": 
>   # optional - value should be between 0 and 1 and greater than
> scaleBackThreshold : default is 0.9
>   "spark.dynamicAllocation.streaming.scaleOutThreshold": 
>   # optional - value should be between 0 and 1 and less than
> scaleOutThreshold : default is 0.5
>   "spark.dynamicAllocation.streaming.scaleBackThreshold": 
>
> The above configs helps us to generate the below configs for app with
> different trigger intervals ( or if they change them for some reason)
>
> spark.dynamicAllocation.schedulerBacklogTimeout,
> spark.dynamicAllocation.sustainedSchedulerBacklogTimeout,
> spark.dynamicAllocation.executorIdleTimeout,
> spark.dynamicAllocation.cachedExecutorIdleTimeout.
>
> While our additional configs have its own limitations. I would like to get
> some feedback if adding such kinds of configs to automate
> the process of calculating the thresholds and their respective configs
> makes sense?
>
> Thank you,
>
> Pavan
>
> On Thu, Mar 28, 2024 at 3:38 PM Pavan Kotikalapudi <
> pkotikalap...@twilio.com> wrote:
>
>> Hi Jungtaek,
>>
>> Sorry for the late reply.
>>
>> I understand the concerns towards finding PMC members, I had similar
>> concerns in the past. Do you think we have something to improve in the SPIP
>> (certain areas) so that it would get traction from PMC members? Or this
>> SPIP might not be a priority to the PMC right now?
>>
>> I agree this change is small enough that it might not be tagged as an
>> SPIP. I started with the template SPIP questions so that it would be easier
>> to understand the limitations of the current system, new solution, how it
>> works, how to use it, limitations etcAs you might have already
>> noticed in the PR, This change is turned off by default, will only work if
>> `spark.dynamicAllocation.streaming.enabled` is true.
>>
>> Regarding the concerns about expertise in DRA,  I will find some core
>> contributors of this module/DRA and tag them to this email with details,
>> Mich has also highlighted the same in the past. Once we get approval from
>> them we can further discuss and enhance this to make the user experience
>> better.
>>
>> Thank you,
>>
>> Pavan
>>
>>
>> On Tue, Mar 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Hi Jungtaek,

Status on current SPARK-24815
:
Thomas Graves is reviewing the draft PR
. I need to add documentation
about the configs and usage details, I am planning to do that this week.
He did mention that it would be great if somebody with experience in
structured streaming would take a look at the algorithm. Will you be able
to review it?

Another point I wanted to discuss is, as you might have already seen
in the design
doc

we
use traditional DRA configs
spark.dynamicAllocation.enabled,
spark.dynamicAllocation.schedulerBacklogTimeout,
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout,
spark.dynamicAllocation.executorIdleTimeout,
spark.dynamicAllocation.cachedExecutorIdleTimeout,

and few new configs

spark.dynamicAllocation.streaming.enabled,
spark.dynamicAllocation.streaming.executorDeallocationRatio,
spark.dynamicAllocation.streaming.executorDeallocationTimeout.

to make the DRA work for structured streaming.

While in the design doc I did mention that we have to calculate  and set
scale out/back thresholds based on the trigger interval
.
We (internally in the company) do have helper functions to auto-generate
the above configs based on trigger interval and the threshold configs (we
also got similar feedback in reviews

).
Here are such configs

  # required - should be greater than 3 seconds as that gives enough
seconds for scaleOut and scaleBack thresholds to work with.
  "spark.sql.streaming.triggerInterval.seconds": 
  # optional - value should be between 0 and 1 and greater than
scaleBackThreshold : default is 0.9
  "spark.dynamicAllocation.streaming.scaleOutThreshold": 
  # optional - value should be between 0 and 1 and less than
scaleOutThreshold : default is 0.5
  "spark.dynamicAllocation.streaming.scaleBackThreshold": 

The above configs helps us to generate the below configs for app with
different trigger intervals ( or if they change them for some reason)

spark.dynamicAllocation.schedulerBacklogTimeout,
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout,
spark.dynamicAllocation.executorIdleTimeout,
spark.dynamicAllocation.cachedExecutorIdleTimeout.

While our additional configs have its own limitations. I would like to get
some feedback if adding such kinds of configs to automate
the process of calculating the thresholds and their respective configs
makes sense?

Thank you,

Pavan

On Thu, Mar 28, 2024 at 3:38 PM Pavan Kotikalapudi 
wrote:

> Hi Jungtaek,
>
> Sorry for the late reply.
>
> I understand the concerns towards finding PMC members, I had similar
> concerns in the past. Do you think we have something to improve in the SPIP
> (certain areas) so that it would get traction from PMC members? Or this
> SPIP might not be a priority to the PMC right now?
>
> I agree this change is small enough that it might not be tagged as an
> SPIP. I started with the template SPIP questions so that it would be easier
> to understand the limitations of the current system, new solution, how it
> works, how to use it, limitations etcAs you might have already
> noticed in the PR, This change is turned off by default, will only work if
> `spark.dynamicAllocation.streaming.enabled` is true.
>
> Regarding the concerns about expertise in DRA,  I will find some core
> contributors of this module/DRA and tag them to this email with details,
> Mich has also highlighted the same in the past. Once we get approval from
> them we can further discuss and enhance this to make the user experience
> better.
>
> Thank you,
>
> Pavan
>
>
> On Tue, Mar 26, 2024 at 8:12 PM Jungtaek Lim 
> wrote:
>
>> Sounds good.
>>
>> One thing I'd like to clarify before shepherding this SPIP is the process
>> itself. Getting enough traction from PMC members is another issue to pass
>> the SPIP vote. Even a vote from committer is not counted. (I don't have a
>> binding vote.) I only see one PMC member (Thomas Graves, not my team) in
>> the design doc and we still don't get positive feedback. So still a long
>> way to go. We need three supporters from PMC members.
>>
>> Another thing is, I get the proposal at a high level, but I don't have
>> actual expertise in DRA. I could review the code in general, but I feel
>> like I'm not qualified to approve the code. We still need an expert on the
>> CORE area, especially who has expertise with DRA. (Could you please
>> annotate the code and enumerate several people who worked on the codebase?)
>> If they need an expertise of streaming to understand how things will work
>> then either you or I can explain, but I can't just approve and merge the
>> code.

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Hi Andrew, Sandy, Jerry, Thomas, marcelo, Whenchen, YangJie, Shixiong,

My apologies. I have tagged soo many of you (on multiple emails), I am in
the process of finding the core contributors of the Dynamic resource
allocation (DRA) feature in apache/spark ,
I could find you folks as some of the core contributing members to this
feature.

We(cc'd) would like to extend the current DRA to work for structured
streaming [SPARK-24815 ]
use-case (based on the heuristics of trigger interval).
Here is the design doc
.
We also have a draft PR  with
initial implementation.

This feature has been running well for the past one year at my company
(twilio) and there are a lot of folks in the community who are interested
in this feature.

Do get the PR to a mergeable state. We would love to leverage your
expertise on DRA. I request you to please review the design doc and the
draft PR, let us know your thoughts and concerns if any. This will hugely
benefit the community utilizing structured streaming applications in their
data pipelines.

Looking forward to hear back from you.

Thank you,

Pavan

On Thu, Mar 28, 2024 at 3:38 PM Pavan Kotikalapudi 
wrote:

> Hi Jungtaek,
>
> Sorry for the late reply.
>
> I understand the concerns towards finding PMC members, I had similar
> concerns in the past. Do you think we have something to improve in the SPIP
> (certain areas) so that it would get traction from PMC members? Or this
> SPIP might not be a priority to the PMC right now?
>
> I agree this change is small enough that it might not be tagged as an
> SPIP. I started with the template SPIP questions so that it would be easier
> to understand the limitations of the current system, new solution, how it
> works, how to use it, limitations etcAs you might have already
> noticed in the PR, This change is turned off by default, will only work if
> `spark.dynamicAllocation.streaming.enabled` is true.
>
> Regarding the concerns about expertise in DRA,  I will find some core
> contributors of this module/DRA and tag them to this email with details,
> Mich has also highlighted the same in the past. Once we get approval from
> them we can further discuss and enhance this to make the user experience
> better.
>
> Thank you,
>
> Pavan
>
>
> On Tue, Mar 26, 2024 at 8:12 PM Jungtaek Lim 
> wrote:
>
>> Sounds good.
>>
>> One thing I'd like to clarify before shepherding this SPIP is the process
>> itself. Getting enough traction from PMC members is another issue to pass
>> the SPIP vote. Even a vote from committer is not counted. (I don't have a
>> binding vote.) I only see one PMC member (Thomas Graves, not my team) in
>> the design doc and we still don't get positive feedback. So still a long
>> way to go. We need three supporters from PMC members.
>>
>> Another thing is, I get the proposal at a high level, but I don't have
>> actual expertise in DRA. I could review the code in general, but I feel
>> like I'm not qualified to approve the code. We still need an expert on the
>> CORE area, especially who has expertise with DRA. (Could you please
>> annotate the code and enumerate several people who worked on the codebase?)
>> If they need an expertise of streaming to understand how things will work
>> then either you or I can explain, but I can't just approve and merge the
>> code.
>>
>> That said, if we succeed in finding one and they review the code and
>> LGTM, I'd rather say not to go with taking the process of SPIP unless the
>> expert reviewing your code requires us to do so. The change you proposed is
>> rather small and does not seem to be invasive (experts can also weigh), and
>> there must never be the case that this feature is turned on by default (as
>> we pointed out limitation). It doesn't look like requiring SPIP, if we
>> carefully document the new change and also clearly describe the limitation.
>> (Also a warning in the codebase that this must not be enabled by default.)
>>
>>
>> On Tue, Mar 26, 2024 at 7:02 PM Pavan Kotikalapudi <
>> pkotikalap...@twilio.com> wrote:
>>
>>> Hi Bhuwan,
>>>
>>> Glad to hear back from you! Very much appreciate your help on reviewing
>>> the design doc/PR and endorsing this proposal.
>>>
>>> Thank you so much @Jungtaek Lim  , @Mich
>>> Talebzadeh   for graciously agreeing to
>>> mentor/shepherd this effort.
>>>
>>> Regarding Twilio copyright in Notice binary file:
>>> Twilio Opensource counsel was involved all through the process, I have
>>> placed it in the project file prior to Twilio signing a CCLA for the spark
>>> project contribution( Aug '23).
>>>
>>> Since the CCLA is signed now, I have removed the twilio copyright from
>>> that file. I didn't get a chance to update the PR after github-actions
>>> closed it.
>>>
>>> Please let me know of next steps needed to 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Hi Jungtaek,

Sorry for the late reply.

I understand the concerns towards finding PMC members, I had similar
concerns in the past. Do you think we have something to improve in the SPIP
(certain areas) so that it would get traction from PMC members? Or this
SPIP might not be a priority to the PMC right now?

I agree this change is small enough that it might not be tagged as an SPIP.
I started with the template SPIP questions so that it would be easier to
understand the limitations of the current system, new solution, how it
works, how to use it, limitations etcAs you might have already
noticed in the PR, This change is turned off by default, will only work if
`spark.dynamicAllocation.streaming.enabled` is true.

Regarding the concerns about expertise in DRA,  I will find some core
contributors of this module/DRA and tag them to this email with details,
Mich has also highlighted the same in the past. Once we get approval from
them we can further discuss and enhance this to make the user experience
better.

Thank you,

Pavan


On Tue, Mar 26, 2024 at 8:12 PM Jungtaek Lim 
wrote:

> Sounds good.
>
> One thing I'd like to clarify before shepherding this SPIP is the process
> itself. Getting enough traction from PMC members is another issue to pass
> the SPIP vote. Even a vote from committer is not counted. (I don't have a
> binding vote.) I only see one PMC member (Thomas Graves, not my team) in
> the design doc and we still don't get positive feedback. So still a long
> way to go. We need three supporters from PMC members.
>
> Another thing is, I get the proposal at a high level, but I don't have
> actual expertise in DRA. I could review the code in general, but I feel
> like I'm not qualified to approve the code. We still need an expert on the
> CORE area, especially who has expertise with DRA. (Could you please
> annotate the code and enumerate several people who worked on the codebase?)
> If they need an expertise of streaming to understand how things will work
> then either you or I can explain, but I can't just approve and merge the
> code.
>
> That said, if we succeed in finding one and they review the code and LGTM,
> I'd rather say not to go with taking the process of SPIP unless the expert
> reviewing your code requires us to do so. The change you proposed is rather
> small and does not seem to be invasive (experts can also weigh), and there
> must never be the case that this feature is turned on by default (as we
> pointed out limitation). It doesn't look like requiring SPIP, if we
> carefully document the new change and also clearly describe the limitation.
> (Also a warning in the codebase that this must not be enabled by default.)
>
>
> On Tue, Mar 26, 2024 at 7:02 PM Pavan Kotikalapudi <
> pkotikalap...@twilio.com> wrote:
>
>> Hi Bhuwan,
>>
>> Glad to hear back from you! Very much appreciate your help on reviewing
>> the design doc/PR and endorsing this proposal.
>>
>> Thank you so much @Jungtaek Lim  , @Mich
>> Talebzadeh   for graciously agreeing to
>> mentor/shepherd this effort.
>>
>> Regarding Twilio copyright in Notice binary file:
>> Twilio Opensource counsel was involved all through the process, I have
>> placed it in the project file prior to Twilio signing a CCLA for the spark
>> project contribution( Aug '23).
>>
>> Since the CCLA is signed now, I have removed the twilio copyright from
>> that file. I didn't get a chance to update the PR after github-actions
>> closed it.
>>
>> Please let me know of next steps needed to bring this draft PR/effort to
>> completion.
>>
>> Thank you,
>>
>> Pavan
>>
>>
>> On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim <
>> kabhwan.opensou...@gmail.com> wrote:
>>
>>> I'm happy to, but it looks like I need to check one more thing about the
>>> license, according to the WIP PR
>>> 
>>> .
>>>
>>> @Pavan Kotikalapudi 
>>> I see you've added the copyright of Twilio in the NOTICE-binary file,
>>> which makes me wonder if Twilio had filed CCLA to the Apache Software
>>> Foundation.
>>>
>>> PMC members can correct me if I'm mistaken, but from my understanding
>>> (and experiences of PMC member in other ASF project), code contribution is
>>> considered as code donation and copyright belongs to ASF. That's why you
>>> can't find the copyright of employers for contributors in the codebase.
>>> What you see copyrights in NOTICE-binary is due to the fact we have binary
>>> dependency and their licenses may require to explicitly mention about
>>> copyright. It's not about direct code contribution.
>>>
>>> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior,
>>> could you please engage with a relevant group in the company (could be a
>>> legal team, or similar with OSS advocate team if there is any) and ensure
>>> that CCLA is filed? The copyright issue is a legal issue, 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Jungtaek Lim]

Sounds good.

One thing I'd like to clarify before shepherding this SPIP is the process
itself. Getting enough traction from PMC members is another issue to pass
the SPIP vote. Even a vote from committer is not counted. (I don't have a
binding vote.) I only see one PMC member (Thomas Graves, not my team) in
the design doc and we still don't get positive feedback. So still a long
way to go. We need three supporters from PMC members.

Another thing is, I get the proposal at a high level, but I don't have
actual expertise in DRA. I could review the code in general, but I feel
like I'm not qualified to approve the code. We still need an expert on the
CORE area, especially who has expertise with DRA. (Could you please
annotate the code and enumerate several people who worked on the codebase?)
If they need an expertise of streaming to understand how things will work
then either you or I can explain, but I can't just approve and merge the
code.

That said, if we succeed in finding one and they review the code and LGTM,
I'd rather say not to go with taking the process of SPIP unless the expert
reviewing your code requires us to do so. The change you proposed is rather
small and does not seem to be invasive (experts can also weigh), and there
must never be the case that this feature is turned on by default (as we
pointed out limitation). It doesn't look like requiring SPIP, if we
carefully document the new change and also clearly describe the limitation.
(Also a warning in the codebase that this must not be enabled by default.)


On Tue, Mar 26, 2024 at 7:02 PM Pavan Kotikalapudi 
wrote:

> Hi Bhuwan,
>
> Glad to hear back from you! Very much appreciate your help on reviewing
> the design doc/PR and endorsing this proposal.
>
> Thank you so much @Jungtaek Lim  , @Mich
> Talebzadeh   for graciously agreeing to
> mentor/shepherd this effort.
>
> Regarding Twilio copyright in Notice binary file:
> Twilio Opensource counsel was involved all through the process, I have
> placed it in the project file prior to Twilio signing a CCLA for the spark
> project contribution( Aug '23).
>
> Since the CCLA is signed now, I have removed the twilio copyright from
> that file. I didn't get a chance to update the PR after github-actions
> closed it.
>
> Please let me know of next steps needed to bring this draft PR/effort to
> completion.
>
> Thank you,
>
> Pavan
>
>
> On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim <
> kabhwan.opensou...@gmail.com> wrote:
>
>> I'm happy to, but it looks like I need to check one more thing about the
>> license, according to the WIP PR
>> 
>> .
>>
>> @Pavan Kotikalapudi 
>> I see you've added the copyright of Twilio in the NOTICE-binary file,
>> which makes me wonder if Twilio had filed CCLA to the Apache Software
>> Foundation.
>>
>> PMC members can correct me if I'm mistaken, but from my understanding
>> (and experiences of PMC member in other ASF project), code contribution is
>> considered as code donation and copyright belongs to ASF. That's why you
>> can't find the copyright of employers for contributors in the codebase.
>> What you see copyrights in NOTICE-binary is due to the fact we have binary
>> dependency and their licenses may require to explicitly mention about
>> copyright. It's not about direct code contribution.
>>
>> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior,
>> could you please engage with a relevant group in the company (could be a
>> legal team, or similar with OSS advocate team if there is any) and ensure
>> that CCLA is filed? The copyright issue is a legal issue, so we have to be
>> conservative and 100% sure that the employer is aware of what is the
>> meaning of donating the code to ASF via reviewing CCLA and relevant doc,
>> and explicitly express that they are OK with it via filing CCLA.
>>
>> You can read the description of agreements on contribution and ICLA/CCLA
>> form from this page.
>> https://www.apache.org/licenses/contributor-agreements.html
>> 
>>
>> Please let me know if this is resolved. This seems to me as a blocker to
>> move on. Please also let me know if the contribution is withdrawn from the
>> employer.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>>
>> On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni
>>  wrote:
>>
>>> Hi Pavan,
>>>
>>> I looked at the PR, and the changes look simple and contained. It would
>>> be useful to add dynamic resource allocation to Spark Structured Streaming.
>>>
>>> Jungtaek. Would you be able to shepherd this change?
>>>
>>>
>>> On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni <
>>> bhuwan.sa...@databricks.com> wrote:
>>>
 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Sounds good.

Thanks again for your help on guiding the effort from discussion/review
through voting phases in the spark dev community.

Thank you,

Pavan

On Tue, Mar 26, 2024 at 4:20 AM Mich Talebzadeh 
wrote:

> Hi Pavan,
>
> Thanks for instigating this proposal. Looks like the proposal is ready and
> has enough votes to be implemented. Having a sheppard will make it more
> fruitful.
>
> I will leave it to @Jungtaek Lim  's
> capable hands to drive it forward.
>
> Will be there to help if needed.
>
> Cheers
>
> Mich Talebzadeh,
> Technologist | Solutions Architect | Data Engineer  | Generative AI
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
> 
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner
> Von
> Braun
> 
> )".
>
>
> On Tue, 26 Mar 2024 at 10:02, Pavan Kotikalapudi 
> wrote:
>
>> Hi Bhuwan,
>>
>> Glad to hear back from you! Very much appreciate your help on reviewing
>> the design doc/PR and endorsing this proposal.
>>
>> Thank you so much @Jungtaek Lim  , @Mich
>> Talebzadeh   for graciously agreeing to
>> mentor/shepherd this effort.
>>
>> Regarding Twilio copyright in Notice binary file:
>> Twilio Opensource counsel was involved all through the process, I have
>> placed it in the project file prior to Twilio signing a CCLA for the spark
>> project contribution( Aug '23).
>>
>> Since the CCLA is signed now, I have removed the twilio copyright from
>> that file. I didn't get a chance to update the PR after github-actions
>> closed it.
>>
>> Please let me know of next steps needed to bring this draft PR/effort to
>> completion.
>>
>> Thank you,
>>
>> Pavan
>>
>>
>> On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim <
>> kabhwan.opensou...@gmail.com> wrote:
>>
>>> I'm happy to, but it looks like I need to check one more thing about the
>>> license, according to the WIP PR
>>> 
>>> .
>>>
>>> @Pavan Kotikalapudi 
>>> I see you've added the copyright of Twilio in the NOTICE-binary file,
>>> which makes me wonder if Twilio had filed CCLA to the Apache Software
>>> Foundation.
>>>
>>> PMC members can correct me if I'm mistaken, but from my understanding
>>> (and experiences of PMC member in other ASF project), code contribution is
>>> considered as code donation and copyright belongs to ASF. That's why you
>>> can't find the copyright of employers for contributors in the codebase.
>>> What you see copyrights in NOTICE-binary is due to the fact we have binary
>>> dependency and their licenses may require to explicitly mention about
>>> copyright. It's not about direct code contribution.
>>>
>>> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior,
>>> could you please engage with a relevant group in the company (could be a
>>> legal team, or similar with OSS advocate team if there is any) and ensure
>>> that CCLA is filed? The copyright issue is a legal issue, so we have to be
>>> conservative and 100% sure that the employer is aware of what is the
>>> meaning of donating the code to ASF via reviewing CCLA and relevant doc,
>>> and explicitly express that they are OK with it via filing CCLA.
>>>
>>> You can read the description of agreements on contribution and ICLA/CCLA
>>> form from this page.
>>> https://www.apache.org/licenses/contributor-agreements.html
>>> 
>>>
>>> Please let me know if this is resolved. This seems to me as a blocker to
>>> move on. Please also let me know if the contribution is withdrawn from the
>>> employer.
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>>
>>> On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni
>>>  wrote:
>>>
 Hi Pavan,

 I looked at the PR, and the changes look simple 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Mich Talebzadeh]

Hi Pavan,

Thanks for instigating this proposal. Looks like the proposal is ready and
has enough votes to be implemented. Having a sheppard will make it more
fruitful.

I will leave it to @Jungtaek Lim  's
capable hands to drive it forward.

Will be there to help if needed.

Cheers

Mich Talebzadeh,
Technologist | Solutions Architect | Data Engineer  | Generative AI
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner  Von
Braun )".


On Tue, 26 Mar 2024 at 10:02, Pavan Kotikalapudi 
wrote:

> Hi Bhuwan,
>
> Glad to hear back from you! Very much appreciate your help on reviewing
> the design doc/PR and endorsing this proposal.
>
> Thank you so much @Jungtaek Lim  , @Mich
> Talebzadeh   for graciously agreeing to
> mentor/shepherd this effort.
>
> Regarding Twilio copyright in Notice binary file:
> Twilio Opensource counsel was involved all through the process, I have
> placed it in the project file prior to Twilio signing a CCLA for the spark
> project contribution( Aug '23).
>
> Since the CCLA is signed now, I have removed the twilio copyright from
> that file. I didn't get a chance to update the PR after github-actions
> closed it.
>
> Please let me know of next steps needed to bring this draft PR/effort to
> completion.
>
> Thank you,
>
> Pavan
>
>
> On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim <
> kabhwan.opensou...@gmail.com> wrote:
>
>> I'm happy to, but it looks like I need to check one more thing about the
>> license, according to the WIP PR
>> 
>> .
>>
>> @Pavan Kotikalapudi 
>> I see you've added the copyright of Twilio in the NOTICE-binary file,
>> which makes me wonder if Twilio had filed CCLA to the Apache Software
>> Foundation.
>>
>> PMC members can correct me if I'm mistaken, but from my understanding
>> (and experiences of PMC member in other ASF project), code contribution is
>> considered as code donation and copyright belongs to ASF. That's why you
>> can't find the copyright of employers for contributors in the codebase.
>> What you see copyrights in NOTICE-binary is due to the fact we have binary
>> dependency and their licenses may require to explicitly mention about
>> copyright. It's not about direct code contribution.
>>
>> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior,
>> could you please engage with a relevant group in the company (could be a
>> legal team, or similar with OSS advocate team if there is any) and ensure
>> that CCLA is filed? The copyright issue is a legal issue, so we have to be
>> conservative and 100% sure that the employer is aware of what is the
>> meaning of donating the code to ASF via reviewing CCLA and relevant doc,
>> and explicitly express that they are OK with it via filing CCLA.
>>
>> You can read the description of agreements on contribution and ICLA/CCLA
>> form from this page.
>> https://www.apache.org/licenses/contributor-agreements.html
>> 
>>
>> Please let me know if this is resolved. This seems to me as a blocker to
>> move on. Please also let me know if the contribution is withdrawn from the
>> employer.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>>
>> On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni
>>  wrote:
>>
>>> Hi Pavan,
>>>
>>> I looked at the PR, and the changes look simple and contained. It would
>>> be useful to add dynamic resource allocation to Spark Structured Streaming.
>>>
>>> Jungtaek. Would you be able to shepherd this change?
>>>
>>>
>>> On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni <
>>> bhuwan.sa...@databricks.com> wrote:
>>>
 Thanks a lot for creating the risk table Pavan. My apologies. I was
 tied up with high priority items for the last couple weeks and could not
 respond. I will review the PR by tomorrow's end, and get back to you.

 Appreciate your patience.

 Thanks
 Bhuwan Sahni

 On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi <
 pkotikalap...@twilio.com> wrote:

> Hi Bhuwan,
>
> I hope the team got a chance to review the draft PR, looking for some
> comments to see if the plan looks alright?. I have updated the document
> about the risks
> 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Hi Bhuwan,

Glad to hear back from you! Very much appreciate your help on reviewing the
design doc/PR and endorsing this proposal.

Thank you so much @Jungtaek Lim  , @Mich
Talebzadeh   for graciously agreeing to
mentor/shepherd this effort.

Regarding Twilio copyright in Notice binary file:
Twilio Opensource counsel was involved all through the process, I have
placed it in the project file prior to Twilio signing a CCLA for the spark
project contribution( Aug '23).

Since the CCLA is signed now, I have removed the twilio copyright from that
file. I didn't get a chance to update the PR after github-actions closed it.

Please let me know of next steps needed to bring this draft PR/effort to
completion.

Thank you,

Pavan


On Tue, Mar 26, 2024 at 12:01 AM Jungtaek Lim 
wrote:

> I'm happy to, but it looks like I need to check one more thing about the
> license, according to the WIP PR
> 
> .
>
> @Pavan Kotikalapudi 
> I see you've added the copyright of Twilio in the NOTICE-binary file,
> which makes me wonder if Twilio had filed CCLA to the Apache Software
> Foundation.
>
> PMC members can correct me if I'm mistaken, but from my understanding (and
> experiences of PMC member in other ASF project), code contribution is
> considered as code donation and copyright belongs to ASF. That's why you
> can't find the copyright of employers for contributors in the codebase.
> What you see copyrights in NOTICE-binary is due to the fact we have binary
> dependency and their licenses may require to explicitly mention about
> copyright. It's not about direct code contribution.
>
> Is Twilio aware of this? Also, if Twilio did not file CCLA in prior, could
> you please engage with a relevant group in the company (could be a legal
> team, or similar with OSS advocate team if there is any) and ensure that
> CCLA is filed? The copyright issue is a legal issue, so we have to be
> conservative and 100% sure that the employer is aware of what is the
> meaning of donating the code to ASF via reviewing CCLA and relevant doc,
> and explicitly express that they are OK with it via filing CCLA.
>
> You can read the description of agreements on contribution and ICLA/CCLA
> form from this page.
> https://www.apache.org/licenses/contributor-agreements.html
> 
>
> Please let me know if this is resolved. This seems to me as a blocker to
> move on. Please also let me know if the contribution is withdrawn from the
> employer.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni
>  wrote:
>
>> Hi Pavan,
>>
>> I looked at the PR, and the changes look simple and contained. It would
>> be useful to add dynamic resource allocation to Spark Structured Streaming.
>>
>> Jungtaek. Would you be able to shepherd this change?
>>
>>
>> On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni <
>> bhuwan.sa...@databricks.com> wrote:
>>
>>> Thanks a lot for creating the risk table Pavan. My apologies. I was tied
>>> up with high priority items for the last couple weeks and could not
>>> respond. I will review the PR by tomorrow's end, and get back to you.
>>>
>>> Appreciate your patience.
>>>
>>> Thanks
>>> Bhuwan Sahni
>>>
>>> On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi <
>>> pkotikalap...@twilio.com> wrote:
>>>
 Hi Bhuwan,

 I hope the team got a chance to review the draft PR, looking for some
 comments to see if the plan looks alright?. I have updated the document
 about the risks
 .(also
 mentioned below). Please confirm if it looks alright?

 *Spark application type*

 *auto-scaling capability*

 *with New auto-scaling capability*

 Spark Batch job

 Works with current DRA

 No - change

 Streaming query without trigger interval

 No implementation

 Can work with this implementation - (have to set certain scale back
 configs based on previous usage pattern) - maybe automate with future work?

 Spark Streaming query with Trigger interval

 No implementation

 With this implementation

 Spark Streaming query with one-time micro batch

 Works with current DRA

 No - change

 Spark Streaming query with

 Availablenow micro batch

 Works with current DRA

 No - change

 Batch + Streaming query (

 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Jungtaek Lim]

I'm happy to, but it looks like I need to check one more thing about the
license, according to the WIP PR
.

@Pavan Kotikalapudi 
I see you've added the copyright of Twilio in the NOTICE-binary file, which
makes me wonder if Twilio had filed CCLA to the Apache Software Foundation.

PMC members can correct me if I'm mistaken, but from my understanding (and
experiences of PMC member in other ASF project), code contribution is
considered as code donation and copyright belongs to ASF. That's why you
can't find the copyright of employers for contributors in the codebase.
What you see copyrights in NOTICE-binary is due to the fact we have binary
dependency and their licenses may require to explicitly mention about
copyright. It's not about direct code contribution.

Is Twilio aware of this? Also, if Twilio did not file CCLA in prior, could
you please engage with a relevant group in the company (could be a legal
team, or similar with OSS advocate team if there is any) and ensure that
CCLA is filed? The copyright issue is a legal issue, so we have to be
conservative and 100% sure that the employer is aware of what is the
meaning of donating the code to ASF via reviewing CCLA and relevant doc,
and explicitly express that they are OK with it via filing CCLA.

You can read the description of agreements on contribution and ICLA/CCLA
form from this page.
https://www.apache.org/licenses/contributor-agreements.html

Please let me know if this is resolved. This seems to me as a blocker to
move on. Please also let me know if the contribution is withdrawn from the
employer.

Thanks,
Jungtaek Lim (HeartSaVioR)


On Mon, Mar 25, 2024 at 11:47 PM Bhuwan Sahni
 wrote:

> Hi Pavan,
>
> I looked at the PR, and the changes look simple and contained. It would be
> useful to add dynamic resource allocation to Spark Structured Streaming.
>
> Jungtaek. Would you be able to shepherd this change?
>
>
> On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni 
> wrote:
>
>> Thanks a lot for creating the risk table Pavan. My apologies. I was tied
>> up with high priority items for the last couple weeks and could not
>> respond. I will review the PR by tomorrow's end, and get back to you.
>>
>> Appreciate your patience.
>>
>> Thanks
>> Bhuwan Sahni
>>
>> On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi <
>> pkotikalap...@twilio.com> wrote:
>>
>>> Hi Bhuwan,
>>>
>>> I hope the team got a chance to review the draft PR, looking for some
>>> comments to see if the plan looks alright?. I have updated the document
>>> about the risks
>>> .(also
>>> mentioned below). Please confirm if it looks alright?
>>>
>>> *Spark application type*
>>>
>>> *auto-scaling capability*
>>>
>>> *with New auto-scaling capability*
>>>
>>> Spark Batch job
>>>
>>> Works with current DRA
>>>
>>> No - change
>>>
>>> Streaming query without trigger interval
>>>
>>> No implementation
>>>
>>> Can work with this implementation - (have to set certain scale back
>>> configs based on previous usage pattern) - maybe automate with future work?
>>>
>>> Spark Streaming query with Trigger interval
>>>
>>> No implementation
>>>
>>> With this implementation
>>>
>>> Spark Streaming query with one-time micro batch
>>>
>>> Works with current DRA
>>>
>>> No - change
>>>
>>> Spark Streaming query with
>>>
>>> Availablenow micro batch
>>>
>>> Works with current DRA
>>>
>>> No - change
>>>
>>> Batch + Streaming query (
>>>
>>> default/
>>>
>>> triggger-interval/
>>>
>>> once/
>>>
>>> availablenow modes), other notebook use cases.
>>>
>>> No implementation
>>>
>>> No implementation
>>>
>>>
>>>
>>> We are more than happy to collaborate on a call to make better progress
>>> on this enhancement. Please let us know.
>>>
>>> Thank you,
>>>
>>> Pavan
>>>
>>> On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>

 Hi Bhuwan et al,

 Thank you for passing on the DataBricks Structured Streaming team's
 review of the SPIP document. FYI, I work closely with Pawan and other
 members to help deliver this piece of work. We appreciate your insights,
 especially regarding the cost savings potential from the PoC.

 Pavan already furnished you with some additional info. Your team's
 point about the SPIP currently addressing a specific use case (single
 streaming query with Processing Time trigger) is well-taken. We agree that
 maintaining simplicity is key, particularly as we explore more general
 resource allocation mechanisms in the future. To address the concerns and
 foster open discussion, The DataBricks team are invited to directly add
 their comments and suggestions to the Jira itself

 [SPARK-24815] Structured Streaming should support dynamic allocation -
 ASF JIRA (apache.org)
 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Bhuwan Sahni]

Hi Pavan,

I looked at the PR, and the changes look simple and contained. It would be
useful to add dynamic resource allocation to Spark Structured Streaming.

Jungtaek. Would you be able to shepherd this change?


On Tue, Mar 19, 2024 at 10:38 AM Bhuwan Sahni 
wrote:

> Thanks a lot for creating the risk table Pavan. My apologies. I was tied
> up with high priority items for the last couple weeks and could not
> respond. I will review the PR by tomorrow's end, and get back to you.
>
> Appreciate your patience.
>
> Thanks
> Bhuwan Sahni
>
> On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi <
> pkotikalap...@twilio.com> wrote:
>
>> Hi Bhuwan,
>>
>> I hope the team got a chance to review the draft PR, looking for some
>> comments to see if the plan looks alright?. I have updated the document
>> about the risks
>> .(also
>> mentioned below). Please confirm if it looks alright?
>>
>> *Spark application type*
>>
>> *auto-scaling capability*
>>
>> *with New auto-scaling capability*
>>
>> Spark Batch job
>>
>> Works with current DRA
>>
>> No - change
>>
>> Streaming query without trigger interval
>>
>> No implementation
>>
>> Can work with this implementation - (have to set certain scale back
>> configs based on previous usage pattern) - maybe automate with future work?
>>
>> Spark Streaming query with Trigger interval
>>
>> No implementation
>>
>> With this implementation
>>
>> Spark Streaming query with one-time micro batch
>>
>> Works with current DRA
>>
>> No - change
>>
>> Spark Streaming query with
>>
>> Availablenow micro batch
>>
>> Works with current DRA
>>
>> No - change
>>
>> Batch + Streaming query (
>>
>> default/
>>
>> triggger-interval/
>>
>> once/
>>
>> availablenow modes), other notebook use cases.
>>
>> No implementation
>>
>> No implementation
>>
>>
>>
>> We are more than happy to collaborate on a call to make better progress
>> on this enhancement. Please let us know.
>>
>> Thank you,
>>
>> Pavan
>>
>> On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>>
>>> Hi Bhuwan et al,
>>>
>>> Thank you for passing on the DataBricks Structured Streaming team's
>>> review of the SPIP document. FYI, I work closely with Pawan and other
>>> members to help deliver this piece of work. We appreciate your insights,
>>> especially regarding the cost savings potential from the PoC.
>>>
>>> Pavan already furnished you with some additional info. Your team's point
>>> about the SPIP currently addressing a specific use case (single streaming
>>> query with Processing Time trigger) is well-taken. We agree that
>>> maintaining simplicity is key, particularly as we explore more general
>>> resource allocation mechanisms in the future. To address the concerns and
>>> foster open discussion, The DataBricks team are invited to directly add
>>> their comments and suggestions to the Jira itself
>>>
>>> [SPARK-24815] Structured Streaming should support dynamic allocation -
>>> ASF JIRA (apache.org)
>>> 
>>> This will ensure everyone involved can benefit from your team's
>>> expertise and facilitate further collaboration.
>>>
>>> Thanks
>>>
>>> Mich Talebzadeh,
>>> Dad | Technologist | Solutions Architect | Engineer
>>> London
>>> United Kingdom
>>>
>>>
>>>view my Linkedin profile
>>> 
>>>
>>>
>>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>> 
>>>
>>>
>>>
>>> *Disclaimer:* The information provided is correct to the best of my
>>> knowledge but of course cannot be guaranteed . It is essential to note
>>> that, as with any advice, quote "one test result is worth one-thousand
>>> expert opinions (Werner
>>> Von
>>> Braun
>>> 
>>> )".
>>>
>>>
>>> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi
>>>  wrote:
>>>
 Thanks Bhuwan and rest of the databricks team for the reviews,

 I appreciate your reviews, was very helpful in evaluating a few options
 that were overlooked earlier (especially about mixed spark apps running on
 notebooks). 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Bhuwan Sahni]

Thanks a lot for creating the risk table Pavan. My apologies. I was tied up
with high priority items for the last couple weeks and could not respond. I
will review the PR by tomorrow's end, and get back to you.

Appreciate your patience.

Thanks
Bhuwan Sahni

On Sun, Mar 17, 2024 at 4:42 PM Pavan Kotikalapudi 
wrote:

> Hi Bhuwan,
>
> I hope the team got a chance to review the draft PR, looking for some
> comments to see if the plan looks alright?. I have updated the document
> about the risks
> .(also
> mentioned below). Please confirm if it looks alright?
>
> *Spark application type*
>
> *auto-scaling capability*
>
> *with New auto-scaling capability*
>
> Spark Batch job
>
> Works with current DRA
>
> No - change
>
> Streaming query without trigger interval
>
> No implementation
>
> Can work with this implementation - (have to set certain scale back
> configs based on previous usage pattern) - maybe automate with future work?
>
> Spark Streaming query with Trigger interval
>
> No implementation
>
> With this implementation
>
> Spark Streaming query with one-time micro batch
>
> Works with current DRA
>
> No - change
>
> Spark Streaming query with
>
> Availablenow micro batch
>
> Works with current DRA
>
> No - change
>
> Batch + Streaming query (
>
> default/
>
> triggger-interval/
>
> once/
>
> availablenow modes), other notebook use cases.
>
> No implementation
>
> No implementation
>
>
>
> We are more than happy to collaborate on a call to make better progress
> on this enhancement. Please let us know.
>
> Thank you,
>
> Pavan
>
> On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh 
> wrote:
>
>>
>> Hi Bhuwan et al,
>>
>> Thank you for passing on the DataBricks Structured Streaming team's
>> review of the SPIP document. FYI, I work closely with Pawan and other
>> members to help deliver this piece of work. We appreciate your insights,
>> especially regarding the cost savings potential from the PoC.
>>
>> Pavan already furnished you with some additional info. Your team's point
>> about the SPIP currently addressing a specific use case (single streaming
>> query with Processing Time trigger) is well-taken. We agree that
>> maintaining simplicity is key, particularly as we explore more general
>> resource allocation mechanisms in the future. To address the concerns and
>> foster open discussion, The DataBricks team are invited to directly add
>> their comments and suggestions to the Jira itself
>>
>> [SPARK-24815] Structured Streaming should support dynamic allocation -
>> ASF JIRA (apache.org)
>> 
>> This will ensure everyone involved can benefit from your team's expertise
>> and facilitate further collaboration.
>>
>> Thanks
>>
>> Mich Talebzadeh,
>> Dad | Technologist | Solutions Architect | Engineer
>> London
>> United Kingdom
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>> 
>>
>>
>>
>> *Disclaimer:* The information provided is correct to the best of my
>> knowledge but of course cannot be guaranteed . It is essential to note
>> that, as with any advice, quote "one test result is worth one-thousand
>> expert opinions (Werner
>> Von
>> Braun
>> 
>> )".
>>
>>
>> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi
>>  wrote:
>>
>>> Thanks Bhuwan and rest of the databricks team for the reviews,
>>>
>>> I appreciate your reviews, was very helpful in evaluating a few options
>>> that were overlooked earlier (especially about mixed spark apps running on
>>> notebooks). Regarding the use-cases, It could handle multiple streaming
>>> queries provided that they are run on the same trigger interval processing
>>> time (very similar to how current batch dra is set up)..but I felt like it
>>> would be beneficial if we separate out streaming queries when setting up
>>> production pipelines.
>>>
>>> Regarding the implementation, here is the draft PR
>>> https://github.com/apache/spark/pull/42352
>>> 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Hi Bhuwan,

I hope the team got a chance to review the draft PR, looking for some
comments to see if the plan looks alright?. I have updated the document
about the risks
.(also
mentioned below). Please confirm if it looks alright?

*Spark application type*

*auto-scaling capability*

*with New auto-scaling capability*

Spark Batch job

Works with current DRA

No - change

Streaming query without trigger interval

No implementation

Can work with this implementation - (have to set certain scale back configs
based on previous usage pattern) - maybe automate with future work?

Spark Streaming query with Trigger interval

No implementation

With this implementation

Spark Streaming query with one-time micro batch

Works with current DRA

No - change

Spark Streaming query with

Availablenow micro batch

Works with current DRA

No - change

Batch + Streaming query (

default/

triggger-interval/

once/

availablenow modes), other notebook use cases.

No implementation

No implementation



We are more than happy to collaborate on a call to make better progress
on this enhancement. Please let us know.

Thank you,

Pavan

On Fri, Mar 1, 2024 at 12:26 PM Mich Talebzadeh 
wrote:

>
> Hi Bhuwan et al,
>
> Thank you for passing on the DataBricks Structured Streaming team's review
> of the SPIP document. FYI, I work closely with Pawan and other members to
> help deliver this piece of work. We appreciate your insights, especially
> regarding the cost savings potential from the PoC.
>
> Pavan already furnished you with some additional info. Your team's point
> about the SPIP currently addressing a specific use case (single streaming
> query with Processing Time trigger) is well-taken. We agree that
> maintaining simplicity is key, particularly as we explore more general
> resource allocation mechanisms in the future. To address the concerns and
> foster open discussion, The DataBricks team are invited to directly add
> their comments and suggestions to the Jira itself
>
> [SPARK-24815] Structured Streaming should support dynamic allocation - ASF
> JIRA (apache.org)
> 
> This will ensure everyone involved can benefit from your team's expertise
> and facilitate further collaboration.
>
> Thanks
>
> Mich Talebzadeh,
> Dad | Technologist | Solutions Architect | Engineer
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
> 
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner
> Von
> Braun
> 
> )".
>
>
> On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi
>  wrote:
>
>> Thanks Bhuwan and rest of the databricks team for the reviews,
>>
>> I appreciate your reviews, was very helpful in evaluating a few options
>> that were overlooked earlier (especially about mixed spark apps running on
>> notebooks). Regarding the use-cases, It could handle multiple streaming
>> queries provided that they are run on the same trigger interval processing
>> time (very similar to how current batch dra is set up)..but I felt like it
>> would be beneficial if we separate out streaming queries when setting up
>> production pipelines.
>>
>> Regarding the implementation, here is the draft PR
>> https://github.com/apache/spark/pull/42352
>> .
>> (already mentioned in ticket SPARK-24815
>> 
>> )
>>
>> I have built it on top of the current Dynamic resource allocation (DRA)
>> algorithm
>> 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Mich Talebzadeh]

Hi Bhuwan et al,

Thank you for passing on the DataBricks Structured Streaming team's review
of the SPIP document. FYI, I work closely with Pawan and other members to
help deliver this piece of work. We appreciate your insights, especially
regarding the cost savings potential from the PoC.

Pavan already furnished you with some additional info. Your team's point
about the SPIP currently addressing a specific use case (single streaming
query with Processing Time trigger) is well-taken. We agree that
maintaining simplicity is key, particularly as we explore more general
resource allocation mechanisms in the future. To address the concerns and
foster open discussion, The DataBricks team are invited to directly add
their comments and suggestions to the Jira itself

[SPARK-24815] Structured Streaming should support dynamic allocation - ASF
JIRA (apache.org) 
This will ensure everyone involved can benefit from your team's expertise
and facilitate further collaboration.

Thanks

Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner  Von
Braun )".


On Fri, 1 Mar 2024 at 19:59, Pavan Kotikalapudi
 wrote:

> Thanks Bhuwan and rest of the databricks team for the reviews,
>
> I appreciate your reviews, was very helpful in evaluating a few options
> that were overlooked earlier (especially about mixed spark apps running on
> notebooks). Regarding the use-cases, It could handle multiple streaming
> queries provided that they are run on the same trigger interval processing
> time (very similar to how current batch dra is set up)..but I felt like it
> would be beneficial if we separate out streaming queries when setting up
> production pipelines.
>
> Regarding the implementation, here is the draft PR
> https://github.com/apache/spark/pull/42352. (already mentioned in ticket
> SPARK-24815 )
>
> I have built it on top of the current Dynamic resource allocation (DRA)
> algorithm
> 
> .
> While current DRA is catered towards batch jobs. This implementation just
> makes few changes to that algorithm to
> - do gradual scale-back. The remove-policy still applies (uses 2 old
> configs we currently have), but we now remove few executors per round of
> evaluation ( I have added 2 configs to tune that)
> - The scale-out process also still uses the same request policy (same uses
> 2 old configs we currently have).
> - while we are using the old configs in the both scale-out/back, the
> difference is that we are now giving configs to them based on the trigger
> interval as our north star.
>
> This implementation is just changes in 2 files to make it work. I have
> made the changes minimal/limited to just the core module of the spark repo.
> 1) to make sure it is applied on primitives of task, stage, job which the
> current dra is already doing. (This will enable us to think about other
> cases like  default and continuous mode can still work provided we have a
> target processing time range we want to achieve)
> 2) We are reusing ExecutorAllocationClient, ExecutorMonitor and listeners
> which are already well tested and working well for batch job use case.
>
> We internally (in the company) have also added helpers so that we have
> less configs to tune. I can contribute that as well, if it makes the dev
> experience better.
>
> Feel free to review the PR, when we decide the direction is alright I will
> start adding the tests as well.
>
> On a side note.  Maybe we should consider some future work to have DRA
> algo per query (batch, streaming queries, mixed etc) rather than per spark
> context.
>
> Thank you,
>
> Pavan
>
>
> On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni
>  wrote:
>
>> Hi Pavan,
>>
>> I am from the DataBricks Structured Streaming team, and we did a review
>> of the SPIP internally. Wanted to pass on the points discussed in the
>> meeting.
>>
>> Thanks for putting together the SPIP document. It's useful to have
>> dynamic resource allocation for Streaming queries, and it's exciting to see
>> the cost saving numbers from your PoC. However, in general we discovered
>> that the SPIP addresses a very particular use-case (single streaming query
>> in Spark cluster with Processing time Trigger). Keeping that in mind, it's
>> useful to make sure that the implementation is simple. Can you please share
>> your PoC implementation to understand 

Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Nivedita VY]

+1

Nivi

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Thanks Bhuwan and rest of the databricks team for the reviews,

I appreciate your reviews, was very helpful in evaluating a few options
that were overlooked earlier (especially about mixed spark apps running on
notebooks). Regarding the use-cases, It could handle multiple streaming
queries provided that they are run on the same trigger interval processing
time (very similar to how current batch dra is set up)..but I felt like it
would be beneficial if we separate out streaming queries when setting up
production pipelines.

Regarding the implementation, here is the draft PR
https://github.com/apache/spark/pull/42352. (already mentioned in ticket
SPARK-24815 )

I have built it on top of the current Dynamic resource allocation (DRA)
algorithm

.
While current DRA is catered towards batch jobs. This implementation just
makes few changes to that algorithm to
- do gradual scale-back. The remove-policy still applies (uses 2 old
configs we currently have), but we now remove few executors per round of
evaluation ( I have added 2 configs to tune that)
- The scale-out process also still uses the same request policy (same uses
2 old configs we currently have).
- while we are using the old configs in the both scale-out/back, the
difference is that we are now giving configs to them based on the trigger
interval as our north star.

This implementation is just changes in 2 files to make it work. I have made
the changes minimal/limited to just the core module of the spark repo.
1) to make sure it is applied on primitives of task, stage, job which the
current dra is already doing. (This will enable us to think about other
cases like  default and continuous mode can still work provided we have a
target processing time range we want to achieve)
2) We are reusing ExecutorAllocationClient, ExecutorMonitor and listeners
which are already well tested and working well for batch job use case.

We internally (in the company) have also added helpers so that we have less
configs to tune. I can contribute that as well, if it makes the dev
experience better.

Feel free to review the PR, when we decide the direction is alright I will
start adding the tests as well.

On a side note.  Maybe we should consider some future work to have DRA algo
per query (batch, streaming queries, mixed etc) rather than per spark
context.

Thank you,

Pavan


On Fri, Mar 1, 2024 at 9:06 AM Bhuwan Sahni
 wrote:

> Hi Pavan,
>
> I am from the DataBricks Structured Streaming team, and we did a review of
> the SPIP internally. Wanted to pass on the points discussed in the meeting.
>
>
> Thanks for putting together the SPIP document. It's useful to have dynamic
> resource allocation for Streaming queries, and it's exciting to see the
> cost saving numbers from your PoC. However, in general we discovered that
> the SPIP addresses a very particular use-case (single streaming query in
> Spark cluster with Processing time Trigger). Keeping that in mind, it's
> useful to make sure that the implementation is simple. Can you please share
> your PoC implementation to understand the code complexity. This would help
> us to ensure that dynamic resource allocation mechanism for Streaming
> queries does not become complicated in the future (if more resource
> allocation mechanisms are added to address other use-cases).
>
> Thanks,
> Bhuwan
>
> On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> Hi Pavan and those who kindly voted for this SPIP
>>
>> Great to have 6+ votes and no -1 and 0. The so-called mass volume is
>> there. The rest is admin matter and how to drive the project forward and
>> yes there is more than one way of skinning the cat. I think we need some
>> flexibility in the rules given the dwindling (IMO) number of comitters who
>> are willing or actively participating. For example, on a similar matter I
>> approached Codi Koeninger who was one of the founders of Spark Streaming,
>> to shepherd a project almost a year back. Sadly he is no longer active and
>> quotes "I haven't been involved lately and would be missing a lot of
>> context." So we need to improvise and see how best we can drive this and
>> similar ones. We wait a short while for a response otherwise I am happy to
>> give a hand if needed and work with you guys to drive this. It is something
>> worthwhile.
>>
>> HTH
>>
>> T
>> Mich Talebzadeh,
>> Dad | Technologist | Solutions Architect | Engineer
>> London
>> United Kingdom
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>> 

RE: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Nivedita VY]

+1

Nivi

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Bhuwan Sahni]

Hi Pavan,

I am from the DataBricks Structured Streaming team, and we did a review of
the SPIP internally. Wanted to pass on the points discussed in the meeting.


Thanks for putting together the SPIP document. It's useful to have dynamic
resource allocation for Streaming queries, and it's exciting to see the
cost saving numbers from your PoC. However, in general we discovered that
the SPIP addresses a very particular use-case (single streaming query in
Spark cluster with Processing time Trigger). Keeping that in mind, it's
useful to make sure that the implementation is simple. Can you please share
your PoC implementation to understand the code complexity. This would help
us to ensure that dynamic resource allocation mechanism for Streaming
queries does not become complicated in the future (if more resource
allocation mechanisms are added to address other use-cases).

Thanks,
Bhuwan

On Fri, Feb 23, 2024 at 11:01 AM Mich Talebzadeh 
wrote:

> Hi Pavan and those who kindly voted for this SPIP
>
> Great to have 6+ votes and no -1 and 0. The so-called mass volume is
> there. The rest is admin matter and how to drive the project forward and
> yes there is more than one way of skinning the cat. I think we need some
> flexibility in the rules given the dwindling (IMO) number of comitters who
> are willing or actively participating. For example, on a similar matter I
> approached Codi Koeninger who was one of the founders of Spark Streaming,
> to shepherd a project almost a year back. Sadly he is no longer active and
> quotes "I haven't been involved lately and would be missing a lot of
> context." So we need to improvise and see how best we can drive this and
> similar ones. We wait a short while for a response otherwise I am happy to
> give a hand if needed and work with you guys to drive this. It is something
> worthwhile.
>
> HTH
>
> T
> Mich Talebzadeh,
> Dad | Technologist | Solutions Architect | Engineer
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner  Von
> Braun )".
>
>
> On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi
>  wrote:
>
>> Thanks for the pointers Mich, will wait for Jungtaek Lee or any other PMC
>> members to respond.
>>
>> aggregating upvotes to this email thread
>>
>> +6
>> Mich Talebzadeh
>> Adam Hobbs
>> Pavan Kotikalapudi
>> Krystal Mitchell
>> Sona Torosyan
>> Aaron Kern
>>
>> Thank you,
>>
>> Pavan
>>
>> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> please check this doc
>>>
>>> Spark Project Improvement Proposals (SPIP) | Apache Spark
>>> 
>>>
>>> and specifically the below extract
>>>
>>> Discussing an SPIP
>>>
>>> All discussion of an SPIP should take place in a public forum,
>>> preferably the discussion attached to the Jira. Any discussions that happen
>>> offline should be made available online for the public via meeting notes
>>> summarizing the discussions.(done)
>>>
>>> During this discussion, one or more shepherds should be identified among
>>> PMC members. (outstanding)
>>>
>>> Once the discussion settles, the shepherd(s) should call for a vote on
>>> the SPIP moving forward on the dev@ list. The vote should be open for
>>> at least 72 hours and follows the typical Apache vote process and passes
>>> upon consensus (at least 3 +1 votes from PMC members and no -1 votes from
>>> PMC members). dev@ should be notified of the vote result.
>>>
>>> If there does not exist at least one PMC member that is committed to
>>> shepherding the change within a month, the SPIP is rejected.
>>>
>>> If a committer does not think a SPIP aligns with long-term project
>>> goals, or is not practical at the point of proposal, the committer should
>>> -1 the SPIP explicitly and give technical justifications.
>>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee can
>>> kindly help the process
>>>
>>> cheers
>>>
>>> Mich Talebzadeh,
>>> Dad | Technologist | Solutions Architect | Engineer
>>> London
>>> United Kingdom
>>>
>>>
>>>view my Linkedin profile
>>> 
>>>
>>>
>>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>> 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Mich Talebzadeh]

Hi Pavan and those who kindly voted for this SPIP

Great to have 6+ votes and no -1 and 0. The so-called mass volume is there.
The rest is admin matter and how to drive the project forward and yes there
is more than one way of skinning the cat. I think we need some flexibility
in the rules given the dwindling (IMO) number of comitters who are willing
or actively participating. For example, on a similar matter I
approached Codi Koeninger who was one of the founders of Spark Streaming,
to shepherd a project almost a year back. Sadly he is no longer active and
quotes "I haven't been involved lately and would be missing a lot of
context." So we need to improvise and see how best we can drive this and
similar ones. We wait a short while for a response otherwise I am happy to
give a hand if needed and work with you guys to drive this. It is something
worthwhile.

HTH

T
Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner  Von
Braun )".


On Fri, 23 Feb 2024 at 17:41, Pavan Kotikalapudi
 wrote:

> Thanks for the pointers Mich, will wait for Jungtaek Lee or any other PMC
> members to respond.
>
> aggregating upvotes to this email thread
>
> +6
> Mich Talebzadeh
> Adam Hobbs
> Pavan Kotikalapudi
> Krystal Mitchell
> Sona Torosyan
> Aaron Kern
>
> Thank you,
>
> Pavan
>
> On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh 
> wrote:
>
>> Hi,
>>
>> please check this doc
>>
>> Spark Project Improvement Proposals (SPIP) | Apache Spark
>> 
>>
>> and specifically the below extract
>>
>> Discussing an SPIP
>>
>> All discussion of an SPIP should take place in a public forum, preferably
>> the discussion attached to the Jira. Any discussions that happen offline
>> should be made available online for the public via meeting notes
>> summarizing the discussions.(done)
>>
>> During this discussion, one or more shepherds should be identified among
>> PMC members. (outstanding)
>>
>> Once the discussion settles, the shepherd(s) should call for a vote on
>> the SPIP moving forward on the dev@ list. The vote should be open for at
>> least 72 hours and follows the typical Apache vote process and passes upon
>> consensus (at least 3 +1 votes from PMC members and no -1 votes from PMC
>> members). dev@ should be notified of the vote result.
>>
>> If there does not exist at least one PMC member that is committed to
>> shepherding the change within a month, the SPIP is rejected.
>>
>> If a committer does not think a SPIP aligns with long-term project goals,
>> or is not practical at the point of proposal, the committer should -1 the
>> SPIP explicitly and give technical justifications.
>> OK a shepherd from PMC members is required. Maybe Jungtaek Lee can kindly
>> help the process
>>
>> cheers
>>
>> Mich Talebzadeh,
>> Dad | Technologist | Solutions Architect | Engineer
>> London
>> United Kingdom
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>> 
>>
>>
>>
>> *Disclaimer:* The information provided is correct to the best of my
>> knowledge but of course cannot be guaranteed . It is essential to note
>> that, as with any advice, quote "one test result is worth one-thousand
>> expert opinions (Werner
>> Von
>> Braun
>> 
>> )".
>>
>>
>> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi
>>  wrote:
>>
>>> Hi Mich,
>>>
>>> We have
>>>
>>> five  +1s till now.
>>>
>>> Mich Talebzadeh
>>> Adam Hobbs
>>> Pavan Kotikalapudi
>>> Krystal Mitchell
>>> Sona Torosyan
>>> (few more in github pr)
>>> +0: None
>>>
>>> -1: None
>>>
>>> Does it pass the required condition as approved?
>>>
>>>

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Thanks for the pointers Mich, will wait for Jungtaek Lee or any other PMC
members to respond.

aggregating upvotes to this email thread

+6
Mich Talebzadeh
Adam Hobbs
Pavan Kotikalapudi
Krystal Mitchell
Sona Torosyan
Aaron Kern

Thank you,

Pavan

On Thu, Feb 22, 2024 at 3:07 PM Mich Talebzadeh 
wrote:

> Hi,
>
> please check this doc
>
> Spark Project Improvement Proposals (SPIP) | Apache Spark
> 
>
> and specifically the below extract
>
> Discussing an SPIP
>
> All discussion of an SPIP should take place in a public forum, preferably
> the discussion attached to the Jira. Any discussions that happen offline
> should be made available online for the public via meeting notes
> summarizing the discussions.(done)
>
> During this discussion, one or more shepherds should be identified among
> PMC members. (outstanding)
>
> Once the discussion settles, the shepherd(s) should call for a vote on the
> SPIP moving forward on the dev@ list. The vote should be open for at
> least 72 hours and follows the typical Apache vote process and passes upon
> consensus (at least 3 +1 votes from PMC members and no -1 votes from PMC
> members). dev@ should be notified of the vote result.
>
> If there does not exist at least one PMC member that is committed to
> shepherding the change within a month, the SPIP is rejected.
>
> If a committer does not think a SPIP aligns with long-term project goals,
> or is not practical at the point of proposal, the committer should -1 the
> SPIP explicitly and give technical justifications.
> OK a shepherd from PMC members is required. Maybe Jungtaek Lee can kindly
> help the process
>
> cheers
>
> Mich Talebzadeh,
> Dad | Technologist | Solutions Architect | Engineer
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
> 
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner
> Von
> Braun
> 
> )".
>
>
> On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi
>  wrote:
>
>> Hi Mich,
>>
>> We have
>>
>> five  +1s till now.
>>
>> Mich Talebzadeh
>> Adam Hobbs
>> Pavan Kotikalapudi
>> Krystal Mitchell
>> Sona Torosyan
>> (few more in github pr)
>> +0: None
>>
>> -1: None
>>
>> Does it pass the required condition as approved?
>>
>>
>> Not sure of that though, nothing about minimum required is mentioned in
>> the past emails.
>>
>> I would request spark PMC members or any others who have done this in the
>> past to understand the process better.
>>
>> Thank you,
>>
>> Pavan
>>
>> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Hi Pavan,
>>>
>>> Do you have a list of votes for this feature by any chance? Does it pass
>>> the required condition as approved?
>>>
>>> HTH
>>>
>>> Mich Talebzadeh,
>>> Dad | Technologist | Solutions Architect | Engineer
>>> London
>>> United Kingdom
>>>
>>>
>>>view my Linkedin profile
>>> 
>>>
>>>
>>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>> 
>>>
>>>
>>>
>>> *Disclaimer:* The information provided is correct to the best of my
>>> knowledge but of course cannot be guaranteed . It is essential to note
>>> that, as with any advice, quote "one test result is worth one-thousand
>>> expert opinions (Werner
>>> Von
>>> Braun
>>> 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Mich Talebzadeh]

+1 for me

Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner  Von
Braun )".


On Fri, 23 Feb 2024 at 16:05, Aaron Kern  wrote:

> +1
>

Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Aaron Kern]

+1

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Mich Talebzadeh]

Hi,

please check this doc

Spark Project Improvement Proposals (SPIP) | Apache Spark


and specifically the below extract

Discussing an SPIP

All discussion of an SPIP should take place in a public forum, preferably
the discussion attached to the Jira. Any discussions that happen offline
should be made available online for the public via meeting notes
summarizing the discussions.(done)

During this discussion, one or more shepherds should be identified among
PMC members. (outstanding)

Once the discussion settles, the shepherd(s) should call for a vote on the
SPIP moving forward on the dev@ list. The vote should be open for at least
72 hours and follows the typical Apache vote process and passes upon
consensus (at least 3 +1 votes from PMC members and no -1 votes from PMC
members). dev@ should be notified of the vote result.

If there does not exist at least one PMC member that is committed to
shepherding the change within a month, the SPIP is rejected.

If a committer does not think a SPIP aligns with long-term project goals,
or is not practical at the point of proposal, the committer should -1 the
SPIP explicitly and give technical justifications.
OK a shepherd from PMC members is required. Maybe Jungtaek Lee can kindly
help the process

cheers

Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner  Von
Braun )".


On Thu, 22 Feb 2024 at 21:52, Pavan Kotikalapudi
 wrote:

> Hi Mich,
>
> We have
>
> five  +1s till now.
>
> Mich Talebzadeh
> Adam Hobbs
> Pavan Kotikalapudi
> Krystal Mitchell
> Sona Torosyan
> (few more in github pr)
> +0: None
>
> -1: None
>
> Does it pass the required condition as approved?
>
>
> Not sure of that though, nothing about minimum required is mentioned in
> the past emails.
>
> I would request spark PMC members or any others who have done this in the
> past to understand the process better.
>
> Thank you,
>
> Pavan
>
> On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh 
> wrote:
>
>> Hi Pavan,
>>
>> Do you have a list of votes for this feature by any chance? Does it pass
>> the required condition as approved?
>>
>> HTH
>>
>> Mich Talebzadeh,
>> Dad | Technologist | Solutions Architect | Engineer
>> London
>> United Kingdom
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>> 
>>
>>
>>
>> *Disclaimer:* The information provided is correct to the best of my
>> knowledge but of course cannot be guaranteed . It is essential to note
>> that, as with any advice, quote "one test result is worth one-thousand
>> expert opinions (Werner
>> Von
>> Braun
>> 
>> )".
>>
>>
>> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi
>>  wrote:
>>
>>> Yes. The PR was closed due to inactivity by github actions..
>>>
>>> The msg
>>> 
>>>  also
>>> says
>>>
>>> > If you'd like to revive this PR, please reopen it and ask a committer
>>> to remove the Stale tag!
>>>
>>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh <
>>> mich.talebza...@gmail.com> wrote:
>>>
 I can see it was closed. Was it because of inactivity?


 Mich Talebzadeh,
 Dad | Technologist | Solutions Architect | Engineer
 London
 United Kingdom


view my Linkedin profile
 


  https://en.everybodywiki.com/Mich_Talebzadeh
 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Hi Mich,

We have

five  +1s till now.

Mich Talebzadeh
Adam Hobbs
Pavan Kotikalapudi
Krystal Mitchell
Sona Torosyan
(few more in github pr)
+0: None

-1: None

Does it pass the required condition as approved?


Not sure of that though, nothing about minimum required is mentioned in the
past emails.

I would request spark PMC members or any others who have done this in the
past to understand the process better.

Thank you,

Pavan

On Thu, Feb 22, 2024 at 3:20 AM Mich Talebzadeh 
wrote:

> Hi Pavan,
>
> Do you have a list of votes for this feature by any chance? Does it pass
> the required condition as approved?
>
> HTH
>
> Mich Talebzadeh,
> Dad | Technologist | Solutions Architect | Engineer
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
> 
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner
> Von
> Braun
> 
> )".
>
>
> On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi
>  wrote:
>
>> Yes. The PR was closed due to inactivity by github actions..
>>
>> The msg
>> 
>>  also
>> says
>>
>> > If you'd like to revive this PR, please reopen it and ask a committer
>> to remove the Stale tag!
>>
>> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> I can see it was closed. Was it because of inactivity?
>>>
>>>
>>> Mich Talebzadeh,
>>> Dad | Technologist | Solutions Architect | Engineer
>>> London
>>> United Kingdom
>>>
>>>
>>>view my Linkedin profile
>>> 
>>>
>>>
>>>  https://en.everybodywiki.com/Mich_Talebzadeh
>>> 
>>>
>>>
>>>
>>> *Disclaimer:* The information provided is correct to the best of my
>>> knowledge but of course cannot be guaranteed . It is essential to note
>>> that, as with any advice, quote "one test result is worth one-thousand
>>> expert opinions (Werner
>>> Von
>>> Braun
>>> 
>>> )".
>>>
>>>
>>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi
>>>  wrote:
>>>
 Hi Spark PMC members,

 I think we have few upvotes for this effort here and more people are
 showing interest (see  PR comments
 
 .)

 Is anyone interested in mentoring and reviewing this effort?

 Also can the repository admin/owner re-open the PR?  ( I guess people
 only with admin access to the repository can do that).

 Thank you,

 Pavan

 On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell
  wrote:

> +1
>
> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote:
> > Thanks for proposing and voting for the feature Mich.
> >
> > adding some references to the thread.
> >
> >- Jira ticket - SPARK-24815
> >
> 

Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Sona Torosyan]

+1

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Mich Talebzadeh]

Hi Pavan,

Do you have a list of votes for this feature by any chance? Does it pass
the required condition as approved?

HTH

Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner  Von
Braun )".


On Thu, 22 Feb 2024 at 10:04, Pavan Kotikalapudi
 wrote:

> Yes. The PR was closed due to inactivity by github actions..
>
> The msg
>  also
> says
>
> > If you'd like to revive this PR, please reopen it and ask a committer to
> remove the Stale tag!
>
> On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh 
> wrote:
>
>> I can see it was closed. Was it because of inactivity?
>>
>>
>> Mich Talebzadeh,
>> Dad | Technologist | Solutions Architect | Engineer
>> London
>> United Kingdom
>>
>>
>>view my Linkedin profile
>> 
>>
>>
>>  https://en.everybodywiki.com/Mich_Talebzadeh
>> 
>>
>>
>>
>> *Disclaimer:* The information provided is correct to the best of my
>> knowledge but of course cannot be guaranteed . It is essential to note
>> that, as with any advice, quote "one test result is worth one-thousand
>> expert opinions (Werner
>> Von
>> Braun
>> 
>> )".
>>
>>
>> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi
>>  wrote:
>>
>>> Hi Spark PMC members,
>>>
>>> I think we have few upvotes for this effort here and more people are
>>> showing interest (see  PR comments
>>> 
>>> .)
>>>
>>> Is anyone interested in mentoring and reviewing this effort?
>>>
>>> Also can the repository admin/owner re-open the PR?  ( I guess people
>>> only with admin access to the repository can do that).
>>>
>>> Thank you,
>>>
>>> Pavan
>>>
>>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell
>>>  wrote:
>>>
 +1

 On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote:
 > Thanks for proposing and voting for the feature Mich.
 >
 > adding some references to the thread.
 >
 >- Jira ticket - SPARK-24815
 >
 
 >- Design Doc
 ><
 https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing>
 
 >
 >- discussion thread
 >
 
 >- PR with initial implementation -
 >https://github.com/apache/spark/pull/42352
 
 >
 > Please vote with:
 >
 > [ ] +1: Accept the proposal and start with the development.
 > [ ] +0
 > [ ] -1: I don’t think this is a good idea because …
 >
 > Thank you,
 >
 > Pavan
 >
 > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh 
 > wrote:
 >
 > >
 > > +1 for me  (non binding)
 > >
 > >
 > >
 > > 

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Yes. The PR was closed due to inactivity by github actions..

The msg 
also
says

> If you'd like to revive this PR, please reopen it and ask a committer to
remove the Stale tag!

On Thu, Feb 22, 2024 at 1:09 AM Mich Talebzadeh 
wrote:

> I can see it was closed. Was it because of inactivity?
>
>
> Mich Talebzadeh,
> Dad | Technologist | Solutions Architect | Engineer
> London
> United Kingdom
>
>
>view my Linkedin profile
> 
>
>
>  https://en.everybodywiki.com/Mich_Talebzadeh
> 
>
>
>
> *Disclaimer:* The information provided is correct to the best of my
> knowledge but of course cannot be guaranteed . It is essential to note
> that, as with any advice, quote "one test result is worth one-thousand
> expert opinions (Werner
> Von
> Braun
> 
> )".
>
>
> On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi
>  wrote:
>
>> Hi Spark PMC members,
>>
>> I think we have few upvotes for this effort here and more people are
>> showing interest (see  PR comments
>> 
>> .)
>>
>> Is anyone interested in mentoring and reviewing this effort?
>>
>> Also can the repository admin/owner re-open the PR?  ( I guess people
>> only with admin access to the repository can do that).
>>
>> Thank you,
>>
>> Pavan
>>
>> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell
>>  wrote:
>>
>>> +1
>>>
>>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote:
>>> > Thanks for proposing and voting for the feature Mich.
>>> >
>>> > adding some references to the thread.
>>> >
>>> >- Jira ticket - SPARK-24815
>>> >
>>> 
>>> >- Design Doc
>>> ><
>>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing>
>>> 
>>> >
>>> >- discussion thread
>>> >
>>> 
>>> >- PR with initial implementation -
>>> >https://github.com/apache/spark/pull/42352
>>> 
>>> >
>>> > Please vote with:
>>> >
>>> > [ ] +1: Accept the proposal and start with the development.
>>> > [ ] +0
>>> > [ ] -1: I don’t think this is a good idea because …
>>> >
>>> > Thank you,
>>> >
>>> > Pavan
>>> >
>>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh 
>>> > wrote:
>>> >
>>> > >
>>> > > +1 for me  (non binding)
>>> > >
>>> > >
>>> > >
>>> > > *Disclaimer:* Use it at your own risk. Any and all responsibility
>>> for any
>>> > > loss, damage or destruction of data or any other property which may
>>> arise
>>> > > from relying on this email's technical content is explicitly
>>> disclaimed.
>>> > > The author will in no case be liable for any monetary damages
>>> arising from
>>> > > such loss, damage or destruction.
>>> > >
>>> > >
>>> > >
>>> >
>>>
>>

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Mich Talebzadeh]

I can see it was closed. Was it because of inactivity?


Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom


   view my Linkedin profile



 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* The information provided is correct to the best of my
knowledge but of course cannot be guaranteed . It is essential to note
that, as with any advice, quote "one test result is worth one-thousand
expert opinions (Werner  Von
Braun )".


On Thu, 22 Feb 2024 at 06:58, Pavan Kotikalapudi
 wrote:

> Hi Spark PMC members,
>
> I think we have few upvotes for this effort here and more people are
> showing interest (see  PR comments
> .)
>
> Is anyone interested in mentoring and reviewing this effort?
>
> Also can the repository admin/owner re-open the PR?  ( I guess people only
> with admin access to the repository can do that).
>
> Thank you,
>
> Pavan
>
> On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell
>  wrote:
>
>> +1
>>
>> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote:
>> > Thanks for proposing and voting for the feature Mich.
>> >
>> > adding some references to the thread.
>> >
>> >- Jira ticket - SPARK-24815
>> >
>> 
>> >- Design Doc
>> ><
>> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing>
>> 
>> >
>> >- discussion thread
>> >
>> 
>> >- PR with initial implementation -
>> >https://github.com/apache/spark/pull/42352
>> 
>> >
>> > Please vote with:
>> >
>> > [ ] +1: Accept the proposal and start with the development.
>> > [ ] +0
>> > [ ] -1: I don’t think this is a good idea because …
>> >
>> > Thank you,
>> >
>> > Pavan
>> >
>> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh 
>> > wrote:
>> >
>> > >
>> > > +1 for me  (non binding)
>> > >
>> > >
>> > >
>> > > *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any
>> > > loss, damage or destruction of data or any other property which may
>> arise
>> > > from relying on this email's technical content is explicitly
>> disclaimed.
>> > > The author will in no case be liable for any monetary damages arising
>> from
>> > > such loss, damage or destruction.
>> > >
>> > >
>> > >
>> >
>>
>

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Hi Spark PMC members,

I think we have few upvotes for this effort here and more people are
showing interest (see  PR comments
.)

Is anyone interested in mentoring and reviewing this effort?

Also can the repository admin/owner re-open the PR?  ( I guess people only
with admin access to the repository can do that).

Thank you,

Pavan

On Tue, Feb 20, 2024 at 2:08 PM Krystal Mitchell 
wrote:

> +1
>
> On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote:
> > Thanks for proposing and voting for the feature Mich.
> >
> > adding some references to the thread.
> >
> >- Jira ticket - SPARK-24815
> >
> 
> >- Design Doc
> ><
> https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing>
> 
> >
> >- discussion thread
> >
> 
> >- PR with initial implementation -
> >https://github.com/apache/spark/pull/42352
> 
> >
> > Please vote with:
> >
> > [ ] +1: Accept the proposal and start with the development.
> > [ ] +0
> > [ ] -1: I don’t think this is a good idea because …
> >
> > Thank you,
> >
> > Pavan
> >
> > On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh 
> > wrote:
> >
> > >
> > > +1 for me  (non binding)
> > >
> > >
> > >
> > > *Disclaimer:* Use it at your own risk. Any and all responsibility for
> any
> > > loss, damage or destruction of data or any other property which may
> arise
> > > from relying on this email's technical content is explicitly
> disclaimed.
> > > The author will in no case be liable for any monetary damages arising
> from
> > > such loss, damage or destruction.
> > >
> > >
> > >
> >
>

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Krystal Mitchell]

+1

On 2024/01/17 17:49:32 Pavan Kotikalapudi wrote:
> Thanks for proposing and voting for the feature Mich.
>
> adding some references to the thread.
>
>- Jira ticket - SPARK-24815
>
>- Design Doc
>
> 
>
>- discussion thread
>
>- PR with initial implementation -
>https://github.com/apache/spark/pull/42352
>
> Please vote with:
>
> [ ] +1: Accept the proposal and start with the development.
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>
> Thank you,
>
> Pavan
>
> On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh 
> wrote:
>
> >
> > +1 for me  (non binding)
> >
> >
> >
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising from
> > such loss, damage or destruction.
> >
> >
> >
>

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Mich Talebzadeh]

Everyone's vote matters whether they are PMC or not. There is no monopoly
here

HTH

Mich Talebzadeh,
Dad | Technologist | Solutions Architect | Engineer
London
United Kingdom


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>


 https://en.everybodywiki.com/Mich_Talebzadeh



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Fri, 19 Jan 2024 at 11:55, Pavan Kotikalapudi
 wrote:

> +1
> If my vote counts.
>
> Does only spark PMC votes count?
>
> Thanks,
>
> Pavan
>
> On Thu, Jan 18, 2024 at 3:19 AM Adam Hobbs
>  wrote:
>
>> +1
>> --
>> *From:* Pavan Kotikalapudi 
>> *Sent:* Thursday, January 18, 2024 4:19:32 AM
>> *To:* Spark dev list 
>> *Subject:* Re: Vote on Dynamic resource allocation for structured
>> streaming [SPARK-24815]
>>
>>
>> CAUTION: This email originated from outside of the organisation. Do not
>> click links or open attachments unless you recognise the sender's full
>> email address and know the content is safe.
>>
>> Thanks for proposing and voting for the feature Mich.
>>
>> adding some references to the thread.
>>
>>- Jira ticket - SPARK-24815
>>
>> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCn0_Z1lI$>
>>- Design Doc
>>
>> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCuAyVt8y$>
>>
>>- discussion thread
>>
>> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCqHoXny8$>
>>- PR with initial implementation -
>>https://github.com/apache/spark/pull/42352
>>
>> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCisLiWaP$>
>>
>> Please vote with:
>>
>> [ ] +1: Accept the proposal and start with the development.
>> [ ] +0
>> [ ] -1: I don’t think this is a good idea because …
>>
>> Thank you,
>>
>> Pavan
>>
>> On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>
>> +1 for me  (non binding)
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> 
>>
>> This communication is intended only for use of the addressee and may
>> contain legally privileged and confidential information.
>> If you are not the addressee or intended recipient, you are notified that
>> any dissemination, copying or use of any of the information is unauthorised.
>>
>> The legal privilege and confidentiality attached to this e-mail is not
>> waived, lost or destroyed by reason of a mistaken delivery to you.
>> If you have received this message in error, we would appreciate an
>> immediate notification via e-mail to contac...@bendigoadelaide.com.au or
>> by phoning 1300 BENDIGO (1300 236 344), and ask that the e-mail be
>> permanently deleted from your system.
>>
>> Bendigo and Adelaide Bank Limited ABN 11 068 049 178
>>
>>
>> 
>>
>

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

+1
If my vote counts.

Does only spark PMC votes count?

Thanks,

Pavan

On Thu, Jan 18, 2024 at 3:19 AM Adam Hobbs
 wrote:

> +1
> --
> *From:* Pavan Kotikalapudi 
> *Sent:* Thursday, January 18, 2024 4:19:32 AM
> *To:* Spark dev list 
> *Subject:* Re: Vote on Dynamic resource allocation for structured
> streaming [SPARK-24815]
>
>
> CAUTION: This email originated from outside of the organisation. Do not
> click links or open attachments unless you recognise the sender's full
> email address and know the content is safe.
>
> Thanks for proposing and voting for the feature Mich.
>
> adding some references to the thread.
>
>- Jira ticket - SPARK-24815
>
> <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCn0_Z1lI$>
>- Design Doc
>
> <https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCuAyVt8y$>
>
>- discussion thread
>
> <https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCqHoXny8$>
>- PR with initial implementation -
>https://github.com/apache/spark/pull/42352
>
> <https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCisLiWaP$>
>
> Please vote with:
>
> [ ] +1: Accept the proposal and start with the development.
> [ ] +0
> [ ] -1: I don’t think this is a good idea because …
>
> Thank you,
>
> Pavan
>
> On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh 
> wrote:
>
>
> +1 for me  (non binding)
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> 
>
> This communication is intended only for use of the addressee and may
> contain legally privileged and confidential information.
> If you are not the addressee or intended recipient, you are notified that
> any dissemination, copying or use of any of the information is unauthorised.
>
> The legal privilege and confidentiality attached to this e-mail is not
> waived, lost or destroyed by reason of a mistaken delivery to you.
> If you have received this message in error, we would appreciate an
> immediate notification via e-mail to contac...@bendigoadelaide.com.au or
> by phoning 1300 BENDIGO (1300 236 344), and ask that the e-mail be
> permanently deleted from your system.
>
> Bendigo and Adelaide Bank Limited ABN 11 068 049 178
>
>
> 
>

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Adam Hobbs]

+1

From: Pavan Kotikalapudi 
Sent: Thursday, January 18, 2024 4:19:32 AM
To: Spark dev list 
Subject: Re: Vote on Dynamic resource allocation for structured streaming 
[SPARK-24815]


CAUTION: This email originated from outside of the organisation. Do not click 
links or open attachments unless you recognise the sender's full email address 
and know the content is safe.


Thanks for proposing and voting for the feature Mich.

adding some references to the thread.

  *   Jira ticket - 
SPARK-24815<https://urldefense.com/v3/__https://issues.apache.org/jira/browse/SPARK-24815__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCn0_Z1lI$>
  *   Design 
Doc<https://urldefense.com/v3/__https://docs.google.com/document/d/1_YmfCsQQb9XhRdKh0ijbc-j8JKGtGBxYsk_30NVSTWo/edit?usp=sharing__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCuAyVt8y$>
  *   discussion 
thread<https://urldefense.com/v3/__https://lists.apache.org/thread/9yx0jnk9h1234joymwlzfx2gh2m8b9bo__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCqHoXny8$>
  *   PR with initial implementation - 
https://github.com/apache/spark/pull/42352<https://urldefense.com/v3/__https://github.com/apache/spark/pull/42352__;!!OkoFT9xN!M8RjO-4PxxtSXLdZ72VEqpLZr9IE1m1Gj4YHrjSKR-6ZwOH-1RMbh-d9RZlvDvxwMrhtlDCGv7l6zFvILPwy_fEyuSdA5k0zCisLiWaP$>

Please vote with:

[ ] +1: Accept the proposal and start with the development.
[ ] +0
[ ] -1: I don’t think this is a good idea because …

Thank you,

Pavan

On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh 
mailto:mich.talebza...@gmail.com>> wrote:

+1 for me  (non binding)



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
damage or destruction of data or any other property which may arise from 
relying on this email's technical content is explicitly disclaimed. The author 
will in no case be liable for any monetary damages arising from such loss, 
damage or destruction.





This communication is intended only for use of the addressee and may contain 
legally privileged and confidential information.
If you are not the addressee or intended recipient, you are notified that any 
dissemination, copying or use of any of the information is unauthorised.

The legal privilege and confidentiality attached to this e-mail is not waived, 
lost or destroyed by reason of a mistaken delivery to you.
If you have received this message in error, we would appreciate an immediate 
notification via e-mail to contac...@bendigoadelaide.com.au or by phoning 1300 
BENDIGO (1300 236 344), and ask that the e-mail be permanently deleted from 
your system.

Bendigo and Adelaide Bank Limited ABN 11 068 049 178

Re: Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Pavan Kotikalapudi]

Thanks for proposing and voting for the feature Mich.

adding some references to the thread.

   - Jira ticket - SPARK-24815
   
   - Design Doc
   


   - discussion thread
   
   - PR with initial implementation -
   https://github.com/apache/spark/pull/42352

Please vote with:

[ ] +1: Accept the proposal and start with the development.
[ ] +0
[ ] -1: I don’t think this is a good idea because …

Thank you,

Pavan

On Wed, Jan 17, 2024 at 9:52 PM Mich Talebzadeh 
wrote:

>
> +1 for me  (non binding)
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>

Vote on Dynamic resource allocation for structured streaming [SPARK-24815]

[Mich Talebzadeh]

+1 for me  (non binding)



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.