Re: Auto/Dynamic scaling in Flink

[罗齐]

Hi Nauroz,

If you’re using Flink 1.5 on Yarn, it supports dynamic task manager allocation 
by default [1]. After skimming the code, it seems to me that in general if 
requested parallelism is larger than available task slots, new task managers 
will be requested via ResourceManager (please correct me if any 
misunderstanding).

We’re also looking into auto scaling issue in Flink, as it’s crucial in our use 
cases (e.g. in batch mode, we will do input with few TMs but output with much 
more TMs). Please kindly let me know if any further information are available.

[1] 
https://stackoverflow.com/questions/38054015/can-yarn-dynamically-allocate-resources-to-flink/38077626#38077626
 


> On Nov 13, 2018, at 8:10 PM, Tzu-Li Chen  wrote:
> 
> Hi,
> 
> Yet Flink does not support auto-scaling. However, there is an umbrella JIRA 
> issue[1]
> to cover the discussion about it. And I think the design doc(draft) 
> attached[2] could help.
> 
> Best,
> tison.
> 
> [1] https://issues.apache.org/jira/browse/FLINK-10407 
> 
> [2] 
> https://docs.google.com/document/d/1XKDXnrp8w45k2jIJNHxpNP2Zmr6FGru3H0dTHwYWJyE/edit
>  
> 
> 
> Tzu-Li (Gordon) Tai mailto:tzuli...@apache.org>> 
> 于2018年11月13日周二 下午8:05写道：
> Hi,
> 
> Flink does not support auto-scaling, yet. Rescaling operations currently are 
> always manual, i.e take a savepoint of the Flink job, and when restoring from 
> the savepoint, define a new parallelism for the job.
> As for the metrics to be used for auto-scaling, I can imagine that it would 
> be possible to base this on top of metrics such as TM throughput, 
> backpressure, etc. AFAIK, there also currently isn't any official design or 
> discussion on going for this.
> 
> Cheers,
> Gordon
> 
> On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani 
>  > wrote:
> Dear Flink Contributors and users,
> 
> I am a PhD student and I was interested to know, using which matrices, and 
> when does Flink performs scaling-in or scaling out of resources? I did search 
> the flink's website where I could only find information about how dynamic 
> scaling is performed in stateless or stateful operator.  It would be 
> interesting to know which matrices Flink uses, and when actually Flink 
> triggers auto-scaling.
> 
> Any link or reference paper with required information is appreciated.
> 
> best regards,
> Nauroz

Re: Auto/Dynamic scaling in Flink

[Tzu-Li Chen]

Hi,

Yet Flink does not support auto-scaling. However, there is an umbrella JIRA
issue[1]
to cover the discussion about it. And I think the design doc(draft)
attached[2] could help.

Best,
tison.

[1] https://issues.apache.org/jira/browse/FLINK-10407
[2]
https://docs.google.com/document/d/1XKDXnrp8w45k2jIJNHxpNP2Zmr6FGru3H0dTHwYWJyE/edit


Tzu-Li (Gordon) Tai  于2018年11月13日周二 下午8:05写道：

> Hi,
>
> Flink does not support auto-scaling, yet. Rescaling operations currently
> are always manual, i.e take a savepoint of the Flink job, and when
> restoring from the savepoint, define a new parallelism for the job.
> As for the metrics to be used for auto-scaling, I can imagine that it
> would be possible to base this on top of metrics such as TM throughput,
> backpressure, etc. AFAIK, there also currently isn't any official design or
> discussion on going for this.
>
> Cheers,
> Gordon
>
> On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani <
> nauroz-khan.nausherw...@etu.univ-cotedazur.fr> wrote:
>
>> Dear Flink Contributors and users,
>>
>> I am a PhD student and I was interested to know, using which matrices,
>> and when does Flink performs scaling-in or scaling out of resources? I did
>> search the flink's website where I could only find information about how
>> dynamic scaling is performed in stateless or stateful operator.  It would
>> be interesting to know which matrices Flink uses, and when actually Flink
>> triggers auto-scaling.
>>
>> Any link or reference paper with required information is appreciated.
>>
>> best regards,
>> Nauroz
>>
>

Re: Auto/Dynamic scaling in Flink

[Tzu-Li (Gordon) Tai]

Hi,

Flink does not support auto-scaling, yet. Rescaling operations currently
are always manual, i.e take a savepoint of the Flink job, and when
restoring from the savepoint, define a new parallelism for the job.
As for the metrics to be used for auto-scaling, I can imagine that it would
be possible to base this on top of metrics such as TM throughput,
backpressure, etc. AFAIK, there also currently isn't any official design or
discussion on going for this.

Cheers,
Gordon

On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani <
nauroz-khan.nausherw...@etu.univ-cotedazur.fr> wrote:

> Dear Flink Contributors and users,
>
> I am a PhD student and I was interested to know, using which matrices, and
> when does Flink performs scaling-in or scaling out of resources? I did
> search the flink's website where I could only find information about how
> dynamic scaling is performed in stateless or stateful operator.  It would
> be interesting to know which matrices Flink uses, and when actually Flink
> triggers auto-scaling.
>
> Any link or reference paper with required information is appreciated.
>
> best regards,
> Nauroz
>

Auto/Dynamic scaling in Flink

[Nauroz Khan Nausherwani]

Dear Flink Contributors and users,

I am a PhD student and I was interested to know, using which matrices, and when 
does Flink performs scaling-in or scaling out of resources? I did search the 
flink's website where I could only find information about how dynamic scaling 
is performed in stateless or stateful operator.  It would be interesting to 
know which matrices Flink uses, and when actually Flink triggers auto-scaling.

Any link or reference paper with required information is appreciated.

best regards,
Nauroz