Re: Support for Auto scaling

[Till Rohrmann]

Hi Sandeep,

it's also not yet supported to dynamically start new TaskManagers when
needed. However, it can be done manually via the taskmanager.sh script.

We're working to enable this feature for 1.3 if everything goes according
to plan.

Cheers,
Till

On Thu, Feb 2, 2017 at 6:49 AM, Tzu-Li (Gordon) Tai 
wrote:

> Hi Sandeep!
>
> While auto scaling jobs in Flink still isn’t possible, in Flink 1.2 you
> will be able to rescale jobs by stopping and restarting.
> This works by taking a savepoint of the job before stopping the job, and
> then redeploy the job with a higher / lower parallelism using the savepoint.
> Upon restarting the job, your states will be redistributed across the new
> operators.
>
> Changing operator / job parallelism on the fly while running is still on
> the future roadmap.
>
> Cheers,
> Gordon
>
>
> On February 2, 2017 at 8:39:39 AM, Meghashyam Sandeep V (
> vr1meghash...@gmail.com) wrote:
>
> Hi Guys,
>
> I currently run flink 1.1.4 streaming jobs in EMR in AWS with yarn. I
> understand that EMR supports auto scaling but Flink doesn't. Is there a
> plan for this support in 1.2.
>
> Thanks,
> Sandeep
>
>

Re: Support for Auto scaling

[Tzu-Li (Gordon) Tai]

Hi Sandeep!

While auto scaling jobs in Flink still isn’t possible, in Flink 1.2 you will be 
able to rescale jobs by stopping and restarting.
This works by taking a savepoint of the job before stopping the job, and then 
redeploy the job with a higher / lower parallelism using the savepoint.
Upon restarting the job, your states will be redistributed across the new 
operators.

Changing operator / job parallelism on the fly while running is still on the 
future roadmap.

Cheers,
Gordon

On February 2, 2017 at 8:39:39 AM, Meghashyam Sandeep V 
(vr1meghash...@gmail.com) wrote:

Hi Guys,

I currently run flink 1.1.4 streaming jobs in EMR in AWS with yarn. I 
understand that EMR supports auto scaling but Flink doesn't. Is there a plan 
for this support in 1.2. 

Thanks,
Sandeep

Support for Auto scaling

[Meghashyam Sandeep V]

Hi Guys,

I currently run flink 1.1.4 streaming jobs in EMR in AWS with yarn. I
understand that EMR supports auto scaling but Flink doesn't. Is there a
plan for this support in 1.2.

Thanks,
Sandeep