Re: FLIP-6 / Kubernetes

[Thomas Weise]

Thanks everyone for the info.

I have used the Flink docker containers to launch a Flink cluster, but what
I'm looking forward to is more along the lines asked by Eron: submitting a
Flink pipeline to a k8s cluster without having to deal with the traditional
Flink cluster. I think this would be very valuable, especially if the
resources can be adjusted dynamically.

Looking forward to discuss more around Flink Forward!

Cheers,
Thomas


On Wed, Mar 21, 2018 at 8:02 AM, Eron Wright  wrote:

> Thanks Stephan for the information, it's exciting.   This is an aspect for
> which the FLIP-6 design doc is lacking detail; somewhere we should work
> through how this will work (e.g. from a job recovery and upgrade
> perspective).   If there's a doc or an open issue, please share, thanks!
>
> On Wed, Mar 21, 2018 at 5:41 AM, Stephan Ewen  wrote:
>
> > Hi Christophe!
> >
> > I could imagine 1.5.1 or 1.6.
> >
> > It is important to point out that you can use Flink with Kubernetes quite
> > well already. That addition is mainly intended to make it more idiomatic
> > for Kubernetes  users.
> >
> > Best,
> > Stephan
> >
> >
> > On Wed, Mar 21, 2018 at 12:03 PM, Christophe Jolif 
> > wrote:
> >
> > > Hi Stephan,
> > >
> > > > This feature did not make it for 1.5, but should come very
> > > soon after.
> > >
> > > Are you talking about a 1.5.1 here? What is the envisioned timeline?
> > >
> > > Thanks!
> > >
> > >
> > > On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen 
> wrote:
> > >
> > > > @Eron That is definitely the way we want to suggest as the way to use
> > k8s
> > > > in the future. This feature did not make it for 1.5, but should come
> > very
> > > > soon after.
> > > >
> > > > @Thomas An implementation of a ResourceManager for k8s should come in
> > the
> > > > near future. Would be happy to jump on a joint FLIP, after the 1.5
> > > release
> > > > and Flink Forward (in three weeks or so).
> > > >
> > > > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright 
> > > wrote:
> > > >
> > > > >  Till, is it possible to package a Flink application as a
> > > self-contained
> > > > > deployment on Kubernetes?  I mean, run a Flink application using
> > 'flink
> > > > > run' such that it launches its own RM/JM and waits for a
> sufficient #
> > > of
> > > > > TMs to join?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann <
> trohrm...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > I forgot to add Flink's K8 documentation [1] which might also be
> > > > helpful
> > > > > > with getting started.
> > > > > >
> > > > > > [1]
> > > > > > https://ci.apache.org/projects/flink/flink-docs-
> > > master/ops/deployment/
> > > > > > kubernetes.html
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <
> > > trohrm...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Thomas,
> > > > > > >
> > > > > > > I think the one way to get started would be to adapt the Flink
> > > docker
> > > > > > > images [1,2] to run with Flink 1.5. Per default, they will use
> > the
> > > > > Flip-6
> > > > > > > components.
> > > > > > >
> > > > > > > Flink 1.5 won't come with a dedicated integration with
> Kubernetes
> > > > which
> > > > > > is
> > > > > > > able to start new pods. However, it should work that you
> manually
> > > or
> > > > by
> > > > > > the
> > > > > > > virtues of an external system start new pods which can be used.
> > > Once
> > > > > the
> > > > > > > new pods have been started and the TaskExecutors have
> registered
> > > one
> > > > > > would
> > > > > > > have to rescale the job manually to the new parallelism.
> > > > > > >
> > > > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> > > > > image.html
> > > > > > > [2] https://hub.docker.com/r/_/flink/
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Till
> > > > > > >
> > > > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise 
> > > > wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> What would be a good starting point to try out Flink on
> > Kubernetes
> > > > > (any
> > > > > > >> examples / tutorials)?
> > > > > > >>
> > > > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> > > > > Kubernetes?
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Thomas
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Christophe
> > >
> >
>

Re: FLIP-6 / Kubernetes

[Eron Wright]

Thanks Stephan for the information, it's exciting.   This is an aspect for
which the FLIP-6 design doc is lacking detail; somewhere we should work
through how this will work (e.g. from a job recovery and upgrade
perspective).   If there's a doc or an open issue, please share, thanks!

On Wed, Mar 21, 2018 at 5:41 AM, Stephan Ewen  wrote:

> Hi Christophe!
>
> I could imagine 1.5.1 or 1.6.
>
> It is important to point out that you can use Flink with Kubernetes quite
> well already. That addition is mainly intended to make it more idiomatic
> for Kubernetes  users.
>
> Best,
> Stephan
>
>
> On Wed, Mar 21, 2018 at 12:03 PM, Christophe Jolif 
> wrote:
>
> > Hi Stephan,
> >
> > > This feature did not make it for 1.5, but should come very
> > soon after.
> >
> > Are you talking about a 1.5.1 here? What is the envisioned timeline?
> >
> > Thanks!
> >
> >
> > On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen  wrote:
> >
> > > @Eron That is definitely the way we want to suggest as the way to use
> k8s
> > > in the future. This feature did not make it for 1.5, but should come
> very
> > > soon after.
> > >
> > > @Thomas An implementation of a ResourceManager for k8s should come in
> the
> > > near future. Would be happy to jump on a joint FLIP, after the 1.5
> > release
> > > and Flink Forward (in three weeks or so).
> > >
> > > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright 
> > wrote:
> > >
> > > >  Till, is it possible to package a Flink application as a
> > self-contained
> > > > deployment on Kubernetes?  I mean, run a Flink application using
> 'flink
> > > > run' such that it launches its own RM/JM and waits for a sufficient #
> > of
> > > > TMs to join?
> > > >
> > > > Thanks!
> > > >
> > > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann  >
> > > > wrote:
> > > >
> > > > > I forgot to add Flink's K8 documentation [1] which might also be
> > > helpful
> > > > > with getting started.
> > > > >
> > > > > [1]
> > > > > https://ci.apache.org/projects/flink/flink-docs-
> > master/ops/deployment/
> > > > > kubernetes.html
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <
> > trohrm...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Thomas,
> > > > > >
> > > > > > I think the one way to get started would be to adapt the Flink
> > docker
> > > > > > images [1,2] to run with Flink 1.5. Per default, they will use
> the
> > > > Flip-6
> > > > > > components.
> > > > > >
> > > > > > Flink 1.5 won't come with a dedicated integration with Kubernetes
> > > which
> > > > > is
> > > > > > able to start new pods. However, it should work that you manually
> > or
> > > by
> > > > > the
> > > > > > virtues of an external system start new pods which can be used.
> > Once
> > > > the
> > > > > > new pods have been started and the TaskExecutors have registered
> > one
> > > > > would
> > > > > > have to rescale the job manually to the new parallelism.
> > > > > >
> > > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> > > > image.html
> > > > > > [2] https://hub.docker.com/r/_/flink/
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise 
> > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> What would be a good starting point to try out Flink on
> Kubernetes
> > > > (any
> > > > > >> examples / tutorials)?
> > > > > >>
> > > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> > > > Kubernetes?
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Thomas
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Christophe
> >
>

Re: FLIP-6 / Kubernetes

[Stephan Ewen]

Hi Christophe!

I could imagine 1.5.1 or 1.6.

It is important to point out that you can use Flink with Kubernetes quite
well already. That addition is mainly intended to make it more idiomatic
for Kubernetes  users.

Best,
Stephan


On Wed, Mar 21, 2018 at 12:03 PM, Christophe Jolif  wrote:

> Hi Stephan,
>
> > This feature did not make it for 1.5, but should come very
> soon after.
>
> Are you talking about a 1.5.1 here? What is the envisioned timeline?
>
> Thanks!
>
>
> On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen  wrote:
>
> > @Eron That is definitely the way we want to suggest as the way to use k8s
> > in the future. This feature did not make it for 1.5, but should come very
> > soon after.
> >
> > @Thomas An implementation of a ResourceManager for k8s should come in the
> > near future. Would be happy to jump on a joint FLIP, after the 1.5
> release
> > and Flink Forward (in three weeks or so).
> >
> > On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright 
> wrote:
> >
> > >  Till, is it possible to package a Flink application as a
> self-contained
> > > deployment on Kubernetes?  I mean, run a Flink application using 'flink
> > > run' such that it launches its own RM/JM and waits for a sufficient #
> of
> > > TMs to join?
> > >
> > > Thanks!
> > >
> > > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann 
> > > wrote:
> > >
> > > > I forgot to add Flink's K8 documentation [1] which might also be
> > helpful
> > > > with getting started.
> > > >
> > > > [1]
> > > > https://ci.apache.org/projects/flink/flink-docs-
> master/ops/deployment/
> > > > kubernetes.html
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann <
> trohrm...@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > I think the one way to get started would be to adapt the Flink
> docker
> > > > > images [1,2] to run with Flink 1.5. Per default, they will use the
> > > Flip-6
> > > > > components.
> > > > >
> > > > > Flink 1.5 won't come with a dedicated integration with Kubernetes
> > which
> > > > is
> > > > > able to start new pods. However, it should work that you manually
> or
> > by
> > > > the
> > > > > virtues of an external system start new pods which can be used.
> Once
> > > the
> > > > > new pods have been started and the TaskExecutors have registered
> one
> > > > would
> > > > > have to rescale the job manually to the new parallelism.
> > > > >
> > > > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> > > image.html
> > > > > [2] https://hub.docker.com/r/_/flink/
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise 
> > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> What would be a good starting point to try out Flink on Kubernetes
> > > (any
> > > > >> examples / tutorials)?
> > > > >>
> > > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> > > Kubernetes?
> > > > >>
> > > > >> Thanks,
> > > > >> Thomas
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Christophe
>

Re: FLIP-6 / Kubernetes

[Christophe Jolif]

Hi Stephan,

> This feature did not make it for 1.5, but should come very
soon after.

Are you talking about a 1.5.1 here? What is the envisioned timeline?

Thanks!


On Tue, Mar 20, 2018 at 9:30 PM, Stephan Ewen  wrote:

> @Eron That is definitely the way we want to suggest as the way to use k8s
> in the future. This feature did not make it for 1.5, but should come very
> soon after.
>
> @Thomas An implementation of a ResourceManager for k8s should come in the
> near future. Would be happy to jump on a joint FLIP, after the 1.5 release
> and Flink Forward (in three weeks or so).
>
> On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright  wrote:
>
> >  Till, is it possible to package a Flink application as a self-contained
> > deployment on Kubernetes?  I mean, run a Flink application using 'flink
> > run' such that it launches its own RM/JM and waits for a sufficient # of
> > TMs to join?
> >
> > Thanks!
> >
> > On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann 
> > wrote:
> >
> > > I forgot to add Flink's K8 documentation [1] which might also be
> helpful
> > > with getting started.
> > >
> > > [1]
> > > https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/
> > > kubernetes.html
> > >
> > > Cheers,
> > > Till
> > >
> > > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann 
> > > wrote:
> > >
> > > > Hi Thomas,
> > > >
> > > > I think the one way to get started would be to adapt the Flink docker
> > > > images [1,2] to run with Flink 1.5. Per default, they will use the
> > Flip-6
> > > > components.
> > > >
> > > > Flink 1.5 won't come with a dedicated integration with Kubernetes
> which
> > > is
> > > > able to start new pods. However, it should work that you manually or
> by
> > > the
> > > > virtues of an external system start new pods which can be used. Once
> > the
> > > > new pods have been started and the TaskExecutors have registered one
> > > would
> > > > have to rescale the job manually to the new parallelism.
> > > >
> > > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> > image.html
> > > > [2] https://hub.docker.com/r/_/flink/
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise 
> wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> What would be a good starting point to try out Flink on Kubernetes
> > (any
> > > >> examples / tutorials)?
> > > >>
> > > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> > Kubernetes?
> > > >>
> > > >> Thanks,
> > > >> Thomas
> > > >>
> > > >
> > > >
> > >
> >
>



-- 
Christophe

Re: FLIP-6 / Kubernetes

[Stephan Ewen]

@Eron That is definitely the way we want to suggest as the way to use k8s
in the future. This feature did not make it for 1.5, but should come very
soon after.

@Thomas An implementation of a ResourceManager for k8s should come in the
near future. Would be happy to jump on a joint FLIP, after the 1.5 release
and Flink Forward (in three weeks or so).

On Tue, Mar 20, 2018 at 4:25 PM, Eron Wright  wrote:

>  Till, is it possible to package a Flink application as a self-contained
> deployment on Kubernetes?  I mean, run a Flink application using 'flink
> run' such that it launches its own RM/JM and waits for a sufficient # of
> TMs to join?
>
> Thanks!
>
> On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann 
> wrote:
>
> > I forgot to add Flink's K8 documentation [1] which might also be helpful
> > with getting started.
> >
> > [1]
> > https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/
> > kubernetes.html
> >
> > Cheers,
> > Till
> >
> > On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann 
> > wrote:
> >
> > > Hi Thomas,
> > >
> > > I think the one way to get started would be to adapt the Flink docker
> > > images [1,2] to run with Flink 1.5. Per default, they will use the
> Flip-6
> > > components.
> > >
> > > Flink 1.5 won't come with a dedicated integration with Kubernetes which
> > is
> > > able to start new pods. However, it should work that you manually or by
> > the
> > > virtues of an external system start new pods which can be used. Once
> the
> > > new pods have been started and the TaskExecutors have registered one
> > would
> > > have to rescale the job manually to the new parallelism.
> > >
> > > [1] https://flink.apache.org/news/2017/05/16/official-docker-
> image.html
> > > [2] https://hub.docker.com/r/_/flink/
> > >
> > > Cheers,
> > > Till
> > >
> > > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise  wrote:
> > >
> > >> Hi,
> > >>
> > >> What would be a good starting point to try out Flink on Kubernetes
> (any
> > >> examples / tutorials)?
> > >>
> > >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on
> Kubernetes?
> > >>
> > >> Thanks,
> > >> Thomas
> > >>
> > >
> > >
> >
>

Re: FLIP-6 / Kubernetes

[Eron Wright]

 Till, is it possible to package a Flink application as a self-contained
deployment on Kubernetes?  I mean, run a Flink application using 'flink
run' such that it launches its own RM/JM and waits for a sufficient # of
TMs to join?

Thanks!

On Mon, Mar 19, 2018 at 2:57 AM, Till Rohrmann  wrote:

> I forgot to add Flink's K8 documentation [1] which might also be helpful
> with getting started.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/
> kubernetes.html
>
> Cheers,
> Till
>
> On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann 
> wrote:
>
> > Hi Thomas,
> >
> > I think the one way to get started would be to adapt the Flink docker
> > images [1,2] to run with Flink 1.5. Per default, they will use the Flip-6
> > components.
> >
> > Flink 1.5 won't come with a dedicated integration with Kubernetes which
> is
> > able to start new pods. However, it should work that you manually or by
> the
> > virtues of an external system start new pods which can be used. Once the
> > new pods have been started and the TaskExecutors have registered one
> would
> > have to rescale the job manually to the new parallelism.
> >
> > [1] https://flink.apache.org/news/2017/05/16/official-docker-image.html
> > [2] https://hub.docker.com/r/_/flink/
> >
> > Cheers,
> > Till
> >
> > On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise  wrote:
> >
> >> Hi,
> >>
> >> What would be a good starting point to try out Flink on Kubernetes (any
> >> examples / tutorials)?
> >>
> >> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on Kubernetes?
> >>
> >> Thanks,
> >> Thomas
> >>
> >
> >
>

Re: FLIP-6 / Kubernetes

[Till Rohrmann]

I forgot to add Flink's K8 documentation [1] which might also be helpful
with getting started.

[1]
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html

Cheers,
Till

On Mon, Mar 19, 2018 at 10:54 AM, Till Rohrmann 
wrote:

> Hi Thomas,
>
> I think the one way to get started would be to adapt the Flink docker
> images [1,2] to run with Flink 1.5. Per default, they will use the Flip-6
> components.
>
> Flink 1.5 won't come with a dedicated integration with Kubernetes which is
> able to start new pods. However, it should work that you manually or by the
> virtues of an external system start new pods which can be used. Once the
> new pods have been started and the TaskExecutors have registered one would
> have to rescale the job manually to the new parallelism.
>
> [1] https://flink.apache.org/news/2017/05/16/official-docker-image.html
> [2] https://hub.docker.com/r/_/flink/
>
> Cheers,
> Till
>
> On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise  wrote:
>
>> Hi,
>>
>> What would be a good starting point to try out Flink on Kubernetes (any
>> examples / tutorials)?
>>
>> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on Kubernetes?
>>
>> Thanks,
>> Thomas
>>
>
>

Re: FLIP-6 / Kubernetes

[Till Rohrmann]

Hi Thomas,

I think the one way to get started would be to adapt the Flink docker
images [1,2] to run with Flink 1.5. Per default, they will use the Flip-6
components.

Flink 1.5 won't come with a dedicated integration with Kubernetes which is
able to start new pods. However, it should work that you manually or by the
virtues of an external system start new pods which can be used. Once the
new pods have been started and the TaskExecutors have registered one would
have to rescale the job manually to the new parallelism.

[1] https://flink.apache.org/news/2017/05/16/official-docker-image.html
[2] https://hub.docker.com/r/_/flink/

Cheers,
Till

On Sun, Mar 18, 2018 at 9:05 PM, Thomas Weise  wrote:

> Hi,
>
> What would be a good starting point to try out Flink on Kubernetes (any
> examples / tutorials)?
>
> Also, will the FLIP-6 work in 1.5 enable dynamic scaling on Kubernetes?
>
> Thanks,
> Thomas
>

FLIP-6 / Kubernetes

[Thomas Weise]

Hi,

What would be a good starting point to try out Flink on Kubernetes (any
examples / tutorials)?

Also, will the FLIP-6 work in 1.5 enable dynamic scaling on Kubernetes?

Thanks,
Thomas