Re: Operators resource requirements on K8s Flink session cluster

[Michaël Melchiore]

Hi Yang,

Thanks for your quick reply.

The Flink K8s documentation distinguishes between standalone and session
deployment mode. I think I will use the latter.
Since my previous mail, I found FLIP-53

which
is precisely the topic of my original question.

So, great progress has been already made to cover my needs. Unfortunately,
I am use DataStreams API which are currently not covered by the initial
implementation. I have asked on the dev mailing list if I could help
bridging this gap.

Regards,

Michaël

Le jeu. 19 déc. 2019 à 04:58, Yang Wang  a écrit :

> Hi Michaël,
>
> Glad to hear that you are going to run Flink workload on Kubernetes.
> AFAIK, we have two
> deployment ways.
> 1. Running Flink standalone session/per-job cluster on K8s. You need to
> calculate how many
> taskmanagers you need and the  per taskmanager. All the
> taskmanager
> will be started by a K8s deployment. You could find more information
> here[1]. In this mode,
> you could be `kubectl scale` to change the replicas of taskmanager if the
> resources are not
> enough for your job.
> 2. Natively running Flink session/per-job on K8s. The session mode has
> been support in
> master branch and will be released in 1.10. The per-job mode is in
> discussion. No matter
> session or per-job, the taskmanager will be allocated dynamically on
> demand. You could
> use a simple command to start a Flink cluster on K8s. More information
> could be found
> here[2].
>
>
> Best,
> Yang
>
> [1].
> https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html
> [2].
> https://docs.google.com/document/d/1-jNzqGF6NfZuwVaFICoFQ5HFFXzF5NVIagUZByFMfBY/edit?usp=sharing
>
>
> Michaël Melchiore  于2019年12月19日周四 上午1:11写道：
>
>> Hello,
>>
>> I plan to run topologies on a Flink session cluster on Kubernetes.
>> In my topologies, operators will have varying resource requirements in
>> term of CPU and RAM.
>> How can I make these informations available from Flink to Kubernetes so
>> the latter takes it into account to optimize its deployment ?
>>
>> I am trying to achieve something similar to Apache Storm/Trident Resource
>> Aware Scheduler
>> .
>>
>> Kind regards,
>>
>> Michaël
>>
>

Re: Operators resource requirements on K8s Flink session cluster

[Yang Wang]

Hi Michaël,

Glad to hear that you are going to run Flink workload on Kubernetes. AFAIK,
we have two
deployment ways.
1. Running Flink standalone session/per-job cluster on K8s. You need to
calculate how many
taskmanagers you need and the  per taskmanager. All the
taskmanager
will be started by a K8s deployment. You could find more information
here[1]. In this mode,
you could be `kubectl scale` to change the replicas of taskmanager if the
resources are not
enough for your job.
2. Natively running Flink session/per-job on K8s. The session mode has been
support in
master branch and will be released in 1.10. The per-job mode is in
discussion. No matter
session or per-job, the taskmanager will be allocated dynamically on
demand. You could
use a simple command to start a Flink cluster on K8s. More information
could be found
here[2].


Best,
Yang

[1].
https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/kubernetes.html
[2].
https://docs.google.com/document/d/1-jNzqGF6NfZuwVaFICoFQ5HFFXzF5NVIagUZByFMfBY/edit?usp=sharing


Michaël Melchiore  于2019年12月19日周四 上午1:11写道：

> Hello,
>
> I plan to run topologies on a Flink session cluster on Kubernetes.
> In my topologies, operators will have varying resource requirements in
> term of CPU and RAM.
> How can I make these informations available from Flink to Kubernetes so
> the latter takes it into account to optimize its deployment ?
>
> I am trying to achieve something similar to Apache Storm/Trident Resource
> Aware Scheduler
> .
>
> Kind regards,
>
> Michaël
>

Operators resource requirements on K8s Flink session cluster

[Michaël Melchiore]

Hello,

I plan to run topologies on a Flink session cluster on Kubernetes.
In my topologies, operators will have varying resource requirements in term
of CPU and RAM.
How can I make these informations available from Flink to Kubernetes so the
latter takes it into account to optimize its deployment ?

I am trying to achieve something similar to Apache Storm/Trident Resource
Aware Scheduler
.

Kind regards,

Michaël