Re: Kubernetes JobManager and TaskManager minimum/maximum resources
But of course the actual memory requirement will largely depend on the type of job, statebackend , number of task slots etc Production TM/JMs usually have much more resources allocated than 2gb/1cpu as you never want to run out of it :) Gyula On Sat, 21 Jan 2023 at 11:17, Gyula Fóra wrote: > Hi! > > I think the examples allocate too many resources by default and we should > reduce it in the yamls. > > 1gb memory and 0.5 cpu should be more than enough , we could probably get > away with even less for example purposes. > > Would you have time trying this out and maybe contributing this > improvement? :) > > Thanks > Gyula > > > On Fri, 20 Jan 2023 at 05:32, Lee Parayno wrote: > >> For application mode FlinkDeployments (maybe even session mode) in >> Kubernetes from the Flink Kubernetes Operator what is the absolute minimum >> amount of CPU and RAM that is required to run the JobManager and >> TaskManager processes? >> >> Some of the example deployment yaml examples have CPU set at 1 full vCPU >> and memory at 2GB (2048 MB). If you factor in JobManager HA, and 1 or more >> TaskManagers (not sure what is the bounding limit for these processes), you >> can be at 3 vCPU and 6 GB memory used just by the “Flink Infrastructure” >> not counting the Job pods. >> >> Has anyone seen a need to have more resources dedicated to these >> processes for some reason? Has anyone run it leaner than this (like with >> 0.5 vCPU and less than 1GB memory) in production? >> >> Comparing this to Google Cloud Platform and the Dataflow Runner, AFAIK >> the only resources utilized (that customers pay for) are the Job instances. >> >> Lee Parayno >> Sent from my iPhone > >
Re: Kubernetes JobManager and TaskManager minimum/maximum resources
Hi! I think the examples allocate too many resources by default and we should reduce it in the yamls. 1gb memory and 0.5 cpu should be more than enough , we could probably get away with even less for example purposes. Would you have time trying this out and maybe contributing this improvement? :) Thanks Gyula On Fri, 20 Jan 2023 at 05:32, Lee Parayno wrote: > For application mode FlinkDeployments (maybe even session mode) in > Kubernetes from the Flink Kubernetes Operator what is the absolute minimum > amount of CPU and RAM that is required to run the JobManager and > TaskManager processes? > > Some of the example deployment yaml examples have CPU set at 1 full vCPU > and memory at 2GB (2048 MB). If you factor in JobManager HA, and 1 or more > TaskManagers (not sure what is the bounding limit for these processes), you > can be at 3 vCPU and 6 GB memory used just by the “Flink Infrastructure” > not counting the Job pods. > > Has anyone seen a need to have more resources dedicated to these processes > for some reason? Has anyone run it leaner than this (like with 0.5 vCPU > and less than 1GB memory) in production? > > Comparing this to Google Cloud Platform and the Dataflow Runner, AFAIK the > only resources utilized (that customers pay for) are the Job instances. > > Lee Parayno > Sent from my iPhone
Kubernetes JobManager and TaskManager minimum/maximum resources
For application mode FlinkDeployments (maybe even session mode) in Kubernetes from the Flink Kubernetes Operator what is the absolute minimum amount of CPU and RAM that is required to run the JobManager and TaskManager processes? Some of the example deployment yaml examples have CPU set at 1 full vCPU and memory at 2GB (2048 MB). If you factor in JobManager HA, and 1 or more TaskManagers (not sure what is the bounding limit for these processes), you can be at 3 vCPU and 6 GB memory used just by the “Flink Infrastructure” not counting the Job pods. Has anyone seen a need to have more resources dedicated to these processes for some reason? Has anyone run it leaner than this (like with 0.5 vCPU and less than 1GB memory) in production? Comparing this to Google Cloud Platform and the Dataflow Runner, AFAIK the only resources utilized (that customers pay for) are the Job instances. Lee Parayno Sent from my iPhone
