Re: Resource Optimization for Flink Job in AWS EMR Cluster

2020-11-05 Thread DEEP NARAYAN Singh
Thanks Prasanna & Till for quick response. Looks like my use case is very similar to yours ,I will try to run multiple containers on the same machine and will update you accordingly. Thanks , -Deep On Thu, Nov 5, 2020 at 2:33 PM Till Rohrmann wrote: > Hi Deep, > > you can increase the

Re: Resource Optimization for Flink Job in AWS EMR Cluster

2020-11-05 Thread Till Rohrmann
Hi Deep, you can increase the average CPU load by reducing the number of overall resources. Having fewer slots over which you can distribute the work should increase the resource usage. Cheers, Till On Thu, Nov 5, 2020 at 9:03 AM Prasanna kumar wrote: > Deep, > > 1) Is it a cpu/memory/io

Re: Resource Optimization for Flink Job in AWS EMR Cluster

2020-11-05 Thread Prasanna kumar
Deep, 1) Is it a cpu/memory/io intensive job ?? Based on that you could allocate resources. >From the question, if the CPU is not utilised , you could run multiple containers on the same machine(tm) ... Following may not be exact case as yours but to give you an idea. Few months back I have

Re: Resource Optimization for Flink Job in AWS EMR Cluster

2020-11-04 Thread Satyaa Dixit
Hi Deep, Thanks for bringing this on table, I'm also facing a similar kind of issue while deploying my flink Job w.r.t resources optimization. Hi Team, It would be much appreciated if someone helps us here. Regards, Satya On Wed, Nov 4, 2020 at 6:33 PM DEEP NARAYAN Singh wrote: > Hi All,

Re: Resource Optimization for Flink Job in AWS EMR Cluster

2020-11-04 Thread DEEP NARAYAN Singh
Hi Guys, Sorry to bother you again.Someone could help me here for clarifying my doubt. Any help will be highly appreciated. Thanks, -Deep On Wed, Nov 4, 2020 at 6:26 PM DEEP NARAYAN Singh wrote: > Hi All, > > I am running a flink streaming job in EMR Cluster with parallelism 21 > having 500

Resource Optimization for Flink Job in AWS EMR Cluster

2020-11-04 Thread DEEP NARAYAN Singh
Hi All, I am running a flink streaming job in EMR Cluster with parallelism 21 having 500 records per second.But still seeing cpu utilization is approximate 5-8 percent. Below is the long running session command in EMR Cluster having 3 instance of type C52xlarge(8vcore, 16 GB memory, AWS