Thanks Prasanna & Till for quick response. Looks like my use case is very similar to yours ,I will try to run multiple containers on the same machine and will update you accordingly.
Thanks , -Deep On Thu, Nov 5, 2020 at 2:33 PM Till Rohrmann <trohrm...@apache.org> wrote: > Hi Deep, > > you can increase the average CPU load by reducing the number of overall > resources. Having fewer slots over which you can distribute the work should > increase the resource usage. > > Cheers, > Till > > On Thu, Nov 5, 2020 at 9:03 AM Prasanna kumar < > prasannakumarram...@gmail.com> > wrote: > > > Deep, > > > > 1) Is it a cpu/memory/io intensive job ?? > > > > Based on that you could allocate resources. > > > > From the question, if the CPU is not utilised , you could run multiple > > containers on the same machine(tm) ... > > > > Following may not be exact case as yours but to give you an idea. > > > > Few months back I have run jobs in emr processing 4-8k per second from > > kafka with paralleism of 8 doing lightweight transformation where end to > > end latency was less than a second (10-50ms). > > > > I used slots where memory allocated is 4GB and JM memory 1gb. Here > > multilple containers ran on the same machine and I got cpu usgae upto > 50%. > > Earlier it was in single digits when just single container ran on a > > machine. > > > > Prasanna. > > > > > > On Thu 5 Nov, 2020, 12:40 Satyaa Dixit, <satyaadi...@gmail.com> wrote: > > > > > Hi Deep, > > > > > > Thanks for bringing this on table, I'm also facing a similar kind of > > issue > > > while deploying my flink Job w.r.t resources optimization. > > > > > > Hi Team, > > > > > > It would be much appreciated if someone helps us here. > > > > > > > > > Regards, > > > Satya > > > > > > On Wed, Nov 4, 2020 at 6:33 PM DEEP NARAYAN Singh < > about.d...@gmail.com> > > > wrote: > > > > > > > Hi All, > > > > > > > > I am running a flink streaming job in EMR Cluster with parallelism 21 > > > > having 500 records per second.But still seeing cpu utilization is > > > > approximate 5-8 percent. > > > > > > > > Below is the long running session command in EMR Cluster having 3 > > > instance > > > > of type C52xlarge(8vcore, 16 GB memory, AWS resource) > > > > > > > > *sudo flink-yarn-session -n 3 -s 7 -jm 4168 -tm 8000 -d* > > > > > > > > Anyone can suggest some configuration to maximize the CPU > utilization? > > > > And Also what would be the standard utilization of CPU for flink job > in > > > > order to achieve the minimum latency? > > > > > > > > Any leads would be appreciated. > > > > > > > > Thanks, > > > > -Deep > > > > > > > > > > > > > -- > > > -------------------------- > > > Best Regards > > > Satya Prakash > > > (M)+91-9845111913 > > > > > >
