[kubernetes-users] Scaled out Pods having inconsistent performances.

[rajitha k]

In our deployment, we have queue consumer pods. The messages are somewhat 
very similar in content and when processing each message we expect pods to 
have minimal deviation in performance across pods. But our observation is 
that scaled pods tend to have different processing times with significant 
deviations.

I've attached a screen shot of a monitoring graph of the processing times 
of pods at a scaled out instance.

Further we have set the CPU and memory limits for this deployment, CPU 
limit and CPU request both are set to '1'. 

The pods are consuming from the same queue. The queue consumers(pods) are 
blocking in nature and processes a single message at a time.

The graph only plots the time taken to process message within the consumer 
excluding any network delays.

We want this processing time to be consistent across scaled pods with a 
minimal deviation.

Hope you could assist us to find the cause and fix to this issue.

Thank you.

<https://lh3.googleusercontent.com/-n02zS67jz4g/WrHwsvt_meI/AAAAAAAAAAc/j15Mhjfgp1E9y3tzZxKqYApiCL9U4CgnwCLcBGAs/s1600/rpeak-processing-time-min.jpg>



deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  labels:
    run:xxxxxxxx
  name: xxxxxxxx
  namespace: default
  
spec:
  selector:
    matchLabels:
      run:xxxxxxxx
  template:
    metadata:
      labels:
        run: xxxxxxxxx
    spec:
      containers:
      - image: gcr.io/xxxxxx/xxxxxxx\:$$version
        imagePullPolicy: IfNotPresent
        name:xxxxxxxxx
        resources:
           limits:
              cpu: "1"
              memory: "256Mi"
           requests:
              cpu: "1"
              memory: "128Mi"
        envFrom:
        - configMapRef:
           name: xxxxxxx-config
      nodeSelector:  
         nodeclass: rpeak-pool

-- 
You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.