I am maintaining a kubernetes cluster having hardware of very different 
configurations. So far it only had machines with 96GB of RAM which worked 
well. Today I added 10 more nodes, each with 32GB of RAM. When a large 
scale experiment was deployed to this new cloud configuration, around 20% 
of the requested pod never got scheduled. They kept on hanging at 
`ContainerCreating` state indefinitely. When I describe a kubernetes node 
on which one of these pods is scheduled, I receive the following,

    # kubectl describe no/<kubernetes_node>
    Name:               <kubernetes_node>
    Roles:              <none>
    Labels:             beta.kubernetes.io/arch=amd64
    Annotations:        node.alpha.kubernetes.io/ttl=0
    Taints:             <none>
    CreationTimestamp:  Fri, 02 Mar 2018 12:58:47 -0800
      Type             Status  LastHeartbeatTime                
 LastTransitionTime                Reason                       Message
      ----             ------  -----------------                
 ------------------                ------                       -------
      OutOfDisk        False   Fri, 02 Mar 2018 18:24:33 -0800   Fri, 02 
Mar 2018 12:58:47 -0800   KubeletHasSufficientDisk     kubelet has 
sufficient disk space available
      MemoryPressure   False   Fri, 02 Mar 2018 18:24:33 -0800   Fri, 02 
Mar 2018 12:58:47 -0800   KubeletHasSufficientMemory   kubelet has 
sufficient memory available
      DiskPressure     False   Fri, 02 Mar 2018 18:24:33 -0800   Fri, 02 
Mar 2018 12:58:47 -0800   KubeletHasNoDiskPressure     kubelet has no disk 
      Ready            True    Fri, 02 Mar 2018 18:24:33 -0800   Fri, 02 
Mar 2018 12:59:17 -0800   KubeletReady                 kubelet is posting 
ready status
      Hostname:    <kubernetes_node>
     cpu:     8
     memory:  32919476Ki
     pods:    110
     cpu:     8
     memory:  32817076Ki
     pods:    110
    System Info:
     Machine ID:                 cb97393b0de14b6ebd4f2eabae6d7690
     System UUID:                00000000-BEEF-0706-0000-0000EFBE0E0F
     Boot ID:                    8aca44a0-9ae9-454e-8f9e-50ae6be3665a
     Kernel Version:             4.4.0-21-generic
     OS Image:                   Ubuntu 16.04 LTS
     Operating System:           linux
     Architecture:               amd64
     Container Runtime Version:  docker://1.13.1
     Kubelet Version:            v1.9.3
     Kube-Proxy Version:         v1.9.3
    ExternalID:                  <kubernetes_node>
    Non-terminated Pods:         (5 in total)
      Namespace                  Name                  CPU Requests  CPU 
Limits  Memory Requests  Memory Limits
      ---------                  ----                  ------------  
----------  ---------------  -------------
      kube-system                calico-node-qpr9n     250m (3%)     0 
(0%)      0 (0%)           0 (0%)
      kube-system                kube-proxy-hqbbd      0 (0%)        0 
(0%)      0 (0%)           0 (0%)
      <kubernetes_user>          minecrawlers-gqjqv    2 (25%)       2 
(25%)     8Gi (25%)        8Gi (25%)
      <kubernetes_user>          minecrawlers-rb984    2 (25%)       2 
(25%)     8Gi (25%)        8Gi (25%)
      <kubernetes_user>          minecrawlers-sjwzd    2 (25%)       2 
(25%)     8Gi (25%)        8Gi (25%)
    Allocated resources:
      (Total limits may be over 100 percent, i.e., overcommitted.)
      CPU Requests  CPU Limits  Memory Requests  Memory Limits
      ------------  ----------  ---------------  -------------
      6250m (78%)   6 (75%)     24Gi (76%)       24Gi (76%)
    Events:         <none>

None of the three pods ever gets scheduled. Describing the first one shows 

    # kubectl describe pods/minecrawlers-gqjqv
    Name:           minecrawlers-gqjqv
    Namespace:      <kubernetes_user>
    Node:           <kubernetes_node>/
    Start Time:     Fri, 02 Mar 2018 17:58:31 -0800
    Labels:         app=minecrawl
    Annotations:    <none>
    Status:         Pending
    Controlled By:  ReplicationController/minecrawlers
        Container ID:   
        Image ID:       
        Port:           <none>
        State:          Waiting
          Reason:       ContainerCreating
        Ready:          False
        Restart Count:  0
          cpu:     2
          memory:  8Gi
          cpu:        2
          memory:     8Gi
        Environment:  <none>
          /dev/shm from dshm (rw)
          /var/run/secrets/kubernetes.io/serviceaccount from 
default-token-5q7vh (ro)
      Type           Status
      Initialized    True 
      Ready          False 
      PodScheduled   True 
        Type:    EmptyDir (a temporary directory that shares a pod's 
        Medium:  Memory
        Type:        Secret (a volume populated by a Secret)
        SecretName:  default-token-5q7vh
        Optional:    false
    QoS Class:       Guaranteed
    Node-Selectors:  <none>
    Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                     node.kubernetes.io/unreachable:NoExecute for 300s
      Type     Reason                  Age                 From            
      ----     ------                  ----                ----            
      Normal   Scheduled               36m                
 default-scheduler                   Successfully assigned 
minecrawlers-gqjqv to <kubernetes_node>
      Normal   SuccessfulMountVolume   36m                 kubelet, 
<kubernetes_node>  MountVolume.SetUp succeeded for volume "dshm"
      Normal   SuccessfulMountVolume   36m                 kubelet, 
<kubernetes_node>  MountVolume.SetUp succeeded for volume 
      Normal   SandboxChanged          35m (x11 over 36m)  kubelet, 
<kubernetes_node>  Pod sandbox changed, it will be killed and re-created.
      Warning  FailedCreatePodSandBox  1m (x541 over 36m)  kubelet, 
<kubernetes_node>  Failed create pod sandbox.

What can possibly be the problem?

You received this message because you are subscribed to the Google Groups 
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

Reply via email to