Nope must be something in your heat templates. ./ete-k8s.sh onap healthdist does a vFW heat template onboarding and distribution (where the zip file is created)
Try that on a fresh robot container and see what the file sizes are. Brian From: Rajendra P Mishra (RP) <[email protected]> Sent: Thursday, April 2, 2020 10:48 AM To: FREEMAN, BRIAN D <[email protected]> Cc: [email protected]; STARK, STEVEN <[email protected]> Subject: Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing ovp validation on reference VNF I increased the size of the storage to 600GB for each node, however the robot pod gets killed everytime I start the test. I noticed that while the test is running, it creates the tmp_heat.zip in /tmp directory from the templates passed by the user. Looks like there is a bug in the code that creates the zip file. The size of the zip file is constantly increasing and ends up consuming the entire drive. For robot I am using the below version of the image: "image: onap/testsuite:1.5.4" Are there some fixes in robot scripts that have been done beyond this version ? Below are some logs from robot pod taken while running the ovp validation tests. ------------------- aarna@anod-master:~$ kubectl get pod -n onap -o=wide | grep robot dev-robot-robot-6d444f4cdd-zsfj7 1/1 Running 0 112m 10.42.3.41 dublin03 <none> <none> aarna@anod-master:~$ kubectl exec -it -n onap dev-robot-robot-6d444f4cdd-zsfj7 -- /bin/bash root@dev-robot-robot-6d444f4cdd-zsfj7:/# ls bin boot dev etc home lib lib64 media mnt opt proc root run sbin share srv sys tmp usr var root@dev-robot-robot-6d444f4cdd-zsfj7:/# df -Th Filesystem Type Size Used Avail Use% Mounted on overlay overlay 584G 163G 421G 28% / tmpfs tmpfs 64M 0 64M 0% /dev tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/vda1 ext4 584G 163G 421G 28% /etc/localtime tmpfs tmpfs 16G 11M 16G 1% /dev/shm 10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 10G 1.9T 1% /share/logs tmpfs tmpfs 16G 12K 16G 1% /run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=oMYtzrFKWMW5gonEV9xw0_8KTkgTIeJgRRf6gqUErso&s=SF41ISVDlCh7coVwiOGQjmI7cAYxG1HntnPhNKvQxGE&e=> tmpfs tmpfs 16G 0 16G 0% /proc/acpi tmpfs tmpfs 16G 0 16G 0% /proc/scsi tmpfs tmpfs 16G 0 16G 0% /sys/firmware root@dev-robot-robot-6d444f4cdd-zsfj7:/# df -Th Filesystem Type Size Used Avail Use% Mounted on overlay overlay 584G 164G 420G 29% / tmpfs tmpfs 64M 0 64M 0% /dev tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/vda1 ext4 584G 164G 420G 29% /etc/localtime tmpfs tmpfs 16G 11M 16G 1% /dev/shm 10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 10G 1.9T 1% /share/logs tmpfs tmpfs 16G 12K 16G 1% /run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=oMYtzrFKWMW5gonEV9xw0_8KTkgTIeJgRRf6gqUErso&s=SF41ISVDlCh7coVwiOGQjmI7cAYxG1HntnPhNKvQxGE&e=> tmpfs tmpfs 16G 0 16G 0% /proc/acpi tmpfs tmpfs 16G 0 16G 0% /proc/scsi tmpfs tmpfs 16G 0 16G 0% /sys/firmware root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates#<mailto:root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates#> ls -lh total 80G -rw-rw-r-- 1 1000 1000 1.4K Mar 31 12:46 base.env -rw-rw-r-- 1 1000 1000 16K Apr 2 07:06 base.yaml -rw-r--r-- 1 root root 80G Apr 2 14:18 tmp_heat.zip root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates#<mailto:root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates#> ls -lh total 81G -rw-rw-r-- 1 1000 1000 1.4K Mar 31 12:46 base.env -rw-rw-r-- 1 1000 1000 16K Apr 2 07:06 base.yaml -rw-r--r-- 1 root root 81G Apr 2 14:18 tmp_heat.zip root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates#<mailto:root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates#> file tmp_heat.zip bash: file: command not found . . root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates#<mailto:root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates#> ls -lh total 101G -rw-rw-r-- 1 1000 1000 1.4K Mar 31 12:46 base.env -rw-rw-r-- 1 1000 1000 16K Apr 2 07:06 base.yaml -rw-r--r-- 1 root root 101G Apr 2 14:20 tmp_heat.zip root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp# df -Th Filesystem Type Size Used Avail Use% Mounted on overlay overlay 584G 209G 375G 36% / tmpfs tmpfs 64M 0 64M 0% /dev tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/vda1 ext4 584G 209G 375G 36% /etc/localtime tmpfs tmpfs 16G 11M 16G 1% /dev/shm 10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 11G 1.9T 1% /share/logs tmpfs tmpfs 16G 12K 16G 1% /run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=oMYtzrFKWMW5gonEV9xw0_8KTkgTIeJgRRf6gqUErso&s=SF41ISVDlCh7coVwiOGQjmI7cAYxG1HntnPhNKvQxGE&e=> tmpfs tmpfs 16G 0 16G 0% /proc/acpi tmpfs tmpfs 16G 0 16G 0% /proc/scsi tmpfs tmpfs 16G 0 16G 0% /sys/firmware -------------------------- thanks, -RP Mishra On Mon, Mar 30, 2020 at 6:30 PM FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>> wrote: Also if the problem is you are filling up /tmp inside your robot container try commenting out this line inside your robot contaiener /var/opt/ONAP/robot/resources/ test_templates/model_test_template.robot #Download CSAR ${catalog_service_id} This will skip the step of downloading the CSAR to the /tmp directory of your robot container which if the csar is large could be the problem. My robot continer is at 32% disk usage after doing 20 or so vFW creations but yours is at 45% so you need to see why your robot container filesystem seems to have a higher utilization. Brian From: FREEMAN, BRIAN D Sent: Monday, March 30, 2020 8:54 AM To: 'Rajendra P Mishra (RP)' <[email protected]<mailto:[email protected]>> Cc: '[email protected]<mailto:[email protected]>' <[email protected]<mailto:[email protected]>>; STARK, STEVEN <[email protected]<mailto:[email protected]>> Subject: RE: [onap-discuss] [OVP][SO] Heatbridge failure while doing ovp validation on reference VNF My azure install was not a full onap install so you may want to turn off optional components if you don’t have enough total ephemeral storage across all your nodes. Brian From: FREEMAN, BRIAN D Sent: Monday, March 30, 2020 8:53 AM To: Rajendra P Mishra (RP) <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; STARK, STEVEN <[email protected]<mailto:[email protected]>> Subject: RE: [onap-discuss] [OVP][SO] Heatbridge failure while doing ovp validation on reference VNF Look at kubectl describe nodes and see how close to your ephemeral storage capacity you are. Some stats from my Azure environment. root@onapdldk-robot-robot-577df6887c-njrc5:/# df -Th Filesystem Type Size Used Avail Use% Mounted on overlay overlay 97G 31G 67G 32% / tmpfs tmpfs 64M 0 64M 0% /dev tmpfs tmpfs 14G 0 14G 0% /sys/fs/cgroup tmpfs tmpfs 14G 158M 14G 2% /dev/shm /dev/sda1 ext4 97G 31G 67G 32% /etc/localtime 173.163.0.4:/dockerdata-nfs/onapdldk-robot/robot/logs nfs4 497G 84G 413G 17% /share/logs tmpfs tmpfs 14G 12K 14G 1% /run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=oMYtzrFKWMW5gonEV9xw0_8KTkgTIeJgRRf6gqUErso&s=SF41ISVDlCh7coVwiOGQjmI7cAYxG1HntnPhNKvQxGE&e=> tmpfs tmpfs 14G 0 14G 0% /proc/acpi tmpfs tmpfs 14G 0 14G 0% /proc/scsi tmpfs tmpfs 14G 0 14G 0% /sys/firmware bf1936@njcdtl01bf1936:~/ONAP/AZURE/ONAPDlDK/oom/kubernetes/robot$ kubectl -n onap top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% aks-nodepool1-97482552-0 883m 11% 18723Mi 78% aks-nodepool1-97482552-1 4375m 55% 17451Mi 72% aks-nodepool1-97482552-2 1849m 23% 12245Mi 51% aks-nodepool1-97482552-3 1478m 18% 17252Mi 71% aks-nodepool1-97482552-4 5382m 68% 21750Mi 90% aks-nodepool1-97482552-5 4617m 59% 14628Mi 61% aks-nodepool1-97482552-6 1179m 15% 19062Mi 79% kubectl describe nodes For node that robot is running: Capacity: attachable-volumes-azure-disk: 32 cpu: 8 ephemeral-storage: 101445900Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 28789864Ki pods: 110 Allocatable: attachable-volumes-azure-disk: 32 cpu: 7820m ephemeral-storage: 93492541286 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 24541288Ki pods: 110 From: Rajendra P Mishra (RP) <[email protected]<mailto:[email protected]>> Sent: Sunday, March 29, 2020 12:06 PM To: FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; STARK, STEVEN <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing ovp validation on reference VNF NFS drive is 2TB Here is the file system space usage in one of the node. ---- aarna@anod-master:~$ df -Th Filesystem Type Size Used Avail Use% Mounted on udev devtmpfs 16G 0 16G 0% /dev tmpfs tmpfs 3.2G 1.3M 3.2G 1% /run /dev/vda1 ext4 196G 87G 110G 45% / tmpfs tmpfs 16G 0 16G 0% /dev/shm tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/vda15 vfat 105M 3.6M 101M 4% /boot/efi 10.240.71.179:/dockerdata-nfs nfs4 1.9T 23G 1.8T 2% /dockerdata-nfs tmpfs tmpfs 3.2G 0 3.2G 0% /run/user/1000 --------- Disk usage inside robot is here. ------- aarna@anod-master:~$ kubectl get pod -n onap | grep robot dev-robot-robot-6d444f4cdd-rrwqj 0/1 Evicted 0 5h2m dev-robot-robot-6d444f4cdd-sn5sl 1/1 Running 0 4h47m aarna@anod-master:~$ kubectl exec -it -n onap dev-robot-robot-6d444f4cdd-sn5sl -- /bin/bash root@dev-robot-robot-6d444f4cdd-sn5sl:/# df -Th Filesystem Type Size Used Avail Use% Mounted on overlay overlay 196G 88G 109G 45% / tmpfs tmpfs 64M 0 64M 0% /dev tmpfs tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/vda1 ext4 196G 88G 109G 45% /etc/localtime tmpfs tmpfs 16G 0 16G 0% /dev/shm 10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 23G 1.8T 2% /share/logs tmpfs tmpfs 16G 12K 16G 1% /run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Vk1goR9Pg3FUGQEz_PEKUp_E_Y4jBvPuiwnpGZijrNM&s=RLmzIwNVRpfgGrovLp8mIHGvUoruK9qxOUINwcTCuaA&e=> tmpfs tmpfs 16G 0 16G 0% /proc/acpi tmpfs tmpfs 16G 0 16G 0% /proc/scsi tmpfs tmpfs 16G 0 16G 0% /sys/firmware ----------- For my ONAP setup, each node is hosted on a VM. Each VM has around 196GB root file system. NFS is setup on a seperate server outside of these VMs. I am trying to increase the size of the root file system from 196GB to 400GB. What I do not understand is why 196GB is not enough ? and where is the space getting consumed. thanks, -RP Mishra On Sun, Mar 29, 2020 at 4:44 PM FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>> wrote: I see now 27g out of 2tb it must be your k8 config is somehow different . What does disk usage inside the robot container look like before it is evicted ? When you created the k8 nodes how much storage did you give eack k8 ? Maybe put robot on its own k8 host ? Sent via the Samsung Galaxy S8, an AT&T 5G Evolution capable smartphone -------- Original message -------- From: Rajendra Prasad Mishra <[email protected]<mailto:[email protected]>> Date: 3/29/20 6:39 AM (GMT-05:00) To: "FREEMAN, BRIAN D" <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>, "STARK, STEVEN" <[email protected]<mailto:[email protected]>> Subject: Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing ovp validation on reference VNF Hi Brian, Steve, Deleting /share/logs and /dockerdata-nfs/dev-robot/robot/logs/ is not helping. Robot pod is still getting killed due to space problems. I had to re-install the whole ONAP to get rid of the problem. But it comes back soon once I run some larger VNF (like vFW). I tried increasing the storage allocated to robot in the values.yaml file before installing ONAP, but I still get the same space problem when I run OVP tests. (below is the diff of the values.yaml file) Is there a clean way to increase the "ephemeral-storage" for robot to some large value so that I can do multiple runs of complex VNF without running into space issues. My /dockerdata-nfs is mounted on a 2TB SSD and only 21GB is used, so we have enough space available. --- aarna@anod-master:~/oom/kubernetes/robot$ git diff values.yaml diff --git a/values.yaml b/values.yaml index 3696057..9cda2c1 100644 --- a/values.yaml +++ b/values.yaml @@ -23,12 +23,12 @@ global: # global defaults # application image repository: nexus3.onap.org:10001<https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=ZlV4kOGHmpRjjv6cp1gx66F9Dv4gkllAc6ExmLAc6L4&e=> image: onap/testsuite:1.5.4 -pullPolicy: Always +pullPolicy: IfNotPresent ubuntuInitImage: oomk8s/ubuntu-init:2.0.0 # flag to enable debugging - application support required -debugEnabled: false +debugEnabled: true ################################################################# # Application configuration defaults. @@ -254,6 +254,6 @@ persistence: ## # storageClass: "-" accessMode: ReadWriteMany - size: 2Gi + size: 4Gi mountPath: /dockerdata-nfs mountSubPath: robot/logs ----------------- Here is the pod that is killed. ----------- aarna@anod-master:~$ kubectl describe pod -n onap dev-robot-robot-6d444f4cdd-bkm4x Name: dev-robot-robot-6d444f4cdd-bkm4x Namespace: onap Priority: 0 Node: dublin05/ Start Time: Fri, 27 Mar 2020 09:30:42 +0000 Labels: app=robot pod-template-hash=6d444f4cdd release=dev-robot Annotations: cni.projectcalico.org/podIP<https://urldefense.proofpoint.com/v2/url?u=http-3A__cni.projectcalico.org_podIP&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=4Ie1PTpTbBekcYyflcwb1Ppzy3DltykU2VI8WFIqGo4&e=>: 10.42.5.41/32<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.42.5.41_32&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=PlvGS8KEgxKtPrUx0Fxgu1icZTwTwj_p31rCbEqFEVw&e=> Status: Failed Reason: Evicted Message: The node was low on resource: ephemeral-storage. Container robot was using 125078316Ki, which exceeds its request of 0. IP: Controlled By: ReplicaSet/dev-robot-robot-6d444f4cdd Containers: robot: Image: nexus3.onap.org:10001/onap/testsuite:1.5.4<https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_testsuite-3A1.5.4&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=uCMjhaGdBG8rdsixT8QAq7iFAB_mV4aWob5OD2Ou_is&e=> Port: 88/TCP Host Port: 0/TCP Liveness: tcp-socket :88 delay=180s timeout=1s period=10s #success=1 #failure=3 Readiness: tcp-socket :88 delay=180s timeout=1s period=10s #success=1 #failure=3 Environment: <none> Mounts: /dev/shm from dshm (rw) /etc/localtime from localtime (ro) /share/config from robot-eteshare (rw) /share/logs from robot-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=62KOK4XwlhhxKcBn85bXtwlUakwbOIegXv08sO1GlqE&e=> from default-token-qd76w (ro) Volumes: robot-logs: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: dev-robot-robot ReadOnly: false dshm: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: <unset> localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: robot-eteshare: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-robot-robot-eteshare-configmap Optional: false default-token-qd76w: Type: Secret (a volume populated by a Secret) SecretName: default-token-qd76w Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute<https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_not-2Dready-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=5IOJYuAKepGEJxmwD8wQHsOePlpnB2x3gbU9Y5XCoIE&e=> for 300s node.kubernetes.io/unreachable:NoExecute<https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_unreachable-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=cNdW_67uaUUVC_JSvX41xLh18gDEnPp4p7za_daCxH0&e=> for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Evicted 25m kubelet, dublin05 The node was low on resource: ephemeral-storage. Container robot was using 125078316Ki, which exceeds its request of 0. Normal Killing 25m kubelet, dublin05 Killing container with id docker://robot:Need to kill Pod ------------------------ thanks for your help, -RP Mishra On Mon, Mar 23, 2020 at 7:07 PM FREEMAN, BRIAN D <[email protected]<mailto:[email protected]>> wrote: IF you have been doing a lot of testing the robot file system can get pretty large (500Meg per test in some cases) Cleanup /share/logs inside the container (that is log files stored on the nfs) and /tmp/csar where the distributed models are stored for use in testing like vCPE Brian From: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> On Behalf Of Rajendra Prasad Mishra Sent: Monday, March 23, 2020 9:25 AM To: STARK, STEVEN <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> Subject: Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing ovp validation on reference VNF Thanks Steve, that worked. I no longer see the openstack authentication issue. However, I am seeing the robot storage issues. When I run the validation test, robot pod is killed because it is consuming more "ephemeral-storage" then allocated. Do you know how to increase the " ephemeral-storage" for robot pod ? This is a standard El Alto installation with no change in default configuration. Why does validation tests take up so much of space (127421328Ki) ----------------- aarna@anod-master:~/oom/kubernetes/robot$ kubectl get pod -n onap | grep robot dev-robot-robot-6d444f4cdd-94kkk 1/1 Running 0 3m49s dev-robot-robot-6d444f4cdd-vspc9 0/1 Evicted 0 3d15h aarna@anod-master:~/oom/kubernetes/robot$ kubectl describe pod -n onap dev-robot-robot-6d444f4cdd-vspc9 Name: dev-robot-robot-6d444f4cdd-vspc9 Namespace: onap Priority: 0 Node: dublin06/ Start Time: Thu, 19 Mar 2020 15:24:57 +0000 Labels: app=robot pod-template-hash=6d444f4cdd release=dev-robot Annotations: cni.projectcalico.org/podIP<https://urldefense.proofpoint.com/v2/url?u=http-3A__cni.projectcalico.org_podIP&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=MLwHGSQ5pPlDSCtDzE_p7lM5vcEUhSDD8zDoC3S4vn4&e=>: 10.42.1.56/32<https://urldefense.proofpoint.com/v2/url?u=http-3A__10.42.1.56_32&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=Pbnhw9Cxfa2A6klTNVsFKrsskeFRF0guKda-qrkjVC4&e=> Status: Failed Reason: Evicted Message: The node was low on resource: ephemeral-storage. Container robot was using 127421328Ki, which exceeds its request of 0. IP: Controlled By: ReplicaSet/dev-robot-robot-6d444f4cdd Containers: robot: Image: nexus3.onap.org:10001/onap/testsuite:1.5.4<https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_testsuite-3A1.5.4&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=zGRkR7bU0ERGCcc4ahXTnuiECXcRN__EkBCF_EOPrsM&e=> Port: 88/TCP Host Port: 0/TCP Liveness: tcp-socket :88 delay=180s timeout=1s period=10s #success=1 #failure=3 Readiness: tcp-socket :88 delay=180s timeout=1s period=10s #success=1 #failure=3 Environment: <none> Mounts: /dev/shm from dshm (rw) /etc/localtime from localtime (ro) /share/config from robot-eteshare (rw) /share/logs from robot-logs (rw) /var/run/secrets/kubernetes.io/serviceaccount<https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=FAoROrnwGV5yFeLt7ZfIM_LtdbNGb7eMT0YJCiD7XLw&e=> from default-token-2mz7g (ro) Volumes: robot-logs: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: dev-robot-robot ReadOnly: false dshm: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: <unset> localtime: Type: HostPath (bare host directory volume) Path: /etc/localtime HostPathType: robot-eteshare: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-robot-robot-eteshare-configmap Optional: false default-token-2mz7g: Type: Secret (a volume populated by a Secret) SecretName: default-token-2mz7g Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute<https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_not-2Dready-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=0b-LBWz7xZSxyHD54a1S970SRnyV92DCFb8OC5hGa6g&e=> for 300s node.kubernetes.io/unreachable:NoExecute<https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_unreachable-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=lUcFt1b-6535RBbsO7aF5SFumNsJvLXBjdo1eIXnxgA&e=> for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Evicted 4m26s kubelet, dublin06 The node was low on resource: ephemeral-storage. Container robot was using 127421328Ki, which exceeds its request of 0. Normal Killing 4m24s kubelet, dublin06 Killing container with id docker://robot:Need to kill Pod ---------------------------------- thanks, -RPMishra On Fri, Mar 20, 2020 at 9:42 PM Steve Stark <[email protected]<mailto:[email protected]>> wrote: Great that helps. It looks like instantiation worked, the part that's failing is some custom heat stack validation after the instantiation. I was able to replicate the failure you're getting. Can you change the admin_tenant value for your identity service to the tenant name instead of the tenant ID and try again? I think in your case the tenant name = admin. This is in the mariadb pod - MariaDB [catalogdb]> update identity_services set admin_tenant="<tenant name>" where id="<identity service id>"; Thanks Steven -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20470): https://lists.onap.org/g/onap-discuss/message/20470 Mute This Topic: https://lists.onap.org/mt/72026138/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
