I increased the size of the storage to 600GB for each node, however the
robot pod gets killed everytime I start the test.
I noticed that while the test is running, it creates the tmp_heat.zip in
/tmp directory from the templates passed by the user.
Looks like there is a bug in the code that creates the zip file. The size
of the zip file is constantly increasing and ends up consuming the entire
drive.
For robot I am using the below version of the image:
"image: onap/testsuite:1.5.4"
Are there some fixes in robot scripts that have been done beyond this
version ?
Below are some logs from robot pod taken while running the ovp validation
tests.
-------------------
aarna@anod-master:~$ kubectl get pod -n onap -o=wide | grep robot
dev-robot-robot-6d444f4cdd-zsfj7 1/1
Running 0 112m 10.42.3.41 dublin03 <none>
<none>
aarna@anod-master:~$ kubectl exec -it -n onap
dev-robot-robot-6d444f4cdd-zsfj7 -- /bin/bash
root@dev-robot-robot-6d444f4cdd-zsfj7:/# ls
bin boot dev etc home lib lib64 media mnt opt proc root run
sbin share srv sys tmp usr var
root@dev-robot-robot-6d444f4cdd-zsfj7:/# df -Th
Filesystem Type Size Used
Avail Use% Mounted on
overlay overlay 584G 163G
421G 28% /
tmpfs tmpfs 64M 0
64M 0% /dev
tmpfs tmpfs 16G 0
16G 0% /sys/fs/cgroup
/dev/vda1 ext4 584G 163G
421G 28% /etc/localtime
tmpfs tmpfs 16G 11M
16G 1% /dev/shm
10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 10G
1.9T 1% /share/logs
tmpfs tmpfs 16G 12K
16G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs tmpfs 16G 0
16G 0% /proc/acpi
tmpfs tmpfs 16G 0
16G 0% /proc/scsi
tmpfs tmpfs 16G 0
16G 0% /sys/firmware
root@dev-robot-robot-6d444f4cdd-zsfj7:/# df -Th
Filesystem Type Size Used
Avail Use% Mounted on
overlay overlay 584G 164G
420G 29% /
tmpfs tmpfs 64M 0
64M 0% /dev
tmpfs tmpfs 16G 0
16G 0% /sys/fs/cgroup
/dev/vda1 ext4 584G 164G
420G 29% /etc/localtime
tmpfs tmpfs 16G 11M
16G 1% /dev/shm
10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 10G
1.9T 1% /share/logs
tmpfs tmpfs 16G 12K
16G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs tmpfs 16G 0
16G 0% /proc/acpi
tmpfs tmpfs 16G 0
16G 0% /proc/scsi
tmpfs tmpfs 16G 0
16G 0% /sys/firmware
root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates# ls -lh
total 80G
-rw-rw-r-- 1 1000 1000 1.4K Mar 31 12:46 base.env
-rw-rw-r-- 1 1000 1000 16K Apr 2 07:06 base.yaml
-rw-r--r-- 1 root root 80G Apr 2 14:18 tmp_heat.zip
root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates# ls -lh
total 81G
-rw-rw-r-- 1 1000 1000 1.4K Mar 31 12:46 base.env
-rw-rw-r-- 1 1000 1000 16K Apr 2 07:06 base.yaml
-*rw-r--r-- 1 root root 81G Apr 2 14:18 tmp_heat.zip*
root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates# file
tmp_heat.zip
bash: file: command not found
.
.
root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates# ls -lh
total 101G
-rw-rw-r-- 1 1000 1000 1.4K Mar 31 12:46 base.env
-rw-rw-r-- 1 1000 1000 16K Apr 2 07:06 base.yaml
-rw-r--r-- 1 root root 101G Apr 2 14:20 tmp_heat.zip
root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp# df -Th
Filesystem Type Size Used
Avail Use% Mounted on
overlay overlay 584G 209G
375G 36% /
tmpfs tmpfs 64M 0
64M 0% /dev
tmpfs tmpfs 16G 0
16G 0% /sys/fs/cgroup
/dev/vda1 ext4 584G 209G
375G 36% /etc/localtime
tmpfs tmpfs 16G 11M
16G 1% /dev/shm
10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 11G
1.9T 1% /share/logs
tmpfs tmpfs 16G 12K
16G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs tmpfs 16G 0
16G 0% /proc/acpi
tmpfs tmpfs 16G 0
16G 0% /proc/scsi
tmpfs tmpfs 16G 0
16G 0% /sys/firmware
--------------------------
thanks,
-RP Mishra
On Mon, Mar 30, 2020 at 6:30 PM FREEMAN, BRIAN D <[email protected]> wrote:
> Also if the problem is you are filling up /tmp inside your robot container
> try commenting out this line inside your robot contaiener
>
>
>
> /var/opt/ONAP/robot/resources/ test_templates/model_test_template.robot
>
> #Download CSAR ${catalog_service_id}
>
>
>
>
>
> This will skip the step of downloading the CSAR to the /tmp directory of
> your robot container which if the csar is large could be the problem.
>
> My robot continer is at 32% disk usage after doing 20 or so vFW creations
> but yours is at 45% so you need to see why your robot container filesystem
> seems to have a higher utilization.
>
>
>
>
>
> Brian
>
>
>
>
>
> *From:* FREEMAN, BRIAN D
> *Sent:* Monday, March 30, 2020 8:54 AM
> *To:* 'Rajendra P Mishra (RP)' <[email protected]>
> *Cc:* '[email protected]' <[email protected]>; STARK,
> STEVEN <[email protected]>
> *Subject:* RE: [onap-discuss] [OVP][SO] Heatbridge failure while doing
> ovp validation on reference VNF
>
>
>
> My azure install was not a full onap install so you may want to turn off
> optional components if you don’t have enough total ephemeral storage across
> all your nodes.
>
>
>
> Brian
>
>
>
>
>
> *From:* FREEMAN, BRIAN D
> *Sent:* Monday, March 30, 2020 8:53 AM
> *To:* Rajendra P Mishra (RP) <[email protected]>
> *Cc:* [email protected]; STARK, STEVEN <[email protected]>
> *Subject:* RE: [onap-discuss] [OVP][SO] Heatbridge failure while doing
> ovp validation on reference VNF
>
>
>
> Look at kubectl describe nodes and see how close to your ephemeral storage
> capacity you are.
>
>
>
>
>
> Some stats from my Azure environment.
>
>
>
> root@onapdldk-robot-robot-577df6887c-njrc5:/# df -Th
>
> Filesystem Type Size Used
> Avail Use% Mounted on
>
> overlay overlay 97G
> 31G 67G 32% /
>
> tmpfs tmpfs 64M
> 0 64M 0% /dev
>
> tmpfs tmpfs 14G
> 0 14G 0% /sys/fs/cgroup
>
> tmpfs tmpfs 14G
> 158M 14G 2% /dev/shm
>
> /dev/sda1 ext4 97G
> 31G 67G 32% /etc/localtime
>
> 173.163.0.4:/dockerdata-nfs/onapdldk-robot/robot/logs nfs4 497G
> 84G 413G 17% /share/logs
>
> tmpfs tmpfs 14G
> 12K 14G 1% /run/secrets/kubernetes.io/serviceaccount
>
> tmpfs tmpfs 14G
> 0 14G 0% /proc/acpi
>
> tmpfs tmpfs 14G
> 0 14G 0% /proc/scsi
>
> tmpfs tmpfs 14G
> 0 14G 0% /sys/firmware
>
>
>
>
>
> bf1936@njcdtl01bf1936:~/ONAP/AZURE/ONAPDlDK/oom/kubernetes/robot$ kubectl
> -n onap top nodes
>
> NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
>
> aks-nodepool1-97482552-0 883m 11% 18723Mi 78%
>
> aks-nodepool1-97482552-1 4375m 55% 17451Mi 72%
>
> aks-nodepool1-97482552-2 1849m 23% 12245Mi 51%
>
> aks-nodepool1-97482552-3 1478m 18% 17252Mi 71%
>
> aks-nodepool1-97482552-4 5382m 68% 21750Mi 90%
>
> aks-nodepool1-97482552-5 4617m 59% 14628Mi 61%
>
> aks-nodepool1-97482552-6 1179m 15% 19062Mi 79%
>
>
>
> kubectl describe nodes
>
>
>
> For node that robot is running:
>
> Capacity:
>
> attachable-volumes-azure-disk: 32
>
> cpu: 8
>
> ephemeral-storage: 101445900Ki
>
> hugepages-1Gi: 0
>
> hugepages-2Mi: 0
>
> memory: 28789864Ki
>
> pods: 110
>
> Allocatable:
>
> attachable-volumes-azure-disk: 32
>
> cpu: 7820m
>
> ephemeral-storage: 93492541286
>
> hugepages-1Gi: 0
>
> hugepages-2Mi: 0
>
> memory: 24541288Ki
>
> pods: 110
>
>
>
> *From:* Rajendra P Mishra (RP) <[email protected]>
> *Sent:* Sunday, March 29, 2020 12:06 PM
> *To:* FREEMAN, BRIAN D <[email protected]>
> *Cc:* [email protected]; STARK, STEVEN <[email protected]>
> *Subject:* Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing
> ovp validation on reference VNF
>
>
>
> NFS drive is 2TB
>
>
>
> Here is the file system space usage in one of the node.
>
> ----
>
> aarna@anod-master:~$ df -Th
> Filesystem Type Size Used Avail Use% Mounted on
> udev devtmpfs 16G 0 16G 0% /dev
> tmpfs tmpfs 3.2G 1.3M 3.2G 1% /run
> /dev/vda1 ext4 196G 87G 110G 45% /
> tmpfs tmpfs 16G 0 16G 0% /dev/shm
> tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock
> tmpfs tmpfs 16G 0 16G 0%
> /sys/fs/cgroup
> /dev/vda15 vfat 105M 3.6M 101M 4% /boot/efi
> 10.240.71.179:/dockerdata-nfs nfs4 1.9T 23G 1.8T 2%
> /dockerdata-nfs
> tmpfs tmpfs 3.2G 0 3.2G 0%
> /run/user/1000
>
> ---------
>
>
>
> Disk usage inside robot is here.
>
>
>
> -------
>
> aarna@anod-master:~$ kubectl get pod -n onap | grep robot
> dev-robot-robot-6d444f4cdd-rrwqj 0/1
> Evicted 0 5h2m
> dev-robot-robot-6d444f4cdd-sn5sl 1/1
> Running 0 4h47m
> aarna@anod-master:~$ kubectl exec -it -n onap
> dev-robot-robot-6d444f4cdd-sn5sl -- /bin/bash
> root@dev-robot-robot-6d444f4cdd-sn5sl:/# df -Th
> Filesystem Type Size Used
> Avail Use% Mounted on
> overlay overlay 196G 88G
> 109G 45% /
> tmpfs tmpfs 64M 0
> 64M 0% /dev
> tmpfs tmpfs 16G 0
> 16G 0% /sys/fs/cgroup
> /dev/vda1 ext4 196G 88G
> 109G 45% /etc/localtime
> tmpfs tmpfs 16G 0
> 16G 0% /dev/shm
> 10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 23G
> 1.8T 2% /share/logs
> tmpfs tmpfs 16G 12K
> 16G 1% /run/secrets/kubernetes.io/serviceaccount
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Vk1goR9Pg3FUGQEz_PEKUp_E_Y4jBvPuiwnpGZijrNM&s=RLmzIwNVRpfgGrovLp8mIHGvUoruK9qxOUINwcTCuaA&e=>
> tmpfs tmpfs 16G 0
> 16G 0% /proc/acpi
> tmpfs tmpfs 16G 0
> 16G 0% /proc/scsi
> tmpfs tmpfs 16G 0
> 16G 0% /sys/firmware
>
> -----------
>
>
>
> For my ONAP setup, each node is hosted on a VM. Each VM has around 196GB
> root file system.
>
> NFS is setup on a seperate server outside of these VMs.
>
>
>
> I am trying to increase the size of the root file system from 196GB to
> 400GB.
>
>
>
> What I do not understand is why 196GB is not enough ? and where is the
> space getting consumed.
>
>
>
> thanks,
>
> -RP Mishra
>
>
>
>
>
> On Sun, Mar 29, 2020 at 4:44 PM FREEMAN, BRIAN D <[email protected]> wrote:
>
> I see now 27g out of 2tb it must be your k8 config is somehow different .
>
>
>
> What does disk usage inside the robot container look like before it is
> evicted ?
>
>
>
> When you created the k8 nodes how much storage did you give eack k8 ?
>
>
>
> Maybe put robot on its own k8 host ?
>
>
>
>
>
> Sent via the Samsung Galaxy S8, an AT&T 5G Evolution capable smartphone
>
>
>
>
>
>
>
> -------- Original message --------
>
> From: Rajendra Prasad Mishra <[email protected]>
>
> Date: 3/29/20 6:39 AM (GMT-05:00)
>
> To: "FREEMAN, BRIAN D" <[email protected]>
>
> Cc: [email protected], "STARK, STEVEN" <[email protected]>
>
> Subject: Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing ovp
> validation on reference VNF
>
>
>
> Hi Brian, Steve,
>
> Deleting /share/logs and /dockerdata-nfs/dev-robot/robot/logs/ is
> not helping. Robot pod is still getting killed due to space problems.
>
>
>
> I had to re-install the whole ONAP to get rid of the problem. But it comes
> back soon once I run some larger VNF (like vFW).
>
>
>
> I tried increasing the storage allocated to robot in the values.yaml file
> before installing ONAP, but I still get the same space problem when I run
> OVP tests.
>
> (below is the diff of the values.yaml file)
>
>
>
> Is there a clean way to increase the "*ephemeral-storage*" for *robot *to
> some large value so that I can do multiple runs of complex VNF
> without running into space issues.
>
> My /dockerdata-nfs is mounted on a 2TB SSD and only 21GB is used, so we
> have enough space available.
>
>
>
> ---
>
> aarna@anod-master:~/oom/kubernetes/robot$ git diff values.yaml
>
> diff --git a/values.yaml b/values.yaml
> index 3696057..9cda2c1 100644
> --- a/values.yaml
> +++ b/values.yaml
> @@ -23,12 +23,12 @@ global: # global defaults
> # application image
> repository: nexus3.onap.org:10001
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=ZlV4kOGHmpRjjv6cp1gx66F9Dv4gkllAc6ExmLAc6L4&e=>
> image: onap/testsuite:1.5.4
> -pullPolicy: Always
> +pullPolicy: IfNotPresent
>
> ubuntuInitImage: oomk8s/ubuntu-init:2.0.0
>
> # flag to enable debugging - application support required
> -debugEnabled: false
> +debugEnabled: true
>
> #################################################################
> # Application configuration defaults.
> @@ -254,6 +254,6 @@ persistence:
> ##
> # storageClass: "-"
> accessMode: ReadWriteMany
> - size: 2Gi
> + size: 4Gi
> mountPath: /dockerdata-nfs
> mountSubPath: robot/logs
>
> -----------------
>
>
>
> Here is the pod that is killed.
>
>
>
> -----------
>
> aarna@anod-master:~$ kubectl describe pod -n onap
> dev-robot-robot-6d444f4cdd-bkm4x
> Name: dev-robot-robot-6d444f4cdd-bkm4x
> Namespace: onap
> Priority: 0
> Node: dublin05/
> Start Time: Fri, 27 Mar 2020 09:30:42 +0000
> Labels: app=robot
> pod-template-hash=6d444f4cdd
> release=dev-robot
> Annotations: cni.projectcalico.org/podIP
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__cni.projectcalico.org_podIP&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=4Ie1PTpTbBekcYyflcwb1Ppzy3DltykU2VI8WFIqGo4&e=>:
> 10.42.5.41/32
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.42.5.41_32&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=PlvGS8KEgxKtPrUx0Fxgu1icZTwTwj_p31rCbEqFEVw&e=>
> Status: Failed
> Reason: Evicted
> Message: The node was low on resource: ephemeral-storage. Container
> robot was using 125078316Ki, which exceeds its request of 0.
> IP:
> Controlled By: ReplicaSet/dev-robot-robot-6d444f4cdd
> Containers:
> robot:
> Image: nexus3.onap.org:10001/onap/testsuite:1.5.4
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_testsuite-3A1.5.4&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=uCMjhaGdBG8rdsixT8QAq7iFAB_mV4aWob5OD2Ou_is&e=>
> Port: 88/TCP
> Host Port: 0/TCP
> Liveness: tcp-socket :88 delay=180s timeout=1s period=10s
> #success=1 #failure=3
> Readiness: tcp-socket :88 delay=180s timeout=1s period=10s
> #success=1 #failure=3
> Environment: <none>
> Mounts:
> /dev/shm from dshm (rw)
> /etc/localtime from localtime (ro)
> /share/config from robot-eteshare (rw)
> /share/logs from robot-logs (rw)
> /var/run/secrets/kubernetes.io/serviceaccount
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=62KOK4XwlhhxKcBn85bXtwlUakwbOIegXv08sO1GlqE&e=>
> from default-token-qd76w (ro)
> Volumes:
> robot-logs:
> Type: PersistentVolumeClaim (a reference to a
> PersistentVolumeClaim in the same namespace)
> ClaimName: dev-robot-robot
> ReadOnly: false
> dshm:
> Type: EmptyDir (a temporary directory that shares a pod's
> lifetime)
> Medium: Memory
> SizeLimit: <unset>
> localtime:
> Type: HostPath (bare host directory volume)
> Path: /etc/localtime
> HostPathType:
> robot-eteshare:
> Type: ConfigMap (a volume populated by a ConfigMap)
> Name: dev-robot-robot-eteshare-configmap
> Optional: false
> default-token-qd76w:
> Type: Secret (a volume populated by a Secret)
> SecretName: default-token-qd76w
> Optional: false
> QoS Class: BestEffort
> Node-Selectors: <none>
> Tolerations: node.kubernetes.io/not-ready:NoExecute
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_not-2Dready-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=5IOJYuAKepGEJxmwD8wQHsOePlpnB2x3gbU9Y5XCoIE&e=>
> for 300s
> node.kubernetes.io/unreachable:NoExecute
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_unreachable-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=cNdW_67uaUUVC_JSvX41xLh18gDEnPp4p7za_daCxH0&e=>
> for 300s
> Events:
> Type Reason Age From Message
> ---- ------ ---- ---- -------
> Warning Evicted 25m kubelet, dublin05 The node was low on resource:
> ephemeral-storage. Container robot was using 125078316Ki, which exceeds its
> request of 0.
> Normal Killing 25m kubelet, dublin05 Killing container with id
> docker://robot:Need to kill Pod
>
> ------------------------
>
>
>
> thanks for your help,
>
> -RP Mishra
>
>
>
> On Mon, Mar 23, 2020 at 7:07 PM FREEMAN, BRIAN D <[email protected]> wrote:
>
> IF you have been doing a lot of testing the robot file system can get
> pretty large (500Meg per test in some cases)
>
>
>
> Cleanup /share/logs inside the container (that is log files stored on the
> nfs) and /tmp/csar where the distributed models are stored for use in
> testing like vCPE
>
>
>
> Brian
>
>
>
>
>
> *From:* [email protected] <[email protected]> *On
> Behalf Of *Rajendra Prasad Mishra
> *Sent:* Monday, March 23, 2020 9:25 AM
> *To:* STARK, STEVEN <[email protected]>
> *Cc:* [email protected]
> *Subject:* Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing
> ovp validation on reference VNF
>
>
>
> Thanks Steve, that worked. I no longer see the openstack authentication
> issue.
>
>
>
> However, I am seeing the robot storage issues. When I run the validation
> test, robot pod is killed because it is consuming more "ephemeral-storage"
> then allocated.
>
> Do you know how to increase the " ephemeral-storage" for robot pod ? This
> is a standard El Alto installation with no change in default configuration.
>
> Why does validation tests take up so much of space (127421328Ki)
>
>
>
> -----------------
>
> aarna@anod-master:~/oom/kubernetes/robot$ kubectl get pod -n onap | grep
> robot
> dev-robot-robot-6d444f4cdd-94kkk 1/1
> Running 0 3m49s
> dev-robot-robot-6d444f4cdd-vspc9 0/1
> Evicted 0 3d15h
> aarna@anod-master:~/oom/kubernetes/robot$ kubectl describe pod -n onap
> dev-robot-robot-6d444f4cdd-vspc9
> Name: dev-robot-robot-6d444f4cdd-vspc9
> Namespace: onap
> Priority: 0
> Node: dublin06/
> Start Time: Thu, 19 Mar 2020 15:24:57 +0000
> Labels: app=robot
> pod-template-hash=6d444f4cdd
> release=dev-robot
> Annotations: cni.projectcalico.org/podIP
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__cni.projectcalico.org_podIP&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=MLwHGSQ5pPlDSCtDzE_p7lM5vcEUhSDD8zDoC3S4vn4&e=>:
> 10.42.1.56/32
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.42.1.56_32&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=Pbnhw9Cxfa2A6klTNVsFKrsskeFRF0guKda-qrkjVC4&e=>
> Status: Failed
> Reason: Evicted
> Message: * The node was low on resource: ephemeral-storage.
> Container robot was using 127421328Ki, which exceeds its request of 0.*
> IP:
> Controlled By: ReplicaSet/dev-robot-robot-6d444f4cdd
> Containers:
> robot:
> Image: nexus3.onap.org:10001/onap/testsuite:1.5.4
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_testsuite-3A1.5.4&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=zGRkR7bU0ERGCcc4ahXTnuiECXcRN__EkBCF_EOPrsM&e=>
> Port: 88/TCP
> Host Port: 0/TCP
> Liveness: tcp-socket :88 delay=180s timeout=1s period=10s
> #success=1 #failure=3
> Readiness: tcp-socket :88 delay=180s timeout=1s period=10s
> #success=1 #failure=3
> Environment: <none>
> Mounts:
> /dev/shm from dshm (rw)
> /etc/localtime from localtime (ro)
> /share/config from robot-eteshare (rw)
> /share/logs from robot-logs (rw)
> /var/run/secrets/kubernetes.io/serviceaccount
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=FAoROrnwGV5yFeLt7ZfIM_LtdbNGb7eMT0YJCiD7XLw&e=>
> from default-token-2mz7g (ro)
> Volumes:
> robot-logs:
> Type: PersistentVolumeClaim (a reference to a
> PersistentVolumeClaim in the same namespace)
> ClaimName: dev-robot-robot
> ReadOnly: false
> dshm:
> Type: EmptyDir (a temporary directory that shares a pod's
> lifetime)
> Medium: Memory
> SizeLimit: <unset>
> localtime:
> Type: HostPath (bare host directory volume)
> Path: /etc/localtime
> HostPathType:
> robot-eteshare:
> Type: ConfigMap (a volume populated by a ConfigMap)
> Name: dev-robot-robot-eteshare-configmap
> Optional: false
> default-token-2mz7g:
> Type: Secret (a volume populated by a Secret)
> SecretName: default-token-2mz7g
> Optional: false
> QoS Class: BestEffort
> Node-Selectors: <none>
> Tolerations: node.kubernetes.io/not-ready:NoExecute
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_not-2Dready-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=0b-LBWz7xZSxyHD54a1S970SRnyV92DCFb8OC5hGa6g&e=>
> for 300s
> node.kubernetes.io/unreachable:NoExecute
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_unreachable-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=lUcFt1b-6535RBbsO7aF5SFumNsJvLXBjdo1eIXnxgA&e=>
> for 300s
> Events:
> Type Reason Age From Message
> ---- ------ ---- ---- -------
>
> * Warning Evicted 4m26s kubelet, dublin06 The node was low on
> resource: ephemeral-storage. Container robot was using 127421328Ki, which
> exceeds its request of 0. Normal Killing 4m24s kubelet, dublin06
> Killing container with id docker://robot:Need to kill Pod*
>
> ----------------------------------
>
>
>
> thanks,
>
> -RPMishra
>
>
>
>
>
> On Fri, Mar 20, 2020 at 9:42 PM Steve Stark <[email protected]> wrote:
>
> Great that helps. It looks like instantiation worked, the part that's
> failing is some custom heat stack validation after the instantiation.
>
> I was able to replicate the failure you're getting. Can you change the
> admin_tenant value for your identity service to the tenant name instead of
> the tenant ID and try again? I think in your case the tenant name = admin.
>
> This is in the mariadb pod -
>
> MariaDB [catalogdb]> update identity_services set admin_tenant="<tenant
> name>" where id="<identity service id>";
>
> Thanks
> Steven
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20469): https://lists.onap.org/g/onap-discuss/message/20469
Mute This Topic: https://lists.onap.org/mt/72026138/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
