Hi Brian, I found the problem. The following code is buggy. We are creating the tmp_heat.zip in the same folder (template_directory) which we are also scanning. As a result the code gets into infinite recursion. (we can test it on the laptop, make sure the two template files are sufficiently large)
This function is called from robot when we run the OVP test. If I comment the sha256() function, my OVP test runs fine. https://github.com/onap/testsuite-python-testing-utils/blob/master/robotframework-onap/listeners/OVPListener.py --------------- def sha256(template_directory): heat_sha = None if os.path.exists(template_directory): zip_file = "{}/tmp_heat.zip".format(template_directory) with ZipFile(zip_file, "w") as zip_obj: for folder_name, subfolders, filenames in os.walk(template_directory): for filename in filenames: file_path = os.path.join(folder_name, filename) zip_obj.write(file_path) with open(zip_file, "rb") as f: bytes = f.read() heat_sha = hashlib.sha256(bytes).hexdigest() os.remove(zip_file) return heat_sha ----------------- thanks, -RP Mishra On Thu, Apr 2, 2020 at 8:31 PM FREEMAN, BRIAN D <[email protected]> wrote: > Nope must be something in your heat templates. > > > > ./ete-k8s.sh onap healthdist does a vFW heat template onboarding and > distribution (where the zip file is created) > > > > Try that on a fresh robot container and see what the file sizes are. > > > > Brian > > > > > > *From:* Rajendra P Mishra (RP) <[email protected]> > *Sent:* Thursday, April 2, 2020 10:48 AM > *To:* FREEMAN, BRIAN D <[email protected]> > *Cc:* [email protected]; STARK, STEVEN <[email protected]> > *Subject:* Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing > ovp validation on reference VNF > > > > I increased the size of the storage to 600GB for each node, however the > robot pod gets killed everytime I start the test. > > > > I noticed that while the test is running, it creates the tmp_heat.zip in > /tmp directory from the templates passed by the user. > > Looks like there is a bug in the code that creates the zip file. The size > of the zip file is constantly increasing and ends up consuming the entire > drive. > > > > For robot I am using the below version of the image: > > "image: onap/testsuite:1.5.4" > > > > Are there some fixes in robot scripts that have been done beyond this > version ? > > > > Below are some logs from robot pod taken while running the ovp validation > tests. > > ------------------- > > aarna@anod-master:~$ kubectl get pod -n onap -o=wide | grep robot > dev-robot-robot-6d444f4cdd-zsfj7 1/1 > Running 0 112m 10.42.3.41 dublin03 <none> > <none> > aarna@anod-master:~$ kubectl exec -it -n onap > dev-robot-robot-6d444f4cdd-zsfj7 -- /bin/bash > root@dev-robot-robot-6d444f4cdd-zsfj7:/# ls > bin boot dev etc home lib lib64 media mnt opt proc root run > sbin share srv sys tmp usr var > root@dev-robot-robot-6d444f4cdd-zsfj7:/# df -Th > Filesystem Type Size Used > Avail Use% Mounted on > overlay overlay 584G 163G > 421G 28% / > tmpfs tmpfs 64M 0 > 64M 0% /dev > tmpfs tmpfs 16G 0 > 16G 0% /sys/fs/cgroup > /dev/vda1 ext4 584G 163G > 421G 28% /etc/localtime > tmpfs tmpfs 16G 11M > 16G 1% /dev/shm > 10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 10G > 1.9T 1% /share/logs > tmpfs tmpfs 16G 12K > 16G 1% /run/secrets/kubernetes.io/serviceaccount > <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=oMYtzrFKWMW5gonEV9xw0_8KTkgTIeJgRRf6gqUErso&s=SF41ISVDlCh7coVwiOGQjmI7cAYxG1HntnPhNKvQxGE&e=> > tmpfs tmpfs 16G 0 > 16G 0% /proc/acpi > tmpfs tmpfs 16G 0 > 16G 0% /proc/scsi > tmpfs tmpfs 16G 0 > 16G 0% /sys/firmware > root@dev-robot-robot-6d444f4cdd-zsfj7:/# df -Th > Filesystem Type Size Used > Avail Use% Mounted on > overlay overlay 584G 164G > 420G 29% / > tmpfs tmpfs 64M 0 > 64M 0% /dev > tmpfs tmpfs 16G 0 > 16G 0% /sys/fs/cgroup > /dev/vda1 ext4 584G 164G > 420G 29% /etc/localtime > tmpfs tmpfs 16G 11M > 16G 1% /dev/shm > 10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 10G > 1.9T 1% /share/logs > tmpfs tmpfs 16G 12K > 16G 1% /run/secrets/kubernetes.io/serviceaccount > <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=oMYtzrFKWMW5gonEV9xw0_8KTkgTIeJgRRf6gqUErso&s=SF41ISVDlCh7coVwiOGQjmI7cAYxG1HntnPhNKvQxGE&e=> > tmpfs tmpfs 16G 0 > 16G 0% /proc/acpi > tmpfs tmpfs 16G 0 > 16G 0% /proc/scsi > tmpfs tmpfs 16G 0 > 16G 0% /sys/firmware > > root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates# ls -lh > total 80G > -rw-rw-r-- 1 1000 1000 1.4K Mar 31 12:46 base.env > -rw-rw-r-- 1 1000 1000 16K Apr 2 07:06 base.yaml > -rw-r--r-- 1 root root 80G Apr 2 14:18 tmp_heat.zip > root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates# ls -lh > total 81G > -rw-rw-r-- 1 1000 1000 1.4K Mar 31 12:46 base.env > -rw-rw-r-- 1 1000 1000 16K Apr 2 07:06 base.yaml > -*rw-r--r-- 1 root root 81G Apr 2 14:18 tmp_heat.zip* > root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates# file > tmp_heat.zip > bash: file: command not found > > . > > . > root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp/vnfdata.5074/templates# ls -lh > total 101G > -rw-rw-r-- 1 1000 1000 1.4K Mar 31 12:46 base.env > -rw-rw-r-- 1 1000 1000 16K Apr 2 07:06 base.yaml > -rw-r--r-- 1 root root 101G Apr 2 14:20 tmp_heat.zip > > > root@dev-robot-robot-6d444f4cdd-zsfj7:/tmp# df -Th > Filesystem Type Size Used > Avail Use% Mounted on > overlay overlay 584G 209G > 375G 36% / > tmpfs tmpfs 64M 0 > 64M 0% /dev > tmpfs tmpfs 16G 0 > 16G 0% /sys/fs/cgroup > /dev/vda1 ext4 584G 209G > 375G 36% /etc/localtime > tmpfs tmpfs 16G 11M > 16G 1% /dev/shm > 10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 11G > 1.9T 1% /share/logs > tmpfs tmpfs 16G 12K > 16G 1% /run/secrets/kubernetes.io/serviceaccount > <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=oMYtzrFKWMW5gonEV9xw0_8KTkgTIeJgRRf6gqUErso&s=SF41ISVDlCh7coVwiOGQjmI7cAYxG1HntnPhNKvQxGE&e=> > tmpfs tmpfs 16G 0 > 16G 0% /proc/acpi > tmpfs tmpfs 16G 0 > 16G 0% /proc/scsi > tmpfs tmpfs 16G 0 > 16G 0% /sys/firmware > > -------------------------- > > > > thanks, > > -RP Mishra > > > > > > > > > > On Mon, Mar 30, 2020 at 6:30 PM FREEMAN, BRIAN D <[email protected]> wrote: > > Also if the problem is you are filling up /tmp inside your robot container > try commenting out this line inside your robot contaiener > > > > /var/opt/ONAP/robot/resources/ test_templates/model_test_template.robot > > #Download CSAR ${catalog_service_id} > > > > > > This will skip the step of downloading the CSAR to the /tmp directory of > your robot container which if the csar is large could be the problem. > > My robot continer is at 32% disk usage after doing 20 or so vFW creations > but yours is at 45% so you need to see why your robot container filesystem > seems to have a higher utilization. > > > > > > Brian > > > > > > *From:* FREEMAN, BRIAN D > *Sent:* Monday, March 30, 2020 8:54 AM > *To:* 'Rajendra P Mishra (RP)' <[email protected]> > *Cc:* '[email protected]' <[email protected]>; STARK, > STEVEN <[email protected]> > *Subject:* RE: [onap-discuss] [OVP][SO] Heatbridge failure while doing > ovp validation on reference VNF > > > > My azure install was not a full onap install so you may want to turn off > optional components if you don’t have enough total ephemeral storage across > all your nodes. > > > > Brian > > > > > > *From:* FREEMAN, BRIAN D > *Sent:* Monday, March 30, 2020 8:53 AM > *To:* Rajendra P Mishra (RP) <[email protected]> > *Cc:* [email protected]; STARK, STEVEN <[email protected]> > *Subject:* RE: [onap-discuss] [OVP][SO] Heatbridge failure while doing > ovp validation on reference VNF > > > > Look at kubectl describe nodes and see how close to your ephemeral storage > capacity you are. > > > > > > Some stats from my Azure environment. > > > > root@onapdldk-robot-robot-577df6887c-njrc5:/# df -Th > > Filesystem Type Size Used > Avail Use% Mounted on > > overlay overlay 97G > 31G 67G 32% / > > tmpfs tmpfs 64M > 0 64M 0% /dev > > tmpfs tmpfs 14G > 0 14G 0% /sys/fs/cgroup > > tmpfs tmpfs 14G > 158M 14G 2% /dev/shm > > /dev/sda1 ext4 97G > 31G 67G 32% /etc/localtime > > 173.163.0.4:/dockerdata-nfs/onapdldk-robot/robot/logs nfs4 497G > 84G 413G 17% /share/logs > > tmpfs tmpfs 14G > 12K 14G 1% /run/secrets/kubernetes.io/serviceaccount > <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=oMYtzrFKWMW5gonEV9xw0_8KTkgTIeJgRRf6gqUErso&s=SF41ISVDlCh7coVwiOGQjmI7cAYxG1HntnPhNKvQxGE&e=> > > tmpfs tmpfs 14G > 0 14G 0% /proc/acpi > > tmpfs tmpfs 14G > 0 14G 0% /proc/scsi > > tmpfs tmpfs 14G > 0 14G 0% /sys/firmware > > > > > > bf1936@njcdtl01bf1936:~/ONAP/AZURE/ONAPDlDK/oom/kubernetes/robot$ kubectl > -n onap top nodes > > NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% > > aks-nodepool1-97482552-0 883m 11% 18723Mi 78% > > aks-nodepool1-97482552-1 4375m 55% 17451Mi 72% > > aks-nodepool1-97482552-2 1849m 23% 12245Mi 51% > > aks-nodepool1-97482552-3 1478m 18% 17252Mi 71% > > aks-nodepool1-97482552-4 5382m 68% 21750Mi 90% > > aks-nodepool1-97482552-5 4617m 59% 14628Mi 61% > > aks-nodepool1-97482552-6 1179m 15% 19062Mi 79% > > > > kubectl describe nodes > > > > For node that robot is running: > > Capacity: > > attachable-volumes-azure-disk: 32 > > cpu: 8 > > ephemeral-storage: 101445900Ki > > hugepages-1Gi: 0 > > hugepages-2Mi: 0 > > memory: 28789864Ki > > pods: 110 > > Allocatable: > > attachable-volumes-azure-disk: 32 > > cpu: 7820m > > ephemeral-storage: 93492541286 > > hugepages-1Gi: 0 > > hugepages-2Mi: 0 > > memory: 24541288Ki > > pods: 110 > > > > *From:* Rajendra P Mishra (RP) <[email protected]> > *Sent:* Sunday, March 29, 2020 12:06 PM > *To:* FREEMAN, BRIAN D <[email protected]> > *Cc:* [email protected]; STARK, STEVEN <[email protected]> > *Subject:* Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing > ovp validation on reference VNF > > > > NFS drive is 2TB > > > > Here is the file system space usage in one of the node. > > ---- > > aarna@anod-master:~$ df -Th > Filesystem Type Size Used Avail Use% Mounted on > udev devtmpfs 16G 0 16G 0% /dev > tmpfs tmpfs 3.2G 1.3M 3.2G 1% /run > /dev/vda1 ext4 196G 87G 110G 45% / > tmpfs tmpfs 16G 0 16G 0% /dev/shm > tmpfs tmpfs 5.0M 0 5.0M 0% /run/lock > tmpfs tmpfs 16G 0 16G 0% > /sys/fs/cgroup > /dev/vda15 vfat 105M 3.6M 101M 4% /boot/efi > 10.240.71.179:/dockerdata-nfs nfs4 1.9T 23G 1.8T 2% > /dockerdata-nfs > tmpfs tmpfs 3.2G 0 3.2G 0% > /run/user/1000 > > --------- > > > > Disk usage inside robot is here. > > > > ------- > > aarna@anod-master:~$ kubectl get pod -n onap | grep robot > dev-robot-robot-6d444f4cdd-rrwqj 0/1 > Evicted 0 5h2m > dev-robot-robot-6d444f4cdd-sn5sl 1/1 > Running 0 4h47m > aarna@anod-master:~$ kubectl exec -it -n onap > dev-robot-robot-6d444f4cdd-sn5sl -- /bin/bash > root@dev-robot-robot-6d444f4cdd-sn5sl:/# df -Th > Filesystem Type Size Used > Avail Use% Mounted on > overlay overlay 196G 88G > 109G 45% / > tmpfs tmpfs 64M 0 > 64M 0% /dev > tmpfs tmpfs 16G 0 > 16G 0% /sys/fs/cgroup > /dev/vda1 ext4 196G 88G > 109G 45% /etc/localtime > tmpfs tmpfs 16G 0 > 16G 0% /dev/shm > 10.240.71.179:/dockerdata-nfs/dev-robot/robot/logs nfs4 1.9T 23G > 1.8T 2% /share/logs > tmpfs tmpfs 16G 12K > 16G 1% /run/secrets/kubernetes.io/serviceaccount > <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=Vk1goR9Pg3FUGQEz_PEKUp_E_Y4jBvPuiwnpGZijrNM&s=RLmzIwNVRpfgGrovLp8mIHGvUoruK9qxOUINwcTCuaA&e=> > tmpfs tmpfs 16G 0 > 16G 0% /proc/acpi > tmpfs tmpfs 16G 0 > 16G 0% /proc/scsi > tmpfs tmpfs 16G 0 > 16G 0% /sys/firmware > > ----------- > > > > For my ONAP setup, each node is hosted on a VM. Each VM has around 196GB > root file system. > > NFS is setup on a seperate server outside of these VMs. > > > > I am trying to increase the size of the root file system from 196GB to > 400GB. > > > > What I do not understand is why 196GB is not enough ? and where is the > space getting consumed. > > > > thanks, > > -RP Mishra > > > > > > On Sun, Mar 29, 2020 at 4:44 PM FREEMAN, BRIAN D <[email protected]> wrote: > > I see now 27g out of 2tb it must be your k8 config is somehow different . > > > > What does disk usage inside the robot container look like before it is > evicted ? > > > > When you created the k8 nodes how much storage did you give eack k8 ? > > > > Maybe put robot on its own k8 host ? > > > > > > Sent via the Samsung Galaxy S8, an AT&T 5G Evolution capable smartphone > > > > > > > > -------- Original message -------- > > From: Rajendra Prasad Mishra <[email protected]> > > Date: 3/29/20 6:39 AM (GMT-05:00) > > To: "FREEMAN, BRIAN D" <[email protected]> > > Cc: [email protected], "STARK, STEVEN" <[email protected]> > > Subject: Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing ovp > validation on reference VNF > > > > Hi Brian, Steve, > > Deleting /share/logs and /dockerdata-nfs/dev-robot/robot/logs/ is > not helping. Robot pod is still getting killed due to space problems. > > > > I had to re-install the whole ONAP to get rid of the problem. But it comes > back soon once I run some larger VNF (like vFW). > > > > I tried increasing the storage allocated to robot in the values.yaml file > before installing ONAP, but I still get the same space problem when I run > OVP tests. > > (below is the diff of the values.yaml file) > > > > Is there a clean way to increase the "*ephemeral-storage*" for *robot *to > some large value so that I can do multiple runs of complex VNF > without running into space issues. > > My /dockerdata-nfs is mounted on a 2TB SSD and only 21GB is used, so we > have enough space available. > > > > --- > > aarna@anod-master:~/oom/kubernetes/robot$ git diff values.yaml > > diff --git a/values.yaml b/values.yaml > index 3696057..9cda2c1 100644 > --- a/values.yaml > +++ b/values.yaml > @@ -23,12 +23,12 @@ global: # global defaults > # application image > repository: nexus3.onap.org:10001 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=ZlV4kOGHmpRjjv6cp1gx66F9Dv4gkllAc6ExmLAc6L4&e=> > image: onap/testsuite:1.5.4 > -pullPolicy: Always > +pullPolicy: IfNotPresent > > ubuntuInitImage: oomk8s/ubuntu-init:2.0.0 > > # flag to enable debugging - application support required > -debugEnabled: false > +debugEnabled: true > > ################################################################# > # Application configuration defaults. > @@ -254,6 +254,6 @@ persistence: > ## > # storageClass: "-" > accessMode: ReadWriteMany > - size: 2Gi > + size: 4Gi > mountPath: /dockerdata-nfs > mountSubPath: robot/logs > > ----------------- > > > > Here is the pod that is killed. > > > > ----------- > > aarna@anod-master:~$ kubectl describe pod -n onap > dev-robot-robot-6d444f4cdd-bkm4x > Name: dev-robot-robot-6d444f4cdd-bkm4x > Namespace: onap > Priority: 0 > Node: dublin05/ > Start Time: Fri, 27 Mar 2020 09:30:42 +0000 > Labels: app=robot > pod-template-hash=6d444f4cdd > release=dev-robot > Annotations: cni.projectcalico.org/podIP > <https://urldefense.proofpoint.com/v2/url?u=http-3A__cni.projectcalico.org_podIP&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=4Ie1PTpTbBekcYyflcwb1Ppzy3DltykU2VI8WFIqGo4&e=>: > 10.42.5.41/32 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.42.5.41_32&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=PlvGS8KEgxKtPrUx0Fxgu1icZTwTwj_p31rCbEqFEVw&e=> > Status: Failed > Reason: Evicted > Message: The node was low on resource: ephemeral-storage. Container > robot was using 125078316Ki, which exceeds its request of 0. > IP: > Controlled By: ReplicaSet/dev-robot-robot-6d444f4cdd > Containers: > robot: > Image: nexus3.onap.org:10001/onap/testsuite:1.5.4 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_testsuite-3A1.5.4&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=uCMjhaGdBG8rdsixT8QAq7iFAB_mV4aWob5OD2Ou_is&e=> > Port: 88/TCP > Host Port: 0/TCP > Liveness: tcp-socket :88 delay=180s timeout=1s period=10s > #success=1 #failure=3 > Readiness: tcp-socket :88 delay=180s timeout=1s period=10s > #success=1 #failure=3 > Environment: <none> > Mounts: > /dev/shm from dshm (rw) > /etc/localtime from localtime (ro) > /share/config from robot-eteshare (rw) > /share/logs from robot-logs (rw) > /var/run/secrets/kubernetes.io/serviceaccount > <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=62KOK4XwlhhxKcBn85bXtwlUakwbOIegXv08sO1GlqE&e=> > from default-token-qd76w (ro) > Volumes: > robot-logs: > Type: PersistentVolumeClaim (a reference to a > PersistentVolumeClaim in the same namespace) > ClaimName: dev-robot-robot > ReadOnly: false > dshm: > Type: EmptyDir (a temporary directory that shares a pod's > lifetime) > Medium: Memory > SizeLimit: <unset> > localtime: > Type: HostPath (bare host directory volume) > Path: /etc/localtime > HostPathType: > robot-eteshare: > Type: ConfigMap (a volume populated by a ConfigMap) > Name: dev-robot-robot-eteshare-configmap > Optional: false > default-token-qd76w: > Type: Secret (a volume populated by a Secret) > SecretName: default-token-qd76w > Optional: false > QoS Class: BestEffort > Node-Selectors: <none> > Tolerations: node.kubernetes.io/not-ready:NoExecute > <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_not-2Dready-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=5IOJYuAKepGEJxmwD8wQHsOePlpnB2x3gbU9Y5XCoIE&e=> > for 300s > node.kubernetes.io/unreachable:NoExecute > <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_unreachable-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=ujPOfc7-AgMLBQA_u0ZbKoJ-ZOlVJU4bhrOnx_O9wQc&s=cNdW_67uaUUVC_JSvX41xLh18gDEnPp4p7za_daCxH0&e=> > for 300s > Events: > Type Reason Age From Message > ---- ------ ---- ---- ------- > Warning Evicted 25m kubelet, dublin05 The node was low on resource: > ephemeral-storage. Container robot was using 125078316Ki, which exceeds its > request of 0. > Normal Killing 25m kubelet, dublin05 Killing container with id > docker://robot:Need to kill Pod > > ------------------------ > > > > thanks for your help, > > -RP Mishra > > > > On Mon, Mar 23, 2020 at 7:07 PM FREEMAN, BRIAN D <[email protected]> wrote: > > IF you have been doing a lot of testing the robot file system can get > pretty large (500Meg per test in some cases) > > > > Cleanup /share/logs inside the container (that is log files stored on the > nfs) and /tmp/csar where the distributed models are stored for use in > testing like vCPE > > > > Brian > > > > > > *From:* [email protected] <[email protected]> *On > Behalf Of *Rajendra Prasad Mishra > *Sent:* Monday, March 23, 2020 9:25 AM > *To:* STARK, STEVEN <[email protected]> > *Cc:* [email protected] > *Subject:* Re: [onap-discuss] [OVP][SO] Heatbridge failure while doing > ovp validation on reference VNF > > > > Thanks Steve, that worked. I no longer see the openstack authentication > issue. > > > > However, I am seeing the robot storage issues. When I run the validation > test, robot pod is killed because it is consuming more "ephemeral-storage" > then allocated. > > Do you know how to increase the " ephemeral-storage" for robot pod ? This > is a standard El Alto installation with no change in default configuration. > > Why does validation tests take up so much of space (127421328Ki) > > > > ----------------- > > aarna@anod-master:~/oom/kubernetes/robot$ kubectl get pod -n onap | grep > robot > dev-robot-robot-6d444f4cdd-94kkk 1/1 > Running 0 3m49s > dev-robot-robot-6d444f4cdd-vspc9 0/1 > Evicted 0 3d15h > aarna@anod-master:~/oom/kubernetes/robot$ kubectl describe pod -n onap > dev-robot-robot-6d444f4cdd-vspc9 > Name: dev-robot-robot-6d444f4cdd-vspc9 > Namespace: onap > Priority: 0 > Node: dublin06/ > Start Time: Thu, 19 Mar 2020 15:24:57 +0000 > Labels: app=robot > pod-template-hash=6d444f4cdd > release=dev-robot > Annotations: cni.projectcalico.org/podIP > <https://urldefense.proofpoint.com/v2/url?u=http-3A__cni.projectcalico.org_podIP&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=MLwHGSQ5pPlDSCtDzE_p7lM5vcEUhSDD8zDoC3S4vn4&e=>: > 10.42.1.56/32 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.42.1.56_32&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=Pbnhw9Cxfa2A6klTNVsFKrsskeFRF0guKda-qrkjVC4&e=> > Status: Failed > Reason: Evicted > Message: * The node was low on resource: ephemeral-storage. > Container robot was using 127421328Ki, which exceeds its request of 0.* > IP: > Controlled By: ReplicaSet/dev-robot-robot-6d444f4cdd > Containers: > robot: > Image: nexus3.onap.org:10001/onap/testsuite:1.5.4 > <https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_testsuite-3A1.5.4&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=zGRkR7bU0ERGCcc4ahXTnuiECXcRN__EkBCF_EOPrsM&e=> > Port: 88/TCP > Host Port: 0/TCP > Liveness: tcp-socket :88 delay=180s timeout=1s period=10s > #success=1 #failure=3 > Readiness: tcp-socket :88 delay=180s timeout=1s period=10s > #success=1 #failure=3 > Environment: <none> > Mounts: > /dev/shm from dshm (rw) > /etc/localtime from localtime (ro) > /share/config from robot-eteshare (rw) > /share/logs from robot-logs (rw) > /var/run/secrets/kubernetes.io/serviceaccount > <https://urldefense.proofpoint.com/v2/url?u=http-3A__kubernetes.io_serviceaccount&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=FAoROrnwGV5yFeLt7ZfIM_LtdbNGb7eMT0YJCiD7XLw&e=> > from default-token-2mz7g (ro) > Volumes: > robot-logs: > Type: PersistentVolumeClaim (a reference to a > PersistentVolumeClaim in the same namespace) > ClaimName: dev-robot-robot > ReadOnly: false > dshm: > Type: EmptyDir (a temporary directory that shares a pod's > lifetime) > Medium: Memory > SizeLimit: <unset> > localtime: > Type: HostPath (bare host directory volume) > Path: /etc/localtime > HostPathType: > robot-eteshare: > Type: ConfigMap (a volume populated by a ConfigMap) > Name: dev-robot-robot-eteshare-configmap > Optional: false > default-token-2mz7g: > Type: Secret (a volume populated by a Secret) > SecretName: default-token-2mz7g > Optional: false > QoS Class: BestEffort > Node-Selectors: <none> > Tolerations: node.kubernetes.io/not-ready:NoExecute > <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_not-2Dready-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=0b-LBWz7xZSxyHD54a1S970SRnyV92DCFb8OC5hGa6g&e=> > for 300s > node.kubernetes.io/unreachable:NoExecute > <https://urldefense.proofpoint.com/v2/url?u=http-3A__node.kubernetes.io_unreachable-3ANoExecute&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=e3d1ehx3DI5AoMgDmi2Fzw&m=67kLb4PSwSX9Fj04dF9MkMMJ0A2LrX9Ze0p6tuDLxo4&s=lUcFt1b-6535RBbsO7aF5SFumNsJvLXBjdo1eIXnxgA&e=> > for 300s > Events: > Type Reason Age From Message > ---- ------ ---- ---- ------- > > * Warning Evicted 4m26s kubelet, dublin06 The node was low on > resource: ephemeral-storage. Container robot was using 127421328Ki, which > exceeds its request of 0. Normal Killing 4m24s kubelet, dublin06 > Killing container with id docker://robot:Need to kill Pod* > > ---------------------------------- > > > > thanks, > > -RPMishra > > > > > > On Fri, Mar 20, 2020 at 9:42 PM Steve Stark <[email protected]> wrote: > > Great that helps. It looks like instantiation worked, the part that's > failing is some custom heat stack validation after the instantiation. > > I was able to replicate the failure you're getting. Can you change the > admin_tenant value for your identity service to the tenant name instead of > the tenant ID and try again? I think in your case the tenant name = admin. > > This is in the mariadb pod - > > MariaDB [catalogdb]> update identity_services set admin_tenant="<tenant > name>" where id="<identity service id>"; > > Thanks > Steven > > > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20497): https://lists.onap.org/g/onap-discuss/message/20497 Mute This Topic: https://lists.onap.org/mt/72026138/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
