+ Mo Dear Michael,
I did noticed once last week a pull that failed for me immediately for no reason and had to restart. Mo, do you think you can join us for a quick debugging session some time today please? Thanks a ton! Jess On Mon Apr 30 11:56:28 2018, frank.obr...@amdocs.com wrote: > LF, > Hi, we have been seeing burst behavior for the last week. This > morning in the integration lab session we see this on multiple > machines. > Downloads are fast when they are running - but completely stall > every min or so for 30-60 sec. > > For example pulling the dmaap container manually - a single pull took > 3 min on one system - 18 on antoher and had sections where the job ran > about 1 to 5 with bursts to 25 Mb/sec (max is usually 10Mb/sec per > stream - up to 5 in parallel - I have seen 25) - but there were times > 30-60 sec where the image download completely paused at 0Mb/sec. > > This behavior is different or is in combination with the traditional > slowdown we used to see - very slow pulls - this issue is new in that > it works fine during the burst - but has sections where all pulls > pause. To be frank - I did see this pause behavior briefly for a > couple seconds back even in Dec. > > > I propose we meet again to test downloads again and get some > volumetrics. > Some details on this and previous issues in > > https://jira.onap.org/browse/OOM-933 > > Time 1130 EST 20180430 > > see slowdown again after our last meeting - a burst behaviour > (burst/hold in 30-60 sec intervals) > > This in an azure system (clean) running a manual docker download - see > attachment on OOM-716<https://jira.onap.org/browse/OOM-716> > > root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0# > history > > 1 cd /var/lib/waagent/custom-script/download/0/ > > 2 ls > > 3 tail -f stdout > > 4 wget > https://jira.onap.org/secure/attachment/11444/prepull_docker_master.sh > > 6 sudo chmod 777 prepull_docker_master.sh > > 7 vi prepull_docker_master.sh > > 8 docker login -u docker -p docker nexus3.onap.org:10001 > > 9 docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > 10 vi prepull_docker_master.sh > > 11 docker pull nexus3.onap.org:10001/onap/vid:1.2.1 > > 12 docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > > > > > root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0# > docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > 1.1.4: Pulling from onap/dmaap/dmaap-mr > > 0c5a750448f5: Pull complete > > e2356a247318: Pull complete > > 12f105c650c3: Pull complete > > 46d4cec7ab72: Pull complete > > f5b0369aa7fc: Pull complete > > a93519a09c8b: Pull complete > > 73ffcbe01622: Waiting > > 071f37eca411: Waiting > > dc33933d1e9b: Waiting > > cc3a1fce1c35: Waiting > > ab292f5d0d16: Waiting > > 9cd00aac030f: Waiting > > > > > > 3 min > > root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0# > docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > 1.1.4: Pulling from onap/dmaap/dmaap-mr > > 0c5a750448f5: Pull complete > > e2356a247318: Pull complete > > 12f105c650c3: Pull complete > > 46d4cec7ab72: Pull complete > > f5b0369aa7fc: Pull complete > > a93519a09c8b: Pull complete > > 73ffcbe01622: Pull complete > > 071f37eca411: Pull complete > > dc33933d1e9b: Pull complete > > cc3a1fce1c35: Pull complete > > ab292f5d0d16: Pull complete > > 9cd00aac030f: Pull complete > > Digest: > sha256:8dff567ab978965f545fc1cadc8060cd34a2599ba586e786795469b32fc670ff > > Status: Downloaded newer image for > nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > testing full predeploy - started 1148 > > 1149 > > ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep pull > | wc -l > > 77 > > 10 sec below > > 32eda82f66ec: Downloading [================> > ] 96.86 MB/298.4 MB > > 32eda82f66ec: Downloading [====================> > ] 124 MB/298.4 MB > > 1153 > > ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep pull > | wc -l > > 75 > > This message and the information contained herein is proprietary and > confidential and subject to the Amdocs policy statement, > > you may review at https://www.amdocs.com/about/email-disclaimer > <https://www.amdocs.com/about/email-disclaimer> _______________________________________________ ONAP-TSC mailing list ONAP-TSC@lists.onap.org https://lists.onap.org/mailman/listinfo/onap-tsc
