Closing this ticket and replying on RT54849 Ryan is helping us adjusting the GC tasks so that they wont slow down our performance as well as adjusting the heap sizes.
Thanks! Jess On Mon Apr 30 14:17:17 2018, jwagantall wrote: > + Mo > > Dear Michael, > > I did noticed once last week a pull that failed for me immediately for > no reason and had to restart. > > Mo, do you think you can join us for a quick debugging session some > time today please? > > Thanks a ton! > Jess > > > > On Mon Apr 30 11:56:28 2018, [email protected] wrote: > > LF, > > Hi, we have been seeing burst behavior for the last week. This > > morning in the integration lab session we see this on multiple > > machines. > > Downloads are fast when they are running - but completely stall > > every min or so for 30-60 sec. > > > > For example pulling the dmaap container manually - a single pull took > > 3 min on one system - 18 on antoher and had sections where the job > > ran > > about 1 to 5 with bursts to 25 Mb/sec (max is usually 10Mb/sec per > > stream - up to 5 in parallel - I have seen 25) - but there were times > > 30-60 sec where the image download completely paused at 0Mb/sec. > > > > This behavior is different or is in combination with the traditional > > slowdown we used to see - very slow pulls - this issue is new in > > that > > it works fine during the burst - but has sections where all pulls > > pause. To be frank - I did see this pause behavior briefly for a > > couple seconds back even in Dec. > > > > > > I propose we meet again to test downloads again and get some > > volumetrics. > > Some details on this and previous issues in > > > > https://jira.onap.org/browse/OOM-933 > > > > Time 1130 EST 20180430 > > > > see slowdown again after our last meeting - a burst behaviour > > (burst/hold in 30-60 sec intervals) > > > > This in an azure system (clean) running a manual docker download - > > see > > attachment on OOM-716<https://jira.onap.org/browse/OOM-716> > > > > root@obrienondemand20180430:/var/lib/waagent/custom- > > script/download/0# > > history > > > > 1 cd /var/lib/waagent/custom-script/download/0/ > > > > 2 ls > > > > 3 tail -f stdout > > > > 4 wget > > https://jira.onap.org/secure/attachment/11444/prepull_docker_master.sh > > > > 6 sudo chmod 777 prepull_docker_master.sh > > > > 7 vi prepull_docker_master.sh > > > > 8 docker login -u docker -p docker nexus3.onap.org:10001 > > > > 9 docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > > > 10 vi prepull_docker_master.sh > > > > 11 docker pull nexus3.onap.org:10001/onap/vid:1.2.1 > > > > 12 docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > > > > > > > > > > > root@obrienondemand20180430:/var/lib/waagent/custom- > > script/download/0# > > docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > > > 1.1.4: Pulling from onap/dmaap/dmaap-mr > > > > 0c5a750448f5: Pull complete > > > > e2356a247318: Pull complete > > > > 12f105c650c3: Pull complete > > > > 46d4cec7ab72: Pull complete > > > > f5b0369aa7fc: Pull complete > > > > a93519a09c8b: Pull complete > > > > 73ffcbe01622: Waiting > > > > 071f37eca411: Waiting > > > > dc33933d1e9b: Waiting > > > > cc3a1fce1c35: Waiting > > > > ab292f5d0d16: Waiting > > > > 9cd00aac030f: Waiting > > > > > > > > > > > > 3 min > > > > root@obrienondemand20180430:/var/lib/waagent/custom- > > script/download/0# > > docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > > > 1.1.4: Pulling from onap/dmaap/dmaap-mr > > > > 0c5a750448f5: Pull complete > > > > e2356a247318: Pull complete > > > > 12f105c650c3: Pull complete > > > > 46d4cec7ab72: Pull complete > > > > f5b0369aa7fc: Pull complete > > > > a93519a09c8b: Pull complete > > > > 73ffcbe01622: Pull complete > > > > 071f37eca411: Pull complete > > > > dc33933d1e9b: Pull complete > > > > cc3a1fce1c35: Pull complete > > > > ab292f5d0d16: Pull complete > > > > 9cd00aac030f: Pull complete > > > > Digest: > > sha256:8dff567ab978965f545fc1cadc8060cd34a2599ba586e786795469b32fc670ff > > > > Status: Downloaded newer image for > > nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 > > > > testing full predeploy - started 1148 > > > > 1149 > > > > ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep > > pull > > | wc -l > > > > 77 > > > > 10 sec below > > > > 32eda82f66ec: Downloading [================> > > ] 96.86 MB/298.4 MB > > > > 32eda82f66ec: Downloading [====================> > > ] 124 MB/298.4 MB > > > > 1153 > > > > ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep > > pull > > | wc -l > > > > 75 > > > > This message and the information contained herein is proprietary and > > confidential and subject to the Amdocs policy statement, > > > > you may review at https://www.amdocs.com/about/email-disclaimer > > <https://www.amdocs.com/about/email-disclaimer> _______________________________________________ ONAP-TSC mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-tsc
