No change in pull speed - a good test create a clean ubuntu 16.04 vm (no previous pull) - install docker, login and pull an image Getting roughly 0.4MB/sec - at least 20x slower than in the past. This is on an AWS EC2 instance On another instance that is running the docker prepull - I saw the 502 timeouts in the nohup.out log - since the restart of nexus3 only 3 images in SO were downloaded for example - normally within 25 min I can download all 40G of onap images.
sudo curl https://releases.rancher.com/install-docker/17.03.sh | sh sudo usermod -aG docker ubuntu ubuntu@ip-172-31-10-98:~$ sudo docker login -u docker -p docker nexus3.onap.org:10001 Login Succeeded # get images from https://git.onap.org/integration/plain/version-manifest/src/main/resources/docker-manifest.csv?h=casablanca # use for example the first line onap/aaf/aaf_agent,2.1.8 # or the prepull script in https://gerrit.onap.org/r/#/c/74780/1/deploy/docker_prepull.sh sudo docker pull nexus3.onap.org:10001/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent 18d680d61657: Pulling fs layer 819d6de9e493: Downloading [======================================> ] 770.7 kB/1.012 MB https://jira.onap.org/browse/LOG-355 If I pull for example onap/ccsdk-dgbuilder-image:0.3.2 - which is 1G - but all but the last 2 layers of 345 and 71mb are not pulled -these just basically sit there - and look to take at least 10+ min based on progress as of 1334 EST ubuntu@ip-172-31-10-98:~$ sudo docker pull nexus3.onap.org:10001/onap/aaf/sms:3.0.1 3.0.1: Pulling from onap/aaf/sms c67f3896b22c: Pull complete 134fac3103be: Pull complete c7fdcaf7f439: Pull complete 590a963725eb: Pull complete 4df2c296b645: Pull complete 15dd1c6ebaf1: Pull complete 76eeb922b789: Pull complete Digest: sha256:d5b64947edb93848acacaa9820234aa29e58217db9f878886b7bafae00fdb436 Status: Downloaded newer image for nexus3.onap.org:10001/onap/aaf/sms:3.0.1 ubuntu@ip-172-31-10-98:~$ sudo docker pull nexus3.onap.org:10001/onap/ccsdk-dgbuilder-image:0.3.2 0.3.2: Pulling from onap/ccsdk-dgbuilder-image 18d680d61657: Already exists 0addb6fece63: Already exists 78e58219b215: Already exists eb6959a66df2: Already exists 4eeb11d61ff9: Downloading [> ] 5.266 MB/345.8 MB 0f81686fe963: Downloading [===> ] 5.266 MB/71.15 MB 4 min later ubuntu@ip-172-31-10-98:~$ sudo docker pull nexus3.onap.org:10001/onap/ccsdk-dgbuilder-image:0.3.2 0.3.2: Pulling from onap/ccsdk-dgbuilder-image 18d680d61657: Already exists 0addb6fece63: Already exists 78e58219b215: Already exists eb6959a66df2: Already exists 4eeb11d61ff9: Downloading [=> ] 10.54 MB/345.8 MB 0f81686fe963: Downloading [=======> ] 10.54 MB/71.15 MB -----Original Message----- From: [email protected] <[email protected]> On Behalf Of Kenny Paul via RT Sent: Tuesday, December 18, 2018 1:14 PM To: [email protected] Cc: [email protected] Subject: [onap-discuss] [ONAP Helpdesk #65794] nexus3 timing out After the restart, these jobs passed now: https://jenkins.onap.org/job/policy-drools-pdp-master-merge-scm-mvn-script/ https://jenkins.onap.org/job/policy-drools-pdp-casablanca-casablanca-merge-scm-mvn-script/ I will keep an eye on Nexus 3 jobs, but please feel free to let me know if issues still occur. Will escalate to INFRA if so, but looks like this might have helped Jess On Tue Dec 18 12:22:23 2018, jwagantall wrote: > I do not see anything unusual in the server. It seems to be operating > well at 18% CPU usage and 14% memory usage which is normal. > > I have restarted the server for now to see if that helps. If not, I > will escalate to out INFRA team > > Thanks > Jess > On Tue Dec 18 12:03:59 2018, jwagantall wrote: > > Looking into the server now .. will report back in a bit.. > > > > Jess > > > > On Tue Dec 18 11:23:45 2018, [email protected] wrote: > > > Looping in Kenny. > > > > > > Thanks, > > > Gildas > > > ONAP Release Manager > > > 1 415 238 6287 > > > > > > > > > -----Original Message----- > > > From: DRAGOSH, PAMELA L (PAM) [mailto:[email protected]] > > > Sent: Tuesday, December 18, 2018 7:34 AM > > > To: [email protected] > > > Cc: LEFEVRE, CATHERINE <[email protected]>; Gildas > > > Lanilis <[email protected]>; [email protected] > > > Subject: Re: [ONAP Helpdesk #65794] AutoReply: nexus3 timing out > > > > > > Catherine and Gildas, > > > > > > Can we get an escalation to the LF on this? I have the following > > > merge job that I have been trying to get done for the last day. It > > > is relevant to the Casablanca Maintenance Release. My team is also > > > in need of installation of new lab to support testing for this > > > release. > > > That cannot happen in a timely manner if we continue to have > > > nexus3 timeouts. > > > > > > https://jenkins.onap.org/job/policy-drools-pdp-casablanca- > > > casablanca- > > > merge-scm-mvn-script/9/ > > > > > > Thanks, > > > > > > Pam > > > > > > On 12/17/18, 3:08 PM, "ONAP Helpdesk via RT" <onap- > > > [email protected]> wrote: > > > > > > Greetings! > > > > > > Starting Dec. 14, 2018 through Jan. 1, 2019 the Linux Foundation > > > will be operating with reduced staff in observance of our office > > > closure. > > > Responses to inbound support requests will be delayed, or > > > otherwise addressed after Jan. 2, 2019. > > > > > > Happy Holidays! > > > > > > Your support ticket regarding: > > > "nexus3 timing out ", > > > has been entered in our ticket tracker. > > > > > > -- > > > The Linux Foundation Support Team > > > > > > ------------------------------------------------------------------ > > > ------- > > > Hi, > > > > > > Per discuss list, people are experiencing timeouts with nexus3. > > > > > > Example merged job that is failing during docker push: > > > > > > https://urldefense.proofpoint.com/v2/url?u=https- > > > 3A__jenkins.onap.org_job_policy-2Ddrools-2Dpdp-2Dmaster-2Dmerge- > > > 2Dscm- > > > 2Dmvn-2Dscript_137&d=DwIDaQ&c=LFYZ- > > > o9_HUMeMTSQicvjIg&r=jwTiArcEj6aUX0HjV0M3dT12gUtk7rC07xpgpVZkS_4&m= > > > Jn68cFc5QcaCx0dIwaVuvoImIAnKFZAjB- > > > SFXV_V7UQ&s=SJ8A4aHbPXqJwbaHGNDaFqIjgTTkQW7pEPCD_I1nD6I&e= > > > > > > thanks, > > > > > > Pam > > > > > > > > > This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14599): https://lists.onap.org/g/onap-discuss/message/14599 Mute This Topic: https://lists.onap.org/mt/28794897/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
