Team, The issue with nexus3.onap.info was likely a routing change Sunday. For those requiring Casablanca 3.0.0-ONAP tagged images for new deployments I will have a 2nd proxy for public consumption up later today. This will free up the LF to address the RC in the new year.
Linux Foundation - I have your back - the quiet period started on the 17th for yourselves, for the rest of us on the 24th. I prototyped a proxy on AWS nexus3.onap.info - 80-100X faster image downloads. I will put up a public one on azure that is on a larger VM for the rest of the community to use (will need a cert - I will add to the jira) - if you use nexus3.onap.cloud please leave a note on the jira I will post details as soon as I get it up - it will take at least 35 hours to saturate the nexus3 proxy with 40G of Casablanca images via the docker_preload.sh script - I will do master later and advise when the proxy is ready. Details on prepping you host VMs to use the nexus3 proxy temporarily. https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy and https://jira.onap.org/browse/TSC-79 # on each host obrienbiometrics:nexus michaelobrien$ scp [email protected]:~/certs/domain.crt . domain.crt obrienbiometrics:nexus michaelobrien$ scp domain.crt [email protected]:~/ domain.crt 100% 2114 15.7KB/s 00:00 # to avoid ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000 Error response from daemon: Get https://nexus3.onap.info:5000/v1/users/: x509: certificate signed by unknown authority # cp cert ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d/nexus3.onap.info:5000 ubuntu@ip-172-31-33-46:~$ sudo cp domain.crt /etc/docker/certs.d/nexus3.onap.info:5000/ca.crt ubuntu@ip-172-31-33-46:~$ sudo systemctl restart docker ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000 Login Succeeded # testing # vm with the image existing - 2 sec ubuntu@ip-172-31-33-46:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 # vm with layers existing except for last 5 - 5 sec ubuntu@a-cd-master:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent 18d680d61657: Already exists .. 20 49e90af50c7d: Already exists 396607166153: Pull complete 8023c56d0234: Pull complete 440441b9ca5e: Pull complete 4376cc11267c: Pull complete acb05d09ff6e: Pull complete Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 # clean AWS VM (clean install of docker) - no pulls yet - 45 sec for everything ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent 18d680d61657: Pulling fs layer 0addb6fece63: Pulling fs layer 78e58219b215: Pulling fs layer eb6959a66df2: Pulling fs layer 321bd3fd2d0e: Pull complete 819d6de9e493: Pull complete 9c5cbae584e0: Pull complete 6b8abae643fc: Pull complete 595cdf95b083: Pull complete d5f35f05364b: Pull complete 33422549438c: Pull complete 507242d06459: Pull complete 797c42cd7990: Pull complete 55cb180bd7a7: Pull complete 8f791662f04f: Pull complete b21eb3946af1: Pull complete 36d0f286a7ec: Pull complete 2150f5ecf4e4: Pull complete 5b1a5109ab06: Pull complete 880ff0160341: Pull complete 14bbbdc46f89: Pull complete 03b5e4771470: Pull complete fdd6eab46e6c: Pull complete 80ff62223f5f: Pull complete e0633204c627: Pull complete 1b3a93e91ad9: Pull complete 29b72f79b735: Pull complete a2cb53787c38: Pull complete 49e90af50c7d: Pull complete 396607166153: Pull complete 8023c56d0234: Pull complete 440441b9ca5e: Pull complete 4376cc11267c: Pull complete acb05d09ff6e: Pull complete Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 ubuntu@ip-172-31-14-34:~$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE nexus3.onap.info:5000/onap/aaf/aaf_agent 2.1.8 090b326a7f11 5 weeks ago 1.14 GB # going to test a same size image directly from the LF - with minimal common layers nexus3.onap.org:10001/onap/testsuite 1.3.2 c4b58baa95e8 3 weeks ago 1.13 GB # 5 min in we are still at 3% - numbers below are a min old ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.org:10001/onap/testsuite:1.3.2 1.3.2: Pulling from onap/testsuite 32802c0cfa4d: Downloading [=============> ] 8.416 MB/32.1 MB da1315cffa03: Download complete fa83472a3562: Download complete f85999a86bef: Download complete 3eca7452fe93: Downloading [=======================> ] 8.517 MB/17.79 MB 9f002f13a564: Downloading [=========================================> ] 8.528 MB/10.24 MB 02682cf43e5c: Waiting dfa9878b26c8: Waiting fdb3a2e7127c: Waiting adab76b27695: Waiting 3e1db7307fd3: Waiting d18a8cc3d4cd: Waiting 973f5d70b3c4: Waiting eb19d05de017: Waiting ac1f3947b000: Waiting 01ae31c1279e: Waiting 24ac662c0884: Waiting 268c36718aa7: Waiting 98559383536b: Waiting 856a1ffe236e: Waiting 80cbab20328c: Waiting caed9c28b75f: Waiting e3c00d7d86d4: Waiting 3109ae4e798e: Waiting 2dc71030fc1d: Waiting f1d763192c3d: Waiting adceeebb9b2b: Waiting 419a6a0c4acd: Waiting 048d8fd47ec5: Waiting 63fc3f31cfdc: Waiting 5441f94df8b4: Waiting 754645df4601: Waiting # in 5 min we get 3% 35/1130Mb - which comes out to 162 min for 1.13G for .org as opposed to 45 sec for .info - which is a 200X slowdown - some of this is due to the fact my nexus3.onap.info is on the same VPC as my test VM - testing on openlab # openlab - 2 min 40 sec which is 3.6 times slower - expected than in AWS - (25 min pulls vs 90min in openlab) - this makes nexus.onap.org 60 times slower in openlab than a proxy running from AWS (2 vCore/16G/ssd VM) ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent 18d680d61657: Pull complete ... acb05d09ff6e: Pull complete Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 #pulling smaller from nexus3.onap.info 2 min 20 - for 36Mb = 0.23Mb/sec - extrapolated to 1.13Gb for above is 5022 sec or 83 min - half the rough calculation above ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.org:10001/onap/aaf/sms:3.0.1 3.0.1: Pulling from onap/aaf/sms c67f3896b22c: Pull complete 134fac3103be: Pull complete c7fdcaf7f439: Pull complete 590a963725eb: Pull complete 4df2c296b645: Pull complete 15dd1c6ebaf1: Pull complete 76eeb922b789: Pull complete Digest: sha256:d5b64947edb93848acacaa9820234aa29e58217db9f878886b7bafae00fdb436 Status: Downloaded newer image for nexus3.onap.org:10001/onap/aaf/sms:3.0.1 # conclusion - nexus3.onap.org is experiencing a routing issue from their DC outbound causing a 80-100x slowdown over a proxy nexus3 - since 20181217 - as local jenkins.onap.org builds complete faster # workaround is to use a nexus3 proxy above and adding to values.yaml global: #repository: nexus3.onap.org:10001 repository: nexus3.onap.info:5000 repositoryCred: user: docker password: docker -----Original Message----- From: [email protected] <[email protected]> On Behalf Of Michael O'Brien Sent: Tuesday, December 18, 2018 1:43 PM To: [email protected]; [email protected]; [email protected] Cc: Soleil, Alain <[email protected]>; Nir Rapaport <[email protected]>; Jim Watson <[email protected]>; Mike Elliott <[email protected]>; Prudence Au <[email protected]>; J.Ram Balasubramanian <[email protected]> Subject: Re: [onap-discuss] [ONAP Helpdesk #65794] nexus3 timing out - still getting 0.4MB/sec - not the usual 10+MB/sec No change in pull speed - a good test create a clean ubuntu 16.04 vm (no previous pull) - install docker, login and pull an image Getting roughly 0.4MB/sec - at least 20x slower than in the past. This is on an AWS EC2 instance On another instance that is running the docker prepull - I saw the 502 timeouts in the nohup.out log - since the restart of nexus3 only 3 images in SO were downloaded for example - normally within 25 min I can download all 40G of onap images. sudo curl https://releases.rancher.com/install-docker/17.03.sh | sh sudo usermod -aG docker ubuntu ubuntu@ip-172-31-10-98:~$ sudo docker login -u docker -p docker nexus3.onap.org:10001 Login Succeeded # get images from https://git.onap.org/integration/plain/version-manifest/src/main/resources/docker-manifest.csv?h=casablanca # use for example the first line onap/aaf/aaf_agent,2.1.8 # or the prepull script in https://gerrit.onap.org/r/#/c/74780/1/deploy/docker_prepull.sh sudo docker pull nexus3.onap.org:10001/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent 18d680d61657: Pulling fs layer 819d6de9e493: Downloading [======================================> ] 770.7 kB/1.012 MB https://jira.onap.org/browse/LOG-355 If I pull for example onap/ccsdk-dgbuilder-image:0.3.2 - which is 1G - but all but the last 2 layers of 345 and 71mb are not pulled -these just basically sit there - and look to take at least 10+ min based on progress as of 1334 EST ubuntu@ip-172-31-10-98:~$ sudo docker pull nexus3.onap.org:10001/onap/aaf/sms:3.0.1 3.0.1: Pulling from onap/aaf/sms c67f3896b22c: Pull complete 134fac3103be: Pull complete c7fdcaf7f439: Pull complete 590a963725eb: Pull complete 4df2c296b645: Pull complete 15dd1c6ebaf1: Pull complete 76eeb922b789: Pull complete Digest: sha256:d5b64947edb93848acacaa9820234aa29e58217db9f878886b7bafae00fdb436 Status: Downloaded newer image for nexus3.onap.org:10001/onap/aaf/sms:3.0.1 ubuntu@ip-172-31-10-98:~$ sudo docker pull nexus3.onap.org:10001/onap/ccsdk-dgbuilder-image:0.3.2 0.3.2: Pulling from onap/ccsdk-dgbuilder-image 18d680d61657: Already exists 0addb6fece63: Already exists 78e58219b215: Already exists eb6959a66df2: Already exists 4eeb11d61ff9: Downloading [> ] 5.266 MB/345.8 MB 0f81686fe963: Downloading [===> ] 5.266 MB/71.15 MB 4 min later ubuntu@ip-172-31-10-98:~$ sudo docker pull nexus3.onap.org:10001/onap/ccsdk-dgbuilder-image:0.3.2 0.3.2: Pulling from onap/ccsdk-dgbuilder-image 18d680d61657: Already exists 0addb6fece63: Already exists 78e58219b215: Already exists eb6959a66df2: Already exists 4eeb11d61ff9: Downloading [=> ] 10.54 MB/345.8 MB 0f81686fe963: Downloading [=======> ] 10.54 MB/71.15 MB -----Original Message----- From: [email protected] <[email protected]> On Behalf Of Kenny Paul via RT Sent: Tuesday, December 18, 2018 1:14 PM To: [email protected] Cc: [email protected] Subject: [onap-discuss] [ONAP Helpdesk #65794] nexus3 timing out After the restart, these jobs passed now: https://jenkins.onap.org/job/policy-drools-pdp-master-merge-scm-mvn-script/ https://jenkins.onap.org/job/policy-drools-pdp-casablanca-casablanca-merge-scm-mvn-script/ I will keep an eye on Nexus 3 jobs, but please feel free to let me know if issues still occur. Will escalate to INFRA if so, but looks like this might have helped Jess On Tue Dec 18 12:22:23 2018, jwagantall wrote: > I do not see anything unusual in the server. It seems to be operating > well at 18% CPU usage and 14% memory usage which is normal. > > I have restarted the server for now to see if that helps. If not, I > will escalate to out INFRA team > > Thanks > Jess > On Tue Dec 18 12:03:59 2018, jwagantall wrote: > > Looking into the server now .. will report back in a bit.. > > > > Jess > > > > On Tue Dec 18 11:23:45 2018, [email protected] wrote: > > > Looping in Kenny. > > > > > > Thanks, > > > Gildas > > > ONAP Release Manager > > > 1 415 238 6287 > > > > > > > > > -----Original Message----- > > > From: DRAGOSH, PAMELA L (PAM) [mailto:[email protected]] > > > Sent: Tuesday, December 18, 2018 7:34 AM > > > To: [email protected] > > > Cc: LEFEVRE, CATHERINE <[email protected]>; Gildas > > > Lanilis <[email protected]>; [email protected] > > > Subject: Re: [ONAP Helpdesk #65794] AutoReply: nexus3 timing out > > > > > > Catherine and Gildas, > > > > > > Can we get an escalation to the LF on this? I have the following > > > merge job that I have been trying to get done for the last day. It > > > is relevant to the Casablanca Maintenance Release. My team is also > > > in need of installation of new lab to support testing for this > > > release. > > > That cannot happen in a timely manner if we continue to have > > > nexus3 timeouts. > > > > > > https://jenkins.onap.org/job/policy-drools-pdp-casablanca- > > > casablanca- > > > merge-scm-mvn-script/9/ > > > > > > Thanks, > > > > > > Pam > > > > > > On 12/17/18, 3:08 PM, "ONAP Helpdesk via RT" <onap- > > > [email protected]> wrote: > > > > > > Greetings! > > > > > > Starting Dec. 14, 2018 through Jan. 1, 2019 the Linux Foundation > > > will be operating with reduced staff in observance of our office > > > closure. > > > Responses to inbound support requests will be delayed, or > > > otherwise addressed after Jan. 2, 2019. > > > > > > Happy Holidays! > > > > > > Your support ticket regarding: > > > "nexus3 timing out ", > > > has been entered in our ticket tracker. > > > > > > -- > > > The Linux Foundation Support Team > > > > > > ------------------------------------------------------------------ > > > ------- > > > Hi, > > > > > > Per discuss list, people are experiencing timeouts with nexus3. > > > > > > Example merged job that is failing during docker push: > > > > > > https://urldefense.proofpoint.com/v2/url?u=https- > > > 3A__jenkins.onap.org_job_policy-2Ddrools-2Dpdp-2Dmaster-2Dmerge- > > > 2Dscm- > > > 2Dmvn-2Dscript_137&d=DwIDaQ&c=LFYZ- > > > o9_HUMeMTSQicvjIg&r=jwTiArcEj6aUX0HjV0M3dT12gUtk7rC07xpgpVZkS_4&m= > > > Jn68cFc5QcaCx0dIwaVuvoImIAnKFZAjB- > > > SFXV_V7UQ&s=SJ8A4aHbPXqJwbaHGNDaFqIjgTTkQW7pEPCD_I1nD6I&e= > > > > > > thanks, > > > > > > Pam > > > > > > > > > This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service> This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#14600): https://lists.onap.org/g/onap-discuss/message/14600 Mute This Topic: https://lists.onap.org/mt/28794897/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
