Team,
Nexus3.onap.info is now at the ccsdk's
Nexus3.onap.cloud is at the end of the aai’s – forgot to post (was asked)
where the cert is – added to
https://jira.onap.org/browse/TSC-79
https://jira.onap.org/secure/attachment/13025/domain_nexus3_onap_cloud.crt
or
https://jira.onap.org/browse/LOG-905
https://jira.onap.org/secure/attachment/13020/domain_nexus3_onap_cloud.crt
Note: docker pulls for images not pulled from nexus3.onap.org will still be
slow until they are in the cache – eta is mid Friday – I will advise when fully
cached.
Remember to rename it via
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy
Thank you
/michael
-----Original Message-----
From: [email protected] via RT <[email protected]>
Sent: Wednesday, December 19, 2018 4:20 AM
To: Michael O'Brien <[email protected]>
Cc: [email protected]; [email protected]
Subject: [ONAP Helpdesk #65866] [linuxfoundation.org #65866] RE: Nexus3 proxy
verified 80-100x faster downloads since 20181217 - I will provide a
nexus3.onap.cloud Azure based proxy until Jan as workaround #nexus3
Hi Brien,
Thanks for setting up the proxy. One question is that how to get the
certificate of nexus3.onap.info? I tried "scp
[email protected]:~/certs/domain.crt<mailto:[email protected]:~/certs/domain.crt>
.", but it failed with ssh public key issues. Without that certificate, I can
not docker login.
-Lianhao
From: [email protected] [mailto:[email protected]] On Behalf Of
Michael O'Brien
Sent: Wednesday, December 19, 2018 8:10 AM
To: [email protected]; [email protected]; [email protected]
Subject: [onap-tsc] Nexus3 proxy verified 80-100x faster downloads since
20181217 - I will provide a nexus3.onap.cloud Azure based proxy until Jan as
workaround #nexus3
Team,
The issue with nexus3.onap.info was likely a routing change Sunday. For
those requiring Casablanca 3.0.0-ONAP tagged images for new deployments I will
have a 2nd proxy for public consumption up later today. This will free up the
LF to address the RC in the new year.
Bottom line is a pull that takes 45 sec on the proxy takes 80-160 min from
nexus3.onap.org only for this week - increasing pull times from 25 min to 35h -
I retested an hour ago at 1730EST.
Linux Foundation - I have your back - the quiet period started on the 17th
for yourselves, for the rest of us on the 24th. I prototyped a proxy on AWS
nexus3.onap.info - 80-100X faster image downloads. I will put up a public one
on azure that is on a larger VM for the rest of the community to use (will need
a cert - I will add to the jira) - if you use nexus3.onap.cloud please leave a
note on the jira
See details on the testing on clean VMs in and out of windriver against both
nexus3.onap.org and nexus3.onap.info.
I will post details as soon as I get it up about the larger azure proxy - it
will take at least 35 hours to saturate the nexus3 proxy with 40G of Casablanca
images via the docker_preload.sh script - I will do master later and advise
when the proxy is ready.
Details on prepping you host VMs to use the nexus3 proxy temporarily.
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy
and
https://jira.onap.org/browse/TSC-79
# on each host
obrienbiometrics:nexus michaelobrien$ scp
[email protected]:~/certs/domain.crt<mailto:[email protected]:~/certs/domain.crt>
.
domain.crt
obrienbiometrics:nexus michaelobrien$ scp domain.crt
[email protected]:~/<mailto:[email protected]:~/>
domain.crt
100% 2114 15.7KB/s 00:00
# to avoid
ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker
nexus3.onap.info:5000
Error response from daemon: Get https://nexus3.onap.info:5000/v1/users/: x509:
certificate signed by unknown authority
# cp cert
ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d
ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d/nexus3.onap.info:5000
ubuntu@ip-172-31-33-46:~$ sudo cp domain.crt
/etc/docker/certs.d/nexus3.onap.info:5000/ca.crt
ubuntu@ip-172-31-33-46:~$ sudo systemctl restart docker
ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker
nexus3.onap.info:5000
Login Succeeded
# testing
# vm with the image existing - 2 sec
ubuntu@ip-172-31-33-46:~$ sudo docker pull
nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
2.1.8: Pulling from onap/aaf/aaf_agent
Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c
Status: Downloaded newer image for
nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
# vm with layers existing except for last 5 - 5 sec
ubuntu@a-cd-master:~$ sudo docker pull
nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
2.1.8: Pulling from onap/aaf/aaf_agent
18d680d61657: Already exists
.. 20
49e90af50c7d: Already exists
396607166153: Pull complete
8023c56d0234: Pull complete
440441b9ca5e: Pull complete
4376cc11267c: Pull complete
acb05d09ff6e: Pull complete
Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c
Status: Downloaded newer image for
nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
# clean AWS VM (clean install of docker) - no pulls yet - 45 sec for everything
ubuntu@ip-172-31-14-34:~$ sudo docker pull
nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
2.1.8: Pulling from onap/aaf/aaf_agent
18d680d61657: Pulling fs layer
0addb6fece63: Pulling fs layer
78e58219b215: Pulling fs layer
eb6959a66df2: Pulling fs layer
321bd3fd2d0e: Pull complete
819d6de9e493: Pull complete
9c5cbae584e0: Pull complete
6b8abae643fc: Pull complete
595cdf95b083: Pull complete
d5f35f05364b: Pull complete
33422549438c: Pull complete
507242d06459: Pull complete
797c42cd7990: Pull complete
55cb180bd7a7: Pull complete
8f791662f04f: Pull complete
b21eb3946af1: Pull complete
36d0f286a7ec: Pull complete
2150f5ecf4e4: Pull complete
5b1a5109ab06: Pull complete
880ff0160341: Pull complete
14bbbdc46f89: Pull complete
03b5e4771470: Pull complete
fdd6eab46e6c: Pull complete
80ff62223f5f: Pull complete
e0633204c627: Pull complete
1b3a93e91ad9: Pull complete
29b72f79b735: Pull complete
a2cb53787c38: Pull complete
49e90af50c7d: Pull complete
396607166153: Pull complete
8023c56d0234: Pull complete
440441b9ca5e: Pull complete
4376cc11267c: Pull complete
acb05d09ff6e: Pull complete
Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c
Status: Downloaded newer image for
nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
ubuntu@ip-172-31-14-34:~$ sudo docker images
REPOSITORY TAG IMAGE ID
CREATED SIZE
nexus3.onap.info:5000/onap/aaf/aaf_agent 2.1.8 090b326a7f11
5 weeks ago 1.14 GB
# going to test a same size image directly from the LF - with minimal common
layers
nexus3.onap.org:10001/onap/testsuite 1.3.2
c4b58baa95e8 3 weeks ago 1.13 GB
# 5 min in we are still at 3% - numbers below are a min old
ubuntu@ip-172-31-14-34:~$ sudo docker pull
nexus3.onap.org:10001/onap/testsuite:1.3.2
1.3.2: Pulling from onap/testsuite
32802c0cfa4d: Downloading [=============> ]
8.416 MB/32.1 MB
da1315cffa03: Download complete
fa83472a3562: Download complete
f85999a86bef: Download complete
3eca7452fe93: Downloading [=======================> ]
8.517 MB/17.79 MB
9f002f13a564: Downloading [=========================================> ]
8.528 MB/10.24 MB
02682cf43e5c: Waiting
dfa9878b26c8: Waiting
fdb3a2e7127c: Waiting
adab76b27695: Waiting
3e1db7307fd3: Waiting
d18a8cc3d4cd: Waiting
973f5d70b3c4: Waiting
eb19d05de017: Waiting
ac1f3947b000: Waiting
01ae31c1279e: Waiting
24ac662c0884: Waiting
268c36718aa7: Waiting
98559383536b: Waiting
856a1ffe236e: Waiting
80cbab20328c: Waiting
caed9c28b75f: Waiting
e3c00d7d86d4: Waiting
3109ae4e798e: Waiting
2dc71030fc1d: Waiting
f1d763192c3d: Waiting
adceeebb9b2b: Waiting
419a6a0c4acd: Waiting
048d8fd47ec5: Waiting
63fc3f31cfdc: Waiting
5441f94df8b4: Waiting
754645df4601: Waiting
# in 5 min we get 3% 35/1130Mb - which comes out to 162 min for 1.13G for .org
as opposed to 45 sec for .info - which is a 200X slowdown - some of this is due
to the fact my nexus3.onap.info is on the same VPC as my test VM - testing on
openlab
# openlab - 2 min 40 sec which is 3.6 times slower - expected than in AWS - (25
min pulls vs 90min in openlab) - this makes nexus.onap.org 60 times slower in
openlab than a proxy running from AWS (2 vCore/16G/ssd VM)
ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull
nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
2.1.8: Pulling from onap/aaf/aaf_agent
18d680d61657: Pull complete
...
acb05d09ff6e: Pull complete
Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c
Status: Downloaded newer image for
nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8
#pulling smaller from nexus3.onap.info 2 min 20 - for 36Mb = 0.23Mb/sec -
extrapolated to 1.13Gb for above is 5022 sec or 83 min - half the rough
calculation above
ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull
nexus3.onap.org:10001/onap/aaf/sms:3.0.1
3.0.1: Pulling from onap/aaf/sms
c67f3896b22c: Pull complete
134fac3103be: Pull complete
c7fdcaf7f439: Pull complete
590a963725eb: Pull complete
4df2c296b645: Pull complete
15dd1c6ebaf1: Pull complete
76eeb922b789: Pull complete
Digest: sha256:d5b64947edb93848acacaa9820234aa29e58217db9f878886b7bafae00fdb436
Status: Downloaded newer image for nexus3.onap.org:10001/onap/aaf/sms:3.0.1
# conclusion - nexus3.onap.org is experiencing a routing issue from their DC
outbound causing a 80-100x slowdown over a proxy nexus3 - since 20181217 - as
local jenkins.onap.org builds complete faster
# workaround is to use a nexus3 proxy above
and adding to values.yaml
global:
#repository: nexus3.onap.org:10001
repository: nexus3.onap.info:5000
repositoryCred:
user: docker
password: docker
This email and the information contained herein is proprietary and confidential
and subject to the Amdocs Email Terms of Service, which you may review at
https://www.amdocs.com/about/email-terms-of-service
This email and the information contained herein is proprietary and confidential
and subject to the Amdocs Email Terms of Service, which you may review at
https://www.amdocs.com/about/email-terms-of-service
<https://www.amdocs.com/about/email-terms-of-service>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#14618): https://lists.onap.org/g/onap-discuss/message/14618
Mute This Topic: https://lists.onap.org/mt/28800220/21656
Mute #nexus3: https://lists.onap.org/mk?hashtag=nexus3&subid=2740164
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
