Jessica/Gary, I have removed the dmaap-messagerouter-docker-master-docker-version-java-daily job from the ci-management.
https://gerrit.onap.org/r/#/c/47415/ Thanks, Sunil From: [email protected] <[email protected]> On Behalf Of Gary Wu Sent: Saturday, May 12, 2018 9:59 AM To: Jessica Wagantall <[email protected]> Cc: Jeremy Phelps <[email protected]>; onap-release <[email protected]>; Anil Belur <[email protected]>; [email protected] Subject: Re: [onap-discuss] Nexus 3 images "disappearing" from the server Hi Jess, Two jobs building to the same docker image does seem odd. Can the dmaap team chime in? Thanks, Gary From: Jessica Wagantall [mailto:[email protected]] Sent: Friday, May 11, 2018 6:00 PM To: Gary Wu <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; onap-release <[email protected]<mailto:[email protected]>>; Gildas Lanilis <[email protected]<mailto:[email protected]>>; Jeremy Phelps <[email protected]<mailto:[email protected]>>; Kenny Paul <[email protected]<mailto:[email protected]>>; Anil Belur <[email protected]<mailto:[email protected]>> Subject: Re: Re: [onap-discuss] Nexus 3 images "disappearing" from the server Gary, Can I get your opinion on this? Looking at dmaap, it seems that this project is using 2 daily templates at the same time: [Image removed by sender. Success] [Image removed by sender. 100%]<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_view_dmaap_job_dmaap-2Dmessagerouter-2Ddocker-2Dmaster-2Ddocker-2Djava-2Ddaily_lastBuild&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=ezwsGB0rGM1bRHkalCR9niiP0cRuKhZe6cx3v80T-EA&e=> dmaap-messagerouter-docker-master-docker-java-daily<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_view_dmaap_job_dmaap-2Dmessagerouter-2Ddocker-2Dmaster-2Ddocker-2Djava-2Ddaily_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=z_mmrTosg1O46sFZjIEkoFDIit7ErUBtKhbCzuSPv2E&e=> 12 hr - #266<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_view_dmaap_job_dmaap-2Dmessagerouter-2Ddocker-2Dmaster-2Ddocker-2Djava-2Ddaily_lastSuccessfulBuild_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=gNV9966El3imHLVofFHmkVhJdQll2YYKiwkN6b_mhgE&e=> N/A 8 min 34 sec [Image removed by sender. Success] [Image removed by sender. 100%]<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_view_dmaap_job_dmaap-2Dmessagerouter-2Ddocker-2Dmaster-2Ddocker-2Dversion-2Djava-2Ddaily_lastBuild&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=wBIml4q6RF21YCZ38UD5WVXczPVO6cZr5Su5DPQmoFA&e=> dmaap-messagerouter-docker-master-docker-version-java-daily<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_view_dmaap_job_dmaap-2Dmessagerouter-2Ddocker-2Dmaster-2Ddocker-2Dversion-2Djava-2Ddaily_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=756UBs7EXjHKwb9Hd3IcUZU0OPzC825EF3qO7UO2X-g&e=> 12 hr - #267<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_view_dmaap_job_dmaap-2Dmessagerouter-2Ddocker-2Dmaster-2Ddocker-2Dversion-2Djava-2Ddaily_lastSuccessfulBuild_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=2-m1Dn1bcH9DoOmCJie15_JvwGbHpaIYZpKSDj6FctM&e=> N/A 9 min 48 sec Both seemed to update a latest tag. I am not saying this is the cause, but it is definitely not right thing to do. Right? thanks! Jess On Fri, May 11, 2018 at 5:49 PM, Jessica Wagantall <[email protected]<mailto:[email protected]>> wrote: Hi Gary, Just a small observation, onap/cli: 2.0.0-SNAPSHOT-20180425T130053Z seems like it will be removed in 45 mins. We have a task that runs every day at 6:30 PT that removes old SNAPSHOTS older than 16 days. I am looking further into your particular case. Thanks! Jess On Fri, May 11, 2018 at 5:43 PM, Jessica Wagantall <[email protected]<mailto:[email protected]>> wrote: Dear Michal, In your case, which is the Jenkins job that needs these versions: policy_handler: nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.policy-handler:2.4.1<https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_org.onap.dcaegen2.platform.policy-2Dhandler-3A2.4.1&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=XHLQZ65DwVrPS0dA9Uxzs75-A5v6IJhYW70h7wdqVUI&e=> service_change_handler: nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.servicechange-handler:1.1.3<https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_org.onap.dcaegen2.platform.servicechange-2Dhandler-3A1.1.3&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=h4jce8kjLp0lUUPumA1TC8-xa5Vc-N3l5QlOfDmhZYE&e=> I think in your particular case this could be a need for these versions to be released and that just hasn't been requested. I can see these binary versions tagged as "...SNAPSHOT.." but they haven't been pushed into the releases repository because no one have requested them Let me know if my info is helpful thanks! Jess On Fri, May 11, 2018 at 2:11 PM, Jessica Wagantall <[email protected]<mailto:[email protected]>> wrote: Thanks Gary, I have updated it since a bit ago and been using it without an issue. Gildas, can we propose to the TSC this upgrade please? It is something we can easily try with about 1 min of downtime while the server restarts with the new version. Thanks! Jess On Fri, May 11, 2018 at 1:37 PM, Gary Wu <[email protected]<mailto:[email protected]>> wrote: Sounds reasonable, except that I’m no longer actively using nexus3ap, and may not be able to give much feedback on how well it’s working. Thanks, Gary From: Jessica Wagantall [mailto:[email protected]<mailto:[email protected]>] Sent: Friday, May 11, 2018 1:27 PM To: [email protected]<mailto:[email protected]> Cc: Gary Wu <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; onap-release <[email protected]<mailto:[email protected]>>; Gildas Lanilis <[email protected]<mailto:[email protected]>>; Jeremy Phelps <[email protected]<mailto:[email protected]>>; Kenny Paul <[email protected]<mailto:[email protected]>>; Anil Belur <[email protected]<mailto:[email protected]>> Subject: Re: Re: [onap-discuss] Nexus 3 images "disappearing" from the server Thanks for your inputs Michael and Gary, Gary, the case you mention makes me wonder if this is a tag problem or the way binaries are being tagged every day. I was chatting with Andy and he mentioned to me that in general Nexus3 reports so many docker issues. We confirmed this is not relates to a disk capacity issues since we have still about 3TB of free space. I am trying something now which I don't guarantee might fix our issue, but we could give it a shot at least. I have upgraded https://nexus3ap.onap.org/<https://urldefense.proofpoint.com/v2/url?u=https-3A__nexus3ap.onap.org_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=qfd33UeQkcSuQa-PKCC6kzwaIt7MpWKzoevXcrJmFh4&e=> to version 3.11.0 and see how it behaves. If everything goes fine and that version is stable, we should upgrade nexus3.onap.org<https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=u4HboPDqDet-qrM2uCdqOT_CwwKdEbqI_2xeKhROoTg&e=> too so that we are at least on the latest version. Can I get your thoughts on this? Thanks! Jess On Thu, May 10, 2018 at 11:44 PM, Michal Ptacek <[email protected]<mailto:[email protected]>> wrote: Hi, not sure if it relates but OOM currently needs for dcae2gen following images policy_handler: nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.policy-handler:2.4.1<https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_org.onap.dcaegen2.platform.policy-2Dhandler-3A2.4.1&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=XHLQZ65DwVrPS0dA9Uxzs75-A5v6IJhYW70h7wdqVUI&e=> service_change_handler: nexus3.onap.org:10001/onap/org.onap.dcaegen2.platform.servicechange-handler:1.1.3<https://urldefense.proofpoint.com/v2/url?u=http-3A__nexus3.onap.org-3A10001_onap_org.onap.dcaegen2.platform.servicechange-2Dhandler-3A1.1.3&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=h4jce8kjLp0lUUPumA1TC8-xa5Vc-N3l5QlOfDmhZYE&e=> both of them are currently NOT available on nexus, the question is whether it relates to this problem Should we just simply start using newer images (fix in OOM ?) I don't know if I can propose that as I am not from DCAE team ... thanks, Michal --------- Original Message --------- Sender : Gary Wu <[email protected]<mailto:[email protected]>> Date : 2018-05-11 07:39 (GMT+1) Title : Re: [onap-discuss] Nexus 3 images "disappearing" from the server To : null<[email protected]<mailto:[email protected]>>, null<[email protected]<mailto:[email protected]>>, null<[email protected]<mailto:[email protected]>>, null<[email protected]<mailto:[email protected]>>, null<[email protected]<mailto:[email protected]>>, null<[email protected]<mailto:[email protected]>>, null<[email protected]<mailto:[email protected]>> Hi Jess, It’s not clear that the “Purge unused docker manifests and images” task is the culprit. If I understand the doc correctly, it’s only supposed to delete images that are no longer associated with any tags. I still see tons of time-stamped SNAPSHOT docker images from a couple of weeks ago still around (e.g. onap/cli: 2.0.0-SNAPSHOT-20180425T130053Z). As of this past couple of hours, onap/dmaap/dmaap-mr:1.1.4 has disappeared. See https://jenkins.onap.org/job/integration-master-version-manifest-verify-java/182/<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_job_integration-2Dmaster-2Dversion-2Dmanifest-2Dverify-2Djava_182_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=Xlw8EXKkn-76oQRYEKmBUeJo97aahxVyC4uKH0wQtKc&e=>. This broke the deployments that were running around that time. Fortunately, I had a local docker cache where I could pull a copy of onap/dmaap/dmaap-mr:1.1.4, and I found that this image has the same sha256 hash as the onap/dmaap/dmaap-mr:latest image currently on nexus3. So the image is still there, but just the tag is missing. Maybe this is another clue to help narrow down the cause. Thanks, Gary From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Jessica Wagantall Sent: Thursday, May 10, 2018 2:55 PM To: [email protected]<mailto:[email protected]>; onap-release <[email protected]<mailto:[email protected]>>; Gildas Lanilis <[email protected]<mailto:[email protected]>>; Jeremy Phelps <[email protected]<mailto:[email protected]>>; Kenny Paul <[email protected]<mailto:[email protected]>>; Anil Belur <[email protected]<mailto:[email protected]>> Subject: [onap-discuss] Nexus 3 images "disappearing" from the server Dear ONAP team, As mentioned today by Helen on the TSC call, it seems that we have experienced an issue with Nexus3 where the dependency images "disappear" for some moment and come back. I was investigating this issue a little bit closer, let me try to explain what I think is happening with Gary's example. In his case, onap/aaf/aaf_cm/manifests/2.1.0-SNAPSHOT<https://urldefense.proofpoint.com/v2/url?u=https-3A__nexus3.onap.org_repository_docker.snapshot_v2_onap_aaf_aaf-5Fcm_manifests_2.1.0-2DSNAPSHOT&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=xN9IDQOpZZiBScGl5PBrPE6e48tJFa0xnqNV-DaVPrQ&e=> (among other AFF images) disappeared on the 9th of may and re-appeared the same day after few hours. Looking at the job that pushed this image https://jenkins.onap.org/view/aaf/job/aaf-authz-master-docker-java-shell-daily/<https://urldefense.proofpoint.com/v2/url?u=https-3A__jenkins.onap.org_view_aaf_job_aaf-2Dauthz-2Dmaster-2Ddocker-2Djava-2Dshell-2Ddaily_&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=YtW6--CWmaAEspCzZ1SgNOId9z350US8f9EcUHq1Noc&e=>, seems like AAF bins were successfully pushed on the 7th and on the 9th but failed on the 8th. At the same time, I believe this rule kicked in: [cid:[email protected]] This rule seems to be scanning for dependencies and will remove any snapshot not being referenced by anyone every day. https://help.sonatype.com/repomanager3/configuration/system-configuration#SystemConfiguration-TypesofTasksandWhentoUseThem<https://urldefense.proofpoint.com/v2/url?u=https-3A__help.sonatype.com_repomanager3_configuration_system-2Dconfiguration-23SystemConfiguration-2DTypesofTasksandWhentoUseThem&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=8EXBdGb2oDx-ql2zjpZcTkKbi-XAxXwkzIXOkrEgwPo&e=> There is not much configuration on this rule to be able to explain what is actually looking for, but I believe this is the cause. This rule might have removed the AAF image pushed on the 7th and, since the AAF jenkins job failed to push a new image on the 8th, the rule might have remove it for some time. Then the job that kicked in on the 9th brought it back. So, here is my suggestions: - We need to make sure our daily jobs are healthy and address any failures on a daily basis to avoid this issue in future - Keeping this rule in place is helping us keep stability on disk space. If we were to remove it we will have grater issues to address. - I have confirmed with Andy and we prefer keeping this known configuration in place to avoid disk usage issues. Let me know if I was clear on my explanation and we can see if keeping an eye on the dailies helps us reducing this occurrences. Thanks a ton! Jess _______________________________________________ onap-discuss mailing list [email protected]<mailto:[email protected]> https://lists.onap.org/mailman/listinfo/onap-discuss<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.onap.org_mailman_listinfo_onap-2Ddiscuss&d=DwMGaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=t27CRRmn7U12dnOvcSPfcQ&m=e7F1deCZniKV5eIqsLVai3QKj_BRh_oAwAFZABzgQU0&s=BsmvxtMbJu9hXez4uLEiNgaL-ed2LX_SzzOnb2oykjM&e=> [cid:[email protected]] [Image removed by sender.]
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
