Gary, Can I get your opinion on this? Looking at dmaap, it seems that this project is using 2 daily templates at the same time: [image: Success] [image: 100%] <https://jenkins.onap.org/view/dmaap/job/dmaap-messagerouter-docker-master-docker-java-daily/lastBuild> dmaap-messagerouter-docker-master-docker-java-daily <https://jenkins.onap.org/view/dmaap/job/dmaap-messagerouter-docker-master-docker-java-daily/> 12 hr - #266 <https://jenkins.onap.org/view/dmaap/job/dmaap-messagerouter-docker-master-docker-java-daily/lastSuccessfulBuild/> N/A 8 min 34 sec [image: Success] [image: 100%] <https://jenkins.onap.org/view/dmaap/job/dmaap-messagerouter-docker-master-docker-version-java-daily/lastBuild> dmaap-messagerouter-docker-master-docker-version-java-daily <https://jenkins.onap.org/view/dmaap/job/dmaap-messagerouter-docker-master-docker-version-java-daily/> 12 hr - #267 <https://jenkins.onap.org/view/dmaap/job/dmaap-messagerouter-docker-master-docker-version-java-daily/lastSuccessfulBuild/> N/A 9 min 48 sec
Both seemed to update a latest tag. I am not saying this is the cause, but it is definitely not right thing to do. Right? thanks! Jess On Fri, May 11, 2018 at 5:49 PM, Jessica Wagantall < [email protected]> wrote: > Hi Gary, > > Just a small observation, > onap/cli: 2.0.0-SNAPSHOT-20180425T130053Z > seems like it will be removed in 45 mins. > > We have a task that runs every day at 6:30 PT that removes old SNAPSHOTS > older than 16 days. > > I am looking further into your particular case. > > Thanks! > Jess > > On Fri, May 11, 2018 at 5:43 PM, Jessica Wagantall < > [email protected]> wrote: > >> Dear Michal, >> >> In your case, which is the Jenkins job that needs these versions: >> policy_handler: nexus3.onap.org:10001/onap/org.onap.dcaegen2 >> .platform.policy-handler:2.4.1 >> service_change_handler: nexus3.onap.org:10001/onap/org.onap. >> dcaegen2.platform.servicechange-handler:1.1.3 >> >> I think in your particular case this could be a need for these versions >> to be released and that just hasn't been requested. >> >> I can see these binary versions tagged as "...SNAPSHOT.." but they >> haven't been pushed into the releases >> repository because no one have requested them >> >> Let me know if my info is helpful >> >> thanks! >> Jess >> >> On Fri, May 11, 2018 at 2:11 PM, Jessica Wagantall < >> [email protected]> wrote: >> >>> Thanks Gary, >>> >>> I have updated it since a bit ago and been using it without an issue. >>> >>> Gildas, can we propose to the TSC this upgrade please? >>> >>> It is something we can easily try with about 1 min of downtime while the >>> server restarts with the new version. >>> >>> Thanks! >>> Jess >>> >>> On Fri, May 11, 2018 at 1:37 PM, Gary Wu <[email protected]> wrote: >>> >>>> Sounds reasonable, except that I’m no longer actively using nexus3ap, >>>> and may not be able to give much feedback on how well it’s working. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Gary >>>> >>>> >>>> >>>> *From:* Jessica Wagantall [mailto:[email protected]] >>>> *Sent:* Friday, May 11, 2018 1:27 PM >>>> *To:* [email protected] >>>> *Cc:* Gary Wu <[email protected]>; [email protected]; >>>> onap-release <[email protected]>; Gildas Lanilis < >>>> [email protected]>; Jeremy Phelps <[email protected]>; >>>> Kenny Paul <[email protected]>; Anil Belur < >>>> [email protected]> >>>> *Subject:* Re: Re: [onap-discuss] Nexus 3 images "disappearing" from >>>> the server >>>> >>>> >>>> >>>> Thanks for your inputs Michael and Gary, >>>> >>>> >>>> >>>> Gary, the case you mention makes me wonder if this is a tag problem or >>>> the way binaries are being tagged every day. >>>> >>>> I was chatting with Andy and he mentioned to me that in general Nexus3 >>>> reports so many docker issues. >>>> >>>> >>>> >>>> We confirmed this is not relates to a disk capacity issues since we >>>> have still about 3TB of free space. >>>> >>>> >>>> >>>> I am trying something now which I don't guarantee might fix our issue, >>>> but we could give it a shot at least. >>>> >>>> I have upgraded https://nexus3ap.onap.org/ to version 3.11.0 and see >>>> how it behaves. >>>> >>>> If everything goes fine and that version is stable, we should upgrade >>>> nexus3.onap.org too so that we are at least >>>> >>>> on the latest version. >>>> >>>> >>>> >>>> Can I get your thoughts on this? >>>> >>>> >>>> >>>> Thanks! >>>> >>>> Jess >>>> >>>> >>>> >>>> On Thu, May 10, 2018 at 11:44 PM, Michal Ptacek < >>>> [email protected]> wrote: >>>> >>>> Hi, >>>> >>>> >>>> >>>> not sure if it relates but OOM currently needs for dcae2gen following >>>> images >>>> >>>> >>>> >>>> policy_handler: nexus3.onap.org:10001/onap/org >>>> .onap.dcaegen2.platform.policy-handler:2.4.1 >>>> service_change_handler: nexus3.onap.org:10001/onap/org >>>> .onap.dcaegen2.platform.servicechange-handler:1.1.3 >>>> >>>> >>>> >>>> both of them are currently NOT available on nexus, the question is >>>> whether it relates to this problem >>>> >>>> >>>> >>>> Should we just simply start using newer images (fix in OOM ?) >>>> >>>> I don't know if I can propose that as I am not from DCAE team ... >>>> >>>> >>>> >>>> thanks, >>>> >>>> Michal >>>> >>>> >>>> >>>> --------- *Original Message* --------- >>>> >>>> *Sender* : Gary Wu <[email protected]> >>>> >>>> *Date* : 2018-05-11 07:39 (GMT+1) >>>> >>>> *Title* : Re: [onap-discuss] Nexus 3 images "disappearing" from the >>>> server >>>> >>>> *To : *null<[email protected]>, null< >>>> [email protected]>, null<[email protected]>, null< >>>> [email protected]>, null<[email protected]>, null< >>>> [email protected]>, null<[email protected]> >>>> >>>> >>>> >>>> Hi Jess, >>>> >>>> >>>> >>>> It’s not clear that the “Purge unused docker manifests and images” task >>>> is the culprit. If I understand the doc correctly, it’s only supposed to >>>> delete images that are no longer associated with any tags. I still see >>>> tons of time-stamped SNAPSHOT docker images from a couple of weeks ago >>>> still around (e.g. onap/cli: 2.0.0-SNAPSHOT-20180425T130053Z). >>>> >>>> >>>> >>>> As of this past couple of hours, onap/dmaap/dmaap-mr:1.1.4 has >>>> disappeared. See https://jenkins.onap.org/job/i >>>> ntegration-master-version-manifest-verify-java/182/. This broke the >>>> deployments that were running around that time. >>>> >>>> >>>> >>>> Fortunately, I had a local docker cache where I could pull a copy of >>>> onap/dmaap/dmaap-mr:1.1.4, and I found that this image has the same sha256 >>>> hash as the onap/dmaap/dmaap-mr:latest image currently on nexus3. So the >>>> image is still there, but just the tag is missing. Maybe this is another >>>> clue to help narrow down the cause. >>>> >>>> >>>> >>>> Thanks, >>>> >>>> Gary >>>> >>>> >>>> >>>> >>>> >>>> *From:* [email protected] [mailto: >>>> [email protected]] *On Behalf Of *Jessica Wagantall >>>> *Sent:* Thursday, May 10, 2018 2:55 PM >>>> *To:* [email protected]; onap-release < >>>> [email protected]>; Gildas Lanilis <[email protected]>; >>>> Jeremy Phelps <[email protected]>; Kenny Paul < >>>> [email protected]>; Anil Belur <[email protected]> >>>> *Subject:* [onap-discuss] Nexus 3 images "disappearing" from the >>>> server >>>> >>>> >>>> >>>> Dear ONAP team, >>>> >>>> >>>> >>>> >>>> >>>> As mentioned today by Helen on the TSC call, it seems that we have >>>> experienced >>>> >>>> >>>> >>>> an issue with Nexus3 where the dependency images "disappear" for some >>>> moment and >>>> >>>> >>>> >>>> come back. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> I was investigating this issue a little bit closer, let me try to >>>> explain what I think is happening >>>> >>>> >>>> >>>> with Gary's example. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> In his case, onap/aaf/aaf_cm/manifests/2.1.0-SNAPSHOT >>>> <https://nexus3.onap.org/repository/docker.snapshot/v2/onap/aaf/aaf_cm/manifests/2.1.0-SNAPSHOT> >>>> (among >>>> other AFF images) disappeared >>>> >>>> >>>> >>>> on the 9th of may and re-appeared the same day after few hours. >>>> >>>> >>>> >>>> Looking at the job that pushed this image https://jenkins.onap.org >>>> /view/aaf/job/aaf-authz-master-docker-java-shell-daily/, >>>> >>>> >>>> >>>> seems like AAF bins were successfully pushed on the 7th and on the 9th >>>> but failed on the 8th. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> At the same time, I believe this rule kicked in: >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> This rule seems to be scanning for dependencies and will remove any >>>> snapshot not being referenced by anyone every day. >>>> >>>> >>>> >>>> https://help.sonatype.com/repomanager3/configuration/system- >>>> configuration#SystemConfiguration-TypesofTasksandWhentoUseThem >>>> >>>> >>>> >>>> There is not much configuration on this rule to be able to explain what >>>> is actually looking for, but I believe this is the cause. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> This rule might have removed the AAF image pushed on the 7th and, since >>>> the AAF jenkins job failed to push a new image on the 8th, >>>> >>>> >>>> >>>> the rule might have remove it for some time. Then the job that kicked >>>> in on the 9th brought it back. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> So, here is my suggestions: >>>> >>>> >>>> >>>> - We need to make sure our daily jobs are healthy and address any >>>> failures on a daily basis to avoid this issue in future >>>> >>>> >>>> >>>> - Keeping this rule in place is helping us keep stability on disk >>>> space. If we were to remove it we will have grater issues to address. >>>> >>>> >>>> >>>> - I have confirmed with Andy and we prefer keeping this known >>>> configuration in place to avoid disk usage issues. >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Let me know if I was clear on my explanation and we can see if keeping >>>> an eye on the dailies helps us reducing this occurrences. >>>> >>>> >>>> >>>> Thanks a ton! >>>> >>>> >>>> >>>> Jess >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> >>>> onap-discuss mailing list >>>> >>>> [email protected] >>>> >>>> https://lists.onap.org/mailman/listinfo/onap-discuss >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> [image: Image removed by sender.] >>>> >>>> >>>> >>> >>> >> >
_______________________________________________ onap-discuss mailing list [email protected] https://lists.onap.org/mailman/listinfo/onap-discuss
