Jamo, Thanks for helping Brady out with that.
The root cause on the failure to checkout the refs was that our git mirrors weren't fully in sync for some reason. I've banged on the replication system a bunch and fixed a few timeouts that I was seeing in the logs. This has fixed the missing ref that was pointed out and I suspect we've fixed it for a couple of other projects that may not have noticed the failure yet either. I'm no longer getting weird replication delays or timeouts and I've forced the replication subsystem to do full replication across all mirror destinations (GitHub for merged changes only and our in cloud mirror for complete mirror) a few times with no errors. We'll definitely keep an eye on the replication log for a bit to make sure we aren't getting any other failures though! -Andy- On 12/22/2017 11:56 AM, Brady Johnson wrote: > Jamo, > > Awesome, thanks for the help. > > Regards, > > *Brady Johnson* > [email protected] <mailto:[email protected]> > > <https://twitter.com/inocybetech><http://www.inocybe.com/><https://www.linkedin.com/company/2661537?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2661537%2Cidx%3A1-1-1%2CtarId%3A1441300264767%2Ctas%3Ainocybe>Screenshot > 2017-02-14 at 10.43.55 AM.png > <https://www.youtube.com/channel/UC9uUWABdPR0Je9Du_15FCkw> > > > > On Fri, Dec 22, 2017 at 1:41 PM, Jamo Luhrsen <[email protected] > <mailto:[email protected]>> wrote: > > Brady, > > On 12/22/2017 08:45 AM, Brady Johnson wrote: > > Andrew, > > > > I have some serious issues with a gerrit patch that I did a 'recheck' > on today, as you can see here [0] and [1] > > > > I get errors in the jenkins job like: > > > > 15:23:29 No JDK named ‘{jdk}’ found > > This looks like it's there in other jobs, passing jobs, so probably > benign. But needs > to get fixed as it would worry me too. > > > > 15:23:45 ERROR: Error fetching remote repo 'origin' > > 15:23:45 hudson.plugins.git.GitException: Failed to fetch from > git://devvexx.opendaylight.org/mirror/sfc > <http://devvexx.opendaylight.org/mirror/sfc> > > <http://devvexx.opendaylight.org/mirror/sfc > <http://devvexx.opendaylight.org/mirror/sfc>> > > > > 15:23:45 stderr: fatal: Couldn't find remote ref > refs/changes/07/65007/3 > > I could reproduce this same problem locally, but can't explain it. > If you fetch patch set 2, > it works. but failed on 3. > > anyway, I pulled your gerrit locally, rebased and pushed ps4 and > that was ok. > > the distro check verify job passed, but the build failed (timed out > in SFT after 90m) > > hope it helps. > > Thanks, > JamO > > > > > > [0] https://git.opendaylight.org/gerrit/#/c/65007/ > <https://git.opendaylight.org/gerrit/#/c/65007/> > > [1] > https://jenkins.opendaylight.org/releng/job/sfc-distribution-check-oxygen/272/console > > <https://jenkins.opendaylight.org/releng/job/sfc-distribution-check-oxygen/272/console> > > > > > > Regards, > > > > *Brady Johnson* > > [email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > > > > <https://twitter.com/inocybetech > <https://twitter.com/inocybetech>><http://www.inocybe.com/ > > <http://www.inocybe.com/>><https://www.linkedin.com/company/2661537?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2661537%2Cidx%3A1-1-1%2CtarId%3A1441300264767%2Ctas%3Ainocybe > > <https://www.linkedin.com/company/2661537?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2661537%2Cidx%3A1-1-1%2CtarId%3A1441300264767%2Ctas%3Ainocybe>>Screenshot > > 2017-02-14 at 10.43.55 AM.png > <https://www.youtube.com/channel/UC9uUWABdPR0Je9Du_15FCkw > <https://www.youtube.com/channel/UC9uUWABdPR0Je9Du_15FCkw>> > > > > > > > > On Thu, Dec 21, 2017 at 8:16 AM, Andrew Grimberg > <[email protected] <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>>> wrote: > > > > We had some issues with the logs getting moved to the new system. > So, > > there's a small window in which some logs didn't actually get > published > > or there's a location but nothing of relevance. > > > > This issue appears to have corrected itself as some of the log syncs > > finished off after we came back up. We do have the old rackspace > logs > > that are still coming over, but those are all >1 historical at this > point. > > > > -Andy- > > > > On 12/20/2017 06:54 PM, Luis Gomez wrote: > > > I am not sure other people but I cannot get any log from sandbox: > > > > > > > https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/openflowplugin-csit-1node-cbench-only-nitrogen/4 > > <https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/openflowplugin-csit-1node-cbench-only-nitrogen/4> > > > <https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/openflowplugin-csit-1node-cbench-only-nitrogen/4 > > <https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/openflowplugin-csit-1node-cbench-only-nitrogen/4>> > > > > > > It says: You don't have permission to access > > > > /sandbox/vex-yul-odl-jenkins-2/openflowplugin-csit-1node-cbench-only-nitrogen/4 > > > on this server. > > > > > >> On Dec 20, 2017, at 5:01 PM, Andrew Grimberg > > >> <[email protected] > <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>> > <mailto:[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>>> > > >> wrote: > > >> > > >> This work has been completed. > > >> > > >> A couple of notes: > > >> > > >> 1) The Jenkins logs have not been fully synced from the old > server. If > > >> you're looking for old logs, they may not be fully available. > You can > > >> get around this by modifying your /etc/hosts file with the > following > > >> entry to get to _old_ logs: > > >> > > >> 72.3.167.142 logs.opendaylight.org > <http://logs.opendaylight.org> <http://logs.opendaylight.org> > <http://logs.opendaylight.org/> > > >> > > >> 2) We forgot to lower the TTL on the DNS entry related to > logs and > > >> javadocs so those entries are going to stick around for up > to an hour > > >> before switching to the new location. As such, logs for > jobs up until > > >> the maintenance may be visible to you while we're still > syncing but > > >> _new_ stored logs may not be. This will correct itself as > DNS expires in > > >> which case you'll be in the same situation as stated above. > > >> > > >> -Andy- > > >> > > >> On 12/20/2017 02:01 PM, Andrew Grimberg wrote: > > >>> Note: This work will be starting shortly. > > >>> > > >>> -Andy- > > >>> > > >>> On 12/19/2017 05:16 PM, Anil Belur wrote: > > >>>> What: LF will perform maintenance on ODL infrastructure > to implement > > >>>> switch over to a git mirror and nexus. > > >>>> > > >>>> When: 2:00p Wed, Dec 20 2017 - 4:30p Wed, Dec 20 2017 PST > (10:00p Wed, > > >>>> Dec 20 - 12:30a Thu, Dec 21 UTC) > > >>>> > > >>>> Why: LF will implement a switch over to a git mirror and > an nexus > > >>>> instance (tentative if the data sync is complete) which > should decrease > > >>>> any lag in accessing resources within the CI environment. > > >>>> > > >>>> Impact: All ODL servers (Gerrit, Jenkins > (sandbox/releng), Nexus, > > >>>> Spectrometer, Sonar, CLM) may be inaccessible during this > time and > > >>>> Jenkins will be put in shutdown mode before the window > starts and any > > >>>> long running Jenkins jobs _will_ be canceled if they > don't complete > > >>>> before the start of the window. > > >>>> > > >>>> Notices will be posted to the mailing lists and in > #opendaylight on > > >>>> Freenode at the start and end of the maintenance. > > >>>> > > >>>> > > https://www.worldtimebuddy.com/?qm=1&lid=100,2174003,8,5746545&h=8&date=2017-12-20&sln=14-16.5 > > <https://www.worldtimebuddy.com/?qm=1&lid=100,2174003,8,5746545&h=8&date=2017-12-20&sln=14-16.5> > > > > <https://www.worldtimebuddy.com/?qm=1&lid=100,2174003,8,5746545&h=8&date=2017-12-20&sln=14-16.5 > > <https://www.worldtimebuddy.com/?qm=1&lid=100,2174003,8,5746545&h=8&date=2017-12-20&sln=14-16.5>> > > >>>> > > >>> > > >> > > >> -- > > >> Andrew J Grimberg > > >> Lead, IT Release Engineering > > >> The Linux Foundation > > >> > > >> _______________________________________________ > > >> release mailing list > > >> [email protected] > <mailto:[email protected]> > <mailto:[email protected] > <mailto:[email protected]>> > <mailto:[email protected] > <mailto:[email protected]> > > <mailto:[email protected] > <mailto:[email protected]>>> > > >> https://lists.opendaylight.org/mailman/listinfo/release > <https://lists.opendaylight.org/mailman/listinfo/release> > <https://lists.opendaylight.org/mailman/listinfo/release > <https://lists.opendaylight.org/mailman/listinfo/release>> > > > > > > > -- > > Andrew J Grimberg > > Lead, IT Release Engineering > > The Linux Foundation > > > > > > _______________________________________________ > > TSC mailing list > > [email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > > https://lists.opendaylight.org/mailman/listinfo/tsc > <https://lists.opendaylight.org/mailman/listinfo/tsc> > <https://lists.opendaylight.org/mailman/listinfo/tsc > <https://lists.opendaylight.org/mailman/listinfo/tsc>> > > > > > > > > > > _______________________________________________ > > TSC mailing list > > [email protected] <mailto:[email protected]> > > https://lists.opendaylight.org/mailman/listinfo/tsc > <https://lists.opendaylight.org/mailman/listinfo/tsc> > > > > -- Andrew J Grimberg Lead, IT Release Engineering The Linux Foundation
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Discuss mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/discuss
