No this is a different issue that I'm actively engaging Rackspace to troubleshoot. I've noticed about 1/5 nodes come online without the ability to route to git.opendaylight.org which we do in our initialization phase to download the releng/builder jenkins scripts to initialize the system. It started appearing sometime Thursday as far as I'm aware.
(part of which sets up the Jenkins user so that we can ssh in) Regards, Thanh On Mon, Aug 15, 2016 at 12:23 PM, Lori Jakab <[email protected]> wrote: > Hi, > > The problem was temporarily fixed back then, but I see issues again. See: > > https://jenkins.opendaylight.org/releng/job/lispflowmapping- > csit-1node-performance-only-carbon/14/console > https://jenkins.opendaylight.org/releng/job/lispflowmapping- > csit-1node-performance-only-carbon/15/console > https://jenkins.opendaylight.org/releng/job/lispflowmapping- > csit-1node-performance-only-carbon/16/console > > Basically: > > [lispflowmapping-csit-1node-performance-only-carbon] $ /bin/bash > /tmp/hudson1296607285891504887.sh > OpenStack IPS are 10.29.13.33,10.29.12.207 > SSH not responding on 10.29.12.207. Retrying in 10 seconds... > Warning: Permanently added '10.29.13.33' (ECDSA) to the list of known hosts. > 10.29.13.33 ubuntu-trusty-mininet-2c-2g-f6d > Successfully copied public keys to slave 10.29.13.33 > Process 9670 successfully copied ssh keys. > SSH not responding on 10.29.12.207. Retrying in 10 seconds... > SSH not responding on 10.29.12.207. Retrying in 10 seconds... > SSH not responding on 10.29.12.207. Retrying in 10 seconds... > SSH not responding on 10.29.12.207. Retrying in 10 seconds... > SSH not responding on 10.29.12.207. Retrying in 10 seconds... > SSH not responding on 10.29.12.207. Retrying in 10 seconds... > SSH not responding on 10.29.12.207. Retrying in 10 seconds... > SSH not responding on 10.29.12.207. Retrying in 10 seconds... > > [...] > > Process 9671 failed to copy ssh keys. > Build step 'Execute shell' marked build as failure > Destroying nodes: [DFW/6ad06adf-6dc5-41f9-8944-80909bc686c9, > DFW/4b15596c-6c15-4127-8b64-829cf929f1ca] > [ssh-agent] Stopped. > > > -Lori > > On Fri, Aug 12, 2016 at 11:27 AM, Lori Jakab <[email protected]> > wrote: > >> On Fri, Aug 12, 2016 at 11:16 AM, Thanh Ha <[email protected]> >> wrote: >> >>> I didn't see this email last night but Venkat notified me last night and >>> fixed the patch with: >>> >>> https://git.opendaylight.org/gerrit/43762 >>> >>> It was caused by another patch I put in place here which I thought I >>> tested but turns out I tested on a csit job which only created 1 node and >>> didn't notice so the bug wasn't caught in testing: >>> >>> https://git.opendaylight.org/gerrit/43716 >>> >> >> Great, thank you both! >> >> -Lori >> >> >>> >>> Regards, >>> Thanh >>> >>> >>> On Fri, Aug 12, 2016 at 12:25 AM, Lori Jakab <[email protected] >>> > wrote: >>> >>>> On Thu, Aug 11, 2016 at 9:53 PM, Lori Jakab <[email protected] >>>> > wrote: >>>> >>>>> +release >>>>> +helpdesk >>>>> >>>> >>>> It looks like Jenkins got scared now that I added helpdesk, and the >>>> jobs started passing :) >>>> >>>> -Lori >>>> >>>> >>>>> >>>>> >>>>> On Thu, Aug 11, 2016 at 3:14 PM, Lori Jakab < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Our performance jobs have been failing consistently for the day, with >>>>>> a message like this (see [0] for example): >>>>>> >>>>>> [lispflowmapping-csit-1node-performance-only-boron] $ /bin/bash >>>>>> /tmp/hudson210770443963292763.sh >>>>>> OpenStack IPS are 10.29.13.20,10.29.13.50 >>>>>> /tmp/hudson210770443963292763.sh: line 31: wait: pid 96109611 is not a >>>>>> child of this shell >>>>>> Process 96109611 failed to copy ssh keys. >>>>>> Warning: Permanently added '10.29.13.50' (ECDSA) to the list of known >>>>>> hosts. >>>>>> Warning: Permanently added '10.29.13.20' (ECDSA) to the list of known >>>>>> hosts. >>>>>> 10.29.13.50 ubuntu-trusty-mininet-2c-2g-491 >>>>>> Successfully copied public keys to slave 10.29.13.50 >>>>>> 10.29.13.20 centos7-java-builder-2c-4g-346 >>>>>> Build step 'Execute shell' marked build as failure >>>>>> >>>>>> >>>>>> Any ideas on what needs to be done to fix it? >>>>>> >>>>>> Thanks, >>>>>> -Lori >>>>>> >>>>>> [0] https://jenkins.opendaylight.org/releng/job/lispflowmapp >>>>>> ing-csit-1node-performance-only-boron/758/console >>>>>> >>>>> >>> >> >
_______________________________________________ infrastructure mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/infrastructure
