Yep. That was an issue. I restarted the ldap server, and things came back up. I need to investigate why the clients aren't properly failing over to the secondary in this situation.
On Sat, Jul 28, 2012 at 10:27 AM, aude <[email protected]> wrote: > Just to follow up, there's obvious issues being reported in > #wikimedia-operations: > > 17:22 < aude> 15:15 <+nagios-wm> PROBLEM - LDAP on virt0 is CRITICAL: > Connection refused > 17:22 < aude> from #wikimedia-operations > 17:22 < aude> 15:59 <+nagios-wm> PROBLEM - Puppetmaster HTTPS on stafford is > CRITICAL: CRITICAL - Socket timeout after 10 seconds > 17:22 < aude> 16:12 <+nagios-wm> RECOVERY - Puppetmaster HTTPS on stafford > is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.002 seconds > > Cheers, > Katie > > > On Sat, Jul 28, 2012 at 7:19 PM, aude <[email protected]> wrote: >> >> I am getting failures in creating new instances on labs today. >> >> >> Jul 28 16:54:24 i-00000371 puppet-agent[3950]: Starting Puppet client >> version 2.7.11 >> Jul 28 16:54:25 i-00000371 puppet-agent[3950]: Could not retrieve catalog >> from remote server: Error 400 on SERVER: Failed when searching for node >> i-00000371.pmtpa.wmflabs: LDAP Search failed >> Jul 28 16:54:25 i-00000371 puppet-agent[3950]: Using cached catalog >> Jul 28 16:54:25 i-00000371 puppet-agent[3950]: Could not retrieve catalog; >> skipping run >> Jul 28 16:55:06 i-00000371 dhclient: DHCPREQUEST of 10.4.0.229 on eth0 to >> 10.4.0.1 port 67 >> Jul 28 16:55:06 i-00000371 dhclient: DHCPACK of 10.4.0.229 from 10.4.0.1 >> Jul 28 16:55:06 i-00000371 dhclient: bound to 10.4.0.229 -- renewal in 53 >> seconds. >> >> >> dhclient repeatedly retries and fails >> >> >> https://labsconsole.wikimedia.org/w/index.php?title=Special:NovaInstance&action=consoleoutput&project=maps&instanceid=i-00000371 >> >> Any ideas what the problem is? probably nothing I can do but wait for Ryan >> or someone? >> >> Cheers, >> Katie >> >> -- >> Board member, Wikimedia District of Columbia >> http://wikimediadc.org >> @wikimediadc / @wikimania2012 >> > > > > -- > Board member, Wikimedia District of Columbia > http://wikimediadc.org > @wikimediadc / @wikimania2012 > > > _______________________________________________ > Labs-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/labs-l > _______________________________________________ Labs-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/labs-l
