We do care! I generally don’t check my email on weekends, and I didn’t get any SMS texts about this, hmmmmMMMM!
This is a known occasional problem, see the last reply in this ticket: https://rt.wikimedia.org/Ticket/Display.html?id=6877 This problem is hard to reproduce. I am waiting for a procurement to be finalized before I start doing some failover tests to troubleshoot this further. In the meantime, if this happens, it is easily fixable. In general, you just need to log into analytics1021 or analytics1022 and run kafka preferred-replica-election , which I just did. See: https://wikitech.wikimedia.org/wiki/Analytics/Kraken/Kafka/Administration#Replica_Elections for more info. Thanks Greg, -Ao On Jun 8, 2014, at 11:08 PM, Greg Grossmeier <[email protected]> wrote: > ----- Forwarded message from Greg Grossmeier <[email protected]> ----- > >> Date: Sun, 8 Jun 2014 20:07:41 -0700 >> From: Greg Grossmeier <[email protected]> >> To: Operations Engineers <[email protected]> >> Subject: Re: Varnishkafka Delivery Errors >> >> <quote name="Greg Grossmeier" date="2014-06-08" time="15:31:54 -0700"> >>> greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" >>> \#wikimedia-operations.06-08.log | wc -l >>> 1354 >> >> Last three days (by Eastern US timezone): >> >> greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" >> \#wikimedia-operations.06-06.log | wc -l >> 0 >> >> greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" >> \#wikimedia-operations.06-07.log | wc -l >> 80 >> >> greg@rose:~/logs/irclogs/2014/Freenode$ grep "Varnishkafka Delivery Errors" >> \#wikimedia-operations.06-08.log | wc -l >> 1833 >> >> Do we care? If not, holy cow does this need to be silenced. >> >> (Also, get this in Logstash :) ) >> >> Greg >> >> >> -- >> | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | >> | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D | > > > > ----- End forwarded message ----- > > -- > | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | > | identi.ca: @greg A18D 1138 8E47 FAC8 1C7D | > _______________________________________________ > Analytics mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/analytics _______________________________________________ Analytics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/analytics
