Re: [Wikimedia-l] Widespread perfomance issues

[Cristian Consonni]

Hi Rogol,

On 16/05/2017 15:44, Faidon Liambotis wrote:
> This was a small outage, affecting only a small portion of European
> traffic (and tiny portion of our global traffic) and not what we'd call
> a "widespread" one -- very far from it.

+1
I think I have to apologize to the Operations team for the the wording I
have chosen for the title of this thread. In hindsight, this issue was
minor.

I was taken aback by the combination of the fact that we (at WM-IT) were
contacted by a journalist (which, in hindsight, put on us some pressure
about trying to understand the causes of the problem); we were in fact
experiencing some problems (both Wikimedia Italia's office and myself at
home use Fastweb as ISP); and, finally, I have misjudged the information
contained in the status page.

> This could have been remediated by either a) the intermediate provider
> fixing or working around the issue b) us disabling the use of that
> provider temporarily c) the affected ISPs (like Fastweb) disabling the
> use of that provider temporarily. In this case, we did not wait for (a)
> or (c) and worked around the issue ourselves, faster than Fastweb etc.
> did (and in less than an hour even since the initial report). I think
> the response from our side and our engineers (and especially Arzhel) was
> pretty stellar.

It was.
Thank you, Faidon, and thanks to all the team for the amazing work you do.

Cristian

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Widespread perfomance issues

[Faidon Liambotis]

Hi Rogol,

I'm not sure what you're referring to when you're talking about a
"recovery plan". These are all pretty standard day-to-day operations for
us.

This was a small outage, affecting only a small portion of European
traffic (and tiny portion of our global traffic) and not what we'd call
a "widespread" one -- very far from it. It wasn't even noticeable in
traffic graphs at all. This was the result of a networking issue in an
intermediate provider between us and the ISPs affected, so nothing that
was in our direct control.

This could have been remediated by either a) the intermediate provider
fixing or working around the issue b) us disabling the use of that
provider temporarily c) the affected ISPs (like Fastweb) disabling the
use of that provider temporarily. In this case, we did not wait for (a)
or (c) and worked around the issue ourselves, faster than Fastweb etc.
did (and in less than an hour even since the initial report). I think
the response from our side and our engineers (and especially Arzhel) was
pretty stellar.

Faidon

On Mon, May 15, 2017 at 09:00:06PM +0100, Rogol Domedonfors wrote:
> Faidon
> 
> Do you believe that your recovery plan was adequate or will you be
> reviewing it in the light of this widespread outage?
> 
> "Rogol"
> 
> On Mon, May 15, 2017 at 12:02 PM, Faidon Liambotis 
> wrote:
> 
> > Hi Cristian,
> >
> > We've had some network connectivity issues with one of our ISPs in our
> > European datacenter that were probably the source of your problems
> > (FastWeb in Italy were among the affected ISPs). This was a localized
> > issue -- only in the European datacenter and one out of six major
> > network carriers and dozens of network connectivity partners.
> >
> > The issue has been worked around as of 09:26 UTC and the situation is
> > being monitored.
> >
> > This issue is being tracked at:
> > https://phabricator.wikimedia.org/T165288
> >
> > Do let us know if this is not the case for you of if you're experiencing
> > any trouble, here, off-list or directly on that Phabricator task.
> >
> > Thanks,
> > Faidon
> > --
> > Faidon Liambotis
> > Principal Operations Engineer
> > Wikimedia Foundation
> >
> > On Mon, May 15, 2017 at 11:02:39AM +0200, Cristian Consonni wrote:
> > > Hi,
> > >
> > > At Wikimedia Italia have been contacted in the last few minutes because
> > > Wikipedia seems unreachable and very slow from Italy.
> > >
> > > The status pages for servers and services signals several perfomance
> > issues:
> > > https://status.wikimedia.org/
> > >
> > > Does somebody know what's going on?
> > >
> > > Thank you.
> > >
> > > Cristian
> > >
> > > ___
> > > Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> > wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> > wiki/Wikimedia-l
> > > New messages to: Wikimedia-l@lists.wikimedia.org
> > > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> > 
> >
> > ___
> > Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> > wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> > wiki/Wikimedia-l
> > New messages to: Wikimedia-l@lists.wikimedia.org
> > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> > 
> >
> ___
> Wikimedia-l mailing list, guidelines at: 
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
> https://meta.wikimedia.org/wiki/Wikimedia-l
> New messages to: Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 
> 

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Widespread perfomance issues

[Rogol Domedonfors]

Faidon

Do you believe that your recovery plan was adequate or will you be
reviewing it in the light of this widespread outage?

"Rogol"

On Mon, May 15, 2017 at 12:02 PM, Faidon Liambotis 
wrote:

> Hi Cristian,
>
> We've had some network connectivity issues with one of our ISPs in our
> European datacenter that were probably the source of your problems
> (FastWeb in Italy were among the affected ISPs). This was a localized
> issue -- only in the European datacenter and one out of six major
> network carriers and dozens of network connectivity partners.
>
> The issue has been worked around as of 09:26 UTC and the situation is
> being monitored.
>
> This issue is being tracked at:
> https://phabricator.wikimedia.org/T165288
>
> Do let us know if this is not the case for you of if you're experiencing
> any trouble, here, off-list or directly on that Phabricator task.
>
> Thanks,
> Faidon
> --
> Faidon Liambotis
> Principal Operations Engineer
> Wikimedia Foundation
>
> On Mon, May 15, 2017 at 11:02:39AM +0200, Cristian Consonni wrote:
> > Hi,
> >
> > At Wikimedia Italia have been contacted in the last few minutes because
> > Wikipedia seems unreachable and very slow from Italy.
> >
> > The status pages for servers and services signals several perfomance
> issues:
> > https://status.wikimedia.org/
> >
> > Does somebody know what's going on?
> >
> > Thank you.
> >
> > Cristian
> >
> > ___
> > Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> wiki/Wikimedia-l
> > New messages to: Wikimedia-l@lists.wikimedia.org
> > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
> ___
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> wiki/Wikimedia-l
> New messages to: Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Widespread perfomance issues

[Cristian Consonni]

Hi Faidon,

On 15/05/2017 13:02, Faidon Liambotis wrote:
> Hi Cristian,
> 
> We've had some network connectivity issues with one of our ISPs in our
> European datacenter that were probably the source of your problems
> (FastWeb in Italy were among the affected ISPs). This was a localized
> issue -- only in the European datacenter and one out of six major
> network carriers and dozens of network connectivity partners.
> 
> The issue has been worked around as of 09:26 UTC and the situation is
> being monitored.
> 
> This issue is being tracked at:
> https://phabricator.wikimedia.org/T165288
> 
> Do let us know if this is not the case for you of if you're experiencing
> any trouble, here, off-list or directly on that Phabricator task.

Thanks for the update.
Keep up the good work.

Ciao,

Cristian

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Widespread perfomance issues

[Alexandros Kosiaris]

Hi,

Probably https://phabricator.wikimedia.org/T165288.

On Mon, May 15, 2017 at 2:04 PM, Gergő Tisza  wrote:
> On Mon, May 15, 2017 at 11:02 AM, Cristian Consonni 
> wrote:
>
>> At Wikimedia Italia have been contacted in the last few minutes because
>> Wikipedia seems unreachable and very slow from Italy.
>>
>> The status pages for servers and services signals several perfomance
>> issues:
>> https://status.wikimedia.org/
>>
>> Does somebody know what's going on?
>>
>
> Maybe related to https://phabricator.wikimedia.org/T165252 (hardware
> problems in one of the caching servers)?
> ___
> Wikimedia-l mailing list, guidelines at: 
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
> https://meta.wikimedia.org/wiki/Wikimedia-l
> New messages to: Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 
> 



-- 
Alexandros Kosiaris 

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Widespread perfomance issues

[Faidon Liambotis]

Hi Cristian,

We've had some network connectivity issues with one of our ISPs in our
European datacenter that were probably the source of your problems
(FastWeb in Italy were among the affected ISPs). This was a localized
issue -- only in the European datacenter and one out of six major
network carriers and dozens of network connectivity partners.

The issue has been worked around as of 09:26 UTC and the situation is
being monitored.

This issue is being tracked at:
https://phabricator.wikimedia.org/T165288

Do let us know if this is not the case for you of if you're experiencing
any trouble, here, off-list or directly on that Phabricator task.

Thanks,
Faidon
--
Faidon Liambotis
Principal Operations Engineer
Wikimedia Foundation

On Mon, May 15, 2017 at 11:02:39AM +0200, Cristian Consonni wrote:
> Hi,
> 
> At Wikimedia Italia have been contacted in the last few minutes because
> Wikipedia seems unreachable and very slow from Italy.
> 
> The status pages for servers and services signals several perfomance issues:
> https://status.wikimedia.org/
> 
> Does somebody know what's going on?
> 
> Thank you.
> 
> Cristian
> 
> ___
> Wikimedia-l mailing list, guidelines at: 
> https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
> https://meta.wikimedia.org/wiki/Wikimedia-l
> New messages to: Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 
> 

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Widespread perfomance issues

[Gergő Tisza]

On Mon, May 15, 2017 at 11:02 AM, Cristian Consonni 
wrote:

> At Wikimedia Italia have been contacted in the last few minutes because
> Wikipedia seems unreachable and very slow from Italy.
>
> The status pages for servers and services signals several perfomance
> issues:
> https://status.wikimedia.org/
>
> Does somebody know what's going on?
>

Maybe related to https://phabricator.wikimedia.org/T165252 (hardware
problems in one of the caching servers)?
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Widespread perfomance issues

[kayode yussuf]

Hi Sam,
if this will help, I think it was your provider. I was able to access Wikipedia 
at that same time last night. Wikipedia is also up as at this morning. Kayode 
Yussuf 

On Monday, May 15, 2017 10:40 AM, Oloruntoba Oyeyele 
 wrote:
 

 Throughout last night (around 7pm WAT till I slept off), I couldn't access
Wikipedia in Nigeria. I thought it was my service provider acting funny at
first. I'll like to know what the problem is, as well.

Sam.

On Mon, May 15, 2017 at 10:03 AM, Cristian Consonni 
wrote:

> On 15/05/2017 11:02, Cristian Consonni wrote:
> > Hi,
> >
> > At Wikimedia Italia have been contacted in the last few minutes because
> > Wikipedia seems unreachable and very slow from Italy.
>
> (Ah, I forgot to mention. by a journalist from ANSA, the national news
> agency)
>
> Cristian
>
> ___
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> wiki/Wikimedia-l
> New messages to: Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 


   
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Widespread perfomance issues

[Oloruntoba Oyeyele]

Throughout last night (around 7pm WAT till I slept off), I couldn't access
Wikipedia in Nigeria. I thought it was my service provider acting funny at
first. I'll like to know what the problem is, as well.

Sam.

On Mon, May 15, 2017 at 10:03 AM, Cristian Consonni 
wrote:

> On 15/05/2017 11:02, Cristian Consonni wrote:
> > Hi,
> >
> > At Wikimedia Italia have been contacted in the last few minutes because
> > Wikipedia seems unreachable and very slow from Italy.
>
> (Ah, I forgot to mention. by a journalist from ANSA, the national news
> agency)
>
> Cristian
>
> ___
> Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/
> wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/
> wiki/Wikimedia-l
> New messages to: Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
>
___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Widespread perfomance issues

[Cristian Consonni]

On 15/05/2017 11:02, Cristian Consonni wrote:
> Hi,
> 
> At Wikimedia Italia have been contacted in the last few minutes because
> Wikipedia seems unreachable and very slow from Italy.

(Ah, I forgot to mention. by a journalist from ANSA, the national news
agency)

Cristian

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

[Wikimedia-l] Widespread perfomance issues

[Cristian Consonni]

Hi,

At Wikimedia Italia have been contacted in the last few minutes because
Wikipedia seems unreachable and very slow from Italy.

The status pages for servers and services signals several perfomance issues:
https://status.wikimedia.org/

Does somebody know what's going on?

Thank you.

Cristian

___
Wikimedia-l mailing list, guidelines at: 
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and 
https://meta.wikimedia.org/wiki/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,