Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Federico Leva (Nemo)]

Leslie Carr, 14/08/2013 22:30:

http://status.wikimedia.org is from external monitors


I don't see any annual or multi-year stats there and I don't think they 
monitor readon-only vs. rw state.


Nemo

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Leslie Carr]

http://status.wikimedia.org is from external monitors

Sent from my mobile. Please excuse the brevity and typos.
On Aug 14, 2013 4:44 AM, "Federico Leva (Nemo)"  wrote:

> Chris McKenna, 14/08/2013 00:00:
>
>> On Tue, 13 Aug 2013, Oliver Keyes wrote:
>>
>>>
>>> I'd like to think Engineering do a pretty good job at uptime for core
>>> services - when was the last time you saw Wikipedia down for any extended
>>> period of time? - regardless of what day of the week it is.
>>>
>>
>> Wikipedia uptime is certianly much better these days than it was around
>> 2005/6 when there were at least two websites dedicated to reporting its
>> status. Back then it wasn't uncommon to experience major slowdowns,
>> periods of non-responsiveness, extended read only periods and edits
>> failing (sometimes silently) due to (iirc) overloaded servers. I can't
>> remember the last time I saw anything like that, but it certainly wasn't
>> 2013.
>>
>
> Btw it would be nice to have stats on uptime, or at least some assessment
> of the progress for the "stabilizing infrastructure" 5 years strategic
> goal. I couldn't find any recently, I asked both on mailing lists and on
> Meta.
>  Stabilizing_the_infrastructure
> **>
>
> Nemo
>
> __**_
> Wikimedia-l mailing list
> Wikimedia-l@lists.wikimedia.**org 
> Unsubscribe: 
> https://lists.wikimedia.org/**mailman/listinfo/wikimedia-l,
>  ?subject=**unsubscribe>
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Federico Leva (Nemo)]

Chris McKenna, 14/08/2013 00:00:

On Tue, 13 Aug 2013, Oliver Keyes wrote:


I'd like to think Engineering do a pretty good job at uptime for core
services - when was the last time you saw Wikipedia down for any extended
period of time? - regardless of what day of the week it is.


Wikipedia uptime is certianly much better these days than it was around
2005/6 when there were at least two websites dedicated to reporting its
status. Back then it wasn't uncommon to experience major slowdowns,
periods of non-responsiveness, extended read only periods and edits
failing (sometimes silently) due to (iirc) overloaded servers. I can't
remember the last time I saw anything like that, but it certainly wasn't
2013.


Btw it would be nice to have stats on uptime, or at least some 
assessment of the progress for the "stabilizing infrastructure" 5 years 
strategic goal. I couldn't find any recently, I asked both on mailing 
lists and on Meta.



Nemo

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Amir E. Aharoni]

2013/8/14 Tim Starling :
> On 14/08/13 07:40, Oliver Keyes wrote:
>> We have weekend support - for core services, which constitute our MediaWiki
>> instances. Git, however, is not a core service - as Max accurately notes,
>> while it makes development finicky and frustrating,
>
> I didn't know anyone used git.wikimedia.org. I think I've only visited
> it once.

I sometimes use it to point people to code, especially new volunteers
who are starting to learn the code.

Other people very frequently send me git.wikimedia.org links when they
are asking me about the code.

So yes, it's not as important as keeping *.wikipedia.org online, but
it is definitely far from negligible.

--
Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי
http://aharoni.wordpress.com
‪“We're living in pieces,
I want to live in peace.” – T. Moore‬

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Tim Starling]

On 14/08/13 07:40, Oliver Keyes wrote:
> We have weekend support - for core services, which constitute our MediaWiki
> instances. Git, however, is not a core service - as Max accurately notes,
> while it makes development finicky and frustrating,

I didn't know anyone used git.wikimedia.org. I think I've only visited
it once.

We're not talking about Git being down, just about one of the two web
viewers being down (the other being GitHub). Web viewers of source
code are a luxury -- they're handy when you're not a developer and so
couldn't be bothered to clone the repository, but they're hardly a
human right.

Despite that, note that Leslie Carr did work on fixing it on the weekend.

-- Tim Starling


___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Chris McKenna]

On Tue, 13 Aug 2013, Oliver Keyes wrote:


I'd like to think Engineering do a pretty good job at uptime for core
services - when was the last time you saw Wikipedia down for any extended
period of time? - regardless of what day of the week it is.


Wikipedia uptime is certianly much better these days than it was around 
2005/6 when there were at least two websites dedicated to reporting its 
status. Back then it wasn't uncommon to experience major slowdowns, 
periods of non-responsiveness, extended read only periods and edits 
failing (sometimes silently) due to (iirc) overloaded servers. I can't 
remember the last time I saw anything like that, but it certainly wasn't 
2013.


The folks responsible for this certainly deserve kudos for their efforts.


Chris McKenna

cmcke...@sucs.org
www.sucs.org/~cmckenna


The essential things in life are seen not with the eyes,
but with the heart

Antoine de Saint Exupery


___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Greg Grossmeier]

A quick update I just sent to wikitech-l (where this conversation should
be happening):

http://lists.wikimedia.org/pipermail/wikitech-l/2013-August/071171.html

Full text below:



As you all are aware, git.wikimedia.org (powered by GitBlit) has been
unstable. The root cause is basically that there is insufficient caching
mechanisms in place exacerbated by a lack of robust rules for web
indexers (Google, Bing, etc).

Currently, the solution is to disallow all indexers from indexing
git.wikimedia.org via the robots.txt file
(https://git.wikimedia.org/robots.txt). We hope to open up indexing in a
more controlled fashion soon.

For now, git.wikimedia.org should be stable for everyday use.

You can follow along with the caching developments on bug 49371:
https://bugzilla.wikimedia.org/show_bug.cgi?id=49371

Thanks to Chad and Ariel, especially, who have been working to make sure
this is resolved quickly and permanently.

Greg

-- 
| Greg GrossmeierGPG: B2FA 27B1 F7EB D327 6B8E |
| identi.ca: @gregA18D 1138 8E47 FAC8 1C7D |


signature.asc
Description: Digital signature
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Oliver Keyes]

We have weekend support - for core services, which constitute our MediaWiki
instances. Git, however, is not a core service - as Max accurately notes,
while it makes development finicky and frustrating, it doesn't impact on
our core mission, which is to provide the sum of human knowledge, for
everybody. It being down for a short period of time is not critical -
Wikipedia being down? Well, I think we'd agree that's a pretty serious
problem.

I'd like to think Engineering do a pretty good job at uptime for core
services - when was the last time you saw Wikipedia down for any extended
period of time? - regardless of what day of the week it is. I recall a few
months ago that we had weird, inconsistent cacheing problems that were
causing trouble for users - Leslie Carr, one of our Ops engineers, worked
11 hours on Friday and (iirc) 14 hours over the weekend to get it fixed.
This isn't the only story in that vein - where there are critical problems
to critical services, they tend to be fixed as promptly as possible, and
staffers pull out all of the stops to make it happen. But it's not
realistic for ops and dev/ops to maintain 24/7 uptime not just for our core
services, but for every single one of the miscellaneous instances we host.
If we had 150 engineers in Platform and Operations, then I might be singing
a different tune, but unfortunately we don't ;).

On the other hand, if you would like to donate enough money to pay for a
team of 6 to maintain our git instance 24/7, I'm sure we could set
something up :P


On 12 August 2013 20:19, Richard Farmbrough wrote:

> I maintained 24/7 support with a team of 6. WMF has 150 staff and does not
> have weekend support.  The tail is wagging the dog.
>
>
>
> __**_
> Wikimedia-l mailing list
> Wikimedia-l@lists.wikimedia.**org 
> Unsubscribe: 
> https://lists.wikimedia.org/**mailman/listinfo/wikimedia-l,
>  ?subject=**unsubscribe>
>



-- 
Oliver Keyes
Community Liaison, Product Development
Wikimedia Foundation
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Richard Farmbrough]

I maintained 24/7 support with a team of 6. WMF has 150 staff and does 
not have weekend support.  The tail is wagging the dog.



___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[MZMcBride]

Huib Laurens wrote:
>I always believed that our servers were monitored 24/7? But nobody seems
>to be around to fix a core part in our systems?

Hi Huib.

You've been around quite a long time, so it shouldn't be new information
to you that the appropriate mailing list for an issue like this is
wikitech-l (where there's already an ongoing thread), not wikimedia-l. And
you should also know that for issues like this, the best place to search
is Bugzilla, as both Nemo and myself have now pointed out (specifically
).

You seem to be a bit confused about core v. non-core services. In the
Wikimedia world, core functionality generally means that the wikis are
accessible for, at the very minimum, read access. Peripheral services and
sites have varying levels of criticality, though there's no rating system
in which git.wikimedia.org (a simple Git repository viewer) would be
considered core (though it being completely down can be considered
critical, as it makes development work more tedious and annoying).

Obviously the operations team has a number of monitoring systems in place.
The public monitoring dashboard is here: .

MZMcBride



___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Federico Leva (Nemo)]

Huib Laurens, 10/08/2013 17:31:

I always believed that our servers where monitored 24/7?


Not magically, despite the amount of unicorns bought by the WMF. You're 
looking for https://bugzilla.wikimedia.org/show_bug.cgi?id=51983



But nobody seems
to be arround to fix a core part in our systems?


No development is really prevented by gitblit being down; business as 
usual continues on gerrit.wikimedia.org and this additional reliability 
is actually what they wanted to achieve by moving gitblit (formerly 
gitweb) to a separate server.


Nemo

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

Re: [Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Martijn Hoekstra]

On Sat, Aug 10, 2013 at 5:31 PM, Huib Laurens  wrote:

> Hello,
>
> I always believed that our servers where monitored 24/7? But nobody seems
> to be arround to fix a core part in our systems?
>
> Huib
>
> -- Forwarded message --
> From: rupert THURNER 
> Date: Sat, Aug 10, 2013 at 5:24 PM
> Subject: [Wikimania-l] git.wikimedia.org dead due to wikimania ;)
> To: "Wikimania general list (open subscription)" <
> wikimani...@lists.wikimedia.org>
>
>
> it seems git.wikimedia.org is down due:
> * no volunteer has access
> * no paid person works on weekends
> * everybody else is at wikimania
>
> rupert.
>


Doesn't this imply that nobody is at Wikimania?


>
> ___
> Wikimania-l mailing list
> wikimani...@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>
>
>
> --
> Met vriendelijke groet,
>
> Huib Laurens
> ___
> Wikimedia-l mailing list
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
> 
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, 

[Wikimedia-l] Fwd: [Wikimania-l] git.wikimedia.org dead due to wikimania ; )

[Huib Laurens]

Hello,

I always believed that our servers where monitored 24/7? But nobody seems
to be arround to fix a core part in our systems?

Huib

-- Forwarded message --
From: rupert THURNER 
Date: Sat, Aug 10, 2013 at 5:24 PM
Subject: [Wikimania-l] git.wikimedia.org dead due to wikimania ;)
To: "Wikimania general list (open subscription)" <
wikimani...@lists.wikimedia.org>


it seems git.wikimedia.org is down due:
* no volunteer has access
* no paid person works on weekends
* everybody else is at wikimania

rupert.

___
Wikimania-l mailing list
wikimani...@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l



-- 
Met vriendelijke groet,

Huib Laurens
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,