Re: [Wikitech-l] Gerrit outage

2019-03-20 Thread Lars Wirzenius
On Wed, Mar 20, 2019 at 06:56:19PM +, Pine W wrote: > I'd like to give a quick thanks to folks who have been dealing with > turbulence. Me too. It's inspirational and awe-inspiring to see people work on this. ___ Wikitech-l mailing list

Re: [Wikitech-l] Gerrit outage

2019-03-20 Thread Pine W
I'd like to give a quick thanks to folks who have been dealing with turbulence. I think that short term mitigation measures sound reasonable, while longer term improvements are planned and developed. Pine ( https://meta.wikimedia.org/wiki/User:Pine )

Re: [Wikitech-l] Gerrit outage

2019-03-19 Thread Alexandros Kosiaris
Hello Fæ, While I understand and agree with your point, I must point out that this 4 days have been hectic on many people from multiple teams. The amount of work to cleanup one person's destructive half hour spree is staggering. We need better tooling for sure to combat this, something that while

Re: [Wikitech-l] Gerrit outage

2019-03-19 Thread
Thanks to everyone who helped sort this out. In some ways, the vandalism neatly demonstrates how Wikimedia projects rely on trust. When these things happen, it is a nice reminder that our open values mean that we should take a light approach to security whenever the potential exposure is always

Re: [Wikitech-l] Gerrit outage

2019-03-19 Thread Alexandros Kosiaris
Gerrit is back up. Almost all of the vandalism has been cleaned up, some minor stuff remains, we will clean that up as well. On Tue, Mar 19, 2019 at 1:42 PM planetenxin wrote: > > Am 19.03.2019 um 12:21 schrieb Andre Klapper: > > planetenxin: Sorry for my previous message, was not meant to be

Re: [Wikitech-l] Gerrit outage

2019-03-19 Thread planetenxin
Am 19.03.2019 um 12:21 schrieb Andre Klapper: > planetenxin: Sorry for my previous message, was not meant to be rude. no worries. Hope, that Gerrit is back alive soon. :-) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org

Re: [Wikitech-l] Gerrit outage

2019-03-19 Thread Andre Klapper
On Tue, 2019-03-19 at 10:49 +, Lewis Cawte via Wikitech-l wrote: > Not everyone is aware that the process of cleaning up the > vandalism/fixing Gerrit includes Gerrit being down temporarily. Right. Should have spent more time to rephrase and explicitly say so. Thanks for pointing that out.

Re: [Wikitech-l] Gerrit outage

2019-03-19 Thread Dan Garry (Deskana)
On Tue, 19 Mar 2019 at 10:50, Lewis Cawte via Wikitech-l < wikitech-l@lists.wikimedia.org> wrote: > Not everyone is aware that the process of cleaning up the vandalism/fixing > Gerrit includes Gerrit being down temporarily. > > Do I need to include a reminder link to WP:AGF / WP:DICK? > That

Re: [Wikitech-l] Gerrit outage

2019-03-19 Thread Lewis Cawte via Wikitech-l
Not everyone is aware that the process of cleaning up the vandalism/fixing Gerrit includes Gerrit being down temporarily. Do I need to include a reminder link to WP:AGF / WP:DICK? -- Lewis Cawte On Tue, 19 Mar 2019, 10:27 Andre Klapper, wrote: > On Tue, 2019-03-19 at 10:59 +0100, planetenxin

Re: [Wikitech-l] Gerrit outage

2019-03-19 Thread Andre Klapper
On Tue, 2019-03-19 at 10:59 +0100, planetenxin wrote: > Gerrit seems to be offline again. Please read the other latest thread on this very mailing list. andre -- Andre Klapper | Bugwrangler / Developer Advocate https://blogs.gnome.org/aklapper/ ___

Re: [Wikitech-l] Gerrit outage

2019-03-19 Thread planetenxin
Gerrit seems to be offline again. > On 16 March 2019, Wikimedia Foundation staff observed suspicious activity > associated with Gerrit and as a precautionary step has taken Gerrit offline > pending investigation. ___ Wikitech-l mailing list

Re: [Wikitech-l] Gerrit outage

2019-03-18 Thread Tyler Cipriani
Hello, As part of cleanup and response Gerrit's use of http tokens has been disabled. You should still be able to use the http REST api using your LDAP password. Gerrit's command-line tools [0] that operate via SSH are also still available. -- Tyler [0].

Re: [Wikitech-l] Gerrit outage

2019-03-17 Thread John Bennett
Hello, Today we have seen Phabricator vandalism from an attacker who was also responsible for the Gerrit outage yesterday. I’d like to clarify a comment I made yesterday and provide as many additional details as I can while still maintaining operational security. While no user accounts were

Re: [Wikitech-l] Gerrit outage

2019-03-16 Thread Paladox via Wikitech-l
The watchlist should be in All-Users. If someone removed your watchlist you can clone that repo. Then in .git/config change refs/heads to just refs/*. Then it should show some refs (I forget which one you check out so you should try them all) you can use git log to see if someone git committed

Re: [Wikitech-l] Gerrit outage

2019-03-16 Thread MGChecker
: Samstag, 16. März 2019 21:14 An: Wikimedia developers Betreff: Re: [Wikitech-l] Gerrit outage Thanks for the updates and for everyone who was or is working on a weekend day. Sometime in the next few weeks if you can publish an incident report that has any sensitive information redacted, I would like

Re: [Wikitech-l] Gerrit outage

2019-03-16 Thread Pine W
Thanks for the updates and for everyone who was or is working on a weekend day. Sometime in the next few weeks if you can publish an incident report that has any sensitive information redacted, I would like to read it. Pine ( https://meta.wikimedia.org/wiki/User:Pine ) On Sat, Mar 16, 2019,

Re: [Wikitech-l] Gerrit outage

2019-03-16 Thread John Bennett
Hello, Gerrit is available again but we are continuing to investigate the suspicious activity. Our preliminary findings point to no users or production systems being compromised and no loss of any confidential information. As we continue to investigate over the next few days we will add any

[Wikitech-l] Gerrit outage

2019-03-16 Thread John Bennett
Hello, On 16 March 2019, Wikimedia Foundation staff observed suspicious activity associated with Gerrit and as a precautionary step has taken Gerrit offline pending investigation. The Wikimedia Foundation's Security, Site Reliability Engineering and Release Engineering teams are investigating