:59 PM Manuel Arostegui
wrote:
> Hello,
>
> I am trying to get this fixed as fast as I can.
>
> There will be brief periods of total unavailability for a section at the
> time. Right now s7 is totally down (the data was almost a week out of date
> anyways).
&
Hello,
I am trying to get this fixed as fast as I can.
There will be brief periods of total unavailability for a section at the
time. Right now s7 is totally down (the data was almost a week out of date
anyways).
Next will be s5.
Sorry for the inconveniences.
Manuel.
On Sun, May 28, 2023 at
Hello,
Tomorrow (Tuesday 15th Feb 2022) at 08:00 AM UTC, we'll set Phabricator to
read-only mode to perform some maintenance on its primary database master:
https://phabricator.wikimedia.org/T301219
Reads will not be affected, but writes will be locked for around 1 minute.
Sorry for the
Hello,
If you don’t use tendril.wikimedia.org or dbtree.wikimedia.org, feel free
to ignore this message.
As of today, tendril is now retired and the main page is replaced with a
list of replacement for different services tendril used to provide:
-
For checking out our dbtree and
Hello,
This will happen in around 45 minutes.
Manuel.
On Mon, Aug 9, 2021 at 1:33 PM Manuel Arostegui
wrote:
> Hello,
>
> On Thursday 12th August at 06:00 AM UTC we'll switch our phabricator
> database master to a different host (
> https://phabricator.wikimedia.org/T288197)
Hello,
On Thursday 12th August at 06:00 AM UTC we'll switch our phabricator
database master to a different host (
https://phabricator.wikimedia.org/T288197)
There'll be 1 or 2 minutes of read-only time. Writes will be blocked, reads
will continue as usual.
Sorry for the inconveniences.
Manuel.
Hello,
This will happen in 40 minutes.
Manuel.
On Tue, Jan 19, 2021 at 4:59 PM Manuel Arostegui
wrote:
> Hello,
>
> We need to restart mariadb daemon on our wikitech master (
> https://phabricator.wikimedia.org/T272388).
>
> When: Thursday 28th January at 09:00AM UTC
Hello,
We need to restart mariadb daemon on our wikitech master (
https://phabricator.wikimedia.org/T272388).
When: Thursday 28th January at 09:00AM UTC
Impact: wikitech will not be available (neither reads nor writes) for 2-5
minutes.
Sorry for any inconvenience.
Manuel.
Hello,
We will be switching over m1 database primary master (
https://phabricator.wikimedia.org/T254556#6247415) , which holds the
etherpad database.
This means that on Thursday 25th at 08:00AM UTC Etherpad will be on
read-only for a few seconds.
Communication will happen on
Hello,
This was successfully done.
MySQL was down for 60 seconds
Gerrit was restarted
OTRS was restarted
recommendation api was restarted
Thanks everyone who helped during the window!
On Thu, Mar 19, 2020 at 9:55 AM Manuel Arostegui
wrote:
> Hello,
>
> This is happening in around
Hello,
This is happening in around 5 minutes.
Manuel.
On Thu, Mar 12, 2020 at 3:04 PM Manuel Arostegui
wrote:
> Hello,
>
> This has been rescheduled from Tuesday 17th 09:00 AM UTC to Thursday 19th
> 09:00 AM UTC
>
> Sorry for any inconvenience.
>
> Manuel.
>
> On
Hello,
This has been rescheduled from Tuesday 17th 09:00 AM UTC to Thursday 19th
09:00 AM UTC
Sorry for any inconvenience.
Manuel.
On Tue, Mar 10, 2020 at 11:52 AM Manuel Arostegui
wrote:
> Hello,
>
> Tuesday 17th at 09:00 AM UTC we will be restarting db1132 as part
> of T
Hello,
Tuesday 17th at 09:00 AM UTC we will be restarting db1132 as part
of T239791.
We will take the opportunity to upgrade MySQL there.
m2 holds the following databases:
- OTRS
- Recommendations API
- Reviewdb (gerrit)
- Debmonitor
We expect read-only time to be around 30-60 seconds, also,
This was done successfully.
Downtime was from 09:00:28 to 09:01:14
On Thu, Feb 20, 2020 at 9:36 AM Manuel Arostegui
wrote:
> This will happen in 25 minutes.
>
> On Wed, Feb 12, 2020 at 2:53 PM Manuel Arostegui
> wrote:
>
>> Hello,
>>
>> *Thursday 20th at
This will happen in 25 minutes.
On Wed, Feb 12, 2020 at 2:53 PM Manuel Arostegui
wrote:
> Hello,
>
> *Thursday 20th at 09:00AM UTC* we'll be restarting and upgrading db1135
> (m2 misc primary database master) as part of
> https://phabricator.wikimedia.org/T239791
>
This was done successfully. Downtime was 56 seconds.
Thanks Mukunda for being around and helping out during the maintenance.
On Thu, Feb 13, 2020 at 6:44 AM Manuel Arostegui
wrote:
> Hello,
>
> This will start in 15 minutes.
>
> On Wed, Feb 12, 2020 at 7:41 PM Manuel Aro
Hello,
This will start in 15 minutes.
On Wed, Feb 12, 2020 at 7:41 PM Manuel Arostegui
wrote:
> Hello,
>
> Thursday 13th at 06:00 AM UTC we'll be restarting phabricator database
> master for https://phabricator.wikimedia.org/T239791 and to get it
> upgraded
Hello,
Thursday 13th at 06:00 AM UTC we'll be restarting phabricator database
master for https://phabricator.wikimedia.org/T239791 and to get it upgraded
as well (https://phabricator.wikimedia.org/T244566)
Phabricator will be on read-only mode for around 1 minute.
Sorry for the inconvenience,
Hello,
*Thursday 20th at 09:00AM UTC* we'll be restarting and upgrading db1135
(m2 misc primary database master) as part of
https://phabricator.wikimedia.org/T239791
The specific task for the upgrade is
https://phabricator.wikimedia.org/T244238
Impact: The following services will be on read-only
Hello,
This was done successfully.
Read only starts: 06:00:24
Read only stops: 06:01:09
Total read only time: 45 seconds
On Tue, Nov 26, 2019 at 6:00 AM Manuel Arostegui
wrote:
> Hello,
>
> This will start in 1h.
>
> Thanks.
> Manuel.
>
> On Thu, Nov 14, 2019 at
Hello,
This will start in 1h.
Thanks.
Manuel.
On Thu, Nov 14, 2019 at 8:25 AM Manuel Arostegui
wrote:
> Hello,
> We have requested a 30 minutes read-only window for s7 (T238046) for the
> 26th November from 06:00-06:30 AM UTC to switchover that section primary
> database mas
On Thu, Nov 21, 2019 at 8:47 PM Željko Filipin
wrote:
> Hi,
>
> for HTML version see
> https://www.mediawiki.org/wiki/Scrum_of_scrums/2019-11-20
>
> Željko
>
> --
>
> = 2019-11-20 =
>
> == Callouts ==
> ** SRE:
> *** DBAS [general update] Upcoming s1 (enwiki) database master switch 14
> Nov:
Hello,
This was done successfully.
Read only starts: 06:01:23
Read only stops: 06:02:04
Total read only time: 41 seconds
On Tue, Nov 19, 2019 at 6:00 AM Manuel Arostegui
wrote:
> Hello,
>
> This will start in 1 hour.
>
> Thanks,
> Manuel.
>
> On Tue, Oct 15, 2019 at 1
Hello,
This will start in 1 hour.
Thanks,
Manuel.
On Tue, Oct 15, 2019 at 10:44 AM Manuel Arostegui
wrote:
> Hello,
> We have requested a 30 minutes read-only window for s6 wikis (T235475) for
> the 19th Nov from 05:00-05:30 AM UTC to switchover that section primary
> database mas
Hello,
We have requested a 30 minutes read-only window for s7 (T238046) for the
26th November from 06:00-06:30 AM UTC to switchover that section primary
database master (T238044)
db1062 is an old host and out of warranty that will be decommissioned
(T217396). The new master will be db1086
We are
Hello,
This was done.
Read only start: 06:00:28
Read only stop: 06:01:39
Total read only time: 01:11 minutes
On Thu, Nov 14, 2019 at 6:00 AM Manuel Arostegui
wrote:
> Hello,
>
> This will start in 1 hour.
>
> Thanks,
> Manuel.
>
> On Mon, Oct 7, 2019 at 11:19 AM M
Hello,
This will start in 1 hour.
Thanks,
Manuel.
On Mon, Oct 7, 2019 at 11:19 AM Manuel Arostegui
wrote:
> Hello,
> We have requested a 30 minutes read-only window for s1 (enwiki) (T234801)
> for the 14th November from 05:00-05:30 AM UTC to switchover that section
> primary dat
Hello,
We have requested a 30 minutes read-only window for s6 wikis (T235475) for
the 19th Nov from 05:00-05:30 AM UTC to switchover that section primary
database master (T235469)
db1061 is an old host and out of warranty that will be decommissioned
(T217396). The new master will be db1131.
We
Hello,
This was done
read only start: 05:00:17
read only stop: 05:00:43
Total read only time: 26 seconds
Thanks
Manuel.
On Tue, Oct 15, 2019 at 6:00 AM Manuel Arostegui
wrote:
> Hello,
>
> This will start in 1 hour.
>
> Thanks,
> Manuel.
>
> On Tue, Oct 1, 2019 at
Hello,
This will start in 1 hour.
Thanks,
Manuel.
On Tue, Oct 1, 2019 at 7:03 AM Manuel Arostegui
wrote:
> Hello,
> We have requested a 30 minutes read-only window for s5 wikis (T234303) for
> the 15th October from 05:00-05:30 AM UTC to switchover that section primary
> dat
Hello,
We have requested a 30 minutes read-only window for s1 (enwiki) (T234801)
for the 14th November from 05:00-05:30 AM UTC to switchover that section
primary database master (T234800)
db1067 is an old host and out of warranty that will be decommissioned
(T217396). The new master will be
Hello,
We have requested a 30 minutes read-only window for s5 wikis (T234303) for
the 15th October from 05:00-05:30 AM UTC to switchover that section primary
database master (T234300)
db1070 is an old host and out of warranty that will be decommissioned
(T217396). The new master will be db1100.
Hello,
s4 (commonswiki) was done:
Read-only start: 05:00:51
Read-only stop: 05:01:42
Total read-only time: 51 seconds
Thanks,
Manuel.
On Thu, Sep 26, 2019 at 6:00 AM Manuel Arostegui
wrote:
> Hello,
>
> The last switchover, from this batch, s4 (commonswiki), will start in 1
Hello,
The last switchover, from this batch, s4 (commonswiki), will start in 1h
https://phabricator.wikimedia.org/T230784
Thanks
Manuel.
On Tue, Aug 20, 2019 at 12:49 PM Manuel Arostegui
wrote:
> Hello,
>
> The following primary database masters will be switched over during the
&
On Tue, Sep 24, 2019 at 6:00 AM Manuel Arostegui
wrote:
> Hello,
>
> s3 master switchover will start in 1h:
> https://phabricator.wikimedia.org/T230783
> List of affected wikis:
> https://raw.githubusercontent.com/wikimedia/operations-mediawiki-config/master/dblists/s3.
to HW malfunction: https://phabricator.wikimedia.org/T233534
Thanks,
Manuel.
On Tue, Aug 20, 2019 at 12:49 PM Manuel Arostegui
wrote:
> Hello,
>
> The following primary database masters will be switched over during the
> next few weeks (more details at https://phabricator.wikimedia
On Tue, Sep 17, 2019 at 6:00 AM Manuel Arostegui
wrote:
> Hello,
>
> s2 primary master switchover will start in 1h:
> https://phabricator.wikimedia.org/T230785
> List of affected wikis:
> https://raw.githubusercontent.com/wikimedia/operations-mediawiki-config/master/dblists
Hello,
s2 primary master switchover will start in 1h:
https://phabricator.wikimedia.org/T230785
List of affected wikis:
https://raw.githubusercontent.com/wikimedia/operations-mediawiki-config/master/dblists/s2.dblist
Thanks,
Manuel.
On Tue, Aug 20, 2019 at 12:49 PM Manuel Arostegui
wrote
This was done.
Read-only starts: Tue Sep 10 16:10:39 UTC 2019
Read-only stops: Tue Sep 10 16:10:45 UTC 2019
Total read-only time: 6 seconds
On Tue, Sep 10, 2019 at 5:00 PM Manuel Arostegui
wrote:
> Hello,
>
> This will happen in 1 hour.
>
> Thanks
> Manuel.
>
> On Thu,
Hello,
This will happen in 1 hour.
Thanks
Manuel.
On Thu, Aug 29, 2019 at 1:00 PM Manuel Arostegui
wrote:
> Hello,
>
> The current primary master for m1 (db1063), which is mostly for internal
> services + etherpad isn't in a great healthy status: it is an old host,
&
s8 (wikidata) master was switched over successfully.
read-only start: Tue Sep 10 05:00:47 UTC 2019
read-only stop: Tue Sep 10 05:02:14 UTC 2019
Total read-only time: 1 minute 27 seconds.
Cheers,
Manuel.
On Tue, Sep 10, 2019 at 6:18 AM Manuel Arostegui
wrote:
> Hello,
>
> s8 (wikidat
Hello,
s8 (wikidata) master will be switched over in 40 minutes.
Manuel.
On Tue, Aug 20, 2019 at 12:49 PM Manuel Arostegui
wrote:
> Hello,
>
> The following primary database masters will be switched over during the
> next few weeks (more details at https://phabricator.wikimedia
This was done successfully.
wikitech read only start: 13:08:40
wikitech read only stop: 13:10:01
Total: 1 minute and 21 seconds
Thanks everyone who helped out!
On Tue, Sep 3, 2019 at 2:05 PM Manuel Arostegui
wrote:
> Hello
>
> This will happen in 55 minutes.
> Thanks
>
>
Hello
This will happen in 55 minutes.
Thanks
Manuel.
On Tue, Aug 6, 2019 at 8:30 AM Manuel Arostegui
wrote:
> Hello,
>
> We are going to switchover m5 (wikitech) primary database master on *Tuesday
> 3rd September at 13:00 UTC *https://phabricator.wikimedia.org/T229657
>
so.
Maybe I can put an User-Notice tag on the phabricator task, so more people
would be aware of that brief interruption?
Again, we are expecting 5-10 seconds of read-only time.
Thanks,
Manuel.
> Pine
> ( https://meta.wikimedia.org/wiki/User:Pine )
>
> On Thu, Aug 29, 2019, 06:28 M
Hello,
The current primary master for m1 (db1063), which is mostly for internal
services + etherpad isn't in a great healthy status: it is an old host,
which needs to be decommissioned and which is having disks failing pretty
much every week (plus disks on predictive failure).
We have decided to
Hello,
The following primary database masters will be switched over during the
next few weeks (more details at https://phabricator.wikimedia.org/T230788):
Impact:
*Writes will be blocked*
*Reads will remain unaffected*
These are the concrete days, hours and affected wikis:
* s8: 10th Sept from
Hello,
We are going to switchover m5 (wikitech) primary database master on *Tuesday
3rd September at 13:00 UTC *https://phabricator.wikimedia.org/T229657
This switchover will also affect some Cloud services that will be handled
on a different announcement.
The current master (db1073) is an old
Hello,
The failover was done successfully.
read-only start: 05:00:50
read-only stop: 05:02:21
Total read-only time: 01:31 minutes
Manuel.
On Tue, Jul 30, 2019 at 6:30 AM Manuel Arostegui
wrote:
> Hello,
>
> This will start in 30 minutes.
>
> Manuel.
>
> On Wed, Jul 10, 2
Hello,
This will start in 30 minutes.
Manuel.
On Wed, Jul 10, 2019 at 9:05 AM Manuel Arostegui
wrote:
> Hello,
> We have requested a 30 minutes read-only window for s8 (wikidata) for the
> 30th July from 05:00-05:30 UTC to switchover our primary wikidata database
> master (T227063
On Sat, Jul 27, 2019 at 4:03 PM Bartosz Dziewoński
wrote:
> The 'searchindex' table uses MyISAM because until recently, InnoDB did
> not support fulltext indexes, which MediaWiki uses for the search. All
> other tables should use InnoDB.
>
> According to https://stackoverflow.com/a/9397060
Hello,
This has been done.
Phabricator read only start: 05:30:44
Phabricator read only stop: 05:31:37
Total read-only time: 53 seconds
Thanks Mukunda for being around to handle the phabricator side of things.
Manuel.
On Thu, Jul 25, 2019 at 7:00 AM Manuel Arostegui
wrote:
> He
Hello,
This maintenance will start in 30 minutes.
Thanks
Manuel.
On Wed, Jul 17, 2019 at 10:37 AM Manuel Arostegui
wrote:
> Hello,
>
> On Thursday 25th July at 05:30 AM UTC we will switchover our current
> primary database master for m3 (phabricator) from db10
Hello,
On Thursday 25th July at 05:30 AM UTC we will switchover our current
primary database master for m3 (phabricator) from db1072 to db1128.+
This is the tracking task for this maintenance:
https://phabricator.wikimedia.org/T228243
db1072, the current db master, is old, and out of warranty
Hello,
We have requested a 30 minutes read-only window for s8 (wikidata) for the
30th July from 05:00-05:30 UTC to switchover our primary wikidata database
master (T227063)
db1071 is an old host and out of warranty that will be decommissioned
(T217396).
This will also unblock some migration
>
Hello Strainu,
Sure, will do. For the record, this affects pretty much all the wikis :-)
# cat echo.dblist
%% all.dblist - nonecho.dblist
# cat nonecho.dblist
legalteamwiki
votewiki
zerowiki
Thanks
Manuel.
>
> În mie., 3 iul. 2019 la 09:10, Manuel Arostegui
> a scris:
> >
This was done.
Read only start: 06:00:36 UTC
Read only stop: 06:01:56 UTC
Total read only time: 01:20 min
On Wed, Jul 3, 2019 at 7:00 AM Manuel Arostegui
wrote:
>
> On Mon, Jul 1, 2019 at 3:48 PM Manuel Arostegui
> wrote:
>
>> Hello,
>>
>> We need to switchover
On Mon, Jul 1, 2019 at 3:48 PM Manuel Arostegui
wrote:
> Hello,
>
> We need to switchover x1 primary master from db1069 to db1120
> https://phabricator.wikimedia.org/T226358
>
> db1069 is a very old host that has hardware issues, it is out of warranty
> and needs to be de
Hello,
We need to switchover x1 primary master from db1069 to db1120
https://phabricator.wikimedia.org/T226358
db1069 is a very old host that has hardware issues, it is out of warranty
and needs to be decommissioned.
Given that x1 cannot be set on read-only on a MW level, we will need to go
This email was received today as it was manually run as part of:
https://phabricator.wikimedia.org/T224804
On Wed, Jun 19, 2019 at 10:56 AM wrote:
>
> Hi Community Metrics team,
>
> This is your automatic monthly Phabricator statistics mail.
>
> Accounts created in (2019-05): 272
> Active
Hello,
This happened successfully.
Read only times (UTC):
Start: 05:01:02
Stop: 05:03:20
Total read only time: 2:18 minutes
On Wed, Jun 19, 2019 at 6:30 AM Manuel Arostegui
wrote:
> Hello,
>
> This will start in 30 minutes.
>
> Manuel.
>
> On Thu, Jun 6, 2019 at 2:4
Hello,
This will start in 30 minutes.
Manuel.
On Thu, Jun 6, 2019 at 2:44 PM Manuel Arostegui
wrote:
> Hello,
>
> We have requested a 30 minutes read-only window for s4 (commonswiki) for
> the 19th June from 05:00-05:30 UTC to switchover our primary commonswiki
> database
Hello,
We have requested a 30 minutes read-only window for s4 (commonswiki) for
the 19th June from 05:00-05:30 UTC to switchover our primary commonswiki
database master - https://phabricator.wikimedia.org/T224516
db1068 is an old host that will be decommissioned (T217396) and which also
suffers
On Wed, May 22, 2019 at 12:12 PM Mukunda Modell
wrote:
> If you do the edits as a bulk edit job I can flag it to run silently which
> will spare people the email spam.
>
I would love to have this done :-)
I already have lots of phab traffic, so keeping this one to a minimum would
be nice!
Hello,
This was done successfully.
Times in UTC:
Read only starts: 05:01:05
Read only stops: 05:03:55
Total read only time: 02:50 minutes
Cheers
On Thu, Apr 11, 2019 at 6:31 AM Manuel Arostegui
wrote:
> Hello,
>
> Reminder: this is happening in 30 minutes.
>
> Manuel.
&g
Hello,
Reminder: this is happening in 30 minutes.
Manuel.
On Fri, Apr 5, 2019 at 8:57 AM Manuel Arostegui
wrote:
> Correction, this is the Thursday 11th of April not the 10th.
> Sorry for the confusion!
>
> On Fri, Apr 5, 2019 at 8:43 AM RhinosF1 Wikipedia
> wro
so it can be translated so I've added it with
>
> https://meta.wikimedia.org/w/index.php?title=Tech/News/2019/15=18991805=18990102
>
> On Fri, 5 Apr 2019 at 07:23, Manuel Arostegui
> wrote:
>
> > Hello,
> >
> > Due to hardware malfunction on the current s3
Hello,
Due to hardware malfunction on the current s3 database primary master
(T219115) we need to failover it to a different host.
We are going to do this on Thursday 10th at 05:00 AM UTC, and we have
requested a 30 minutes window (T220080) as we have to go read only for all
the wikis living on
This was done.
Read only ON at: 07:01:00
Read only OFF at: 07:04:20
Total time read only time: 03:20 minutes
If you see something strange, please let us know.
Thanks
On Thu, Jan 17, 2019 at 7:40 AM Manuel Arostegui
wrote:
> This will start in 20 minutes.
>
> Thanks.
>
> On We
This will start in 20 minutes.
Thanks.
On Wed, Jan 16, 2019 at 2:02 PM Manuel Arostegui
wrote:
> Hello,
>
> Due to the recent issues with the PDUs on both A2 (T213748) and A3
> (T213859) we need to swap s3 db primary master (T213858), as it lives on A2.
>
> We are going to d
Hello,
Due to the recent issues with the PDUs on both A2 (T213748) and A3
(T213859) we need to swap s3 db primary master (T213858), as it lives on A2.
We are going to do this on Thursday 17th at 07:00 AM UTC, and we have
requested a 30 minutes window (T213864) as we have to go read only for all
On Sat, Oct 13, 2018 at 4:38 AM Pine W wrote:
> Forwarding news of data loss. :( Hopefully it will all be recoverable.
>
>
Hello,
We believe no data has been lost - the data inserted during the incident is
still at our codfw DC (the active DC at the time), as there were no issues
on that
gt; ( https://meta.wikimedia.org/wiki/User:Pine )
>
>
> On Thu, Sep 6, 2018 at 9:22 AM Manuel Arostegui
> wrote:
>
> > On Wed, Sep 5, 2018 at 8:49 PM, Pine W wrote:
> >
> > > Thanks as usual for these, Grace.
> > >
> > > I feel good if I can
On Wed, Sep 5, 2018 at 8:49 PM, Pine W wrote:
> Thanks as usual for these, Grace.
>
> I feel good if I can vaguely understand 50% of what's in these notes.
>
> I have an elementary question, and I hope that I'm not testing anyone's
> patience with this question. Do the switchover/switchback
Hello,
We suffered an unexpected issue with the phabricator database replica,
which I believe it is used to generate these stats.
We will hopefully have it back up today and we can re-run the script.
Sorry for the inconveniences,
Manuel.
On Wed, Aug 1, 2018 at 3:01 AM, יגאל חיטרון wrote:
>
On Tue, Jan 9, 2018 at 6:50 AM, Manuel Arostegui <maroste...@wikimedia.org>
wrote:
> Hello,
>
> We are starting in 10 minutes
>
> On Wed, Jan 3, 2018 at 10:36 AM, Manuel Arostegui <
> maroste...@wikimedia.org> wrote:
>
>> Hello everyone
>>
>> H
76 matches
Mail list logo
