Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-12-12T08:37:00Z] Remove old backup directory from db1116 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo,
Marostegui added a comment.
If there is no space issues on that host, there is no harm in leaving it a bit longer I would say.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, MarosteguiCc:
Banyek added a comment.
on db1116 we still have the backup dir:
drwxr-xr-x 8 mysql mysql 8.0K Oct 11 13:21 sqldata.s8_BACKUP_T206743
I guess we shall it remove now, as this ticket is resolved (and we still remember the issue) ?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
jcrespo added a comment.
@Pigsonthewing I hope my comment at Wikidata Village Pump was helpful- if you think that is ok, I would suggest closing this task, and open a different one to track the merges of old history (this was to track the recovery from backups)?TASK
Addshore added a comment.
In T206743#4691328, @jcrespo wrote:
@Addshore I thought you had communicated to wikidata users about that? Apparently not, or @Pigsonthewing didn't see it, could you link your messages to him?
I'll leave that to @Lea_Lacroix_WMDE. This is still on the needs
jcrespo added a comment.
@Addshore I thought you had communicated to wikidata users about that? Apparently not, or @Pigsonthewing didn't see it, could you link your messages to him?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Marostegui added a comment.
I guess it is because what @Addshore described at T206743#4662054 ?TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespo, MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing,
Marostegui added a comment.
Final round of tables that have been checked and are clean:
abuse_filter_action
abuse_filter_history
betafeatures_user_counts
content
content_models
global_block_whitelist
ipblocks
math
mathoid
ores_model
protected_titles
querycache_info
revision_comment_temp
Marostegui added a comment.
The following tables are empty so no need to check:
actor
config
filearchive
filejournal
globalblocks
hidden
image
image_comment_temp
interwiki
ipblocks_restrictions
job
l10n_cache
oldimage
revision_actor_temp
searchindex
securepoll_cookie_match
securepoll_options
Marostegui added a comment.
wb_terms is finished and it is all clean.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek, Pigsonthewing, Nikerabbit, gerritbot, WMDE-leszek,
Marostegui added a comment.
More tables to check after the ones checked at: T206743#4685983
abuse_filter_log (T206743#4685983) was fixed by Jaime (I have rechecked again and it is indeed now consistent).
The following tables are clean between db1087 and db1092:
abuse_filter
category
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-23T06:39:09Z] Stop replication on db1092 and db1087 for checking T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
Marostegui added a comment.
These tables are fine:
logging
revision
slots
text
Starting to check:
abuse_filter
category
change_tag
change_tag_def
cu_changes
externallinks
geo_tags
imagelinks
ip_changes
iwlinks
linter
log_search
ores_classification
page
page_props
page_restrictions
Addshore added a comment.
In T206743#4685334, @jcrespo wrote:
@Addshore did you sent to wikidata users the list you compiled to check bot activity?
YesTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
Marostegui added a comment.
The following tables have been re-checked on db1092 (recloned from codfw) vs db1087 (manually fixed):
abuse_filter_log
archive
babel
bv2013_edits
bv2015_edits
bv2017_edits
categorylinks
comment
cu_log
ipblocks
langlinks
module_deps
object_cache
querycache
querycachetwo
jcrespo added a comment.
After many fixes during the weekend, wb_terms also fixed on labs, all hosts should be consistent now, doing some extra checks to verify everything is good.
@Addshore did you sent to wikidata users the list you compiled to check bot activity?TASK
Marostegui added a comment.
Update: pagelinks is now being re-imported on labs (this table is around 136G) so it will take a whileTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn,
Marostegui added a comment.
Update from Jaime 18th Oct 16:05: s8 core hosts all finished getting fixed (pending labs)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: ArielGlenn, Banyek,
Marostegui added a comment.
In T206743#4676466, @ArielGlenn wrote:
In T206743#4675755, @Banyek wrote:
on db1124 with instance s8 we have a repliation error as
Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032;
ArielGlenn added a comment.
In T206743#4675755, @Banyek wrote:
on db1124 with instance s8 we have a repliation error as
Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the
Marostegui added a comment.
Update from 17th at 19:04 from Jaime:
all tables except wb_terms, which is half done, should be equal on the s8 masterTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc:
Banyek added a comment.
on db1124 with instance s8 we have a repliation error as
Last_Error: Could not execute Delete_rows_v1 event on table wikidatawiki.pagelinks; Can't find record in 'pagelinks', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log db1087-bin.003073,
Marostegui added a comment.
Update from Jaime at 12:10 UTC
All tables on s8 master fixed except pagelinks and wb_termsTASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MarosteguiCc: Pigsonthewing, Nikerabbit,
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-17T06:52:01Z] fixing s8 master drifts T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc:
Marostegui added a comment.
Update from yesterday at around 14:00UTC
@jcrespo has done an amazing job of manually checking and fixing most of the tables on db1087 (which is the labs master, and it is not as easy to reclone as the others). He's gone thru all the tables to check and fix them.
Right
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-16T06:05:03Z] stopping db1092 and db1087 in sync T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc:
Addshore added a comment.
I have prepared https://www.wikidata.org/wiki/User:Addshore/2018/10/DC_Switch_Issue which includes the pages that will need to be reviewed.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Marostegui added a comment.
So as Jaime said at T206743#4666169 all the pooled replicas in eqiad have now the content from codfw, so that is now consistent.
What is pending is db1087 (depooled) which is the master for db1124 (labsdb master) and labsdb, and of course the master, db1071 which
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T13:16:48Z] stopping db1092 and db1087 in sync T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc:
jcrespo added a comment.
I'll generate a list once all slaves that serve mw traffic are re imaged :)
This is true right now for all pooled read only replicas, but I need to fix the master.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Addshore added a comment.
In T206743#4665661, @jcrespo wrote:
@Addshore The main task now is to check pages and pages from revisions that were lost and then restored, and check edits done since last wednesday on those pages, as, while the data has not been lost, those may have put an edit over
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T09:46:16Z] Synchronized wmf-config/db-eqiad.php: Depool db1092 for recloning - T206743 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
jcrespo added a comment.
@Addshore The main task now is to check pages and pages from revisions that were lost and then restored, and check edits done since last wednesday on those pages, as, while the data has not been lost, those may have put an edit over an older version as the latest, hiding
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T08:46:38Z] Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 - T206743 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 467285 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1104
https://gerrit.wikimedia.org/r/467285TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Addshore added a comment.
I wrote a little crappy but effective query that I can just run on all of the shards to see if anything else was affected.
select max(rev_id), min(rev_id), count(*), max(rev_id) - min(rev_id) as diff from ( select rev_id, rev_timestamp from revision order by rev_id DESC
jcrespo added a comment.
@Pigsonthewing Please don't get worry- as I said at T206743#4658620 only primary data was recovered, cache tables may show still wrong data until we fully recover everything this week, showing some weird data. Please be patient, but don't touch anything that seems wrong,
gerritbot added a comment.
Change 467285 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1104
https://gerrit.wikimedia.org/r/467285TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T07:24:08Z] Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 48s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T07:12:16Z] Synchronized wmf-config/db-eqiad.php: Depool db1104 - T206743 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 467261 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104
https://gerrit.wikimedia.org/r/467261TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 467261 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1104
https://gerrit.wikimedia.org/r/467261TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 467260 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Fully repool db1109
https://gerrit.wikimedia.org/r/467260TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 467260 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Fully repool db1109
https://gerrit.wikimedia.org/r/467260TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 467252 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1109
https://gerrit.wikimedia.org/r/467252TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 467252 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Slowly repool db1109
https://gerrit.wikimedia.org/r/467252TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-15T05:16:12Z] Stop MySQL on db1109 for recloning - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc:
gerritbot added a comment.
Change 467249 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109
https://gerrit.wikimedia.org/r/467249TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 467249 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1109
https://gerrit.wikimedia.org/r/467249TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Pigsonthewing added a comment.
The item I gave as an example above still shows the wrong image:
https://www.wikidata.org/wiki/Q2058295
It is also missing other data. Note this diff, the most recent edit (as I write):
https://www.wikidata.org/w/index.php?title=Q2058295=762891812=745457545
as it
Addshore added a comment.
Here is a list of all pages that will likely be affected already with data being lost in the revisions that have since been restored:
MariaDB [wikidatawiki]> SELECT DISTINCT rev_page, page_title, page_namespace, page_touched
-> FROM revision AS revision
-> INNER
Marostegui added a comment.
root@neodymium:/home/marostegui# ./section s8 | while read host port; do echo $host; mysql.py -h$host:$port wikidatawiki -BN -e "select page_id, page_title, page_latest from page where page_id = 99480;";done
labsdb1011.eqiad.wmnet
99480 Q97215 745463455
Addshore added a comment.
Using the example that I had in T206743#4661943 the pages that were fixable should now be fixed
MariaDB [wikidatawiki]> select page_id, page_title, page_latest from page where page_id = 99480;
+-++-+
| page_id | page_title | page_latest |
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-12T18:37:06Z] modified attachLatest.php script finished running over 9395 pages T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Addshore added a comment.
In T206743#4662033, @Marostegui wrote:
In T206743#4662023, @Addshore wrote:
I'll run a maint script to fix the page_latest of the 9000ish pages that have an incorrect one so that we don't end up with more inconsistencies / broken stuff moving forward.
But what will
Marostegui added a comment.
In T206743#4662023, @Addshore wrote:
I'll run a maint script to fix the page_latest of the 9000ish pages that have an incorrect one so that we don't end up with more inconsistencies / broken stuff moving forward.
But what will happen with the correct data we have in
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-12T18:25:26Z] running modified attachLatest.php script over ~9000 pages on wikidatawiki (with added wait for slaves) T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Addshore added a comment.
I'll run a maint script to fix the page_latest of the 9000ish pages that have an incorrect one so that we don't end up with more inconsistencies / broken stuff moving forward.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Marostegui added a comment.
In T206743#4661943, @Addshore wrote:
Which means there are ~9433 pages that now probably have the wrong page_latest, for example:
We are fixing that by recloning the hosts (T206743#4658362) from codfw, which have the correct data:
See how db1099 and db1101 already
Marostegui added a comment.
We are also fully restoring the eqiad hosts from codfw which has all the fine data.
The fix done on Friday was a quick fix to get all the data in there.
On eqiad, db1099 and db1101 should already have the proper data. They were recloned from codfwTASK
Addshore added a comment.
It looks like we could fix these with the attachLatest.php maint script but it would need a little bit of modification.TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui,
Addshore added a comment.
Which means there are ~9433 pages that now probably have the wrong page_latest, for example:
MariaDB [wikidatawiki]> select page_id, page_title, page_latest from page where page_id = 99480;
+-++-+
| page_id | page_title | page_latest |
Addshore added a comment.
In T206743#4661704, @Pigsonthewing wrote:
I'm pretty sure I added an image:
https://commons.wikimedia.org/wiki/File:Sandy_Wollaston_(cropped).jpg
to:
https://www.wikidata.org/wiki/Q2058295
that day (though of course its possible I was distracted and didn't complete
Pigsonthewing added a comment.
I'm pretty sure I added an image:
https://commons.wikimedia.org/wiki/File:Sandy_Wollaston_(cropped).jpg
to:
https://www.wikidata.org/wiki/Q2058295
that day (though of course its possible I was distracted and didn't complete the task); it's not there now.TASK
Addshore added a comment.
In T206743#4661592, @Lea_Lacroix_WMDE wrote:
User:Jason.nlw told me that during the effected period, he was in the middle of a batch upload, and some data was not restored yet.
Lea_Lacroix_WMDE added a comment.
User:Jason.nlw told me that during the effected period, he was in the middle of a batch upload, and some data was not restored yet.
https://www.wikidata.org/w/index.php?title=Special:Contributions=20180913095817=500=user=Jason.nlw===2018-09-13=2018-09-13
Can
gerritbot added a comment.
Change 466710 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Repool db1101:3318, db2085:3318, db2083
https://gerrit.wikimedia.org/r/466710TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 466710 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad,db-codfw.php: Repool db1101:3318, db2085:3318, db2083
https://gerrit.wikimedia.org/r/466710TASK
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T15:12:50Z] Stop MySQL on db2085:3318 to reclone db1101:3318 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
gerritbot added a comment.
Change 466658 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318
https://gerrit.wikimedia.org/r/466658TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 466658 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Depool db1101:3318
https://gerrit.wikimedia.org/r/466658TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 466652 merged by jenkins-bot:
[operations/mediawiki-config@master] db-eqiad.php: Repool db1099:3318
https://gerrit.wikimedia.org/r/466652TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
gerritbot added a comment.
Change 466652 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-eqiad.php: Repool db1099:3318
https://gerrit.wikimedia.org/r/466652TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
jcrespo added a comment.
We believe the most important issues (missing revisions, pages and users, and making accessible the content) has been fixed. We will lower the UBN after doing some additional sanity checks. Having said that, for the following days, some missing things like user
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T14:30:44Z] Synchronized wmf-config/db-eqiad.php: T206743: mariadb: Depool db1087 (duration: 00m 49s)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T14:28:17Z] depooling db1087 (T206743)TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, StashbotCc: gerritbot,
jcrespo added a comment.
applied a fix to:
db1109
db1071
db1104
db1101:3318
db1092
dbstore1002TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: gerritbot, WMDE-leszek, jcrespo, mark,
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T13:23:44Z] Stop MySQL on db2083 to reclone db1116:3318 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui,
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T13:20:24Z] Stop MySQL on db1116:3318 to reclone it from db2083 - T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To:
gerritbot added a comment.
Change 466594 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2083
https://gerrit.wikimedia.org/r/466594TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
Marostegui added a comment.
s8 host cloning process
labsdb1011
labsdb1010
labsdb1009
dbstore1002 (not sure we should reclone it, it will be replaced "soon")
db1124 (sanitarium)
db1116 (backup source)
db1109
db1104 (candidate master)
db1101 (recentchanges)
db1099 (recentchanges)
db1092
gerritbot added a comment.
Change 466594 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2083
https://gerrit.wikimedia.org/r/466594TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL
jcrespo added a comment.
We know the exact timestamps of missing rows,
from
db1071-bin.007238:795791989 2018-09-13 09:08:17 on the active (codfw) master: db2045-bin.005879:1036765620
to
db1071-bin.007238:796727644 2018-09-13 09:58:26 on the active (codfw) master: db2045-bin.005880:132937482
We
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T11:10:24Z] Stop MYSQL on db2085:3318 and db1099:3318 T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui,
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-10-11T11:09:49Z] Stop MYSQL on db2088:3318 and db1099:3318 T206743TASK DETAILhttps://phabricator.wikimedia.org/T206743EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui,
85 matches
Mail list logo
