jcrespo added a project: DBA.
TASK DETAILhttps://phabricator.wikimedia.org/T174044EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Aklapper, daniel, E1presidente, Ramsey-WMF, Jmmuguerza, SandraF_WMF, GoranSMilovanovic, QZanden, Acer
jcrespo removed a project: Blocked-on-schema-change.jcrespo added a comment.
No actionables here (yet), add #blocked-on-schema-change when it actually is blocked on that.TASK DETAILhttps://phabricator.wikimedia.org/T174044EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
jcrespo added a comment.
@Anomie is right, adding it now would only spam us and deplay actually blocked changes. Add the DBA to tell us we will be involved in the future, as a heads up, which I already did (the Blocked external is the meaning here, not the not db team, we *will* be involved, just
jcrespo reassigned this task from jcrespo to daniel.
TASK DETAILhttps://phabricator.wikimedia.org/T173269EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, jcrespoCc: WMDE-leszek, jcrespo, Peachey88, Marostegui, Krenair, Addshore, TerraCodes, Jay8g
jcrespo closed this task as "Resolved".jcrespo claimed this task.jcrespo added a comment.
Resolving for now.TASK DETAILhttps://phabricator.wikimedia.org/T173269EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: WMDE-leszek, jcrespo,
jcrespo added a project: CirrusSearch.Herald added projects: Discovery, Discovery-Search.
TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi
jcrespo added a comment.
So this is deployed into production, we did a test run and it seems to work as intended.
I left a "disable" patch https://gerrit.wikimedia.org/r/373507 and instructions to deploy there, in case fellow ops have to do some emergency thing to disable, so it is alr
jcrespo added a comment.
@Ladsgroup: This should be easy to fix:
root@terbium:/var/log/wikidata$ ls -lha rebuildTermSql*
-rw-rw-r-- 1 www-data www-data 1.7K Aug 25 06:30 rebuildTermSqlIndex.log
-rw-r--r-- 1 www-data www-data 130 Aug 8 13:15 rebuildTermSqlIndex.log-20170810.gz
-rw-rw-r-- 1 www
jcrespo added a comment.
This is probably a symptom and not a cause, but I wanted to comment it anyway in case it was interesting:
There seems to be higher than usual hhvm exceptions:
https://logstash.wikimedia.org/goto/80fa5708f0a5e9da4be9f4630969b72e
Most of those, at least the ones
jcrespo raised the priority of this task from "High" to "Unbreak Now!".jcrespo claimed this task.jcrespo added a comment.
The introduction of wikidata events on recentchanges has converted the "light" recentchanges table into a monolithical 500GB table:
common
jcrespo added a comment.
We can do it on commons and ruwiki, which are probably the most affected ones, plus some on s3.TASK DETAILhttps://phabricator.wikimedia.org/T171027EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: D3r1ck01, matej_suchanek
jcrespo removed a project: User-notice.
TASK DETAILhttps://phabricator.wikimedia.org/T171027EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Finavon, WMDE-leszek, saper, Masti, Lydia_Pintscher, D3r1ck01, matej_suchanek, Ankry, Ladsgroup, Lsanabria
jcrespo added a comment.
BTW, the decision was already mentioned as the 4th option of @Catrope suggestions "Disable Wikidata RC on large wikis until we have a more scalable implementation of the feature". I think nobody predicted how bad things were at the time.TASK D
jcrespo reopened subtask T171027: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis as "Open".
TASK DETAILhttps://phabricator.wikimedia.org/T90435EMAIL PREFERENCEShttps://phabricator.wikimedi
jcrespo reopened this task as "Open".
TASK DETAILhttps://phabricator.wikimedia.org/T171027EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Finavon, WMDE-leszek, saper, Masti, Lydia_Pintscher, D3r1ck01, matej_suchanek, Ankry, Ladsgroup,
jcrespo closed this task as "Resolved".jcrespo added a project: User-notice.jcrespo added a comment.
Notification for users: We are going to disable wikidata recentchanges (meaning, changes on pages on other wikis coming from changes done on wikidata; the recentchanges at wikidata is not
jcrespo closed subtask T171027: "2062 Read timeout is reached" DBQueryError when trying to load specific users' watchlists (with +1000 articles) on several wikis as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T90435EMAIL PREFERENCEShttps://phabricator.wikimedi
jcrespo added a comment.
This is the rate on the last 7 days of Apiqueryrecentchanges:
https://logstash.wikimedia.org/goto/7a5cebd94f0120ead4ca10a34f6b2b54
Coincidentally, it is much larger on ruwiki and commons, even if ruwiki was added as backend a borrowed large server.TASK DETAILhttps
jcrespo added a comment.
I think the materialization is a wrong approach, and we should try to materialize the changes only on query. E.g.:
Watchlist x page x wikidata_usage JOINS being queried on real time. Of course, that makes the query an order of magnitude slower and more complex but- some
jcrespo added a comment.
This is the direct cause of T180564.TASK DETAILhttps://phabricator.wikimedia.org/T178661EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: chasemp, jcrespoCc: bd808, chasemp, Magnus, hoo, daniel, jcrespo, Marostegui, Aklapper, Ladsgroup
jcrespo edited projects, added Wikidata; removed Toolforge.
TASK DETAILhttps://phabricator.wikimedia.org/T181486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Ladsgroup, jcrespo, Aklapper, Magnus, Lahi, Gq86, GoranSMilovanovic, QZanden, Wikidata
jcrespo added a comment.
@Magnus- you probably didn't see Ladsgroup comment, apparently it requires editing, and not purging for the table to repopulate.TASK DETAILhttps://phabricator.wikimedia.org/T181486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo added a comment.
@Aklapper Probably, but I would close that one, as that should not be happening right now, unless you have reports saying it is again.TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo removed jcrespo as the assignee of this task.jcrespo added a comment.
This need hardware provisioning, and that means budget, and that means a detailed plan with our overall ok.TASK DETAILhttps://phabricator.wikimedia.org/T176277EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings
jcrespo added a comment.
In the longer term we should make some changes to maintain-views so that it can drop a single view without needing to rebuild all of the views for the database.
Also, given that the "new" servers can be depooled almost at will, metadata locking can be avoided,
jcrespo closed this task as "Declined".jcrespo added a comment.
We are happy with the configuration on both eqiad and codfw, we do not need to test testwiki- we did it with wikidatawiki easily.TASK DETAILhttps://phabricator.wikimedia.org/T180694EMAIL PREFERENCEShttps://phabricator.wik
jcrespo added a comment.
Thanks, Ladsgroup, for starters we were thinking of preparing the topology changes for s8 on codfw (which if it breaks it wouldn't be a huge deal because it is passive right now). Here is the change:
https://gerrit.wikimedia.org/r/391835
Moving testwikidatawiki would
jcrespo renamed this task from "Test testwikidatawiki on s8" to "Test moving testwikidatawiki database to s8 replica set on Wikimedia".jcrespo updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTIONIn order to test that moving wikidatawiki on production to s8 w
jcrespo created this task.jcrespo added projects: Operations, Wikidata, MediaWiki-Configuration, DBA.
TASK DESCRIPTIONIn order to test that moving wikidatawiki on production to s8 will not break things, we have thought of moving one non-user impacting wiki first to test everything is ok. Testwiki
jcrespo added a comment.
@Ladsgroup After looking at database logs, I can confirm there is something really bad with the wikidata new item logic- each request has to lock the table, which esentially serialized creations, puts a limit on the number of items that can be created per second (in a bad
jcrespo added a comment.
@Ladsgroup Without looking, I imagine a bot overloaded the creation, something that should be easy to check on recentchanges. Of course, that doesn't take away from avoiding or reducing the lock as much as possible.TASK DETAILhttps://phabricator.wikimedia.org/T183341EMAIL
jcrespo added a comment.
@Billinghurst I was actually praising you for the report, for taking the time, and saying sorry it happened :-)TASK DETAILhttps://phabricator.wikimedia.org/T183341EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo
jcrespo added a comment.
Yes, the slowdown should avoid issues until a fix is conceived.TASK DETAILhttps://phabricator.wikimedia.org/T183341EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Lydia_Pintscher, Daniel_Mietchen, mark, jcrespo, Marostegui
jcrespo added a comment.
The LIMIT 1 FOR UPDATE (plus what marostegui comments) indicates that is not a lag problem, but a contention problem (Error: 1205 Lock wait timeout exceeded)- many items wanting to lock the same rows at the same time. There is nothing for the DBAs to do here, code should
jcrespo edited projects, added MediaWiki-Database; removed DBA.jcrespo added a comment.
@Billinghurst In non-technical terms, it seems to be an overload of some kind- something that normally doesn't happen, but that the code should avoid from happening in any case (e.g. it could be an unrelated
jcrespo added a comment.
Full table scans are indeed expected and preferred when tables have 0 to a only a few rows. However, I think Bstorm is right to be concerned, we have seen lately an increase in load on labsdbs, but I have yet to check if it is related to the usage of views or just
jcrespo added a comment.
Yes, that is much better.TASK DETAILhttps://phabricator.wikimedia.org/T191391EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: Aklapper, Lucas_Werkmeister_WMDE, Jonas, jcrespo, Ladsgroup, Lahi, Gq86
jcrespo added a comment.
This would have been very useful in this case https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.htmlTASK DETAILhttps://phabricator.wikimedia.org/T194270EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Marostegui
jcrespo added a comment.
Most if not all of these seem to have gone since yesterday's train (need more time to check if absolutly all are gone or just the bulk of it).TASK DETAILhttps://phabricator.wikimedia.org/T191282EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
jcrespo added a comment.
The indexes seem disproportionally large compared to the data. Could the table be split somehow in a 1:1 relationship that could make sense (e.g. index_table with just columns to search and data_table with almost just the primary index).TASK DETAILhttps
jcrespo lowered the priority of this task from "High" to "Normal".jcrespo added a comment.
@mmodell Most of the errors are gone, but some are still happening. I think this is no longer high, but if it has an easy fix, those should be looked at by the code owners:
https://lo
jcrespo raised the priority of this task from "Normal" to "High".
TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Marostegui, jcrespoCc: Stashbot, gerritbot, Marostegui, Aklapper, Lu
jcrespo added a comment.
TermSqlIndex::getMatchingTerms is still failing, it is right now the top failing query among all mediawiki databases:
Wikibase\Lib\Store\Sql\TermSqlIndex::getMatchingTerms 10.64.16.85 2062 Read timeout is reached (10.64.16.85) SELECT term_entity_type,term_type
jcrespo reassigned this task from jcrespo to Addshore.Herald added a project: User-Addshore.
TASK DETAILhttps://phabricator.wikimedia.org/T196172EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, jcrespoCc: ops-monitoring-bot, Marostegui, jcrespo
jcrespo moved this task from Triage to Done on the DBA board.jcrespo added a comment.
I don't see huge issues on the current master- I would solve this as resolved and at some point thing about the epic parent (T108944) to improve the infrastructure.TASK DETAILhttps://phabricator.wikimedia.org
jcrespo added a comment.
I thought that was the query that broke things, am I wrong?TASK DETAILhttps://phabricator.wikimedia.org/T194273EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: Lydia_Pintscher, Mholloway, Stashbot, gerritbot
jcrespo added a comment.
Don't repool it unless it's back to the old status.
So, to clarify, we can remove all data here and eliminate the extra accounts?TASK DETAILhttps://phabricator.wikimedia.org/T191391EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo added a comment.
Still ongoing:
Wikibase\Lib\Store\Sql\TermSqlIndex::getMatchingTerms 10.64.32.113 2062 Read timeout is reached (10.64.32.113) SELECT term_entity_type,term_type,term_language,term_text,term_weight,term_full_entity_id FROM `wb_terms`WHERE ((term_language = 'en
jcrespo closed subtask T196047: reimage db2083 back into wikidata (s8) as "Resolved".
TASK DETAILhttps://phabricator.wikimedia.org/T191391EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, jcrespoCc: Marostegui, Aklapper, Lucas_Werkmeister_W
jcrespo added a comment.
Clarification, Tomorrow 7th, not 6th.TASK DETAILhttps://phabricator.wikimedia.org/T196172EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Anomie, jcrespoCc: Marostegui, jcrespo, Aklapper, aude, Addshore, Anomie, Jdforrester-WMF
jcrespo added a comment.
The plan is to reimage those servers into stretch tomorrow (thursday 6th June) and reload its data from production. I can also deploy grants for performance_schema/sys T195578 there so you have more data for profiling.
During reimage/upgrade/load, access in read-write
jcrespo added a comment.
The plan is to reimage those servers into stretch tomorrow (thursday 7th June) and reload its data from production. I can also deploy grants for performance_schema/sys T195578 there so you have more data for profiling.
During reimage/upgrade/load, access in read-write
jcrespo claimed this task.
TASK DETAILhttps://phabricator.wikimedia.org/T196172EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Marostegui, jcrespo, Aklapper, aude, Addshore, Anomie, Jdforrester-WMF, gerritbot, Abit, daniel, Lahi, PDrouin-WMF, Gq86
jcrespo added a parent task: Restricted Task.
TASK DETAILhttps://phabricator.wikimedia.org/T198049EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Aklapper, hoo, Lydia_Pintscher, daniel, Ladsgroup, Marostegui, AndyTan, Davinaclare77, Qtn1293, Lahi
jcrespo added a comment.
51,715 exceptions with:
[{exception_id}] {exception_url} Wikimedia\Rdbms\DBReplicationWaitError from line 426 of /srv/mediawiki/php-1.32.0-wmf.8/includes/libs/rdbms/lbfactory/LBFactory.php: Could not wait for replica DBs to catch up to db1071TASK DETAILhttps
jcrespo added a comment.
Edits halved, which is consistent with a wikidata outage:
F22621659: Screenshot_20180625_081919.png
No long running queries on master or db1099:3318:
F22621676: Screenshot_20180625_081013.png
F22621675: Screenshot_20180625_080008.png
Revisions were 10 times less than
jcrespo added a comment.
Let me ask in another way- which percentage, even with a lot of error, of work dispatch creates on s8 (wikidata), if we include jobqueue, webrequests and other traffic.
I would put at least 2 servers for redundancy reasons.TASK DETAILhttps://phabricator.wikimedia.org
jcrespo added a project: Wikidata.jcrespo added a comment.
Most of the warnings (I suppose the logging was changed to understand which component relates to) seem to be related to wikibase client jobs right now:
https://logstash.wikimedia.org/app/kibana#/dashboard/DBQueryTASK DETAILhttps
jcrespo added a comment.
@Ladsgroup after the table rebuilding on dbstore2001, purging is going at a good pace:
https://grafana.wikimedia.org/dashboard/db/mysql?panelId=11=1=codfw%20prometheus%2Fops=dbstore2001=13318=1524683251842=1524726451842
As soon as the counter gets close to 0, the row
jcrespo renamed this task from "Drop replication of wb_entity_per_page in labs" to "Drop wb_entity_per_page views in labs".jcrespo updated the task description. (Show Details)
CHANGES TO TASK DESCRIPTIONWe've got https://gerrit.wikimedia.org/r/382694 merged, and DBAs will t
jcrespo added a comment.
Further comments/issues should go to the parent task T177208TASK DETAILhttps://phabricator.wikimedia.org/T181645EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lea_Lacroix_WMDE, jcrespoCc: Joe, Rschen7754, Izno, Ptolusque, revi
jcrespo added a comment.
Based on the comments, I highly suspect this is related to T180918TASK DETAILhttps://phabricator.wikimedia.org/T183101EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Imarlier, jcrespoCc: jcrespo, Legoktm, Imarlier, aaron, Aklapper
jcrespo added a comment.
Anything that would edit the database would either be disabled or generate an error- preferences and watchlist edition is handled on the database, so it would not be available. It is important to note that while we require a 30 minute window, we expect, if things go well
jcrespo assigned this task to bd808.jcrespo added a comment.
I think cloud stuff is ready -although there is a bit of lag at the moment on wikidatawiki rightnow 12K seconds-, only things pending would be cloud specific setup such as dns changes, script changes and https://tools.wmflabs.org/replag
jcrespo added a comment.
@Izno in the event of an error happening, because the retry that Ladsgroup says also fails, and it is not fixed by us later, a purge or null edit on the origin wiki should fix any inconsistency. Sadly, this is theoretically impossible to fix unless we block as read only
jcrespo added a comment.
BTW, this is starting in less than 20 minutes.TASK DETAILhttps://phabricator.wikimedia.org/T181645EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lea_Lacroix_WMDE, jcrespoCc: Rschen7754, Izno, Ptolusque, revi, Silraks, Steinsplitter
jcrespo added a comment.
I think there is a different core returned between killed (or other error)-although I think in this case there is an exception, timed out and acquired, but I may be wrong. Note that on that ticket, I mention waits for slave can start being killed from now on to avoid
jcrespo added a comment.
This is maybe a defect on the deployment of the badges extension/Wikidata or a breakage on apache redirects. However, I have open other languages, that show the image on the browser, and all generate a 404- the image comes from a base64 encoding, not from the static image
jcrespo added a parent task: T42810: Wikibase badges (tracking).
TASK DETAILhttps://phabricator.wikimedia.org/T186815EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Superyetkin, Aklapper, Davinaclare77, Qtn1293, Lahi, Gq86
jcrespo added a subtask: T186815: Badges not displaying on trwiki.
TASK DETAILhttps://phabricator.wikimedia.org/T42810EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Taketa, DixonD, Aklapper, Liuxinyu970226, Bugreporter, Wikidata-bugs, Lazowik
jcrespo added a comment.
@Superyetkin As per Ladsgroup, it seems the CSS you added here is not correct https://tr.wikipedia.org/w/index.php?title=MediaWiki%3AVector.css=revision=18969658=18388018 Please double check it.
@Ladsgroup Apparently, that reference is not the only one, I think
jcrespo added a comment.
curl 'https://en.wikipedia.org/w/load.php?debug=false=en=ext.cite.styles%7Cext.echo.badgeicons%7Cext.echo.styles.badge%7Cext.uls.interlanguage%7Cext.visualEditor.desktopArticleTarget.noscript%7Cext.wikimediaBadges%7Cmediawiki.legacy.commonPrint%2Cshared
jcrespo added a comment.
okTASK DETAILhttps://phabricator.wikimedia.org/T186815EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Superyetkin, jcrespoCc: Ladsgroup, Lydia_Pintscher, jcrespo, Superyetkin, Aklapper, Davinaclare77, Qtn1293, Lahi, Gq86
jcrespo added a comment.
Sqoops the wbc_entity_usage tables
Also, which db server is being used? analytics-replica/analytics store/dbstore1002 is ok to do that, others are not.TASK DETAILhttps://phabricator.wikimedia.org/T187521EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
jcrespo added a comment.
@Magnus I agree with you it is a bug, and I have updated the tags and the title to reflect that, it is now on the backlog for the #Wikidata team.TASK DETAILhttps://phabricator.wikimedia.org/T181486EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
jcrespo added a project: User-notice.jcrespo added a comment.
Maybe it is too late now? This is happening in exactly 1 week.TASK DETAILhttps://phabricator.wikimedia.org/T181645EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lea_Lacroix_WMDE, jcrespoCc: Elitre
jcrespo added a comment.
This seem to be:
ChangeTags::tagUsageStatistics 10.64.16.76 2062 Read timeout is reached (10.64.16.76) SELECT ct_tag,count(*) AS `hitcount` FROM `change_tag` GROUP BY ct_tag ORDER BY hitcount DESC
That query is taking over 60 seconds, which is a limit imposed
jcrespo removed a project: User-notice.jcrespo added a comment.
That seems right, according to https://meta.wikimedia.org/wiki/Tech/News/For_contributors#What_is_typically_not_included : "Changes only relevant for one language or one wiki"TASK DETAILhttps://phabricator.wikimedia.org/T1
jcrespo added a project: User-notice.jcrespo added a comment.
Maybe it is relevant, as the context why read only is needed is "to provide dedicated database resources to Wikidata due to its growth", which is kind of a big deal?TASK DETAILhttps://phabricator.wikimedia.org/T1
jcrespo added a comment.
As a note, on some parts of the world, in particular, SFO, it will be still the 8th.TASK DETAILhttps://phabricator.wikimedia.org/T181645EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lea_Lacroix_WMDE, jcrespoCc: JStrodt_WMDE
jcrespo added a comment.
this will affect the ability to edit the language links on other wikis, yes?
It will indeed affect that, as that is now entirely handled on Wikidata.TASK DETAILhttps://phabricator.wikimedia.org/T181645EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel
jcrespo added a project: Data-Services.jcrespo added subscribers: Andrew, bd808.jcrespo added a comment.
@bd808 @Andrew This contains the self-actionable part of T181643 (mostly dns-related changes, I assume, for wikireplicas), although probably that was probably already know it was pending
jcrespo created this task.jcrespo triaged this task as "Normal" priority.jcrespo added projects: Operations, MediaWiki-Maintenance-scripts, Wikidata.Herald added a subscriber: Aklapper.
TASK DESCRIPTIONBy grepping s7:
modules/mediawiki/manifests/maintenance/refreshlinks.pp:
jcrespo added subscribers: Joe, akosiaris, Dzahn, ArielGlenn.jcrespo added a comment.
^see if that patch makes senseTASK DETAILhttps://phabricator.wikimedia.org/T184179EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: ArielGlenn, Dzahn, akosiaris
jcrespo removed a project: Security.jcrespo added a comment.
ok, sorry, for a second I thought you were executing that- on that code is guaranteed to be sanitized.TASK DETAILhttps://phabricator.wikimedia.org/T184812EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo reopened this task as "Open".jcrespo raised the priority of this task from "Normal" to "Unbreak Now!".Herald added a subscriber: TerraCodes.
TASK DETAILhttps://phabricator.wikimedia.org/T184812EMAIL PREFERENCEShttps://phabricator.wikimedia.org/s
jcrespo reopened subtask T184812: Enable constraint result caching on Wikidata as "Open".
TASK DETAILhttps://phabricator.wikimedia.org/T181060EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDE, jcrespoCc: gerritbot, Aklap
jcrespo added a comment.
F14034462: Screenshot_20180226_185900.pngTASK DETAILhttps://phabricator.wikimedia.org/T184812EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lucas_Werkmeister_WMDE, jcrespoCc: greg, Ladsgroup, jcrespo, Marostegui, TerraCodes, Stashbot
jcrespo added a comment.
https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m=10=1=db1109=eth0=1519578882341=1519665282341TASK DETAILhttps://phabricator.wikimedia.org/T184812EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo added a project: Security.jcrespo added a comment.
SELECT $startOpts $vars $from $useIndex $ignoreIndex " .
"WHERE $conds $preLimitTail";
That looks like a recipe for an sql injection, how did this pass security review?TASK DETAILhttps://phabricator.wikimedia.o
jcrespo added a comment.
@Ladsgroup check the other existing groups, but unless it has a huge overlap with on of them, we should agree on a new one e.g. "dispatch". For that (or even if we reuse an existing one) we should add more servers to that group, assuming it is considerable lo
jcrespo added a comment.
I am not too worried about exceptions/error messages, I only pointed those in case it helped debug the real issues, the ones I mentioned at T198049#4310282 (in first appearance caused by a single production host having issues).TASK DETAILhttps://phabricator.wikimedia.org
jcrespo added a comment.
@tstarling Please stop writes going to *s2* unless they have already finished, there are plans to put s2 in read only without much pre-warning (unscheduled read only) T201694- I don't know exactly what is going to be the plan (a logical failover or a network maintenance
jcrespo added a comment.
I still have no idea how this could happen (also these errors don't seem to be on mwlog1001?!).
Does dumpJson.php call any function that could force any check on all replicas to fail (and because it has old configuration, it fails on a depooled non-slow host?TASK
jcrespo added a comment.
Related: T135851 T164488#3530600 T162593#3175313 T162807#3882309 T163190#3246365 T160509#3110389TASK DETAILhttps://phabricator.wikimedia.org/T202032EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: tstarling, jcrespoCc: Aklapper, daniel
jcrespo added a comment.
In T183488#4506824, @daniel wrote:
We did pretty extensive testing of this on the test copies (db and db1112). We could have a script that does basic sanity checks, but I'm not sure what to test, beyond the trivial. All I can think of is:
all revision rows have
jcrespo added a comment.
So the script will make sure to not miss any content, but doesn't assure it introduces extra one, or duplicates some, that is why I mention the usage of extra checks. Takes very little time (1 long running query per check) and validates the whole process. The same thing
jcrespo added a comment.
I mentioned this to Tim, and he said to better mention it to Brad (and Daniel?), that it would be nice to run a read-only process to check the consistency of the migration- it should take much less time than the writes and would avoid headaches given the issues
jcrespo added a comment.
Let's keep an eye on performance metrics, specially https://grafana.wikimedia.org/dashboard/db/save-timing Although with the estimations given T183488#4480579, it may be desirable to make things faster rather and short than extended for a long period of time.
Meanwhile, I
jcrespo added a comment.
does not try to refresh it periodically
But db1101 never was a dump host, but rc, and it used to have load 1.TASK DETAILhttps://phabricator.wikimedia.org/T147169EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc
301 - 400 of 581 matches
Mail list logo
