jcrespo added a comment.
@hoo you are right, but check my comment
https://phabricator.wikimedia.org/T101502#1341481 on a very similar issue on
https://phabricator.wikimedia.org/T101502.
We need to validate that change on all queries from the same piece of code (not
only for this particular
jcrespo added a subscriber: jcrespo.
jcrespo added a comment.
First comments from a MySQL DBA, assuming that goes to production:
- Some tables lack of a `PRIMARY KEY`.** This should be a requirement for new
tables**.
- character strings do not have a specified `CHARACTER SET`. Check that those
jcrespo added a blocked task: T17441: Some tables lack unique or primary keys,
may allow confusing duplicate data.
TASK DETAIL
https://phabricator.wikimedia.org/T102992
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, Tamslo
jcrespo added a comment.
@aude If you are increasing the priority based on my comment, don't. Regular
databases can handle the extra load (for now), the main issue is in research
databases or labs, where they contain all dbs, or when there is maintenance and
we are in a degraded state
jcrespo added a comment.
Could this job have caused:
/* JobRunner::commitMasterChanges 127.0.0.1 */
GET_LOCK('jobrunner-serial-commit', 30) AS lockstatus /*
31bdf25cc42f95f4f4a9b493cafe0299 db1024 plwiki 11s */ /* localhost */
we had another large lag spike around the same time than other
jcrespo added a comment.
I am almost sure that the first part of my previous comment was not caused by
this job, but a very large LinksUpdate::incrTableUpdate, sorry about that.
TASK DETAIL
https://phabricator.wikimedia.org/T107319
EMAIL PREFERENCES
https://phabricator.wikimedia.org
jcrespo added a comment.
By @hoo request, I have applied the schema change also to testwikidatawiki.
Please, in the future, for us silly DBAs :-), we need affected wikis very
explicitly.
TASK DETAIL
https://phabricator.wikimedia.org/T99459
EMAIL PREFERENCES
https
jcrespo added a subscriber: jcrespo.
TASK DETAIL
https://phabricator.wikimedia.org/T107602
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Joe, jcrespo
Cc: jcrespo, Legoktm, gerritbot, Smalyshev, BBlack, Joe, daniel, RobLa-WMF,
Aklapper, aude
jcrespo raised the priority of this task from High to Unbreak Now!.
jcrespo added a subscriber: ori.
jcrespo added a comment.
Sorry, reverting back to Unbreak now, because this is breaking almost all wikis
and the problems have been spoted now on production machines, not only on
support
jcrespo lowered the priority of this task from Unbreak Now! to High.
jcrespo added a comment.
This is the lag report of db1028.
F1511025: Screenshot from 2015-08-15 11:30:15.png
https://phabricator.wikimedia.org/F1511025
Seems to have calmed for now.
TASK DETAIL
https
jcrespo added a comment.
And here it is actual data of an example spike:
Host UserSchema Client Source Thread Transaction Runtime
Stamp
db1038wikiusersimplewiki mw1001 - 5096510924
30A18B0D6 15s 2015-08-15 10:04:05
UPDATE
jcrespo added a subscriber: jcrespo.
jcrespo added a comment.
This query, which has to be executed in a serialized way due to replication, is
causing a lag problem on some of the less powerful database slaves (db1047),
and extra load on the rest of the servers. This is specially true for s3
jcrespo moved this task to Backlog on the Database workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T116404
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: aude
jcrespo added a comment.
I do not see this happening on enwiki. Checking on other wikis/hosts.
TASK DETAIL
https://phabricator.wikimedia.org/T116404
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: aude, daniel, hoo, Aklapper, jcrespo
jcrespo moved this task to In progress on the Database workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T116404
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc
jcrespo closed this task as a duplicate of T118372: OAuth broken.
TASK DETAIL
https://phabricator.wikimedia.org/T118363
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Sjoerddebruin, yuvipanda, Magnus, Harmonia_Amanda, Aklapper, Dzahn
jcrespo reopened blocking task T118372: OAuth broken as "Open".
TASK DETAIL
https://phabricator.wikimedia.org/T118363
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Sjoerddebruin, yuvipanda, Magnus, Harmonia_Amanda, Aklap
jcrespo moved this task to Blocked external/Not db team on the Database
workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T118162
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo added a comment.
> the connections should be pooled
We only pool connections at server side- there is not such a thing as client
connection pooling in mediawiki right now.
TASK DETAIL
https://phabricator.wikimedia.org/T118162
EMAIL PREFERENCES
https://phabricator.wikimedia.
jcrespo added a comment.
To update the latest issues identified:
- As this creates one connection per wiki, it ends up opening 1800 connections
to s3 servers (actual measure)
- Most of this connections are idle, not doing actual work, which makes them
unnecesary, while creating the same issue
jcrespo added a comment.
We can go into details of terminology, but reusing connections != pool. There
are many differences, some of which require persistent connections. But the key
factor (specially in this case) is that in a pool of connections there is an
**upper limit** of maximum
jcrespo added a comment.
ok, if it is explained and expected, then it is not urgent.
A higher of updates does not necessarily imply less performance, (it could mean
less lag, for example, if the updates are smaller). However, there could be an
overhead in round trips- I will let you own/decide
jcrespo added a comment.
The trend has not finished:
F2911313: db1072.png <https://phabricator.wikimedia.org/F2911313>
TASK DETAIL
https://phabricator.wikimedia.org/T117398
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Perfo
jcrespo moved this task to Backlog on the Database workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T116404
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: aude
jcrespo added a comment.
The initial issue still happens, although now the query is consistently slow
every time on both servers.
TASK DETAIL
https://phabricator.wikimedia.org/T116404
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc
jcrespo moved this task to In progress on the Database workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T116404
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc
jcrespo raised the priority of this task from "Normal" to "High".
TASK DETAIL
https://phabricator.wikimedia.org/T116404
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: aude, daniel, hoo, Aklapper, jcrespo, Wikidata-b
jcrespo added a comment.
A similar thing is happening on zhwiki for a different query- the optimizer
seems to have some bug for that wiki in particular?
TASK DETAIL
https://phabricator.wikimedia.org/T116404
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
jcrespo added a comment.
Can confirm last seen on db1018:
SELECT DISTINCT eu_entity_id FROM `wbc_entity_usage` WHERE eu_entity_id IN
('Q10864210','Q10866766','Q10874855','Q10877844','Q10877846','Q10878314','Q10879635',
'Q10880445','Q10882043','Q10887655','Q10890010','Q10890075
jcrespo created this task.
jcrespo added a subscriber: jcrespo.
jcrespo added projects: Performance, Database, Wikidata, operations.
Herald added a subscriber: Aklapper.
TASK DESCRIPTION
This is mainly observed on [[
https://tendril.wikimedia.org/host/view/db1052.eqiad.wmnet/3306 | s1
jcrespo added a comment.
You are opening one connection per wiki. That is wrong. Locking per wiki will
serve nothing.
TASK DETAIL
https://phabricator.wikimedia.org/T118162
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Tobi_WMDE_SW
jcrespo added a comment.
| 206189024 | wikiadmin | 10.64.32.13:49847 | ruwikibooks |
Sleep |
| 206189066 | wikiadmin | 10.64.32.13:49848 | sahwiki |
Sleep |
| 206189112 | wikiadmin | 10.64.32.13:49850 | fiwikinews |
Sleep
jcrespo closed this task as "Resolved".
jcrespo claimed this task.
jcrespo added a comment.
Resolving, and hopefuly future issues can be resolved in further tickets. I am
ok with the current state.
TASK DETAIL
https://phabricator.wikimedia.org/T118162
EMAIL PREFERENC
jcrespo created this task.
jcrespo added a subscriber: jcrespo.
jcrespo added projects: Wikidata, MediaWiki-Database.
Herald added a subscriber: Aklapper.
TASK DESCRIPTION
There seems to be a large number of API calls to wikidata creating a high
number of deadlocks like this (450/hour
jcrespo closed this task as "Declined".
jcrespo claimed this task.
jcrespo added a comment.
I'm more than ok with the work you are doing, I was just suggesting making the
code faster :-)
TASK DETAIL
https://phabricator.wikimedia.org/T111535
EMAIL PREFERENC
jcrespo added a subscriber: jcrespo.
TASK DETAIL
https://phabricator.wikimedia.org/T111769
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: jcrespo, aude, Aklapper, Wikidata-bugs
___
Wikidata
jcrespo added a comment.
@JanZerebecki not sure if you are asking me: Yes, IF that is the case (I cannot
say)- batching will actually be worse than no batching at all- as it will be
slower than a single query and more open to clashing with other queries without
any of the advantages (reduced
jcrespo added a blocked task: T30599: Deadlock tracking bug (tracking).
TASK DETAIL
https://phabricator.wikimedia.org/T111535
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: aude, JanZerebecki, Addshore, Aklapper, jcrespo, Wikidata-bugs
jcrespo moved this task to Backlog on the Database workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T111769
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: daniel
jcrespo added a comment.
I set the priority to low "nice to fix it, but it is not breaking anything
now". Feel free to disagree with me.
TASK DETAIL
https://phabricator.wikimedia.org/T111535
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
T
jcrespo triaged this task as "Low" priority.
TASK DETAIL
https://phabricator.wikimedia.org/T111535
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: aude, JanZerebecki, Addshore, Aklapper, jcrespo, Wik
jcrespo added a comment.
It could be an enqueue issue, if thousands of those queries are sent at the
same time (due to bot traffic, for example), they would be enqueued on server
side and executed with controlled concurrency. I could check the binary logs to
see if that is the case
jcrespo removed a blocking task: Restricted Task.
TASK DETAIL
https://phabricator.wikimedia.org/T111954
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Smalyshev, Karima, jkroll, Wikidata-bugs, Jdouglas, aude, Deskana,
Manybubbles
jcrespo closed blocking task Restricted Task as "Invalid".
TASK DETAIL
https://phabricator.wikimedia.org/T111954
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Smalyshev, Karima, jkroll, Wikidata-bugs, Jdouglas, aud
jcrespo moved this task to Backlog on the Database workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T116404
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: aude
jcrespo added projects: DBA, Blocked-on-schema-change.
jcrespo added a comment.
@hoo: new procedure (I am trying to advertise this as much as possible so
changes are not lost): https://wikitech.wikimedia.org/wiki/Schema_changes
TASK DETAIL
https://phabricator.wikimedia.org/T62539
EMAIL
jcrespo added a subscriber: jcrespo.
TASK DETAIL
https://phabricator.wikimedia.org/T114019
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: ArielGlenn, jcrespo
Cc: jcrespo, Bianjiang, madhuvishy, Milimetric, RobLa-WMF, GWicke, TTO,
zhuyifei1999
jcrespo moved this task to Backlog on the DBA workboard.
TASK DETAIL
https://phabricator.wikimedia.org/T62539
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: hoo, Krenair
jcrespo added a subscriber: jcrespo.
jcrespo added a comment.
Diff between cached and rendered:
P2457 wikivoyage format <https://phabricator.wikimedia.org/P2457>
Visual comparison, for reference:
F3184405: wikivoyage_before.png <https://phabricator.wikimedia.org/F3184405&g
jcrespo added a comment.
@hoo thanks for taking the time, I will reward following the procedure with
preference on the queue of pending changes. Sadly, I cannot guarantee this to
be done immediately, but probably will be scheduled for the week of the 19th of
January.
TASK DETAIL
https
jcrespo added a comment.
This may be one of the reasons of https://phabricator.wikimedia.org/T122429.
TASK DETAIL
https://phabricator.wikimedia.org/T111769
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: daniel, jcrespo, aude, Aklapper
jcrespo added a comment.
I am thinking of killing and banning these connections right now from s3
because it is breaking our database servers. It is still creating 1000-2000
idle connections to db1035.
TASK DETAIL
https://phabricator.wikimedia.org/T118162
EMAIL PREFERENCES
https
jcrespo added a comment.
> Now Jynus sais that he doesn't believe that fix is going to help our
> situation. I'm curious about why he believes that.
Connection re-use does not work if you open 900 connections at the same time
every 3 minutes. I have been saying that for a long time. If
jcrespo closed this task as a duplicate of T119315: labsdb1002 and labsdb1003
crashed, affecting replication.
TASK DETAIL
https://phabricator.wikimedia.org/T119382
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Aklapper, Magnus
jcrespo reopened this task as "Open".
jcrespo added a subscriber: ArielGlenn.
jcrespo added a comment.
This issue happened again today between 4:26 and 7:20 (I may have done
something to end it because I had to do some unrelated server changes):
https://logstash.wikimedia.org/
jcrespo added a comment.
@hoo, my apologies- I thought it had been already deployed. Let me keep it
open until it deploys so that 3rd parties can see it can happen for now- you
can move it if you need it on the
https://phabricator.wikimedia.org/tag/wikidata/ dashboard.
TASK DETAIL
https
jcrespo moved this task from Triage to In progress on the DBA board.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
WORKBOARD
https://phabricator.wikimedia.org/project/board/1060/
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: hoo, jcrespo
jcrespo added a comment.A slight comment here: I know the second query is way better than the the first one- there could be an even better solution; but just doing this change would be a huge improvement (at least 100x faster).TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL
jcrespo added a comment.@daniel I think this is a case of prematurely optimizing. It is true that things like:
foreach ... { 'SELECT' }
Are usually considered bad practices, but for trying to minimize "round-trip time", we are actually doing a way worse query. Your query c
jcrespo added a comment.This is not a problem with the servers, the query planner, or the indexing:
MariaDB db1068 commonswiki > EXPLAIN SELECT DISTINCT eu_entity_id FROM `wbc_entity_usage` WHERE eu_entity_id IN ('Q148475','Q54919','Q423048','Q2494649','Q13219454','Q131454','Q36
jcrespo added a comment.I like your approach @daniel, starting simple, then optimize with a more complex option. Thank you!TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Zppix
jcrespo added a comment.
There are a couple of long-running tasks on the master, from snapshot1003-
that is strange, but not sure it is an issue.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo created this task.
Herald added subscribers: Zppix, Aklapper.
TASK DESCRIPTION
There is a huge amount of connections to the master, as many that there are
5000 connection errors per minute.
While I cannot yet put a reason to it, most of the connections that fail seem
jcrespo edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Aklapper, jcrespo, Zppix, Minhnv-2809, Volans, D3r1ck01, Izno, Luke081515,
Wikidata-bugs, aude
jcrespo edited the task description.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: JanZerebecki, hoo, aude, Aklapper, jcrespo, Zppix, Minhnv-2809, Volans,
D3r1ck01, Izno, Luke081515
jcrespo added a comment.
Termininja, you do not need to write only once per minute, something like 1
POST/write per second is unlikely to cause issues, just wait for one edit to
finish before starting the next one (DO NOT SENT MULTIPLE REQUEST IN PARALLEL).
I will tell the admins
jcrespo added a comment.
Termininja, let's try to fix this. Can you run edits exclusively in a serial
way, as suggested, even if that means they will be slower?
TASK DETAIL
https://phabricator.wikimedia.org/T135471
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel
jcrespo closed this task as "Resolved".
jcrespo claimed this task.
jcrespo added a comment.
Your account has been already unblocked.
TASK DETAIL
https://phabricator.wikimedia.org/T135471
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To:
jcrespo added a comment.
I do not know what happened there:
F4098270: Screenshot from 2016-06-01 11:36:39.png
<https://phabricator.wikimedia.org/F4098270>
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings
jcrespo added a comment.
A high sustained level of errors seem to have gone at 21:40-21:45 yesterday.
The errors continue, but only in smaller spikes. We'll see if it returns when
user activity grows more.
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https
jcrespo added a comment.
@JanZerebecki - our max_connections is very high: 1 simultaneous
connections- because we use connection pooling at server side. We almost never
reach that- mediawiki servers get saturated first; what fails is the connection
timeout, which is set to 3 seconds
jcrespo added a comment.
Sorry I wasn't specific enough. This issue is only happening on db1049:
s5-master (Wikidata master).
TASK DETAIL
https://phabricator.wikimedia.org/T136598
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc
jcrespo added a comment.
If you ask for wikis, this is the distribution:
filter: `wikibase-addUsagesForPage and 10.64.16.144`
(10.64.16.144 is s5-master)
dewiki (777) enwiki (691) wikidatawiki (487) frwiki (423) zhwiki (415)
commonswiki (384) svwiki (375) itwiki (287) ptwiki
jcrespo added a subscriber: hoo.jcrespo added a comment.
With 'dump', I sometimes mean 'vslow', too (e.g. for terbium).TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: hoo, ArielGlenn, jcrespo
jcrespo added a comment.
@ArielGlenn : let's identify the reason why this is happening before changing things- we may implement proxying/etcd config before changing any logic. The important part here and now is that probably some mediawiki class is not using the dump role, maybe related
jcrespo added a comment.
These jobs need access to the main DBs
What is a "main db" and what is the difference with a 'vslow' slave? You are accessing a testing slave, that will be put down at any moment (or will block & kill terbium traffic).TASK DETAILhttps://phabricator
jcrespo added a comment.
Let me show you the weight of these servers:
's5' => array(
'vslow' => array(
'db1045' => 1,
),
'dump' => array(
'db1045' => 1,
),
'api' => array(
'db1070' => 1,
'db1071' => 1,
),
'watchlist' => array(
'db1026'
jcrespo added a comment.
As I said:
If these create light-weight queries only, lets disconnect and connect after some amount of seconds. If dump hosts are too slow, let's give them better resources. If there is a problem with the connection framework, let'x fix it with the addition of a proxy
jcrespo added a comment.
This is not an emergency, I handled that, but it should be definitely 'high'- I suspect it is what it is causing queries such as dumps fail/go slow. T138291 has a different root cause, but I assume it is related to this.
There has been dewiki bot (api) users complaining
jcrespo added a comment.
However, there are potentially several dozen places where we call LoadBalancer::getConnection (we try not to hog the connection, but only get it from the LB when we need it - so we do that often). We'd have to somehow loop this parameter through to all the places where we
jcrespo added a comment.
Probably related to T138208, I would set that as unbreak now.TASK DETAILhttps://phabricator.wikimedia.org/T138291EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: jcrespo, Stashbot, hoo, Melderick, Zppix, D3r1ck01, Izno
jcrespo added a comment.
Right now my only worries are for wikidata, because they create large amount of connection errors and are very visible- I can check on other shards, but even if they do, it would be very low priority (no infrastructure issues there).TASK DETAILhttps
jcrespo added a comment.
10.64.48.26 is db1071, a regular-traffic db.TASK DETAILhttps://phabricator.wikimedia.org/T138208EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: hoo, ArielGlenn, jcrespo, Zppix, D3r1ck01, Izno, Wikidata-bugs, aude
jcrespo added a comment.
@Dzahn check your mail.TASK DETAILhttps://phabricator.wikimedia.org/T134017EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: jcrespoCc: Jdforrester-WMF, RobH, Lydia_Pintscher, StevenJ81, hoo, aude, Multichill, VIGNERON, Nikki, chasemp
jcrespo added a project: Wikidata.jcrespo added a comment.
This is not only happening for dumps, terbium is also wrongly using main-dbs (which are still on testing) for long-running queries, which cause long periods of connection issues: https://logstash.wikimedia.org/#dashboard/temp
jcrespo closed this task as "Resolved".jcrespo added a comment.
Master connection issues not seen for a week.TASK DETAILhttps://phabricator.wikimedia.org/T136598EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: Stashbot, ArielGle
jcrespo created blocking task T137972: Use UNION ALL instead of UNION for bulk wbc_entity_usage queries.
TASK DETAILhttps://phabricator.wikimedia.org/T137539EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hoo, jcrespoCc: gerritbot, Zppix, Aklapper, hoo, aude
jcrespo edited the task description. (Show Details)
EDIT DETAILS...This maybe circumstantial, and not even related, but there is in some cases a slight performance improvement while using `UNION ALL` instead of `UNION`, the first not neededing a temporary table and being able to return rows
jcrespo created this task.jcrespo added projects: Performance, Wikidata.
TASK DESCRIPTIONHi, with the resolution on the previous issue (T137539), performance has increased, however, I have detected an increase on the number of temporary tables:
F4172090: Screenshot from 2016-06-16 17:49:41.png
jcrespo added a comment.
As a related, but probably offtopic, you may want to have a look at this page:
https://www.wikidata.org/wiki/Wikidata:List_of_properties/Summary_table
TASK DETAIL
https://phabricator.wikimedia.org/T123867
EMAIL PREFERENCES
https://phabricator.wikimedia.org
jcrespo added a comment.
Actually, now that I see, both could be responsible, as both are reported
around 15:30 UTC.
> Given we have statement based replication
Not 100% truth everywhere anymore. See our puppet config.
> these should show up for the slaves as well, in case they run for
jcrespo added a comment.
Last 2 days there were 251 hits of
DELETE /* MessageGroupStats::clear 127.0.0.1 */
FROM `translate_groupstats`
WHERE tgs_group = 'page-Wikidata:List of properties/Summary table'
AND tgs_lang = 'en'
/* e567ea5f86b3e2b07cd283d8aa9270a9 db1058 wikidatawiki 11s
jcrespo added a blocked task: T95501: Fix causes of slave lag and get it to
under 5 seconds at peak (tracking).
TASK DETAIL
https://phabricator.wikimedia.org/T122429
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: JanZerebecki, hoo
jcrespo added a comment.
Could be related to either https://phabricator.wikimedia.org/T122429 or
https://phabricator.wikimedia.org/T109943.
TASK DETAIL
https://phabricator.wikimedia.org/T123867
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences
jcrespo removed blocking tasks: T109179: Migrate MySQLs to use ROW-based
replication (tracking), T95501: Fix causes of slave lag and get it to under 5
seconds at peak (tracking).
TASK DETAIL
https://phabricator.wikimedia.org/T122429
EMAIL PREFERENCES
https://phabricator.wikimedia.org
jcrespo added a blocked task: T95501: Fix causes of slave lag and get it to
under 5 seconds at peak (tracking).
TASK DETAIL
https://phabricator.wikimedia.org/T123867
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: Aklapper, StudiesWorld
jcrespo added a blocked task: T109179: Migrate MySQLs to use ROW-based
replication (tracking).
TASK DETAIL
https://phabricator.wikimedia.org/T122429
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jcrespo
Cc: JanZerebecki, hoo, Glaisher, Aklapper
jcrespo added a comment.
I wanted to be added to these kind of task because this is already done for
*all tables* for operations-related tasks, so it may duplicate efforts.
Improvements have to be made, though.
TASK DETAIL
https://phabricator.wikimedia.org/T68025
EMAIL PREFERENCES
https
jcrespo added a comment.
> Is this monitored publicly?
That is part of the improvements, it is currently non-public (NDA required). It
will probably be on grafana soon, but we are migrating the monitoring backend
system at the same time (being tested on
http://prometheus.wmflabs.org/graf
jcrespo added a comment.
NDA only link (example for db1058, s5-master):
https://tendril.wikimedia.org/report/table_status?host=db1058=wikidata
This is NDA-only because it is unlikely, but possible that confidential
information may appear there, related to user's privacy (query log).
Tendril
1 - 100 of 581 matches
Mail list logo
