[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

EBernhardson added a comment. To do something about this some number needs to be decided on. What is the appropriate amount of acceptable lag? From the search engine perspective a minute or even five is considered fully working. TASK DETAIL https://phabricator.wikimedia.org/T287563

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

Esc3300 added a comment. I'm not sure if everybody knows what "60s" or "162s" (what may seem quick) actually means to editors: it's the time they wait between creating the item and adding it as a statement value (which there seems long). Personally, I paste qids as search isn't

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

Lydia_Pintscher added a comment. We have editors experiencing problems like this again. Can you have another look please? TASK DETAIL https://phabricator.wikimedia.org/T287563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel,

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

Gehel closed this task as "Resolved". Gehel claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T287563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: So9q, Pchelolo, hnowlan, EBernhardson, dcausse, Nikki, Aklapper,

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

So9q added a comment. @EBernhardson That is a very nice script! Thanks for annotating it, so I almost understand everything going on.  I just ran it for a few minutes and I saw lags of 70-162 seconds. The highest being of a lexeme. TASK DETAIL https://phabricator.wikimedia.org/T287563

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

EBernhardson added a comment. I wrote a quick script (P17040 ) to read the public event streams to get revision timestamps, and then poll the mediawiki api's to find out which revision we have indexed in search. I haven't run any fancy stats over

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

EBernhardson added a comment. Current rates and backlog on ElasticaWrite look reasonable, p99 backlog is under 10m, roughly comparable to the numbers we saw in eqiad. Overall looks healthy. Quasi relatedly though, the backlog on cirrusSearchIncomingLinkCount has increased recently. The

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

MPhamWMF moved this task from needs triage to Current work on the Discovery-Search board. MPhamWMF edited projects, added Discovery-Search (Current work); removed Discovery-Search. TASK DETAIL https://phabricator.wikimedia.org/T287563 WORKBOARD

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

hnowlan added a subscriber: Pchelolo. hnowlan added a comment. In T287563#7246465 , @dcausse wrote: > Hi @hnowlan in case you have ideas, we're investigating why the `cirrusSearchElasticaWrite` job is being backlogged more frequently

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

dcausse added a subscriber: hnowlan. dcausse added a comment. Hi @hnowlan in case you have ideas, we're investigating why the `cirrusSearchElasticaWrite` job is being backlogged more frequently since the switch. There does not seem to be more messages produced to it nor we see more

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

dcausse added a comment. The ElasticaWrite job seems to be receiving roughly the same amount of messages (150/s per partition on average) before and after the switch. Looking at the partitioned topic ElasticWrite it's heavily backlogged since the switch: F34569267: Capture d’écran du

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

EBernhardson added a comment. Checked a few things, looking for why this job would have increased it's backlog. Most notable metric change is that eqiad was processing ElasticaWrites at a rate of ~150 for the partitioned topic. codfw is currently averaging over 250, Where did the extra 100

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

dcausse added a comment. Looking at the metrics of the `cirrusElasticaWrite` job it seems that its backlog time was greatly degraded just after the DC switch (June 28). I don't see anything particular in other dashboards that could explain such difference in latency. TASK DETAIL

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

Addshore added a project: wdwb-tech. TASK DETAIL https://phabricator.wikimedia.org/T287563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore Cc: Nikki, Aklapper, Lydia_Pintscher, Invadibot, MPhamWMF, maantietaja, CBogen, Akuckartz, Nandana,

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

Nikki added a comment. Lexemes are also taking longer than they used to to show up, although I haven't checked whether it's taking as long as items. TASK DETAIL https://phabricator.wikimedia.org/T287563 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/

[Wikidata-bugs] [Maniphest] T287563: slow indexing of new Items on Wikidata?

Lydia_Pintscher created this task. Lydia_Pintscher added projects: Elasticsearch, Wikidata. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Discovery-Search. TASK DESCRIPTION We have some anecdodal evidence that recently indexing of new Items in the