gerritbot added a comment.
Change 413899 merged by jenkins-bot:
[operations/mediawiki-config@master] Add configuration for CirrusSearch to instantly index new Wikidata items
https://gerrit.wikimedia.org/r/413899TASK DETAILhttps://phabricator.wikimedia.org/T183053EMAIL
gerritbot added a comment.
Change 413492 merged by jenkins-bot:
[mediawiki/extensions/CirrusSearch@master] Allow some wikis to instantly index newly created articles
https://gerrit.wikimedia.org/r/413492TASK DETAILhttps://phabricator.wikimedia.org/T183053EMAIL
gerritbot added a comment.
Change 413899 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/mediawiki-config@master] Add configuration for CirrusSearch to instantly index new Wikidata items
https://gerrit.wikimedia.org/r/413899TASK
gerritbot added a comment.
Change 413492 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[mediawiki/extensions/CirrusSearch@master] Allow some wikis to instantly index newly created articles
https://gerrit.wikimedia.org/r/413492TASK
gerritbot added a comment.
Change 399466 merged by jenkins-bot:
[operations/mediawiki-config@master] Lower ElasticSearch index refresh interval for Wikidata to 5s
https://gerrit.wikimedia.org/r/399466TASK DETAILhttps://phabricator.wikimedia.org/T183053EMAIL
EBernhardson added a comment.
sending the basic info semi-synchronously (from DeferredUpdates, which will run in the same process as the edit but after closing the connection to the user) should be ok. Actually generating a "basic" set instead of the full thing might be more difficult than
Smalyshev added a comment.
We could maybe just sent basic info to ES when saving a new article, synchronously (not sure if it's a good idea, just putting it out there) and then let the jobs update it with full data. In the minus side, we'll get one extra document write which is then immediately
EBernhardson added a comment.
We may still need to look into the special-case of newly created pages being indexed from the web request, rather than being punted into the job queue. cirrusSearchLinksUpdatePrioritized, which performs the actual generation of a document and write to elasticsearch,
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2018-01-02T18:37:19Z] T183053 update index.refresh_interval for wikidatawiki_{content,general} on eqiad to 5sTASK DETAILhttps://phabricator.wikimedia.org/T183053EMAIL
Smalyshev added a comment.
Looks to me that 5s is working fine. I'll add a config patch.TASK DETAILhttps://phabricator.wikimedia.org/T183053EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: gerritbot, Stashbot, debt, jhsoby, Lydia_Pintscher,
gerritbot added a comment.
Change 399466 had a related patch set uploaded (by Smalyshev; owner: Smalyshev):
[operations/mediawiki-config@master] Lower refresh interval for Wikidata to 5s
https://gerrit.wikimedia.org/r/399466TASK DETAILhttps://phabricator.wikimedia.org/T183053EMAIL
EBernhardson added a comment.
Took some measurements of refresh rate averaged over 5 minutes pre and post-deployment. Overall it's perhaps a 15% increase in refresh/minute across the cluster. Disk IO graphs don't show anything particularly interesting. There will certainly be more merge volume as
Stashbot added a comment.
Mentioned in SAL (#wikimedia-operations) [2017-12-20T20:31:14Z] T183053 update elasticsearch settings for wikidatawiki_content on codfw to use: index.refresh_interval=5sTASK DETAILhttps://phabricator.wikimedia.org/T183053EMAIL
dcausse added a comment.
Same for me I'd be for trying to increase the refresh rate on wikidata_content.TASK DETAILhttps://phabricator.wikimedia.org/T183053EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: dcausseCc: debt, jhsoby, Lydia_Pintscher, EBernhardson,
Smalyshev added a comment.
I agree that we should try to lower the refresh rate to 5s and see whether it works.TASK DETAILhttps://phabricator.wikimedia.org/T183053EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: SmalyshevCc: debt, jhsoby, Lydia_Pintscher,
EBernhardson added a comment.
It seems there are a couple options here, my thoughts:
Reduce the default refresh interval
By default we use a 30 second refresh interval for all wikis. This means that 30 seconds worth of updates get bundled together into a single update. Updates are not searchable
Smalyshev added a comment.
So item creation rate is about 85k per day, or very close to one per second. Bots seem to dominate that though, so for real users it will be lower. Also, some of those are probably tools like QuickStatements for which it also could be fine to have the regular delay -
Lydia_Pintscher added a comment.
The first graph on https://grafana.wikimedia.org/dashboard/db/wikidata-datamodel?refresh=30m=1 shows the number of new items created over time.
For the particular problem indeed bots could be taken out. The make up the biggest part of new page creations on Wikidata
dcausse added a comment.
If a large majority of such usecases involve searching the entity id (QXXX) of the newly created item we can perform an additional db match to compensate the lag of the search index.
It's what we do for normal wikis, a db match is run in addition to the query sent to the
19 matches
Mail list logo
