[Wikidata-bugs] [Maniphest] T255706: [SW] [WB-Client] [TECH] Wikibase\Client\Usage\Sql\EntityUsageTable::addUsages Deadlock
aaron added a comment. I noticed that addUsages() uses $this->db->replication()->wait(); which uses LBFactory::waitForReplication(). Doesn't that mean it's waiting for replicas without committing each batch? It seems like it would just hold more and more locks while waiting for unrelated events to replicate. Maybe AddUsagesForPageJob could use self::JOB_NO_EXPLICIT_TRX_ROUND (like RefreshLinksJob) and LBFactory::commitAndWaitForReplication() could be used instead. Perhaps addUsagesForPage() could wrap the update in a named lock based on the page ID, similar to LinksUpdate::acquirePageLock. Maybe addUsedEntities() could also do a SELECT first to see which rows are already there. I see the job spec for AddUsagesForPageJob has "removeDuplicates" but the parameters includes "usages". Maybe successive edits result in slight differences in params that bypass de-duplication? TASK DETAIL https://phabricator.wikimedia.org/T255706 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: ItamarWMDE, Ladsgroup, Krinkle, eprodromou, aaron, Michael, Aklapper, thcipriani, Danny_Benjafield_WMDE, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, Akuckartz, darthmon_wmde, Rosalie_WMDE, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Verdy_p, Wikidata-bugs, aude, Jdforrester-WMF, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T237249: Globally track Scribunto LuaSandbox profiling data
aaron closed this task as a duplicate of T225969: Per template/Lua module profiling with Grafana dashboards. TASK DETAIL https://phabricator.wikimedia.org/T237249 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Aklapper, Lydia_Pintscher, hoo, LennardHofmann, Astuthiodit_1, karapayneWMDE, Invadibot, maantietaja, ItamarWMDE, Akuckartz, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, SundanceRaphael, _jensen, rosalieper, Scott_WUaS, Izno, alex-mashin, Wikidata-bugs, aude, Dinoguy1000, jayvdb, MrStradivarius, Jackmcbarn, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T316487: Wikibase cannot save properties on SQLite with PHP 8.1
aaron added a comment. I don't think wikibase, or MediaWiki code generally, should be so tightly coupled to the driver returning strings. Ideally, I somewhat prefer using native PHP integers...but the our sqlite/mysql/postgres Database subclasses should be consistent. We can possibly use ATTR_STRINGIFY_FETCHES for now. TASK DETAIL https://phabricator.wikimedia.org/T316487 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Krinkle, CtrlZvi, Aklapper, Hellket777, LisafBia6531, Astuthiodit_1, 786, Biggs657, karapayneWMDE, Invadibot, maantietaja, Juan90264, Alter-paule, Beast1978, ItamarWMDE, Un1tY, Akuckartz, Hook696, darthmon_wmde, Kent7301, joker88john, CucyNoiD, Nandana, Gaboe420, Giuliamocci, Cpaulf30, Rayssa-, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, Lewizho99, Maathavan, _jensen, rosalieper, Agabi10, Neuronton, Scott_WUaS, Wikidata-bugs, aude, Dinoguy1000, waldyrious, Lydia_Pintscher, Nikerabbit, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T298682: Wikibase\Repo\Store\Sql\SqlIdGenerator::generateNewId deadlock on wb_id_counters when running selenium tests in parallel
aaron added a comment. \Wikibase\Repo\Store\Sql\SqlIdGenerator definitely looks prone to deadlocks. It should probably work more like TableNameStore (named locks + auto-commit trx). TASK DETAIL https://phabricator.wikimedia.org/T298682 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, Aklapper, kostajh, hashar, Invadibot, maantietaja, Akuckartz, DannyS712, Nandana, lucamauri, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, zeljkofilipin, Jdforrester-WMF, Addshore, Mbch331 ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T195792: Create Mediawiki DB abstraction for individual query timeouts
aaron moved this task from Inbox to Radar on the Performance-Team board. aaron edited projects, added Performance-Team (Radar); removed Performance-Team. TASK DETAIL https://phabricator.wikimedia.org/T195792 WORKBOARD https://phabricator.wikimedia.org/project/board/1212/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: CDanis, Kormat, dpifke, danshick-wmde, Aklapper, Legoktm, Marostegui, jcrespo, aaron, daniel, Addshore, Invadibot, maantietaja, Akuckartz, WDoranWMF, holger.knust, EvanProdromou, Nandana, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Dinoguy1000, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T195792: Create Mediawiki DB abstraction for individual query timeouts
aaron merged a task: T293536: MediaWiki should support setting a read query time limit. aaron added subscribers: dpifke, Kormat, CDanis. Restricted Application added a project: Performance-Team. TASK DETAIL https://phabricator.wikimedia.org/T195792 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: CDanis, Kormat, dpifke, danshick-wmde, Aklapper, Legoktm, Marostegui, jcrespo, aaron, daniel, Addshore, Invadibot, maantietaja, Akuckartz, WDoranWMF, holger.knust, EvanProdromou, Nandana, Rayssa-, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wikidata-bugs, aude, Dinoguy1000, Mbch331, Jay8g ___ Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org
[Wikidata-bugs] [Maniphest] T213089: Upgrade memcached cluster to Debian Stretch/Buster
aaron added a comment. Regarding RedisLockManager (it only needs 2 of the 3 host to be reachable). If one of them is depooled or refuses connections, no one should notice any disruption. For otherwise unreachable servers, there is a 2 second timeout (and the redis server will be avoided for the rest of the request). TASK DETAIL https://phabricator.wikimedia.org/T213089 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Ladsgroup, Krenair, elukey, jijiki, faidon, Paladox, Aklapper, Krinkle, aaron, Joe, MoritzMuehlenhoff, lmata, wkandek, JMeybohm, Akuckartz, WDoranWMF, holger.knust, EvanProdromou, Legado_Shulgin, Nandana, Jony, Davinaclare77, Qtn1293, Techguru.pc, Lahi, Gq86, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, LawExplorer, Vali.matei, Zppix, _jensen, rosalieper, Agabi10, Scott_WUaS, Pchelolo, Wong128hk, Wikidata-bugs, aude, Mbch331, Rxy, Jay8g, fgiunchedi, Dzahn ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T246456: Performance review of Wikidata Bridge
aaron added a comment. From the perspective of popular/major articles, likely to have infoboxes, the extra 42.1 KB for loading the "app" JS doesn't seem crazy. I've looked through code several times and it seems reasonable. Testing with fast/slow 3G doesn't reveal obnoxious reflows or delay either. Having the edit link go directly to a Q page when the JS hasn't fully loaded felt somewhat jarring, though I don't image that happening often. I don't see much editing at all given how discrete the icon is (a good thing). The bootstrapping "init" JS is also pretty tiny. It does have a fair number of module references in the using() call for pages with editable elements. OTOH, those seem to be loaded anyway, with the "app" being the only new thing triggered. The DOM search for editable entity links is just a simple CSS selector call with reasonable metadata extraction. I don't see (nor did I perceive) any CPU use or long task issue there. The client <=> api.php layer looks reasonable and well abstracted. Given the low-key nature of the GUI, I don't foresee any obvious edit rate, DB overhead, nor contention issues. I don't see any reason to block the wikidata-bridge deployment and consider this task resolved from my end. TASK DETAIL https://phabricator.wikimedia.org/T246456 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Gilles, Jakob_WMDE, Lydia_Pintscher, Addshore, Krinkle, Aklapper, darthmon_wmde, Michael, Nandana, Jony, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T246456: Performance review of Wikidata Bridge
aaron added a comment. I've been looking at this from time to time, and haven't found anything real problems yet. Some of the things I'm looking out for are: - Pageview critical path effects: - Bytes (JS) - Bytes (CSS+images) - Page load delay - First input delay - Content reflows - Post-load pageview interaction effects: - Hover delays - Input delays - Backend effects - DB I/O usage - DB contention issues - Search index store usage - Key/value store usage - Cache usage I'll be looking at the frontend some more this weekend, but I expect to sign-off as "LGTM" monday. TASK DETAIL https://phabricator.wikimedia.org/T246456 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Gilles, Jakob_WMDE, Lydia_Pintscher, Addshore, Krinkle, Aklapper, darthmon_wmde, Michael, Nandana, Jony, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T246456: Performance review of Wikidata Bridge
aaron added a comment. In T246456#6116523 <https://phabricator.wikimedia.org/T246456#6116523>, @darthmon_wmde wrote: > hey @aaron, @Gilles, > > could you, please, give us an update on this task? could you also tell us something we could tackle proactively that may help? > > We are currently implementing the last stories and that information would help us enormously to shape our next steps. > > Thanks a lot in advance! Sorry, I was reading through the documentation before a 1 week vacation, and then getting distracted by debugging T249069 <https://phabricator.wikimedia.org/T249069> and coordinating on T236414 <https://phabricator.wikimedia.org/T236414> and T250205 <https://phabricator.wikimedia.org/T250205> related issues. Getting back to this is my next priority after fixing T249069 <https://phabricator.wikimedia.org/T249069>. TASK DETAIL https://phabricator.wikimedia.org/T246456 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Gilles, Jakob_WMDE, Lydia_Pintscher, Addshore, Krinkle, Aklapper, darthmon_wmde, Sarai-WMDE, Michael, Nandana, Jony, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, rosalieper, Scott_WUaS, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T248147: Wikimedia\Rdbms\Database::normalizeUpsertKeys called with deprecated parameter style: the unique key array should be a string or array of string arrays ge
aaron closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T248147 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Krinkle, Jdforrester-WMF, jijiki, aaron, Bawolff, Aklapper, thcipriani, hashar, Marostegui, NavinRizwi, Prondubuisi, eprodromou, Iflorez, darthmon_wmde, alaa_wmde, 94rain, DannyS712, Nandana, Analytics.mediafiles, kostajh, Jony, Force_Radical, Lahi, Gq86, Xover, GoranSMilovanovic, Vexations, Jayprakash12345, QZanden, LawExplorer, Tshrinivasan, _jensen, rosalieper, Agabi10, Taiwania_Justo, Scott_WUaS, Pchelolo, Jonas, Info-farmer, Wikidata-bugs, aude, Candalua, Huji, Lydia_Pintscher, Tpt, Mbch331, Rxy, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T157651: sql.php runs LoadExtensionSchemaUpdates
aaron added a comment.
In T157651#6039302 <https://phabricator.wikimedia.org/T157651#6039302>, @Tgr
wrote:
> I would suggest the opposite: keep `sql.php`, drop `patchSql.php`. I don't
think many people are familiar with the latter (compare patchSql
<https://www.mediawiki.org/w/index.php?title=Special:Search=500=0=1=1=1=1=1=1=%22patchSql.php%22={}>
vs sql
<https://www.mediawiki.org/w/index.php?title=Special:Search=500=0=1=1=1=1=1=1=%22sql.php%22={}>
docs for example) and I don't think it's terribly useful - passing a file path
is more user-friendly than passing a patch name. And it does not even replace
the schema vars, meaning it's actively harmful to anyone who uses table
prefixes or non-default table settings.
> So, IMO
>
> - keep `mysql.php` which is indeed widely used at least in Wikimedia
production for debugging, more user-friendly than sql.php (which channels query
output through PHP which does weird things to it) and not problematic (it
already requires a write flag for performing any changes - although that relies
on a master/slave distinction so that could be improved, cf T249683#6039238
<https://phabricator.wikimedia.org/T249683#6039238> - and does not accidentally
run updares).
> - kill `patchSql.php` which is IMO pretty useless. (Probably worth a
wikitech question to ensure it is indeed not used.)
> - keep `sql.php` manual debugging mode, which is the only way to debug a
non-MySQL server, but require an explicit `--debug` flag used. Do not invoke
the updater (not even for variable transformations) when that's used, it seems
pointless and just extra code path exposure (cf the fatal error it gave during
the incident).
> - keep `sql.php` for running scripts but require a `--write` flag like
`mysql.php` does for scripts that change data. (I would even separate an admin
mode for schema changes and a write mode for data changes via a restricted
user.)
> - if `sql.php` is invoked with no script file and `--debug` flag just exit
with an error (and without creating a DatabaseUpdater or doing anything else
nontrivial).
This all makes sense to me. Since DDLs are not transactional in mariadb and
so on, it adds an extra layer of disruption over "DELETE WHERE 1=1".
Permissions with temp tables could get annoying though (Database.php read-only
enforcement already has to look out for them). Starting off with a --write flag
seems doable for now. It would be nice to make it use a replica DB by default
(if --write is missing and --replicadb is not already set to a host or *).
TASK DETAIL
https://phabricator.wikimedia.org/T157651
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: aaron
Cc: Jdforrester-WMF, kostajh, WMDE-leszek, Addshore, Anomie, alaa_wmde, daniel,
Ladsgroup, Pablo-WMDE, kchapman, Krinkle, Catrope, Reception123, gerritbot,
aaron, Aklapper, Tgr, Blissjay007, Oblanco79, Alter-paule, NavinRizwi,
Beast1978, Un1tY, Demian, eprodromou, Hook696, Daryl-TTMG, RomaAmorRoma,
E.S.A-Sheild, darthmon_wmde, Kent7301, Meekrab2012, joker88john, 94rain,
CucyNoiD, Nandana, Lens0021, NebulousIris, jijiki, Gaboe420, Jony, Versusxo,
Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86,
Af420, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382,
45Jayjay1969, Jayprakash12345, Th3d3v1ls, Ramalepe, Liugev6, QZanden,
EnricoCNC, LawExplorer, Vali.matei, WSH1906, Lewizho99, Maathavan, elukey,
_jensen, rosalieper, Agabi10, Taiwania_Justo, Scott_WUaS, Pchelolo, SBisson,
Wong128hk, Wikidata-bugs, aude, Dcljr, Bawolff, Gryllida, jeblad, ArielGlenn,
He7d3r, Mbch331, Rxy, Jay8g, akosiaris
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Claimed] T248147: Wikimedia\Rdbms\Database::normalizeUpsertKeys called with deprecated parameter style: the unique key array should be a string or array of string arrays g
aaron claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T248147 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Krinkle, Jdforrester-WMF, jijiki, aaron, Bawolff, Aklapper, thcipriani, hashar, Marostegui, NavinRizwi, Prondubuisi, eprodromou, Iflorez, darthmon_wmde, alaa_wmde, 94rain, DannyS712, Nandana, Analytics.mediafiles, kostajh, Jony, Force_Radical, Lahi, Gq86, Xover, GoranSMilovanovic, Vexations, Jayprakash12345, QZanden, LawExplorer, Tshrinivasan, _jensen, rosalieper, Agabi10, Taiwania_Justo, Scott_WUaS, Pchelolo, Jonas, Info-farmer, Wikidata-bugs, aude, Candalua, Huji, Lydia_Pintscher, Tpt, Mbch331, Rxy, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T183993: Investigate & Fix "According to a SiteLinkLookup Q47013093 is linked to frwiki while it is not or it does not exist" in logs
aaron added a comment. In T183993#5801637 <https://phabricator.wikimedia.org/T183993#5801637>, @Addshore wrote: >> If they are more than one case ( I checked and it seems it's two cases in the past 7 days) We need to make sure tools or Wikibase itself put some time between edits (I think this is made using Wikibase itself) We should ask user what they used that made this happen. > > Shouldn't chronology protector help us here? It probably is relevant, though without looking at the code or API request pattern in question, I can't say whether anything is wrong off hand. Does these errors occur to users who themsleves just made an edit or is it other users attempting to access something recently edited by someone else? Are there relevant uses of WANObjectCache that do not use getCacheSetOptions()? TASK DETAIL https://phabricator.wikimedia.org/T183993 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Addshore, aaron Cc: aaron, Krinkle, Catrope, daniel, Mbch331, Lucas_Werkmeister_WMDE, WMDE-leszek, Ladsgroup, Addshore, Aklapper, darthmon_wmde, Nandana, Lahi, Gq86, Pablo-WMDE, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Scott_WUaS, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Jdforrester-WMF, Rxy, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T227758: [investigate] purging strategy
aaron added a comment. Note that CdnCacheUpdate queues a purge to happen X seconds later to help deal with lag (mediawiki-config has $wgCdnReboundPurgeDelay at 11). If lag gets near that amount, then $wgCdnMaxageLagged will kick in. TASK DETAIL https://phabricator.wikimedia.org/T227758 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Michael, aaron, Lucas_Werkmeister_WMDE, Addshore, Pablo-WMDE, Aklapper, Lydia_Pintscher, darthmon_wmde, DannyS712, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Closed] T221577: Wikimedia\Rdbms\LBFactory::getEmptyTransactionTicket: LinksUpdate does not have outer scope
aaron closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T221577 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: kchapman, DannyS712, WMDE-leszek, alaa_wmde, Ladsgroup, Lydia_Pintscher, daniel, akosiaris, JTannerWMF, Krinkle, Jdforrester-WMF, Marostegui, aaron, TerraCodes, Liuxinyu970226, Aklapper, jcrespo, E.S.A-Sheild, darthmon_wmde, Premeditated, joker88john, CucyNoiD, Nandana, NebulousIris, Banyek, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, V4switch, LawExplorer, Vali.matei, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Jonas, Wong128hk, Niharika, Wikidata-bugs, matthiasmullie, aude, Dinoguy1000, Fabrice_Florin, MaxSem, Matanya, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T212550: Implement support for ChronologyProtection in RDF export
aaron added a comment. In T212550#5124323 <https://phabricator.wikimedia.org/T212550#5124323>, @Smalyshev wrote: > @aaron Is there any docs how to use the client ID on the client side? I see there's support for `cpPosIndex` cookie which is `$index@$time#$clientId` but if I have only client ID that is not going to help me. So I am not sure how would one use it - I can't find any trace of `ChronologyClientId` HTTP header. Right, I'll patch Setup.php to use the header if cpPosIndex does not already contain it via the cookie. I thought it did this already, but apparently not (probably since nothing uses that pattern yet). TASK DETAIL https://phabricator.wikimedia.org/T212550 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Krinkle, Ottomata, Aklapper, Smalyshev, aaron, alaa_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Vali.matei, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T212550: Implement support for ChronologyProtection in RDF export
aaron added a comment. In T212550#5124280 <https://phabricator.wikimedia.org/T212550#5124280>, @Krinkle wrote: > Another question for @aaron as well - If we start storing this chronologyprotector field in kafka etc. that means it can survive longer and no longer has an expiry like we do for cookies. I vaguely recall an issue in the past where clients that kept the value too long caused subsequent requests to stall for many seconds as MW mistook the unknown value for a value that hasn't appeared yet. > > It may've been mitigated already, but if not, we'll need to sort that out before this goes live. That was cpPosIndex for cookies, which was changed to include timestamp/clientId so that old values are ignored server-side. Use of client IDs alone will never cause wait delay on the position store itself, which is fine for this sort of EventBus use case; the DB changes and kafka enqueue happen in the master DC and the jobs run in the master DC as well, so there is no need to wait on cross-DC position store replication. By not doing so, it also avoids any risk of weird cases causing wait timeout delay. TASK DETAIL https://phabricator.wikimedia.org/T212550 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Krinkle, Ottomata, Aklapper, Smalyshev, aaron, alaa_wmde, joker88john, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Imarlier, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, Lucas_Werkmeister_WMDE, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, merbst, LawExplorer, Vali.matei, WSH1906, Lewizho99, Maathavan, _jensen, rosalieper, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T212550: Implement support for ChronologyProtection in RDF export
aaron added a comment. I think you can just add a method, similar to LBFactory::getChronologyProtectorTouched, that exposes the client ID, maybe call it LBFactory::getChronologyProtectorClientId(); TASK DETAIL https://phabricator.wikimedia.org/T212550 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Aklapper, Smalyshev, aaron, alaa_wmde, Nandana, Lahi, Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst, LawExplorer, _jensen, rosalieper, Jonas, Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T124196: Notice: Unable to unserialize job_params in some CirrusSearch jobs (when using JobQueueDB)
aaron added a comment. Why does the job itself have all of the transformed text rather than just a revision/page ID and use them to derive the transformed text? I get that some metadata is not stored elsewhere and would have to go in the job. TASK DETAIL https://phabricator.wikimedia.org/T124196 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, Krinkle, Rudloff, Physikerwelt, GFXDude2010, Zoglun, hoo, aude, Aklapper, alaa_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Avner, Gehel, _jensen, rosalieper, FloNight, Wikidata-bugs, jayvdb, Mbch331, Jay8g, jeremyb ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T218197: CacheAwarePropertyInfoStore & CachingPropertyInfoLookup should use WANObjectCache instead of BagOStuff
aaron added a comment. Yes, and CacheAwarePropertyInfoStore should use delete() or such for purges rather than set(). Using set() would only effect one DC. TASK DETAIL https://phabricator.wikimedia.org/T218197 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, Aklapper, elukey, Addshore, alaa_wmde, Nandana, Lahi, Gq86, GoranSMilovanovic, QZanden, LawExplorer, _jensen, rosalieper, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194299: Lock wait timeout exceeded in SqlIdGenerator::generateNewId
aaron added a comment. That sounds right.TASK DETAILhttps://phabricator.wikimedia.org/T194299EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Addshore, aaronCc: Fnielsen, Lucas_Werkmeister_WMDE, Ladsgroup, aaron, TerraCodes, Stashbot, Banyek, Paladox, mmodell, daniel, Addshore, Aklapper, gerritbot, hoo, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, _jensen, D3r1ck01, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Jdforrester-WMF, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T93142: [Task] Look into Wikibase use of memcached to see what needs broadcasted purges
aaron added a comment. I see EntityRevisionCache and CacheAwarePropertyInfoStore seems to use set() on invalidation. Also, EntityRevisionCache and CachingEntityRevisionLookup, and PopulateInterwiki seems to call delete() on a non-WAN cache instance.TASK DETAILhttps://phabricator.wikimedia.org/T93142EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Addshore, aaron, daniel, JanZerebecki, Lydia_Pintscher, aude, Aklapper, Nandana, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, LawExplorer, Vali.matei, _jensen, D3r1ck01, Volker_E, Wikidata-bugs, GWicke, Dinoguy1000, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194299: Lock wait timeout exceeded in SqlIdGenerator::generateNewId
aaron added a comment. In T194299#4714630, @daniel wrote: In T194299#4714614, @aaron wrote: openConnection is badly named and still reuses connections. You'd probably want getConnection with CONN_TRX_AUTO I hate this hack. This may *still* re-use connections, if anything else used CONN_TRX_AUTO. We should have CONN_NEW. Reuse is fine here, it's just confusing for something called "openConnection". CONN_TRX_AUTO will just reuse other CONN_TRX_AUTO connections.TASK DETAILhttps://phabricator.wikimedia.org/T194299EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: aaron, TerraCodes, Stashbot, Banyek, Paladox, mmodell, daniel, Addshore, Aklapper, gerritbot, hoo, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Jdforrester-WMF, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T194299: Lock wait timeout exceeded in SqlIdGenerator::generateNewId
aaron added a comment. openConnection is badly named and still reuses connections. You'd probably want getConnection with CONN_TRX_AUTOTASK DETAILhttps://phabricator.wikimedia.org/T194299EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: aaron, TerraCodes, Stashbot, Banyek, Paladox, mmodell, daniel, Addshore, Aklapper, gerritbot, hoo, CucyNoiD, Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, lisong, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Jdforrester-WMF, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T200420: Wikidata dispatching stuck
aaron added a comment. Ah, right, I read that ternary backwards, <<$maxTime < PHP_INT_MAX ? PHP_INT_MAX : 1>>.TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Nikki, zeljkofilipin, aaron, gerritbot, Stashbot, WMDE-leszek, Addshore, TerraCodes, Liuxinyu970226, matej_suchanek, Aklapper, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T200420: Wikidata dispatching stuck
aaron added a comment. In T200420#4453134, @Addshore wrote: Something to note, because the locks are no longer in the DB, we end up selecting the same 15 or so wikis that are locked all of the time. It could be that the other wikis actually don't have locks: before using the redis lock manager the status of the lock from the db was also in the select so that locked dbs would not be selected at all. You mean the condition on chd_touched in getCandidateClients()? That would roughly correlate with lock state. It's odd that engageClientLock() is blocking for the SQL coordinator but not the LockManager-using subclass (Database::lock is 5s by default). That would make the LM-based selectClient() more likely to not find anything and return false (if there are a few busy wikis), as it only does one pass with no usleep(). Since dispatchMaxTime is not PHP_INT_MAX (but 720) and I don't see --max-passes in puppet, then max-passes for dispatchChanges.php runs is 1, right? Thus, once selectClient() returns false, then dispatchChanges.php would exit afaik. The loop/delay (idle-delay) logic would not happen in such cases.TASK DETAILhttps://phabricator.wikimedia.org/T200420EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: zeljkofilipin, aaron, gerritbot, Stashbot, WMDE-leszek, Addshore, TerraCodes, Liuxinyu970226, matej_suchanek, Aklapper, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Lewizho99, Maathavan, Jonas, Wikidata-bugs, aude, Lydia_Pintscher, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T182322: ChronologyProtector breaks if two requests write different sets of databases
aaron added a comment. Yes, https://gerrit.wikimedia.org/r/396546 .TASK DETAILhttps://phabricator.wikimedia.org/T182322EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Johan, Darwinius, Aklapper, Lea_Lacroix_WMDE, Addshore, Ladsgroup, Joe, Trizek-WMF, jhsoby, Envlh, Framawiki, daniel, hoo, Simon_Villeneuve, Manu1400, MisterSynergy, Fnielsen, Jay8g, TerraCodes, aude, aaron, Krinkle, Jonas, thiemowmde, Lucas_Werkmeister_WMDE, Steinsplitter, Anvilaquarius, Anomalocaris, Sjoerddebruin, Imarlier, GerritBot, MusikAnimal, Adrian1985, Cpaulf30, Rayssa-, Lahi, Gq86, Baloch007, Darkminds3113, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, LawExplorer, Vali.matei, Lewizho99, Maathavan, Luke081515, Wikidata-bugs, TheDJ, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007
aaron added a comment. In T181385#3806113, @gerritbot wrote: Change 394779 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz): [mediawiki/core@master] Try to opportunistically flush statsd data in maintenance scripts https://gerrit.wikimedia.org/r/394779 This seems more general and preferable than the other patch, but leaving both here to look at.TASK DETAILhttps://phabricator.wikimedia.org/T181385EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: gerritbot, Legoktm, Addshore, Krinkle, demon, TerraCodes, Jay8g, Liuxinyu970226, Stashbot, Aklapper, daniel, Lydia_Pintscher, ArielGlenn, aude, hoo, Imarlier, Lahi, Gq86, Baloch007, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Maathavan, Wikidata-bugs, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T181385: Wikidata entity dumpers stuck with 100% CPU on snapshot1007
aaron added a comment. How long do these run? The sample rate in config is set to be extremely low. So perhaps: The buffering class buffers things that won't even be saved The buffering could be disable in CLI mode TASK DETAILhttps://phabricator.wikimedia.org/T181385EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: gerritbot, Legoktm, Addshore, Krinkle, demon, TerraCodes, Jay8g, Liuxinyu970226, Stashbot, Aklapper, daniel, Lydia_Pintscher, ArielGlenn, aude, hoo, Imarlier, Lahi, Gq86, Baloch007, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Maathavan, Wikidata-bugs, Svick, Mbch331, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173696: Cache constraint check results
aaron added a comment. Probably hotTTR is way to high. It's really "expected time till refresh given 1 hit/sec". With 50/min, you'd get maybe 2 updates (new values) per regex. I'll put up a patch for that.TASK DETAILhttps://phabricator.wikimedia.org/T173696EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: WMDE-leszek, aaronCc: Krinkle, aaron, gerritbot, Ladsgroup, daniel, Aklapper, Jonas, Lucas_Werkmeister_WMDE, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173696: Cache constraint check results
aaron added a comment. In T173696#3696294, @Lucas_Werkmeister_WMDE wrote: I did a bunch of requests against https://www.wikidata.org/w/api.php?action="">, which checks a format constraint for “title”. It’s always the same regex and only a handful of different values (17). But while I could see a sharp rise in requests in Grafana corresponding to the times when I sent those requests (permalink), most of them are still cache misses. I’m not sure how to interpret that – it seems values aren’t entering the cache map very often? What was the request rate and time window?TASK DETAILhttps://phabricator.wikimedia.org/T173696EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: WMDE-leszek, aaronCc: Krinkle, aaron, gerritbot, Ladsgroup, daniel, Aklapper, Jonas, Lucas_Werkmeister_WMDE, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173696: Cache constraint check results
aaron added a comment. In T173696#3690945, @Lucas_Werkmeister_WMDE wrote: Reopening. This task is supposed to be for caching results in general, which isn’t done yet at all, though we had a lot of discussion on caching regex checks specifically here, which in hindsight should’ve been in a separate task. Also, IMO the regex caching isn’t done yet, since the Grafana stats are pretty unsatisfactory. (Perhaps we should repurpose this task to be just about regex checking, open a new one for general caching, and reshuffle the parent tasks so that this one is a child of the new task?) The cache hit rate will be low if their is low traffic (as of now with a few people using a JS tool), since not much will get populated and items will expire. Something like increasing the cache set() rate as the last-modified time of the cache entry increases would work. It would also result in much more entries stored that might not get a lot of use though.TASK DETAILhttps://phabricator.wikimedia.org/T173696EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: WMDE-leszek, aaronCc: Krinkle, aaron, gerritbot, Ladsgroup, daniel, Aklapper, Jonas, Lucas_Werkmeister_WMDE, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T72710: StorageException in EditEntityActionTest::testActionForPage (edit-already-exists) and related failures
aaron closed subtask T42451: "Transaction already in progress" error in sqlite as "Resolved". TASK DETAILhttps://phabricator.wikimedia.org/T72710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Lydia_Pintscher, aaronCc: gerritbot, Wikidata-bugs, aude, Lydia_Pintscher, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Izno, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173696: Cache constraint check results
aaron added a comment.
In T173696#3620700, @Lucas_Werkmeister_WMDE wrote:
Interesting idea! It feels a bit weird to implement logic like this on top of the cache (I thought that’s the cache’s job?), but you’re the expert :) it sounds like it makes a lot of sense, at least, since the set of regexes is mostly static and the set of values is highly dynamic, with some very commonly used values.
I think I’ll remove the “don’t bother” microtime check, though, since it seems that even for an extremely simple query like SELECT (1 AS ?x) {}, the query service rarely responds in less than 0.04 seconds, and never in less than 0.02 seconds (tested from a Cloud VPS system within the Eqiad cluster).
That seems to make sense.
@aaron do you mind if I make that a separate commit and credit you as the git commit author?
Sure.
One thing that could also be tweaked in that code is to track the last timestamp an entry was touched and prune ones older than a certain number of days. For busy regexes, that stops long-tail cruft from accumulating and reduce memcached I/O in bytes. Another thing to tune would be the max size.TASK DETAILhttps://phabricator.wikimedia.org/T173696EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Krinkle, aaron, gerritbot, Ladsgroup, daniel, Aklapper, Jonas, Lucas_Werkmeister_WMDE, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173696: Cache constraint check results
aaron added a comment.
If want to avoid flooding cache with rarely used long-tail combinations, maybe something like this could be done:
$textHash = hash( 'sha256', $text );
$cacheMap = $this->cache->getWithSetCallback(
$this->cache->makeKey(
'WikibaseQualityConstraints', // extension
'regex', // action
'WDQS-Java', // regex flavor
hash( 'sha256', $regex )
),
WANObjectCache::TTL_DAY,
function ( $curCacheMap ) use ( $text, $regex, $textHash ) {
// Initialize the cache map if not set
if ( $curCacheMap === false ) {
return [];
}
// Refresh triggered by hotTTR...
// Add regex-text check result to top of the LRU map if present
if ( isset( $curCacheMap[$textHash] ) ) {
return [ $textHash => $curCacheMap[$textHash] ] + $curCacheMap;
}
// Get the regex-text check result
$value = (int)$this->matchesRegularExpressionWithSparql( $text, $regex );
// Add regex-text check to the bottom 3/8s of the LRU map
$index = intval( count( $curCacheMap ) * 5/8 );
$newCacheMap = [];
$pos = 0;
foreach ( $curCacheMap as $k => $v ) {
if ( $pos == $index ) {
$newCacheMap[$textHash] = $value; // inject
++$pos;
}
if ( $pos++ >= 100 ) {
break; // prune to size
}
$newCacheMap[$k] = $v; // preserve
}
return $newCacheMap;
},
[
// Once map is > 1 sec old, consider refreshing
'ageNew' => 1,
// Increase likely of refresh to certainty once 1 minute old;
// the most common keys are more likely to trigger this
'hotTTR' => 60,
// avoid querying cache servers multiple times in a request
'pcTTL' => $cache::PROC_TTL_LONG
]
);
$value = isset( $cacheMap[$textHash] )
? $cacheMap[$textHash]
: (int)$this->matchesRegularExpressionWithSparql( $text, $regex );TASK DETAILhttps://phabricator.wikimedia.org/T173696EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Krinkle, aaron, gerritbot, Ladsgroup, daniel, Aklapper, Jonas, Lucas_Werkmeister_WMDE, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Lewizho99, Maathavan, Agabi10, Izno, Wikidata-bugs, aude, Mbch331___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. Those refreshLInks jobs (from wikibase) are the only ones that use multiple titles per job, so they will be a lot slower (seems to be 50 pages/job) than the regular ones from MediaWiki core. That is a bit on the slow side for a run time of a non-rare job type (e.g. TMH or GWT).TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Nikerabbit, Mholloway, Legoktm, ema, Joe, GWicke, Nemo_bis, Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Eevans, mobrovac, Hardikj, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. In T173710#3571046, @EBernhardson wrote: In T173710#3571009, @Legoktm wrote: Could we always bump page_touched, but only send the purges to varnish if the timestamp is within the past four days? Would that let us run the older jobs faster since if I understand correctly the throttling is to avoid overloading varnish with purges? Unfortunately the throttling still happens regardless of page touched. Throttling isn't based on actual purges performed but on the number of work items in a job. Work items are a simple count of pages in the job, rather than how many pages will actually be purged. Changing this behavior would basically increase the number of purges we send to varnish. Seems simple enough to make a Job::getEffectedWorkItems(), defaulting to getWorkItems() but updated by the backlink jobs during run(). The getBackoffTimeToWait() call in JobRunner could use the effected count and be moved down a bit, after the Job::run() call.TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Legoktm, ema, Joe, GWicke, Nemo_bis, Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, Vali.matei, Avner, Zppix, debt, Gehel, FloNight, Izno, Eevans, mobrovac, Hardikj, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. In T173710#3570037, @Joe wrote: Correcting myself after a discussion with @ema: since we have up to 4 cache layers (at most), we should process any job with a root timestamp newer than 4 times the cache TTL cap. So anything older than 4 days should be safely discardable. This would account for about 1% of jobs according to Gwicke's sampling, but I suspect that under large pressure the distribution could get significantly worse. I'd be careful about parser cache and an extensions using page_touched to validated cached values. Discarding jobs might break some assumptions there.TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: ema, Joe, GWicke, Nemo_bis, Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune, Vali.matei, Avner, Zppix, debt, Gehel, FloNight, Izno, Eevans, mobrovac, Hardikj, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment.
As far as retries go, the attempts hash for wikidatawiki:htmlCacheUpdate has few entries with run counts no greater than 3. The onl incrementing code is doPop() in MediaWiki, the same code that made them go up to 3 to begin with. If the same job ran many times, I'd expect there to be very high values there.
> aaron@terbium:~$ mwscript eval.php wikidatawiki
>
> error_reporting( E_ALL );
> require("/home/aaron/eval_job_check.php");
> foreach ( $wmfLocalServices['jobqueue_redis'] as $tag => $host ) { sanityCheckJQHost( $host, wfWikiId(), 'htmlCacheUpdate' ); }
array(6) {
["743f54ce7b8843d8b6e4ec081f633508"]=>
string(1) "3"
["ee20490772484aae905592ce6a4bc22c"]=>
string(1) "3"
["a45d1c46edc8450a90da89668cbe1924"]=>
string(1) "3"
["0083c49d9dec492d99ee7ea95ab25403"]=>
string(1) "3"
["b1f4cb9f1b9c4402b9f8da2348d6a46f"]=>
string(1) "3"
["2edd120f3b1a42edb3645d2dd777bf81"]=>
string(1) "3"
}
array(3) {
["65d41242504d4e4198b1213da1d3536c"]=>
string(1) "3"
["c2ceaffe86274a56b3b491899e3e3594"]=>
string(1) "3"
["f38d9c0116e7438b9c8d9a8ae6f9430e"]=>
string(1) "3"
}
array(3) {
["720afb9160b542b896820a8d069910c2"]=>
string(1) "3"
["3407d8dd224840c2bf79c36b55bc311a"]=>
string(1) "3"
["1f67fd5e59914a4686bee0877c4b935f"]=>
string(1) "3"
}
array(2) {
["9aa931c3f3444cc0bd9bfa8ff3097062"]=>
string(1) "3"
["a5bc6d9346f84a87ad4829edf096b977"]=>
string(1) "3"
}
array(1) {
["46677062e9e74d048541f1b8dab3c63a"]=>
string(1) "3"
}
array(3) {
["45b63ee504dd4f798956f6900079f452"]=>
string(1) "3"
["7992a032bebf45b9a686991dc29a24b4"]=>
string(1) "3"
["935f44b5c3d64dd392f29e8e8e94963b"]=>
string(1) "3"
}
array(3) {
["ef36284c42da45cfa667419c820d17c6"]=>
string(1) "3"
["6803b9c714b545a59d1830c5ab55ec60"]=>
string(1) "3"
["832aa8f83c1f475dabc56e256f22ea84"]=>
string(1) "3"
}
array(2) {
["dc12814a0b214c6d94f054aca4201115"]=>
string(1) "3"
["60fc4b1e6b354189982add7dfabccf25"]=>
string(1) "3"
}
array(4) {
["935610fad21c4d2eb8336cb594f57afb"]=>
string(1) "3"
["0a4bfacccd8d48258f8b7689b99f3180"]=>
string(1) "3"
["a381aed77ce94bec872aaebf8b96016b"]=>
string(1) "3"
["e8b9c16c9d3848c38ab0c44556a7d2e4"]=>
string(1) "3"
}
array(4) {
["6f4dd16a084d486dab52658a4ea54c37"]=>
string(1) "3"
["0e7f6a92e6eb4bb8a121047f869c3f6e"]=>
string(1) "3"
["fd18564d792f4d9f82f45a1e42c46973"]=>
string(1) "3"
["3b786e6a7dfd4f2fb4a9f924f160fcba"]=>
string(1) "3"
}
array(4) {
["0b13d238fa554706a08a5b2160a66e1e"]=>
string(1) "3"
["8c35814276f04985b7158081acfb8dbf"]=>
string(1) "3"
["1c3accd2123a4159aa7ee2e95628ad29"]=>
string(1) "3"
["d5a4d5bb391d4192ab1af5a9caee9f46"]=>
string(1) "3"
}
array(3) {
["ce5df11aaecc4bf9a641787c9bc41e9e"]=>
string(1) "3"
["16bd168b60e348bfab39e7a8921a99a1"]=>
string(1) "3"
["93ff062e5b00463e9efcda7604274112"]=>
string(1) "3"
}
A page with 1 million backlinks would have (job divisions)x(leaf jobs per division) + (jobs that just divide into other jobs) = ( 1e6 / 300 * 3 + 1e6 / 300 = ~13334 job runs (if none failed), and they would all have the same rootJobTimestamp. The number of jobs with the same minute prefix would be higher (different rootJobSignature values though). The only thing odd about the table @GWicke posted is how old some root job descendants are.
Since job divisions go to the end of the queue (like any other job pushed), it will make it trickier to reason about timing. The oldest job in the queue might be to a page with a lot of backlinks. Each division puts the leaf and remnant (the one to divide) jobs at the end of the queue. The runners have to burn through the queue to get to the remnant job. This cycle repeats until it's done. When the queue has any serious length, this means it might take a long time to finish some old template backlink refresh/purge. During the increase, jobs kept piling up, meaning each iteration of old many-backlink job would take a long time to even get to the next division, stretching it out further than just a continuous one-off backlog of b
[Wikidata-bugs] [Maniphest] [Commented On] T174422: Make dbBatchSize in WikiPageUpdater configurable
aaron added a comment. In T174422#3566350, @Krinkle wrote: @Ladsgroup @aaron Would $wgUpdateRowsPerQuery be appropiate here, too? Or is it important for this particular query to use a different batch size? Most of the pure waiting in the job will be for replication (the throttling just makes the worker request finish and the slot go to another wiki). It seems worth considering to just use that setting.TASK DETAILhttps://phabricator.wikimedia.org/T174422EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, aaronCc: Aklapper, fgiunchedi, aaron, Krinkle, Jdforrester-WMF, WMDE-leszek, jcrespo, EBernhardson, gerritbot, Emijrp, Mr.Ibrahem, Magnus, Sjoerddebruin, Bugreporter, Pasleim, XXN, Harej, Daniel_Mietchen, Agabi10, Stashbot, daniel, Liuxinyu970226, Peachey88, BBlack, Andreasmperu, Nemo_bis, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Maathavan, Izno, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. Though this bit is problematic: "page_touched < " . $dbw->addQuotes( $dbw->timestamp( $touchTimestamp ) ) ...seems like that comparison should use rootJobTimestamp if present.TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. Ignored purges still count as work items, yes. Rebound purges could explain some of the number. Also, given the backlog, lots of them probably had actually different rootJobTimestamps. MediaWiki can de-duplicate those when it's the same backlinked page X being edited several times by ignoring the older timestamp ones. It's trickier when templates A and B are edited and the backlinks overlap. Sometimes that gets caught, other times both purges to same page happen. If htmlCacheUpdate queue was LIFO instead of FIFO, then the higher timestamp purges would run first more often and the lower ones would no-op given the SELECT query...that might be where the most de-duplication opportunities are missed. It mostly relies on non-parallel execution of jobs causing the range->root job division, and leaf job execution for different template/file edits to be intertwined. Whether the job with the higher rootJobTimestamp runs first or vise versa is luck based. When it's the former, then the purge is de-duplicated on the DB/CDN layer. Making that queue LIFO would nullify the rootJobSignature/timestamp deduplication however (e.g. several edits to template A). I guess visually, the limitations on per-page deduplication is like: Edit to A (t1): Queue: JobA, [tail: left, head:right] Edit to B (t2): Queue: JobB, , JobA, As jobs run: Queue: JobAremnant,JobAleaf1, ..., JobAleaf500, , , Queue: JobBremnant,JobBleaf1, ..., JobBleaf500, , JobAremnant,JobAleaf1, ..., JobAleaf500, So the page A jobs from t1 run and *then* later the B jobs from t2. This tends to repeat as the remnant jobs divide up info leaf jobs. Any common pages in those leaf jobs will likely have page_touched hit twice (first t1 and then t2). The queue doesn't "know" that a later job will touch some of the same pages with a higher value, obviating the need for the first purges (aside from avoiding purge starvation in pathological cases).TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. Note that for de-duplication, as long as the job has rootJobTimestamp set, it will ignore rows already touched (page_touched) to a higher/equal value, and likewise not send purges to the corresponding pages. So the CDN aspects *should* already have lots of de-duplication, the job spam notwithstanding.TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Andreasmperu, BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. In T173710#3551156, @aaron wrote: Secondary purges where for dealing with replication lag scenarios, not lost purges. That was one extra purge (2X). One easy change I can see to not use CdnCacheUpdate from HtmlCacheUpdateJob (but still for the pages directly being edited). There is already processing delay anyway (and if there is none, there less likely to be replag, though not guaranteed), so there is less "de facto" use in a secondary purge for backlinks. That said, there is still some extension or underlying user pattern that I suspect is the underlying cause. Sacrificing rebound purges will cut purges in half, and it's easy to do, hence my patch above.TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. Secondary purges where for dealing with replication lag scenarios, not lost purges. That was one extra purge (2X). One easy change I can see to not use CdnCacheUpdate from HtmlCacheUpdateJob (but still for the pages directly being edited). There is already processing delay anyway (and if there is none, there less likely to be replag, though not guaranteed), so there is less "de facto" use in a secondary purge for backlinks.TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: BBlack, Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. In T173710#3548223, @daniel wrote: In T173710#3547580, @aaron wrote: In other words, base jobs for entities that will divide up and purge all backlinks to the given entity. Note that each job has two entries. Wait - each job has two entries? You mean, there are duplicates inserted, and not pruned?... No, just two entries in the screen output of that script (STARTING and DONE).TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Peachey88, Liuxinyu970226, daniel, Stashbot, Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T173710: Job queue is increasing non-stop
aaron added a comment. From mwscript maintenance/runJobs.php wikidatawiki --type htmlCacheUpdate --nothrottle --maxjobs 100 | grep "IsSelf=1" I can see almost all of the jobs are things like: 2017-08-24 01:15:39 htmlCacheUpdate Q36985371 table=pagelinks recursive=1 rootJobIsSelf=1 rootJobSignature=904df933392e17eb9d3b70fb34b393ce7e24c4be rootJobTimestamp=20170817131048 requestId=WZWV1gpAANEAACEvJp4AAABK (uuid=fdabaff29f96432fbb7b538162406ede,timestamp=1502975448,QueuePartition=rdb1-6381) In other words, base jobs for entities that will divide up and purge all backlinks to the given entity. Note that each job has two entries. Looking at the ratio of base jobs from the first X jobs run via script, I get: aaron@terbium:~$ mwscript maintenance/runJobs.php wikidatawiki --type htmlCacheUpdate --nothrottle --maxjobs 1000 | grep "IsSelf=1" | wc -l 1980 Which is 990/1000 jobs from the run. Even 10,000 yeilds 19844, so 9922/1000 jobs.TASK DETAILhttps://phabricator.wikimedia.org/T173710EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Agabi10, Daniel_Mietchen, Harej, XXN, Pasleim, Bugreporter, Sjoerddebruin, Magnus, Mr.Ibrahem, Emijrp, gerritbot, Reedy, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Project Column] T173710: Job queue is increasing non-stop
aaron moved this task from Inbox to Radar on the Performance-Team board.aaron edited projects, added Performance-Team (Radar); removed Performance-Team. TASK DETAILhttps://phabricator.wikimedia.org/T173710WORKBOARDhttps://phabricator.wikimedia.org/project/board/1212/EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: gerritbot, Reedy, EBernhardson, Esc3300, jcrespo, WMDE-leszek, Jdforrester-WMF, Krinkle, aaron, fgiunchedi, Aklapper, Ladsgroup, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, EBjune, Vali.matei, Avner, Lewizho99, Zppix, Maathavan, debt, Gehel, FloNight, Izno, Wikidata-bugs, aude, jayvdb, faidon, Mbch331, Jay8g, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis
aaron added a comment. I commented on the patch, it's a METHOD mismatch problem, so the commit/wait steps don't happen (just the one big one like before).TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, aaronCc: PokestarFan, Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis
aaron added a comment. In T164173#3420723, @daniel wrote: @aaron another question: does RefreshLinksJob also purge the CDN cache automatically? should it? It does update the parser cache... It saves the cache as a convenience in some cases (since the relevant htmlCacheUpdate job uses the rootJobTimestamp in setting page_touched, so some cache misses can be avoided by doing so). The only CDN purges refreshLinks does is minor thinks like file description pages (since they list usage links).TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: daniel, aaronCc: Agabi10, gerritbot, Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, Lordiis, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Vali.matei, Lewizho99, Minhnv-2809, Zppix, Maathavan, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis
aaron added a comment. I also wonder why some of those log warnings come from close() and others have the proper commitMasterChanges() bit in the stack trace. Normally, there should be nothing to commit by close() and it is just commits for sanity.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Krinkle, aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unassigned] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis
aaron removed aaron as the assignee of this task. TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: aaron, MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis
aaron added a comment.
In T164173#3343495, @aaron wrote:
@daniel , can you look into the amount of purges happening in ChangeNotification jobs? I don't see any throttling or lag checks on in the job code.
/rpc/RunJobs.php?wiki=commonswiki=ChangeNotification=60=300M
Expectation (maxAffected <= 500) by JobRunner::run not met (actual: 89936):
[transaction f7dd6f writes to 10.64.48.23 (commonswiki)]
#0 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/TransactionProfiler.php(294): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/database/Database.php(2888): Wikimedia\Rdbms\TransactionProfiler->transactionWritingOut()
#2 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/database/Database.php(756): Wikimedia\Rdbms\Database->commit()
#3 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1084): Wikimedia\Rdbms\Database->close()
#4 (): Closure$Wikimedia\Rdbms\LoadBalancer::closeAll()
#5 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1485): call_user_func_array()
#6 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1085): Wikimedia\Rdbms\LoadBalancer->forEachOpenConnection()
#7 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1076): Wikimedia\Rdbms\LoadBalancer->closeAll()
#8 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1665): Wikimedia\Rdbms\LoadBalancer->disable()
#9 (): Wikimedia\Rdbms\LoadBalancer->__destruct()
#10 {main}
If those are page_touched updates, that would a pretty large number of purges to make in one job at once.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: MZMcBride, daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Subscribers] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis
aaron added a subscriber: daniel.aaron added a comment. @daniel , can you look into the amount of purges happening in ChangeNotification jobs? I don't see any throttling or lag checks on in the job code.TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: daniel, Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Triaged] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis
aaron triaged this task as "Normal" priority. TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Ladsgroup, hoo, Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T164173: Cache invalidations coming from the JobQueue are causing lag on several wikis
aaron added a project: Wikidata. TASK DETAILhttps://phabricator.wikimedia.org/T164173EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Marostegui, Aklapper, jcrespo, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Minhnv-2809, Zppix, Izno, Luke081515, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, Krenair, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unassigned] T124418: Investigate massive increase in htmlCacheUpdate jobs in Dec/Jan
aaron removed aaron as the assignee of this task. TASK DETAILhttps://phabricator.wikimedia.org/T124418EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: GWicke, ArielGlenn, Krinkle, Peter, EBernhardson, Smalyshev, gerritbot, Legoktm, daniel, hoo, aude, Lydia_Pintscher, JanZerebecki, MZMcBride, Luke081515, aaron, faidon, Joe, ori, BBlack, Aklapper, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Vali.matei, Zppix, Izno, Wikidata-bugs, Mbch331, Jay8g, fgiunchedi___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T154596: SpecialModifyEntity master queries on page views
aaron edited projects, added Wikidata; removed WMF-deploy-2016-10-25_(1.28.0-wmf.23), WMF-deploy-2016-09-13_(1.28.0-wmf.19), MW-1.28-release-notes. TASK DETAILhttps://phabricator.wikimedia.org/T154596EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Joe, Krenair, PleaseStand, gerritbot, Aklapper, Gilles, Nemo_bis, MZMcBride, Glaisher, Johan, daniel, Amire80, Liuxinyu970226, aaron, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g, akosiaris___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T151993: Implement ChangeDispatchCoordinator based on RedisLockManager
aaron added a comment. If the owner fatals, the lock will have to expire (the TTL depends on the LockManager instance config and/or whether the context is CLI or web).TASK DETAILhttps://phabricator.wikimedia.org/T151993EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, aaronCc: gerritbot, Ladsgroup, hoo, Lydia_Pintscher, Jonas, Aklapper, jcrespo, Marostegui, daniel, aaron, Th3d3v1ls, Ramalepe, Liugev6, Vali.matei, Lewizho99, Minhnv-2809, Maathavan, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GWicke, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T151993: Implement ChangeDispatchCoordinator based on RedisLockManager
aaron added a comment. There is no analogous method. Maybe a non-blocking engageClientLock() call can replace the isClientLockUsed() call? Seems like chd_lock is used to determine whether to do a non-blocking check first before a blocking acquisition (which could race anyway, which I guess just effects runtime?).TASK DETAILhttps://phabricator.wikimedia.org/T151993EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: Ladsgroup, aaronCc: gerritbot, Ladsgroup, hoo, Lydia_Pintscher, Jonas, Aklapper, jcrespo, Marostegui, daniel, aaron, Th3d3v1ls, Ramalepe, Liugev6, Vali.matei, Lewizho99, Minhnv-2809, Maathavan, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GWicke, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T151681: DispatchChanges: Avoid long-lasting connections to the master DB
aaron added a comment. There is also a flip-side to automatically dropping on connection loss, which is that loss can happen (possibly due to the net_wait_timeout options) while the connection to DBs actually being updated stays alive. In that case, multiple threads could run on the same client wiki. Not sure if that actually happens though, there would be reconnection events in logstash if it did.TASK DETAILhttps://phabricator.wikimedia.org/T151681EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: aaron, Marostegui, jcrespo, Aklapper, Jonas, Lydia_Pintscher, hoo, daniel, Vali.matei, Minhnv-2809, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GWicke, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T151681: DispatchChanges: Avoid long-lasting connections to the master DB
aaron added a comment. How often would locks be dropped? Using ScopedLock would handle exceptions in non-lock code. The shutdown handler usually catches SIGINT. I guess there are still fatal errors, though I'd hope that sort of thing would be rare. In that case, the redis lock manager used by our FileBackend instances could be reused.TASK DETAILhttps://phabricator.wikimedia.org/T151681EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: aaron, Marostegui, jcrespo, Aklapper, Jonas, Lydia_Pintscher, hoo, daniel, Vali.matei, Minhnv-2809, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GWicke, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T147169: Make sure Wikibase dump maintenance scripts solely use the "dump" db group
aaron added a comment. Does getLazyConnectionRef() help here?TASK DETAILhttps://phabricator.wikimedia.org/T147169EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: thiemowmde, Addshore, aaron, Aklapper, Smalyshev, Lydia_Pintscher, jcrespo, aude, daniel, hoo, D3r1ck01, Izno, Wikidata-bugs, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T148419: Patch adding caches to MediaWikiServices caused Cognate services to not correctly register
aaron added a comment. Are they are more useful/direct traces?TASK DETAILhttps://phabricator.wikimedia.org/T148419EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: WMDE-Fisch, Tobi_WMDE_SW, aaron, Addshore, Aklapper, D3r1ck01, Izno, Wikidata-bugs, aude, Darkdadaah, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T145819: Jobs invoking SiteConfiguration::getConfig cause HHVM to fail updating the bytecode cache due to being filesize limited to 512MBytes
aaron added a comment. It should be a goal to replace usage of the maintenance config script, but not high priority IMO.TASK DETAILhttps://phabricator.wikimedia.org/T145819EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: matmarex, thcipriani, Anomie, aaron, MZMcBride, Tobi_WMDE_SW, FastLizard4, JJMC89, zeljkofilipin, Lydia_Pintscher, daniel, aude, Addshore, Aklapper, greg, Legoktm, demon, gerritbot, Stashbot, hashar, Lewizho99, Maathavan, D3r1ck01, Liudvikas, Izno, Luke081515, Wikidata-bugs, ArielGlenn, JanZerebecki, Mbch331, Jay8g, Joe, jeremyb, mmodell___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T146529: 5-10 times more rows are currently read from the s5 master than from all other masters together
aaron added a comment.
Also seeing:
Expectation (masterConns <= 0) by ApiMain::setRequestExpectations not met:
[connect to 10.64.16.144 (wikidatawiki)]
#0 /srv/mediawiki/php-1.28.0-wmf.20/includes/libs/rdbms/TransactionProfiler.php(156): TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.28.0-wmf.20/includes/libs/rdbms/loadbalancer/LoadBalancer.php(573): TransactionProfiler->recordConnection()
#2 /srv/mediawiki/php-1.28.0-wmf.20/includes/dao/DBAccessBase.php(61): LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(175): DBAccessBase->getConnection()
#4 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(76): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->selectRevisionInformationMultiple()
#5 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(199): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->loadRevisionInformation()
#6 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(164): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->doFetch()
#7 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(84): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation()
#8 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(156): Wikibase\Lib\Store\WikiPageEntityRevisionLookup->getEntityRevision()
#9 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(138): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision()
#10 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(156): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision()
#11 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(138): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision()
#12 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/repo/includes/Api/GetEntities.php(255): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision()
#13 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/repo/includes/Api/GetEntities.php(237): Wikibase\Repo\Api\GetEntities->getEntityRevision()
#14 /srv/mediawiki/php-1.28.0-wmf.20/extensions/Wikidata/extensions/Wikibase/repo/includes/Api/GetEntities.php(129): Wikibase\Repo\Api\GetEntities->getEntityRevisionsFromEntityIds()
#15 /srv/mediawiki/php-1.28.0-wmf.20/includes/api/ApiMain.php(1427): Wikibase\Repo\Api\GetEntities->execute()
#16 /srv/mediawiki/php-1.28.0-wmf.20/includes/api/ApiMain.php(511): ApiMain->executeAction()
#17 /srv/mediawiki/php-1.28.0-wmf.20/includes/api/ApiMain.php(482): ApiMain->executeActionWithErrorHandling()
#18 /srv/mediawiki/php-1.28.0-wmf.20/api.php(83): ApiMain->execute()
#19 /srv/mediawiki/w/api.php(3): include()
#20 {main}TASK DETAILhttps://phabricator.wikimedia.org/T146529EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: aaron, Aklapper, jcrespo, Marostegui, Vali.matei, Minhnv-2809, D3r1ck01, Izno, Luke081515, Wikidata-bugs, aude, GWicke, Mbch331, Jay8g, Krenair___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T146079: s5-master contention caused (?) by refreshlinksprioritized/addUsagesForPage jobs running for all wikis
aaron added a comment. Does addUsages() get called when no other writes are pending commit? If so, you can do the usual getEmptyTransactionTicket/commitAndWaitForReplication dance. If not, you'd have to pass the ticked down from above...TASK DETAILhttps://phabricator.wikimedia.org/T146079EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: aaron, Lydia_Pintscher, daniel, aude, hoo, Aklapper, jcrespo, Marostegui, Minhnv-2809, D3r1ck01, Izno, Luke081515, Wikidata-bugs, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T145819: Jobs invoking SiteConfiguration::getConfig cause HHVM to fail updating the bytecode cache due to being filesize limited to 512MBytes
aaron added a comment. The backport should get the maintenance script call rate back to the old status quo (rare), so re-deploy is worth attempting.TASK DETAILhttps://phabricator.wikimedia.org/T145819EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: thcipriani, Anomie, aaron, MZMcBride, Tobi_WMDE_SW, FastLizard4, JJMC89, zeljkofilipin, Lydia_Pintscher, daniel, aude, Addshore, Aklapper, greg, Legoktm, demon, gerritbot, Stashbot, hashar, Lewizho99, Maathavan, D3r1ck01, Liudvikas, Izno, Luke081515, Wikidata-bugs, ArielGlenn, JanZerebecki, Mbch331, Jay8g, Joe, jeremyb, mmodell___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T142664: ChangeNotification writeQueryTime warnings
aaron created this task.aaron added a project: Wikidata.Herald added a subscriber: Aklapper. TASK DESCRIPTIONI keep seeing this in DBPerformance.log: Expectation (writeQueryTime <= 5) by JobRunner::run not met (actual: 5.3946626186371): [transaction 3d09beb22c0e writes to 10.64.16.30 (rowiki)] Maybe there are some updates that should be batched (with lag checks)...TASK DETAILhttps://phabricator.wikimedia.org/T142664EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Aklapper, aaron, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T140955: Wikibase\Repo\Store\WikiPageEntityStore::updateWatchlist: Automatic transaction with writes in progress (from DatabaseBase::query (LinkCache::addLinkObj)
aaron merged a task: T140967: WikiPageEntityStore::updateWatchlist breaks implicit transactions. TASK DETAILhttps://phabricator.wikimedia.org/T140955EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: hashar, aaron, hoo, Lydia_Pintscher, daniel, aude, jcrespo, Aklapper, TerraCodes, greg, Luke081515, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Merged] T140967: WikiPageEntityStore::updateWatchlist breaks implicit transactions
aaron closed this task as a duplicate of T140955: Wikibase\Repo\Store\WikiPageEntityStore::updateWatchlist: Automatic transaction with writes in progress (from DatabaseBase::query (LinkCache::addLinkObj)), performing implicit commit!. TASK DETAILhttps://phabricator.wikimedia.org/T140967EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Aklapper, aaron, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T140955: Wikibase\Repo\Store\WikiPageEntityStore::updateWatchlist: Automatic transaction with writes in progress (from DatabaseBase::query (LinkCache::addLinkObj)
aaron added a comment. The watchlist one is likely fixed by d484555db6b734ef56edf2d521dbcfb54170c7a6 in core . The extension code looks OK. The other one is caused by using deadlockLoop(), which I'd suggest removing.TASK DETAILhttps://phabricator.wikimedia.org/T140955EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: hashar, aaron, hoo, Lydia_Pintscher, daniel, aude, jcrespo, Aklapper, TerraCodes, greg, Luke081515, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T140955: Wikibase\Repo\Store\WikiPageEntityStore::updateWatchlist: Automatic transaction with writes in progress (from DatabaseBase::query (LinkCache::addLin
aaron added a comment. Premature commits like this make multi-DB transactions less safe. Only Job/DeferrableUpdate/Maintenance code should be flushing transactions. If something needs its own transaction it should use some sort of deferred update.TASK DETAILhttps://phabricator.wikimedia.org/T140955EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: hashar, aaron, hoo, Lydia_Pintscher, daniel, aude, jcrespo, Aklapper, TerraCodes, greg, Luke081515, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g, Krenair___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T140968: Wikibase refreshLinks hook breaks implicit transactions
aaron created this task.aaron added projects: Wikidata, MediaWiki-Database.Herald added a subscriber: Aklapper. TASK DESCRIPTIONFrom logstash: DatabaseBase::deadlockLoop: Automatic transaction with writes in progress (from DatabaseBase::query (LinksUpdate::acquirePageLock)), performing implicit commit!TASK DETAILhttps://phabricator.wikimedia.org/T140968EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Aklapper, aaron, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T140967: WikiPageEntityStore::updateWatchlist breaks implicit transactions
aaron created this task.aaron added projects: Wikidata, MediaWiki-Database.Herald added a subscriber: Aklapper. TASK DESCRIPTIONFrom logstash: Wikibase\Repo\Store\WikiPageEntityStore::updateWatchlist: Automatic transaction with writes in progress (from DatabaseBase::query (LinkCache::addLinkObj)), performing implicit commit!TASK DETAILhttps://phabricator.wikimedia.org/T140967EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: Aklapper, aaron, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331, Jay8g___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T110399: WikiPageEntityMetaDataLookup querying DB master on HTTP GET
aaron added a comment. Editing needs to do conflict detection anyway, since fresh data sent to the client becomes stale while they wait anyway, which has to be solved with conflict detection. The second case should use slaves too, just like templates/files use slaves, which works fine.TASK DETAILhttps://phabricator.wikimedia.org/T110399EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: daniel, hoo, aude, Joe, Krenair, PleaseStand, gerritbot, Aklapper, aaron, Gilles, Nemo_bis, MZMcBride, Glaisher, D3r1ck01, Izno, Wikidata-bugs, Mbch331___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Edited] T110399: WikiPageEntityMetaDataLookup querying DB master on HTTP GET
aaron edited the task description.TASK DETAILhttps://phabricator.wikimedia.org/T110399EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: hoo, aude, Joe, Krenair, PleaseStand, gerritbot, Aklapper, aaron, Gilles, Nemo_bis, MZMcBride, Glaisher, D3r1ck01, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T110399: WikiPageEntityMetaDataLookup querying DB master on HTTP GET
aaron changed the title from "WikiPageEntityMetaDataLookup querying DB master on GET" to "WikiPageEntityMetaDataLookup querying DB master on HTTP GET".TASK DETAILhttps://phabricator.wikimedia.org/T110399EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: hoo, aude, Joe, Krenair, PleaseStand, gerritbot, Aklapper, aaron, Gilles, Nemo_bis, MZMcBride, Glaisher, D3r1ck01, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Retitled] T110399: WikiPageEntityMetaDataLookup querying DB master on GET
aaron changed the title from "SpecialModifyEntity querying DB master on GET" to "WikiPageEntityMetaDataLookup querying DB master on GET".aaron edited the task description.TASK DETAILhttps://phabricator.wikimedia.org/T110399EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: aaronCc: hoo, aude, Joe, Krenair, PleaseStand, gerritbot, Aklapper, aaron, Gilles, Nemo_bis, MZMcBride, Glaisher, D3r1ck01, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Raised Priority] T135485: [Bug] File deletion problem on commons.wikimedia.org
aaron raised the priority of this task from "High" to "Unbreak Now!". Herald added subscribers: Luke081515, TerraCodes, Urbanecm. TASK DETAIL https://phabricator.wikimedia.org/T135485 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Urbanecm, TerraCodes, Luke081515, Riley_Huntley, matmarex, Raymond, aaron, Green_Giant, McZusatz, Krenair, Storkk, aude, hoo, daniel, Pokefan95, Aklapper, Steinsplitter, Wdwd, Zppix, D3r1ck01, Izno, Wikidata-bugs, El_Grafo, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T135485: [Bug] File deletion problem on commons.wikimedia.org
aaron added a comment. If the # of rows it might delete has no sane upper bound, then it should be a job. If it's just 100s at most, then it could use a DeferredUpdate class IMO. TASK DETAIL https://phabricator.wikimedia.org/T135485 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, Green_Giant, McZusatz, Krenair, Storkk, aude, hoo, daniel, Pokefan95, Aklapper, Steinsplitter, Wdwd, Zppix, Riley_Huntley, D3r1ck01, Izno, Wikidata-bugs, El_Grafo, Mbch331, Jay8g, fgiunchedi ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T110399: SpecialModifyEntity querying DB master on GET
aaron added a comment. This is now one of the two main remaining causes of warnings in this log. TASK DETAIL https://phabricator.wikimedia.org/T110399 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aude, Joe, Krenair, PleaseStand, gerritbot, bd808, Aklapper, aaron, Gilles, Nemo_bis, MZMcBride, Glaisher, D3r1ck01, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T108929: [Task] only fallback to master in WikiPageEntityMetaDataLookup when explicitly requested
aaron added a comment. Has anyone had a chance to look at this lately? The patch doesn't seem to load for me. TASK DETAIL https://phabricator.wikimedia.org/T108929 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: hoo, aaron Cc: aaron, Aklapper, aude, D3r1ck01, Izno, Wikidata-bugs, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Up For Grabs] T133422: DB master queries for entities on parse
aaron placed this task up for grabs. TASK DETAIL https://phabricator.wikimedia.org/T133422 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Joe, Krenair, PleaseStand, gerritbot, bd808, Aklapper, Gilles, Nemo_bis, MZMcBride, Glaisher, Johan, daniel, Amire80, aaron, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T133422: DB master queries for entities on parse
aaron created this task. TASK DESCRIPTION Expectation (masterConns <= 0) by MediaWiki::main not met: [connect to 10.64.16.144 (wikidatawiki)] TransactionProfiler.php line 311 calls wfBacktrace() TransactionProfiler.php line 146 calls TransactionProfiler->reportExpectationViolated() LoadBalancer.php line 571 calls TransactionProfiler->recordConnection() DBAccessBase.php line 61 calls LoadBalancer->getConnection() WikiPageEntityMetaDataLookup.php line 162 calls DBAccessBase->getConnection() WikiPageEntityMetaDataLookup.php line 71 calls Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->selectRevisionInformationMultiple() PrefetchingWikiPageEntityMetaDataAccessor.php line 193 calls Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->loadRevisionInformation() PrefetchingWikiPageEntityMetaDataAccessor.php line 162 calls Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->doFetch() WikiPageEntityRevisionLookup.php line 81 calls Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation() CachingEntityRevisionLookup.php line 149 calls Wikibase\Lib\Store\WikiPageEntityRevisionLookup->getEntityRevision() CachingEntityRevisionLookup.php line 132 calls Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision() CachingEntityRevisionLookup.php line 149 calls Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision() CachingEntityRevisionLookup.php line 132 calls Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision() RevisionBasedEntityLookup.php line 44 calls Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision() RedirectResolvingEntityLookup.php line 51 calls Wikibase\Lib\Store\RevisionBasedEntityLookup->getEntity() EntityRetrievingTermLookup.php line 131 calls Wikibase\DataModel\Services\Lookup\RedirectResolvingEntityLookup->getEntity() EntityRetrievingTermLookup.php line 116 calls Wikibase\DataModel\Services\Lookup\EntityRetrievingTermLookup->fetchFingerprint() EntityRetrievingTermLookup.php line 64 calls Wikibase\DataModel\Services\Lookup\EntityRetrievingTermLookup->getFingerprint() UsageTrackingTermLookup.php line 64 calls Wikibase\DataModel\Services\Lookup\EntityRetrievingTermLookup->getLabels() LanguageFallbackLabelDescriptionLookup.php line 55 calls Wikibase\Client\Usage\UsageTrackingTermLookup->getLabels() WikibaseLuaBindings.php line 138 calls Wikibase\Lib\Store\LanguageFallbackLabelDescriptionLookup->getLabel() Scribunto_LuaWikibaseLibrary.php line 349 calls Wikibase\Client\DataAccess\Scribunto\WikibaseLuaBindings->getLabel() Engine.php line 403 calls Wikibase\Client\DataAccess\Scribunto\Scribunto_LuaWikibaseLibrary->getLabel() - line - calls Scribunto_LuaSandboxCallback->__call() Engine.php line 316 calls LuaSandboxFunction->call() LuaCommon.php line 240 calls Scribunto_LuaSandboxInterpreter->callFunction() LuaCommon.php line 881 calls Scribunto_LuaEngine->executeModule() Hooks.php line 121 calls Scribunto_LuaModule->invoke() Parser.php line 3807 calls ScribuntoHooks::invokeHook() Parser.php line 3542 calls Parser->callParserFunction() Preprocessor_Hash.php line 1075 calls Parser->braceSubstitution() Preprocessor_Hash.php line 1514 calls PPFrame_Hash->expand() Parser.php line 3681 calls PPTemplateFrame_Hash->cachedExpand() Preprocessor_Hash.php line 1075 calls Parser->braceSubstitution() Parser.php line 3356 calls PPFrame_Hash->expand() Parser.php line 1249 calls Parser->replaceVariables() Parser.php line 446 calls Parser->internalParse() WikitextContent.php line 331 calls Parser->parse() AbstractContent.php line 497 calls WikitextContent->fillParserOutput() PoolWorkArticleView.php line 140 calls AbstractContent->getParserOutput() PoolCounterWork.php line 123 calls PoolWorkArticleView->doWork() Article.php line 666 calls PoolCounterWork->execute() ViewAction.php line 44 calls Article->view() MediaWiki.php line 503 calls ViewAction->show() MediaWiki.php line 288 calls MediaWiki->performAction() MediaWiki.php line 745 calls MediaWiki->performRequest() MediaWiki.php line 519 calls MediaWiki->main() index.php line 43 calls MediaWiki->run() index.php line 3 calls include() TASK DETAIL https://phabricator.wikimedia.org/T133422 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Joe, Krenair, PleaseStand, gerritbot, bd808, Aklapper, Gilles, Nemo_bis, MZMcBride, Glaisher, Johan, daniel, Amire80, aaron, D3r1ck01, Izno, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T88986: [Task] Avoid inconsistencies in Wikidata editing due to slave lag
aaron removed a blocked task: T88445: MediaWiki multi-datacenter investigation and work. TASK DETAIL https://phabricator.wikimedia.org/T88986 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, Addshore, Lydia_Pintscher, JanZerebecki, hoo, aude, Aklapper, daniel, Minhnv-2809, Volans, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T88986: [Task] Avoid inconsistencies in Wikidata editing due to slave lag
aaron added a comment. In https://phabricator.wikimedia.org/T88986#1692565, @aude wrote: > @aaron https://phabricator.wikimedia.org/T108929 is the one issue I am aware of that should be fixed asap. I'm not sure what else... Is anyone one looking at that? I think I still see it in the logs from time to time by happenstance. TASK DETAIL https://phabricator.wikimedia.org/T88986 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, Addshore, Lydia_Pintscher, JanZerebecki, hoo, aude, Aklapper, daniel, Minhnv-2809, D3r1ck01, Izno, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Declined] T69117: Performance review of PubSubHubbub extension
aaron closed this task as "Declined". aaron claimed this task. Herald removed a subscriber: Liuxinyu970226. TASK DETAIL https://phabricator.wikimedia.org/T69117 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Jimkont, Wikidata-bugs, greg, Anomie, ori, wikibugs-l-list, aaron, KartikMistry, Lydia_Pintscher, daniel ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T122360: ChangeNotification jobs may be doing too much work at once
aaron created this task. aaron added subscribers: aaron, BBlack. aaron added a project: Wikidata. Herald added subscribers: StudiesWorld, Aklapper. TASK DESCRIPTION I see errors in the logs like: ``` LoadBalancer::commitAll 10.64.48.26 2006MySQL server has gone away (10.64.48.26)COMMIT ``` ``` Number of requested IDs (79370) is too high ``` The later error corresponds to a large CDN purge call (with ~80k urls). I hope that doesn't cause UDP flood once the UID limit is removed (wmf10). Maybe the job run() method could be split up? Even internally, by batching purges with short sleep in between calling commitAll() between long running methods if needed. Alternatively, it could subdivide...but I'm not too familiar with exactly what it does to know if that would work. TASK DETAIL https://phabricator.wikimedia.org/T122360 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: BBlack, Aklapper, aaron, StudiesWorld, Wikidata-bugs, aude, Mbch331 ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Changed Status] T103912: [Task] Ex:WikibaseQualityExternalValidation - performance review of Special:CrossCheck
aaron changed the task status from "Open" to "Stalled". aaron added a comment. Was this deployed already? Perhaps this can just be closed. TASK DETAIL https://phabricator.wikimedia.org/T103912 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: StudiesWorld, Lydia_Pintscher, aaron, Wikibase-Quality-External-Validation, Aklapper, Liuxinyu970226, Andreasburmeister, csteipp, Tamslo, Jonaskeutel, JanZerebecki, soeren.oldag, dpatrick, Luke081515, Wikidata-bugs, aude, Bawolff, Mbch331, Krenair, Legoktm ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Unblock] T75456: DatabaseBase::deadlockLoop: Transaction already in progress (from RefreshLinks::fixLinksFromArticle)
aaron closed blocking task T45575: PHP notices: Explicit commit of implicit transaction and Transaction already in progress as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T75456 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Lydia_Pintscher, aaron Cc: gerritbot, demon, Aklapper, Wikidata-bugs, aude, Lydia_Pintscher, hoo, Luke081515, Mbch331, Jay8g, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
aaron added a subscriber: aaron. aaron added a comment. See "+channel:DBPerformance +message:"*connections made*"" at logstash.wikimedia.org. I see lots of concurrent connections from mw1152 scripts (if reuseConnection was called properly I'd assume there would be ~7 or so, since going from DB to DB on the same shard is just a matter of USE clauses). TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T118162: Wikibase dispatchChanges.php runs slow, creates an absurd amount of database connections
aaron added a comment. I'll try to look into whether the problem is in core or not, though I need to spend time on some other tasks too. TASK DETAIL https://phabricator.wikimedia.org/T118162 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, jcrespo, gerritbot, daniel, aude, hoo, Lydia_Pintscher, Addshore, Aklapper, Joe, Wikidata-bugs, Mbch331, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Updated] T117398: number of database updates multiplied x3 since 29 October
aaron added a subscriber: aaron. aaron added a comment. Sounds like the the result of fixing the rpc/RunJobs to properly run jobs till the 30 sec limit rather than 1 at a time (which wasted huge amounts of time in setup overhead and caused massive job backlogs, particularly for the 'enqueue' and 'refreshLinks' queues). Keeping up means more DB traffic. This was noticed by Erik and fixed by me on 2015-10-29. 06:30 logmsgbot: aaron@tin Synchronized rpc/RunJobs.php: https://phabricator.wikimedia.org/rOMWC29ccbd24839f4360f022403f7075460d072ade40 (duration: 00m 17s) Related: https://phabricator.wikimedia.org/T117304 In any case, if the run rate for any type of job is too high for some reason, then runner count and $wgJobBackoffThrottling can be adjusted. TASK DETAIL https://phabricator.wikimedia.org/T117398 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, Performance-Team, Aklapper, jcrespo, Wikidata-bugs, aude, GWicke, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T88986: [Task] Avoid inconsistencies due to slave lag
aaron added a subscriber: aaron. aaron added a comment. If this bug is wikidata-specific, can someone update the title to reflect that? Also, is anyone working on this yet? TASK DETAIL https://phabricator.wikimedia.org/T88986 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aaron, Addshore, Lydia_Pintscher, JanZerebecki, hoo, aude, Aklapper, daniel, Wikidata-bugs, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Up For Grabs] T110399: SpecialModifyEntity querying DB master on GET
aaron placed this task up for grabs. aaron set Security to None. TASK DETAIL https://phabricator.wikimedia.org/T110399 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: aude, Joe, Krenair, PleaseStand, gerritbot, bd808, Aklapper, aaron, Gilles, Nemo_bis, MZMcBride, Glaisher, Wikidata-bugs ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T103912: [Task] Ex:WikibaseQualityExternalValidation - performance review of Special:CrossCheck
aaron added a comment. The JOIN for query #2 does not seem to have an index. wbqev_identifier_properties has: PRIMARY KEY (identifier_pid, dump_id) ...but nothing like (dump_id). Is the wbqev_identifier_properties table going to pruned of older dumps or will it just keep growing? Query #3 would require a few clever index dives to run well. I'm not sure how smart MariaDB is here. Has anyone tried "EXPLAIN format=json" with a good dataset (doesn't have to be full size, but a few 100k items). My fear would be having just the (dump_id) index part used and scanning happening for the other two IN()s. On the revision table, two IN()s work fine: EXPLAIN EXTENDED SELECT * FROM revision FORCE INDEX(rev_page_id) WHERE rev_page IN (5043734,3535,6234) AND rev_id IN (343535,23255,33626); stdClass Object ( [id] => 1 [select_type] => SIMPLE [table] => revision [type] => range [possible_keys] => rev_page_id [key] => rev_page_id [key_len] => 8 [ref] => [rows] => 9 [filtered] => 100.00 [Extra] => Using index condition ) For wbqev_external_data, the only index with pid has it as the third part of the index. CREATE INDEX /*i*/dump_id_external_id_pid ON /*_*/wbqev_external_data (dump_id, external_id, pid); ...so mariadb will need to handle 3 levels of IN nesting to avoid scanning extra rows. It may work just fine, though it would be nice to know for sure. I'm having trouble finding documentation about wbqev_external_data. The PK is just an opaque rowid field. How many rows might there be for any given (dump_id, external_id, pid) combination? What decides that number? TASK DETAIL https://phabricator.wikimedia.org/T103912 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Lydia_Pintscher, aaron, Wikibase-Quality-External-Validation, Aklapper, Liuxinyu970226, Andreasburmeister, csteipp, Tamslo, Jonaskeutel, JanZerebecki, soeren.oldag, dpatrick, Wikidata-bugs, aude, Krenair ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Created] T110399: SpecialModifyEntity querying DB master on GET
aaron created this task. aaron claimed this task. aaron added subscribers: Glaisher, MZMcBride, Nemo_bis, Gilles, aaron, Aklapper, bd808, gerritbot, PleaseStand, Krenair, Joe. aaron added projects: Availability, Wikidata. TASK DESCRIPTION [GET] Expectation (masterConns = 0) by MediaWiki::main not met: [connect to 10.64.32.28 (wikidatawiki)] TransactionProfiler.php line 307 calls wfBacktrace() TransactionProfiler.php line 146 calls TransactionProfiler-reportExpectationViolated() LoadBalancer.php line 545 calls TransactionProfiler-recordConnection() DBAccessBase.php line 61 calls LoadBalancer-getConnection() WikiPageEntityMetaDataLookup.php line 161 calls DBAccessBase-getConnection() WikiPageEntityMetaDataLookup.php line 70 calls Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup-selectRevisionInformationMultiple() PrefetchingWikiPageEntityMetaDataAccessor.php line 148 calls Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup-loadRevisionInformation() WikiPageEntityRevisionLookup.php line 78 calls Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor-loadRevisionInformation() SpecialWikibaseRepoPage.php line 163 calls Wikibase\Lib\Store\WikiPageEntityRevisionLookup-getEntityRevision() SpecialModifyEntity.php line 127 calls Wikibase\Repo\Specials\SpecialWikibaseRepoPage-loadEntity() SpecialSetSiteLink.php line 103 calls Wikibase\Repo\Specials\SpecialModifyEntity-prepareArguments() SpecialModifyEntity.php line 76 calls Wikibase\Repo\Specials\SpecialSetSiteLink-prepareArguments() SpecialPage.php line 384 calls Wikibase\Repo\Specials\SpecialModifyEntity-execute() SpecialPageFactory.php line 553 calls SpecialPage-run() MediaWiki.php line 249 calls SpecialPageFactory::executePath() MediaWiki.php line 683 calls MediaWiki-performRequest() MediaWiki.php line 474 calls MediaWiki-main() index.php line 41 calls MediaWiki-run() index.php line 3 calls include() TASK DETAIL https://phabricator.wikimedia.org/T110399 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: aaron Cc: Joe, Krenair, PleaseStand, gerritbot, bd808, Aklapper, aaron, Gilles, Nemo_bis, MZMcBride, Glaisher, Wikidata-bugs, aude, Malyacko ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Project] [Updated] Wikidata-Query-Service
aaron removed a member: aaron. PROJECT DETAIL https://phabricator.wikimedia.org/project/profile/891/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gage, ksmith, Jdouglas, DanielFriesen, hoo, Addshore, Tpt, JeroenDeDauw, Joe, Eloquence, aude, Tobi_WMDE_SW, Wikidata-bugs, daniel, MaxSem, jkroll, JanZerebecki, Smalyshev, Manybubbles, GWicke ___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
