Addshore added a comment.
The only other thing I was going to add (forgot before i hit submit on the
last post)
Within the cluster varnish cached results for entities return much faster
than the php returned results (of course)
| entity | varnish result | php result | page selection |
| Q1.ttl?flavour=dump | ~0.06-0.07s | ~0.6-0.7s | randomish |
| Q64.ttl?flavour=dump | ~0.15-0.16s | ~2.3-2.5s | randomish |
| Q100.ttl?flavour=dump | ~0.13-0.14s | ~2s | randomish |
| Q55886027.ttl?flavour=dump | ~0.14s | ~7-17s? | LongPages |
| Q2911127.ttl?flavour=dump | ~~0.02s | 0.06s | ShortPages |
|
Data was gathered from a prod mw host with requests like the following
cat curl-format.txt
time_namelookup: %{time_namelookup}\n time_connect:
%{time_connect}\n time_appconnect: %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n time_redirect:
%{time_redirect}\n time_starttransfer: %{time_starttransfer}\n
----------\n time_total: %{time_total}\n
curl -w "@curl-format.txt" -o /dev/null -s
"https://www.wikidata.org/wiki/Special:EntityData/Q2911127.ttl?flavour=dump&addshore=17";
I guess the wdqs internal machines would have comparable response times?
It's hard to really figure anything concrete out from this but the wdqs
updater / updaters would potentially spend a lot less time waiting for
responses (maybe they already do them async?) if they hit varnish more?
Doing some terrible maths and looking at the smallest possible time saving
for a short page, so 0.04s saved by hitting the cache and assuming 1 million
edits in a day (based on the comment above, even though right now the wdqs
updater does a small amount of batching so makes less requests) 1000000*0.04 =
40,000s = =~11 hours per host?
This doesn't really help if the slowest part of the process is actually
writing the data to blazegraph, but 11 hours in a 24 hour period is still
pretty significant. I hope the Java updater does some amount of async work
(writing to blazegraph while getting the next data ready?)
TASK DETAIL
https://phabricator.wikimedia.org/T217897
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Addshore
Cc: Addshore, Smalyshev, BBlack, Aklapper, Gehel, alaa_wmde, Legado_Shulgin,
Nandana, thifranc, AndyTan, Davinaclare77, Qtn1293, Lahi, Gq86,
Lucas_Werkmeister_WMDE, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, EBjune,
merbst, LawExplorer, Zppix, _jensen, rosalieper, Jonas, Xmlizer, Wong128hk,
jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, faidon,
Mbch331, Jay8g, fgiunchedi
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
