dcausse created this task.
dcausse added projects: Wikidata-Query-Service,
Discovery-Wikidata-Query-Service-Sprint.
Restricted Application added a subscriber: Aklapper.
Restricted Application added a project: Wikidata.
TASK DESCRIPTION
We currently have no way to anticipate (offline) how prod performance can be
affected by a optimization made on the code base or by tweaking blazegraph
settings.
The technique would be to replay queries logged in production to test server
deployed with and without the changes and evaluate the differences.
Replay queries
--------------
@Igorkim78 suggests using this script snippet to collect numbers:
echo "md5,http_code,execution_seconds,result_bytes,result_rows" && awk -v
RS="\n---\n" '{ print "read -d '\'\'' QUERY << EOM\n"$0"\nEOM\n RESULT=`curl -s
-X POST -H 'accept:text/csv' -w
'%{http_code},%{time_starttransfer},%{size_download}' --data-binary @-
https://query.wikidata.org/sparql <<EOF2\nquery="$0"\nEOF2`\n echo `echo
\"$QUERY\" | md5sum | tr -d \" -\"`,`echo \"$RESULT\" | tail -n 1`,`echo
\"$RESULT\" | tail -n +3 | wc -l` " }' querymix.txt | sh
The input file looks like:
select * {
?s ?p ?o
} limit 1
---
select * {
?s ?p ?o
} limit 2
And the output file looks like:
md5,http_code,execution_seconds,result_bytes,result_rows
f01b6f2b0f403054502d60bdec9ebc1d,200,0.604,127,1
5d92d9891ae8a64acac0269511406a29,200,0.671,199,2
Collect queries
---------------
The sole queries that can currently be extracted are the GET requests from
the webrequests logs in hive. This will not include large queries sent by users
using a POST request.
T101013 <https://phabricator.wikimedia.org/T101013> should fix this by adding
a dedicated dataset of queries collected from the wdqs backend.
TASK DETAIL
https://phabricator.wikimedia.org/T234968
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: dcausse
Cc: dcausse, Igorkim78, Aklapper, darthmon_wmde, DannyS712, Nandana, Lahi,
Gq86, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, _jensen, rosalieper, Cirdan, Jonas, Xmlizer, jkroll, Smalyshev,
Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
