[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2019-12-04T10:31:10Z] rolling restart of wdqs for config change (event logging) - T101013 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phab

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 554357 **merged** by Ottomata: [operations/puppet@production] Use mediawiki_analytics_events, no mediawiki_events camus job for sparql-query https://gerrit.wikimedia.org/r/554357 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCE

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 554357 had a related patch set uploaded (by Ottomata; owner: Ottomata): [operations/puppet@production] Use mediawiki_analytics_events, no mediawiki_events camus job for sparql-query https://gerrit.wikimedia.org/r/554357 TASK DETAIL https://phabricato

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 554353 **merged** by Ottomata: [operations/puppet@production] Import sparql-query events in mediawiki_analytics camus job https://gerrit.wikimedia.org/r/554353 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabric

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 554353 had a related patch set uploaded (by Ottomata; owner: Ottomata): [operations/puppet@production] Import sparql-query events in mediawiki_analytics camus job https://gerrit.wikimedia.org/r/554353 TASK DETAIL https://phabricator.wikimedia.org/T10

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 554327 **merged** by Ottomata: [operations/puppet@production] Import *sparql-query topics and refine them https://gerrit.wikimedia.org/r/554327 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.or

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 554327 had a related patch set uploaded (by Ottomata; owner: Ottomata): [operations/puppet@production] Import *sparql-query topics and refine them https://gerrit.wikimedia.org/r/554327 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFE

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 549081 **merged** by Gehel: [operations/puppet@production] [wdqs] configure eventgate endpoint for sparql/query events https://gerrit.wikimedia.org/r/549081 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricato

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 548764 **merged** by Ottomata: [operations/deployment-charts@master] eventgate-analytics - stream config for new sparql-query streams https://gerrit.wikimedia.org/r/548764 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES htt

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

dcausse added a comment. @JAllemandou no, I moved this done too early and it was closed before being available. The code is not yet deployed. I'll ping you once that is available. TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

JAllemandou added a comment. Does this being closed mean we can access data on kafka? TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, JAllemandou Cc: Igorkim78, JAllemandou, Ottomata,

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 550754 **merged** by Gehel: [operations/puppet@production] [wdqs] Fix event service configuration https://gerrit.wikimedia.org/r/550754 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settin

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 550754 had a related patch set uploaded (by DCausse; owner: DCausse): [operations/puppet@production] [wdqs] Fix event service configuration https://gerrit.wikimedia.org/r/550754 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 548779 **merged** by jenkins-bot: [wikidata/query/rdf@master] Make the stream name configurable for sparql/query events https://gerrit.wikimedia.org/r/548779 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricat

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

Ottomata added a comment. > Sorry about this stupid queue name No worries, its beta ┐｜･ิω･ิ#｜┌ TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, Ottomata Cc: Igorkim78, JAllemandou,

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 549081 had a related patch set uploaded (by DCausse; owner: DCausse): [operations/puppet@production] [wdqs] configure eventgate endpoint for sparql/query events https://gerrit.wikimedia.org/r/549081 TASK DETAIL https://phabricator.wikimedia.org/T1010

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

dcausse added a comment. Tested sending an event to eventgate in beta generated by new code: curl -XPOST -H"Content-Type: application/json; charset=UTF-8" http://deployment-eventgate-3.deployment-prep.eqiad.wmflabs:8192/v1/events -d '[{"namespace":"ns123","query":"select","format":"xml

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 548779 had a related patch set uploaded (by DCausse; owner: DCausse): [wikidata/query/rdf@master] Make the stream name configurable for sparql/query events https://gerrit.wikimedia.org/r/548779 TASK DETAIL https://phabricator.wikimedia.org/T101013 E

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 548764 had a related patch set uploaded (by Ottomata; owner: Ottomata): [operations/deployment-charts@master] eventgate-analytics - stream config for new stream wdqs.sparql-query https://gerrit.wikimedia.org/r/548764 TASK DETAIL https://phabricator.w

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 543004 **merged** by jenkins-bot: [wikidata/query/rdf@master] Add sparql query event gate generator https://gerrit.wikimedia.org/r/543004 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/sett

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 545563 **merged** by Ottomata: [mediawiki/event-schemas@master] Add http.response_headers to sparql/query/1.0.0 https://gerrit.wikimedia.org/r/545563 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikim

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 545563 had a related patch set uploaded (by DCausse; owner: DCausse): [mediawiki/event-schemas@master] Add http.response_headers to sparql/query/1.0.0 https://gerrit.wikimedia.org/r/545563 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

dcausse added a comment. @Igorkim78 thanks for the suggestion, we could perhaps log all response HTTP headers (we already log all request HTTP headers). @Ottomata thanks for the merge, I'll perhaps send a quick patch to add this before asking you to deploy anything. TASK DETAIL https://p

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

Ottomata added a comment. sparql/query schema is merged. We'll need to do an eventgate-analytics k8s deploy before it can be used. Let me know when you want to start testing this. We can also update eventgate-analytics in deployment-prep first if you can/want to test there. TASK DETAIL

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 541563 **merged** by Ottomata: [mediawiki/event-schemas@master] Add sparql/query 1.0.0 https://gerrit.wikimedia.org/r/541563 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/em

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 543004 had a related patch set uploaded (by DCausse; owner: DCausse): [wikidata/query/rdf@master] WIP: Add sparql query event gate generator https://gerrit.wikimedia.org/r/543004 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

Ottomata added a comment. > do we need to do something on the refinery/hadoop side to create the hive table Depending on the name of the stream, it will probably need whitelisted in a camus a refine job. > and/or add a purge mechanism for the 90days retention I believe the tabl

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

dcausse added a comment. @Ottomata I updated the task description to indicate the steps needed to make this happen, I probably missed something so could you take a quick look and let us know I missed something important here (e.g. do we need to do something on the refinery/hadoop side to cre

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

dcausse added a comment. @Ottomata absolutely this is for analysis purposes TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Ottomata, Smalyshev, Deskana, Aklapper, 4748kitoko, Hook6

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

Ottomata added a comment. Q: I assume you'll want this to go into the eventgate-analytics instance, yes? I.e. You won't be building production services on this stream, just using it for offline analytics (well, offline performance testing). TASK DETAIL https://phabricator.wikimedia.org/T1

[Wikidata-bugs] [Maniphest] [Commented On] T101013: Log Wikidata Query Service queries to the event gate infrastructure

gerritbot added a comment. Change 541563 had a related patch set uploaded (by DCausse; owner: DCausse): [mediawiki/event-schemas@master] Add wdqs/sparqlquery 0.0.1 https://gerrit.wikimedia.org/r/541563 TASK DETAIL https://phabricator.wikimedia.org/T101013 EMAIL PREFERENCES https://