[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

dcausse removed projects: Patch-For-Review, Discovery-Search (Current work). dcausse removed dcausse as the assignee of this task. dcausse added a comment. Made https://github.com/nomoa/flink-python-demo but stopped actively working on this for the moment, have hit issues with python env.

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

Stashbot added a comment. Mentioned in SAL (#wikimedia-releng) [2020-12-04T18:46:06Z] Zuul: Added CI for Flink Swift Plugin T262942 TASK DETAIL https://phabricator.wikimedia.org/T262942 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

gerritbot added a comment. Change 643279 **merged** by jenkins-bot: [integration/config@master] jjb / Zuul: Added CI for Flink Swift Plugin https://gerrit.wikimedia.org/r/643279 TASK DETAIL https://phabricator.wikimedia.org/T262942 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

gerritbot added a comment. Change 643021 **abandoned** by ZPapierski: [wikidata/query/rdf@master] Flink Swift FS plugin with TempAuth Reason: Moved to own repo - https://gerrit.wikimedia.org/r/c/wikidata/query/flink-swift-plugin/ /643257 https://gerrit.wikimedia.org/r/643021

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

gerritbot added a comment. Change 643279 had a related patch set uploaded (by ZPapierski; owner: ZPapierski): [integration/config@master] Added Flink Swift Plugin https://gerrit.wikimedia.org/r/643279 TASK DETAIL https://phabricator.wikimedia.org/T262942 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

gerritbot added a comment. Change 643257 had a related patch set uploaded (by ZPapierski; owner: ZPapierski): [wikidata/query/flink-swift-plugin@master] Flink Swift FS plugin with TempAuth https://gerrit.wikimedia.org/r/643257 TASK DETAIL https://phabricator.wikimedia.org/T262942

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T262942 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse, gerritbot Cc: Aklapper, Ottomata, Gehel, dcausse, CDanis, Zbyszko, Alter-paule, Beast1978,

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

gerritbot added a comment. Change 643021 had a related patch set uploaded (by ZPapierski; owner: ZPapierski): [wikidata/query/rdf@master] Flink Swift FS plugin with TempAuth https://gerrit.wikimedia.org/r/643021 TASK DETAIL https://phabricator.wikimedia.org/T262942 EMAIL

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

dcausse added a comment. Yes it definitely can support such queries e.g (extract all api requests from mediawiki.apiaction grouped by their action param and database where the avg backend time is > 100ms over a 1 minute window). SELECT TUMBLE_START(dt, INTERVAL '1' MINUTE) as

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

Ottomata added a comment. Hm, I think we need to find a use case that can be done using Stream SQL repl. I don't think SRE will deploy a Java app e.g. during a DDoS. Can Flink's SQL REPL do something like

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

dcausse added a comment. Looking at existing solutions based on flink in this area I don't think this is a good fit for the table API and/or SQL unless the usecase is relatively simple (does not require fine control on the state nor specific timers). Most solutions I've seen describe a

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

dcausse claimed this task. dcausse moved this task from Ready for Development to In Progress on the Discovery-Search (Current work) board. TASK DETAIL https://phabricator.wikimedia.org/T262942 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

CBogen set the point value for this task to "13". TASK DETAIL https://phabricator.wikimedia.org/T262942 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: CBogen Cc: Aklapper, Ottomata, Gehel, dcausse, CDanis, Zbyszko, CBogen, Akuckartz, 4748kitoko,

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

Zbyszko moved this task from All WDQS-related tasks to Current work on the Wikidata-Query-Service board. Zbyszko added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T262942 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

mforns edited projects, added Analytics-Radar; removed Analytics. TASK DETAIL https://phabricator.wikimedia.org/T262942 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: mforns Cc: Aklapper, Ottomata, Gehel, dcausse, CDanis, Zbyszko, CBogen, Akuckartz,

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

Gehel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T262942 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Ottomata, Gehel, dcausse, CDanis, Zbyszko, CBogen, Akuckartz, 4748kitoko, darthmon_wmde,

[Wikidata-bugs] [Maniphest] T262942: PoC on anomaly detection with Flink

Gehel created this task. Gehel added projects: Wikidata-Query-Service, Analytics. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION As the team having a use case on Flink (WDQS Streaming Updater) we want to explore sharing