[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2021-01-06 Thread Gehel
Gehel closed subtask T246004: Spike: Can/should Swift be used as Flink checkpoint backend? as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-12-04 Thread dcausse
dcausse added a subtask: T269451: Possible flink optimizations/cleanups. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-11-23 Thread Gehel
Gehel closed subtask T266750: The streaming updater consumer should stop accumulating patches if it cannot handle them as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-11-23 Thread Gehel
Gehel closed subtask T255399: Prepare wdqs1009 to run the streaming updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-11-09 Thread Gehel
Gehel closed subtask T256882: The streaming updater should support page undeletes as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-11-09 Thread Gehel
Gehel closed subtask T265452: Add a configurable restart strategy to the streaming updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-11-09 Thread Gehel
Gehel closed subtask T266318: Clarify dependencies on codehale dropwizards as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-11-02 Thread dcausse
dcausse added a subtask: T267001: Compute page properties information at munge time. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-11-01 Thread dcausse
dcausse added a subtask: T266986: The metrics of the streaming-updater-consumer should be visible in grafana. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-30 Thread Gehel
Gehel added a subtask: T266470: Expose wdqs1009 to wdqs users and gather feedback. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-30 Thread Gehel
Gehel added a subtask: T261119: Architecture review of Flink based WDQS Streaming Updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-30 Thread Gehel
Gehel added a subtask: T266750: The streaming updater consumer should stop accumulating patches if it cannot handle them. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-26 Thread Gehel
Gehel closed subtask T256875: The streaming updater should support page deletions as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-26 Thread Gehel
Gehel closed subtask T256949: The streaming updater should support suppressed deletes as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-23 Thread dcausse
dcausse added a subtask: T266321: Determine flink metrics configuration and backend when running from k8s. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-23 Thread dcausse
dcausse added a subtask: T266318: Clarify dependencies on codehale dropwizards. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-14 Thread dcausse
dcausse added a subtask: T265452: Add a configurable restart strategy to the streaming updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-13 Thread Mstyles
Mstyles reopened subtask T256875: The streaming updater should support page deletions as Open. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-13 Thread Mstyles
Mstyles closed subtask T256875: The streaming updater should support page deletions as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

2020-10-06 Thread Gehel
Gehel renamed this task from "EPIC: Rework the WDQS updater as an event driven application" to "[Epic] Rework the WDQS updater as an event driven application". TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-10-05 Thread Gehel
Gehel closed subtask T262896: http://wikiba.se/ontology#wikiGroup; triples should be marked as linked/unlinked shared statements as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-28 Thread Gehel
Gehel closed subtask T258240: Refactor Options handling in Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-28 Thread Gehel
Gehel closed subtask T251096: [WDQS Streaming Updater] Organise module structure as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-15 Thread dcausse
dcausse added a subtask: T262896: triples should be marked as "linked/unlinked shared statements". TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-14 Thread Gehel
Gehel closed subtask T256946: Add a custom watermark generator that supports low rate streams as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-14 Thread Gehel
Gehel closed subtask T248449: [WDQS Streaming Updater] Add error handling for Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-14 Thread Gehel
Gehel closed subtask T258687: The streaming updater should read its events from multiple DC streams as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-11 Thread dcausse
dcausse added a subtask: T262676: Make the pipeline more robust when backfilling data generated during a DC switch. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-02 Thread dcausse
dcausse updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-02 Thread dcausse
dcausse added a comment. In T244590#6429672 , @Thadguidry wrote: >> - the output of this is a simple event without any data saying: do a diff between rev X and Y, fully delete entity QXYZ, ... > > Is that supposed to be "data

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-02 Thread Thadguidry
Thadguidry added a comment. > - the output of this is a simple event without any data saying: do a diff between rev X and Y, fully delete entity QXYZ, ... Is that supposed to be "data saving" ? > rdf diff generation: materialize the command and fetch the data from wikibase and send

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-02 Thread Thadguidry
Thadguidry updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Thadguidry Cc: tfmorris, revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-09-01 Thread Gehel
Gehel closed subtask T258684: Use RichAsyncFunction for GenerateEntityDiffPatchOperation as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris, revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-08-19 Thread Gehel
Gehel added a subtask: T244341: Stop using blank nodes for encoding SomeValue and OWL constraints in WDQS. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris, revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-08-17 Thread Gehel
Gehel closed subtask T257325: Upgrade to flink 1.11 as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris, revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-08-17 Thread Gehel
Gehel closed subtask T257795: GenerateEntityDiffPatchOperationUnitTest fails randomly as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris, revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-08-17 Thread Gehel
Gehel closed subtask T248451: [WDQS Streaming Updater] Custom parallelism configuration for Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-23 Thread dcausse
dcausse added a subtask: T258687: The streaming updater should read its events from multiple DC streams. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: tfmorris, revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-23 Thread dcausse
dcausse removed a subtask: T258683: Switch GenerateEntityDiffPatchOperation to a RichFunction. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: tfmorris, revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-23 Thread dcausse
dcausse added a subtask: T258684: Use RichAsyncFunction for GenerateEntityDiffPatchOperation. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: tfmorris, revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-23 Thread dcausse
dcausse added a subtask: T258683: Switch GenerateEntityDiffPatchOperation to a RichFunction. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: tfmorris, revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-17 Thread Zbyszko
Zbyszko added a subtask: T258240: Refactor Options handling in Streaming Updater . TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: revi, Mholloway, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-13 Thread Gehel
Gehel closed subtask T251275: [WDQS Streaming Updater] Update blazegraph based on the content present in the streaming updater output kafka stream as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-13 Thread Gehel
Gehel closed subtask T251270: The streaming updater should produce its events to kafka as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-13 Thread Gehel
Gehel closed subtask T245728: Add a component to generate a diff between two entity revisions as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-13 Thread Gehel
Gehel closed subtask T248464: [WDQS Streaming Updater] Implement ouput format in Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-13 Thread Gehel
Gehel closed subtask T248452: [WDQS Streaming Updater] Deploy and configure Streaming Updater Hadoop YARN as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

2020-07-13 Thread dcausse
dcausse added a subtask: T257795: GenerateEntityDiffPatchOperationUnitTest fails randomly. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: revi, Mholloway, Ladsgroup, Multichill,