[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T289836: Upgrade the WDQS streaming updater to latest flink (1.16) as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Mohammed_Sadat_WMDE, So9q,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T279541: Add a reconciliation strategy to the wdqs streaming updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Mohammed_Sadat_WMDE, So9q,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T301108: Migrate wdqs1010 to the Flink based Streaming Updater and cleanup left over pieces of the old updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T299222: Properly configure logback for W[CD]QS streaming updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Mohammed_Sadat_WMDE, So9q,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T301108: Migrate wdqs1010 to the Flink based Streaming Updater and cleanup left over pieces of the old updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T299222: Properly configure logback for W[CD]QS streaming updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Mohammed_Sadat_WMDE, So9q, Lydia_Pintscher,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T261119: Architecture review of Flink based WDQS Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Mohammed_Sadat_WMDE, So9q,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Ottomata added a comment. ※\(^o^)/※ TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel, Ottomata Cc: Mohammed_Sadat_WMDE, So9q, Lydia_Pintscher, VladimirAlexiev, karapayneWMDE, MPhamWMF,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed this task as "Resolved". Gehel claimed this task. Gehel added a comment. Completed! TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Mohammed_Sadat_WMDE, So9q,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T276469: Cookbooks and / or operation procedures are adapted for the new Flink based WDQS Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T276467: Ensure we have proper monitoring / alerting on the new Flink based WDQS Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T288231: Deploy the wdqs streaming updater to production as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Mohammed_Sadat_WMDE, So9q, Lydia_Pintscher,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse closed subtask T266321: Determine flink metrics configuration and backend when running from k8s as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

MPhamWMF closed subtask T105427: Need a way for WDQS updater to become aware of suppressed deletes as Declined. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: MPhamWMF Cc: Mohammed_Sadat_WMDE,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse closed subtask T286890: Checkpoint _metadata has grown up to 70Mb as Declined. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Mohammed_Sadat_WMDE, So9q, Lydia_Pintscher,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Lydia_Pintscher merged a task: T290061: improve WD update performance. Lydia_Pintscher added subscribers: VladimirAlexiev, Lydia_Pintscher, So9q, Mohammed_Sadat_WMDE. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T283591: StateExtractionJob is too slow as Declined. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: karapayneWMDE, MPhamWMF, Daniel_Mietchen, Thadguidry, tfmorris,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T286938: Create a plan for a final streaming updater rollout as source of truth for blazegraph instances as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T289836: Upgrade to latest flink (1.13.2). TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: karapayneWMDE, MPhamWMF, Daniel_Mietchen, Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T264006: Deploy Flink (rdf-streaming-updater) to kubernetes (k8s) as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: karapayneWMDE, MPhamWMF,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse removed a parent task: T285710: WDQS lag detection required manual adjustment during DC switchover. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: MPhamWMF, Daniel_Mietchen,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T288231: Deploy the wdqs streaming updater to production. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: MPhamWMF, Daniel_Mietchen, Thadguidry, tfmorris,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a parent task: T285710: WDQS lag detection required manual adjustment during DC switchover. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: MPhamWMF, Daniel_Mietchen,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T283599: Add support for schema evolution of serialized objects as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: MPhamWMF, Daniel_Mietchen, Thadguidry,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T277443: The streaming updater consumer should log information when divergences are detected as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: MPhamWMF,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T199219: WDQS should use internal endpoint to communicate to Wikidata as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: MPhamWMF, Daniel_Mietchen,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Zbyszko closed subtask T247058: Deployment strategy and hardware requirement for new Flink based WDQS updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: MPhamWMF,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Zbyszko closed subtask T262676: Make the pipeline more robust when backfilling data generated during a DC switch as Declined. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: MPhamWMF,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T286890: Checkpoint _metadata has grown up to 70Mb. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: MPhamWMF, Daniel_Mietchen, Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T275133: Limit query parallelism from Flink based WDQS updater to Wikidata as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: MPhamWMF, Daniel_Mietchen,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: MPhamWMF, Daniel_Mietchen, Thadguidry, tfmorris, revi, Ladsgroup, Multichill, darthmon_wmde,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T266470: Expose wdqs1009 to wdqs users and gather feedback as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: MPhamWMF, Daniel_Mietchen, Thadguidry,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T276750: Add means to upgrade the flink code even when incompatible serialization changes are involved as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T279541: Add a reconciliation strategy to the wdqs streaming updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: MPhamWMF, Daniel_Mietchen, Thadguidry,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

MPhamWMF added a comment. announcement that new Streaming Updater is available on the test server here: https://www.wikidata.org/wiki/Wikidata:Project_chat#New_WDQS_Streaming_Updater_now_available_on_pre-production_test_server_for_feedback TASK DETAIL

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T277443: The streaming updater consumer should log information when divergences are detected. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Daniel_Mietchen,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T276750: Add a mean to upgrade the flink code even when incompatible serialization changes are involved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T276469: Cookbooks and / or operation procedures are adapted for the new Flink based WDQS Streaming Updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T276467: Ensure we have proper monitoring / alerting on the new Flink based WDQS Streaming Updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T269451: Possible flink optimizations/cleanups as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Daniel_Mietchen, Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T199219: WDQS should use internal endpoint to communicate to Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Daniel_Mietchen, Thadguidry, tfmorris,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T275133: Limit query parallelism from Flink based WDQS updater to Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Daniel_Mietchen, Thadguidry, tfmorris,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T266986: The metrics of the streaming-updater-consumer should be visible in grafana as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Daniel_Mietchen,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T246004: Spike: Can/should Swift be used as Flink checkpoint backend? as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T269451: Possible flink optimizations/cleanups. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T266750: The streaming updater consumer should stop accumulating patches if it cannot handle them as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc:

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T255399: Prepare wdqs1009 to run the streaming updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T256882: The streaming updater should support page undeletes as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T265452: Add a configurable restart strategy to the streaming updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T266318: Clarify dependencies on codehale dropwizards as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T267001: Compute page properties information at munge time. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T266986: The metrics of the streaming-updater-consumer should be visible in grafana. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T266470: Expose wdqs1009 to wdqs users and gather feedback. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T261119: Architecture review of Flink based WDQS Streaming Updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel added a subtask: T266750: The streaming updater consumer should stop accumulating patches if it cannot handle them. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T256875: The streaming updater should support page deletions as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel closed subtask T256949: The streaming updater should support suppressed deletes as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T266321: Determine flink metrics configuration and backend when running from k8s. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T266318: Clarify dependencies on codehale dropwizards. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

dcausse added a subtask: T265452: Add a configurable restart strategy to the streaming updater. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Mstyles reopened subtask T256875: The streaming updater should support page deletions as Open. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Mstyles closed subtask T256875: The streaming updater should support page deletions as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: [Epic] Rework the WDQS updater as an event driven application

Gehel renamed this task from "EPIC: Rework the WDQS updater as an event driven application" to "[Epic] Rework the WDQS updater as an event driven application". TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T262896: http://wikiba.se/ontology#wikiGroup; triples should be marked as linked/unlinked shared statements as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T258240: Refactor Options handling in Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T251096: [WDQS Streaming Updater] Organise module structure as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

dcausse added a subtask: T262896: triples should be marked as "linked/unlinked shared statements". TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T256946: Add a custom watermark generator that supports low rate streams as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T248449: [WDQS Streaming Updater] Add error handling for Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris, revi,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T258687: The streaming updater should read its events from multiple DC streams as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Thadguidry, tfmorris,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

dcausse added a subtask: T262676: Make the pipeline more robust when backfilling data generated during a DC switch. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

dcausse updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Thadguidry, tfmorris, revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

dcausse added a comment. In T244590#6429672 , @Thadguidry wrote: >> - the output of this is a simple event without any data saying: do a diff between rev X and Y, fully delete entity QXYZ, ... > > Is that supposed to be "data

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Thadguidry added a comment. > - the output of this is a simple event without any data saying: do a diff between rev X and Y, fully delete entity QXYZ, ... Is that supposed to be "data saving" ? > rdf diff generation: materialize the command and fetch the data from wikibase and send

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Thadguidry updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Thadguidry Cc: tfmorris, revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3, Smalyshev,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T258684: Use RichAsyncFunction for GenerateEntityDiffPatchOperation as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris, revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel added a subtask: T244341: Stop using blank nodes for encoding SomeValue and OWL constraints in WDQS. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris, revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T257325: Upgrade to flink 1.11 as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris, revi, Mholloway, Ladsgroup, Multichill, darthmon_wmde, Iamamz3,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T257795: GenerateEntityDiffPatchOperationUnitTest fails randomly as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris, revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T248451: [WDQS Streaming Updater] Custom parallelism configuration for Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: tfmorris,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

dcausse added a subtask: T258687: The streaming updater should read its events from multiple DC streams. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: tfmorris, revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

dcausse removed a subtask: T258683: Switch GenerateEntityDiffPatchOperation to a RichFunction. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: tfmorris, revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

dcausse added a subtask: T258684: Use RichAsyncFunction for GenerateEntityDiffPatchOperation. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: tfmorris, revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

dcausse added a subtask: T258683: Switch GenerateEntityDiffPatchOperation to a RichFunction. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: tfmorris, revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Zbyszko added a subtask: T258240: Refactor Options handling in Streaming Updater . TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zbyszko Cc: revi, Mholloway, Ladsgroup, Multichill,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T251275: [WDQS Streaming Updater] Update blazegraph based on the content present in the streaming updater output kafka stream as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T251270: The streaming updater should produce its events to kafka as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T245728: Add a component to generate a diff between two entity revisions as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway, Ladsgroup,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T248464: [WDQS Streaming Updater] Implement ouput format in Streaming Updater as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

Gehel closed subtask T248452: [WDQS Streaming Updater] Deploy and configure Streaming Updater Hadoop YARN as Resolved. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: revi, Mholloway,

[Wikidata-bugs] [Maniphest] T244590: EPIC: Rework the WDQS updater as an event driven application

dcausse added a subtask: T257795: GenerateEntityDiffPatchOperationUnitTest fails randomly. TASK DETAIL https://phabricator.wikimedia.org/T244590 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: revi, Mholloway, Ladsgroup, Multichill,