[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles, Gehel Cc: Aklapper, dcausse, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301,

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles reopened this task as "Open". TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, dcausse, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696, darthmon_wmde,

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. Mstyles closed this task as "Resolved". Mstyles added a comment. All patches are merged and items can be deleted successfully TASK DETAIL https://phabricator.wikimedia.org/T256875

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 630855 **merged** by jenkins-bot: [wikidata/query/rdf@master] Add some UpdatePatchAccumulator unit tests https://gerrit.wikimedia.org/r/630855 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles added a comment. Delete functionality is complete, optimization work in progress to ensure that patches created on the streaming updater consumer that get sent to blazegraph do not contain statements from entities that will be deleted TASK DETAIL

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 630855 had a related patch set uploaded (by Mstyles; owner: DCausse): [wikidata/query/rdf@master] Add some UpdatePatchAccumulator unit tests https://gerrit.wikimedia.org/r/630855 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 629765 **merged** by jenkins-bot: [wikidata/query/rdf@master] delete entity integration test https://gerrit.wikimedia.org/r/629765 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 629234 **merged** by jenkins-bot: [wikidata/query/rdf@master] remove entities from blazegraph https://gerrit.wikimedia.org/r/629234 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles added a comment. Wrapping up the last phase of deleting an entity from blazegraph. I have verified that items to be deleted are present in the streaming updater output. TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 629765 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] delete entity integration test https://gerrit.wikimedia.org/r/629765 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 629234 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] remove entities from blazegraph https://gerrit.wikimedia.org/r/629234 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 623452 **merged** by jenkins-bot: [wikidata/query/rdf@master] refine unit tests https://gerrit.wikimedia.org/r/623452 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 621356 **merged** by jenkins-bot: [wikidata/query/rdf@master] add page delete to updater pipeline https://gerrit.wikimedia.org/r/621356 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 623452 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] refine unit tests https://gerrit.wikimedia.org/r/623452 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 623035 **abandoned** by Mstyles: [wikidata/query/rdf@master] allow deleted object to be created Reason: I misunderstood the intended logic https://gerrit.wikimedia.org/r/623035 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 623035 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] allow deleted object to be created https://gerrit.wikimedia.org/r/623035 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles added a comment. Wrapping up the flink pipeline work, a patch is out for the integration tests and there's some more test cases to be added for clarity on the Decide Mutation Operation to ensure that all delete cases are covered. Pipeline consumer work is up next TASK DETAIL

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 621349 **merged** by jenkins-bot: [wikidata/query/rdf@master] use SuccessfulOp as higher level abstraction https://gerrit.wikimedia.org/r/621349 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 618634 **merged** by jenkins-bot: [wikidata/query/rdf@master] Adds delete case to DecideMutationOperation https://gerrit.wikimedia.org/r/618634 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 621356 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] add revision create to updater pipeline https://gerrit.wikimedia.org/r/621356 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 621349 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] use SuccessfulOp as higher level abstraction https://gerrit.wikimedia.org/r/621349 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles added a comment. Clearing the state is out of the question, I think. Right now, we mark the revision we see a delete at and then we don't allow a Rev Create Event. However, this means that any regular RevCreate events don't allow the item to be reimported ever again. I think it

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

dcausse added a comment. In T256875#6364443 , @Mstyles wrote: > What do we want to do if we get a PageDelete event for the same item multiple times? Right now we're not tracking deletes, so I'm assuming multiple deletes for the same

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 618159 **merged** by jenkins-bot: [wikidata/query/rdf@master] Make Deserialization Schema generic https://gerrit.wikimedia.org/r/618159 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 617761 **merged** by jenkins-bot: [wikidata/query/rdf@master] Add page delete stream to incoming streams https://gerrit.wikimedia.org/r/617761 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 618634 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] Adds delete case to DecideMutationOperation https://gerrit.wikimedia.org/r/618634 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles added a comment. What do we want to do if we get a PageDelete event for the same item multiple times? Right now we're not tracking deletes, so I'm assuming multiple deletes for the same item probably can't do that much harm. TASK DETAIL https://phabricator.wikimedia.org/T256875

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 618159 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] Make Deserialization Schema generic https://gerrit.wikimedia.org/r/618159 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles added a comment. Shared model delete functionality is complete, currently in the process of adding delete functionality to the flink pipeline TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles, gerritbot Cc: Aklapper, dcausse, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696,

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 617761 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] Add page delete stream to incoming streams https://gerrit.wikimedia.org/r/617761 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles, Maintenance_bot Cc: Aklapper, dcausse, CBogen, Akuckartz, darthmon_wmde, Nandana,

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 616633 **merged** by jenkins-bot: [wikidata/query/rdf@master] add delete operation https://gerrit.wikimedia.org/r/616633 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles, gerritbot Cc: Aklapper, dcausse, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696,

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

gerritbot added a comment. Change 616633 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] add delete operation https://gerrit.wikimedia.org/r/616633 TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, dcausse, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314,

[Wikidata-bugs] [Maniphest] T256875: The streaming updater should support page deletions

Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T256875 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, dcausse, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86,