[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Lucas_Werkmeister_WMDE added a comment. AFAIK: The “legacy” updater queries the recent changes via the API , then gets the RDF for the edited entities from Special:EntityData, and compares that to the live data in the query service to

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Hannah_Bast added a comment. @Harej Thanks for the quick reply, James! Are you saying that the script is scraping + parsing https://www.wikidata.org/wiki/Special:RecentChanges to obtain the triples to be added and deleted? Or is there a different way to access that page, which gives you the

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Hannah_Bast added a comment. I just looked into https://github.com/wikimedia/wikidata-query-rdf, which provides a tool `runUpdate.sh`. When I run it for a Blazegraph instance with exactly one triple of the form `

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Harej added a comment. In T294133#9531815 , @Hannah_Bast wrote: > 1. Which API is this script (or rather the underlying Update.java

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

lbowmaker removed a project: Data Engineering and Event Platform Team. lbowmaker moved this task from Data Eng/Event Platform to Icebox (not considered in current quarter) on the Data-Engineering board. TASK DETAIL https://phabricator.wikimedia.org/T294133 WORKBOARD

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

JArguello-WMF added a project: Data Engineering and Event Platform Team. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JArguello-WMF Cc: MPhamWMF, KingsleyIdehen, Gehel, Jheald, tfmorris, Sj,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Sj added a subscriber: MPhamWMF. Sj added a comment. Thanks @Gehel for catching the duplicate. Suggested edits for this (merged) issue: - The merged issue should be for **making public the data WDQS uses for live updates**. A specific implementation such as "**...using

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Gehel added a comment. Note that there is additional context in T330521 . TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: KingsleyIdehen,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Gehel merged a task: T330521: Make WDQS update stream public. Gehel added subscribers: Sj, danshick-wmde, tfmorris, Jheald, Gehel, KingsleyIdehen. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

dcausse added a parent task: T330521: Make WDQS update stream public. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Ottomata added a comment. (Oh, past me said this already... :p) TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, EChetty,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Ottomata added a comment. > Here you must consume only one. Actually, this is curious. These are really distinct streams. We probably should have named them differently. We can't just pick eqiad.rdf-streaming-updater.mutation when the user connects to EventStreams in eqiad, because

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Restricted Application added a project: Data-Engineering. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RBrounley_WMF Cc: RBrounley_WMF, Harej, Ottomata, Aklapper, dcausse, EChetty, Invadibot,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

odimitrijevic moved this task from Incoming to Event Platform on the Analytics board. odimitrijevic lowered the priority of this task from "High" to "Medium". TASK DETAIL https://phabricator.wikimedia.org/T294133 WORKBOARD https://phabricator.wikimedia.org/project/board/11/ EMAIL

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

MPhamWMF triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: MPhamWMF Cc: Ottomata, Aklapper, dcausse, EChetty, Invadibot, MPhamWMF, maantietaja, RBrounley_WMF,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Ottomata added a comment. Oh, we'll also want to add create event schema and add it to the schema repo. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Ottomata, Aklapper,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Ottomata added projects: Analytics, Event-Platform. Ottomata added a comment. > Here you must consume only one. Should we expose both? We'll need to declare these streams in EventStreammConfig, likely each as a distinct stream overriding the list of topics that make up the stream.

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Ottomata, Aklapper, dcausse, Invadibot, MPhamWMF, maantietaja, RBrounley_WMF,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

dcausse added a subscriber: Ottomata. dcausse added a project: EventStreams. dcausse updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Ottomata, Aklapper,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

dcausse updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T294133 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: Aklapper, dcausse, MPhamWMF, CBogen, Namenlos314, Gq86, Lucas_Werkmeister_WMDE, EBjune, merbst,

[Wikidata-bugs] [Maniphest] T294133: Expose rdf-streaming-updater.mutation content through EventStreams

dcausse created this task. dcausse added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION As a consumer of the wikidata content I want to be able to have access the same RDF data the WMF WDQS servers use to perform their live updates so