[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Ottomata closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata Cc: Zbyszko, Mholloway, MPhamWMF, Gehel, Aklapper, JAllemandou, dcausse, Mstyles, Ottomata, BTulli

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Zbyszko added a comment. @Ottomata thanks! TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, Zbyszko Cc: Zbyszko, Mholloway, MPhamWMF, Gehel, Aklapper, JAllemandou, dcausse, Mstyles, O

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Ottomata added a project: Analytics-Kanban. Ottomata added a comment. Yes! We've done this now that we are using Gobblin instead of Camus. Moving this to our Kanban so we can ACK and close it this week. TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://pha

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Zbyszko added a comment. @Ottomata can this be closed? TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, Zbyszko Cc: Zbyszko, Mholloway, MPhamWMF, Gehel, Aklapper, JAllemandou, dcausse,

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-07-08T14:52:29Z] Synchronized wmf-config/InitialiseSettings.php: Add consumers.analytics_hadoop-ingestion stream config settings for automated gobblin imports - T271232 T2739

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668124 **merged** by Ottomata: [operations/mediawiki-config@master] Add a consumers.analytics-hadoop setting to automate ingestion of streams into HDFS https://gerrit.wikimedia.org/r/668124 TASK DETAIL https://phabricator.wikimedia.org/T273901 EM

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 701177 **merged** by Ottomata: [wikimedia-event-utilities@master] Support getting EventStreamConfig settings by JsonPointer path https://gerrit.wikimedia.org/r/701177 TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 701177 had a related patch set uploaded (by Ottomata; author: Ottomata): [wikimedia-event-utilities@master] Support getting EventStreamConfig settings by JsonPointer path https://gerrit.wikimedia.org/r/701177 TASK DETAIL https://phabricator.wikim

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668144 **merged** by Ottomata: [operations/deployment-charts@master] eventgate-main should also use schemas/event/secondary repo https://gerrit.wikimedia.org/r/668144 TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668144 had a related patch set uploaded (by Ottomata; owner: Ottomata): [operations/deployment-charts@master] eventgate-main should also use schemas/event/secondary repo https://gerrit.wikimedia.org/r/668144 TASK DETAIL https://phabricator.wikimedia.

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668139 **merged** by Ottomata: [operations/deployment-charts@master] Bump to 2021-03-03-172637-production to get rdf_streaming_updater schemas https://gerrit.wikimedia.org/r/668139 TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFEREN

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668139 had a related patch set uploaded (by Ottomata; owner: Ottomata): [operations/deployment-charts@master] Bump to 2021-03-03-172637-production to get rdf_streaming_updater schemas https://gerrit.wikimedia.org/r/668139 TASK DETAIL https://phabrica

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668135 **merged** by Ottomata: [eventgate-wikimedia@master] Bump schemas/event/secondary sha to get rdf_streaming_updater schemas https://gerrit.wikimedia.org/r/668135 TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https:/

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668135 had a related patch set uploaded (by Ottomata; owner: Ottomata): [eventgate-wikimedia@master] Bump schemas/event/secondary sha to get rdf_streaming_updater schemas https://gerrit.wikimedia.org/r/668135 TASK DETAIL https://phabricator.wikimedia

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-03-03T17:13:55Z] Synchronized wmf-config/InitialiseSettings.php: Set destination_event_serivce: eventgate-main for rdf-streaming-updater streams - T273901 (duration: 01m 08s)

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668131 **merged** by Ottomata: [operations/mediawiki-config@master] Set destination_event_serivce: eventgate-main for rdf-streaming-updater streams https://gerrit.wikimedia.org/r/668131 TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PRE

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668131 had a related patch set uploaded (by Ottomata; owner: Ottomata): [operations/mediawiki-config@master] Set destination_event_serivce: eventgate-main for rdf-streaming-updater streams https://gerrit.wikimedia.org/r/668131 TASK DETAIL https://pha

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Ottomata added a subscriber: Mholloway. Ottomata added a comment. Ok here's the idea: I add a new EventStreamConfig settings block called `consumers`, where we can add consumers by name, and then put in relevant settings for them. Those consumers would be responsible for using those set

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668124 had a related patch set uploaded (by Ottomata; owner: Ottomata): [operations/mediawiki-config@master] Add a consumers.analytics-hadoop setting to automate ingestion of streams intod HDFS https://gerrit.wikimedia.org/r/668124 TASK DETAIL https:

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2021-03-03T16:28:07Z] Synchronized wmf-config/InitialiseSettings.php: canary_events_enabled: true for rdf-streaming-updater streams - T273901 (duration: 01m 49s) TASK DETAIL ht

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668119 **merged** by jenkins-bot: [operations/mediawiki-config@master] Set canary_events_enabled: true for rdf-streaming-updater streams https://gerrit.wikimedia.org/r/668119 TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, gerritbot Cc: MPhamWMF, Gehel, Aklapper, JAllemandou, dcausse, Mstyles, Ottomata, Alter-paule

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

gerritbot added a comment. Change 668119 had a related patch set uploaded (by Ottomata; owner: Ottomata): [operations/mediawiki-config@master] Set canary_events_enabled: true for rdf-streaming-updater streams https://gerrit.wikimedia.org/r/668119 TASK DETAIL https://phabricator.wikim

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

MPhamWMF added a comment. For additional context, we are planning on deploying the new Streaming Updater on March 15 TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Ottomata, MPhamWMF Cc: MPha

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Gehel added a project: Discovery-Search (Current work). Gehel moved this task from Tracking to Current work on the Wikidata-Query-Service board. TASK DETAIL https://phabricator.wikimedia.org/T273901 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES https://p

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Gehel added a comment. @Ottomata this is going to block the deployment of the WDQS Flink based Streaming Updater. Any chance you could raise the priority? TASK DETAIL https://phabricator.wikimedia.org/T273901 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailprefere

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

fdans moved this task from Incoming to Event Platform on the Analytics board. fdans triaged this task as "Medium" priority. TASK DETAIL https://phabricator.wikimedia.org/T273901 WORKBOARD https://phabricator.wikimedia.org/project/board/11/ EMAIL PREFERENCES https://phabricator.wikimedia.or

[Wikidata-bugs] [Maniphest] T273901: Automate event stream ingestion into HDFS for streams that don't use EventGate

Ottomata created this task. Ottomata added projects: Wikidata-Query-Service, Analytics, Event-Platform. Restricted Application added a subscriber: Aklapper. Restricted Application added a project: Wikidata. TASK DESCRIPTION Currently, we configure Camus jobs to import events based on which event