[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles, Gehel Cc: dcausse, dduvall, jeena, akosiaris, Aklapper, Gehel, Mstyles, Alter-paule, Beast1978,

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

jeena added a comment. To summarize the rest of our meeting on 11/2/2020, we also briefly talked about how to manage the flink part of the image, which is currently downloaded when building with blubber. My and @dduvall's suggestion was to build flink into separate image to use as a base

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Mstyles moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. Mstyles added a comment. I definitely agree that pipeline lib could be used for Java projects, but adding that current functionality with this project is out of scope. We are moving

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

dduvall added a comment. In T265512#6642612 , @Mstyles wrote: > The part that is really important to our team is getting our docker image into the WMF docker registry so that we can use it to deploy our Flink cluster to the WMF

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

akosiaris added a comment. In T265512#6648623 , @Gehel wrote: > > My (limited) understanding of the usual pipelines is that we expect a 1-to-1 mapping between project source code, packaged artifact and deployment. That

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Gehel added a comment. A few additional notes / context in no particular order: `streaming-updater-producer` is strongly coupled with the rest of the wikidata/query/rdf project. It might be possible to externalize it to it's own repo, but it will still have dependencies on the rest of

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Mstyles added a comment. The projects are all related and there are dependencies. Separating the streaming-updater-producer would actually be a significant task. I think that would be a separate project to possibly consider in the future. TASK DETAIL

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

gerritbot added a comment. Change 643312 **merged** by jenkins-bot: [integration/config@master] Follow-up 823287e4f : Fix name of repo to flink-rdf-streaming-updater https://gerrit.wikimedia.org/r/643312

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

gerritbot added a comment. Change 643312 had a related patch set uploaded (by Jforrester; owner: Jforrester): [integration/config@master] Follow-up 823287e4f : Fix name of repo to flink-rdf-streaming-updater

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Stashbot added a comment. Mentioned in SAL (#wikimedia-releng) [2020-11-24T17:11:35Z] Zuul: Install pipeline CI for rdf-streaming-updater T265512 TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

gerritbot added a comment. Change 643297 **merged** by jenkins-bot: [integration/config@master] [rdf-streaming-updater] Add pipeline CI https://gerrit.wikimedia.org/r/643297 TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

gerritbot added a comment. Change 643297 had a related patch set uploaded (by Mstyles; owner: Mstyles): [integration/config@master] add pipeline config to ci https://gerrit.wikimedia.org/r/643297 TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

jeena added a comment. In T265512#6642612 , @Mstyles wrote: > Currently for #2, "Build and verify streaming-updater-producer from source, tracking changes to your project repo." > That already happens in a different process, via a

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Mstyles added a comment. For #1, that's is correct. We are currently downloading Flink from the internet. I don't think that's the best idea long term, but it seems fine for now. Currently for #2, "Build and verify streaming-updater-producer from source, tracking changes to your

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

dduvall added a comment. Just now getting up to speed on this, so please bear with me if I'm not fully understanding the problem. PipelineLib and Blubber are primarily meant for managing CI workflows (building images from project source, executing tests, etc.) that respond to changes

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

gerritbot added a comment. Change 642599 **abandoned** by Mstyles: [wikidata/query/rdf@master] add pipeline config Reason: going to use different repo instead, see 643090 https://gerrit.wikimedia.org/r/642599 TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Mstyles added a comment. Instead of trying to skip CI for every commit, the easiest thing to do would be to move the pipeline directory into its own repo. It's not using any of the code in the current repo anyways. TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: dduvall, jeena, akosiaris, Aklapper, Gehel, Mstyles, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz,

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

akosiaris added subscribers: jeena, dduvall. akosiaris added a comment. In T265512#6637980 , @Mstyles wrote: > @akosiaris it was unclear to me whether we need the promote section in the pipeline config. I'm referring to this:

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Mstyles moved this task from Ready for Development to Needs review on the Discovery-Search (Current work) board. Mstyles added a subscriber: akosiaris. Mstyles added a comment. @akosiaris it was unclear to me whether we need the promote section in the pipeline config. I'm referring to this:

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: Aklapper, Gehel, Mstyles, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz, Hook696,

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

gerritbot added a comment. Change 642599 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] add pipeline config https://gerrit.wikimedia.org/r/642599 TASK DETAIL https://phabricator.wikimedia.org/T265512 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Gehel triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T265512 WORKBOARD https://phabricator.wikimedia.org/project/board/1227/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Aklapper, Gehel, Mstyles,

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Mstyles moved this task from All WDQS-related tasks to Current work on the Wikidata-Query-Service board. Mstyles added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T265512 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL

[Wikidata-bugs] [Maniphest] T265512: Set up Pipeline Configuration in WDQS repo

Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service. TASK DESCRIPTION We need to define the pipeline configuration for WDQS. This is done by creating a config file