[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles, Gehel Cc: hashar, Zbyszko, akosiaris, Ottomata, dcausse, Aklapper, Gehel, Mstyles, Alter-paule,

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Mstyles moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board. Mstyles added a comment. blubberfile is done and the docker image is present in the wikimedia docker repository TASK DETAIL https://phabricator.wikimedia.org/T265504 WORKBOARD

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

gerritbot added a comment. Change 643090 **merged** by jenkins-bot: [wikidata/query/flink-rdf-streaming-updater@master] move pipeline directory to separate repo https://gerrit.wikimedia.org/r/643090 TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

gerritbot added a subscriber: hashar. gerritbot added a comment. Change 643378 **abandoned** by Mstyles: [integration/config@master] remove test stage Reason: @Hashar thank you for explaining the necessity of the test step. I will put a test in!

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

gerritbot added a comment. Change 643378 had a related patch set uploaded (by Mstyles; owner: Mstyles): [integration/config@master] remove test stage https://gerrit.wikimedia.org/r/643378 TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

gerritbot added a comment. Change 635074 **abandoned** by Mstyles: [wikidata/query/rdf@master] add pipeline directory Reason: going to use different repo instead, see https://gerrit.wikimedia.org/r/c/wikidata/query/flink-rdf-streaming-updater/ /643090

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

gerritbot added a comment. Change 643090 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/flink-rdf-streaming-updater@master] move pipeline directory to separate repo https://gerrit.wikimedia.org/r/643090 TASK DETAIL

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

akosiaris added a comment. In T265504#6615197 , @Mstyles wrote: > @akosiaris I started using the new Java images that you uploaded. I wasn't able to install gpg in the build process. There are some conflicts. We can skip gpg

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Mstyles added a comment. @akosiaris I started using the new Java images that you uploaded. I wasn't able to install gpg in the build process. There are some conflicts. We can skip gpg verification of the Flink tar, but I don't think that's a good idea. I will continue to do some debugging.

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

gerritbot added a comment. Change 635743 **merged** by Alexandros Kosiaris: [operations/docker-images/production-images@master] Add new java images https://gerrit.wikimedia.org/r/635743 TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

akosiaris added a comment. In T265504#6598430 , @Mstyles wrote: > @akosiaris when you get some time, can you please take another look at https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/635074 Yes, I will. This hasn't fallen

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Mstyles added a comment. @akosiaris when you get some time, can you please take another look at https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/635074 TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Zbyszko added a comment. > I hope my answer helps as well. Yes it did, thank you! It might've all came from the fact I wasn't present during your first meeting, but know I have much better perspective on how to review the code. TASK DETAIL https://phabricator.wikimedia.org/T265504

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

akosiaris added a comment. In T265504#6580948 , @Zbyszko wrote: >> Could you elaborate on that a bit? > > Sure, here goes: We are using Apache Flink[1] as a platform for our event processing we do to feed Wikidata Query Service.

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Zbyszko added a comment. > Could you elaborate on that a bit? Sure, here goes: We are using Apache Flink[1] as a platform for our event processing we do to feed Wikidata Query Service. We've want to move to Flink deployment to Kubernetes, hence this ticket. Apache Flink provides it's

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

akosiaris added a comment. In T265504#6580645 , @Zbyszko wrote: > @akosiaris I see, makes sense. I still would like to solve the issue with replicating the original dockerfile - can we deploy Flink images to our registry - even if we'd

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Zbyszko added a comment. @akosiaris I see, makes sense. I still would like to solve the issue with replicating the original dockerfile - can we deploy Flink images to our registry - even if we'd need to fork Flink docker repo? TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

akosiaris added a comment. In T265504#6578280 , @Zbyszko wrote: > @akosiaris Can we base a blubber enabled project on a 3rd party docker image, provided on docker hub? I was wondering if we have to replicate original dockerfile here

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Zbyszko added a comment. @akosiaris Can we base a blubber enabled project on a 3rd party docker image, provided on docker hub? I was wondering if we have to replicate original dockerfile here (I'd rather base of their image to reduce future maintenance). TASK DETAIL

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

gerritbot added a comment. Change 635743 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris): [operations/docker-images/production-images@master] Add new java images https://gerrit.wikimedia.org/r/635743 TASK DETAIL

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Mstyles added a comment. The original Flink dockerfile exposes ports, which is not an option in blubber. I'm hoping that any networking/ports issues can be resolved

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Mstyles added a comment. In order to test the image created by the blubberfile, do the following in the rdf repo `cd .pipeline` `blubber blubber.yaml production | docker build --tag blubber-flink-test- --file - .` -> this builds the image and tags it `docker run blubber-flink-test-`

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles, gerritbot Cc: Aklapper, Gehel, Mstyles, Alter-paule, Beast1978, CBogen, Un1tY, Akuckartz,

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

gerritbot added a comment. Change 635074 had a related patch set uploaded (by Mstyles; owner: Mstyles): [wikidata/query/rdf@master] add pipeline directory https://gerrit.wikimedia.org/r/635074 TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Mstyles claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, darthmon_wmde, Nandana, Namenlos314, Lahi, Gq86,

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Mstyles added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T265504 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Mstyles Cc: Aklapper, Gehel, Mstyles, CBogen, Akuckartz, darthmon_wmde, Nandana,

[Wikidata-bugs] [Maniphest] T265504: Create Blubberfile in WDQS repo

Mstyles created this task. Mstyles added projects: Wikidata, Wikidata-Query-Service. TASK DESCRIPTION We need to create a blubberfile in order to use the WMF pipeline to deploy to Kubernetes. Blubberfiles are higher level