dduvall added a comment.
Just now getting up to speed on this, so please bear with me if I'm not fully
understanding the problem.
PipelineLib and Blubber are primarily meant for managing CI workflows
(building images from project source, executing tests, etc.) that respond to
changes to the project repo (Gerrit events) in which their configuration files
live and result in production deployable images. So this use case may be a bit
incongruent with that model. It could also be workable albeit in a way slightly
against the grain. Just something to keep in mind.
In general, it seems like you're wanting to:
1. Keep a published base image for Flink up to date, tracking some stable
version from upstream.
2. Build and verify `streaming-updater-producer` from source, tracking
changes to your project repo.
3. Build and publish a production deployable image that has Flink installed
via the base image and has `streaming-updater-producer` installed into
`/opt/flink/usrlib`.
Is that right? If so, I suggest:
Managing the Flink base image some other way
--------------------------------------------
Since it seems like you want to bump the version only periodically and based
on a version number, I think maybe `operations/docker-images/production-images`
would be right for that, though I see that @akosiaris already mentioned that as
a possibility with the downside that every change would require SRE review.
Alternatively, you could move the parts related to building the Flink base
image to a separate repo (like you mentioned) and continue to use
PipelineLib/Blubber. Again, Blubber is currently very opinionated and geared
toward runnable applications built from local source, not for base images that
rely on a lot of externally download stuff, but it could work. If you got this
route, you'd probably want to create a periodic or manual CI job that invokes
the PipelineLib pipeline since it's not responding to change particulate repo
changes in Gerrit.
Building and testing `streaming-updater-producer` via Blubber/PipelineLib
-------------------------------------------------------------------------
Looks like you're currently fetching the jar from Archiva which has been
built by some previous CI job. This is actually the part I would suggest using
Blubber for.
Something like:
version: v4
base: docker-registry.wikimedia.org/releng/maven-java8
variants:
build:
builder:
command: [mvn, clean, compile]
requirements: [.]
entrypoint: [mvn] # allow for generic use of this variant in running
tests, etc.
A `.pipeline/config.yaml` that defines a test pipeline for this would look
something like:
pipelines:
test: # scheduled to run during test (CR submission) and gate-and-submit
(CR+2)
blubberfile: blubber.yaml
stages:
- name: build-and-verify
build: build
run:
arguments: verify # mvn verify
Integrating and verifying the production deployable image using
Blubber/PipelineLib
-----------------------------------------------------------------------------------
Expand your `.pipeline/blubber.yaml` to include a production variant that
integrates the Flink base image and your packaged JAR.
version: v4
base: docker-registry.wikimedia.org/releng/maven-java8
variants:
# [build variant from before]
prep:
includes: [build]
builder:
command: [mvn, clean, package]
production:
base: docker-registry.wikimedia.org/flink:3.2.1 # the base image for
flink that you've published elsewhere
copies:
- from: prep
source: /srv/app/path/to/your/jar
destination: /opt/flink/usrlib/
And expand the `.pipeline/config.yaml` to include publishing:
pipelines:
# [omitted test pipeline from before]
publish: # scheduled to run post-merge
blubberfile: blubber.yaml
stages:
- name: publish
build: production
publish:
image: true
This is where you could also include a test deployment to staging with
end-to-end tests and/or an automated version bump in deployment-charts via
`promote`.
TASK DETAIL
https://phabricator.wikimedia.org/T265512
EMAIL PREFERENCES
https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: Mstyles, dduvall
Cc: dduvall, jeena, akosiaris, Aklapper, Gehel, Mstyles, Alter-paule,
Beast1978, CBogen, Un1tY, Akuckartz, Hook696, Kent7301, joker88john, CucyNoiD,
Nandana, Namenlos314, Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420,
Bsandipan, Lucas_Werkmeister_WMDE, GoranSMilovanovic, QZanden, EBjune, merbst,
LawExplorer, Lewizho99, Maathavan, _jensen, rosalieper, Scott_WUaS, Jonas,
Xmlizer, jkroll, Wikidata-bugs, Jdouglas, aude, Tobias1984, Manybubbles, Mbch331
_______________________________________________
Wikidata-bugs mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
