Hi Philipp, yes, what you are proposing sounds good from my side. We got an access token configured by Infra and I played a little bit with Github Actions and it's great fun. I also asked Infra to setup Nexus, so I'd propose to try the approach you were outlining thereafter.
Concerning your last point, AFAIK we shouldn't do releases from a CI pipeline as we did in the pre-ASF era. Instead, a release manager from the community would lead the preparation of the release, call the vote and is also responsible for pushing convenience libraries to our channels after a successful vote. I hope that Justin will comment in case I'm wrong here. Dominik -----Original Message----- From: Philipp Zehnder <[email protected]> Sent: Thursday, December 12, 2019 4:17 PM To: [email protected] Subject: Re: CI Pipelines Hi Dominik, I would also prefer to have a separate Docker Hub Repo. Regarding the build process: Would it be possible use Jenkins to push the SNAPSHOTs into Nexus (for PE developers) and a separate build pipeline with for example with GitHub actions which first builds the .jar files and then the containers without pulling or pushing to Nexus? For Releases we could have a separate build pipeline which we can trigger manually to build ‘release docker images’ directly with the artifacts from Nexus. Cheers, Philipp > On 11. Dec 2019, at 22:55, Dominik Riemer <[email protected]> wrote: > > Hi, > > I'm currently trying to figure out what could be the best CI strategy > in the ASF infrastructure. > > > > For StreamPipes development, the following things are important for a > good development experience: > > - Users with different roles (e.g., UI dev, backend dev, pipeline > element dev) should be able to work independently on components > without needing to setup the whole development stack > > - We currently manage this in Docker, e.g., when you are developing > on the UI, all other components and services run in Docker, so that UI > developers don't need to care about how to setup a Maven project and > vice versa > > - When developing pipeline elements (which are in the repo > incubator-streampipes-extensions), SNAPSHOTs of incubator-streampipes > artifacts need to be fetched from a Maven repository, so that pipeline > element development can happen based on the latest SNAPSHOT version > > > > Currently, we use a build pipeline in Gitlab CI as follows (taking the > example of the core at incubator-streampipes): > > - Build Maven artifacts > > - Build UI > > - Build Docker Images and deploy these images to Docker Hub > > - Deploy Maven artifacts to Nexus > > This was quite a smooth and integrated process. > > > > So after having talked to Chris and INFRA, it seems that only Jenkins > is allowed to publish to the Apache Maven Nexus, but Jenkins can only > push to the official Apache Docker Hub account. As our development is > heavily focused around Docker (we are building around 20 Docker Images > for the individual services), I'd personally prefer to use the 2nd way > proposed in the Docker Hub release policy ( > <https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGui > deline > s> > https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuid > elines , e.g., having an apache-streampipes organization account on > Docker Hub) as it gives users a much better overview on what images > are currently available. However, this won't work from Jenkins as it's > not possible to push to a non-ASF-namespace. > > > > I played around a little bit with Jenkins pipelines and while > Docker-based builds are generally supported, I'm not yet 100% sure how > easy it is to share artifacts over stages. > > > > So I guess we have two options, both approaches have pros and cons: > > - Use Jenkins for the complete pipeline and publish our images to the > ASF Docker Hub > > - Use Jenkins only to build and publish the Maven artifacts to the > ASF Nexus and some other CI pipeline outside of Jenkins (Travis or > Github > Actions) to push to an apache-streampipes account on Docker Hub > > > > What do you think? Maybe there's also a better option I'm overlooking > ;-) > > > > Dominik > > >
