Hi Dominik, I would also prefer to have a separate Docker Hub Repo.
Regarding the build process: Would it be possible use Jenkins to push the SNAPSHOTs into Nexus (for PE developers) and a separate build pipeline with for example with GitHub actions which first builds the .jar files and then the containers without pulling or pushing to Nexus? For Releases we could have a separate build pipeline which we can trigger manually to build ‘release docker images’ directly with the artifacts from Nexus. Cheers, Philipp > On 11. Dec 2019, at 22:55, Dominik Riemer <rie...@apache.org> wrote: > > Hi, > > I'm currently trying to figure out what could be the best CI strategy in the > ASF infrastructure. > > > > For StreamPipes development, the following things are important for a good > development experience: > > - Users with different roles (e.g., UI dev, backend dev, pipeline > element dev) should be able to work independently on components without > needing to setup the whole development stack > > - We currently manage this in Docker, e.g., when you are developing > on the UI, all other components and services run in Docker, so that UI > developers don't need to care about how to setup a Maven project and vice > versa > > - When developing pipeline elements (which are in the repo > incubator-streampipes-extensions), SNAPSHOTs of incubator-streampipes > artifacts need to be fetched from a Maven repository, so that pipeline > element development can happen based on the latest SNAPSHOT version > > > > Currently, we use a build pipeline in Gitlab CI as follows (taking the > example of the core at incubator-streampipes): > > - Build Maven artifacts > > - Build UI > > - Build Docker Images and deploy these images to Docker Hub > > - Deploy Maven artifacts to Nexus > > This was quite a smooth and integrated process. > > > > So after having talked to Chris and INFRA, it seems that only Jenkins is > allowed to publish to the Apache Maven Nexus, but Jenkins can only push to > the official Apache Docker Hub account. As our development is heavily > focused around Docker (we are building around 20 Docker Images for the > individual services), I'd personally prefer to use the 2nd way proposed in > the Docker Hub release policy ( > <https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuideline > s> > https://cwiki.apache.org/confluence/display/INCUBATOR/DistributionGuidelines > , e.g., having an apache-streampipes organization account on Docker Hub) as > it gives users a much better overview on what images are currently > available. However, this won't work from Jenkins as it's not possible to > push to a non-ASF-namespace. > > > > I played around a little bit with Jenkins pipelines and while Docker-based > builds are generally supported, I'm not yet 100% sure how easy it is to > share artifacts over stages. > > > > So I guess we have two options, both approaches have pros and cons: > > - Use Jenkins for the complete pipeline and publish our images to the > ASF Docker Hub > > - Use Jenkins only to build and publish the Maven artifacts to the > ASF Nexus and some other CI pipeline outside of Jenkins (Travis or Github > Actions) to push to an apache-streampipes account on Docker Hub > > > > What do you think? Maybe there's also a better option I'm overlooking ;-) > > > > Dominik > > >
