Aldrin, +1 to separate repository (bullet #2). The basic premise that Docker releases should happen separate from the main distribution is spot on. I think a separate repository would help keep this separation.
I tend to believe that the future of NiFi distributions will be via containerization. Making the Docker components somewhat a standalone initiative will help drive changes and innovation in this area. I'd like to help see Docker become a first-class citizen for distributing, running and upgrading NiFi. Thanks, Adam On Thu, Sep 21, 2017 at 9:11 AM, Aldrin Piri <aldrinp...@gmail.com> wrote: > Hey folks, > > ** This message turned out to be more detailed than anticipated. In > summary, I propose consolidating Docker/container work with a separate > release process outside of the repository they are packaging. Full > thoughts and background follow. Any input would be appreciated! > > --- > > I've been working through providing some additional Docker capabilities for > the project and wanted to share some thoughts as well as possible plans to > help us be a bit more nimble and responsive to curating Dockerfiles and > their respective images on DockerHub. > > As a bit of context, we currently have the core NiFi project captured in > two Dockerfiles, one that is used in conjunction with a Maven plugin for > creating an image during the NiFi build (dockermaven), and another that is > used for building tagged releases on Docker Hub (dockerhub). Both of these > artifacts, currently, reside in a nifi-docker project and are activated via > Maven profile, (-P docker). > > We've seen at times that this is a very coupled process and limits our > flexibility. For instance, we had an ill-placed 'chown' which caused a > duplicating layer and causes our image to be doubly large. While this has > been remedied, given current release processes, this is included with the > core nifi release and we have been unable to rectify that issue. > > Another issue is a very specific sequence of actions that needs to happen > with the current release for artifacts to be triggered correctly in Docker > Hub. This can be seen in Section 6 of the release guide [1]. While there > are ways to rectify this if the timing isn't quite right and/or an error is > made, it can impose an additional burden on the INFRA team to facilitate > these requests as there currently is no capability for PMCs to manage their > Docker repositories directly. > > Ultimately, I think we should consider a separate release process for NiFi > Docker, and any associated efforts that may relate to those files. In this > context, NiFi is encompassing of all projects/efforts in the project. > Additional efforts could comprise of examples of configuring NiFi to be > secured or clustered, receive data from MiNiFi instances, or using Docker > Compose or other orchestration frameworks. I have also noticed a number of > different areas across our work that are using Docker for integration > testing purposes. With some planning and coordination, we could likely > consolidate many of these core resources/templates to allow us to reuse > them across efforts. > > I believe there are two approaches from an organizational standpoint that > let us execute on the separate release process effectively: > > 1.) Condense all Docker artifacts into the current NiFi repository [2]. We > update our release for NiFi to exclude the Docker subtree to carry out our > normal release flow and provide the build/tooling for the Docker subtree to > be released on its own. > > 2.) Establish a new git repository to handle Docker and any other > containerization efforts and migrate all existing resources into a file > structure that makes sense. > > My inclination is toward (2). > > Regardless of path chosen above, this frees us to handle updates and > improvements to container efforts when needed. Any time we wanted to > release updates to Docker images, we could perform a separate release on > either the subtree of (1) or the repository of (2) and reference the > associated latest artifacts of NiFi. > > If you've made it this far, thanks for working through the wall of text and > would appreciate any thoughts or comments. > > [1] http://nifi.apache.org/release-guide.html > [2] https://git-wip-us.apache.org/repos/asf?p=nifi.git >
