Hi Joe, Thanks for confirming my suspicions. I have seen NIFIREG-260 <https://jira.apache.org/jira/browse/NIFIREG-260> and the associated PR on GitHub <https://github.com/apache/nifi-registry/pull/177>, but it’s been almost a year since the last time the PR was published to. I think the best way for my team to move forward is to use a single central registry and implement some process for coordinating our efforts. It would be great if Nifi Registry implemented a way to add arbitrary tags to a process group so we can tag which versions have been approved for production / staging / etc. In the meantime I was thinking of maintaining a mapping of flow versions and tags externally and using it in automatic deployment scripts. So, for example I can spin up a production environment of my product using the production-tagged version of a process group rather than the latest. This is better than what we have currently, but won't have the concept of a release branch.
I also saw there was the concept of registry aliases. I think this might help with pushing nested versioned flows from one registry to the other. I have a hunch that this approach will uncover more issues. My hunch is that the bucketID and flowID will be different and if I change the definition of the alias I will get errors like bucket XXXXX not found. Another issue I see would be that if Version X of a nested flow might mean something different on the destination registry than on the source registry since versions are autoincremented and are not in sync from registry to registry. In the next couple of weeks i'll investigate this. Having multiple registries would make us feel safer about deploying flows to production, but we want to validate the correctness first. -Eric On Mon, Feb 24, 2020 at 10:45 PM Joe Witt <[email protected]> wrote: > Eric > > You wrote a great email and all your points make a lot of sense. I'll be > blunt and state - we didn't design for that and we should have. > > When we launched the NiFi Registry we were seeing a lot of large scale > monolith NIFi clusters being deployed where each cluster ran the flows of > many different teams and/or use cases. We did not expect though how > common/popular it would be that multiple flow developers would be working > within the same process group. But that is definitely what they want to do > today. > > We thought the notion of branching would be useful but not nearly as > quickly as it has been. We thought multiple registries and sharing across > them would be of interest but again just not this fast. > > The ability to support versioned flows which reference other versioned > flows - your TLDR was pretty darn spot on. > > We solved the problem we saw in front of us with the NiFi Registry. What > we didn't see/anticipate, in my opinion was just how much pent up > demand/need there was for a true full git-like versioning experience for > flow management in NiFi. > > I think there are quite a few folks focused in this area so you can expect > some strong improvements here just not clear on when. > > In the mean time I'd tell you to the extent you can build around a central > NiFi Registry, compose versioned flows of other versioned flows, and limit > authorship of a versioned flow to a single author at any one time unless > the other author is going to fork from a particular version to make a new > versioned flow. > > Again great e-mail. Your expectation of what it should do is very > fair/right. > > Thanks > Joe > > On Mon, Feb 24, 2020 at 10:32 PM Eric Secules <[email protected]> wrote: > >> Hello everyone, >> >> Im starting to use nifi and nifi registry on my development team and >> we're running into issues working together on the same versioned process >> groups. >> The nifi registry doesn't support branching, merging and code review >> nativly so we all have ended up developing on the same branch on the same >> registry instance and doing in-person peer review. Is there a better way >> for teams to develop process groups for nifi? >> >> What we've tried: >> I tried to set up my own registry on my local machine where I do >> development and make incremental changes. Then when I am ready to "merge" I >> import the process group to the central registry from my local registry. >> The main issue with this is that there's no mechanism for merge if the >> central registry and my local registry have diverged. The other issue is >> when a versioned process group containing other versioned process groupss >> is moved from local-reg to central-reg the inner PGs still say their source >> is local-reg despite the containing PG moving from local-reg to >> central-reg. This becomes a problem in production environments which would >> only be connected to central-reg. Tldr; Moving nested versioned flows >> between registries is complicated. >> >> I've also tried backing up my local registry to a separate branch in git >> and manually merging it with the branch that central-reg backs up to, but >> these git branches are glorified backups and the registry doesn't seem to >> be built to pull updates from them. On top of that doing a code review on >> the generated JSON describing a process group is difficult and I ran into >> several merge conflicts testing out a very simple merge where the target >> branch diverged slightly from the feature branch. >> >> Does anyone have any other approaches that have succeeded on teams with >> multiple people developing on the same set of process groups? >> >> Thanks, >> Eric >> >
