Eric You wrote a great email and all your points make a lot of sense. I'll be blunt and state - we didn't design for that and we should have.
When we launched the NiFi Registry we were seeing a lot of large scale monolith NIFi clusters being deployed where each cluster ran the flows of many different teams and/or use cases. We did not expect though how common/popular it would be that multiple flow developers would be working within the same process group. But that is definitely what they want to do today. We thought the notion of branching would be useful but not nearly as quickly as it has been. We thought multiple registries and sharing across them would be of interest but again just not this fast. The ability to support versioned flows which reference other versioned flows - your TLDR was pretty darn spot on. We solved the problem we saw in front of us with the NiFi Registry. What we didn't see/anticipate, in my opinion was just how much pent up demand/need there was for a true full git-like versioning experience for flow management in NiFi. I think there are quite a few folks focused in this area so you can expect some strong improvements here just not clear on when. In the mean time I'd tell you to the extent you can build around a central NiFi Registry, compose versioned flows of other versioned flows, and limit authorship of a versioned flow to a single author at any one time unless the other author is going to fork from a particular version to make a new versioned flow. Again great e-mail. Your expectation of what it should do is very fair/right. Thanks Joe On Mon, Feb 24, 2020 at 10:32 PM Eric Secules <[email protected]> wrote: > Hello everyone, > > Im starting to use nifi and nifi registry on my development team and we're > running into issues working together on the same versioned process groups. > The nifi registry doesn't support branching, merging and code review > nativly so we all have ended up developing on the same branch on the same > registry instance and doing in-person peer review. Is there a better way > for teams to develop process groups for nifi? > > What we've tried: > I tried to set up my own registry on my local machine where I do > development and make incremental changes. Then when I am ready to "merge" I > import the process group to the central registry from my local registry. > The main issue with this is that there's no mechanism for merge if the > central registry and my local registry have diverged. The other issue is > when a versioned process group containing other versioned process groupss > is moved from local-reg to central-reg the inner PGs still say their source > is local-reg despite the containing PG moving from local-reg to > central-reg. This becomes a problem in production environments which would > only be connected to central-reg. Tldr; Moving nested versioned flows > between registries is complicated. > > I've also tried backing up my local registry to a separate branch in git > and manually merging it with the branch that central-reg backs up to, but > these git branches are glorified backups and the registry doesn't seem to > be built to pull updates from them. On top of that doing a code review on > the generated JSON describing a process group is difficult and I ran into > several merge conflicts testing out a very simple merge where the target > branch diverged slightly from the feature branch. > > Does anyone have any other approaches that have succeeded on teams with > multiple people developing on the same set of process groups? > > Thanks, > Eric >
