Re: Suggestions for Flow Development Lifestyle

[Emanuel Oliveira]

i know that when we export xml as template, ids are not preserved (what we
have on canvas vs on generated temp[ate xml).
I wonder about ids from deployment from nifi registry are they preserved ?


Best Regards,
*Emanuel Oliveira*



On Tue, Feb 25, 2020 at 8:57 PM Kevin Doran  wrote:

> I’ve always thought along the lines Otto suggests, that eventually, given
> some way of formatting the diff, there would also be some visual tool in
> the ecosystem that would help visualize that diff and could be used
> specifically in the context of reviewing/merging changes.
>
> Lots of good discussion on this thread, thanks all!
>
> Kevin
>
> On Feb 25, 2020, at 15:44, Otto Fowler  wrote:
>
> -or- when diffing, some other representation could be presented other than
> straight xml.
>
> The diff ( if run / visualized in nifi ) needn’t be byte for byte, it can
> be logical can’t it?
>
> On February 25, 2020 at 14:58:24, Eric Secules (esecu...@gmail.com) wrote:
>
> Hi Ken,
>
>
>> I feel like there are also issues in trying to build a pull-request-like
>> workflow, in that most NiFi design is done visually on the canvas but there
>> isn't a good way, that I know of, to represent differences between versions
>> in a visual way. Even if you enabled pull request-like functionality, I
>> don't know that I would expect users to approve changes by examining diffs
>> in the XML files.
>>
>
> I agree, doing PR can be improved by starting up a nifi instance and
> registry on top of the feature branch, so you can view the changes in a
> safe environment before approving. But this doesn't solve the issue of
> merging diverged versions. For that we'd need a visual representation of a
> diff. Maybe entering a 3 way diff mode with side-by-side canvases and no
> active processors where the differences are highlighted. Doing merge
> resolution at the XML level is a non-starter for me.
>
> I've tried out restarting the registry which works for receiving new
> changes. I have also had some trouble with loosing previous version history
> when restarting a registry, but the main issue would still be merging
> diverged branches. Maybe it would help to have some checkout mechanism and
> return to the days of centralized version control if merging is too
> difficult to implement.
>
> -Eric
>
> On Tue, Feb 25, 2020 at 9:36 AM Ken Swanson  wrote:
>
>> I have a lot of interest in this too, as my team has run into these
>> issues as well.
>>
>> I feel like there are also issues in trying to build a pull-request-like
>> workflow, in that most NiFi design is done visually on the canvas but there
>> isn't a good way, that I know of, to represent differences between versions
>> in a visual way. Even if you enabled pull request-like functionality, I
>> don't know that I would expect users to approve changes by examining diffs
>> in the XML files.
>>
>> However, I did have one thing to add that I hope might help:
>> On Tue, Feb 25, 2020 at 12:32 AM Eric Secules  wrote:
>>
>>> I've also tried backing up my local registry to a separate branch in git
>>> and manually merging it with the branch that central-reg backs up to, but
>>> these git branches are glorified backups and the registry doesn't seem to
>>> be built to pull updates from them. On top of that doing a code review on
>>> the generated JSON describing a process group is difficult and I ran into
>>> several merge conflicts testing out a very simple merge where the target
>>> branch diverged slightly from the feature branch.
>>>
>>
>> From what I've seen, the registry will use any updates that are in the
>> repo when it starts up. So you should be able to do something like this:
>>
>>1. Make the registry pull (or clone) from the backup as part of its
>>startup process
>>2. Restart the registry when a merge to master is made. AFAIK this
>>should not affect any NiFi instances that are connected to the registry;
>>they should continue to work once the registry comes back up
>>
>> It's a bit clunky but I think it works.
>>
>> -Ken
>>
>
>

Re: Suggestions for Flow Development Lifestyle

[Kevin Doran]

I’ve always thought along the lines Otto suggests, that eventually, given some 
way of formatting the diff, there would also be some visual tool in the 
ecosystem that would help visualize that diff and could be used specifically in 
the context of reviewing/merging changes.

Lots of good discussion on this thread, thanks all!

Kevin

> On Feb 25, 2020, at 15:44, Otto Fowler  wrote:
> 
> -or- when diffing, some other representation could be presented other than 
> straight xml.
> 
> The diff ( if run / visualized in nifi ) needn’t be byte for byte, it can be 
> logical can’t it? 
> 
> On February 25, 2020 at 14:58:24, Eric Secules (esecu...@gmail.com 
> ) wrote:
> 
>> Hi Ken,
>>  
>> I feel like there are also issues in trying to build a pull-request-like 
>> workflow, in that most NiFi design is done visually on the canvas but there 
>> isn't a good way, that I know of, to represent differences between versions 
>> in a visual way. Even if you enabled pull request-like functionality, I 
>> don't know that I would expect users to approve changes by examining diffs 
>> in the XML files.
>> 
>> I agree, doing PR can be improved by starting up a nifi instance and 
>> registry on top of the feature branch, so you can view the changes in a safe 
>> environment before approving. But this doesn't solve the issue of merging 
>> diverged versions. For that we'd need a visual representation of a diff. 
>> Maybe entering a 3 way diff mode with side-by-side canvases and no active 
>> processors where the differences are highlighted. Doing merge resolution at 
>> the XML level is a non-starter for me. 
>> 
>> I've tried out restarting the registry which works for receiving new 
>> changes. I have also had some trouble with loosing previous version history 
>> when restarting a registry, but the main issue would still be merging 
>> diverged branches. Maybe it would help to have some checkout mechanism and 
>> return to the days of centralized version control if merging is too 
>> difficult to implement.
>> 
>> -Eric
>> 
>> On Tue, Feb 25, 2020 at 9:36 AM Ken Swanson > > wrote:
>> I have a lot of interest in this too, as my team has run into these issues 
>> as well.
>> 
>> I feel like there are also issues in trying to build a pull-request-like 
>> workflow, in that most NiFi design is done visually on the canvas but there 
>> isn't a good way, that I know of, to represent differences between versions 
>> in a visual way. Even if you enabled pull request-like functionality, I 
>> don't know that I would expect users to approve changes by examining diffs 
>> in the XML files.
>> 
>> However, I did have one thing to add that I hope might help:
>> On Tue, Feb 25, 2020 at 12:32 AM Eric Secules > > wrote:
>> I've also tried backing up my local registry to a separate branch in git and 
>> manually merging it with the branch that central-reg backs up to, but these 
>> git branches are glorified backups and the registry doesn't seem to be built 
>> to pull updates from them. On top of that doing a code review on the 
>> generated JSON describing a process group is difficult and I ran into 
>> several merge conflicts testing out a very simple merge where the target 
>> branch diverged slightly from the feature branch.
>> 
>> From what I've seen, the registry will use any updates that are in the repo 
>> when it starts up. So you should be able to do something like this:
>> Make the registry pull (or clone) from the backup as part of its startup 
>> process
>> Restart the registry when a merge to master is made. AFAIK this should not 
>> affect any NiFi instances that are connected to the registry; they should 
>> continue to work once the registry comes back up
>> It's a bit clunky but I think it works.
>> 
>> -Ken

Re: Suggestions for Flow Development Lifestyle

[Otto Fowler]

 -or- when diffing, some other representation could be presented other than
straight xml.

The diff ( if run / visualized in nifi ) needn’t be byte for byte, it can
be logical can’t it?

On February 25, 2020 at 14:58:24, Eric Secules (esecu...@gmail.com) wrote:

Hi Ken,


> I feel like there are also issues in trying to build a pull-request-like
> workflow, in that most NiFi design is done visually on the canvas but there
> isn't a good way, that I know of, to represent differences between versions
> in a visual way. Even if you enabled pull request-like functionality, I
> don't know that I would expect users to approve changes by examining diffs
> in the XML files.
>

I agree, doing PR can be improved by starting up a nifi instance and
registry on top of the feature branch, so you can view the changes in a
safe environment before approving. But this doesn't solve the issue of
merging diverged versions. For that we'd need a visual representation of a
diff. Maybe entering a 3 way diff mode with side-by-side canvases and no
active processors where the differences are highlighted. Doing merge
resolution at the XML level is a non-starter for me.

I've tried out restarting the registry which works for receiving new
changes. I have also had some trouble with loosing previous version history
when restarting a registry, but the main issue would still be merging
diverged branches. Maybe it would help to have some checkout mechanism and
return to the days of centralized version control if merging is too
difficult to implement.

-Eric

On Tue, Feb 25, 2020 at 9:36 AM Ken Swanson  wrote:

> I have a lot of interest in this too, as my team has run into these issues
> as well.
>
> I feel like there are also issues in trying to build a pull-request-like
> workflow, in that most NiFi design is done visually on the canvas but there
> isn't a good way, that I know of, to represent differences between versions
> in a visual way. Even if you enabled pull request-like functionality, I
> don't know that I would expect users to approve changes by examining diffs
> in the XML files.
>
> However, I did have one thing to add that I hope might help:
> On Tue, Feb 25, 2020 at 12:32 AM Eric Secules  wrote:
>
>> I've also tried backing up my local registry to a separate branch in git
>> and manually merging it with the branch that central-reg backs up to, but
>> these git branches are glorified backups and the registry doesn't seem to
>> be built to pull updates from them. On top of that doing a code review on
>> the generated JSON describing a process group is difficult and I ran into
>> several merge conflicts testing out a very simple merge where the target
>> branch diverged slightly from the feature branch.
>>
>
> From what I've seen, the registry will use any updates that are in the
> repo when it starts up. So you should be able to do something like this:
>
>1. Make the registry pull (or clone) from the backup as part of its
>startup process
>2. Restart the registry when a merge to master is made. AFAIK this
>should not affect any NiFi instances that are connected to the registry;
>they should continue to work once the registry comes back up
>
> It's a bit clunky but I think it works.
>
> -Ken
>

Re: Suggestions for Flow Development Lifestyle

[Eric Secules]

Hi Ken,


> I feel like there are also issues in trying to build a pull-request-like
> workflow, in that most NiFi design is done visually on the canvas but there
> isn't a good way, that I know of, to represent differences between versions
> in a visual way. Even if you enabled pull request-like functionality, I
> don't know that I would expect users to approve changes by examining diffs
> in the XML files.
>

I agree, doing PR can be improved by starting up a nifi instance and
registry on top of the feature branch, so you can view the changes in a
safe environment before approving. But this doesn't solve the issue of
merging diverged versions. For that we'd need a visual representation of a
diff. Maybe entering a 3 way diff mode with side-by-side canvases and no
active processors where the differences are highlighted. Doing merge
resolution at the XML level is a non-starter for me.

I've tried out restarting the registry which works for receiving new
changes. I have also had some trouble with loosing previous version history
when restarting a registry, but the main issue would still be merging
diverged branches. Maybe it would help to have some checkout mechanism and
return to the days of centralized version control if merging is too
difficult to implement.

-Eric

On Tue, Feb 25, 2020 at 9:36 AM Ken Swanson  wrote:

> I have a lot of interest in this too, as my team has run into these issues
> as well.
>
> I feel like there are also issues in trying to build a pull-request-like
> workflow, in that most NiFi design is done visually on the canvas but there
> isn't a good way, that I know of, to represent differences between versions
> in a visual way. Even if you enabled pull request-like functionality, I
> don't know that I would expect users to approve changes by examining diffs
> in the XML files.
>
> However, I did have one thing to add that I hope might help:
> On Tue, Feb 25, 2020 at 12:32 AM Eric Secules  wrote:
>
>> I've also tried backing up my local registry to a separate branch in git
>> and manually merging it with the branch that central-reg backs up to, but
>> these git branches are glorified backups and the registry doesn't seem to
>> be built to pull updates from them. On top of that doing a code review on
>> the generated JSON describing a process group is difficult and I ran into
>> several merge conflicts testing out a very simple merge where the target
>> branch diverged slightly from the feature branch.
>>
>
> From what I've seen, the registry will use any updates that are in the
> repo when it starts up. So you should be able to do something like this:
>
>1. Make the registry pull (or clone) from the backup as part of its
>startup process
>2. Restart the registry when a merge to master is made. AFAIK this
>should not affect any NiFi instances that are connected to the registry;
>they should continue to work once the registry comes back up
>
> It's a bit clunky but I think it works.
>
> -Ken
>

Re: Suggestions for Flow Development Lifestyle

[Joe Witt]

Eric

Yep sounds good - you clearly have a good handle on this and a good vision
for where it could be.  Please do engage more on this topic with the team
as we make improvements going forward.

Thanks!

On Tue, Feb 25, 2020 at 11:02 AM Eric Secules  wrote:

> Hi Joe,
>
> Thanks for confirming my suspicions. I have seen NIFIREG-260
>  and the associated PR
> on GitHub , but it’s
> been almost a year since the last time the PR was published to. I think the
> best way for my team to move forward is to use a single central registry
> and implement some process for coordinating our efforts. It would be great
> if Nifi Registry implemented a way to add arbitrary tags to a process group
> so we can tag which versions have been approved for production / staging /
> etc. In the meantime I was thinking of maintaining a mapping of flow
> versions and tags externally and using it in automatic deployment scripts.
> So, for example I can spin up a production environment of my product using
> the production-tagged version of a process group rather than the latest.
> This is better than what we have currently, but won't have the concept of a
> release branch.
>
> I also saw there was the concept of registry aliases. I think this might
> help with pushing nested versioned flows from one registry to the other. I
> have a hunch that this approach will uncover more issues. My hunch is that
> the bucketID and flowID will be different and if I change the definition of
> the alias I will get errors like bucket X not found. Another issue I
> see would be that if Version X of a nested flow might mean something
> different on the destination registry than on the source registry since
> versions are autoincremented and are not in sync from registry to registry.
> In the next couple of weeks i'll investigate this. Having multiple
> registries would make us feel safer about deploying flows to production,
> but we want to validate the correctness first.
>
> -Eric
>
> On Mon, Feb 24, 2020 at 10:45 PM Joe Witt  wrote:
>
>> Eric
>>
>> You wrote a great email and all your points make a lot of sense.  I'll be
>> blunt and state - we didn't design for that and we should have.
>>
>> When we launched the NiFi Registry we were seeing a lot of large scale
>> monolith NIFi clusters being deployed where each cluster ran the flows of
>> many different teams and/or use cases.  We did not expect though how
>> common/popular it would be that multiple flow developers would be working
>> within the same process group.  But that is definitely what they want to do
>> today.
>>
>> We thought the notion of branching would be useful but not nearly as
>> quickly as it has been.  We thought multiple registries and sharing across
>> them would be of interest but again just not this fast.
>>
>> The ability to support versioned flows which reference other versioned
>> flows - your TLDR was pretty darn spot on.
>>
>> We solved the problem we saw in front of us with the NiFi Registry.  What
>> we didn't see/anticipate, in my opinion was just how much pent up
>> demand/need there was for a true full git-like versioning experience for
>> flow management in NiFi.
>>
>> I think there are quite a few folks focused in this area so you can
>> expect some strong improvements here just not clear on when.
>>
>> In the mean time I'd tell you to the extent you can build around a
>> central NiFi Registry, compose versioned flows of other versioned flows,
>> and limit authorship of a versioned flow to a single author at any one time
>> unless the other author is going to fork from a particular version to make
>> a new versioned flow.
>>
>> Again great e-mail.  Your expectation of what it should do is very
>> fair/right.
>>
>> Thanks
>> Joe
>>
>> On Mon, Feb 24, 2020 at 10:32 PM Eric Secules  wrote:
>>
>>> Hello everyone,
>>>
>>> Im starting to use nifi and nifi registry on my development team and
>>> we're running into issues working together on the same versioned process
>>> groups.
>>> The nifi registry doesn't support branching, merging and code review
>>> nativly so we all have ended up developing on the same branch on the same
>>> registry instance and doing in-person peer review. Is there a better way
>>> for teams to develop process groups for nifi?
>>>
>>> What we've tried:
>>> I tried to set up my own registry on my local machine where I do
>>> development and make incremental changes. Then when I am ready to "merge" I
>>> import the process group to the central registry from my local registry.
>>> The main issue with this is that there's no mechanism for merge if the
>>> central registry and my local registry have diverged. The other issue is
>>> when a versioned process group containing other versioned process groupss
>>> is moved from local-reg to central-reg the inner PGs still say their source
>>> is local-reg despite the containing PG moving from local-reg to

Re: Suggestions for Flow Development Lifestyle

[Eric Secules]

Hi Joe,

Thanks for confirming my suspicions. I have seen NIFIREG-260
 and the associated PR on
GitHub , but it’s been
almost a year since the last time the PR was published to. I think the best
way for my team to move forward is to use a single central registry and
implement some process for coordinating our efforts. It would be great if
Nifi Registry implemented a way to add arbitrary tags to a process group so
we can tag which versions have been approved for production / staging /
etc. In the meantime I was thinking of maintaining a mapping of flow
versions and tags externally and using it in automatic deployment scripts.
So, for example I can spin up a production environment of my product using
the production-tagged version of a process group rather than the latest.
This is better than what we have currently, but won't have the concept of a
release branch.

I also saw there was the concept of registry aliases. I think this might
help with pushing nested versioned flows from one registry to the other. I
have a hunch that this approach will uncover more issues. My hunch is that
the bucketID and flowID will be different and if I change the definition of
the alias I will get errors like bucket X not found. Another issue I
see would be that if Version X of a nested flow might mean something
different on the destination registry than on the source registry since
versions are autoincremented and are not in sync from registry to registry.
In the next couple of weeks i'll investigate this. Having multiple
registries would make us feel safer about deploying flows to production,
but we want to validate the correctness first.

-Eric

On Mon, Feb 24, 2020 at 10:45 PM Joe Witt  wrote:

> Eric
>
> You wrote a great email and all your points make a lot of sense.  I'll be
> blunt and state - we didn't design for that and we should have.
>
> When we launched the NiFi Registry we were seeing a lot of large scale
> monolith NIFi clusters being deployed where each cluster ran the flows of
> many different teams and/or use cases.  We did not expect though how
> common/popular it would be that multiple flow developers would be working
> within the same process group.  But that is definitely what they want to do
> today.
>
> We thought the notion of branching would be useful but not nearly as
> quickly as it has been.  We thought multiple registries and sharing across
> them would be of interest but again just not this fast.
>
> The ability to support versioned flows which reference other versioned
> flows - your TLDR was pretty darn spot on.
>
> We solved the problem we saw in front of us with the NiFi Registry.  What
> we didn't see/anticipate, in my opinion was just how much pent up
> demand/need there was for a true full git-like versioning experience for
> flow management in NiFi.
>
> I think there are quite a few folks focused in this area so you can expect
> some strong improvements here just not clear on when.
>
> In the mean time I'd tell you to the extent you can build around a central
> NiFi Registry, compose versioned flows of other versioned flows, and limit
> authorship of a versioned flow to a single author at any one time unless
> the other author is going to fork from a particular version to make a new
> versioned flow.
>
> Again great e-mail.  Your expectation of what it should do is very
> fair/right.
>
> Thanks
> Joe
>
> On Mon, Feb 24, 2020 at 10:32 PM Eric Secules  wrote:
>
>> Hello everyone,
>>
>> Im starting to use nifi and nifi registry on my development team and
>> we're running into issues working together on the same versioned process
>> groups.
>> The nifi registry doesn't support branching, merging and code review
>> nativly so we all have ended up developing on the same branch on the same
>> registry instance and doing in-person peer review. Is there a better way
>> for teams to develop process groups for nifi?
>>
>> What we've tried:
>> I tried to set up my own registry on my local machine where I do
>> development and make incremental changes. Then when I am ready to "merge" I
>> import the process group to the central registry from my local registry.
>> The main issue with this is that there's no mechanism for merge if the
>> central registry and my local registry have diverged. The other issue is
>> when a versioned process group containing other versioned process groupss
>> is moved from local-reg to central-reg the inner PGs still say their source
>> is local-reg despite the containing PG moving from local-reg to
>> central-reg. This becomes a problem in production environments which would
>> only be connected to central-reg. Tldr; Moving nested versioned flows
>> between registries is complicated.
>>
>> I've also tried backing up my local registry to a separate branch in git
>> and manually merging it with the branch that central-reg backs up to, but
>> these git branches are glorified 

Re: Suggestions for Flow Development Lifestyle

[Ken Swanson]

I have a lot of interest in this too, as my team has run into these issues
as well.

I feel like there are also issues in trying to build a pull-request-like
workflow, in that most NiFi design is done visually on the canvas but there
isn't a good way, that I know of, to represent differences between versions
in a visual way. Even if you enabled pull request-like functionality, I
don't know that I would expect users to approve changes by examining diffs
in the XML files.

However, I did have one thing to add that I hope might help:
On Tue, Feb 25, 2020 at 12:32 AM Eric Secules  wrote:

> I've also tried backing up my local registry to a separate branch in git
> and manually merging it with the branch that central-reg backs up to, but
> these git branches are glorified backups and the registry doesn't seem to
> be built to pull updates from them. On top of that doing a code review on
> the generated JSON describing a process group is difficult and I ran into
> several merge conflicts testing out a very simple merge where the target
> branch diverged slightly from the feature branch.
>

>From what I've seen, the registry will use any updates that are in the repo
when it starts up. So you should be able to do something like this:

   1. Make the registry pull (or clone) from the backup as part of its
   startup process
   2. Restart the registry when a merge to master is made. AFAIK this
   should not affect any NiFi instances that are connected to the registry;
   they should continue to work once the registry comes back up

It's a bit clunky but I think it works.

-Ken

Re: Suggestions for Flow Development Lifestyle

[Mike Thomsen]

In 2018, we had a similar experience, and we resolved a lot of potential
merge conflicts by dividing the work load so that people weren't working on
the same pieces at the same time. It was mainly a matter of shaping the
agile stories so that you didn't have 2-3 people stepping on each other's
toes.

The Registry could use some improvements in the areas that Joe cites, but
in practice I've found that working on communication and agile processes
can stop a lot of it from every happening.

On Tue, Feb 25, 2020 at 12:12 PM Darren Govoni  wrote:

> You could try using git and pull requests, merges, code review process.
> Just have to export and import your flows as templates.
>
> Alternatively, if Nifi registry was built on git API (local or remote
> repos etc) then it would all "just work" the way you describe.
> --
> *From:* Eric Secules 
> *Sent:* Tuesday, February 25, 2020 1:31 AM
> *To:* users@nifi.apache.org 
> *Subject:* Suggestions for Flow Development Lifestyle
>
> Hello everyone,
>
> Im starting to use nifi and nifi registry on my development team and we're
> running into issues working together on the same versioned process groups.
> The nifi registry doesn't support branching, merging and code review
> nativly so we all have ended up developing on the same branch on the same
> registry instance and doing in-person peer review. Is there a better way
> for teams to develop process groups for nifi?
>
> What we've tried:
> I tried to set up my own registry on my local machine where I do
> development and make incremental changes. Then when I am ready to "merge" I
> import the process group to the central registry from my local registry.
> The main issue with this is that there's no mechanism for merge if the
> central registry and my local registry have diverged. The other issue is
> when a versioned process group containing other versioned process groupss
> is moved from local-reg to central-reg the inner PGs still say their source
> is local-reg despite the containing PG moving from local-reg to
> central-reg. This becomes a problem in production environments which would
> only be connected to central-reg. Tldr; Moving nested versioned flows
> between registries is complicated.
>
> I've also tried backing up my local registry to a separate branch in git
> and manually merging it with the branch that central-reg backs up to, but
> these git branches are glorified backups and the registry doesn't seem to
> be built to pull updates from them. On top of that doing a code review on
> the generated JSON describing a process group is difficult and I ran into
> several merge conflicts testing out a very simple merge where the target
> branch diverged slightly from the feature branch.
>
> Does anyone have any other approaches that have succeeded on teams with
> multiple people developing on the same set of process groups?
>
> Thanks,
> Eric
>

Re: Suggestions for Flow Development Lifestyle

[Darren Govoni]

You could try using git and pull requests, merges, code review process. Just 
have to export and import your flows as templates.

Alternatively, if Nifi registry was built on git API (local or remote repos 
etc) then it would all "just work" the way you describe.

From: Eric Secules 
Sent: Tuesday, February 25, 2020 1:31 AM
To: users@nifi.apache.org 
Subject: Suggestions for Flow Development Lifestyle

Hello everyone,

Im starting to use nifi and nifi registry on my development team and we're 
running into issues working together on the same versioned process groups.
The nifi registry doesn't support branching, merging and code review nativly so 
we all have ended up developing on the same branch on the same registry 
instance and doing in-person peer review. Is there a better way for teams to 
develop process groups for nifi?

What we've tried:
I tried to set up my own registry on my local machine where I do development 
and make incremental changes. Then when I am ready to "merge" I import the 
process group to the central registry from my local registry. The main issue 
with this is that there's no mechanism for merge if the central registry and my 
local registry have diverged. The other issue is when a versioned process group 
containing other versioned process groupss is moved from local-reg to 
central-reg the inner PGs still say their source is local-reg despite the 
containing PG moving from local-reg to central-reg. This becomes a problem in 
production environments which would only be connected to central-reg. Tldr; 
Moving nested versioned flows between registries is complicated.

I've also tried backing up my local registry to a separate branch in git and 
manually merging it with the branch that central-reg backs up to, but these git 
branches are glorified backups and the registry doesn't seem to be built to 
pull updates from them. On top of that doing a code review on the generated 
JSON describing a process group is difficult and I ran into several merge 
conflicts testing out a very simple merge where the target branch diverged 
slightly from the feature branch.

Does anyone have any other approaches that have succeeded on teams with 
multiple people developing on the same set of process groups?

Thanks,
Eric

Re: Suggestions for Flow Development Lifestyle

[Joe Witt]

Eric

You wrote a great email and all your points make a lot of sense.  I'll be
blunt and state - we didn't design for that and we should have.

When we launched the NiFi Registry we were seeing a lot of large scale
monolith NIFi clusters being deployed where each cluster ran the flows of
many different teams and/or use cases.  We did not expect though how
common/popular it would be that multiple flow developers would be working
within the same process group.  But that is definitely what they want to do
today.

We thought the notion of branching would be useful but not nearly as
quickly as it has been.  We thought multiple registries and sharing across
them would be of interest but again just not this fast.

The ability to support versioned flows which reference other versioned
flows - your TLDR was pretty darn spot on.

We solved the problem we saw in front of us with the NiFi Registry.  What
we didn't see/anticipate, in my opinion was just how much pent up
demand/need there was for a true full git-like versioning experience for
flow management in NiFi.

I think there are quite a few folks focused in this area so you can expect
some strong improvements here just not clear on when.

In the mean time I'd tell you to the extent you can build around a central
NiFi Registry, compose versioned flows of other versioned flows, and limit
authorship of a versioned flow to a single author at any one time unless
the other author is going to fork from a particular version to make a new
versioned flow.

Again great e-mail.  Your expectation of what it should do is very
fair/right.

Thanks
Joe

On Mon, Feb 24, 2020 at 10:32 PM Eric Secules  wrote:

> Hello everyone,
>
> Im starting to use nifi and nifi registry on my development team and we're
> running into issues working together on the same versioned process groups.
> The nifi registry doesn't support branching, merging and code review
> nativly so we all have ended up developing on the same branch on the same
> registry instance and doing in-person peer review. Is there a better way
> for teams to develop process groups for nifi?
>
> What we've tried:
> I tried to set up my own registry on my local machine where I do
> development and make incremental changes. Then when I am ready to "merge" I
> import the process group to the central registry from my local registry.
> The main issue with this is that there's no mechanism for merge if the
> central registry and my local registry have diverged. The other issue is
> when a versioned process group containing other versioned process groupss
> is moved from local-reg to central-reg the inner PGs still say their source
> is local-reg despite the containing PG moving from local-reg to
> central-reg. This becomes a problem in production environments which would
> only be connected to central-reg. Tldr; Moving nested versioned flows
> between registries is complicated.
>
> I've also tried backing up my local registry to a separate branch in git
> and manually merging it with the branch that central-reg backs up to, but
> these git branches are glorified backups and the registry doesn't seem to
> be built to pull updates from them. On top of that doing a code review on
> the generated JSON describing a process group is difficult and I ran into
> several merge conflicts testing out a very simple merge where the target
> branch diverged slightly from the feature branch.
>
> Does anyone have any other approaches that have succeeded on teams with
> multiple people developing on the same set of process groups?
>
> Thanks,
> Eric
>

Suggestions for Flow Development Lifestyle

[Eric Secules]

Hello everyone,

Im starting to use nifi and nifi registry on my development team and we're
running into issues working together on the same versioned process groups.
The nifi registry doesn't support branching, merging and code review
nativly so we all have ended up developing on the same branch on the same
registry instance and doing in-person peer review. Is there a better way
for teams to develop process groups for nifi?

What we've tried:
I tried to set up my own registry on my local machine where I do
development and make incremental changes. Then when I am ready to "merge" I
import the process group to the central registry from my local registry.
The main issue with this is that there's no mechanism for merge if the
central registry and my local registry have diverged. The other issue is
when a versioned process group containing other versioned process groupss
is moved from local-reg to central-reg the inner PGs still say their source
is local-reg despite the containing PG moving from local-reg to
central-reg. This becomes a problem in production environments which would
only be connected to central-reg. Tldr; Moving nested versioned flows
between registries is complicated.

I've also tried backing up my local registry to a separate branch in git
and manually merging it with the branch that central-reg backs up to, but
these git branches are glorified backups and the registry doesn't seem to
be built to pull updates from them. On top of that doing a code review on
the generated JSON describing a process group is difficult and I ran into
several merge conflicts testing out a very simple merge where the target
branch diverged slightly from the feature branch.

Does anyone have any other approaches that have succeeded on teams with
multiple people developing on the same set of process groups?

Thanks,
Eric