Hi Eric, So I looked into this September last year. I found that a variant of master driven development was the best course of action, for collaboration on a single Nifi flow.
As far as I can tell the git integration deliberately designed to be as simple a s possible, the down side of this is that git functionality like branching just doesn't work with the git integration. I briefly looked into seeing if I could sidestep a few of the issues by changing how the flow version information was stored, but I ran into several road blocks around adding support for history re-writes....... Edward On Fri, 15 May 2020, 01:17 Eric Secules, <esecu...@gmail.com> wrote: > Hi Ami, > > I'm testing our NiFi application using a blackbox approach. We have a test > harness that stages input files and waits for the corresponding outputs, > then validates the content of the output. NiFi runs in a docker container > (see https://hub.docker.com/r/apache/nifi/) and you can automate starting > it up and orchestrating a test harness for your application in any > competent CI/CD pipeline. > > Jorge, I don't understand your comment. NiFi registry can be synced to a > git repo as a backup mechanism, but you can't simply make branches, work > and merge them back, because even the smallest change in a flow will create > a massive diff in the Registry's backup git repo. It would be great if NiFi > had a way of highlighting the differences between flow versions on the > canvas and allow you to choose which elements to keep and which to discard > when resolving a merge conflict. > > -Eric > > On Thu, May 14, 2020 at 12:00 PM Jorge Machado <jom...@me.com> wrote: > >> I think we could improve Nifi by hooking up to GitHub somehow. So that we >> don’t need the registry… >> >> On 14. May 2020, at 18:49, Ami Goldenberg <ami.g...@gmail.com> wrote: >> >> Hi Eric, >> Would love to know, what kind of tests do you write/run for NiFi? We were >> just researching this topic. >> >> On Thu, May 14, 2020 at 6:38 PM Eric Secules <esecu...@gmail.com> wrote: >> >>> Hi Michal, >>> >>> I'm also using a single registry for development and production. It >>> doesn't help with collaborating on the same process group as there is way >>> for it to reconcile merge conflicts. Instead, the registry will earn you >>> that you're about to overwrite someone else's changes. Another pain of >>> concurrent development is there's no concept of a PR and no visual diff of >>> your local changes making review difficult. I've backed our registry up to >>> git I've set up a CI pipeline in Azure Devops which runs our tests every >>> time a new version of a process group is checked in. It's better than >>> nothing but I'd rather nifi implemented git flow. Developing as a team on >>> nifi and nifi registry is like your whole team developing and pushing >>> directly to master. >>> >>> >>> -Eric >>> >>> On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <jom...@me.com> wrote: >>> >>>> Hi, >>>> >>>> Managing xml is always hard I think. Last time I need to do something >>>> similar we used https://nifi.apache.org/registry.html >>>> Works pretty well It was already 2 Years ago. Maybe now there is >>>> something better >>>> >>>> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external < >>>> michal.sl...@dhl.com> wrote: >>>> >>>> Hello, >>>> >>>> may I ask you for recommendations for development and CI/CD in NiFi? >>>> Pls let me describe our situation…I am a developer from DHL currently >>>> working on project including NiFi. It is part of our core and it is >>>> responsible for handling incoming data streams, data transormation and >>>> put it into various elastic search indexes (and queues). >>>> >>>> Up to now development was quite straightforward as only one developer >>>> did it. But we have extended our team recently and now we face problems how >>>> to correctly maintain development for more developers working in >>>> parallel and then CI/CD of it as we have classical dev/test/uat/project >>>> env. structure. >>>> >>>> Pls do you have any recommadation how to achive it? For now in general >>>> is enought. Its good to mention that currently we work with NiFi version >>>> 1.8…tried to upgrade to 1.9. but some of components failed so the >>>> upgrade was postponed. But with new features in 1.10. and 1.11. we head >>>> to uprade to these versions. >>>> >>>> Maybe if we can arrange a call it would be great! >>>> >>>> With regards, >>>> Michal Sláma >>>> >>>> >>>> >>>> *This message is from DHL Information Services (Europe) s.r.o. and may >>>> contain confidential business information. It is intended solely for the >>>> use of the individual to whom it is addressed. If you are not the intended >>>> recipient please contact the sender and delete this message and any >>>> attachment from your system. Unauthorized publication, use, dissemination, >>>> forwarding, printing or copying of this email and its attachments is >>>> strictly prohibited.* >>>> >>>> >>>> >>
