Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Adam, I think your statements about “definitely a total sum of work that is greater” are true for a specific audience. As someone who routinely reviews PRs and handles release management tasks, I know where I would like to see improvements. I also write feature code (though not as often

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Adam, To your point, we currently have the situation you described when dependencies exist between NiFi and NiFi Registry, where one must be released before the other. In my opinion, it is better to be able to verify functionality in smaller atomic change-sets and limited scope over a larger

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Andy - fair points. Note that by definition, the process you describe is harder (requires more maneuvers). Maybe it's warranted/justified for the desired integrity that you are after, but it's most definitely a total sum of work that is greater. Your registry example is really good. In your

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

I think by definition, a contribution _must_ fit into a single repository. This will force developers to carefully consider the boundaries between modules and build clean abstractions. If you are a new contributor, I would be surprised if you are making a single (logical) contribution that

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Bryan, I think both of your points are actually closely related, and they somewhat speak to my thoughts/concerns about splitting the repository. I would argue that one PR that affects multiple modules in a single repository is _easier_ to review than multiple PRs that affect single modules. In

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Two other points to throw out there... 1) I think something to consider is how the management of pull requests would be impacted, since that is the main form of contribution. Separate repos forces pull requests to stay scoped to a given module, making for more straight forward reviews. It also

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Thanks Adam and Edward! This is exactly the type of discussion I was hoping a detailed and specific proposal would generate, so thanks for the input. I'll reply to each of you in turn: Adam, It is true that a repo-per-project approach is not required. I've worked on projects that do it both ways

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

To be honest and to your point Joe, the thing that optimizes the RM duties should probably be preferred in all of this. There is so much overhead for the release manager, that lubricating the RM process probably trumps a lot of my concerns. I think there's real concern for making the project

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

I think Nifi would really benefit from this. One thing I think that, should be looked into is something I noticed while trying to get to grips with the Nifi source, At the start of the year I did a small exercise to track how a call from the API to start and stop a processor translates to a

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

I think the concerns around user management are valid, are they not? Overhead in JIRA goes up (assigning rights to users in JIRA is multiplied). Risk to new contributors is high, because each isolated repository has its own life and code contribution styles. Maybe the actual apache infra

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

to clarify user management for infra is not a prob. it is an ldap group. repo creation is self service as well amd group access is tied to that. release artifact is the source we produce. this is typically correlated to a tag of the repo. if we have all source in one repo it isnt clear to me

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Just as a point of discussion, I'm not entirely sure that splitting into multiple physical git repositories is actually adding any value. I think it's worth consideration that all the (good) changes being proposed are done under a single mono-repository model. If we split into multiple

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

I agree that this looks great. I think Mike’s idea is worth considering as well. I would hope, that as part of this effort some thought will be given to enhancing the developer documentation around the modules would be given as well. On July 10, 2019 at 18:15:21, Mike Thomsen

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

I agree. It's very well thought out. One change to consider is splitting the extensions further into two separate repos. One that would serve as a standard library of sorts for other component developers and another that would include everything else. Things like the Record API would go into the

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Thanks Kevin, this looks really promising. Updating the link here as I think the page may have moved: https://cwiki.apache.org/confluence/display/NIFI/NiFi+Project+and+Repository+Restructuring Andy

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

Hi NiFi Dev Community, Jeff Storck, Bryan Bende, and I have been collaborating back and forth on a proposal for how to restructure the NiFi source code into smaller Maven projects and repositories based on the discussion that took place awhile back on this thread. I'm reviving this older thread

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

I am also in favor of splitting the nifi maven project up into smaller projects with independent release cycles in order to decouple development at well defined boundaries/interfaces and also to facilitate code reuse. In anticipation of eventually working towards a NiFi 2.0 that introduces bigger

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

I am a strong +1 on the separation and reducing the build time. With that in mind, I think the process I brought up yesterday [1] of signing our artifacts with GPG as part of the Maven build is paramount, because we would now be consuming core code across multiple projects/repositories, so

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

In regards to "We 'could' also split out the 'nifi-api'...", NiFi 2.0 would also be a good time to look at more clearly defining the separation between the UI and the framework. Where nifi-api is the contract between the extensions and the framework, the NiFi Rest api is the contract between the

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

In the same category of challenges that Peter pointed out, it might be difficult for Travis to build the "framework" and "extensions" projects if there are changes in a PR that affect both projects. Is there a good way in Travis to have the workspace/maven repo shared between projects in a single

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

yeah that is a good point. We 'could' also split out the 'nifi-api' which is the only 'contract' between extensions and core. Not sure the juice is worth the squeeze but then again we rarely change that thing. On Thu, May 30, 2019 at 11:23 AM Peter Wicks (pwicks) wrote: > One more "not

RE: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

One more "not awesome" would be that core changes that affect extensions will be a little harder to test. If I make a core change that changes the signature of an interface/etc... I'll need to do some extra work to make sure I don't break extensions that use it. Still worth it, just one more