Re: Use a different runtime

2019-07-12 Thread Andy LoPresto
Mike, The short answer is “this is not as simple as replacing one class”. The longer answer is that you may be interested in the work by Mark Payne and Sam Hjemfelt to provide an additional runtime in NiFi (referred to as stateless NiFi [1]). Hope this helps. [1]

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Andy LoPresto
Adam, I think your statements about “definitely a total sum of work that is greater” are true for a specific audience. As someone who routinely reviews PRs and handles release management tasks, I know where I would like to see improvements. I also write feature code (though not as often

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Jeff
Adam, To your point, we currently have the situation you described when dependencies exist between NiFi and NiFi Registry, where one must be released before the other. In my opinion, it is better to be able to verify functionality in smaller atomic change-sets and limited scope over a larger

Use a different runtime

2019-07-12 Thread HWANG, MICHAEL (MICHAEL)
Hello I want to use Nifi's UI and APIs (i.e. registry) but not use the Nifi scheduler, controller, runtime. I want to prototype hooking in my own scheduler, controller, runtime. How involved would this be? Would it be more involved than replacing the class

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Adam Taft
Andy - fair points. Note that by definition, the process you describe is harder (requires more maneuvers). Maybe it's warranted/justified for the desired integrity that you are after, but it's most definitely a total sum of work that is greater. Your registry example is really good. In your

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Andy LoPresto
I think by definition, a contribution _must_ fit into a single repository. This will force developers to carefully consider the boundaries between modules and build clean abstractions. If you are a new contributor, I would be surprised if you are making a single (logical) contribution that

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Adam Taft
Bryan, I think both of your points are actually closely related, and they somewhat speak to my thoughts/concerns about splitting the repository. I would argue that one PR that affects multiple modules in a single repository is _easier_ to review than multiple PRs that affect single modules. In

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Bryan Bende
Two other points to throw out there... 1) I think something to consider is how the management of pull requests would be impacted, since that is the main form of contribution. Separate repos forces pull requests to stay scoped to a given module, making for more straight forward reviews. It also

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Kevin Doran
Thanks Adam and Edward! This is exactly the type of discussion I was hoping a detailed and specific proposal would generate, so thanks for the input. I'll reply to each of you in turn: Adam, It is true that a repo-per-project approach is not required. I've worked on projects that do it both ways

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Adam Taft
To be honest and to your point Joe, the thing that optimizes the RM duties should probably be preferred in all of this. There is so much overhead for the release manager, that lubricating the RM process probably trumps a lot of my concerns. I think there's real concern for making the project

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Edward Armes
I think Nifi would really benefit from this. One thing I think that, should be looked into is something I noticed while trying to get to grips with the Nifi source, At the start of the year I did a small exercise to track how a call from the API to start and stop a processor translates to a

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Adam Taft
I think the concerns around user management are valid, are they not? Overhead in JIRA goes up (assigning rights to users in JIRA is multiplied). Risk to new contributors is high, because each isolated repository has its own life and code contribution styles. Maybe the actual apache infra

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Joe Witt
to clarify user management for infra is not a prob. it is an ldap group. repo creation is self service as well amd group access is tied to that. release artifact is the source we produce. this is typically correlated to a tag of the repo. if we have all source in one repo it isnt clear to me

Re: [EXT] [discuss] Splitting NiFi framework and extension repos and releases

2019-07-12 Thread Adam Taft
Just as a point of discussion, I'm not entirely sure that splitting into multiple physical git repositories is actually adding any value. I think it's worth consideration that all the (good) changes being proposed are done under a single mono-repository model. If we split into multiple