Team, I have drafted an initial version of the NiFi Improvement Proposal Process:
https://cwiki.apache.org/confluence/display/NIFI/NiFi+Improvement+Proposal+Process The document outlines the scope of project components requiring a proposal, as well as the requirements for a proposal, and the review process. The general approach for review and approval is designed to be similar to the release process. The requirements for a proposal are similar to other project improvement proposals. I recommend creating a new Jira project for handling Improvement Proposals, as opposed to using Confluence pages. Please review the draft and I will look to call for a vote within the next week. As mentioned earlier, with the nifi-api module being the core contract for internal and external extensions, I would like to include moving the nifi-api to a separate Git repository as part of the vote for instituting the Improvement Proposal Process. If this proves to be too much of a concern, we can divide the question and vote separately on the Improvement Proposal Process and moving the nifi-api. Thanks for the feedback! Regards, David Handermann On Wed, Aug 21, 2024 at 11:09 AM David Handermann <exceptionfact...@apache.org> wrote: > > Thanks for the replies! > > I will move forward with drafting a NiFi Improvements Proposal page to > keep the discussion going. > > Regards, > David Handermann > > On Tue, Aug 20, 2024 at 3:22 PM Mark Payne <marka...@hotmail.com> wrote: > > > > David, > > > > +1 to all of this. Especially for guaranteeing compatibility and > > maintainability, I think instrumenting a more > > formal approach for updating the API is a step in the right direction. The > > huge amount of purging, cleanup, > > and refactoring that has gone into 2.0 helps to highlight the importance of > > ensuring maintainability. > > > > Thanks > > -Mark > > > > > > > On Aug 20, 2024, at 2:00 AM, Joe Witt <joe.w...@gmail.com> wrote: > > > > > > David > > > > > > Yeah well written and good points. > > > > > > I dont think this says we would relax our existing commitment we have > > > shown > > > for things such as the http based api but rather is calling out specific > > > things which we will make changes to even more formalized. > > > > > > To that end +1 on the concept of proposals and +1 to breaking out the > > > nifi-api (fundamental designed extension points for flow development) in > > > particular. > > > > > > Thanks > > > Joe > > > > > > On Fri, Aug 16, 2024 at 8:20 PM Adam Taft <a...@adamtaft.com> wrote: > > > > > >> These are all really appreciated concepts, David. Thank you for putting > > >> the > > >> thoughts and time in on this! > > >> > > >> Regarding this: > > >>> The NiFi REST API does not have quite the same level of concern, but may > > >> warrant inclusion. > > >> > > >> I hear what you're saying. However, the REST API (from my > > >> observation/experience) has gathered quite a number of useful tools and > > >> "hacks" for NiFi. Quite often, many different monitoring and alerting > > >> tools > > >> have been developed against the REST API by third parties and/or > > >> integrators of NiFi against their internal workflows. Having stable API > > >> versioning in the REST API possibly makes just as much sense as having > > >> the > > >> same for the nifi-api itself. This is a prime entry point for extensions > > >> and other features developed alongside NiFi, maybe even the weird stuff > > >> that you can't do with the nifi-api directly. > > >> > > >> Food for thought of course, but I would hope that we can treat the REST > > >> API > > >> as a proper first-class citizen in terms of documented versioning. It > > >> turns > > >> out it's quite a useful means for interacting with a running NiFi > > >> instance. > > >> > > >> /Adam > > >> > > >> On Fri, Aug 16, 2024 at 9:59 AM David Handermann < > > >> exceptionfact...@apache.org> wrote: > > >> > > >>> Team, > > >>> > > >>> As we wrap up remaining items for NiFi 2, we should consider how to > > >>> continue improving the quality and maintainability of the NiFi > > >>> ecosystem. > > >>> > > >>> One primary focus of NiFi 2 has been the reduction of technical debt, > > >>> which involved the removal of numerous modules and thousands of lines > > >>> of code. In that process, it is worth highlighting that the core NiFi > > >>> API, and the NiFi Framework API, and the NiFi REST API have had > > >>> comparatively few breaking changes. This a testament to the quality of > > >>> the API design itself. The NiFi Version Schema and API Compatibility > > >>> [1] has provided a strong direction thus far. > > >>> > > >>> With that background, we should consider adopting a more formal > > >>> process around changes that impact the fundamental API contracts that > > >>> NiFi provides. NiFi Feature Proposals [2] have provided elements of > > >>> this in the past, but did not include approval requirements. Kafka [3] > > >>> and Airflow [4] have more structured improvement proposal processes, > > >>> and that is what we should adopt going forward. > > >>> > > >>> Part of moving in this direction requires identifying the areas that > > >>> would require going through the Improvement Proposal process itself. > > >>> At minimum, this should include the nifi-api [5] module. The > > >>> nifi-framework-api [6] is also worth consideration for inclusion in > > >>> this category. The NiFi REST API does not have quite the same level of > > >>> concern, but may warrant inclusion. > > >>> > > >>> As part of this discussion, we should consider separating the nifi-api > > >>> module into its own repository, with its own versioning scheme. This > > >>> will provide a helpful distinction in terms of the scope of changes, > > >>> and allow the API to be released independently of the application, > > >>> providing strong version compatibility guarantees. > > >>> > > >>> Based on feedback for the general idea, I would be glad to draft a > > >>> NiFi Improvement Proposal page, outlining the recommended steps in > > >>> more detail so we can come to consensus on how this should work. > > >>> > > >>> Regards, > > >>> David Handermann > > >>> > > >>> [1] > > >>> > > >> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility > > >>> [2] > > >>> https://cwiki.apache.org/confluence/display/NIFI/NiFi+Feature+Proposals > > >>> [3] > > >>> > > >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > >>> [4] > > >>> > > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals > > >>> [5] https://github.com/apache/nifi/tree/main/nifi-api > > >>> [6] https://github.com/apache/nifi/tree/main/nifi-framework-api > > >>> > > >> > >
