Hi David, Currently we are in the process of implementing support for the NiFi python processors in MiNiFi C++. Probably in the next open source release this feature will be available, so the available NiFi Python processors will be usable in MiNiFi C++ as well. I think this idea would help with the collaboration of supporting these processors in both Java and C++ projects. It would certainly make release verification easier to be able to concentrate only on the Python processors if they are released separately.
My concern is how would the automatic testing and verification would work in this scenario? Would all the testing of the Python processors be moved to the new repository and would be tested there separately, with both NiFi and MiNiFi C++, or only with NiFi, or all of the testing would remain in the respective client repositories? Regards, Gabor Gyimesi On Fri, 26 Jan 2024 at 14:11, David Handermann <exceptionfact...@apache.org> wrote: > Pierre, > > Thanks for the reply, and noting the potential concern with the > ability to find these components. > > I think there are several ways we can address this concern, both for > optional Java components, and for Python components in a separate > repository. > > For Python components in particular, we could add direct links to > published versions on the main download page, calling out their > availability in the official PyPI repository. Although this would need > to be denoted as a non-official release channel for Apache purposes, > this is common practice in other projects, and follows the approach we > already have for container images on Docker Hub. > > In addition to linking from the download page, we could publish the > generated documentation for these components. The current process for > publishing generated documentation is based on the convenience binary, > but with some adjustments, we could publish the documentation for the > Python components as well. This is a good prompt to start doing this > for the optional Java components, and I plan to look at doing this for > the next release with optional Java components. > > To your last question, it is worth noting that any binary releases > would fall into the category of convenience builds. Initially, I think > the NiFi framework release would not include the Python extension > components. However, having a few short steps on installation, linked > from the download page, seems like it would provide a way forward. > > Regards, > David Handermann > > On Fri, Jan 26, 2024 at 1:22 AM Pierre Villard > <pierre.villard...@gmail.com> wrote: > > > > Hi David, > > > > While I agree with your summary, I have a concern here which is about > user > > awareness of this feature. We've seen in the past: as soon as we don't > > include NARs in the convenience binary, we see that users have no clue > > about those NARs (and some are super powerful/useful). I agree that > python > > is a bit different because it requires a user action to enable it in the > > first place but I still think that including the components in the > > convenience binary of Apache NiFi would drive user awareness, adoption, > etc. > > > > If we have a separated repo with its own release cycle can we imagine a > > process where, when releasing Apache NiFi, it'd include whatever is the > > latest version of the Python repo? Or something along those lines? > > > > Pierre > > > > Le ven. 26 janv. 2024 à 08:01, David Handermann < > exceptionfact...@apache.org> > > a écrit : > > > > > Team, > > > > > > As we get closer to a full release of Apache NiFi 2.0.0, we have an > > > important opportunity to set the direction for future development of > > > Python-based Processors. > > > > > > The introduction of native Python support presents a number of new > > > integration opportunities, and it also raises questions about > > > maintenance and versioning. As the journey to NiFi 2.0.0 has shown, it > > > requires significant effort to coordinate maintenance and > > > modernization across hundreds of project modules. Although the > > > internal project structure has maintained helpful separation of API > > > and implementation, the current release strategy highlights the > > > challenges of verifying multiple layers of changes. Introducing a new > > > programming language provides greater possibilities, but also makes it > > > more difficult to maintain a single repository with a single > > > versioning strategy. > > > > > > I propose creating a new Git repository named nifi-python-extensions, > > > which would have its own versioning and release process. This would > > > contain the extensions now under the module of the same name in the > > > NiFi repository. Having a separate repository and release process for > > > Python-based extensions has the following advantages: > > > > > > 1. Clean separation between NiFi APIs for Python and Python-based > > > Processors > > > 2. Independent release cycles for Python-based Processors > > > 3. Focused release verification and testing on Python-based modules > > > > > > These advantages can also enable more rapid iteration on Python-based > > > Processors, without impacting the NiFi Framework or requiring new > > > releases at that level. Although this would require a separate > > > installation process for Python-based components, this could follow an > > > approach similar to what is already required for optional Java-based > > > components. > > > > > > Thanks in advance for your consideration. > > > > > > Regards, > > > David Handermann > > > Apache NiFi PMC Member > > > >
