You are totally fine to engage on this thread Sean. We appreciate it
On Tue, Aug 11, 2026 at 8:17 PM Sean Hunter via dev <[email protected]> wrote: > Hi David, > > As a user, rather than contributor, I think my primary concern is one of > operations and implementation. What alternative would be available? We used > the 1.x version and were quite excited to move to 2.0 with its more native > integration that was easier to troubleshoot. We also do build a NAR as part > of our pipeline process, and I suspect since we used a blog as guidance to > develop that process, that others likely do as well. Our use case is a > particularly nasty data transform (Windows logs of some sort, naturally) > that’s reasonable to write and execute in Python. It’s readable, while > still being performant enough for our needs, and seemingly not causing > system stability issues for us. > > I do understand and take your points. What I’d like to better understand > are the options users have. Do you have any thoughts? > > Thank you and have a great day! > Sean > > PS - Please let me know if this is too far off topic for the dev mailing > list and I’ll take it elsewhere. > > From: David Handermann <[email protected]> > Date: Tuesday, August 11, 2026 at 8:38 PM > To: [email protected] <[email protected]> > Subject: [DISCUSS] Future Support for Python Processors? > > Be Careful: Email is from an External Source > Use caution when opening emails and do not click on links or attachments > from unknown senders. For more resources, visit security.cat.com/phishing. > > > Team, > > The introduction of support for Python Processors was one of the major > new features of Apache NiFi 2. In light of experience, however, I > believe it is time to reconsider future support for this extension > strategy. The idea of removing this feature may raise some concerns, > but it is important to consider the options based on scalability, > security, and supportability. > > One of the greatest benefits of Python-based Processors is rapid > iteration. Although this is a strength during the development phase, > it can become a serious weakness when moving to production. > > NiFi is fundamentally a Java-based application, thus tuning resource > allocation is a key part of maintaining a production system. Running a > Python Processor requires one or more independent Python processes, > each with their own memory consumption and CPU usage patterns. What > might work at a small scale on a development machine breaks down when > attempting to run at production scale, without careful tuning. This is > challenging enough with Java-based components, but global settings > like heap and non-heap memory provide clear boundaries. Python > Processors, as currently implemented, do not support this level of > control. Beyond process control, operations that require communicating > between the NiFi JVM and Python processes involve socket > communication, which can be expensive for frequent calls. Thus an > apparent initial advantage of Python Processors becomes a major > problem when moving beyond limited types of operations. > > When it comes to security, Python Processors can involve more serious > risks than bundled Java extensions. > > With NiFi support for downloading dependencies at runtime, it is > essential for an administrator to provide a secure supply chain > strategy for retrieving these dependencies. Although the NiFi > framework also supports packaging Python Processors as NAR bundles, > this packaging capability is not currently part of the Apache NiFi > project. The project could build on third party work, but as Python > packaging is less opinionated, this is an additional maintenance > question that quickly becomes more involved. This combination of > supply chain security and lack of packaging support raises security > challenges. > > Supportability is always a question for any feature in an open source > project, but Python Processors are a particular point of concern. > Providing NiFi framework support requires a strong knowledge of the > NiFi project itself, and a strong knowledge of Python. Project > maintainers and contributors usually have expertise in particular > areas, but it is rare to find both. More to the point, however, the > Py4J project, which NiFi depends on for integration, has seen limited > maintenance, with the last tagged version dating to January 2025. > Without robust architecture support, the NiFi project cannot provide a > strong framework for future implementation. > > Taking these issues together, I believe the project should consider > deprecating support for Python Processors. Similar to other > deprecations, actual removal would not occur until a major release > version, and that would be a separate conversation. > > Rather than continuing to extend the life of Python Processor support, > it seems better to invest limited project resources into maintaining > and improving core functionality. > > Regards, > David Handermann > Apache NiFi PMC Chair > >
