Team, The introduction of support for Python Processors was one of the major new features of Apache NiFi 2. In light of experience, however, I believe it is time to reconsider future support for this extension strategy. The idea of removing this feature may raise some concerns, but it is important to consider the options based on scalability, security, and supportability.
One of the greatest benefits of Python-based Processors is rapid iteration. Although this is a strength during the development phase, it can become a serious weakness when moving to production. NiFi is fundamentally a Java-based application, thus tuning resource allocation is a key part of maintaining a production system. Running a Python Processor requires one or more independent Python processes, each with their own memory consumption and CPU usage patterns. What might work at a small scale on a development machine breaks down when attempting to run at production scale, without careful tuning. This is challenging enough with Java-based components, but global settings like heap and non-heap memory provide clear boundaries. Python Processors, as currently implemented, do not support this level of control. Beyond process control, operations that require communicating between the NiFi JVM and Python processes involve socket communication, which can be expensive for frequent calls. Thus an apparent initial advantage of Python Processors becomes a major problem when moving beyond limited types of operations. When it comes to security, Python Processors can involve more serious risks than bundled Java extensions. With NiFi support for downloading dependencies at runtime, it is essential for an administrator to provide a secure supply chain strategy for retrieving these dependencies. Although the NiFi framework also supports packaging Python Processors as NAR bundles, this packaging capability is not currently part of the Apache NiFi project. The project could build on third party work, but as Python packaging is less opinionated, this is an additional maintenance question that quickly becomes more involved. This combination of supply chain security and lack of packaging support raises security challenges. Supportability is always a question for any feature in an open source project, but Python Processors are a particular point of concern. Providing NiFi framework support requires a strong knowledge of the NiFi project itself, and a strong knowledge of Python. Project maintainers and contributors usually have expertise in particular areas, but it is rare to find both. More to the point, however, the Py4J project, which NiFi depends on for integration, has seen limited maintenance, with the last tagged version dating to January 2025. Without robust architecture support, the NiFi project cannot provide a strong framework for future implementation. Taking these issues together, I believe the project should consider deprecating support for Python Processors. Similar to other deprecations, actual removal would not occur until a major release version, and that would be a separate conversation. Rather than continuing to extend the life of Python Processor support, it seems better to invest limited project resources into maintaining and improving core functionality. Regards, David Handermann Apache NiFi PMC Chair
