Julien Ledoux created NIFI-14368:
------------------------------------
Summary: Python Processor can not use parent class properties as
PropertyDependency
Key: NIFI-14368
URL: https://issues.apache.org/jira/browse/NIFI-14368
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 2.3.0, 2.2.0, 2.1.0
Environment: Official Docker Images NiFi 2.x running on Ubuntu 22.04.4
LTS and Docker v27.0.3
Reporter: Julien Ledoux
Attachments: debug_inheritance.zip
When using a Python parent class (multiple inheritance) to store common
properties and methods for a collection processors, it is not possible to use
parent class properties as a PropertyDependency in its child classes.
This approach worked correctly in NiFi versions 2.0.0-M4 and 2.0.0. However,
starting with NiFi 2.1.0, referencing a parent class property as a
PropertyDependency causes a crash of the processors using this pattern.
Here is the error message from 2.3.0 version:
{code:java}
2025-03-14 14:06:52,561 ERROR [python-log-344] py4j.java_gateway There was an
exception while executing the Python Proxy on the Python Side.Traceback (most
recent call last): File
"/opt/nifi/nifi-current/python/framework/py4j/java_gateway.py", line 2506, in
_call_proxy return_value = getattr(self.pool[obj_id], method)(*params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File
"/opt/nifi/nifi-current/./python/framework/Controller.py", line 77, in
reloadProcessor self.extensionManager.reload_processor(processorType,
version, workDirectory) File
"/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 135, in
reload_processor self.__gather_extension_details(module_file,
extension_home, dependencies_bundled, work_dir) File
"/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 239, in
__gather_extension_details classes_and_details =
self.__get_processor_classes_and_details(module_file, extension_home,
dependencies_bundled)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 269,
in __get_processor_classes_and_details details =
ProcessorInspection.get_processor_details(class_node, module_file,
extension_home, dependencies_bundled=dependencies_bundled)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
154, in get_processor_details property_descriptions =
get_property_descriptions(class_node, module_string_constants)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File
"/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line 288, in
get_property_descriptions visitor.visit(class_node) File
"/usr/lib/python3.11/ast.py", line 418, in visit return visitor(node)
^^^^^^^^^^^^^ File "/usr/lib/python3.11/ast.py", line 426, in generic_visit
self.visit(item) File "/usr/lib/python3.11/ast.py", line 418, in visit
return visitor(node) ^^^^^^^^^^^^^ File
"/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line 94, in
visit_Assign value = self.resolve_dependencies(keyword.value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File
"/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line 61, in
resolve_dependencies variable_name = dependency.args[0].id
^^^^^^^^^^^^^^^^^^^^^AttributeError: 'Attribute' object has no attribute
'id'2025-03-14 14:06:52,564 WARN [Validate Components Thread-3]
o.a.n.p.processor.FlowFileSourceProxy
PythonProcessor[type=AnotherFlowFileSourceProcessor,
id=9453f6c2-0195-1000-0d8a-2de7d7b09bca] Failed to perform validation for
Python Processor PythonProcessor[type=AnotherFlowFileSourceProcessor,
id=9453f6c2-0195-1000-0d8a-2de7d7b09bca]; assuming invalidpy4j.Py4JException:
An exception was raised by the Python Proxy. Return Message: Traceback (most
recent call last): File
"/opt/nifi/nifi-current/python/framework/py4j/java_gateway.py", line 2506, in
_call_proxy return_value = getattr(self.pool[obj_id], method)(*params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File
"/opt/nifi/nifi-current/./python/framework/Controller.py", line 77, in
reloadProcessor self.extensionManager.reload_processor(processorType,
version, workDirectory) File
"/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 135, in
reload_processor self.__gather_extension_details(module_file,
extension_home, dependencies_bundled, work_dir) File
"/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 239, in
__gather_extension_details classes_and_details =
self.__get_processor_classes_and_details(module_file, extension_home,
dependencies_bundled)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 269,
in __get_processor_classes_and_details details =
ProcessorInspection.get_processor_details(class_node, module_file,
extension_home, dependencies_bundled=dependencies_bundled)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
154, in get_processor_details property_descriptions =
get_property_descriptions(class_node, module_string_constants)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File
"/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line 288, in
get_property_descriptions visitor.visit(class_node) File
"/usr/lib/python3.11/ast.py", line 418, in visit return visitor(node)
^^^^^^^^^^^^^ File "/usr/lib/python3.11/ast.py", line 426, in generic_visit
self.visit(item) File "/usr/lib/python3.11/ast.py", line 418, in visit
return visitor(node) ^^^^^^^^^^^^^ File
"/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line 94, in
visit_Assign value = self.resolve_dependencies(keyword.value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File
"/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line 61, in
resolve_dependencies variable_name = dependency.args[0].id
^^^^^^^^^^^^^^^^^^^^^AttributeError: 'Attribute' object has no attribute 'id'
at py4j.Protocol.getReturnValue(Protocol.java:476) at
org.apache.nifi.py4j.client.PythonProxyInvocationHandler.invoke(PythonProxyInvocationHandler.java:87)
at jdk.proxy6/jdk.proxy6.$Proxy109.reloadProcessor(Unknown Source)
at
org.apache.nifi.py4j.StandardPythonProcessorBridge.reload(StandardPythonProcessorBridge.java:186)
at
org.apache.nifi.python.processor.PythonProcessorProxy.reload(PythonProcessorProxy.java:357)
at
org.apache.nifi.python.processor.PythonProcessorProxy.customValidate(PythonProcessorProxy.java:199)
at
org.apache.nifi.components.AbstractConfigurableComponent.validate(AbstractConfigurableComponent.java:130)
at
org.apache.nifi.controller.AbstractComponentNode.computeValidationErrors(AbstractComponentNode.java:912)
at
org.apache.nifi.controller.StandardProcessorNode.computeValidationErrors(StandardProcessorNode.java:1054)
at
org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:847)
at
org.apache.nifi.controller.AbstractComponentNode.performValidation(AbstractComponentNode.java:861)
at
org.apache.nifi.components.validation.StandardValidationTrigger.trigger(StandardValidationTrigger.java:52)
at
org.apache.nifi.components.validation.TriggerValidationTask.run(TriggerValidationTask.java:58)
at org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583) {code}
Because of this issue, upgrades beyond NiFi 2.0.0 are blocked for anyone
relying on this inheritance pattern.
I've attached a simple processor structure that uses such pattern.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
