Julien Ledoux created NIFI-14233:
------------------------------------
Summary: Python extensions: PropertyDependency crashes when using
an imported property
Key: NIFI-14233
URL: https://issues.apache.org/jira/browse/NIFI-14233
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 2.2.0, 2.1.0
Environment: Official Docker Images 2.0.0 stable running on Ubuntu
22.04.4 LTS and Docker v27.0.3
Reporter: Julien Ledoux
Attachments: python_extension.tar.gz
When developing Python processors for NiFi, it is often necessary to share
properties and methods across multiple processors by placing them into a shared
module and importing them into each processor class.
This approach worked correctly in NiFi versions 2.0.0-M4 and 2.0.0. However,
starting with NiFi 2.1.0, referencing an imported property as a
PropertyDependency causes a crash.
For example, in NiFi 2.2.0, using an imported property in property_descriptors
still works fine:
{code:java}
from debug.modules.MyModule import MY_MODULE_PROPERTY
from nifiapi.flowfilesource import FlowFileSource
class MyFlowFileSourceProcessor(FlowFileSource):
...
property_descriptors = [
MY_MODULE_PROPERTY
]{code}
But if the same property is used as a dependency, NiFi fails to load the
processor:
{code:java}
from debug.modules.MyModule import MY_MODULE_PROPERTY
from nifiapi.flowfilesource import FlowFileSource
from nifiapi.properties import PropertyDescriptor, PropertyDependency,
StandardValidators, ExpressionLanguageScope
class MyFlowFileSourceProcessor(FlowFileSource):
...
MY_PROPERTY = PropertyDescriptor(
name="Number of Spaces",
description="Number of spaces to use for pretty-printing",
validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
expression_language_scope=ExpressionLanguageScope.ENVIRONMENT,
default_value="4",
required=True,
dependencies=[
PropertyDependency(MY_MODULE_PROPERTY, ("false"))
]
){code}
This results in a KeyError: 'MY_MODULE_PROPERTY' exception, preventing the
extension from loading:
{code:java}
2025-02-04 15:33:43,065 ERROR [python-log-288] python.ExtensionManager Failed
to load Python extensions from module file
/opt/nifi/nifi-current/python_extensions/development/debug/__init__.py. This
module will be ignored.
Traceback (most recent call last):
File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 202,
in __discover_extensions_from_paths
self.__gather_extension_details(module_file, path, dependencies_bundled,
work_dir)
File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 237,
in __gather_extension_details
self.__gather_extension_details(child_module_file, extension_home,
dependencies_bundled, work_dir, local_dependencies=local_dependencies)
File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 239,
in __gather_extension_details
classes_and_details = self.__get_processor_classes_and_details(module_file,
extension_home, dependencies_bundled)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 269,
in __get_processor_classes_and_details
details = ProcessorInspection.get_processor_details(class_node,
module_file, extension_home, dependencies_bundled=dependencies_bundled)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
154, in get_processor_details
property_descriptions = get_property_descriptions(class_node,
module_string_constants)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
288, in get_property_descriptions
visitor.visit(class_node)
File "/usr/lib/python3.11/ast.py", line 418, in visit
return visitor(node)
^^^^^^^^^^^^^
File "/usr/lib/python3.11/ast.py", line 426, in generic_visit
self.visit(item)
File "/usr/lib/python3.11/ast.py", line 418, in visit
return visitor(node)
^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
94, in visit_Assign
value = self.resolve_dependencies(keyword.value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
62, in resolve_dependencies
if not self.discovered_property_descriptors[variable_name]:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
KeyError: 'MY_MODULE_PROPERTY{code}
Because of this issue, upgrades beyond NiFi 2.0.0 are blocked for anyone
relying on this import pattern.
This bug may be related to changes introduced in NIFI-13997.
I've attached a simple processor structure that uses such pattern
:[^python_extension.tar.gz]
*Steps to Reproduce*
# Create a Python module that exports a property descriptor (e.g.,
MY_MODULE_PROPERTY).
# Import that property into a custom Python processor class.
# Attempt to use the imported property as a dependency in another
PropertyDescriptor.
# Observe that the processor fails to load with a KeyError during startup.
*Expected Behavior*
The processor should allow using imported properties in both the
property_descriptors list and in PropertyDependency, as in earlier versions of
NiFi (2.0.0-M4 and 2.0.0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
