[
https://issues.apache.org/jira/browse/NIFI-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann updated NIFI-14233:
------------------------------------
Fix Version/s: 2.8.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Python Processor can not use imported properties as PropertyDependency
> ----------------------------------------------------------------------
>
> Key: NIFI-14233
> URL: https://issues.apache.org/jira/browse/NIFI-14233
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0
> Environment: Official Docker Images NiFi 2.x running on Ubuntu
> 22.04.4 LTS and Docker v27.0.3
> Reporter: Julien Ledoux
> Assignee: Pierre Villard
> Priority: Blocker
> Labels: extensions, python
> Fix For: 2.8.0
>
> Attachments: debug_modules.zip
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> When developing Python processors for NiFi, it is often necessary to share
> properties and methods across multiple processors by placing them into a
> shared module and importing them into each processor class.
> This approach worked correctly in NiFi versions 2.0.0-M4 and 2.0.0. However,
> starting with NiFi 2.1.0, referencing an imported property as a
> PropertyDependency causes a crash.
> For example, in NiFi 2.2.0, using an imported property in
> property_descriptors still works fine:
> {code:java}
> from MyModule import MY_MODULE_PROPERTY
> from nifiapi.flowfilesource import FlowFileSource
> class MyFlowFileSourceProcessor(FlowFileSource):
> ...
> property_descriptors = [
> MY_MODULE_PROPERTY
> ]{code}
> But if the same property is used as a dependency, NiFi fails to load the
> processor:
> {code:java}
> from MyModule import MY_MODULE_PROPERTY
> from nifiapi.flowfilesource import FlowFileSource
> from nifiapi.properties import PropertyDescriptor, PropertyDependency,
> StandardValidators, ExpressionLanguageScope
> class MyFlowFileSourceProcessor(FlowFileSource):
> ...
> MY_PROPERTY = PropertyDescriptor(
> name="Number of Spaces",
> description="Number of spaces to use for pretty-printing",
> validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
> expression_language_scope=ExpressionLanguageScope.ENVIRONMENT,
> default_value="4",
> required=True,
> dependencies=[
> PropertyDependency(MY_MODULE_PROPERTY, ("false"))
> ]
> ){code}
> This results in a KeyError: 'MY_MODULE_PROPERTY' exception, preventing the
> extension from loading:
> {code:java}
> 2025-02-04 15:33:43,065 ERROR [python-log-288] python.ExtensionManager Failed
> to load Python extensions from module file
> /opt/nifi/nifi-current/python_extensions/development/debug/__init__.py. This
> module will be ignored.
> Traceback (most recent call last):
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 202, in __discover_extensions_from_paths
> self.__gather_extension_details(module_file, path, dependencies_bundled,
> work_dir)
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 237, in __gather_extension_details
> self.__gather_extension_details(child_module_file, extension_home,
> dependencies_bundled, work_dir, local_dependencies=local_dependencies)
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 239, in __gather_extension_details
> classes_and_details =
> self.__get_processor_classes_and_details(module_file, extension_home,
> dependencies_bundled)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 269, in __get_processor_classes_and_details
> details = ProcessorInspection.get_processor_details(class_node,
> module_file, extension_home, dependencies_bundled=dependencies_bundled)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 154, in get_processor_details
> property_descriptions = get_property_descriptions(class_node,
> module_string_constants)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 288, in get_property_descriptions
> visitor.visit(class_node)
> File "/usr/lib/python3.11/ast.py", line 418, in visit
> return visitor(node)
> ^^^^^^^^^^^^^
> File "/usr/lib/python3.11/ast.py", line 426, in generic_visit
> self.visit(item)
> File "/usr/lib/python3.11/ast.py", line 418, in visit
> return visitor(node)
> ^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 94, in visit_Assign
> value = self.resolve_dependencies(keyword.value)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 62, in resolve_dependencies
> if not self.discovered_property_descriptors[variable_name]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
> KeyError: 'MY_MODULE_PROPERTY{code}
> Because of this issue, upgrades beyond NiFi 2.0.0 are blocked for anyone
> relying on this import pattern.
> I've attached a simple processor structure that uses such pattern :
> [^debug_modules.zip]
> *Steps to Reproduce*
> # Create a Python module that exports a property descriptor (e.g.,
> MY_MODULE_PROPERTY).
> # Import that property into a custom Python processor class.
> # Attempt to use the imported property as a dependency in another
> PropertyDescriptor.
> # Observe that the processor fails to load with a KeyError during startup.
> *Expected Behavior*
> The processor should allow using imported properties in both the
> property_descriptors list and in PropertyDependency, as in earlier versions
> of NiFi (2.0.0-M4 and 2.0.0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
