[
https://issues.apache.org/jira/browse/NIFI-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18003543#comment-18003543
]
David Handermann commented on NIFI-14233:
-----------------------------------------
Thanks for working on this issue [~himanshu123nifi]! As noted on the pull
request, please review the pull request checklist and confirm that automated
unit tests are passing before submitting changes. There may be some
environmental differences in GitHub, but the initial set of changes failed on
all platforms, so confirming a successful complete build is an important step.
Glad to review the changes pending successful automated builds.
> Python Processor can not use imported properties as PropertyDependency
> ----------------------------------------------------------------------
>
> Key: NIFI-14233
> URL: https://issues.apache.org/jira/browse/NIFI-14233
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 2.1.0, 2.2.0, 2.3.0
> Environment: Official Docker Images NiFi 2.x running on Ubuntu
> 22.04.4 LTS and Docker v27.0.3
> Reporter: Julien Ledoux
> Assignee: Himanshu sahu
> Priority: Blocker
> Labels: extensions, python
> Attachments: debug_modules.zip
>
>
> When developing Python processors for NiFi, it is often necessary to share
> properties and methods across multiple processors by placing them into a
> shared module and importing them into each processor class.
> This approach worked correctly in NiFi versions 2.0.0-M4 and 2.0.0. However,
> starting with NiFi 2.1.0, referencing an imported property as a
> PropertyDependency causes a crash.
> For example, in NiFi 2.2.0, using an imported property in
> property_descriptors still works fine:
> {code:java}
> from MyModule import MY_MODULE_PROPERTY
> from nifiapi.flowfilesource import FlowFileSource
> class MyFlowFileSourceProcessor(FlowFileSource):
> ...
> property_descriptors = [
> MY_MODULE_PROPERTY
> ]{code}
> But if the same property is used as a dependency, NiFi fails to load the
> processor:
> {code:java}
> from MyModule import MY_MODULE_PROPERTY
> from nifiapi.flowfilesource import FlowFileSource
> from nifiapi.properties import PropertyDescriptor, PropertyDependency,
> StandardValidators, ExpressionLanguageScope
> class MyFlowFileSourceProcessor(FlowFileSource):
> ...
> MY_PROPERTY = PropertyDescriptor(
> name="Number of Spaces",
> description="Number of spaces to use for pretty-printing",
> validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
> expression_language_scope=ExpressionLanguageScope.ENVIRONMENT,
> default_value="4",
> required=True,
> dependencies=[
> PropertyDependency(MY_MODULE_PROPERTY, ("false"))
> ]
> ){code}
> This results in a KeyError: 'MY_MODULE_PROPERTY' exception, preventing the
> extension from loading:
> {code:java}
> 2025-02-04 15:33:43,065 ERROR [python-log-288] python.ExtensionManager Failed
> to load Python extensions from module file
> /opt/nifi/nifi-current/python_extensions/development/debug/__init__.py. This
> module will be ignored.
> Traceback (most recent call last):
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 202, in __discover_extensions_from_paths
> self.__gather_extension_details(module_file, path, dependencies_bundled,
> work_dir)
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 237, in __gather_extension_details
> self.__gather_extension_details(child_module_file, extension_home,
> dependencies_bundled, work_dir, local_dependencies=local_dependencies)
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 239, in __gather_extension_details
> classes_and_details =
> self.__get_processor_classes_and_details(module_file, extension_home,
> dependencies_bundled)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 269, in __get_processor_classes_and_details
> details = ProcessorInspection.get_processor_details(class_node,
> module_file, extension_home, dependencies_bundled=dependencies_bundled)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 154, in get_processor_details
> property_descriptions = get_property_descriptions(class_node,
> module_string_constants)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 288, in get_property_descriptions
> visitor.visit(class_node)
> File "/usr/lib/python3.11/ast.py", line 418, in visit
> return visitor(node)
> ^^^^^^^^^^^^^
> File "/usr/lib/python3.11/ast.py", line 426, in generic_visit
> self.visit(item)
> File "/usr/lib/python3.11/ast.py", line 418, in visit
> return visitor(node)
> ^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 94, in visit_Assign
> value = self.resolve_dependencies(keyword.value)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 62, in resolve_dependencies
> if not self.discovered_property_descriptors[variable_name]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
> KeyError: 'MY_MODULE_PROPERTY{code}
> Because of this issue, upgrades beyond NiFi 2.0.0 are blocked for anyone
> relying on this import pattern.
> I've attached a simple processor structure that uses such pattern :
> [^debug_modules.zip]
> *Steps to Reproduce*
> # Create a Python module that exports a property descriptor (e.g.,
> MY_MODULE_PROPERTY).
> # Import that property into a custom Python processor class.
> # Attempt to use the imported property as a dependency in another
> PropertyDescriptor.
> # Observe that the processor fails to load with a KeyError during startup.
> *Expected Behavior*
> The processor should allow using imported properties in both the
> property_descriptors list and in PropertyDependency, as in earlier versions
> of NiFi (2.0.0-M4 and 2.0.0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
