[ https://issues.apache.org/jira/browse/NIFI-14233 ]
Himanshu sahu deleted comment on NIFI-14233:
--------------------------------------
was (Author: JIRAUSER309871):
I’ve raised a PR to resolve the issue with `Property Dependency` resolution for
imported `Property Descriptor` values in Python processors.
PR: https://github.com/apache/nifi/pull/10070
(Please review — this fixes the Key Error issue when referencing imported
properties)
Summary of Fix:
1.Updated `resolve_dependencies()` in `ProcessorInspection.py` to check
`module_string_constants` fallback.
2.Gracefully skips unresolved descriptors with proper logging.
3.Added test processors using shared imports to validate the fix.
Manual and unit testing completed. Everything is working as expected.
Please review and let me know if anything further is required.
> Python Processor can not use imported properties as PropertyDependency
> ----------------------------------------------------------------------
>
> Key: NIFI-14233
> URL: https://issues.apache.org/jira/browse/NIFI-14233
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 2.1.0, 2.2.0, 2.3.0
> Environment: Official Docker Images NiFi 2.x running on Ubuntu
> 22.04.4 LTS and Docker v27.0.3
> Reporter: Julien Ledoux
> Assignee: Himanshu sahu
> Priority: Blocker
> Labels: extensions, python
> Attachments: debug_modules.zip
>
>
> When developing Python processors for NiFi, it is often necessary to share
> properties and methods across multiple processors by placing them into a
> shared module and importing them into each processor class.
> This approach worked correctly in NiFi versions 2.0.0-M4 and 2.0.0. However,
> starting with NiFi 2.1.0, referencing an imported property as a
> PropertyDependency causes a crash.
> For example, in NiFi 2.2.0, using an imported property in
> property_descriptors still works fine:
> {code:java}
> from MyModule import MY_MODULE_PROPERTY
> from nifiapi.flowfilesource import FlowFileSource
> class MyFlowFileSourceProcessor(FlowFileSource):
> ...
> property_descriptors = [
> MY_MODULE_PROPERTY
> ]{code}
> But if the same property is used as a dependency, NiFi fails to load the
> processor:
> {code:java}
> from MyModule import MY_MODULE_PROPERTY
> from nifiapi.flowfilesource import FlowFileSource
> from nifiapi.properties import PropertyDescriptor, PropertyDependency,
> StandardValidators, ExpressionLanguageScope
> class MyFlowFileSourceProcessor(FlowFileSource):
> ...
> MY_PROPERTY = PropertyDescriptor(
> name="Number of Spaces",
> description="Number of spaces to use for pretty-printing",
> validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
> expression_language_scope=ExpressionLanguageScope.ENVIRONMENT,
> default_value="4",
> required=True,
> dependencies=[
> PropertyDependency(MY_MODULE_PROPERTY, ("false"))
> ]
> ){code}
> This results in a KeyError: 'MY_MODULE_PROPERTY' exception, preventing the
> extension from loading:
> {code:java}
> 2025-02-04 15:33:43,065 ERROR [python-log-288] python.ExtensionManager Failed
> to load Python extensions from module file
> /opt/nifi/nifi-current/python_extensions/development/debug/__init__.py. This
> module will be ignored.
> Traceback (most recent call last):
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 202, in __discover_extensions_from_paths
> self.__gather_extension_details(module_file, path, dependencies_bundled,
> work_dir)
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 237, in __gather_extension_details
> self.__gather_extension_details(child_module_file, extension_home,
> dependencies_bundled, work_dir, local_dependencies=local_dependencies)
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 239, in __gather_extension_details
> classes_and_details =
> self.__get_processor_classes_and_details(module_file, extension_home,
> dependencies_bundled)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 269, in __get_processor_classes_and_details
> details = ProcessorInspection.get_processor_details(class_node,
> module_file, extension_home, dependencies_bundled=dependencies_bundled)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 154, in get_processor_details
> property_descriptions = get_property_descriptions(class_node,
> module_string_constants)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 288, in get_property_descriptions
> visitor.visit(class_node)
> File "/usr/lib/python3.11/ast.py", line 418, in visit
> return visitor(node)
> ^^^^^^^^^^^^^
> File "/usr/lib/python3.11/ast.py", line 426, in generic_visit
> self.visit(item)
> File "/usr/lib/python3.11/ast.py", line 418, in visit
> return visitor(node)
> ^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 94, in visit_Assign
> value = self.resolve_dependencies(keyword.value)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 62, in resolve_dependencies
> if not self.discovered_property_descriptors[variable_name]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
> KeyError: 'MY_MODULE_PROPERTY{code}
> Because of this issue, upgrades beyond NiFi 2.0.0 are blocked for anyone
> relying on this import pattern.
> I've attached a simple processor structure that uses such pattern :
> [^debug_modules.zip]
> *Steps to Reproduce*
> # Create a Python module that exports a property descriptor (e.g.,
> MY_MODULE_PROPERTY).
> # Import that property into a custom Python processor class.
> # Attempt to use the imported property as a dependency in another
> PropertyDescriptor.
> # Observe that the processor fails to load with a KeyError during startup.
> *Expected Behavior*
> The processor should allow using imported properties in both the
> property_descriptors list and in PropertyDependency, as in earlier versions
> of NiFi (2.0.0-M4 and 2.0.0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
