[
https://issues.apache.org/jira/browse/NIFI-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18043526#comment-18043526
]
Julien Ledoux commented on NIFI-14233:
--------------------------------------
Hi NiFi team,
I hope you’re doing well.
I’m reaching out to follow up on this ticket regarding the Python processor
compatibility issue introduced after the 2.1.0 release.
We reported this problem back in February, and it still prevents us from
upgrading beyond NiFi 2.0.0 in production. Our pipelines rely heavily on custom
Python processors, and without this fix, we cannot adopt the latest versions of
NiFi, including important security, feature, and stability improvements, nor
package our processors as NARs (since Python was 3.9 in 2.0.0, while the
Datavolo Hatch NAR plugin requires Python 3.11).
Custom Python processor support was introduced as a key enhancement in NiFi
2.0, and we continue to rely on it heavily. We hope to see full compatibility
restored in upcoming releases.
I fully understand the team has many priorities, but given the impact on our
deployments, could you please let us know if there are any updates on the patch
timeline, or if there is anything we can provide to help move the issue forward?
Even an estimated target version or a confirmation of the planned approach
would greatly help us plan our releases.
Thank you very much for your time and support.
We truly appreciate your efforts and look forward to your guidance on the next
steps.
Best regards,
Julien
> Python Processor can not use imported properties as PropertyDependency
> ----------------------------------------------------------------------
>
> Key: NIFI-14233
> URL: https://issues.apache.org/jira/browse/NIFI-14233
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0
> Environment: Official Docker Images NiFi 2.x running on Ubuntu
> 22.04.4 LTS and Docker v27.0.3
> Reporter: Julien Ledoux
> Assignee: Himanshu sahu Ksolve NiFi Expert
> Priority: Blocker
> Labels: extensions, python
> Attachments: debug_modules.zip
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When developing Python processors for NiFi, it is often necessary to share
> properties and methods across multiple processors by placing them into a
> shared module and importing them into each processor class.
> This approach worked correctly in NiFi versions 2.0.0-M4 and 2.0.0. However,
> starting with NiFi 2.1.0, referencing an imported property as a
> PropertyDependency causes a crash.
> For example, in NiFi 2.2.0, using an imported property in
> property_descriptors still works fine:
> {code:java}
> from MyModule import MY_MODULE_PROPERTY
> from nifiapi.flowfilesource import FlowFileSource
> class MyFlowFileSourceProcessor(FlowFileSource):
> ...
> property_descriptors = [
> MY_MODULE_PROPERTY
> ]{code}
> But if the same property is used as a dependency, NiFi fails to load the
> processor:
> {code:java}
> from MyModule import MY_MODULE_PROPERTY
> from nifiapi.flowfilesource import FlowFileSource
> from nifiapi.properties import PropertyDescriptor, PropertyDependency,
> StandardValidators, ExpressionLanguageScope
> class MyFlowFileSourceProcessor(FlowFileSource):
> ...
> MY_PROPERTY = PropertyDescriptor(
> name="Number of Spaces",
> description="Number of spaces to use for pretty-printing",
> validators=[StandardValidators.POSITIVE_INTEGER_VALIDATOR],
> expression_language_scope=ExpressionLanguageScope.ENVIRONMENT,
> default_value="4",
> required=True,
> dependencies=[
> PropertyDependency(MY_MODULE_PROPERTY, ("false"))
> ]
> ){code}
> This results in a KeyError: 'MY_MODULE_PROPERTY' exception, preventing the
> extension from loading:
> {code:java}
> 2025-02-04 15:33:43,065 ERROR [python-log-288] python.ExtensionManager Failed
> to load Python extensions from module file
> /opt/nifi/nifi-current/python_extensions/development/debug/__init__.py. This
> module will be ignored.
> Traceback (most recent call last):
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 202, in __discover_extensions_from_paths
> self.__gather_extension_details(module_file, path, dependencies_bundled,
> work_dir)
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 237, in __gather_extension_details
> self.__gather_extension_details(child_module_file, extension_home,
> dependencies_bundled, work_dir, local_dependencies=local_dependencies)
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 239, in __gather_extension_details
> classes_and_details =
> self.__get_processor_classes_and_details(module_file, extension_home,
> dependencies_bundled)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line
> 269, in __get_processor_classes_and_details
> details = ProcessorInspection.get_processor_details(class_node,
> module_file, extension_home, dependencies_bundled=dependencies_bundled)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 154, in get_processor_details
> property_descriptions = get_property_descriptions(class_node,
> module_string_constants)
>
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 288, in get_property_descriptions
> visitor.visit(class_node)
> File "/usr/lib/python3.11/ast.py", line 418, in visit
> return visitor(node)
> ^^^^^^^^^^^^^
> File "/usr/lib/python3.11/ast.py", line 426, in generic_visit
> self.visit(item)
> File "/usr/lib/python3.11/ast.py", line 418, in visit
> return visitor(node)
> ^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 94, in visit_Assign
> value = self.resolve_dependencies(keyword.value)
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> File "/opt/nifi/nifi-current/python/framework/ProcessorInspection.py", line
> 62, in resolve_dependencies
> if not self.discovered_property_descriptors[variable_name]:
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
> KeyError: 'MY_MODULE_PROPERTY{code}
> Because of this issue, upgrades beyond NiFi 2.0.0 are blocked for anyone
> relying on this import pattern.
> I've attached a simple processor structure that uses such pattern :
> [^debug_modules.zip]
> *Steps to Reproduce*
> # Create a Python module that exports a property descriptor (e.g.,
> MY_MODULE_PROPERTY).
> # Import that property into a custom Python processor class.
> # Attempt to use the imported property as a dependency in another
> PropertyDescriptor.
> # Observe that the processor fails to load with a KeyError during startup.
> *Expected Behavior*
> The processor should allow using imported properties in both the
> property_descriptors list and in PropertyDependency, as in earlier versions
> of NiFi (2.0.0-M4 and 2.0.0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
