Fabian Reiber created NIFI-14621:
------------------------------------
Summary: Failed to import external python dependencies in nar
extensions managed by nifi registry
Key: NIFI-14621
URL: https://issues.apache.org/jira/browse/NIFI-14621
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 2.3.0
Environment: Podman 5.5.0
Debian 12
Nifi 2.3.0
Nifi-Registry 1.26.0
Reporter: Fabian Reiber
I wrote a simple Python processor with an external dependency, created a nar
file with hatch, uploaded the file to a Nifi Registry and configured the [Nifi
nar registry
property|https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#nifi-registry-nar-provider]
in {{nifi.properties}} to use this registry. After a restart of Nifi, it logs
an error that the dependency can't be found:
{code:python}
2025-06-02 12:35:18,000 ERROR [python-log-212] py4j.java_gateway There was an
exception while executing the Python Proxy on the Python Side.
Traceback (most recent call last):
File "/opt/nifi/nifi-current/python/framework/py4j/java_gateway.py", line
2506, in _call_proxy
return_value = getattr(self.pool[obj_id], method)(*params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/./python/framework/Controller.py", line 71, in
createProcessor
processorClass = self.extensionManager.getProcessorClass(processorType,
version, work_dir)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 105,
in getProcessorClass
processor_class = self.__load_extension_module(module_file,
details.local_dependencies)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nifi/nifi-current/python/framework/ExtensionManager.py", line 365,
in __load_extension_module
module_spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File
"/opt/nifi/nifi-current/work/nar/extensions/processors::processors-nar::0.0.1::444c6774-eb67-4bf2-abab-25beb1631736.nar-unpacked/processors/hello_world.py",
line 1, in <module>
import redis
ModuleNotFoundError: No module named 'redis'
{code}
Regardless of whether it makes sense to use redis in a separate python module,
it does not work with pandas or any other third party dependency as well.
But if I replace the colons of the filename of the automatically downloaded nar
file
{{processors::processors-nar::0.0.1::444c6774-eb67-4bf2-abab-25beb1631736.nar}}
with dashes
{{processors-processors-nar-0.0.1-444c6774-eb67-4bf2-abab-25beb1631736.nar}} it
works.
It does not explain why the python environment does not find the external
dependencies. I would suggest, that the loader fails parsing the nar file due
to the colons. I tested it with Nifi 2.3.0 but would expect, that 2.4.0 is
affected as well, because I did not find any release note which indicates a fix.
The workaround is not suitable. For any new version or new processor, I have to
rename it, after all nar files were downloaded.
I found out that the
[NiFiRegistryExtensionBundleMetadata|https://github.com/apache/nifi/blob/rel/nifi-2.3.0/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/extension/NiFiRegistryExtensionBundleMetadata.java#L27]
class uses the colons as a filename separator.
I used [this how
to|https://datavolo.io/2024/05/how-to-package-and-deploy-python-processors-for-apache-nifi/]
to deploy a nar file with hatch. After the nar file was created with {{hatch
build --target nar}}, I uploaded it to the registry with {{curl -F
file=@dist/processors-0.0.1.nar
http://localhost/nifi-registry-api/buckets/4a7579d3-c41e-4a9b-a3db-9f57340c9e29/bundles/nifi-nar}}.
If you need more information, please let me know :)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
