sgoel2be24-cyber opened a new pull request, #74531: URL: https://github.com/apache/airflow/pull/74531
On RHEL and Fedora, a fresh virtualenv install with the Kafka provider reports `Plugin 'kafka_event_producer' already registered, skipping` as a plugin import error (in the logs and in the UI), even though the plugin is installed once. These distros set `sys.platlibdir` to `lib64`, so `site` puts both `<venv>/lib64/pythonX.Y/site-packages` and `<venv>/lib/pythonX.Y/site-packages` on `sys.path`. `venv` creates `lib64` as a symlink to `lib`, and `site` doesn't resolve symlinks when it deduplicates paths. `importlib.metadata.distributions()` returns a distribution once per `sys.path` entry it is found on, so `entry_points_with_dist()` yields every `airflow.plugins` entry point twice. Its docstring already says this and leaves deduplication to the caller. Provider discovery does that (it skips package names it has already seen), but `_load_entrypoint_plugins` didn't, so each entry-point plugin was loaded twice. Since #66649 the second copy is recorded as a plugin import error, which is what made this visible. `_load_entrypoint_plugins` now skips an entry point it has already seen. `EntryPoint` objects compare and hash on name, value and group, so the second copy of the same distribution is dropped before it's loaded. The change is in the shared `plugins_manager` library, so it covers both airflow-core and the Task SDK. Of the providers in the report, only Kafka ships an entry-point plugin. OpenLineage, Databricks, Edge3, Hive, Informatica and Common AI plugins are affected the same way. Tested locally: - Reproduced on macOS by appending a symlink to `site-packages` to `sys.path`. On `main`, `plugins_manager._get_plugins()` reports `Plugin 'OpenLineageProviderPlugin' already registered, skipping` as an import error, under the same `<dist>==<version>: EntryPoint(...)` key as in the issue. With this change, all 7 entry-point plugins in the workspace load once and there are no import errors - The new test in `shared/plugins_manager` fails on `main` (the plugin is loaded twice) and passes with the change, on Python 3.11 and 3.12 - `pytest` on airflow-core's plugin tests (plugins manager, `airflow plugins` CLI, plugins API routes and datamodels): 103 passed. Task SDK `test_plugins_manager.py`: 2 passed - In `shared/plugins_manager`, `test_entrypoint_plugin_errors_dont_raise_exceptions` fails when the whole file runs, on `main` as well. `_get_grouped_entry_points` is cached, so the `distributions()` mock from the earlier test leaks into it. It passes on its own, and this PR doesn't touch it - `prek` pre-commit and manual stages pass, including `mypy-shared-plugins_manager`. `breeze verify` lists the core test suites, which need Docker, and there's no Docker on my machine, so they'll run in CI closes: #74502 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
