jscheffl commented on code in PR #36537:
URL: https://github.com/apache/airflow/pull/36537#discussion_r1445451492
##########
dev/preinstalled_providers.json:
##########
@@ -0,0 +1,9 @@
+[
+ "http",
+ "common.io",
+ "common.sql",
+ "ftp",
+ "http",
+ "imap",
+ "sqlite"
Review Comment:
I was "naively" testing and miserably failed. Used the current branch,
created a new venv, then called `pip install -e .`, followed by `pip install -e
.[ssh,docker]`. But then calling `airflow standalone` failed miserably with:
```
(.test-hatch) jscheffl@hp860g9:~/Workspace/airflow$ airflow standalone
[2024-01-08T23:52:25.061+0100] {configuration.py:1194} ERROR - No module
named 'flask_appbuilder'
[2024-01-08T23:52:25.064+0100] {cli_parser.py:76} ERROR - cannot load CLI
commands from auth manager
Traceback (most recent call last):
File "/home/jscheffl/Workspace/airflow/airflow/configuration.py", line
1192, in getimport
return import_string(full_qualified_path)
File "/home/jscheffl/Workspace/airflow/airflow/utils/module_loading.py",
line 39, in import_string
module = import_module(module_path)
File "/usr/lib/python3.10/importlib/__init__.py", line 126, in
import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in
_call_with_frames_removed
File
"/home/jscheffl/Workspace/airflow/airflow/providers/fab/auth_manager/fab_auth_manager.py",
line 52, in <module>
from airflow.providers.fab.auth_manager.models import Permission, Role,
User
File
"/home/jscheffl/Workspace/airflow/airflow/providers/fab/auth_manager/models/__init__.py",
line 27, in <module>
from flask_appbuilder.models.sqla import Model
ModuleNotFoundError: No module named 'flask_appbuilder'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/jscheffl/Workspace/airflow/airflow/cli/cli_parser.py", line
73, in <module>
auth_mgr = get_auth_manager_cls()
File
"/home/jscheffl/Workspace/airflow/airflow/www/extensions/init_auth_manager.py",
line 37, in get_auth_manager_cls
auth_manager_cls = conf.getimport(section="core", key="auth_manager")
File "/home/jscheffl/Workspace/airflow/airflow/configuration.py", line
1195, in getimport
raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: The object could not be loaded.
Please check "auth_manager" key in "core" section. Current value:
"airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager".
[2024-01-08T23:52:25.148+0100] {providers_manager.py:970} WARNING -
Exception when importing 'airflow.hooks.filesystem.FSHook' from
'airflow.hooks.filesystem' package: No module named 'flask_appbuilder'
Traceback (most recent call last):
File "/home/jscheffl/Workspace/airflow/.test-hatch/bin/airflow", line 8,
in <module>
sys.exit(main())
File "/home/jscheffl/Workspace/airflow/airflow/__main__.py", line 54, in
main
conf = write_default_airflow_configuration_if_needed()
File "/home/jscheffl/Workspace/airflow/airflow/configuration.py", line
1989, in write_default_airflow_configuration_if_needed
conf.write(
File "/home/jscheffl/Workspace/airflow/airflow/configuration.py", line
673, in write
with
self.make_sure_configuration_loaded(with_providers=include_providers):
File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/home/jscheffl/Workspace/airflow/airflow/configuration.py", line
527, in make_sure_configuration_loaded
ProvidersManager()._initialize_providers_configuration()
File "/home/jscheffl/Workspace/airflow/airflow/utils/singleton.py", line
32, in __call__
cls._instances[cls] = super().__call__(*args, **kwargs)
File "/home/jscheffl/Workspace/airflow/airflow/providers_manager.py", line
445, in __init__
self._init_airflow_core_hooks()
File "/home/jscheffl/Workspace/airflow/airflow/providers_manager.py", line
471, in _init_airflow_core_hooks
self._hook_provider_dict[hook_info.connection_type] = HookClassProvider(
AttributeError: 'NoneType' object has no attribute 'connection_type'
```
To fix it I needed to install fab provider - missing in the default! `pip
install -e .[fab]`
so fab should be installed by default as dependency. Anyway while updating
this I recommend sorting alphabetically... and have you seen http is duplictae?
:-D
```suggestion
"common.io",
"common.sql",
"fab",
"ftp",
"http",
"imap",
"sqlite"
```
##########
dev/preinstalled_providers.json:
##########
@@ -0,0 +1,9 @@
+[
+ "http",
+ "common.io",
+ "common.sql",
+ "ftp",
+ "http",
+ "imap",
+ "sqlite"
Review Comment:
Other question anyway, just thinking of beautification - why adding a new
JSON file? Have you considered using a YAML instead?
##########
docs/apache-airflow/extra-packages-ref.rst:
##########
@@ -97,7 +106,7 @@ with a consistent set of dependencies based on constraint
files provided by Airf
.. code-block:: bash
:substitutions:
- pip install apache-airflow[google,amazon,apache.spark]==|version| \
+ pip install apache-airflow[google,amazon,apache-spark]==|version| \
Review Comment:
For me _all_ is consistent now and here, was reading pep-0685 and all sounds
resonable. Resolving here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
