potiuk commented on PR #36537:
URL: https://github.com/apache/airflow/pull/36537#issuecomment-1883270253
> I was "naively" testing and miserably failed. Used the current branch,
created a new venv, then called pip install -e ., followed by pip install -e
.[ssh,docker]. But then calling airflow standalone failed miserably with:
> ModuleNotFoundError: No module named 'flask_appbuilder'
OK @jscheffl . I **think** I found a good solution that seemed impossible at
first - I think we can have cake and eat it too.
* the `flask_application_builder` package and other dependencies are not
really NEEDED by anything else than webserver.. They are needed for two things
only:
- to retrieve connection UI widgets from Hooks when providers are
discovered
- to instantiate `auth_manager` class for webserver
Both cases - we only need them for webserver. For scheduler, worker,
triggerer and most unit tests, they are not really needed.
* in the future we can optimise bootstrapping of Airflow CLI to completely
skip those steps - but in case of generally if information provided by the two
- provider connection widgets, and auth_manger, is not available, nothing bad
happens. Things will work as usual.
* so, we can selectively handle import errors on provider's widet retrieval.
This way connection's widget information will be missing, but it is only used
by Webserver, so it's ok to just raise warnings (there will be a lot of them -
for each connection) and continue as usual. The warnings will say that widgets
will not be available for hook class X.
* similarly when auth_manager is not initialized, we can just raise a
warning that "webserver will not work" and continue (unless we start webserver
of course in which case we should print detailed exception and exit with error).
Eventually, when we do it this way, we will eat cake and have it too.
* we do not need to install `fab` packages by default in the `pip install
-e .` case, so that "naive" installable installation will be somewhat usable
(i.e. airflow CLI will work, ,most of the unit tests will work etc. etc). It
will also install considerably faster than `pip install -e ".[devel]"`
* however in all the documentation we will present `[devel]` as the
**right** way of installing truly complete (but minimal) development
environment. It has things like `pytest` - with all extensions we use,
'request_mock', `coverage`, `async` libraries, `black`, `ruff`, `pre-commit`,
`mypy` - generally everything you'd need from a complete development
environment. That one will also include all pre-installed provider's
dependencies (this way we will have generally the same enviornment as after
regular airflow installation pulls pre-installed providers - inlcuing fab,
sqlite and other pre-installed ones.
* similarly all the `hatch` managed environments in our pyproject.toml
will have `devel` as default extra
I think this is the best approach that nicely combines the consistency of
our extras/devel dependencies with the ability of quickly installing `bare`
editable airflow with `pure` airflow core requirements only (and serve naive
users who will just run `pip install -e .`.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
