potiuk commented on PR #29465: URL: https://github.com/apache/airflow/pull/29465#issuecomment-1604017790
We have an issue about it https://github.com/apache/airflow/issues/30764 already and yeah. I think it needs a bit closer look. I think it might be quite tricky to solve - our setup.py is pretty, well complex, so for now I kept on ignoring it and I think we will need to solve it possibly by generally modernizing out package build tool configuration - part of it has been done in #31378 . What makes it difficult is that we want to keep the possibilities of: 1) Being able to run `pip install -e .` and have all "airfow + providers" in one editable environment 2) Similarly - have the environment in Breeze where both airflow and providers are immediately "editable" - i.e. installed from sources so that you do not have to reinstall provider package every time you edit the source code of it. 3) But also have an option to only install airflow, without the providers and install the providers from `pypi` (for CI and testing and constraint generation). So far this has been achieved by `INSTALL_PROVIDERS_FROM_SOURCES` env variable that determines the approach taken (and in breeze for example `INSTALL_PROVIDERS_FROM_SOURCES` is set to `true`. There is also quite complex setup.py code that has been handling that, but in recent releases of `pip` and `setuptools` some of the `hacks` we were using to achieve that got subtly brokent and we need to figure out another approach. It might also be that we will only be able to do it "well" when we switch to a "proper" structure for provider's code, following https://github.com/apache/airflow/pull/28291 (scripts) and https://github.com/apache/airflow/pull/28292 (POC migration) - but then solving the above a little contradicting requirements might be tricky. I've been following what's going on in Python PyPa and maybe we should switch to [flit](https://github.com/pypa/flit) or [hatch](https://github.com/pypa/hatch) - both are part of `PyPa` and both have great maintainers who are working on them. I believe that in hatch (listening to recent podcast of hatch creator: https://talkpython.fm/episodes/show/408/hatch-a-modern-python-workflow there is an idea or maybe even an early implementation already for something similar to our case where you have multiple projects in monorepo that should be edit-installable together. And my preference would be rather than invent something on our own, tocontribute some work to hatch to make it capable of doing what we want to do. I think this is a great opportunity if our goals and ideas are aligned, because we could provide a nice testing ground for such case for hatch. @uranusjr - I think you are the best person here to have a say and recommendations what we should do and likely you even have some good relation with hatch maintainer and we could jointly do something together and maybe even help to set the standards for the Python packaging worlds ? We could setup a small team around it and work on both - contributing things to hatch (if the direction would be aligned) and applying it to Airflow at the same time. Maybe we should brainstorm a little here or in #30764 on what we should do and how we should approach it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
