potiuk commented on PR #27264: URL: https://github.com/apache/airflow/pull/27264#issuecomment-1458962013
> We also use Airflow with hardly any providers. Just cncf+celery+docker, to orchestrate our business logic in taskflow. It still gives plenty of value in this configuration. Sure. And for that case, I think you can try/use Airflow with Python 3.11 already, I think there are no problems with it, I think. There is nothing to **prevent** airflow from being installed and run in 3.11 environment, it just does not mark explicitly support for it and (see below) we do not have some artifacts produced for 3.11 - such as constraints, images. > Isn't the idea of providers to decouple the core from dependency explosion, exactly to prevent ending up in situation such as this where dependency deep inside a single provider blocks the rest. No. This is not the reason of decoupling providers. The reason for decoupling providers is to enable the users to be able to upgrade and downgrate them separately from Airlfow core, thus allowing to use new (or rely on old) behaviours of providers independently from the Airflow version used. Airlfow without providers have very limited value. I understand there are some usages like yours, but this is a "niche" case (I believe) and we have many more users who are using very popular providers. > Maybe this is already possible if one installs airflow manually via pip? Yes. It should work. > Right now I've mostly used the pre-packaged slim- docker images, that don't come with any providers installed. Could it be an option to release slim images for 3.11 while waiting for providers to release the full images? You are quite free to build your own image if you want - with 3.11. This should be rather straightforward, I can help you with that if you have trouble with it. I just do not want to **just** release slim image and say we are "3.11" ready - because I honestly think from vast majority of users perspective, we are not. The main reason why "support for 3.11" is needed realy is: * support for constraints so that users can install airflow + selected providers in a repeatable way * support for images that contain popular, predefined set of providers. Making airflow "support 3.11" is mostly about those two points. When we are releasing airflow, we release it as a source code and we provide some convenience packages as well for users (Pypi packages, container images). When we say "we support 3.11" we have to follow what we promised and say "And this and that is supported". Surely we **could** release only pypi package and slim image, but it would mean that a lot of the regular workflows of our users would not work. For exampel Anyone who uses `pip install apache-airflow[google]` would fail to do so (and in rather nasty way as pip will start back-tracking and it will take 10s of minutes until it decides to give up. And if someone expects "apache/airlfow:python-3.11" image to be there, they will be disappointed (either because it will not be there or because it will have some crucial differences vs. 3.10). So when we say "airflow supports 3.11", it is much more than "apache-airflow package can be installed and run on 3.11". If we are to exclude the most popular packages, when we say "we support 3.11", then it would have to be stated very clearly and even if we do, there will be many people surprised. In fact I see very little reason to say we have "3.11" version support if we do not release constraints nor images for the most important providers. So I would use that as a "last resort" really - I know for a fact, for example Google has quite a big "dependency bump" coming soon for their provider, hopefully snowflake and others will follow the suite. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
