potiuk commented on pull request #11950: URL: https://github.com/apache/airflow/pull/11950#issuecomment-784527287
> i.e. it seems sub optimal to require that every provider support the new python version before airflow can be updated as that could lead to less popular providers blocking the rest of the community. Yeah. It is suboptimal. And we already had a number of problems because of that. But the way it works currently is that - unforatunately - all providers and core run in one python env and this means they have to share dependencies. Not much we can do now, and while we ha some ideas how to improve that, this is - in most cases - not as big problem as it seems. In most cases we have lower-bound dependencies (rather than upper-bound) and different providers have only small subset of overlapping dependencies. So there are just a few providers that are problematic - and snowflake is one of them. The benefits of not having to introduce isolation between providers are important - simplicity, debuggability, easy deployment, easy management of envs, upgrades etc. Are important. and we developed the whole CI process to keep the "golden" dependencies up-to-date. There are really two problematic providers/dependencies: * apache-beam * snowflake If we can help to sort them out, we should be really good - without having to introduce another approach. Taking into account that 3 of Airflow Committers /PMC members are now @ Snowflake - I think we can improve the former. And 3 committers of beam are also with us. so....... ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
