potiuk commented on issue #33241: URL: https://github.com/apache/airflow/issues/33241#issuecomment-1670447311
There is no points in installing extra dependencies with constraints. It's wrong. See our documentation. I will simply copy explanation from https://github.com/apache/airflow/issues/33169#issuecomment-1667720954 This is a misunderstanding on how constraints work. When you read details on constraints, the main (and only) reason for constrainst is to provide a reproducible installation for airflow and all community providers released at the moment of airflow release). However that does NOT include case when you are installing optional dependencies of some of the dependencies of airflow. If they are not included during installation of airflow, one of it's extras or one of it's providers constraints might be conflicting. There are no guarantees of any sort. People also often make the mistake to install anything "new" with constraints on top of airflow. This is wrong. This is now clearly describe in the docuemntation (as Elad pointed out) and I am going to give a whole talk about it at the upcoming Airflow Summit - https://airflowsummit.org/sessions/2023/mastering-dependencies-the-airflow-way/ - recording will be published after that. Constraints are there to provide reproducible installation of Airflow. Full stop. Constraints is something that you SHOULD use if you are installing airflow with constraints in a clear venv. And this where constraints role ends. If you are adding anything on top of it, constraints makes simply no sense. You should use the usual dependency mechanism. Using snowflake-connector-python[secure-local-storage] adds a different dependency specified in "secure-local-storage' extra of snowflake-connector-python. And as long it does not conflct with other requirements of other packages, this is fine to upgrade or downgrade the version. Ths is PRECISELY why we are using constraints - because the do not limit you in upgrading and downgrading other packages as your OTHER dependencies/new requirements you specify might need them. Also, it would be great if we could figure out what all versions require keyring. I'll more than happy to investigate this. This is a very good question. But this is a question for snowflake-connector-python team, documentation and changelog. If you choose one of our dependencies (snowflake-connector-python) and decide to manually install an optional dependency of it (this is is what [secure-local-storage] extra does), then you should check whatever the documentation, changelog etc. of snowflake-connector-python. Airflow has 650 dependencies (including transitive ones). It would be entirely infisible to bring to airflow documentation or anyhow to our user's attention - every single variation of every single optional extra of every single dependency. This is likely a matrix of likely of a 10s of milions of entries including all combinations. This is why - when you are actually diverging and installing some "other" dependencies than the ones that default airflow installation covers, it's up to the user to make sure that all the dependencies are ok. So - look at snowflake-connector-python documentation and see what they say. If the documentation is lacking - look at the package content and requirements they have. We cannot do much here in Airflow about all those optonal things. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
