potiuk commented on issue #57431: URL: https://github.com/apache/airflow/issues/57431#issuecomment-3477951360
> [@potiuk](https://github.com/potiuk) thanks for the reminder. I have a question, the fact that we use airflow 3.1.1 image, shouldn't we be forced to use apache-airflow==3.1.1 even if not specified? a constraint? No. The fact that it is a container image with specific version of airflow preinstalled, does not change how `pip` works - it considers all installed packages as simple set of packages and it has no notion of which package is the one that should be kept in the version it was installed with - we have to explicitly tell `pip` that we want to keep the package in the version we want: This is **PRECISELY** what this suggested way of doing it does: ``` pip install apache-airflow=3.1.1 your_packages ``` It's meaning, when you run it where airflow 3.1.1 is installed is: "Please install your_packages, while also making sure that apache-airflow remains at 3.1.1 but feel free to modify any of the other packages that are conflicting" - that's how the exact instruction to `pip` resolver looks like. This instruction on the other hand: ``` pip install your_packages ``` Means - "Please install your_packages, and feel free to upgrade or downgrade any other packages that you might find conflicting" (including apache-airflow - there is **nothing special** about apache-airflow package from `pip` point of view - it's one of the 700+ packages installed and it's not "less" or "more" important than others (unless you explicitly tell that to `pip`). As far as I know the way our documentation explains is the only way how you can make it work in the intended way. This is not how Pip works - there is nothing we can do about it. I **did** consider adding apache-airflow=X.Y.Z to constraints but it makes completely no sense - because the problem you experience is when you DO NOT use constraints. Our constraints (and this is very clearly specified in the docs) are ONLY supposed to be used in the scenario where you install airflow for the first time: ``` pip install apache-airflow[extras]==3.1.1 --constraints ....3.1.1.... ``` You are already using version specification in both places and this is the ONLY time you should use constraints of ours. Full stop. In all other cases: a) if you do not use constraints, pip might freely decide what it does with ANY package that is installed - including airflow. And we cannot do **anything** about it - other than telling pip to pin airflow (which is exactly what we explain in our docs) pip install apache-airflow==3.1.1 your_packages b) if you do use constraints, then this is wrong approach. If you already installed airflow with constraints as explained above. you could have and still want to add any extra packages WITH those constraints, the proper way is to add your packages to that initial command ``` pip install apache-airflow[extras]==3.1.1 --constraints ....3.1.1.... your_packages ``` Besides it's also impossible that airflow will get downgraded when you use constraints anyway. If you add any package that woudl cause airflow to be downgraded, then constraints will actually cause conflict before that - if there is no conflict with Airflow 3.1.1 constraints, Also in the future when package.lock [PEP 751](https://peps.python.org/pep-0751/) will be a supported way of installing packages in a reproducible way, we will be able to replace our custom constraints mechanism with it, and there is an idea being discussed (I discussed it on Python's discord no later than yesterday here - https://discuss.python.org/t/pre-pep-add-ability-to-install-a-package-with-reproducible-dependencies/99497/33 ) - when we standardize using `pipx` and using the `pep-751` lockfile to do such installations, we **might** be able to get a standard feature for it. As I explained there - there are two reasons for having such standardisation in place: > I would love if I would not have to tell the user two things: > > * where to get constraints from > * that they should pin their “driving” project to the original version when they are updating the environment As you see - I am well aware the need to "pin" the "driving' project is a friction and non-obvious thing for users like you. However - currently - we do not seem to have any tooling that could help with it and it needs some standardisation effort to happen. Of course - if you find any other to make it happens - feel absolutely free to open a PR and describe it in our docs or maybe even make a PR to update our Dockerfiles to make it works "frictionlessly" for our users, but so far I have not found such a solution - and we opted for documenting it hoping that: * those who read it will follow * those who don't read it will fail, come here and we will direct them to to the docs so that they read them -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
