Yes - what Lars wrote about Python changing. This is nothing "airflow" specific - you cannot just "upgrade" installed packages with python - but that's basic python stuff. Also yes - you should know which extras you wanted to install and use the same extras when you upgrade if you want to reproducibly install Airflow + the providers you selected.
And to add to it- if you've modified and added something later - it's up to you to make sure it is working and not conflicting the documentation of "reproducible installation" **JUST** describes installation of Airflow and it's providers. If you want to install/upgrade anything else, you will likely have to repeat what you've done before, maybe choose other versions of other packages etc. - depends on your needs, It's impossible to know what else you want to install - so what we have described is what Airflow + community providers have - this is where the boundaries of "reproducible installation" described in our docs is. And then - of course - if you need to upgrade/downgrade any providers/other dependencies, you still can do it if you have some specific expectations - or for example want to upgrade some dependencies to latest verisons etc. But that's again - next step after you reproducibly install airflow and you are on your own there Also for future reference and questions - all this is all described in our docs "Upgrading Airflow together with providers" in "Installation and upgrades scenarios" in our docs: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-and-upgrade-scenarios So I suggest using it as reference. J. On Thu, Nov 9, 2023 at 9:11 PM Lars Winderling <lars.winderl...@posteo.de> wrote: > Hi Ben, > > I think that should work. I would just like to mention to include the list > of extras if installed, i.e. > > pip install --upgrade 'apache-airflow[celery,docker]==2.7.3' -c ' > https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.9.txt > ' > > and in case you upgrade your python version, make sure to create a new > venv first before installing airflow into it, upgrading in-place will not > work. We're also running in vms, and we create a new env for every upgrade, > so we can easily rollback if the pip installation fails e.g. because of > custom dependencies and the like. Sorry if I've stated only obvious points. > > Best, Lars > > > On 9 November 2023 20:37:23 CET, Ben Hancock <bhanc...@alm.com.INVALID> > wrote: > >> On 11/4/2023 4:57 AM, Jarek Potiuk wrote: >> >> If you want to make sure released Airflow installs in a reproducible >>> way from the scratch - now and in the future, the only way to achieve >>> that is described here: >>> >>> >>> https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html >>> >>> It involves using constraints. It only works with `pip`. There are >>> no other ways and other tools that can be achieved easily, so we >>> strongly recommend you use `pip` when installing Airflow. >>> >>> Example installation command for airflow with celery extra and >>> Python version 3.8 is here: >>> >>> pip install "apache-airflow[celery]==2.7.2" --constraint " >>> >>> https://raw.githubusercontent.com/apache/airflow/constraints-2.7.2/constraints-3.8.txt >>> >> >> Thanks for this reminder and tutorial. When upgrading an Airflow >> instance installed with virtualenv, should users interact with >> constraints files the same way? >> >> Here's the sequence I'm imagining, per the upgrade docs[1], if I were >> trying to upgrade to the latest version with Python 3.9 >> >> (venv) $ pip install --upgrade apache-airflow==2.7.3 \ >> >>> --constraint >>> "https://raw.githubusercontent.com/apache/airflow/constraints-2.7.3/constraints-3.9.txt"; >>> >> >> (venv) $ airflow db migrate >> >> Is that the right sequence? Is there anything else that should be kept >> in mind? >> >> - Ben >> >> [1]: >> https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading.html >> ------------------------------ >> To unsubscribe, e-mail: users-unsubscr...@airflow.apache.org >> For additional commands, e-mail: users-h...@airflow.apache.org >> >>
