potiuk commented on issue #20168: URL: https://github.com/apache/airflow/issues/20168#issuecomment-993527948
> To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen? The big difference is that when you hace your Dockerfile and resulting image.You build your image in your CI/CD frameworrk and you can run checks for it. You could for example verify airflow version, verify whether all imports are done correctly, verify if basic commands work. All that LONG before you start hitting that in production. This is the best common sense engineering practice. And this is the main reasons why containers, isolation has been introduced for. To make a "frozen" environment that you can securely test and deploy in production. If you are using Helm/K8S/Docker but then you try to use it as if it was "local venv" - you are using the worst of both worlds - volatitlty, lack of guarantees, lack of stability and no opportunity to test things before deploying - all those are coming with containers but you refuse to make use of it, at the same time terribly complicating your setup. It's literallly like using a super-modern automated drilll to hit a nail. Not a good idea. Yes you can do it and it will do the job eventually. But it's quite expensive and dangerous thing to do. > To me it's still a bit of a puzzle how a pip dependency turns the table on the main application and effectively overwrites it. Seems quite wild. If I do the pip install in a dockerfile will the same not happen? I have no idea how the `extraPipPackages` work under the hood (and no desire to learn). We deliberately refused to even implement it when asked by the users in the [official Airflow Helm Chart](https://airflow.apache.org/docs/helm-chart/stable/index.html) - anticipating this kind of problems . I think this is more for @thesuperzapper to debug. PIP has some switches (for example `--eager-upgrade` upgrade strategy, that might cause it, or maybe it is simply a result of using `--upgrade` in the pip install. No idea. No desire to find out. Don't do it. If you on the other hand build your own image and follow the [nice documentation](https://airflow.apache.org/docs/docker-stack/build.html) we have with plenty of examples, where you directly control the `pip` commands used I am quite sure you would not have the same problem. Even if you had - let me repeat it again - you would have a chance to find out and debug before it automatically migrated your database. And this is what I heartily recommend. BTW. Docker image testing is precisely what we do in our CI/CD for airflow. With every single PR in Airflow we verify that the image does what it is supposed to do. Heartily recommend. See here for an inspiration (code available https://github.com/apache/airflow/blob/main/docker_tests/test_prod_image.py): https://github.com/apache/airflow/runs/4519400249?check_suite_focus=true#step:6:310 ``` docker_tests/test_prod_image.py::TestCommands::test_without_command docker_tests/test_prod_image.py::TestCommands::test_airflow_command [gw0] [ 3%] PASSED docker_tests/test_prod_image.py::TestCommands::test_without_command docker_tests/test_prod_image.py::TestCommands::test_airflow_version [gw1] [ 6%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_command docker_tests/test_prod_image.py::TestCommands::test_python_version [gw1] [ 10%] PASSED docker_tests/test_prod_image.py::TestCommands::test_python_version docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed [gw0] [ 13%] PASSED docker_tests/test_prod_image.py::TestCommands::test_airflow_version docker_tests/test_prod_image.py::TestCommands::test_bash_version [gw1] [ 16%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_required_providers_are_installed docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] [gw0] [ 20%] PASSED docker_tests/test_prod_image.py::TestCommands::test_bash_version docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict [gw1] [ 23%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[amazon-import_names0] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] [gw0] [ 26%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_pip_dependencies_conflict docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] [gw1] [ 30%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[async-import_names1] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] [gw0] [ 33%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[cncf.kubernetes-import_names4] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] [gw1] [ 36%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[azure-import_names2] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] [gw0] [ 40%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[dask-import_names5] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] [gw1] [ 43%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[celery-import_names3] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] [gw0] [ 46%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[docker-import_names6] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] [gw0] [ 50%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[hashicorp-import_names10] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] [gw1] [ 53%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[elasticsearch-import_names7] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] [gw0] [ 56%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[ldap-import_names11] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] [gw1] [ 60%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[google-import_names8] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] [gw0] [ 63%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[mysql-import_names12] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] [gw1] [ 66%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[grpc-import_names9] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] [gw0] [ 70%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[postgres-import_names13] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] [gw1] [ 73%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[pyodbc-import_names14] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] [gw0] [ 76%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[redis-import_names15] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] [gw1] [ 80%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sendgrid-import_names16] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] [gw0] [ 83%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[sftp/ssh-import_names17] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] [gw1] [ 86%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[slack-import_names18] docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] [gw0] [ 90%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[statsd-import_names19] docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root [gw1] [ 93%] PASSED docker_tests/test_prod_image.py::TestPythonPackages::test_check_dependencies_imports[virtualenv-import_names20] docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root [gw1] [ 96%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_run_custom_python_packages_as_root [gw0] [100%] PASSED docker_tests/test_prod_image.py::TestExecuteAsRoot::test_execute_airflow_as_root ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
