potiuk edited a comment on issue #18794: URL: https://github.com/apache/airflow/issues/18794#issuecomment-937786632
This is all expected. Actually your Python 3.9 installation likely fails as well, but a bit differently (and you did not notice it) - it simply refused to install anything because the `hive` provider cannot be found for python 3.9. The Python Hive Provider (at the time of release) did not work with Python 3.9 See dependencies in: https://airflow.apache.org/docs/apache-airflow-providers-apache-hive/stable/index.html and it's not installable for Python 3.9. As PyHive did not install for Python 3.9 so we excluded it from being available for Python 3.9. The reason why your packages cannot be installed is that they require "build-essential" which is not added into the production image because it is optimized for size - the Docker image is optimized for size, multi-segmented and we only use build essential in the first segment to save several hundreds of MB. This is all nicely explained in https://airflow.apache.org/docs/docker-stack/build.html You can also watch my talk about it here: from last year's Airflow Summit https://www.youtube.com/watch?v=wDr3Y7q2XoI You have two options if you want to add packages requiring compilation: 1) Extend the image adding build essentials as expleined here: https://airflow.apache.org/docs/docker-stack/build.html#example-when-you-add-packages-requiring-compilation - but this will make your image much bigger 2) Custom-build your image from the scratch: https://airflow.apache.org/docs/docker-stack/build.html#customizing-the-image - this will help you to build highly optimized image, but it requires a bit more elaborated process and using Airflow sources for that. You can see the calculations of how much you can safe by going the custom route. It's your choice whether you want to pay the price of having bigger images vs. simpler build process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
