potiuk commented on issue #4543: [AIRFLOW-3718] [WIP] Multi-layered version of the docker image URL: https://github.com/apache/airflow/pull/4543#issuecomment-473732065 @fokko @ash - one more comment - I am now ready to involve community. I actually managed to hugely simplify the layer structure. I did a lot of testing - including all the timing calculation - how much we save at each stage, whether the layers are needed, whether the wheel cache layer is really helping. All this is documented now in https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-10+Multi-layered+and+multi-stage+official+Airflow+image From those tests it seems that wheel cache did more harm than good. It complicated the whole setup and increased number of images we had to build and eventually only provided marginal savings. It took much longer to build the images in general (bigger Dockerfile, more layers etc.). Also when I disabled CYTHON compilation for Cassandra driver, pip installation time went down by 5(!) minutes or so and then it started to be comparable with the installation from wheels. So my most recent proposal is even more simplified - we now have only three stages in the Dockerfile and two images in DockerHub (just the slim Airflow Image and fat Airflow CI image). And it's also a bit easier to explain and understand. I will explain it in the community devlist email.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
