potiuk edited a comment on issue #4543: [AIRFLOW-3718] [WIP] Multi-layered version of the docker image URL: https://github.com/apache/airflow/pull/4543#issuecomment-470987408 @ashb @fokko - I will continue working on it later today but again some progress info: * I optimised down the number of images/Dockerfiles needed. I found an easy workaround for lack of ARG support in 'COPY --from'. This way we have now 3 images less and 3 dockerfiles less. It started to look manageable and reasonable. * We have now those images (per python version): * Basic Airflow apt dependencies: [**_airflow/latest-2.7-deps-v2.0.0.dev0_**] * Airflow image = Basic Airflow apt dependencies + PIP packages installed + airflow installed [**_airflow/latest-2.7_**, **_airflow/latest-2.7-v2.0.0.dev0_**] - this is what we currently build * CI apt dependencies = Basic Airflow apt dependencies + those needed for CI [**_latest-3.7-ci-deps-v2.0.0.dev0_**] * CI PIPCACHE image = CI apt Dependencies + compiled wheels for all packages [**_latest-2.7-pipcache-v2.0.0.dev0_**] * CI image = CI apt dependencies + /cache from PIPCACHE + PIP packages installed from the cached wheels + airflow installed [**_latest-2.7-ci_**, **_latest-2.7-ci-v2.0.0.dev0_**] This starts looking much easier to explain ( I will draw some DAG with dependencies when it comes to documentation). So this might be the final setup. * I added back NPM - it is now using 'apt ci' instead of 'apt install' (it uses package-lock.json instead of updating it). It is also added now with incremental layering architecture - it will only be run when any of the www sources change rather than any other sources. This will make non-WWW related changes build faster as well - using the NPM layer from cache. Now I continue fixing all the tests.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
