potiuk edited a comment on issue #4543: [AIRFLOW-3718] Multi-layered version of the docker image URL: https://github.com/apache/airflow/pull/4543#issuecomment-455999297 @ashb, @Fokko - I am more than happy to help with solving the dependency issues as well. I hope to have more time to work on it soon (than when I had couple of months back). I just am not sure that this is the same issue but this one can help with the dependency one. I think if we finally agree that layered approach is workable, it might open up some new ways of dealing with dependencies and solving dependencies issue might actually get solved more easily. And likely - we won't have to do much if we move to layered approach for CI images as well. What we get with the layered solution is a kind of "temporary stability" of image. Unless setup.py or base image changes, the dependencies in base image will remain pretty stable. We could use very similar approach in CI Docker images: move it to the airflow repository, and skip "pip install' after every commit. I think that might solve most of concerns I had about stability of CI builds (and why I originated the thread). Most of the builds in CI will be "stable" - they will use cached dependencies from the last successfull build of the CI image. It's only when someone upgrades setup.py where the transient dependencies might cause installation problems (and it's the right moment to solve them). At the same time we will always get early warnings about dependency problems with the "latest-clean" image that will be run with every commit - and this will happen without disturbing the work of people running their builds using the latest working CI image. The good thing will be that the tests will continue to run for the users that will make simple commits - using the cached dependencies even if "latest-clean" image will fail with transient dependencies problems. Until the "clean" build is fixed test will continue to run using the latest cached dependencies. It's great for the regular committers (because their work won't be disturbed by external factors they can do nothing about). On the other hands, solving dependency issues will be at the hands of those who already deal with setup.py changes in their own changes. This looks like frictionless, self-healing setup that might provide both stability for committers and "eventual consistency" for maintainers and we will not have to change the approach where we have "upper-open" requirements in airflow. Last but not least (but this is quite independent) - we could think about freezing the requirements in the scripts that you use to prepare the release. But I think this is quite a different task.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
