potiuk edited a comment on issue #4543: [AIRFLOW-3718] [WIP] Multi-layered version of the docker image URL: https://github.com/apache/airflow/pull/4543#issuecomment-471218351 No problem. I was also thinking about it. There are two possible aproaches I thought about: 1) Keep what I proposed - right now the requirements will be "frozen" implicitely for CI builds. We can have a separate "clean" build where everything will install from scratch. We have now full support for it and we coudl setup a separate Travis build just doing it (not running tests, just running build afresh from a master branch notifying everyone. That would be like a global ("the requirements we have now in master are unstable and we need to fix them really soon" warning). I like this idea as it would not impact the people who just commit few line fixes but then maintainers could have enough lead time to fix it. Hovever it is not in line with the "explicit is better than implicit" Zen of python. 2) Another option is something akin to NPM .lock file. We could run something like `pip freeze > ci-requirements.txt` and save it in the repo (equivalent to npm's .lock file). This could be run at every 'pip install -e .' (custom install class in setup.py) - but then we would have to freeze those requirements with "ci_devel" extras. I think this kind of worklfow as described in 2) was one of the main motivations of why in NPM they changed 'npm install' behaviour (and somehow explains why 'npm ci' is hard to discover/not obvious). With 'npm install' anyone running it will now get the ".lock" file updated and will be encouraged to commit it together with package.json change. If we write custom setup.py Install method, that could actually work the same way - ci-requirements.txt will get updated and we can use those requirements rather than setup.py to install requirements for CI builds (with pip install -r). With the multi-layered Dockerfile I have now it should be quite easy to do this conditionally only for CI builds . I start to like the 2)nd approach more and more as I think about it :). It's explicit and might actually work. What do you think @ashb ?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
