potiuk commented on issue #4938: [AIRFLOW-4117] Multi-staging Image - Travis CI tests [Step 3/3] URL: https://github.com/apache/airflow/pull/4938#issuecomment-511208280 Hey @ashb - as you suggested I unified the scripts into two _util.sh and _in_container_utils.sh which are universally used this is in 54fd4c2 . What's even more - I also converted the last remaining script (checking the licences) to use docker as well (but this time full CI image as it needs Java). so we have fully dockerized CI build now. I also played a bit with the whole environment also on Linux and I think I solved the problem that Stephen reported. When you run tests in container a lot of .pyc files were created using root user on Linux if they were not created before in the host. That caused problems when you try to switch branches and directories are supposed to be deleted (they cannot be because .pyc files are root owned in the host as well). I read a bit and I simply did two things: 1) finding and deleting all .pyc files on entering docker container 2) I disabled generation of .pyc files by setting PYTHONDONTWRITEBYTECODE="true". It also slows down a little importing (but I have not noticed any slow-down - I will see how it impacts tests on Travis with the current build). Those .pyc files are not as efficient now with multi-processor CPUS and relatively slow docker OSXFS I think it might even be faster without those .pyc bytecode files - this is in ec8182d . Last, but not least - I also ported some of my later changes to run-test scripts (following your screencast) - right now it performs database intialization only first time it is run per environment (or when you force it with `-i` flag) and it allows to specify `-s` flag to suppress capturing output by nosetest (with some comment explaining how to do ipdb debugging) and -v to switch to verbose output with colors. This is in 4e0b9b2 I think once it succeeds, it should be more than ready for the prime time.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
