Hey Stefan, All the important problems should be fixed now! You can pull the latest changes from my branch and retry:
- The warning with /root/.docker/config.json is nothing to worry about (it tries to access your host configuration for credentials - but we do not need any credentials for the environment). - The CLEAN_FILES unbound variable was an interesting one - not sure why it was not set at the first run and set it on the other (likely because CLEAN_FILES is propagated through docker-compose settings and set to empty after initialisation) but I protected against such case now - Running flake/pylint is not supposed to be run from the container but from the host. I looked at the docs and realised it was not at all clear :). It was obvious for me but not for someone new :). That's big value of such tests. I updated CONTRIBUTING.md with a note and also added extra protection. When you try those scripts from container you will get the following error now: *You are inside the Airflow docker container!You should only run this script from the host.Learn more about how we develop and test airflow in:* *https://github.com/apache/airflow/blob/master/CONTRIBUTING.md <https://github.com/apache/airflow/blob/master/CONTRIBUTING.md> * Please try to run them from the host (and after pull you can try from the container and you will see the error). It will run a bit slower now because it checks whether docker should be rebuilt but soon I will optimise it away with pre-commit hooks (I already have a POC working). Anyone else wants to try - feel free and let me know! J. On Tue, Jul 9, 2019 at 7:03 PM Jarek Potiuk <[email protected]> wrote: > Thanks Stefan! - will look into all those and fix :). Those are some of > the small things that should be super easy to fix. > > The root user for Linux is is one of the differences Mac <> Linux that is > super difficult to resolve (in current Docker) when you mount local > volumes. The problem is that inside the container you will have the same > uid/gid as your host user but you will not have that user inside the > container (and then you will have no permissions to run airflow inside the > container). Unfortunately mapping the users host <-> container is not easy > (requires engine restart and cannot be done per-container). And we cannot > assume fixed UID (different host users have different ids). > > Here is detailed explanation from our PR: > https://github.com/apache/airflow/pull/4938#issuecomment-507790924 > > This problem does not occur on MacOS because on Mac there is a (slower) > osxfs filesystem used that maps users from host to linux VM -> and then to > container. I am going to workaround it very soon for Linux as well.. In the > follow-up "simplified development environment" AIP-7 those root-owned files > will be cleaned up on entering the environment and you will have a simple > command to clean them up. Maybe not a perfect solution but it is better > than manual cleanup with sudo. > > J. > > > On Tue, Jul 9, 2019 at 6:38 PM Stefan Seelmann <[email protected]> > wrote: > >> Hi Jarek, >> >> tested quickly on ArchLinux. Overall looks very good, thanks for all the >> effort! >> >> >> When building the images for the first time I saw the following warning: >> ``` >> WARNING: Error loading config file: /root/.docker/config.json: stat >> /root/.docker/config.json: permission denied >> ``` >> >> >> When running `./scripts/ci/local_ci_enter_environment.sh` the first time >> it fails: >> ``` >> $ PYTHON_VERSION=3.6 BACKEND=postgres ENV=docker >> ./scripts/ci/local_ci_enter_environment.sh >> /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh: line 57: >> CLEAN_FILES: unbound variable >> ``` >> I added `CLEAN_FILES=false` to the command then it worked. Next time the >> additional env variable was not required. >> >> >> Within the dev env I was able to run test using `run-tests`. >> >> >> However running flake8 or pylint fails, seems the `hooks` folder is not >> mounted into the docker container: >> ``` >> /opt/airflow /opt/airflow >> scripts/ci/ci_flake8.sh: line 32: ./hooks/build: No such file or directory >> ``` >> >> >> Another observation: Files and folders created by the environment have >> user/group `root` (examples are `logs` and `__pycache__`). I assume the >> reason is that the container runs as user root. Maybe it's possible to >> run the container with uid/gid of the curren user using `-u $(id >> -u):$(id -g)`? >> >> >> Kind Regards, >> Stefan >> >> >> > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
