Now I'm seeing build errors while running tests, on my fork and Apache repo:
airflow@eab495ad15f0:/app$ tox -e py35-backend_postgres-env_docker PYTHON_VERSION=3 py35-backend_postgres-env_docker create: /app/.tox/py35-backend_postgres-env_docker py35-backend_postgres-env_docker installdeps: wheel, codecov py35-backend_postgres-env_docker installed: apache-airflow===2.0.0.dev0-,certifi==2018.11.29,chardet==3.0.4,codecov==2.0.15,coverage==4.5.2,idna==2.8,requests==2.21.0,urllib3==1.24.1 py35-backend_postgres-env_docker run-test-pre: PYTHONHASHSEED='3899480037' py35-backend_postgres-env_docker runtests: commands[0] | pip wheel --no-use-pep517 --progress-bar off -w /home/airflow/.wheelhouse -f /home/airflow/.wheelhouse -e '.[devel_ci]' Looking in links: /home/airflow/.wheelhouse Obtaining file:///app Collecting alembic<1.0,>=0.9 (from apache-airflow===2.0.0.dev0-) File was already downloaded /home/airflow/.wheelhouse/alembic-0.9.10-py2.py3-none-any.whl Collecting bleach~=2.1.3 (from apache-airflow===2.0.0.dev0-) File was already downloaded /home/airflow/.wheelhouse/bleach-2.1.4-py2.py3-none-any.whl Collecting configparser<3.6.0,>=3.5.0 (from apache-airflow===2.0.0.dev0-) File was already downloaded /home/airflow/.wheelhouse/configparser-3.5.3-py3-none-any.whl Collecting croniter<0.4,>=0.3.17 (from apache-airflow===2.0.0.dev0-) LOG ABBREVIATED... Skipping secretstorage, due to already being wheel. Skipping jeepney, due to already being wheel. Building wheels for collected packages: apache-airflow Building wheel for apache-airflow (setup.py) ... done Running setup.py clean for apache-airflow Failed to build apache-airflow ERROR: Failed to build one or more wheels ERROR: InvocationError for command '/app/.tox/py35-backend_postgres-env_docker/bin/pip wheel --no-use-pep517 --progress-bar off -w /home/airflow/.wheelhouse -f /home/airflow/.wheelhouse -e .[devel_ci]' (exited with code 1) _______________________________________________ summary ________________________________________________ ERROR: py35-backend_postgres-env_docker: commands failed airflow@eab495ad15f0:/app$ On Sun, Feb 3, 2019 at 11:31 AM David Cavaletto <[email protected]> wrote: > Appears to have been bad permissions on the files on my local checkout of > my repo. I re-cloned it and its working now. I did need to use --user flag > to get the pip install to succeed. Would you like me to submit a PR to > change the docs? > > > > On Sun, Feb 3, 2019 at 11:26 AM David Cavaletto <[email protected]> > wrote: > >> Accidentally replied directly to Ash, not to dev mailing list. >> >> Same issue. >> >> It does install with sudo, but I know you're not supposed to use sudo >> with pip. >> That seemed to let me run the test command, but I had to run it with sudo >> too. >> >> So one interesting note, it works on the Apache Airflow repo I have >> cloned, but not on my branch (which also on master and up to date with >> Apache Airflow repo). I'll dig in to see if there is something different. >> >> >> >> On Sun, Feb 3, 2019 at 11:06 AM Ash Berlin-Taylor <[email protected]> wrote: >> >>> Try `pip install --user -e .` >>> >>> Instructions may need updating? >>> >>> On 3 February 2019 16:58:39 GMT, David Cavaletto <[email protected]> >>> wrote: >>>> >>>> My id on both my host and docker image is 1000. >>>> >>>> On Sat, Feb 2, 2019 at 4:33 PM Stefan Seelmann <[email protected]> >>>> wrote: >>>> >>>> What is your user ID (id -u)? I assume that happens when your user ID on >>>>> the host is not 1000, then the mounted /app volume is not writeable by >>>>> the airflow user within the container which UID is 1000. >>>>> >>>>> On 2/2/19 10:04 PM, David Cavaletto wrote: >>>>> >>>>>> I'm following the instructions for testing using docker-compose. After >>>>>> running `docker-compose -f scripts/ci/docker-compose.yml run >>>>>> airflow-testing bash` I land on the container as expected. >>>>>> >>>>>> But then running `pip install -e .[devel]` fails with a permission >>>>>> error. >>>>>> Upgrading pip also fails with a permission error. >>>>>> >>>>>> Am I missing something? Shouldn't this work out of the box? Any >>>>>> >>>>> suggestions? >>>>> >>>>>> >>>>>> >>>>>> Below is the complete output. >>>>>> >>>>>> ➜ caddac.airflow git:(master) docker-compose -f >>>>>> scripts/ci/docker-compose.yml run airflow-testing bash >>>>>> Starting ci_postgres_1 ... done >>>>>> Starting ci_mongo_1 ... >>>>>> Starting ci_rabbitmq_1 ... done >>>>>> Starting ci_cassandra_1 ... done >>>>>> Starting ci_mysql_1 ... done >>>>>> Starting ci_krb5-kdc-server_1 ... done >>>>>> Starting ci_mongo_1 ... done >>>>>> airflow@415da8f8db0e:/app$ pip install -e .[devel] >>>>>> Obtaining file:///app >>>>>> Complete output from command python setup.py egg_info: >>>>>> gitpython not found: Cannot compute the git version. >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> >>>>>> "/usr/local/lib/python3.5/dist-packages/setuptools/command/easy_install.py", >>>>> >>>>>> line 466, in check_site_dir >>>>>> open(testfile, 'w').close() >>>>>> PermissionError: [Errno 13] Permission denied: >>>>>> './.eggs/test-easy-install-24.write-test' >>>>>> >>>>>> During handling of the above exception, another exception occurred: >>>>>> >>>>>> Traceback (most recent call last): >>>>>> File "<string>", line 1, in <module> >>>>>> File "/app/setup.py", line 417, in <module> >>>>>> do_setup() >>>>>> File "/app/setup.py", line 412, in do_setup >>>>>> python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*', >>>>>> File >>>>>> >>>>> "/usr/local/lib/python3.5/dist-packages/setuptools/__init__.py", >>>>> >>>>>> line 139, in setup >>>>>> _install_setup_requires(attrs) >>>>>> File >>>>>> >>>>> "/usr/local/lib/python3.5/dist-packages/setuptools/__init__.py", >>>>> >>>>>> line 134, in _install_setup_requires >>>>>> dist.fetch_build_eggs(dist.setup_requires) >>>>>> File "/usr/local/lib/python3.5/dist-packages/setuptools/dist.py", >>>>>> line 514, in fetch_build_eggs >>>>>> replace_conflicting=True, >>>>>> File >>>>>> "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line >>>>>> 777, in resolve >>>>>> replace_conflicting=replace_conflicting >>>>>> File >>>>>> "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line >>>>>> 1060, in best_match >>>>>> return self.obtain(req, installer) >>>>>> File >>>>>> "/usr/local/lib/python3.5/dist-packages/pkg_resources/__init__.py", line >>>>>> 1072, in obtain >>>>>> return installer(requirement) >>>>>> File "/usr/local/lib/python3.5/dist-packages/setuptools/dist.py", >>>>>> line 580, in fetch_build_egg >>>>>> cmd.ensure_finalized() >>>>>> File "/usr/lib/python3.5/distutils/cmd.py", line 107, in >>>>>> ensure_finalized >>>>>> self.finalize_options() >>>>>> File >>>>>> >>>>>> "/usr/local/lib/python3.5/dist-packages/setuptools/command/easy_install.py", >>>>> >>>>>> line 324, in finalize_options >>>>>> self.check_site_dir() >>>>>> File >>>>>> >>>>>> "/usr/local/lib/python3.5/dist-packages/setuptools/command/easy_install.py", >>>>> >>>>>> line 469, in check_site_dir >>>>>> self.cant_write_to_target() >>>>>> File >>>>>> >>>>>> "/usr/local/lib/python3.5/dist-packages/setuptools/command/easy_install.py", >>>>> >>>>>> line 532, in cant_write_to_target >>>>>> raise DistutilsError(msg) >>>>>> distutils.errors.DistutilsError: can't create or remove files in >>>>>> install directory >>>>>> >>>>>> The following error occurred while trying to add or remove files in >>>>>> >>>>> the >>>>> >>>>>> installation directory: >>>>>> >>>>>> [Errno 13] Permission denied: >>>>>> './.eggs/test-easy-install-24.write-test' >>>>>> >>>>>> The installation directory you specified (via --install-dir, >>>>>> >>>>> --prefix, >>>>> >>>>>> or >>>>>> the distutils default setting) was: >>>>>> >>>>>> ./.eggs >>>>>> >>>>>> Perhaps your account does not have write access to this directory? >>>>>> >>>>> If >>>>> >>>>>> the >>>>>> installation directory is a system-owned directory, you may need to >>>>>> sign in >>>>>> as the administrator or "root" account. If you do not have >>>>>> administrative >>>>>> access to this machine, you may wish to choose a different >>>>>> >>>>> installation >>>>> >>>>>> directory, preferably one that is listed in your PYTHONPATH >>>>>> >>>>> environment >>>>> >>>>>> variable. >>>>>> >>>>>> For information on other options, you may wish to consult the >>>>>> documentation at: >>>>>> >>>>>> https://setuptools.readthedocs.io/en/latest/easy_install.html >>>>>> >>>>>> Please make the appropriate changes for your system and try again. >>>>>> ------------------------------ >>>>>> Command "python setup.py egg_info" failed with error code 1 in /app/ >>>>>> You are using pip version 18.0, however version 19.0.1 is available. >>>>>> You should consider upgrading via the 'pip install --upgrade pip' >>>>>> >>>>> command. >>>>> >>>>>> airflow@415da8f8db0e:/app$ pip install --upgrade pip >>>>>> Collecting pip >>>>>> Using cached >>>>>> >>>>>> https://files.pythonhosted.org/packages/46/dc/7fd5df840efb3e56c8b4f768793a237ec4ee59891959d6a215d63f727023/pip-19.0.1-py2.py3-none-any.whl >>>>> >>>>>> Installing collected packages: pip >>>>>> Found existing installation: pip 18.0 >>>>>> Uninstalling pip-18.0: >>>>>> Could not install packages due to an EnvironmentError: [Errno 13] >>>>>> Permission denied: '/usr/bin/pip' >>>>>> Consider using the `--user` option or check the permissions. >>>>>> >>>>>> You are using pip version 18.0, however version 19.0.1 is available. >>>>>> You should consider upgrading via the 'pip install --upgrade pip' >>>>>> >>>>> command. >>>>> >>>>>> airflow@415da8f8db0e:/app$ >>>>>> >>>>>> >>>>> >>>>>
