[
https://issues.apache.org/jira/browse/BEAM-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16544635#comment-16544635
]
Ahmet Altay commented on BEAM-4749:
-----------------------------------
This was happening to people doing builds/lints locally. It is breaking the
build. It is also breaking the installation of a pre-built SDK in MacOS. There
are no links because it was all local runs. I was able to reproduce on my
laptop as well. I do not know if I ran the xcode command before, I can try to
run again with that.
Is it possible to gracefully fallback to avro in client environments when
fastavro is not installed. As long as we can run with fastavro in the
distributed environment (in linux containers) it will capture primary
performance improvement.
The other options is that, we can ask fastavro project to provide working
pre-built files for macos.
(I do not know about the status of Windows. I suspect it might also have
issues. I am most worried about end users failing to install pre-build SDKs)
> fastavro breaks macos tests
> ---------------------------
>
> Key: BEAM-4749
> URL: https://issues.apache.org/jira/browse/BEAM-4749
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Ahmet Altay
> Assignee: Ryan Williams
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Recent addition of the fastavro dependency is breaking python linter in
> macos. At least for some cases, because it requires a compiler.
> Could we optionally depend on fastavro, and fallback to regular avro package?
>
> Log:
> *> Task :beam-sdks-python:lintPy27*
> ERROR: invocation failed (exit code 1), logfile:
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
> ERROR: actionid: py27-lint
> msg: installpkg
> cmdargs:
> ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python',
> '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip',
> 'install',
> '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
>
> Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
> Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
> Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
> Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/a7/0a/b08ba5cef63c675e8442c2bf1cbcef90c8b9f824be2202d492f0cedb0913/fastavro-0.19.7.tar.gz]
> Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/66/89/4a90caabd51c17686cbb48a9bbe8c592c4be929c0d2542d2ffde76b0d671/grpcio-1.13.0-cp27-cp27m-macosx_10_12_x86_64.whl]
> Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
> Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
> Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
> Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl]
> Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/4f/56/a21f2d077ceae7fd521c0ed31fb8bb1c7f13ffbb09bf7dd27de6cf6bad08/protobuf-3.6.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
> Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl]
> Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz]
> Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
> Requirement already satisfied: six<1.12,>=1.9 in
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from
> apache-beam==2.6.0.dev0) (1.11.0)
> Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl]
> Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl]
> Requirement already satisfied: future<1.0.0,>=0.16.0 in
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from
> apache-beam==2.6.0.dev0) (0.16.0)
> Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl]
> Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl]
> Requirement already satisfied: setuptools>=18.0 in
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from
> fastavro==0.19.7->apache-beam==2.6.0.dev0) (39.2.0)
> Requirement already satisfied: enum34>=1.0.4 in
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from
> grpcio<2,>=1.8->apache-beam==2.6.0.dev0) (1.1.6)
> Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Collecting requests>=2.7.0 (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl]
> Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl]
> Collecting funcsigs>=1; python_version < "3.3" (from
> mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl]
> Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl]
> Collecting pyasn1>=0.1.7 (from
> oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/a0/70/2c27740f08e477499ce19eefe05dbcae6f19fdc49e9e82ce4768be0643b9/pyasn1-0.4.3-py2.py3-none-any.whl]
> Collecting pyasn1-modules>=0.0.5 (from
> oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/19/02/fa63f7ba30a0d7b925ca29d034510fc1ffde53264b71b4155022ddf3ab5d/pyasn1_modules-0.2.2-py2.py3-none-any.whl]
> Collecting chardet<3.1.0,>=3.0.2 (from
> requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl]
> Collecting idna<2.8,>=2.5 (from
> requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl]
> Collecting urllib3<1.24,>=1.21.1 (from
> requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl]
> Collecting certifi>=2017.4.17 (from
> requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl]
> Building wheels for collected packages: apache-beam, fastavro, pyyaml
> Running setup.py bdist_wheel for apache-beam: started
> Running setup.py bdist_wheel for apache-beam: finished with status 'done'
> Stored in directory:
> /Users/relax/Library/Caches/pip/wheels/a3/18/08/60ce57e27e8b04460419c9463b9b6da19a8aab9fb64afc612f
> Running setup.py bdist_wheel for fastavro: started
> Running setup.py bdist_wheel for fastavro: finished with status 'error'
> Complete output from command
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u
> -c "import setuptools,
> tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d
> /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-wheel-p6UGao
> --python-tag cp27:
> running bdist_wheel
> running build
> running build_py
> creating build
> creating build/lib.macosx-10.4-x86_64-2.7
> creating build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validate_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validation_py.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> running build_ext
> building 'fastavro._read' extension
> creating build/temp.macosx-10.4-x86_64-2.7
> creating build/temp.macosx-10.4-x86_64-2.7/fastavro
> gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall
> -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7
> -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
> xcrun: error: invalid active developer path
> (/Library/Developer/CommandLineTools), missing xcrun at:
> /Library/Developer/CommandLineTools/usr/bin/xcrun
> error: command 'gcc' failed with exit status 1
>
> ----------------------------------------
> Failed building wheel for fastavro
> Running setup.py clean for fastavro
> Running setup.py bdist_wheel for pyyaml: started
> Running setup.py bdist_wheel for pyyaml: finished with status 'done'
> Stored in directory:
> /Users/relax/Library/Caches/pip/wheels/03/05/65/bdc14f2c6e09e82ae3e0f13d021e1b6b2481437ea2f207df3f
> Successfully built apache-beam pyyaml
> Failed to build fastavro
> Installing collected packages: avro, crcmod, dill, fastavro, futures, grpcio,
> docopt, chardet, idna, urllib3, certifi, requests, hdfs, httplib2, pbr,
> funcsigs, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pytz,
> pyyaml, pyvcf, typing, nose, pyhamcrest, apache-beam
> Running setup.py install for fastavro: started
> Running setup.py install for fastavro: finished with status 'error'
> Complete output from command
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u
> -c "import setuptools,
> tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record
> /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt
> --single-version-externally-managed --compile --install-headers
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro:
> running install
> running build
> running build_py
> creating build
> creating build/lib.macosx-10.4-x86_64-2.7
> creating build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_py.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validate_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validation_py.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/validation.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> running build_ext
> building 'fastavro._read' extension
> creating build/temp.macosx-10.4-x86_64-2.7
> creating build/temp.macosx-10.4-x86_64-2.7/fastavro
> gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall
> -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7
> -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
> xcrun: error: invalid active developer path
> (/Library/Developer/CommandLineTools), missing xcrun at:
> /Library/Developer/CommandLineTools/usr/bin/xcrun
> error: command 'gcc' failed with exit status 1
>
> ----------------------------------------
> Command
> "/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python
> -u -c "import setuptools,
> tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record
> /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt
> --single-version-externally-managed --compile --install-headers
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro"
> failed with error code 1 in
> /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/
>
> py27-lint installed:
> astroid==1.6.5,avro==1.8.2,backports.functools-lru-cache==1.5,configparser==3.5.0,crcmod==1.7,dill==0.2.6,enum34==1.1.6,flake8==3.5.0,future==0.16.0,isort==4.2.15,lazy-object-proxy==1.3.1,mccabe==0.6.1,pycodestyle==2.3.1,pyflakes==1.6.0,pylint==1.7.2,singledispatch==3.4.0.3,six==1.11.0,wrapt==1.10.11
> ___________________________________ summary
> ____________________________________
> ERROR: py27-lint: InvocationError for command
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip
> install
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]
> (see
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log)
> (exited with code 1)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
