[
https://issues.apache.org/jira/browse/BEAM-4749?focusedWorklogId=122440&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-122440
]
ASF GitHub Bot logged work on BEAM-4749:
----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Jul/18 17:24
Start Date: 12/Jul/18 17:24
Worklog Time Spent: 10m
Work Description: aaltay opened a new pull request #5939: [BEAM-4749]
Install fastavro only in linux
URL: https://github.com/apache/beam/pull/5939
Installing fastavro breaks in MacOS at compilation time. This in turn
prevents SDK from being installed. Until we have tests and know that fastavro
installs cleanly in other OSes I will limit it to linux only.
R: @udim
cc: @ryan-williams
------------------------
Follow this checklist to help us incorporate your contribution quickly and
easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA
issue, if applicable. This will automatically link the pull request to the
issue.
- [ ] If this contribution is large, please file an Apache [Individual
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone
(e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/)
| --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/)
| [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)
| --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)
</br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/)
| --- | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 122440)
Time Spent: 10m
Remaining Estimate: 0h
> fastavro breaks macos tests
> ---------------------------
>
> Key: BEAM-4749
> URL: https://issues.apache.org/jira/browse/BEAM-4749
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
> Reporter: Ahmet Altay
> Assignee: Chamikara Jayalath
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Recent addition of the fastavro dependency is breaking python linter in
> macos. At least for some cases, because it requires a compiler.
> Could we optionally depend on fastavro, and fallback to regular avro package?
>
> Log:
> *> Task :beam-sdks-python:lintPy27*
> ERROR: invocation failed (exit code 1), logfile:
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log
> ERROR: actionid: py27-lint
> msg: installpkg
> cmdargs:
> ['/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python',
> '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip',
> 'install',
> '/Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]']
>
> Processing ./target/.tox/dist/apache-beam-2.6.0.dev0.zip
> Collecting avro<2.0.0,>=1.8.1 (from apache-beam==2.6.0.dev0)
> Collecting crcmod<2.0,>=1.7 (from apache-beam==2.6.0.dev0)
> Collecting dill==0.2.6 (from apache-beam==2.6.0.dev0)
> Collecting fastavro==0.19.7 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/a7/0a/b08ba5cef63c675e8442c2bf1cbcef90c8b9f824be2202d492f0cedb0913/fastavro-0.19.7.tar.gz]
> Collecting grpcio<2,>=1.8 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/66/89/4a90caabd51c17686cbb48a9bbe8c592c4be929c0d2542d2ffde76b0d671/grpcio-1.13.0-cp27-cp27m-macosx_10_12_x86_64.whl]
> Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.6.0.dev0)
> Collecting httplib2<=0.11.3,>=0.8 (from apache-beam==2.6.0.dev0)
> Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/e6/35/f187bdf23be87092bd0f1200d43d23076cee4d0dec109f195173fd3ebc79/mock-2.0.0-py2.py3-none-any.whl]
> Collecting oauth2client<5,>=2.0.1 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/82/d8/3eab58811282ac7271a081ba5c0d4b875ce786ca68ce43e2a62ade32e9a8/oauth2client-4.1.2-py2.py3-none-any.whl]
> Collecting protobuf<4,>=3.5.0.post1 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/4f/56/a21f2d077ceae7fd521c0ed31fb8bb1c7f13ffbb09bf7dd27de6cf6bad08/protobuf-3.6.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl]
> Collecting pytz<=2018.4,>=2018.3 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/dc/83/15f7833b70d3e067ca91467ca245bae0f6fe56ddc7451aa0dc5606b120f2/pytz-2018.4-py2.py3-none-any.whl]
> Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/4a/85/db5a2df477072b2902b0eb892feb37d88ac635d36245a72a6a69b23b383a/PyYAML-3.12.tar.gz]
> Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.6.0.dev0)
> Requirement already satisfied: six<1.12,>=1.9 in
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from
> apache-beam==2.6.0.dev0) (1.11.0)
> Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/0d/4d/4e5985d075d241d686a1663fa1f88b61d544658d08c1375c7c6aac32afc3/typing-3.6.4-py2-none-any.whl]
> Collecting futures<4.0.0,>=3.1.1 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl]
> Requirement already satisfied: future<1.0.0,>=0.16.0 in
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from
> apache-beam==2.6.0.dev0) (0.16.0)
> Collecting nose>=1.3.7 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl]
> Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl]
> Requirement already satisfied: setuptools>=18.0 in
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from
> fastavro==0.19.7->apache-beam==2.6.0.dev0) (39.2.0)
> Requirement already satisfied: enum34>=1.0.4 in
> ./target/.tox/py27-lint/lib/python2.7/site-packages (from
> grpcio<2,>=1.8->apache-beam==2.6.0.dev0) (1.1.6)
> Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Collecting requests>=2.7.0 (from hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl]
> Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/b3/5d/c196041ffdf3e34ba206db6d61d1f893a75e1f3435699ade9bd65e089a3d/pbr-4.0.4-py2.py3-none-any.whl]
> Collecting funcsigs>=1; python_version < "3.3" (from
> mock<3.0.0,>=1.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/69/cb/f5be453359271714c01b9bd06126eaf2e368f1fddfff30818754b5ac2328/funcsigs-1.0.2-py2.py3-none-any.whl]
> Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl]
> Collecting pyasn1>=0.1.7 (from
> oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/a0/70/2c27740f08e477499ce19eefe05dbcae6f19fdc49e9e82ce4768be0643b9/pyasn1-0.4.3-py2.py3-none-any.whl]
> Collecting pyasn1-modules>=0.0.5 (from
> oauth2client<5,>=2.0.1->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/19/02/fa63f7ba30a0d7b925ca29d034510fc1ffde53264b71b4155022ddf3ab5d/pyasn1_modules-0.2.2-py2.py3-none-any.whl]
> Collecting chardet<3.1.0,>=3.0.2 (from
> requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl]
> Collecting idna<2.8,>=2.5 (from
> requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl]
> Collecting urllib3<1.24,>=1.21.1 (from
> requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl]
> Collecting certifi>=2017.4.17 (from
> requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.6.0.dev0)
> Using cached
> [https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl]
> Building wheels for collected packages: apache-beam, fastavro, pyyaml
> Running setup.py bdist_wheel for apache-beam: started
> Running setup.py bdist_wheel for apache-beam: finished with status 'done'
> Stored in directory:
> /Users/relax/Library/Caches/pip/wheels/a3/18/08/60ce57e27e8b04460419c9463b9b6da19a8aab9fb64afc612f
> Running setup.py bdist_wheel for fastavro: started
> Running setup.py bdist_wheel for fastavro: finished with status 'error'
> Complete output from command
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u
> -c "import setuptools,
> tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d
> /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-wheel-p6UGao
> --python-tag cp27:
> running bdist_wheel
> running build
> running build_py
> creating build
> creating build/lib.macosx-10.4-x86_64-2.7
> creating build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validate_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validation_py.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/validation.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> running build_ext
> building 'fastavro._read' extension
> creating build/temp.macosx-10.4-x86_64-2.7
> creating build/temp.macosx-10.4-x86_64-2.7/fastavro
> gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall
> -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7
> -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
> xcrun: error: invalid active developer path
> (/Library/Developer/CommandLineTools), missing xcrun at:
> /Library/Developer/CommandLineTools/usr/bin/xcrun
> error: command 'gcc' failed with exit status 1
>
> ----------------------------------------
> Failed building wheel for fastavro
> Running setup.py clean for fastavro
> Running setup.py bdist_wheel for pyyaml: started
> Running setup.py bdist_wheel for pyyaml: finished with status 'done'
> Stored in directory:
> /Users/relax/Library/Caches/pip/wheels/03/05/65/bdc14f2c6e09e82ae3e0f13d021e1b6b2481437ea2f207df3f
> Successfully built apache-beam pyyaml
> Failed to build fastavro
> Installing collected packages: avro, crcmod, dill, fastavro, futures, grpcio,
> docopt, chardet, idna, urllib3, certifi, requests, hdfs, httplib2, pbr,
> funcsigs, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pytz,
> pyyaml, pyvcf, typing, nose, pyhamcrest, apache-beam
> Running setup.py install for fastavro: started
> Running setup.py install for fastavro: finished with status 'error'
> Complete output from command
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python -u
> -c "import setuptools,
> tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record
> /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt
> --single-version-externally-managed --compile --install-headers
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro:
> running install
> running build
> running build_py
> creating build
> creating build/lib.macosx-10.4-x86_64-2.7
> creating build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_schema_py.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/write.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_write_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__init__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_py.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_read_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validate_common.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_validation_py.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/six.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/_timezone.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/__main__.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/const.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/schema.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/read.py -> build/lib.macosx-10.4-x86_64-2.7/fastavro
> copying fastavro/validation.py ->
> build/lib.macosx-10.4-x86_64-2.7/fastavro
> running build_ext
> building 'fastavro._read' extension
> creating build/temp.macosx-10.4-x86_64-2.7
> creating build/temp.macosx-10.4-x86_64-2.7/fastavro
> gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -Os -Wall
> -Wstrict-prototypes -I/Library/GoogleCorpSupport/Python/2.7/include/python2.7
> -c fastavro/_read.c -o build/temp.macosx-10.4-x86_64-2.7/fastavro/_read.o
> xcrun: error: invalid active developer path
> (/Library/Developer/CommandLineTools), missing xcrun at:
> /Library/Developer/CommandLineTools/usr/bin/xcrun
> error: command 'gcc' failed with exit status 1
>
> ----------------------------------------
> Command
> "/Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python
> -u -c "import setuptools,
> tokenize;__file__='/private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/setup.py';f=getattr(tokenize,
> 'open', open)(__file__);code=f.read().replace('\r\n',
> '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record
> /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-record-sAEfXi/install-record.txt
> --single-version-externally-managed --compile --install-headers
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/include/site/python2.7/fastavro"
> failed with error code 1 in
> /private/var/folders/7p/4pkdvcd52b36bzwq9lts1frh000r4y/T/pip-install-WX5Ntf/fastavro/
>
> py27-lint installed:
> astroid==1.6.5,avro==1.8.2,backports.functools-lru-cache==1.5,configparser==3.5.0,crcmod==1.7,dill==0.2.6,enum34==1.1.6,flake8==3.5.0,future==0.16.0,isort==4.2.15,lazy-object-proxy==1.3.1,mccabe==0.6.1,pycodestyle==2.3.1,pyflakes==1.6.0,pylint==1.7.2,singledispatch==3.4.0.3,six==1.11.0,wrapt==1.10.11
> ___________________________________ summary
> ____________________________________
> ERROR: py27-lint: InvocationError for command
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/python
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/bin/pip
> install
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/dist/apache-beam-2.6.0.dev0.zip[test]
> (see
> /Users/relax/beam-gradle/beam/sdks/python/target/.tox/py27-lint/log/py27-lint-2.log)
> (exited with code 1)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
