[
https://issues.apache.org/jira/browse/BEAM-7527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880806#comment-16880806
]
Udi Meiri commented on BEAM-7527:
---------------------------------
I reproduced the PKG-INFO error using this command (and reducing the tests to
just BigQueryQueryToTableIT:
{code:java}
for i in `seq 100`; do ../../gradlew
:sdks:python:test-suites:direct:py35:postCommitIT
:sdks:python:test-suites:direct:py36:postCommitIT
:sdks:python:test-suites:direct:py37:postCommitIT || break; done{code}
Error:
{code:java}
Installing collected packages: apache-beam
Found existing installation: apache-beam 2.15.0.dev0
Uninstalling apache-beam-2.15.0.dev0:
Successfully uninstalled apache-beam-2.15.0.dev0
Running setup.py develop for apache-beam
ERROR: Complete output from command
/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/bin/python3.5
-c 'import setuptools,
tokenize;__file__='"'"'/usr/local/google/home/ehudm/src/beam/sdks/python/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop
--no-deps:
ERROR: running develop
running egg_info
writing entry points to apache_beam.egg-info/entry_points.txt
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing top-level names to apache_beam.egg-info/top_level.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
/usr/local/google/home/ehudm/src/beam/sdks/python/setup.py:177: UserWarning:
Some syntactic constructs of Python 3 are not yet fully supported by Apache
Beam.
'Some syntactic constructs of Python 3 are not yet fully supported by '
Rolling back uninstall of apache-beam
/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/setuptools/dist.py:472:
UserWarning: Normalizing '2.15.0.dev' to '2.15.0.dev0'
normalized_version,
warning: no files found matching 'README.md'
Moving to
/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/apache-beam.egg-link
warning: no files found matching 'NOTICE'
from /tmp/pip-uninstall-w0mv7gcp/apache-beam.egg-link
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
running build_ext
Creating
/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/apache-beam.egg-link
(link to .)
Adding apache-beam 2.15.0.dev0 to easy-install.pth file
Traceback (most recent call last):
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/pkg_resources/__init__.py",
line 2672, in version
return self._version
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/pkg_resources/__init__.py",
line 2806, in __getattr__
raise AttributeError(attr)
AttributeError: _version
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/local/google/home/ehudm/src/beam/sdks/python/setup.py", line 238, in
<module>
'test': generate_protos_first(test),
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/setuptools/__init__.py",
line 145, in setup
return distutils.core.setup(**attrs)
File "/usr/lib/python3.5/distutils/core.py", line 148, in setup
dist.run_commands()
File "/usr/lib/python3.5/distutils/dist.py", line 955, in run_commands
self.run_command(cmd)
File "/usr/lib/python3.5/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/local/google/home/ehudm/src/beam/sdks/python/setup.py", line 166, in
run
super(cmd, self).run()
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/setuptools/command/develop.py",
line 38, in run
self.install_for_development()
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/setuptools/command/develop.py",
line 156, in install_for_development
self.process_distribution(None, self.dist, not self.no_deps)
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/setuptools/command/easy_install.py",
line 727, in process_distribution
self.update_pth(dist)
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/setuptools/command/easy_install.py",
line 1215, in update_pth
self.pth_file.add(dist) # add new entry
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/setuptools/command/easy_install.py",
line 1662, in add
Environment.add(self, dist)
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/pkg_resources/__init__.py",
line 1037, in add
dists.sort(key=operator.attrgetter('hashcmp'), reverse=True)
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/pkg_resources/__init__.py",
line 2592, in hashcmp
self.parsed_version,
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/pkg_resources/__init__.py",
line 2639, in parsed_version
self._parsed_version = parse_version(self.version)
File
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/lib/python3.5/site-packages/pkg_resources/__init__.py",
line 2680, in version
raise ValueError(msg, self)
ValueError: ("Missing 'Version:' header and/or PKG-INFO file at path:
/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam.egg-info/PKG-INFO",
apache-beam 2.15.0.dev0 (/usr/local/google/home/ehudm/src/beam/sdks/python))
----------------------------------------
ERROR: Command
"/usr/local/google/home/ehudm/src/beam/build/gradleenv/1398941889/bin/python3.5
-c 'import setuptools,
tokenize;__file__='"'"'/usr/local/google/home/ehudm/src/beam/sdks/python/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop
--no-deps" failed with error code 1 in
/usr/local/google/home/ehudm/src/beam/sdks/python/
{code}
> Beam Python integration test suites are flaky: ModuleNotFoundError
> ------------------------------------------------------------------
>
> Key: BEAM-7527
> URL: https://issues.apache.org/jira/browse/BEAM-7527
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Valentyn Tymofieiev
> Assignee: Mark Liu
> Priority: Major
>
> I am seeing several errors in Python SDK Integration test suites, such as
> Dataflow ValidatesRunner and Python PostCommit that fail due to one of the
> autogenerated files not being found.
> For example:
> {noformat}
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/__init__.py:84:
> UserWarning: Running the Apache Beam SDK on Python 3 is not yet fully
> supported. You may encounter buggy behavior or missing features.
> 'Running the Apache Beam SDK on Python 3 is not yet fully supported. '
> Failure: ModuleNotFoundError (No module named 'beam_runner_api_pb2') ...
> ERROR
> ======================================================================
> ERROR: Failure: ModuleNotFoundError (No module named 'beam_runner_api_pb2')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/site-packages/nose/failure.py",
> line 39, in runTest
> raise self.exc_val.with_traceback(self.tb)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/site-packages/nose/loader.py",
> line 418, in loadTestsFromName
> addr.filename, addr.module)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/site-packages/nose/importer.py",
> line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/site-packages/nose/importer.py",
> line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/imp.py",
> line 245, in load_module
> return load_package(name, filename)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/imp.py",
> line 217, in load_package
> return _load(spec)
> File "<frozen importlib._bootstrap>", line 684, in _load
> File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
> File "<frozen importlib._bootstrap_external>", line 678, in exec_module
> File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/__init__.py",
> line 97, in <module>
> from apache_beam import coders
> File "/home/jenkins/jenkins-slave/workspace/beam_Pos
> tCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/coders/__init__.py", line
> 19, in <module>
> from apache_beam.coders.coders import *
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/coders/coders.py",
> line 32, in <module>
> from apache_beam.coders import coder_impl
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/coders/coder_impl.py",
> line 44, in <module>
> from apache_beam.utils import windowed_value
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/utils/windowed_value.py",
> line 34, in <module>
> from apache_beam.utils.timestamp import MAX_TIMESTAMP
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/utils/timestamp.py",
> line 34, in <module>
> from apache_beam.portability import common_urns
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/portability/common_urns.py",
> line 25, in <module>
> from apache_beam.portability.api import metrics_pb2
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/portability/api/metrics_pb2.py",
> line 16, in <module>
> import beam_runner_api_pb2 as beam__runner__api__pb2
> ModuleNotFoundError: No module named 'beam_runner_api_pb2'
> {noformat}
> {noformat}
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/__init__.py:84:
> UserWarning: Running the Apache Beam SDK on Python 3 is not yet fully
> supported. You may encounter buggy behavior or missing features.
> 'Running the Apache Beam SDK on Python 3 is not yet fully supported. '
> Failure: ModuleNotFoundError (No module named 'endpoints_pb2') ...
> ERROR
> ======================================================================
> ERROR: Failure: ModuleNotFoundError (No module named 'endpoints_pb2')
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/site-packages/nose/failure.py",
> line 39, in runTest
> raise self.exc_val.with_traceback(self.tb)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/site-packages/nose/loader.py",
> line 418, in loadTestsFromName
> addr.filename, addr.module)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/site-packages/nose/importer.py",
> line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/site-packages/nose/importer.py",
> line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/imp.py",
> line 245, in load_module
> return load_package(name, filename)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/build/gradleenv/-1734967053/lib/python3.6/imp.py",
> line 217, in load_package
> return _load(spec)
> File "<frozen importlib._bootstrap>", line 684, in _load
> File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
> File "<frozen importlib._bootstrap_external>", line 678, in exec_module
> File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/__init__.py",
> line 97, in <module>
> from apache_beam import coders
> File "/home/jenkins/jenkins-slave/workspace/beam_PostCommi
> t_Py_VR_Dataflow/src/sdks/python/apache_beam/coders/__init__.py", line 19, in
> <module>
> from apache_beam.coders.coders import *
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/coders/coders.py",
> line 32, in <module>
> from apache_beam.coders import coder_impl
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/coders/coder_impl.py",
> line 44, in <module>
> from apache_beam.utils import windowed_value
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/utils/windowed_value.py",
> line 34, in <module>
> from apache_beam.utils.timestamp import MAX_TIMESTAMP
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/utils/timestamp.py",
> line 34, in <module>
> from apache_beam.portability import common_urns
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/portability/common_urns.py",
> line 24, in <module>
> from apache_beam.portability.api import beam_runner_api_pb2
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/sdks/python/apache_beam/portability/api/beam_runner_api_pb2.py",
> line 16, in <module>
> import endpoints_pb2 as endpoints__pb2
> ModuleNotFoundError: No module named 'endpoints_pb2'
> {noformat}
> The rootcause is not clear, I suspect that it may be related to the way we
> parallelize execution of Python test suites for 2.7, 3.5, 3.6, 3.7.
> cc: [~altay] [~markflyhigh] [~Juta] [~frederik]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
