See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/1221/changes>
Changes:
[robertwb] Add support for reading/writing headers to text files
------------------------------------------
[...truncated 2622 lines...]
>>> RUNNING DATAFLOW RUNNER VALIDATESRUNNER TESTS
python setup.py nosetests \
-a ValidatesRunner --test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--sdk_location=$SDK_LOCATION \
--job_name=$JOBNAME_VR_TEST \
--requirements_file=postcommit_requirements.txt \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/setuptools/dist.py>:333:
UserWarning: Normalizing '0.6.0.dev' to '0.6.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam_sdk.egg-info/requires.txt
writing apache_beam_sdk.egg-info/PKG-INFO
writing top-level names to apache_beam_sdk.egg-info/top_level.txt
writing dependency_links to apache_beam_sdk.egg-info/dependency_links.txt
writing entry points to apache_beam_sdk.egg-info/entry_points.txt
reading manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
Searching for pyhamcrest<2.0,>=1.9
Best match: PyHamcrest 1.9.0
Processing PyHamcrest-1.9.0-py2.7.egg
Using
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/.eggs/PyHamcrest-1.9.0-py2.7.egg>
test_empty_side_outputs (apache_beam.transforms.ptransform_test.PTransformTest)
...
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136:
UserWarning: Using fallback coder for typehint: Union[SideOutputValue, int].
warnings.warn('Using fallback coder for typehint: %r.' % typehint)
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136:
UserWarning: Using fallback coder for typehint: Any.
warnings.warn('Using fallback coder for typehint: %r.' % typehint)
DEPRECATION: pip install --download has been deprecated and will be removed in
the future. Pip now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318:
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name
Indication) extension to TLS is not available on this platform. This may cause
the server to present an incorrect TLS certificate, which can cause validation
failures. You can upgrade to a newer version of Python to solve this. For more
information, see
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122:
InsecurePlatformWarning: A true SSLContext object is not available. This
prevents urllib3 from configuring SSL appropriately and may cause certain SSL
connections to fail. You can upgrade to a newer version of Python to solve
this. For more information, see
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
File was already downloaded
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging
appdirs pyparsing
ok
test_par_do_with_multiple_outputs_and_using_return
(apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION: pip
install --download has been deprecated and will be removed in the future. Pip
now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging
appdirs pyparsing
ok
test_par_do_with_multiple_outputs_and_using_yield
(apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION: pip
install --download has been deprecated and will be removed in the future. Pip
now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:318:
SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name
Indication) extension to TLS is not available on this platform. This may cause
the server to present an incorrect TLS certificate, which can cause validation
failures. You can upgrade to a newer version of Python to solve this. For more
information, see
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
SNIMissingWarning
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py>:122:
InsecurePlatformWarning: A true SSLContext object is not available. This
prevents urllib3 from configuring SSL appropriately and may cause certain SSL
connections to fail. You can upgrade to a newer version of Python to solve
this. For more information, see
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
File was already downloaded
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging
appdirs pyparsing
ok
test_undeclared_side_outputs
(apache_beam.transforms.ptransform_test.PTransformTest) ... DEPRECATION: pip
install --download has been deprecated and will be removed in the future. Pip
now has a download command that should be used instead.
Collecting pyhamcrest (from -r postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/PyHamcrest-1.9.0.tar.gz
Collecting mock (from -r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/mock-2.0.0.tar.gz
Collecting setuptools (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/setuptools-34.1.1.zip
Collecting six (from pyhamcrest->-r postcommit_requirements.txt (line 1))
File was already downloaded /tmp/dataflow-requirements-cache/six-1.10.0.tar.gz
Collecting funcsigs>=1 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded
/tmp/dataflow-requirements-cache/funcsigs-1.0.2.tar.gz
Collecting pbr>=0.11 (from mock->-r postcommit_requirements.txt (line 2))
File was already downloaded /tmp/dataflow-requirements-cache/pbr-1.10.0.tar.gz
Collecting packaging>=16.8 (from setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/packaging-16.8.tar.gz
Collecting appdirs>=1.4.0 (from setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/appdirs-1.4.0.tar.gz
Collecting pyparsing (from packaging>=16.8->setuptools->pyhamcrest->-r
postcommit_requirements.txt (line 1))
File was already downloaded
/tmp/dataflow-requirements-cache/pyparsing-2.1.10.zip
Successfully downloaded pyhamcrest mock setuptools six funcsigs pbr packaging
appdirs pyparsing
ok
test_as_dict_with_unique_labels
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_and_as_dict_side_inputs
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_with_unique_labels
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_list_without_unique_labels
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_with_different_defaults_with_unique_labels
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_as_singleton_with_different_defaults_without_unique_labels
(apache_beam.transforms.sideinputs_test.SideInputsTest) ...
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/sideinputs_test.py>:241:
DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
e.exception.message.startswith(
ok
test_as_singleton_without_unique_labels
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_default_value_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_empty_singleton_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
test_iterable_side_input
(apache_beam.transforms.sideinputs_test.SideInputsTest) ... ok
----------------------------------------------------------------------
Ran 14 tests in 1191.035s
OK
# Run wordcount on the Google Cloud Dataflow service
# and validate job that finishes successfully.
echo ">>> RUNNING TEST DATAFLOW RUNNER py-wordcount"
>>> RUNNING TEST DATAFLOW RUNNER py-wordcount
python setup.py nosetests \
-a IT --test-pipeline-options=" \
--runner=TestDataflowRunner \
--project=$PROJECT \
--staging_location=$GCS_LOCATION/staging-wordcount \
--temp_location=$GCS_LOCATION/temp-wordcount \
--output=$GCS_LOCATION/py-wordcount-cloud/output \
--sdk_location=$SDK_LOCATION \
--job_name=$JOBNAME_E2E_WC \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/local/lib/python2.7/site-packages/setuptools/dist.py>:333:
UserWarning: Normalizing '0.6.0.dev' to '0.6.0.dev0'
normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam_sdk.egg-info/requires.txt
writing apache_beam_sdk.egg-info/PKG-INFO
writing top-level names to apache_beam_sdk.egg-info/top_level.txt
writing dependency_links to apache_beam_sdk.egg-info/dependency_links.txt
writing entry points to apache_beam_sdk.egg-info/entry_points.txt
reading manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'apache_beam_sdk.egg-info/SOURCES.txt'
Searching for pyhamcrest<2.0,>=1.9
Best match: PyHamcrest 1.9.0
Processing PyHamcrest-1.9.0-py2.7.egg
Using
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/.eggs/PyHamcrest-1.9.0-py2.7.egg>
test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT) ...
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/io/gcsio.py>:109:
DeprecationWarning: object() takes no parameters
super(GcsIO, cls).__new__(cls, storage_client))
<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/coders/typecoders.py>:136:
UserWarning: Using fallback coder for typehint: Any.
warnings.warn('Using fallback coder for typehint: %r.' % typehint)
ERROR
======================================================================
ERROR: test_wordcount_it (apache_beam.examples.wordcount_it_test.WordCountIT)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount_it_test.py";,>
line 54, in test_wordcount_it
wordcount.run(test_pipeline.get_full_options_as_args(**extra_opts))
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/examples/wordcount.py";,>
line 101, in run
result = p.run()
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py";,>
line 163, in run
return self.runner.run(self)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/test/test_dataflow_runner.py";,>
line 32, in run
self.result = super(TestDataflowRunner, self).run(pipeline)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py";,>
line 167, in run
super(DataflowRunner, self).run(pipeline)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/runner.py";,>
line 113, in run
pipeline.visit(RunVisitor(self))
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py";,>
line 186, in visit
self._root_transform().visit(visitor, self, visited)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py";,>
line 423, in visit
part.visit(visitor, pipeline, visited)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py";,>
line 423, in visit
part.visit(visitor, pipeline, visited)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py";,>
line 423, in visit
part.visit(visitor, pipeline, visited)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py";,>
line 423, in visit
part.visit(visitor, pipeline, visited)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/pipeline.py";,>
line 426, in visit
visitor.visit_transform(self)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/runner.py";,>
line 108, in visit_transform
self.runner.run_transform(transform_node)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/runner.py";,>
line 183, in run_transform
return m(transform_node)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py";,>
line 395, in run_ParDo
transform_node.transform.side_output_tags)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/runners/dataflow_runner.py";,>
line 267, in _add_step
DisplayData.create_from(transform_node.transform).items])
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py";,>
line 152, in create_from
return cls(has_display_data._namespace(), has_display_data.display_data())
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py";,>
line 83, in __init__
self._populate_items(display_data_dict)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py";,>
line 91, in _populate_items
element.display_data())
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py";,>
line 83, in __init__
self._populate_items(display_data_dict)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/transforms/display.py";,>
line 91, in _populate_items
element.display_data())
File
"<https://builds.apache.org/job/beam_PostCommit_Python_Verify/ws/sdks/python/apache_beam/io/textio.py";,>
line 321, in display_data
self.append_trailing_newlines,
AttributeError: '_TextSink' object has no attribute 'append_trailing_newlines'
-------------------- >> begin captured logging << --------------------
oauth2client.contrib.multistore_file: WARNING: The
oauth2client.contrib.multistore_file module has been deprecated and will be
removed in the next release of oauth2client. Please migrate to
multiprocess_file_storage.
root: WARNING: snappy is not installed; some tests will be skipped.
root: WARNING: Tensorflow is not installed, so skipping some tests.
root: DEBUG: PValue computed by read/Read (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by split (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by pair_with_one (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by group (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by count (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by write/Write/WriteImpl/DoOnce (tag 1): refcount:
2 => 1
root: DEBUG: PValue computed by write/Write/WriteImpl/InitializeWrite (tag 1):
refcount: 1 => 0
root: DEBUG: PValue computed by format (tag 1): refcount: 1 => 0
root: DEBUG: PValue computed by
write/Write/WriteImpl/ViewAsSingleton(write|Write|WriteImpl|InitializeWrite.None)/CreatePCollectionView
(tag 1): refcount: 2 => 1
root: ERROR: Error while visiting write/Write/WriteImpl/WriteBundles
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
Ran 1 test in 3.569s
FAILED (errors=1)
Build step 'Execute shell' marked build as failure
