See <https://builds.apache.org/job/beam_PostCommit_Py_ValCont/6364/display/redirect?page=changes>
Changes: [rohde.samuel] Make GroupByKey a primitive [rohde.samuel] Move ReifyWindows etc to DirectRunner, don't subclass from GBK [rohde.samuel] remove references to _GroupByKeyOnly [rohde.samuel] add back the ReifyWindows to the GBK, some internal tests depend on it. [chamikara] Clarifies Dataflow execution environment model [chamikara] Fixes typos. [rohde.samuel] Skip checking input determinism for GBKs embedded in x-lang transforms. [chamikara] Clarifies text. [rehmanmuradali0] Update changes.md ------------------------------------------ [...truncated 769.89 KB...] --nologcapture \ --processes=1 \ --process-timeout=900 \ --with-xunitmp \ --xunitmp-file=$XUNIT_FILE \ --ignore-files '.*py3\d?\.py$' \ --test-pipeline-options=" \ --runner=TestDataflowRunner \ --project=$PROJECT \ --region=$REGION \ --worker_harness_container_image=$CONTAINER:$TAG \ --staging_location=$GCS_LOCATION/staging-validatesrunner-test \ --temp_location=$GCS_LOCATION/temp-validatesrunner-test \ --output=$GCS_LOCATION/output \ --sdk_location=$SDK_LOCATION \ --num_workers=1" setup.py:262: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only. 'You are using Apache Beam with Python 2. ' <https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/setuptools/dist.py>:476: UserWarning: Normalizing '2.23.0.dev' to '2.23.0.dev0' normalized_version, running nosetests running egg_info <https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/gen_protos.py>:241: UserWarning: Installing grpcio-tools is recommended for development. warnings.warn('Installing grpcio-tools is recommended for development.') INFO:gen_protos:Skipping proto regeneration: all files up to date writing requirements to apache_beam.egg-info/requires.txt writing apache_beam.egg-info/PKG-INFO writing top-level names to apache_beam.egg-info/top_level.txt writing dependency_links to apache_beam.egg-info/dependency_links.txt writing entry points to apache_beam.egg-info/entry_points.txt reading manifest file 'apache_beam.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching 'README.md' warning: no files found matching 'NOTICE' warning: no files found matching 'LICENSE' writing manifest file 'apache_beam.egg-info/SOURCES.txt' <https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/__init__.py>:82: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only. 'You are using Apache Beam with Python 2. ' WARNING:root:python-snappy is not installed; some tests will be skipped. WARNING:root:Tensorflow is not installed, so skipping some tests. WARNING:apache_beam.runners.interactive.interactive_environment:Interactive Beam requires Python 3.5.3+. WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features. WARNING:apache_beam.runners.interactive.interactive_environment:You cannot use Interactive Beam features when you are not in an interactive environment such as a Jupyter notebook or ipython terminal. WARNING:root:Make sure that locally built Python SDK docker image has Python 2.7 interpreter. WARNING:root:Make sure that locally built Python SDK docker image has Python 2.7 interpreter. test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) ... ERROR test_metrics_fnapi_it (apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest) ... ERROR ====================================================================== ERROR: test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) ---------------------------------------------------------------------- Traceback (most recent call last): File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",> line 812, in run test(orig) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py",> line 45, in __call__ return self.run(*arg, **kwarg) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py",> line 133, in run self.runTest(result) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py",> line 151, in runTest test(result) File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__ return self.run(*args, **kwds) File "/usr/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount_it_test.py",> line 54, in test_wordcount_fnapi_it self._run_wordcount_it(wordcount.run, experiment='beam_fn_api') File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount_it_test.py",> line 91, in _run_wordcount_it save_main_session=False) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/examples/wordcount.py",> line 121, in run result = p.run() File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py",> line 512, in run self._options).run(False) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py",> line 525, in run return self.runner.run_pipeline(self, self._options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",> line 75, in run_pipeline self.result.cancel() File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1679, in cancel return self.state File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1617, in state self._update_job() File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 1570, in _update_job self._job = self._runner.dataflow_client.get_job(self.job_id()) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/utils/retry.py",> line 236, in wrapper return fun(*args, **kwargs) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",> line 828, in get_job response = self._client.projects_locations_jobs.Get(request) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",> line 699, in Get config, request, global_params=global_params) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",> line 729, in _RunMethod http, http_request, **opts) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",> line 350, in MakeRequest check_response_func=check_response_func) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",> line 400, in _MakeRequestNoRetry redirections=redirections, connection_type=connection_type) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/oauth2client/transport.py",> line 169, in new_request redirections, connection_type) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/oauth2client/transport.py",> line 169, in new_request redirections, connection_type) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/httplib2/__init__.py",> line 2189, in request cachekey, File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/httplib2/__init__.py",> line 1845, in _request conn, request_uri, method, body, headers File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/httplib2/__init__.py",> line 1786, in _conn_request response = conn.getresponse() File "/usr/lib/python2.7/httplib.py", line 1165, in getresponse response.begin() File "/usr/lib/python2.7/httplib.py", line 463, in begin version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 419, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib/python2.7/socket.py", line 480, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/python2.7/ssl.py", line 756, in recv return self.read(buflen) File "/usr/lib/python2.7/ssl.py", line 643, in read v = self._sslobj.read(len) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",> line 276, in signalhandler raise TimedOutException() TimedOutException: 'test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT)' ====================================================================== ERROR: test_metrics_fnapi_it (apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",> line 812, in run test(orig) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py",> line 45, in __call__ return self.run(*arg, **kwarg) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py",> line 133, in run self.runTest(result) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/case.py",> line 151, in runTest test(result) File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__ return self.run(*args, **kwds) File "/usr/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py",> line 57, in test_metrics_fnapi_it result = self.run_pipeline(experiment='beam_fn_api') File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline_test.py",> line 45, in run_pipeline return dataflow_exercise_metrics_pipeline.apply_and_run(p) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_exercise_metrics_pipeline.py",> line 167, in apply_and_run | 'm_out' >> beam.FlatMap( File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/ptransform.py",> line 562, in __ror__ result = p.apply(self, pvalueish, label) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py",> line 654, in apply pvalueish_result = self.runner.apply(transform, pvalueish, self._options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 153, in apply return super(DataflowRunner, self).apply(transform, input, options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py",> line 198, in apply return m(transform, input, options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py",> line 228, in apply_PTransform return transform.expand(input) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2625, in expand | Map(coder.decode).with_output_types(self.get_output_type())) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pvalue.py",> line 140, in __or__ return self.pipeline.apply(ptransform, self) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py",> line 654, in apply pvalueish_result = self.runner.apply(transform, pvalueish, self._options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 153, in apply return super(DataflowRunner, self).apply(transform, input, options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py",> line 198, in apply return m(transform, input, options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py",> line 228, in apply_PTransform return transform.expand(input) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/core.py",> line 2616, in expand return pcoll | Reshuffle() File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pvalue.py",> line 140, in __or__ return self.pipeline.apply(ptransform, self) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py",> line 654, in apply pvalueish_result = self.runner.apply(transform, pvalueish, self._options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 153, in apply return super(DataflowRunner, self).apply(transform, input, options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py",> line 198, in apply return m(transform, input, options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py",> line 228, in apply_PTransform return transform.expand(input) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/util.py",> line 715, in expand | 'RemoveRandomKeys' >> Map(lambda t: t[1]).with_input_types( File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pvalue.py",> line 140, in __or__ return self.pipeline.apply(ptransform, self) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py",> line 654, in apply pvalueish_result = self.runner.apply(transform, pvalueish, self._options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",> line 153, in apply return super(DataflowRunner, self).apply(transform, input, options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py",> line 198, in apply return m(transform, input, options) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/runners/runner.py",> line 228, in apply_PTransform return transform.expand(input) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/transforms/util.py",> line 686, in expand | FlatMap(restore_timestamps).with_output_types(Any)) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pvalue.py",> line 140, in __or__ return self.pipeline.apply(ptransform, self) File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/pipeline.py",> line 680, in apply 'passthrough_pcollection_output_ids', default=False): File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/apache_beam/options/pipeline_options.py",> line 833, in lookup_experiment for experiment in self.experiments: File "<https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/container/venv/python/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",> line 276, in signalhandler raise TimedOutException() TimedOutException: 'test_metrics_fnapi_it (apache_beam.runners.dataflow.dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest)' ---------------------------------------------------------------------- XML: nosetests-beam_python2.7_sdk.xml ---------------------------------------------------------------------- XML: <https://builds.apache.org/job/beam_PostCommit_Py_ValCont/ws/src/sdks/python/nosetests.xml> ---------------------------------------------------------------------- Ran 2 tests in 904.689s FAILED (errors=2) cleanup_container Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk:20200521-181539 Untagged: us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk@sha256:6feb2e5e188a564889431b5fed087df808e84715f4629558fc3900e190bf473c Deleted: sha256:ab0e1c4b9fb2710f1aaf59064e399cdefbf9021e9f6f69be42844e026679cfa2 Deleted: sha256:c425afaed643ca42cc119f4e78ebcf4b89d0537636422d0fff727a82e3cc3776 Deleted: sha256:7ee2b87f3955434d237a3679b906c94f840372fa158ec1bb30b845a36d0e94c0 Deleted: sha256:2cd3b991dbb44c0d7fdade714ed967749684b293b152ba34d2124f0e01a28436 Deleted: sha256:e8ab3a48c7a780162781b9f207feefc21101357c86c62b26adaf00ada015e1f6 Deleted: sha256:8fc1bb25a463d5bff8ac73056f1fa5f44f88a57daae2ad0113a8b84de186f8cd Deleted: sha256:554dbf8f233ffda31733e6c5c8135e9690d7a85e53bd7f4d7c8d28fb6727fc6a Deleted: sha256:c82d6289d94e2410b0016df4c62442f46f2dc0e51fc27c27be39824b81c3684b Deleted: sha256:a9a511590d71ec105d2b54454dc2afcdb11d40e6abed6e79b04f4d2a3bd9e04e Deleted: sha256:84ee335305f72e5ac69c3930f1c634fd406b5bd1dddebdffb53e581211b4467b Deleted: sha256:45af70cdc20c8a29ee89501aa2b6ab17ac744fc39d2f425b9a661456fc7c9291 Deleted: sha256:64d1b84f580405d7a4a27b633060f67fdf293f962bd10a95c143846a3a08cee4 Deleted: sha256:2580553c8302efb0e5969ab7f2b78554a042cac3a5a636a7fa71dc999ab777c1 Deleted: sha256:58f40a60aea09d7a89f8f515a7e7feefc86972ec79295cf8ab98a6119cfd32f1 Deleted: sha256:9dedc970d3e61ae384ca37212ca4ce6fd4ab17d31c9982f685576f02878489cd Deleted: sha256:257d98a1d6474b8cf89da73dc6574754c83993be93e7ec1ef2bf8158e6bfe9de Deleted: sha256:dec1f32047b47c05ad5d2eda917c1e0ab1df7b287fd2193fd76ffb486a59b4ad Deleted: sha256:5f8315b076eaffa926118cae3780a4e2264ef3defeaedd19b5ef640481ec3245 Deleted: sha256:1262faff7f15abbc5a73be35c42a1e96552744b2a2b55510ae5ca6b3a95e106b Deleted: sha256:ce12861ea9f71b81d77d19971acd9e51f45ff3236007d6e7d2aa6d18af92b9fc Deleted: sha256:85ae4cb99d5048505e9950c7220837b57cdd86fb31243ed8e6ad16781c9099cc Deleted: sha256:5133083760b36360519f4b8aab0beb0929bfd6b6a82c852218478d43ff7cf4b9 Deleted: sha256:4b915bc0fbd43487d32d8710844dfd47a07982ccd48b199250284d5e53b412eb Deleted: sha256:d884000ae6b47f04fb556da26ce7ca60d1a93c0b75658fc73c12d9d289789f0b Deleted: sha256:5888b6a24a9c64605121f30f909c6ad910a2e25485c990604a2f46b5adff01b5 Deleted: sha256:745ec2b5f53e727f19b8c71723d6486ffcc051e0484dbf82dda619c117a9f7d3 Deleted: sha256:668ca6432a9b37038804eee5caa6a71ea6c8efa880f60ccb5877650d60d05ac6 Deleted: sha256:09845613e31fd42b825088ae82d823be7dadd8c6465876b9f9322ee630a3b856 Digests: - us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk@sha256:6feb2e5e188a564889431b5fed087df808e84715f4629558fc3900e190bf473c Associated tags: - 20200521-181539 Tags: - us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk:20200521-181539 Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk:20200521-181539]. Deleted [us.gcr.io/apache-beam-testing/jenkins/beam_python2.7_sdk@sha256:6feb2e5e188a564889431b5fed087df808e84715f4629558fc3900e190bf473c]. Removed the container Build step 'Execute shell' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: builds-unsubscr...@beam.apache.org For additional commands, e-mail: builds-h...@beam.apache.org