See <https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/25/display/redirect?page=changes>
Changes: [kedin] [SQL] Add support for DQL to PubsubJsonTable [kedin] [SQL] Add integration test for Pubsub Json Table DLQ [apilloud] [SQL] Add jdbc:beam: [apilloud] [SQL] Use CalciteConnection as context [ankurgoenka] Copyting dependency.py and dependency_test.py [ankurgoenka] reformatting code [ankurgoenka] Introducing classes [ankurgoenka] gruping public and non public methods [ankurgoenka] Renaming method and updating reference for file_copy and file_download [ankurgoenka] Removing Google specific code and unused code in stager. [ankurgoenka] Removing reference to Google/GCS/Dataflow and restructuring a bit of [ankurgoenka] Fixing Stager Tests [ankurgoenka] Making dependency.py use stager.py and fixing dependency_test.py test [ankurgoenka] Moving stager to portability [ankurgoenka] Removing stage_job_resources from dependencies.py and other minor [ankurgoenka] Merging filehandler in stager [ankurgoenka] renaming GCSStager to _ParameterizedStager [ankurgoenka] Making a few methods static ------------------------------------------ [...truncated 156.14 KB...] File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/utils/retry.py";,> line 180, in wrapper return fun(*args, **kwargs) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py";,> line 485, in create_job self.create_job_description(job) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py";,> line 511, in create_job_description StringIO(job.proto_pipeline.SerializeToString())) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py";,> line 467, in stage_file response = self._storage_client.objects.Insert(request, upload=upload) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py";,> line 971, in Insert download=download) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py";,> line 720, in _RunMethod http, http_request, **opts) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py";,> line 346, in MakeRequest check_response_func=check_response_func) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py";,> line 396, in _MakeRequestNoRetry redirections=redirections, connection_type=connection_type) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py";,> line 175, in new_request redirections, connection_type) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py";,> line 282, in request connection_type=connection_type) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py";,> line 1609, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py";,> line 1351, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py";,> line 1307, in _conn_request response = conn.getresponse() File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse response.begin() File "/usr/lib/python2.7/httplib.py", line 453, in begin version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 409, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib/python2.7/socket.py", line 480, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/python2.7/ssl.py", line 756, in recv return self.read(buflen) File "/usr/lib/python2.7/ssl.py", line 643, in read v = self._sslobj.read(len) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,> line 276, in signalhandler raise TimedOutException() TimedOutException: 'test_multi_valued_singleton_side_input (apache_beam.transforms.sideinputs_test.SideInputsTest)' ====================================================================== ERROR: test_multiple_empty_outputs (apache_beam.transforms.ptransform_test.PTransformTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,> line 812, in run test(orig) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py";,> line 45, in __call__ return self.run(*arg, **kwarg) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py";,> line 133, in run self.runTest(result) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py";,> line 151, in runTest test(result) File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__ return self.run(*args, **kwds) File "/usr/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/transforms/ptransform_test.py";,> line 271, in test_multiple_empty_outputs pipeline.run() File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/testing/test_pipeline.py";,> line 102, in run result = super(TestPipeline, self).run(test_runner_api) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/pipeline.py";,> line 389, in run self.to_runner_api(), self.runner, self._options).run(False) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/pipeline.py";,> line 402, in run return self.runner.run_pipeline(self) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,> line 67, in run_pipeline self.wait_until_in_state(PipelineState.CANCELLED, timeout=300) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,> line 87, in wait_until_in_state job_state = self.result.state File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,> line 1033, in state self._update_job() File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,> line 1011, in _update_job self._job = self._runner.dataflow_client.get_job(self.job_id()) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/utils/retry.py";,> line 180, in wrapper return fun(*args, **kwargs) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py";,> line 624, in get_job response = self._client.projects_locations_jobs.Get(request) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py";,> line 656, in Get config, request, global_params=global_params) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py";,> line 720, in _RunMethod http, http_request, **opts) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py";,> line 346, in MakeRequest check_response_func=check_response_func) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py";,> line 396, in _MakeRequestNoRetry redirections=redirections, connection_type=connection_type) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py";,> line 175, in new_request redirections, connection_type) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/oauth2client/transport.py";,> line 282, in request connection_type=connection_type) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py";,> line 1609, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py";,> line 1351, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/httplib2/__init__.py";,> line 1307, in _conn_request response = conn.getresponse() File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse response.begin() File "/usr/lib/python2.7/httplib.py", line 453, in begin version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 409, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib/python2.7/socket.py", line 480, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/python2.7/ssl.py", line 756, in recv return self.read(buflen) File "/usr/lib/python2.7/ssl.py", line 643, in read v = self._sslobj.read(len) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,> line 276, in signalhandler raise TimedOutException() TimedOutException: 'test_multiple_empty_outputs (apache_beam.transforms.ptransform_test.PTransformTest)' ====================================================================== ERROR: test_par_do_with_multiple_outputs_and_using_return (apache_beam.transforms.ptransform_test.PTransformTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,> line 812, in run test(orig) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py";,> line 45, in __call__ return self.run(*arg, **kwarg) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py";,> line 133, in run self.runTest(result) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py";,> line 151, in runTest test(result) File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__ return self.run(*args, **kwds) File "/usr/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/transforms/ptransform_test.py";,> line 244, in test_par_do_with_multiple_outputs_and_using_return pipeline.run() File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/testing/test_pipeline.py";,> line 102, in run result = super(TestPipeline, self).run(test_runner_api) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/pipeline.py";,> line 389, in run self.to_runner_api(), self.runner, self._options).run(False) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/pipeline.py";,> line 402, in run return self.runner.run_pipeline(self) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,> line 67, in run_pipeline self.wait_until_in_state(PipelineState.CANCELLED, timeout=300) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,> line 90, in wait_until_in_state time.sleep(5) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,> line 276, in signalhandler raise TimedOutException() TimedOutException: 'test_par_do_with_multiple_outputs_and_using_return (apache_beam.transforms.ptransform_test.PTransformTest)' ====================================================================== ERROR: test_par_do_with_multiple_outputs_and_using_yield (apache_beam.transforms.ptransform_test.PTransformTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,> line 812, in run test(orig) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py";,> line 45, in __call__ return self.run(*arg, **kwarg) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py";,> line 133, in run self.runTest(result) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py";,> line 151, in runTest test(result) File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__ return self.run(*args, **kwds) File "/usr/lib/python2.7/unittest/case.py", line 329, in run testMethod() File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/transforms/ptransform_test.py";,> line 228, in test_par_do_with_multiple_outputs_and_using_yield pipeline.run() File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/testing/test_pipeline.py";,> line 102, in run result = super(TestPipeline, self).run(test_runner_api) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/pipeline.py";,> line 389, in run self.to_runner_api(), self.runner, self._options).run(False) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/pipeline.py";,> line 402, in run return self.runner.run_pipeline(self) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,> line 67, in run_pipeline self.wait_until_in_state(PipelineState.CANCELLED, timeout=300) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py";,> line 87, in wait_until_in_state job_state = self.result.state File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,> line 1033, in state self._update_job() File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,> line 1011, in _update_job self._job = self._runner.dataflow_client.get_job(self.job_id()) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/utils/retry.py";,> line 180, in wrapper return fun(*args, **kwargs) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py";,> line 624, in get_job response = self._client.projects_locations_jobs.Get(request) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py";,> line 656, in Get config, request, global_params=global_params) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py";,> line 722, in _RunMethod return self.ProcessHttpResponse(method_config, http_response, request) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py";,> line 728, in ProcessHttpResponse self.__ProcessHttpResponse(method_config, http_response, request)) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py";,> line 611, in __ProcessHttpResponse response_type, http_response.content) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py";,> line 442, in DeserializeMessage message = encoding.JsonToMessage(response_type, data) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/encoding.py";,> line 104, in JsonToMessage return _ProtoJsonApiTools.Get().decode_message(message_type, message) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/encoding.py";,> line 290, in decode_message message_type, result) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/protorpclite/protojson.py";,> line 211, in decode_message message = self.__decode_dictionary(message_type, dictionary) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/protorpclite/protojson.py";,> line 284, in __decode_dictionary for item in value] File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/encoding.py";,> line 312, in decode_field field.message_type, json.dumps(value)) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/encoding.py";,> line 291, in decode_message result = _ProcessUnknownEnums(result, encoded_message) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/encoding.py";,> line 512, in _ProcessUnknownEnums decoded_message = json.loads(encoded_message) File "/usr/lib/python2.7/json/__init__.py", line 339, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 364, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode obj, end = self.scan_once(s, idx) File "<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py";,> line 276, in signalhandler raise TimedOutException() TimedOutException: 'test_par_do_with_multiple_outputs_and_using_yield (apache_beam.transforms.ptransform_test.PTransformTest)' ---------------------------------------------------------------------- XML: <https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/nosetests.xml> ---------------------------------------------------------------------- Ran 16 tests in 1803.745s FAILED (errors=16) > Task :beam-sdks-python:validatesRunnerTests FAILED :beam-sdks-python:validatesRunnerTests (Thread[Task worker for ':' Thread 10,5,main]) completed. Took 30 mins 4.98 secs. FAILURE: Build failed with an exception. * Where: Build file '<https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/ws/src/sdks/python/build.gradle'> line: 211 * What went wrong: Execution failed for task ':beam-sdks-python:validatesRunnerTests'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org BUILD FAILED in 30m 23s 3 actionable tasks: 3 executed Publishing build scan... https://gradle.com/s/2p5btlutl6yym Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected]
