Juta Staes created BEAM-6870:
--------------------------------
Summary: python 3 test_hourly_team_score_it fails with dataflow
job id already exists
Key: BEAM-6870
URL: https://issues.apache.org/jira/browse/BEAM-6870
Project: Beam
Issue Type: Sub-task
Components: sdk-py-core
Reporter: Juta Staes
This IT test on python 3 fails with "Already exists job ..."
{code:java}
14:42:03 ======================================================================
14:42:03 ERROR: test_hourly_team_score_it
(apache_beam.examples.complete.game.hourly_team_score_it_test.HourlyTeamScoreIT)
14:42:03 ----------------------------------------------------------------------
14:42:03 Traceback (most recent call last):
14:42:03 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py",
line 89, in test_hourly_team_score_it
14:42:03 self.test_pipeline.get_full_options_as_args(**extra_opts))
14:42:03 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py",
line 295, in run
14:42:03 }, options.view_as(GoogleCloudOptions).project))
14:42:03 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
line 426, in __exit__
14:42:03 self.run().wait_until_finish()
14:42:03 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
line 406, in run
14:42:03 self._options).run(False)
14:42:03 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
line 419, in run
14:42:03 return self.runner.run_pipeline(self, self._options)
14:42:03 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
line 64, in run_pipeline
14:42:03 self.result.wait_until_finish(duration=wait_duration)
14:42:03 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
line 1217, in wait_until_finish
14:42:03 (self.state, getattr(self._runner, 'last_error_msg', None)), self)
14:42:03 apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException:
Dataflow pipeline failed. State: FAILED, Error:
14:42:03 Traceback (most recent call last):
14:42:03 File "apache_beam/runners/common.py", line 732, in
apache_beam.runners.common.DoFnRunner.process
14:42:03 File "apache_beam/runners/common.py", line 555, in
apache_beam.runners.common.PerWindowInvoker.invoke_process
14:42:03 File "apache_beam/runners/common.py", line 624, in
apache_beam.runners.common.PerWindowInvoker._invoke_per_window
14:42:03 File "apache_beam/runners/common.py", line 816, in
apache_beam.runners.common._OutputProcessor.process_outputs
14:42:03 File "apache_beam/runners/common.py", line 831, in
apache_beam.runners.common._OutputProcessor.process_outputs
14:42:03 File
"/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_file_loads.py",
line 385, in process
14:42:03 create_disposition=self.create_disposition)
14:42:03 File
"/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 560, in perform_load_job
14:42:03 write_disposition=write_disposition)
14:42:03 File
"/usr/local/lib/python3.5/site-packages/apache_beam/utils/retry.py", line 195,
in wrapper
14:42:03 return fun(*args, **kwargs)
14:42:03 File
"/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 327, in _insert_load_job
14:42:03 response = self.client.jobs.Insert(request)
14:42:03 File
"/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 342, in Insert
14:42:03 upload=upload, upload_config=upload_config)
14:42:03 File
"/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line
731, in _RunMethod
14:42:03 return self.ProcessHttpResponse(method_config, http_response,
request)
14:42:03 File
"/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line
737, in ProcessHttpResponse
14:42:03 self.__ProcessHttpResponse(method_config, http_response, request))
14:42:03 File
"/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line
604, in __ProcessHttpResponse
14:42:03 http_response, method_config=method_config, request=request)
14:42:03 apitools.base.py.exceptions.HttpConflictError: HttpError accessing
<https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/jobs?alt=json>:
response: <{'status': '409', 'content-type': 'application/json;
charset=UTF-8', 'x-content-type-options': 'nosniff', 'transfer-encoding':
'chunked', 'cache-control': 'private, max-age=0', '-content-encoding': 'gzip',
'date': 'Wed, 20 Mar 2019 13:29:44 GMT', 'server': 'GSE', 'content-length':
'364', 'x-xss-protection': '1; mode=block', 'vary': 'Origin, X-Origin',
'x-frame-options': 'SAMEORIGIN', 'expires': 'Wed, 20 Mar 2019 13:29:44 GMT'}>,
content <{
14:42:03 "error": {
14:42:03 "errors": [
14:42:03 {
14:42:03 "domain": "global",
14:42:03 "reason": "duplicate",
14:42:03 "message": "Already Exists: Job
apache-beam-testing:US.beam_load_2019_03_20_132147_71_1fea0de5d8e357a03e39484b3ebe0123_0"
14:42:03 }
14:42:03 ],
14:42:03 "code": 409,
14:42:03 "message": "Already Exists: Job
apache-beam-testing:US.beam_load_2019_03_20_132147_71_1fea0de5d8e357a03e39484b3ebe0123_0"
14:42:03 }
14:42:03 }
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
