[
https://issues.apache.org/jira/browse/BEAM-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797387#comment-16797387
]
Juta Staes commented on BEAM-6870:
----------------------------------
The test also fails in isolation because the retry is triggered somehow. I did
not yet look into why this happens.
> python 3 test_hourly_team_score_it fails with bigquery job id already exists
> ----------------------------------------------------------------------------
>
> Key: BEAM-6870
> URL: https://issues.apache.org/jira/browse/BEAM-6870
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Juta Staes
> Priority: Major
>
> This IT test on python 3 fails with "Already exists job ..."
> {code:java}
> 14:42:03
> ======================================================================
> 14:42:03 ERROR: test_hourly_team_score_it
> (apache_beam.examples.complete.game.hourly_team_score_it_test.HourlyTeamScoreIT)
> 14:42:03
> ----------------------------------------------------------------------
> 14:42:03 Traceback (most recent call last):
> 14:42:03 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/complete/game/hourly_team_score_it_test.py",
> line 89, in test_hourly_team_score_it
> 14:42:03 self.test_pipeline.get_full_options_as_args(**extra_opts))
> 14:42:03 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/complete/game/hourly_team_score.py",
> line 295, in run
> 14:42:03 }, options.view_as(GoogleCloudOptions).project))
> 14:42:03 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 426, in __exit__
> 14:42:03 self.run().wait_until_finish()
> 14:42:03 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> 14:42:03 self._options).run(False)
> 14:42:03 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 14:42:03 return self.runner.run_pipeline(self, self._options)
> 14:42:03 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
> line 64, in run_pipeline
> 14:42:03 self.result.wait_until_finish(duration=wait_duration)
> 14:42:03 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
> line 1217, in wait_until_finish
> 14:42:03 (self.state, getattr(self._runner, 'last_error_msg', None)),
> self)
> 14:42:03
> apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException:
> Dataflow pipeline failed. State: FAILED, Error:
> 14:42:03 Traceback (most recent call last):
> 14:42:03 File "apache_beam/runners/common.py", line 732, in
> apache_beam.runners.common.DoFnRunner.process
> 14:42:03 File "apache_beam/runners/common.py", line 555, in
> apache_beam.runners.common.PerWindowInvoker.invoke_process
> 14:42:03 File "apache_beam/runners/common.py", line 624, in
> apache_beam.runners.common.PerWindowInvoker._invoke_per_window
> 14:42:03 File "apache_beam/runners/common.py", line 816, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> 14:42:03 File "apache_beam/runners/common.py", line 831, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> 14:42:03 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_file_loads.py",
> line 385, in process
> 14:42:03 create_disposition=self.create_disposition)
> 14:42:03 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py",
> line 560, in perform_load_job
> 14:42:03 write_disposition=write_disposition)
> 14:42:03 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/utils/retry.py", line
> 195, in wrapper
> 14:42:03 return fun(*args, **kwargs)
> 14:42:03 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/bigquery_tools.py",
> line 327, in _insert_load_job
> 14:42:03 response = self.client.jobs.Insert(request)
> 14:42:03 File
> "/usr/local/lib/python3.5/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
> line 342, in Insert
> 14:42:03 upload=upload, upload_config=upload_config)
> 14:42:03 File
> "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line
> 731, in _RunMethod
> 14:42:03 return self.ProcessHttpResponse(method_config, http_response,
> request)
> 14:42:03 File
> "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line
> 737, in ProcessHttpResponse
> 14:42:03 self.__ProcessHttpResponse(method_config, http_response,
> request))
> 14:42:03 File
> "/usr/local/lib/python3.5/site-packages/apitools/base/py/base_api.py", line
> 604, in __ProcessHttpResponse
> 14:42:03 http_response, method_config=method_config, request=request)
> 14:42:03 apitools.base.py.exceptions.HttpConflictError: HttpError accessing
> <https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/jobs?alt=json>:
> response: <{'status': '409', 'content-type': 'application/json;
> charset=UTF-8', 'x-content-type-options': 'nosniff', 'transfer-encoding':
> 'chunked', 'cache-control': 'private, max-age=0', '-content-encoding':
> 'gzip', 'date': 'Wed, 20 Mar 2019 13:29:44 GMT', 'server': 'GSE',
> 'content-length': '364', 'x-xss-protection': '1; mode=block', 'vary':
> 'Origin, X-Origin', 'x-frame-options': 'SAMEORIGIN', 'expires': 'Wed, 20 Mar
> 2019 13:29:44 GMT'}>, content <{
> 14:42:03 "error": {
> 14:42:03 "errors": [
> 14:42:03 {
> 14:42:03 "domain": "global",
> 14:42:03 "reason": "duplicate",
> 14:42:03 "message": "Already Exists: Job
> apache-beam-testing:US.beam_load_2019_03_20_132147_71_1fea0de5d8e357a03e39484b3ebe0123_0"
> 14:42:03 }
> 14:42:03 ],
> 14:42:03 "code": 409,
> 14:42:03 "message": "Already Exists: Job
> apache-beam-testing:US.beam_load_2019_03_20_132147_71_1fea0de5d8e357a03e39484b3ebe0123_0"
> 14:42:03 }
> 14:42:03 }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
