[
https://issues.apache.org/jira/browse/BEAM-8446?focusedWorklogId=334354&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-334354
]
ASF GitHub Bot logged work on BEAM-8446:
----------------------------------------
Author: ASF GitHub Bot
Created on: 25/Oct/19 20:27
Start Date: 25/Oct/19 20:27
Worklog Time Spent: 10m
Work Description: pabloem commented on pull request #9858: [BEAM-8446]
Adding a test checking the wait for BQ jobs
URL: https://github.com/apache/beam/pull/9858#discussion_r339226407
##########
File path: sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py
##########
@@ -424,6 +425,90 @@ def test_records_traverse_transform_with_mocks(self):
assert_that(jobs,
equal_to([job_reference]), label='CheckJobs')
+ @unittest.skipIf(sys.version_info[0] == 2,
+ 'Mock pickling problems in Py 2')
+ @mock.patch('time.sleep')
+ def test_wait_for_job_completion(self, sleep_mock):
+ job_references = [bigquery_api.JobReference(),
+ bigquery_api.JobReference()]
+ job_references[0].projectId = 'project1'
+ job_references[0].jobId = 'jobId1'
+ job_references[1].projectId = 'project1'
+ job_references[1].jobId = 'jobId2'
+
+ job_1_waiting = mock.Mock()
+ job_1_waiting.status.state = 'RUNNING'
+ job_2_done = mock.Mock()
+ job_2_done.status.state = 'DONE'
+ job_2_done.status.errorResult = None
+
+ job_1_done = mock.Mock()
+ job_1_done.status.state = 'DONE'
+ job_1_done.status.errorResult = None
+
+ bq_client = mock.Mock()
+ bq_client.jobs.Get.side_effect = [
+ job_1_waiting,
+ job_2_done,
+ job_1_done,
+ job_2_done]
+
+ waiting_dofn = bqfl.WaitForBQJobs(bq_client)
+
+ dest_list = [(i, job) for i, job in enumerate(job_references)]
+
+ with TestPipeline('DirectRunner') as p:
+ references = beam.pvalue.AsList(p | 'job_ref' >> beam.Create(dest_list))
+ outputs = (p
+ | beam.Create([''])
+ | beam.ParDo(waiting_dofn, references))
+
+ assert_that(outputs,
+ equal_to(dest_list))
+
+ sleep_mock.assert_called_once()
+
+ @unittest.skipIf(sys.version_info[0] == 2,
+ 'Mock pickling problems in Py 2')
+ @mock.patch('time.sleep')
+ def test_one_job_failed_after_waiting(self, sleep_mock):
+ job_references = [bigquery_api.JobReference(),
+ bigquery_api.JobReference()]
+ job_references[0].projectId = 'project1'
+ job_references[0].jobId = 'jobId1'
+ job_references[1].projectId = 'project1'
+ job_references[1].jobId = 'jobId2'
+
+ job_1_waiting = mock.Mock()
+ job_1_waiting.status.state = 'RUNNING'
+ job_2_done = mock.Mock()
+ job_2_done.status.state = 'DONE'
+ job_2_done.status.errorResult = None
+
+ job_1_error = mock.Mock()
+ job_1_error.status.state = 'DONE'
Review comment:
yes, it returns DONE with an error report
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 334354)
Time Spent: 6h (was: 5h 50m)
> apache_beam.io.gcp.bigquery_write_it_test.BigQueryWriteIntegrationTests.test_big_query_write_new_types
> is flaky
> ---------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-8446
> URL: https://issues.apache.org/jira/browse/BEAM-8446
> Project: Beam
> Issue Type: New Feature
> Components: test-failures
> Reporter: Boyuan Zhang
> Assignee: Pablo Estrada
> Priority: Major
> Time Spent: 6h
> Remaining Estimate: 0h
>
> test_big_query_write_new_types appears to be flaky in
> beam_PostCommit_Python37 test suite.
> https://builds.apache.org/job/beam_PostCommit_Python37/733/
> https://builds.apache.org/job/beam_PostCommit_Python37/739/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
