[
https://issues.apache.org/jira/browse/BEAM-7437?focusedWorklogId=267212&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-267212
]
ASF GitHub Bot logged work on BEAM-7437:
----------------------------------------
Author: ASF GitHub Bot
Created on: 26/Jun/19 01:49
Start Date: 26/Jun/19 01:49
Worklog Time Spent: 10m
Work Description: udim commented on pull request #8934: [BEAM-7437] Add
streaming flag to BQ streaming inserts IT test
URL: https://github.com/apache/beam/pull/8934#discussion_r297456280
##########
File path: sdks/python/apache_beam/io/gcp/tests/bigquery_matcher.py
##########
@@ -165,6 +169,33 @@ def describe_mismatch(self, pipeline_result,
mismatch_description):
.append_text(self.actual_data)
+class BigqueryFullResultStreamingMatcher(BigqueryFullResultMatcher):
+ """
+ Matcher that verifies Bigquery data with given query.
+
+ Fetch Bigquery data with given query, compare to the expected data.
+ This matcher polls BigQuery until the no. of records in BigQuery is
+ equal to the no. of records in expected data.
+ A timeout can be specified.
+ """
+
+ def __init__(self, project, query, data, timeout=MAX_TIMESTAMP):
Review comment:
Could you make the default timeout to be finite. Would 5/10/15 minutes work?
(the less the better - having post-commit tests hang until their 2 hour
deadline is not fun. It easier to debug where the test timed out if it raises a
TimeoutError and the stacktrace is printed)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 267212)
Time Spent: 3h (was: 2h 50m)
> Integration Test for BQ streaming inserts for streaming pipelines
> -----------------------------------------------------------------
>
> Key: BEAM-7437
> URL: https://issues.apache.org/jira/browse/BEAM-7437
> Project: Beam
> Issue Type: Test
> Components: io-python-gcp
> Affects Versions: 2.12.0
> Reporter: Tanay Tummalapalli
> Assignee: Tanay Tummalapalli
> Priority: Minor
> Labels: test
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Integration Test for BigQuery Sink using Streaming Inserts for streaming
> pipelines.
> Integration tests currently exist for batch pipelines, it can also be added
> for streaming pipelines using TestStream. This will be a precursor to the
> failing integration test to be added for [BEAM-6611|
> https://issues.apache.org/jira/browse/BEAM-6611].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
