[
https://issues.apache.org/jira/browse/BEAM-7721?focusedWorklogId=293700&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-293700
]
ASF GitHub Bot logged work on BEAM-7721:
----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Aug/19 07:55
Start Date: 13/Aug/19 07:55
Worklog Time Spent: 10m
Work Description: mwalenia commented on pull request #9295: [BEAM-7721]
Add write test to BigQueryIOIT
URL: https://github.com/apache/beam/pull/9295#discussion_r313262635
##########
File path:
sdks/java/io/bigquery-io-tests/src/test/java/org/apache/beam/sdk/bigqueryiotests/BigQueryIOIT.java
##########
@@ -113,33 +120,34 @@ public static void setup() throws IOException {
SyntheticOptions.fromJsonString(options.getSourceOptions(),
SyntheticSourceOptions.class);
bigQueryMetricsDataset = options.getBigQueryDataset();
bigQueryMetricsTable = options.getBigQueryTable();
- bigQuerySourceDataset = options.getBigQuerySourceDataset();
- bigQuerySourceTable = options.getBigQuerySourceTable();
+ bigQueryTestDataset = options.getBigQueryTestDataset();
+ bigQueryTestTable = options.getBigQueryTestTable();
+ useStreamingWrites = options.getUseStreamingWrites();
BigQueryOptions bigQueryOptions = BigQueryOptions.newBuilder().build();
tableQualifier =
String.format(
- "%s:%s.%s", bigQueryOptions.getProjectId(), bigQuerySourceDataset,
bigQuerySourceTable);
- if (!checkForSourceTable()) {
- createAndFillSourceTable();
- }
+ "%s:%s.%s", bigQueryOptions.getProjectId(), bigQueryTestDataset,
bigQueryTestTable);
}
@AfterClass
public static void tearDown() {
BigQueryOptions options = BigQueryOptions.newBuilder().build();
BigQuery client = options.getService();
- TableId tableId =
- TableId.of(options.getProjectId(), bigQuerySourceDataset,
bigQuerySourceTable);
+ TableId tableId = TableId.of(options.getProjectId(), bigQueryTestDataset,
bigQueryTestTable);
client.delete(tableId);
}
- private static void createAndFillSourceTable() {
-
- Pipeline pipeline = Pipeline.create();
+ private void testWrite() {
+ Pipeline pipeline = Pipeline.create(options);
+ BigQueryIO.Write.Method method =
+ useStreamingWrites
+ ? BigQueryIO.Write.Method.STREAMING_INSERTS
+ : BigQueryIO.Write.Method.FILE_LOADS;
pipeline
.apply("Read from source", Read.from(new
SyntheticBoundedSource(sourceOptions)))
.apply("Map records", ParDo.of(new MapKVToV()))
+ .apply("Gather time", ParDo.of(new TimeMonitor<>(NAMESPACE,
WRITE_TIME_METRIC_NAME)))
Review comment:
An oversight, I fixed it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 293700)
Time Spent: 8h 10m (was: 8h)
> Measure throughput in BigQueryIO read in Java
> ---------------------------------------------
>
> Key: BEAM-7721
> URL: https://issues.apache.org/jira/browse/BEAM-7721
> Project: Beam
> Issue Type: Improvement
> Components: testing
> Reporter: Michal Walenia
> Assignee: Michal Walenia
> Priority: Major
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> Collect bytes/time metric in BQ read tests in Java SDK
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
