akashorabek commented on code in PR #34078:
URL: https://github.com/apache/beam/pull/34078#discussion_r1973732882
##########
it/google-cloud-platform/src/test/java/org/apache/beam/it/gcp/bigquery/BigQueryIOST.java:
##########
@@ -296,11 +297,10 @@ private void
generateDataAndWrite(BigQueryIO.Write<byte[]> writeIO) throws IOExc
.withSchema(schema)
.withCustomGcsTempLocation(ValueProvider.StaticValueProvider.of(tempLocation)));
- String runnerV2Experiment = "use_runner_v2";
String experiments =
configuration.writeMethod.equals(STORAGE_API_AT_LEAST_ONCE_METHOD)
- ? runnerV2Experiment + ",streaming_mode_at_least_once"
- : runnerV2Experiment;
+ ? GcpOptions.STREAMING_ENGINE_EXPERIMENT +
",streaming_mode_at_least_once"
+ : GcpOptions.STREAMING_ENGINE_EXPERIMENT;
Review Comment:
In the test, we first count the number of written records inside the write
pipeline as a separate PTransform. Then, after the write process is complete,
we fetch the row count directly from the BigQuery table and compare these two
values. When using the exactly_once method (STORAGE_WRITE_API), the number of
records in the BigQuery table should not exceed the value obtained from the
write pipeline. However, for some reason, the value in the BigQuery table is
higher when using runner_v2. Because of this, the test fails at the assertion
of the number of rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
