reuvenlax commented on code in PR #25723:
URL: https://github.com/apache/beam/pull/25723#discussion_r1131553718
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java:
##########
@@ -758,7 +800,24 @@ public void outputWithTimestamp(
BigQueryStorageApiInsertError output, org.joda.time.Instant
timestamp) {
context.output(failedRowsTag, output, timestamp,
GlobalWindow.INSTANCE);
}
- });
+ };
+ @Nullable OutputReceiver<TableRow> successfulRowsReceiver = null;
+ if (successfulRowsTag != null) {
+ successfulRowsReceiver =
+ new OutputReceiver<TableRow>() {
Review Comment:
hmm, the key point here is that the output elements need to have the same
timestamps as the input elements (because then you can just reapply the Window
function). This is a bit tricky for the exactly-once sink, as the
GroupIntoBatches loses the individual element timestamps (you just get a single
timestamp for the batch element).
I'm trying to think of a way to make this work without adding a lot of extra
cost or complexity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
