johnjcasey commented on code in PR #22347:
URL: https://github.com/apache/beam/pull/22347#discussion_r931504562
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWritesShardedRecords.java:
##########
@@ -553,6 +553,11 @@ public void onTimer(
OutputReceiver<KV<String, Operation>> o,
BoundedWindow window) {
// Stream is idle - clear it.
+ // Note: this is best effort. We are explicitly emiting a timestamp that
is before
+ // the default output timestamp, which means that in some cases (usually
when draining
+ // a pipeline) this finalize element will be dropped as late. This is
usually ok as
+ // BigQuery will eventually garbage collect the stream. We attempt to
finalize idle streams
+ // merely to remove the pressure of large numbers of orphaned streams
from BigQuery.
Review Comment:
what is the consequence of orphaned streams?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
