igorbernstein2 commented on code in PR #17823:
URL: https://github.com/apache/beam/pull/17823#discussion_r899279150
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java:
##########
@@ -844,6 +874,22 @@ public void processElement(ProcessContext c, BoundedWindow
window) throws Except
failures.add(new BigtableWriteException(c.element(),
exception));
}
});
+ if (config.getDataflowThrottleReporting()) {
+ long delta = 0;
+ ResourceLimiterStats stats =
+ ResourceLimiterStats.getInstance(
+ new BigtableInstanceName(
+ config.getProjectId().get(),
config.getInstanceId().get()));
Review Comment:
Please cache this in the constructor
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java:
##########
@@ -739,6 +754,16 @@ public Write withEmulator(String emulatorHost) {
return
toBuilder().setBigtableConfig(config.withEmulator(emulatorHost)).build();
}
+ /* This is an experimental feature that may get changed in the future
+ *
+ * Returns a new {@link BigtableIO.Write} that will report amount of time
throttling to Dataflow
+ */
+ @Experimental
+ public Write withDataflowThrottleReporting() {
Review Comment:
I would add a bool param here. It will allow you to transition this to be
opt out later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
