ajamato commented on a change in pull request #12070:
URL: https://github.com/apache/beam/pull/12070#discussion_r450478316
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamSource.java
##########
@@ -219,7 +232,15 @@ private synchronized boolean readNextRecord() throws
IOException {
}
fractionConsumedFromPreviousResponse =
fractionConsumedFromCurrentResponse;
- ReadRowsResponse currentResponse = responseIterator.next();
+ ReadRowsResponse currentResponse;
+ Stopwatch stopwatch = Stopwatch.createStarted();
Review comment:
It's probably fine to not use an option, since it occurs infrequently.
Though it may be a good idea to add an options as a precaution. Making it opt
in. We can turn it on by default later. IMO.
It may be a lot of effort to do this with the state sampler. So the approach
you have is probably best to start with.
Other approaches would be to run the job through some sort of profiler.
Though I am not sure how to do that today with any of our runners. Possibly it
could be setup with direct runner. I think that would help you to get plenty of
useful info on the problem as well. May be worth asking on the dev list about
that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
