ahmedabu98 commented on code in PR #25392:
URL: https://github.com/apache/beam/pull/25392#discussion_r1109125140
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageStreamBundleSource.java:
##########
@@ -160,8 +161,9 @@ public long getEstimatedSizeBytes(PipelineOptions options) {
@Override
public List<? extends OffsetBasedSource<T>> split(
long desiredBundleSizeBytes, PipelineOptions options) {
- // A stream source can't be split without reading from it due to
server-side liquid sharding.
- // TODO: Implement dynamic work rebalancing.
+ // This method is only called for initial splits. Since this class will
always be a child source
+ // of BigQueryStorageSourceBase, all splits here will be handled by
`splitAtFraction()`. As a
+ // result, this is a no-op.
Review Comment:
FYI because it was mentioned previously as `TODO: Implement dynamic work
rebalancing`, splitAtFraction() _is_ dynamic work rebalancing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
