reuvenlax commented on code in PR #38783:
URL: https://github.com/apache/beam/pull/38783#discussion_r3369601917
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiLoads.java:
##########
@@ -457,6 +466,52 @@ private void addErrorCollections(
}
}
+ /**
+ * A {@link DoFn} that applies a composite sharding key to incoming records
to optimize BigQuery
+ * Storage API throughput.
+ *
+ * <p>This transform manages the balance between connection count (resource
overhead) and
+ * processing parallelism by distributing data across {@code numShards}
buckets:
+ *
+ * <ul>
+ * <li><b>Data Affinity:</b> By using a composite key {@code KV<DestT,
Integer>}, this transform
Review Comment:
I don't understand this comment. <dest, 1> and <dest, 2> are two different
keys and can end up on different workers. The fact that they have the same
prefix generally doesn't mean anything for most runners.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
