stankiewicz commented on code in PR #38783:
URL: https://github.com/apache/beam/pull/38783#discussion_r3371885897
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiLoads.java:
##########
@@ -457,6 +466,52 @@ private void addErrorCollections(
}
}
+ /**
+ * A {@link DoFn} that applies a composite sharding key to incoming records
to optimize BigQuery
+ * Storage API throughput.
+ *
+ * <p>This transform manages the balance between connection count (resource
overhead) and
+ * processing parallelism by distributing data across {@code numShards}
buckets:
+ *
+ * <ul>
+ * <li><b>Data Affinity:</b> By using a composite key {@code KV<DestT,
Integer>}, this transform
Review Comment:
Reuven, I've updated the description as it wasn't accurate. Goal is to group
by destination+shard, not by prefix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
