Naireen commented on a change in pull request #16795:
URL: https://github.com/apache/beam/pull/16795#discussion_r809801173
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiLoads.java
##########
@@ -100,48 +102,23 @@ public WriteResult
expandTriggered(PCollection<KV<DestinationT, ElementT>> input
// Handle triggered, low-latency loads into BigQuery.
PCollection<KV<DestinationT, ElementT>> inputInGlobalWindow =
input.apply("rewindowIntoGlobal", Window.into(new GlobalWindows()));
-
- // First shard all the records.
- // TODO(reuvenlax): Add autosharding support so that users don't have to
pick a shard count.
- PCollection<KV<ShardedKey<DestinationT>, byte[]>> shardedRecords =
+ PCollection<KV<ShardedKey<DestinationT>, Iterable<byte[]>>> groupedRecords
=
inputInGlobalWindow
.apply("Convert", new
StorageApiConvertMessages<>(dynamicDestinations, bqServices))
.apply(
- "AddShard",
- ParDo.of(
- new DoFn<KV<DestinationT, byte[]>,
KV<ShardedKey<DestinationT>, byte[]>>() {
- int shardNumber;
-
- @Setup
- public void setup() {
- shardNumber =
ThreadLocalRandom.current().nextInt(numShards);
- }
-
- @ProcessElement
- public void processElement(
- @Element KV<DestinationT, byte[]> element,
- OutputReceiver<KV<ShardedKey<DestinationT>, byte[]>>
o) {
- DestinationT destination = element.getKey();
- ByteBuffer buffer = ByteBuffer.allocate(Integer.BYTES);
- buffer.putInt(++shardNumber % numShards);
- o.output(
- KV.of(ShardedKey.of(destination, buffer.array()),
element.getValue()));
- }
- }))
- .setCoder(KvCoder.of(ShardedKey.Coder.of(destinationCoder),
ByteArrayCoder.of()));
-
- PCollection<KV<ShardedKey<DestinationT>, Iterable<byte[]>>> groupedRecords
=
- shardedRecords.apply(
- "GroupIntoBatches",
- GroupIntoBatches.<ShardedKey<DestinationT>, byte[]>ofByteSize(
- MAX_BATCH_SIZE_BYTES, (byte[] e) -> (long) e.length)
- .withMaxBufferingDuration(triggeringFrequency));
+ "GroupIntoBatches",
+ GroupIntoBatches.<DestinationT, byte[]>ofByteSize(
+ MAX_BATCH_SIZE_BYTES, (byte[] e) -> (long) e.length)
+ .withMaxBufferingDuration(triggeringFrequency)
+ .withShardedKey());
Review comment:
Done, thanks for pointing that out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
