tejasiyer-dev commented on code in PR #40543:
URL: https://github.com/apache/beam/pull/40543#discussion_r4607014555
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -1114,6 +1208,301 @@ public PCollection<FileResult<DestinationT>>
expand(PCollection<UserT> input) {
}
}
+ /**
+ * Streaming write that sends large bundles straight to temporary files.
+ *
+ * <p>Small bundles, and records that would exceed the per-bundle writer
limit, take the usual
+ * grouped path ({@link WriteShardedBundlesToTempFiles} when a shard count
is set, or the {@link
+ * GroupIntoBatches} auto-sharded path otherwise). All files then drop their
shard number and are
+ * grouped per destination, so finalization assigns shard numbers across
both paths.
+ */
+ private class WriteShardedBundlesWithDirectWrite
+ extends PTransform<PCollection<UserT>,
PCollection<List<FileResult<DestinationT>>>> {
+ private final Coder<DestinationT> destinationCoder;
+ private final Coder<FileResult<DestinationT>> fileResultCoder;
+ private final @Nullable PCollectionView<Integer> numShardsView;
+ private final long thresholdBytes;
+
+ private WriteShardedBundlesWithDirectWrite(
+ Coder<DestinationT> destinationCoder,
+ Coder<FileResult<DestinationT>> fileResultCoder,
+ @Nullable PCollectionView<Integer> numShardsView,
+ long thresholdBytes) {
+ this.destinationCoder = destinationCoder;
+ this.fileResultCoder = fileResultCoder;
+ this.numShardsView = numShardsView;
+ this.thresholdBytes = thresholdBytes;
+ }
+
+ @Override
+ public PCollection<List<FileResult<DestinationT>>>
expand(PCollection<UserT> input) {
+ TupleTag<UserT> smallBundlesTag = new TupleTag<>("smallBundles");
+ TupleTag<UserT> largeBundlesTag = new TupleTag<>("largeBundles");
+ TupleTag<FileResult<DestinationT>> directFilesTag = new
TupleTag<>("directFiles");
+ TupleTag<UserT> spilledRecordsTag = new TupleTag<>("spilledRecords");
+
+ // Evaluate the threshold per window, pane and destination, i.e. per
output file.
+ PCollectionTuple lifted =
+ input.apply(
+ "LiftLargeBundles",
+ BundleLifter.of(smallBundlesTag, largeBundlesTag,
thresholdBytes, input.getCoder())
+ .withKeyFn(new
DestinationKeyFn<>(getDynamicDestinations())));
+
+ PCollectionTuple routed =
+ lifted
+ .get(largeBundlesTag)
+ .setCoder(input.getCoder())
+ .apply(
+ "DirectWriteLargeBundles",
+ ParDo.of(new DirectWriteRecordsFn(spilledRecordsTag,
input.getCoder()))
+ .withSideInputs(getSideInputs())
+ .withOutputTags(
+ directFilesTag,
+ TupleTagList.of(ImmutableList.of(spilledRecordsTag,
BAD_RECORD_TAG))));
+ addErrorCollection(routed);
+
+ PCollection<UserT> smallBundles =
+
PCollectionList.of(lifted.get(smallBundlesTag).setCoder(input.getCoder()))
+ .and(routed.get(spilledRecordsTag).setCoder(input.getCoder()))
+ .apply("FlattenSmallBundles", Flatten.pCollections())
+ .setCoder(input.getCoder());
+ PCollection<FileResult<DestinationT>> groupedFiles;
+ if (getComputeNumShards() != null || getNumShardsProvider() != null) {
Review Comment:
Done. Direct write is now a pre-step in expand: it writes the large keys
directly, and the rest goes through the existing sharded or autosharded write.
Temp files from both are then flattened, grouped and finalized together.
WriteShardedBundlesWithDirectWrite is removed, and pipelines without direct
write are unchanged.
One difference from your sketch is the grouped path keeps write + group in
one step as is, and the direct write case has its own branch that writes,
flattens in the direct files, then groups. I left the grouped step alone to
avoid changing step names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]