tejasiyer-dev commented on code in PR #40543:
URL: https://github.com/apache/beam/pull/40543#discussion_r4607014555


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -1114,6 +1208,301 @@ public PCollection<FileResult<DestinationT>> 
expand(PCollection<UserT> input) {
     }
   }
 
+  /**
+   * Streaming write that sends large bundles straight to temporary files.
+   *
+   * <p>Small bundles, and records that would exceed the per-bundle writer 
limit, take the usual
+   * grouped path ({@link WriteShardedBundlesToTempFiles} when a shard count 
is set, or the {@link
+   * GroupIntoBatches} auto-sharded path otherwise). All files then drop their 
shard number and are
+   * grouped per destination, so finalization assigns shard numbers across 
both paths.
+   */
+  private class WriteShardedBundlesWithDirectWrite
+      extends PTransform<PCollection<UserT>, 
PCollection<List<FileResult<DestinationT>>>> {
+    private final Coder<DestinationT> destinationCoder;
+    private final Coder<FileResult<DestinationT>> fileResultCoder;
+    private final @Nullable PCollectionView<Integer> numShardsView;
+    private final long thresholdBytes;
+
+    private WriteShardedBundlesWithDirectWrite(
+        Coder<DestinationT> destinationCoder,
+        Coder<FileResult<DestinationT>> fileResultCoder,
+        @Nullable PCollectionView<Integer> numShardsView,
+        long thresholdBytes) {
+      this.destinationCoder = destinationCoder;
+      this.fileResultCoder = fileResultCoder;
+      this.numShardsView = numShardsView;
+      this.thresholdBytes = thresholdBytes;
+    }
+
+    @Override
+    public PCollection<List<FileResult<DestinationT>>> 
expand(PCollection<UserT> input) {
+      TupleTag<UserT> smallBundlesTag = new TupleTag<>("smallBundles");
+      TupleTag<UserT> largeBundlesTag = new TupleTag<>("largeBundles");
+      TupleTag<FileResult<DestinationT>> directFilesTag = new 
TupleTag<>("directFiles");
+      TupleTag<UserT> spilledRecordsTag = new TupleTag<>("spilledRecords");
+
+      // Evaluate the threshold per window, pane and destination, i.e. per 
output file.
+      PCollectionTuple lifted =
+          input.apply(
+              "LiftLargeBundles",
+              BundleLifter.of(smallBundlesTag, largeBundlesTag, 
thresholdBytes, input.getCoder())
+                  .withKeyFn(new 
DestinationKeyFn<>(getDynamicDestinations())));
+
+      PCollectionTuple routed =
+          lifted
+              .get(largeBundlesTag)
+              .setCoder(input.getCoder())
+              .apply(
+                  "DirectWriteLargeBundles",
+                  ParDo.of(new DirectWriteRecordsFn(spilledRecordsTag, 
input.getCoder()))
+                      .withSideInputs(getSideInputs())
+                      .withOutputTags(
+                          directFilesTag,
+                          TupleTagList.of(ImmutableList.of(spilledRecordsTag, 
BAD_RECORD_TAG))));
+      addErrorCollection(routed);
+
+      PCollection<UserT> smallBundles =
+          
PCollectionList.of(lifted.get(smallBundlesTag).setCoder(input.getCoder()))
+              .and(routed.get(spilledRecordsTag).setCoder(input.getCoder()))
+              .apply("FlattenSmallBundles", Flatten.pCollections())
+              .setCoder(input.getCoder());
+      PCollection<FileResult<DestinationT>> groupedFiles;
+      if (getComputeNumShards() != null || getNumShardsProvider() != null) {

Review Comment:
   Done. Direct write is now a pre-step in expand: it writes the large keys 
directly, and the rest goes through the existing sharded or autosharded write. 
Temp files from both are then flattened, grouped and finalized together. 
WriteShardedBundlesWithDirectWrite is removed, and pipelines without direct 
write are unchanged.
   
   One difference from your sketch is the grouped path keeps write + group in 
one step as is, and the direct write case has its own branch that writes, 
flattens in the direct files, then groups. I left the grouped step alone to 
avoid changing step names.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to