tejasiyer-dev commented on code in PR #40543:
URL: https://github.com/apache/beam/pull/40543#discussion_r4606703748


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -1114,6 +1208,301 @@ public PCollection<FileResult<DestinationT>> 
expand(PCollection<UserT> input) {
     }
   }
 
+  /**
+   * Streaming write that sends large bundles straight to temporary files.
+   *
+   * <p>Small bundles, and records that would exceed the per-bundle writer 
limit, take the usual
+   * grouped path ({@link WriteShardedBundlesToTempFiles} when a shard count 
is set, or the {@link
+   * GroupIntoBatches} auto-sharded path otherwise). All files then drop their 
shard number and are
+   * grouped per destination, so finalization assigns shard numbers across 
both paths.
+   */
+  private class WriteShardedBundlesWithDirectWrite
+      extends PTransform<PCollection<UserT>, 
PCollection<List<FileResult<DestinationT>>>> {
+    private final Coder<DestinationT> destinationCoder;
+    private final Coder<FileResult<DestinationT>> fileResultCoder;
+    private final @Nullable PCollectionView<Integer> numShardsView;
+    private final long thresholdBytes;
+
+    private WriteShardedBundlesWithDirectWrite(
+        Coder<DestinationT> destinationCoder,
+        Coder<FileResult<DestinationT>> fileResultCoder,
+        @Nullable PCollectionView<Integer> numShardsView,
+        long thresholdBytes) {
+      this.destinationCoder = destinationCoder;
+      this.fileResultCoder = fileResultCoder;
+      this.numShardsView = numShardsView;
+      this.thresholdBytes = thresholdBytes;
+    }
+
+    @Override
+    public PCollection<List<FileResult<DestinationT>>> 
expand(PCollection<UserT> input) {
+      TupleTag<UserT> smallBundlesTag = new TupleTag<>("smallBundles");
+      TupleTag<UserT> largeBundlesTag = new TupleTag<>("largeBundles");
+      TupleTag<FileResult<DestinationT>> directFilesTag = new 
TupleTag<>("directFiles");
+      TupleTag<UserT> spilledRecordsTag = new TupleTag<>("spilledRecords");

Review Comment:
   Done. Added withMaxLiftedKeys(int) to BundleLifter. WriteFiles sets it to 
maxNumWritersPerBundle, so the spill tag and spill logic are gone.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to