tejasiyer-dev commented on code in PR #40543:
URL: https://github.com/apache/beam/pull/40543#discussion_r4607169029


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -441,6 +456,26 @@ public WriteFiles<UserT, DestinationT, OutputT> 
withBatchMaxBufferingDuration(
     return 
toBuilder().setBatchMaxBufferingDuration(batchMaxBufferingDuration).build();
   }
 
+  /**
+   * Returns a new {@link WriteFiles} that fuses with preceding stages and 
writes large input from a
+   * single bundle directly to temporary files.
+   *
+   * <p>Records are buffered per window, pane and destination until their 
encoded size reaches
+   * {@code thresholdBytes}; those that reach it are written straight to 
temporary files. Smaller
+   * groups keep the existing grouped path so output files stay large. Shard 
numbers are assigned at
+   * finalization, which may override {@link #withNumShards(int)} (the number 
of files per window
+   * can differ from the configured number of shards).
+   *
+   * <p>Direct writes are only used for unbounded inputs with non-merging 
windows and discarding

Review Comment:
   You're right it doesn't have to just be unbounded inputs. For bounded inputs 
with a fixed shard count, it is preferable to go through a shuffle, so I 
enabled direct write for that case too.
   
   Without a fixed shard count, bounded inputs already write each bundle 
directly with WriteUnshardedBundlesToTempFiles.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to