tejasiyer-dev commented on code in PR #40543:
URL: https://github.com/apache/beam/pull/40543#discussion_r4607169029
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -441,6 +456,26 @@ public WriteFiles<UserT, DestinationT, OutputT>
withBatchMaxBufferingDuration(
return
toBuilder().setBatchMaxBufferingDuration(batchMaxBufferingDuration).build();
}
+ /**
+ * Returns a new {@link WriteFiles} that fuses with preceding stages and
writes large input from a
+ * single bundle directly to temporary files.
+ *
+ * <p>Records are buffered per window, pane and destination until their
encoded size reaches
+ * {@code thresholdBytes}; those that reach it are written straight to
temporary files. Smaller
+ * groups keep the existing grouped path so output files stay large. Shard
numbers are assigned at
+ * finalization, which may override {@link #withNumShards(int)} (the number
of files per window
+ * can differ from the configured number of shards).
+ *
+ * <p>Direct writes are only used for unbounded inputs with non-merging
windows and discarding
Review Comment:
You're right it doesn't have to just be unbounded inputs. For bounded inputs
with a fixed shard count, it is preferable to go through a shuffle, so I
enabled direct write for that case too.
Without a fixed shard count, bounded inputs already write each bundle
directly with WriteUnshardedBundlesToTempFiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]