scwhittle commented on code in PR #40543:
URL: https://github.com/apache/beam/pull/40543#discussion_r4642551529


##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -137,21 +144,27 @@ public abstract class WriteFiles<UserT, DestinationT, 
OutputT>
   @Internal
   public static final Class<? extends WriteFiles> CONCRETE_CLASS = 
AutoValue_WriteFiles.class;
 
-  // The maximum number of file writers to keep open in a single bundle at a 
time, since file
-  // writers default to 64mb buffers. This comes into play when writing 
per-window or dynamic
-  // destination files. The first 20 files from a single WriteFiles transform 
will write files
-  // inline in the transform. Anything beyond that might be spilled to shuffle 
(default) or cause
+  // The maximum number of file writers to keep open in a single bundle at a 
time,
+  // since file
+  // writers default to 64mb buffers. This comes into play when writing 
per-window
+  // or dynamic
+  // destination files. The first 20 files from a single WriteFiles transform 
will
+  // write files

Review Comment:
   can all the comment wrapping changes be reverted?



##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -315,34 +339,42 @@ public WriteFiles<UserT, DestinationT, OutputT> 
withNumShards(
    * #withRunnerDeterminedSharding()}):</b>
    *
    * <table>
-   *   <tr>
-   *     <th>Configuration</th>
-   *     <th>Behavior when {@code maxNumWritersPerBundle} is reached</th>
-   *     <th>Worker Memory Consumption</th>
-   *     <th>Shuffle Cost</th>
-   *     <th>Output File Size / Count</th>
-   *   </tr>
-   *   <tr>
-   *     <td><b>Default (Spill to Shuffle)</b><br>{@code 
maxNumWritersPerBundle > 0},<br>{@code evictWritersWhenFull = false}</td>
-   *     <td>Keeps first {@code N} writers open; spills remaining records to a 
{@link GroupByKey} shuffle stage</td>
-   *     <td>Bounded ({@code <= N} buffers per bundle)</td>
-   *     <td>High if many records spill across shuffle</td>
-   *     <td>Fewer, larger files</td>
-   *   </tr>
-   *   <tr>
-   *     <td><b>LRU Writer Eviction</b><br>{@code maxNumWritersPerBundle > 
0},<br>{@code evictWritersWhenFull = true}</td>
-   *     <td>Flushes and closes the least recently used (LRU) open writer to 
open a new writer inline</td>
-   *     <td>Bounded ({@code <= N} buffers per bundle)</td>
-   *     <td>None (no shuffle stage for unwritten records)</td>
-   *     <td>May produce more/smaller files (minimal if input is ordered by 
destination, high if random)</td>
-   *   </tr>
-   *   <tr>
-   *     <td><b>No Spilling</b><br>{@link #withNoSpilling()} ({@code 
maxNumWritersPerBundle = -1})</td>
-   *     <td>Opens a new writer for every destination in the bundle without 
limit</td>
-   *     <td>Unbounded (risk of OOM with many destinations)</td>
-   *     <td>None (no shuffle stage for unwritten records)</td>
-   *     <td>Fewer, larger files (1 file per destination per bundle)</td>
-   *   </tr>
+   * <tr>

Review Comment:
   also revert



##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -579,8 +633,141 @@ public WriteFilesResult<DestinationT> 
expand(PCollection<UserT> input) {
         (getComputeNumShards() == null) ? null : 
input.apply(getComputeNumShards());
 
     boolean fixedSharding = getComputeNumShards() != null || 
getNumShardsProvider() != null;
+    // Records that take the grouped path, and files written directly (if any).
+    PCollection<UserT> groupedInput = input;
+    @Nullable PCollection<FileResult<DestinationT>> directFiles = null;
+    Long directWriteThreshold = getDirectWriteByteThreshold();
+    if (directWriteThreshold != null) {
+      List<String> disabledReasons = new ArrayList<>();
+      // Direct writes are only supported for unbounded inputs as of now.

Review Comment:
   rm comment



##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -579,8 +633,141 @@ public WriteFilesResult<DestinationT> 
expand(PCollection<UserT> input) {
         (getComputeNumShards() == null) ? null : 
input.apply(getComputeNumShards());
 
     boolean fixedSharding = getComputeNumShards() != null || 
getNumShardsProvider() != null;
+    // Records that take the grouped path, and files written directly (if any).
+    PCollection<UserT> groupedInput = input;
+    @Nullable PCollection<FileResult<DestinationT>> directFiles = null;
+    Long directWriteThreshold = getDirectWriteByteThreshold();
+    if (directWriteThreshold != null) {
+      List<String> disabledReasons = new ArrayList<>();
+      // Direct writes are only supported for unbounded inputs as of now.
+      if (input.isBounded() == IsBounded.BOUNDED) {
+        disabledReasons.add("the input is bounded");
+      }
+      // WriteFiles allows merging windows when a fixed shard count is set 
(merged by
+      // GroupIntoShards's GroupByKey), whereas direct writes run before any
+      // GroupByKey and would
+      // use unmerged windows.
+      if (input.getWindowingStrategy().needsMerge()) {
+        disabledReasons.add("the windowing strategy uses merging windows");
+      }
+      // Accumulating panes re-emit earlier records in every firing, but 
direct writes
+      // write each
+      // record only once.
+      if (input.getWindowingStrategy().getMode() == 
AccumulationMode.ACCUMULATING_FIRED_PANES) {
+        disabledReasons.add("the windowing strategy accumulates fired panes");
+      }
+      if (disabledReasons.isEmpty()) {
+        TupleTag<KV<DestinationT, UserT>> keyedTag = new 
TupleTag<>("keyedRecords");
+        TupleTag<KV<DestinationT, UserT>> smallBundlesTag = new 
TupleTag<>("smallBundles");
+        TupleTag<KV<DestinationT, UserT>> largeBundlesTag = new 
TupleTag<>("largeBundles");
+        TupleTag<FileResult<DestinationT>> directFilesTag = new 
TupleTag<>("directFiles");
+
+        PCollectionTuple keyed =
+            input.apply(
+                "KeyByDestination",
+                ParDo.of(new KeyByDestinationFn(input.getCoder()))
+                    .withSideInputs(getSideInputs())
+                    .withOutputTags(keyedTag, 
TupleTagList.of(BAD_RECORD_TAG)));
+        addErrorCollection(keyed);
+        PCollection<KV<DestinationT, UserT>> keyedInput =
+            keyed.get(keyedTag).setCoder(KvCoder.of(destinationCoder, 
input.getCoder()));
+
+        // Evaluate the threshold per window, pane and destination, i.e. per 
output
+        // file. Lifting
+        // at most maxNumWritersPerBundle keys means the direct write never 
runs out of
+        // writers.
+        BundleLifter<KV<DestinationT, UserT>> lifter =
+            BundleLifter.of(
+                    smallBundlesTag,
+                    largeBundlesTag,
+                    directWriteThreshold,
+                    new ValueCoderSizer<>(input.getCoder()))
+                .withKeyFn(KV::getKey);
+        if (getMaxNumWritersPerBundle() > 0) {
+          lifter = lifter.withMaxLiftedKeys(getMaxNumWritersPerBundle());
+        }
+        PCollectionTuple lifted = keyedInput.apply("LiftLargeBundles", lifter);
+
+        PCollectionTuple written =
+            lifted
+                .get(largeBundlesTag)
+                .apply(
+                    "DirectWriteLargeBundles",
+                    ParDo.of(
+                            new WriteKeyedUnshardedTempFilesFn(
+                                destinationCoder,
+                                input.getCoder(),
+                                Metrics.counter(WriteFiles.class, 
"directWriteBundles")))
+                        .withSideInputs(getSideInputs())
+                        .withOutputTags(directFilesTag, 
TupleTagList.of(BAD_RECORD_TAG)));
+        addErrorCollection(written);
+        directFiles = written.get(directFilesTag).setCoder(fileResultCoder);
+        groupedInput =
+            lifted
+                .get(smallBundlesTag)
+                .apply("DropDestinationKey", Values.create())
+                .setCoder(input.getCoder());

Review Comment:
   don't think this setcoder is needed, seems like it would just propagate from 
values



##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -579,8 +633,141 @@ public WriteFilesResult<DestinationT> 
expand(PCollection<UserT> input) {
         (getComputeNumShards() == null) ? null : 
input.apply(getComputeNumShards());
 
     boolean fixedSharding = getComputeNumShards() != null || 
getNumShardsProvider() != null;
+    // Records that take the grouped path, and files written directly (if any).
+    PCollection<UserT> groupedInput = input;
+    @Nullable PCollection<FileResult<DestinationT>> directFiles = null;
+    Long directWriteThreshold = getDirectWriteByteThreshold();
+    if (directWriteThreshold != null) {
+      List<String> disabledReasons = new ArrayList<>();
+      // Direct writes are only supported for unbounded inputs as of now.
+      if (input.isBounded() == IsBounded.BOUNDED) {
+        disabledReasons.add("the input is bounded");
+      }
+      // WriteFiles allows merging windows when a fixed shard count is set 
(merged by
+      // GroupIntoShards's GroupByKey), whereas direct writes run before any
+      // GroupByKey and would
+      // use unmerged windows.
+      if (input.getWindowingStrategy().needsMerge()) {
+        disabledReasons.add("the windowing strategy uses merging windows");
+      }
+      // Accumulating panes re-emit earlier records in every firing, but 
direct writes
+      // write each
+      // record only once.
+      if (input.getWindowingStrategy().getMode() == 
AccumulationMode.ACCUMULATING_FIRED_PANES) {
+        disabledReasons.add("the windowing strategy accumulates fired panes");
+      }
+      if (disabledReasons.isEmpty()) {
+        TupleTag<KV<DestinationT, UserT>> keyedTag = new 
TupleTag<>("keyedRecords");
+        TupleTag<KV<DestinationT, UserT>> smallBundlesTag = new 
TupleTag<>("smallBundles");
+        TupleTag<KV<DestinationT, UserT>> largeBundlesTag = new 
TupleTag<>("largeBundles");
+        TupleTag<FileResult<DestinationT>> directFilesTag = new 
TupleTag<>("directFiles");
+
+        PCollectionTuple keyed =

Review Comment:
   Maybe it would look better in the UI if this was grouped under a transform 
that is "WriteFilesDirectly"? then that could be expanded to see the keybydest, 
lift, directwritelargebundles, dropkey but otherwise appears cleaner.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to