tejasiyer-dev commented on code in PR #40543:
URL: https://github.com/apache/beam/pull/40543#discussion_r4648311208
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -315,34 +339,42 @@ public WriteFiles<UserT, DestinationT, OutputT>
withNumShards(
* #withRunnerDeterminedSharding()}):</b>
*
* <table>
- * <tr>
- * <th>Configuration</th>
- * <th>Behavior when {@code maxNumWritersPerBundle} is reached</th>
- * <th>Worker Memory Consumption</th>
- * <th>Shuffle Cost</th>
- * <th>Output File Size / Count</th>
- * </tr>
- * <tr>
- * <td><b>Default (Spill to Shuffle)</b><br>{@code
maxNumWritersPerBundle > 0},<br>{@code evictWritersWhenFull = false}</td>
- * <td>Keeps first {@code N} writers open; spills remaining records to a
{@link GroupByKey} shuffle stage</td>
- * <td>Bounded ({@code <= N} buffers per bundle)</td>
- * <td>High if many records spill across shuffle</td>
- * <td>Fewer, larger files</td>
- * </tr>
- * <tr>
- * <td><b>LRU Writer Eviction</b><br>{@code maxNumWritersPerBundle >
0},<br>{@code evictWritersWhenFull = true}</td>
- * <td>Flushes and closes the least recently used (LRU) open writer to
open a new writer inline</td>
- * <td>Bounded ({@code <= N} buffers per bundle)</td>
- * <td>None (no shuffle stage for unwritten records)</td>
- * <td>May produce more/smaller files (minimal if input is ordered by
destination, high if random)</td>
- * </tr>
- * <tr>
- * <td><b>No Spilling</b><br>{@link #withNoSpilling()} ({@code
maxNumWritersPerBundle = -1})</td>
- * <td>Opens a new writer for every destination in the bundle without
limit</td>
- * <td>Unbounded (risk of OOM with many destinations)</td>
- * <td>None (no shuffle stage for unwritten records)</td>
- * <td>Fewer, larger files (1 file per destination per bundle)</td>
- * </tr>
+ * <tr>
Review Comment:
Done
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -137,21 +144,27 @@ public abstract class WriteFiles<UserT, DestinationT,
OutputT>
@Internal
public static final Class<? extends WriteFiles> CONCRETE_CLASS =
AutoValue_WriteFiles.class;
- // The maximum number of file writers to keep open in a single bundle at a
time, since file
- // writers default to 64mb buffers. This comes into play when writing
per-window or dynamic
- // destination files. The first 20 files from a single WriteFiles transform
will write files
- // inline in the transform. Anything beyond that might be spilled to shuffle
(default) or cause
+ // The maximum number of file writers to keep open in a single bundle at a
time,
+ // since file
+ // writers default to 64mb buffers. This comes into play when writing
per-window
+ // or dynamic
+ // destination files. The first 20 files from a single WriteFiles transform
will
+ // write files
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]