scwhittle commented on code in PR #40543:
URL: https://github.com/apache/beam/pull/40543#discussion_r4642551529
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -137,21 +144,27 @@ public abstract class WriteFiles<UserT, DestinationT,
OutputT>
@Internal
public static final Class<? extends WriteFiles> CONCRETE_CLASS =
AutoValue_WriteFiles.class;
- // The maximum number of file writers to keep open in a single bundle at a
time, since file
- // writers default to 64mb buffers. This comes into play when writing
per-window or dynamic
- // destination files. The first 20 files from a single WriteFiles transform
will write files
- // inline in the transform. Anything beyond that might be spilled to shuffle
(default) or cause
+ // The maximum number of file writers to keep open in a single bundle at a
time,
+ // since file
+ // writers default to 64mb buffers. This comes into play when writing
per-window
+ // or dynamic
+ // destination files. The first 20 files from a single WriteFiles transform
will
+ // write files
Review Comment:
can all the comment wrapping changes be reverted?
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -315,34 +339,42 @@ public WriteFiles<UserT, DestinationT, OutputT>
withNumShards(
* #withRunnerDeterminedSharding()}):</b>
*
* <table>
- * <tr>
- * <th>Configuration</th>
- * <th>Behavior when {@code maxNumWritersPerBundle} is reached</th>
- * <th>Worker Memory Consumption</th>
- * <th>Shuffle Cost</th>
- * <th>Output File Size / Count</th>
- * </tr>
- * <tr>
- * <td><b>Default (Spill to Shuffle)</b><br>{@code
maxNumWritersPerBundle > 0},<br>{@code evictWritersWhenFull = false}</td>
- * <td>Keeps first {@code N} writers open; spills remaining records to a
{@link GroupByKey} shuffle stage</td>
- * <td>Bounded ({@code <= N} buffers per bundle)</td>
- * <td>High if many records spill across shuffle</td>
- * <td>Fewer, larger files</td>
- * </tr>
- * <tr>
- * <td><b>LRU Writer Eviction</b><br>{@code maxNumWritersPerBundle >
0},<br>{@code evictWritersWhenFull = true}</td>
- * <td>Flushes and closes the least recently used (LRU) open writer to
open a new writer inline</td>
- * <td>Bounded ({@code <= N} buffers per bundle)</td>
- * <td>None (no shuffle stage for unwritten records)</td>
- * <td>May produce more/smaller files (minimal if input is ordered by
destination, high if random)</td>
- * </tr>
- * <tr>
- * <td><b>No Spilling</b><br>{@link #withNoSpilling()} ({@code
maxNumWritersPerBundle = -1})</td>
- * <td>Opens a new writer for every destination in the bundle without
limit</td>
- * <td>Unbounded (risk of OOM with many destinations)</td>
- * <td>None (no shuffle stage for unwritten records)</td>
- * <td>Fewer, larger files (1 file per destination per bundle)</td>
- * </tr>
+ * <tr>
Review Comment:
also revert
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -579,8 +633,141 @@ public WriteFilesResult<DestinationT>
expand(PCollection<UserT> input) {
(getComputeNumShards() == null) ? null :
input.apply(getComputeNumShards());
boolean fixedSharding = getComputeNumShards() != null ||
getNumShardsProvider() != null;
+ // Records that take the grouped path, and files written directly (if any).
+ PCollection<UserT> groupedInput = input;
+ @Nullable PCollection<FileResult<DestinationT>> directFiles = null;
+ Long directWriteThreshold = getDirectWriteByteThreshold();
+ if (directWriteThreshold != null) {
+ List<String> disabledReasons = new ArrayList<>();
+ // Direct writes are only supported for unbounded inputs as of now.
Review Comment:
rm comment
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -579,8 +633,141 @@ public WriteFilesResult<DestinationT>
expand(PCollection<UserT> input) {
(getComputeNumShards() == null) ? null :
input.apply(getComputeNumShards());
boolean fixedSharding = getComputeNumShards() != null ||
getNumShardsProvider() != null;
+ // Records that take the grouped path, and files written directly (if any).
+ PCollection<UserT> groupedInput = input;
+ @Nullable PCollection<FileResult<DestinationT>> directFiles = null;
+ Long directWriteThreshold = getDirectWriteByteThreshold();
+ if (directWriteThreshold != null) {
+ List<String> disabledReasons = new ArrayList<>();
+ // Direct writes are only supported for unbounded inputs as of now.
+ if (input.isBounded() == IsBounded.BOUNDED) {
+ disabledReasons.add("the input is bounded");
+ }
+ // WriteFiles allows merging windows when a fixed shard count is set
(merged by
+ // GroupIntoShards's GroupByKey), whereas direct writes run before any
+ // GroupByKey and would
+ // use unmerged windows.
+ if (input.getWindowingStrategy().needsMerge()) {
+ disabledReasons.add("the windowing strategy uses merging windows");
+ }
+ // Accumulating panes re-emit earlier records in every firing, but
direct writes
+ // write each
+ // record only once.
+ if (input.getWindowingStrategy().getMode() ==
AccumulationMode.ACCUMULATING_FIRED_PANES) {
+ disabledReasons.add("the windowing strategy accumulates fired panes");
+ }
+ if (disabledReasons.isEmpty()) {
+ TupleTag<KV<DestinationT, UserT>> keyedTag = new
TupleTag<>("keyedRecords");
+ TupleTag<KV<DestinationT, UserT>> smallBundlesTag = new
TupleTag<>("smallBundles");
+ TupleTag<KV<DestinationT, UserT>> largeBundlesTag = new
TupleTag<>("largeBundles");
+ TupleTag<FileResult<DestinationT>> directFilesTag = new
TupleTag<>("directFiles");
+
+ PCollectionTuple keyed =
+ input.apply(
+ "KeyByDestination",
+ ParDo.of(new KeyByDestinationFn(input.getCoder()))
+ .withSideInputs(getSideInputs())
+ .withOutputTags(keyedTag,
TupleTagList.of(BAD_RECORD_TAG)));
+ addErrorCollection(keyed);
+ PCollection<KV<DestinationT, UserT>> keyedInput =
+ keyed.get(keyedTag).setCoder(KvCoder.of(destinationCoder,
input.getCoder()));
+
+ // Evaluate the threshold per window, pane and destination, i.e. per
output
+ // file. Lifting
+ // at most maxNumWritersPerBundle keys means the direct write never
runs out of
+ // writers.
+ BundleLifter<KV<DestinationT, UserT>> lifter =
+ BundleLifter.of(
+ smallBundlesTag,
+ largeBundlesTag,
+ directWriteThreshold,
+ new ValueCoderSizer<>(input.getCoder()))
+ .withKeyFn(KV::getKey);
+ if (getMaxNumWritersPerBundle() > 0) {
+ lifter = lifter.withMaxLiftedKeys(getMaxNumWritersPerBundle());
+ }
+ PCollectionTuple lifted = keyedInput.apply("LiftLargeBundles", lifter);
+
+ PCollectionTuple written =
+ lifted
+ .get(largeBundlesTag)
+ .apply(
+ "DirectWriteLargeBundles",
+ ParDo.of(
+ new WriteKeyedUnshardedTempFilesFn(
+ destinationCoder,
+ input.getCoder(),
+ Metrics.counter(WriteFiles.class,
"directWriteBundles")))
+ .withSideInputs(getSideInputs())
+ .withOutputTags(directFilesTag,
TupleTagList.of(BAD_RECORD_TAG)));
+ addErrorCollection(written);
+ directFiles = written.get(directFilesTag).setCoder(fileResultCoder);
+ groupedInput =
+ lifted
+ .get(smallBundlesTag)
+ .apply("DropDestinationKey", Values.create())
+ .setCoder(input.getCoder());
Review Comment:
don't think this setcoder is needed, seems like it would just propagate from
values
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -579,8 +633,141 @@ public WriteFilesResult<DestinationT>
expand(PCollection<UserT> input) {
(getComputeNumShards() == null) ? null :
input.apply(getComputeNumShards());
boolean fixedSharding = getComputeNumShards() != null ||
getNumShardsProvider() != null;
+ // Records that take the grouped path, and files written directly (if any).
+ PCollection<UserT> groupedInput = input;
+ @Nullable PCollection<FileResult<DestinationT>> directFiles = null;
+ Long directWriteThreshold = getDirectWriteByteThreshold();
+ if (directWriteThreshold != null) {
+ List<String> disabledReasons = new ArrayList<>();
+ // Direct writes are only supported for unbounded inputs as of now.
+ if (input.isBounded() == IsBounded.BOUNDED) {
+ disabledReasons.add("the input is bounded");
+ }
+ // WriteFiles allows merging windows when a fixed shard count is set
(merged by
+ // GroupIntoShards's GroupByKey), whereas direct writes run before any
+ // GroupByKey and would
+ // use unmerged windows.
+ if (input.getWindowingStrategy().needsMerge()) {
+ disabledReasons.add("the windowing strategy uses merging windows");
+ }
+ // Accumulating panes re-emit earlier records in every firing, but
direct writes
+ // write each
+ // record only once.
+ if (input.getWindowingStrategy().getMode() ==
AccumulationMode.ACCUMULATING_FIRED_PANES) {
+ disabledReasons.add("the windowing strategy accumulates fired panes");
+ }
+ if (disabledReasons.isEmpty()) {
+ TupleTag<KV<DestinationT, UserT>> keyedTag = new
TupleTag<>("keyedRecords");
+ TupleTag<KV<DestinationT, UserT>> smallBundlesTag = new
TupleTag<>("smallBundles");
+ TupleTag<KV<DestinationT, UserT>> largeBundlesTag = new
TupleTag<>("largeBundles");
+ TupleTag<FileResult<DestinationT>> directFilesTag = new
TupleTag<>("directFiles");
+
+ PCollectionTuple keyed =
Review Comment:
Maybe it would look better in the UI if this was grouped under a transform
that is "WriteFilesDirectly"? then that could be expanded to see the keybydest,
lift, directwritelargebundles, dropkey but otherwise appears cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]