arunpandianp commented on code in PR #40543:
URL: https://github.com/apache/beam/pull/40543#discussion_r4559492447
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java:
##########
@@ -579,6 +610,22 @@ public WriteFilesResult<DestinationT>
expand(PCollection<UserT> input) {
(getComputeNumShards() == null) ? null :
input.apply(getComputeNumShards());
boolean fixedSharding = getComputeNumShards() != null ||
getNumShardsProvider() != null;
+ Long directWriteThreshold = getDirectWriteByteThreshold();
+ if (directWriteThreshold != null
+ && getSideInputs().isEmpty()
+ && input.isBounded() == IsBounded.UNBOUNDED) {
Review Comment:
We should disable direct write when the window is merging like in
https://github.com/tejasiyer-dev/beam/blob/b1fe445521135fe623ef2d69ae1dd11361dbf1e4/sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java#L573
when directWriteThreshold is present and if we are not enabling it, should
we throw an error or fallback? When falling back adding a log saying direct
write is disabled with a reason would be nice to have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
