scwhittle commented on code in PR #39473:
URL: https://github.com/apache/beam/pull/39473#discussion_r3783250274
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java:
##########
Review Comment:
bytessinked currently is just the output messages. There is a todo to
measure state. I think that we should update the flushing of key state to add
in the state bytes so that we can stop merging keys if each key is modifying
state and it adds up. Then we will be less likely to need to truncate since
the sinkbytes limit is lower than the max limit.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingModeExecutionContext.java:
##########
@@ -717,6 +718,25 @@ private void validateCommitRequestSize() {
return;
}
+ // Look at budgetHandle instead of executedWorks because when intermediate
work items are
Review Comment:
Comment was a little unclear at first. Maybe something like the following?
// If this is a multi-key work item, then we need to retry all of the
individual work items without merging so that we can identify large commits to
truncate. We determine the work items that were part of the bundle by looking
at the budgethandle instead of executedWorks because validateCommitRequestSize
is called when transitioning and the handle has been updated but executedWorks
has not.
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/KeyGroupWorkQueue.java:
##########
@@ -193,6 +199,10 @@ private void unlinkNode(Node node) {
if (firstNode == keyGroupWorkList.tail) {
return null;
}
+
+ // MultiKeyBatchingDisabled items should not be in keyGroupWorkList
+ checkState(!Node.isMultiKeyBatchingDisabled(firstNode.task));
Review Comment:
do we need `Node.` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
