scwhittle commented on a change in pull request #11289: [BEAM-9660]: Add an
explicit check for integer overflow.
URL: https://github.com/apache/beam/pull/11289#discussion_r402466421
##########
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/StreamingDataflowWorker.java
##########
@@ -1372,11 +1372,13 @@ private void process(
WorkItemCommitRequest commitRequest = outputBuilder.build();
int byteLimit = maxWorkItemCommitBytes;
int commitSize = commitRequest.getSerializedSize();
- int estimatedCommitSize = commitSize < 0 ? Integer.MAX_VALUE :
commitSize;
-
- // Detect overflow of integer serialized size or if the byte limit was
exceeded.
- windmillMaxObservedWorkItemCommitBytes.addValue(estimatedCommitSize);
- if (estimatedCommitSize > byteLimit) {
+ // Detect overflow of integer serialized size.
+ if (commitSize < 0) {
+ throw KeyCommitTooLargeException.causedBy(computationId, byteLimit,
commitRequest);
Review comment:
We still would prefer to send a truncation commit (from
buildWorkItemTruncationRequest). That reports the commit as too large but
sends the necessary info to the backend to attempt to truncate the work item
for reprocessing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
