Steve Koonce created BEAM-8554:
----------------------------------
Summary: Use WorkItemCommitRequest protobuf fields to signal that
a WorkItem needs to be broken up
Key: BEAM-8554
URL: https://issues.apache.org/jira/browse/BEAM-8554
Project: Beam
Issue Type: Improvement
Components: runner-dataflow
Reporter: Steve Koonce
+Background:+
When a WorkItemCommitRequest is generated that's bigger than the permitted size
(> ~180 MB), a KeyCommitTooLargeException is logged (_not thrown_) and the
request is still sent to the service. The service rejects the commit, but
breaks up input messages that were bundled together and adds them to new,
smaller work items that will later be pulled and re-tried - likely without
generating another commit that is too large.
When a WorkItemCommitRequest is generated that's too large to be sent back to
the service (> 2 GB), a KeyCommitTooLargeException is thrown and nothing is
sent back to the service.
+Proposed Improvement+
In both cases, prevent the doomed, large commit item from being sent back to
the service. Instead send flags in the commit request signaling that the
current work item led to a commit that is too large and the work item should be
broken up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
