Sam Whittle created BEAM-11910:
----------------------------------
Summary: Increase subsequent page size for bags after the first
Key: BEAM-11910
URL: https://issues.apache.org/jira/browse/BEAM-11910
Project: Beam
Issue Type: Bug
Components: runner-dataflow
Reporter: Sam Whittle
Currently the page size of bags requested from the streaming dataflow backend
is always 8MB. In pipelines with large bags this can limit throughput as it
results in more round-trips to the backend. In particular with Streaming
Engine this is noticable due to increased latency.
I propose using 8MB for the first bag fetch and then doubling the limit for
subsequent paginations
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
