Daniel Mills created BEAM-10984:
-----------------------------------
Summary: Dataflow's UnboundedReaderIterator should support
overriding default bundle size targets
Key: BEAM-10984
URL: https://issues.apache.org/jira/browse/BEAM-10984
Project: Beam
Issue Type: Improvement
Components: runner-dataflow
Reporter: Daniel Mills
Assignee: Reza ardeshir rokni
Currently, the UnboundedReaderIterator will read until 10k elements have been
read or 10s have passed. This works for most pipelines, but is insufficient
for pipelines that either require very high throughput or require low latency.
We should make these values controllable via a PipelineOption, probably under
DataflowPipelineDebugOptions.
The constants are defined here:
https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/WorkerCustomSources.java#L757
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
