[
https://issues.apache.org/jira/browse/BEAM-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17001079#comment-17001079
]
Yichi Zhang commented on BEAM-8998:
-----------------------------------
throttling is introduced to avoid expensive scheduling problem mentioned in
BEAM-5791
> Avoid excessive bundle progress polling in Dataflow Runner
> ----------------------------------------------------------
>
> Key: BEAM-8998
> URL: https://issues.apache.org/jira/browse/BEAM-8998
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Yichi Zhang
> Priority: Major
>
> Dataflow Java runner uses 0.1 secs interval for polling bundle progress from
> SDK Harness, and use the result to decide whether data transfer should be
> throttled. This can potentially overload SDK Harness.
> We should try to come up with a way to avoid the throttling and lower the
> bundle progress request frequency significantly.
>
> Code reference:
> frequency setting:
> [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java#L296]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
