[
https://issues.apache.org/jira/browse/BEAM-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yichi Zhang updated BEAM-8998:
------------------------------
Description:
Dataflow Java runner uses 0.1 secs interval for polling bundle progress from
SDK Harness, and use the result to decide whether data delivery should be
throttled. This can potentially overload SDK Harness.
We should try to come up with a way to avoid the throttling and lower the
bundle progress request frequency significantly.
Code reference:
frequency setting:
[https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java#L296]
was:
Dataflow Java runner uses 0.1 secs interval for polling bundle progress from
SDK Harness, and use the result to decide whether data delivery should be
throttled. This can potentially overload SDK Harness.
We should try to come up with a way to avoid the throttling and lower the
bundle progress request frequency significantly.
> Avoid excessive bundle progress polling in Dataflow Runner
> ----------------------------------------------------------
>
> Key: BEAM-8998
> URL: https://issues.apache.org/jira/browse/BEAM-8998
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Yichi Zhang
> Priority: Major
>
> Dataflow Java runner uses 0.1 secs interval for polling bundle progress from
> SDK Harness, and use the result to decide whether data delivery should be
> throttled. This can potentially overload SDK Harness.
> We should try to come up with a way to avoid the throttling and lower the
> bundle progress request frequency significantly.
>
> Code reference:
> frequency setting:
> [https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/BeamFnMapTaskExecutor.java#L296]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
