Robert Bradshaw created BEAM-9945:
-------------------------------------
Summary: Use consistent element count for progress counter.
Key: BEAM-9945
URL: https://issues.apache.org/jira/browse/BEAM-9945
Project: Beam
Issue Type: Improvement
Components: beam-model, sdk-go, sdk-java-harness, sdk-py-harness
Reporter: Robert Bradshaw
Fix For: 2.21.0
This influences how the SDK communicates progress and how splitting is
performed.
We currently inspect the element count of the output (which has inconsistent
definitions across SDKs, see BEAM-9934). Instead, we can move to using an
explicit, separate metric.
This can currently lead to incorrect progress reporting and in some cases even
a crash in the UW depending on the SDK, and makes it more difficult (but not
impossible) to fix element counts in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
