Abacn commented on code in PR #36750:
URL: https://github.com/apache/beam/pull/36750#discussion_r2518785151
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java:
##########
@@ -101,8 +141,16 @@ protected RestrictionTrackerObserverWithProgress(
}
@Override
- public synchronized Progress getProgress() {
- return ((HasProgress) delegate).getProgress();
+ public Progress getProgress() {
Review Comment:
There were cases of inaccurate progress confused runner: prior to #26352
boundedsource as restriction tracker (the scenario of Dataflow runner v2
evaluating boundedsource) always reports zero progress, and the effect was
slowness and runner not splitting sources, from the PR description. At least if
the stale progress is zero it should be minimum concern (happened before).
If the stale progress is a positive number, this is similar to long running
DoFn.process we often observes (in case of Dataflow, there is a straggler
alert) but the pipeline can still work as intended. Let me write a pipeline and
test this scenario, would a stale positive prgress make runner eagerly issue
trySplit, or other unexpected behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
