Abacn commented on code in PR #36750:
URL: https://github.com/apache/beam/pull/36750#discussion_r2527997431
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java:
##########
@@ -101,8 +141,16 @@ protected RestrictionTrackerObserverWithProgress(
}
@Override
- public synchronized Progress getProgress() {
- return ((HasProgress) delegate).getProgress();
+ public Progress getProgress() {
Review Comment:
> So the caller needs to know what restriction it is relative to. How
should the caller be sure which restriction the Progress is for?
The caller stacktrace, in SDK harness, is
```
org.apache.beam.fn.harness.control.ProcessBundleHandler#progress
org.apache.beam.fn.harness.control.ProcessBundleHandler#intermediateMonitoringData
org.apache.beam.fn.harness.control.BundleProgressReporter.InMemory#updateIntermediateMonitoringData
org.apache.beam.fn.harness.FnApiDoFnRunner.<anonymous
BundleProgressReporter, under case
PTransformTranslation.SPLITTABLE_PROCESS_SIZED_ELEMENTS_AND_RESTRICTIONS_URN>.updateIntermediateMonitoringData
org.apache.beam.fn.harness.FnApiDoFnRunner#getProgress
```
it has a member `currentTracker` that is a restriction tracker:
https://github.com/apache/beam/blob/c8d7ca02867ced1cb3cd0476dbcaed48cedd8884/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java#L750
The caller's restriction tracker is known (assigned in
processElementForWindowObservingSizedElementAndRestriction). The restriction
tracker's getProgress call simply calls delegate's getProgress:
https://github.com/apache/beam/blob/c8d7ca02867ced1cb3cd0476dbcaed48cedd8884/sdks/java/core/src/main/java/org/apache/beam/sdk/fn/splittabledofn/RestrictionTrackers.java#L105
where delegate in this case is a `BoundedSourceAsSDFRestrictionTracker`
It's getProgress call invokes `currentReader.getFractionConsumed()`. I
understand at this point restriction tracker relies on the underlying reader to
know the progress. After this change, when getProgress stucks, it reports the
last claimed progress, which I consider it is still a valid value.
Would these information sufficient to answer the concern here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
