umustafi commented on a change in pull request #3394:
URL: https://github.com/apache/gobblin/pull/3394#discussion_r707693945
##########
File path:
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/AbstractJobLauncher.java
##########
@@ -663,7 +663,7 @@ public void apply(JobListener jobListener, JobContext
jobContext)
@VisibleForTesting
public static long sumWorkUnitsSizes (WorkUnitStream workUnitStream) {
Collection<WorkUnit> workUnits =
JobLauncherUtils.flattenWorkUnits(workUnitStream.getMaterializedWorkUnitCollection());
- long totalSizeInBytes = workUnits.stream().mapToLong(wu ->
wu.getPropAsLong(ServiceConfigKeys.WORK_UNIT_SIZE)).sum();
+ long totalSizeInBytes = workUnits.stream().mapToLong(wu ->
wu.getPropAsLong(ServiceConfigKeys.WORK_UNIT_SIZE, 1)).sum();
Review comment:
We had a case where a user started a retention job with progress
reporting that failed since the Retention Source does not set a value for this
property. However, I also noticed that if all the workunits have size 0 and
progress reporting is turned on then there will be a divide by 0 error
[here](https://jarvis.corp.linkedin.com/codesearch/result/?name=TaskStateCollectorService.java&path=gobblin-elr%2Fgobblin-runtime%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fgobblin%2Fruntime&reponame=linkedin%2Fgobblin-elr#281).
I defaulted to using "1" for the size so progress reporting will default to
being based on # work units. Although retention jobs will occur very quickly
the progress will appear "100%" instead of "0%" or "Not calculated" if I
handled the error within the TaskStateCollectorService.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
