umustafi commented on code in PR #3656:
URL: https://github.com/apache/gobblin/pull/3656#discussion_r1131470132
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java:
##########
@@ -292,17 +355,23 @@ private void scheduleSpecsFromCatalog() {
Spec spec = batchOfSpecsIterator.next();
try {
addSpecHelperMethod(spec);
- urisLeftToSchedule.remove(spec.getUri());
+ totalAddSpecTime += this.eachCompleteAddSpecValue; // this is
updated by each call to onAddSpec
+ actualNumFlowsScheduled += 1;
} catch (Exception e) {
// If there is an uncaught error thrown during compilation, log it
and continue adding flows
_log.error("Could not schedule spec {} from flowCatalog due to ",
spec, e);
}
+ urisLeftToSchedule.remove(spec.getUri());
}
startOffset += this.loadSpecsBatchSize;
- // This count is used to ensure the average spec get time is calculated
accurately for the last batch which may be
- // smaller than the loadSpecsBatchSize
- averageGetSpecTimeValue = (batchGetEndTime - batchGetStartTime) /
batchOfSpecs.size();
+ totalGetTime += batchGetEndTime - batchGetStartTime;
+ // Don't skew the average get spec time value with the last batch that
may be very small
+ if (batchOfSpecs.size() == this.loadSpecsBatchSize) {
Review Comment:
These are good call outs, in practice only the last batch size I expect to
fall short of batchSize but it's better to be more robust in case the batch
size was set extremely large and covered all specs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
