Will-Lo commented on code in PR #3516:
URL: https://github.com/apache/gobblin/pull/3516#discussion_r909846027
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java:
##########
@@ -322,6 +326,19 @@ public AddSpecResponse onAddSpec(Spec addedSpec) {
return new AddSpecResponse<>(response);
}
+ // Check quota limits against run immediately flows or adhoc flows before
saving the schedule
+ if (!jobConfig.containsKey(ConfigurationKeys.JOB_SCHEDULE_KEY) ||
PropertiesUtils.getPropAsBoolean(jobConfig,
ConfigurationKeys.FLOW_RUN_IMMEDIATELY, "false")) {
Review Comment:
The scheduled + runImmediately flow would be rejected completely here since
the check is done right after compilation and before the job is added to the
scheduled map.
If the flow that exists is updated to be runImmediately and exceeds quota,
the runImmediate change would be rejected (and any other modifications done on
the flow). The flow would retain its previous state and then ran on its old
schedule.
If a flow that doesn't exist has runImmediately and exceeds quota, it is
rejected from the system entirely.
So the current model in the PR is that any changes that involve running a
flow immediately when the quota is exceeded is rejected. I think it's possible
to move to a future state where changes to a flow is not dependent on the
flow's execution schedule, but it would require a lot of refactoring. I think
it can be grouped with other changes we talked about, such as the faster
feedback when running adhoc flows. I think a possible future state would be
where users can save flows independently of whether or not the flow is
scheduled, and the execution schedule would be treated entirely separate
(similar to ADF and some other systems).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
