Will-Lo commented on code in PR #3550:
URL: https://github.com/apache/gobblin/pull/3550#discussion_r965390221
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobScheduler.java:
##########
@@ -335,7 +335,9 @@ public AddSpecResponse onAddSpec(Spec addedSpec) {
if (quotaManager.isPresent()) {
// QuotaManager has idempotent checks for a dagNode, so this check
won't double add quotas for a flow in the DagManager
try {
+ // todo : we should probably check quota for all of the start nodes
quotaManager.get().checkQuota(dag.getNodes().get(0));
+ ((FlowSpec)
addedSpec).getConfigAsProperties().setProperty(ServiceConfigKeys.GOBBLIN_SERVICE_ADHOC_FLOW,
"true");
Review Comment:
Would it make sense if we added this in `createFlowSpecForConfig()` under
`FlowConfigLocalResourceHandler.java`? Seems like this property can be used in
a lot of areas outside of just this feature
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/flow/MultiHopFlowCompiler.java:
##########
@@ -281,6 +281,12 @@ public Dag<JobExecutionPlan> compileFlow(Spec spec) {
Instrumented.markMeter(flowCompilationSuccessFulMeter);
Instrumented.updateTimer(flowCompilationTimer, System.nanoTime() -
startTime, TimeUnit.NANOSECONDS);
+ if
(Boolean.parseBoolean(flowSpec.getConfigAsProperties().getProperty(ServiceConfigKeys.GOBBLIN_SERVICE_ADHOC_FLOW)))
{
+ // todo : we should probably set it on all of the start nodes
Review Comment:
I don't think it should be set for all the nodes because multihop flows
would break. When each hop is completed it would decrement the quota and
re-increment the quota for the next hop. If it's set for all nodes then this
scenario would need to be handled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
