[
https://issues.apache.org/jira/browse/GOBBLIN-1650?focusedWorklogId=774261&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-774261
]
ASF GitHub Bot logged work on GOBBLIN-1650:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 24/May/22 22:12
Start Date: 24/May/22 22:12
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #3511:
URL: https://github.com/apache/gobblin/pull/3511#discussion_r880993561
##########
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/UserQuotaManager.java:
##########
@@ -159,6 +179,9 @@ public boolean releaseQuota(Dag.DagNode<JobExecutionPlan>
dagNode) {
String proxyUserKey = DagManagerUtils.getUserQuotaKey(proxyUser,
dagNode);
decrementQuotaUsage(proxyUserToJobCount, proxyUserKey);
}
+ String flowGroup =
ConfigUtils.getString(dagNode.getValue().getJobSpec().getConfig(),
+ ConfigurationKeys.FLOW_GROUP_KEY, "");
+ decrementQuotaUsage(flowGroupToJobCount,
DagManagerUtils.getFlowGroupQuotaKey(flowGroup, dagNode));
String serializedRequesters =
DagManagerUtils.getSerializedRequesterList(dagNode);
if (serializedRequesters != null) {
Review Comment:
I have a test from a previous PR in`DagManagerTest` that also tested for
releaseQuota API in terms of the larger flow lifecycle.
`releaseQuota` is only called when a flow is completed so besides that the
only scenario where the count decrements is when an increment fails due to
exceeding the quota, but this is tested in the below test
`testUserAndFlowGroupQuotaMultipleUsersAdd` since a failed increment is
followed with a decrement then increment
Issue Time Tracking
-------------------
Worklog Id: (was: 774261)
Time Spent: 50m (was: 40m)
> Allow GaaS to enforce quotas by flowgroup
> -----------------------------------------
>
> Key: GOBBLIN-1650
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1650
> Project: Apache Gobblin
> Issue Type: New Feature
> Reporter: William Lo
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Certain GaaS flows can have a large number of flows that can cause
> instability on dependent services. We want to be able to control the
> throughput of flows from a flowgroup granularity on top of a user
> granularity. We keep the quota configuration separate as there can be many
> users submitting flows to the same flowgroup.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
