[
https://issues.apache.org/jira/browse/GOBBLIN-1656?focusedWorklogId=778303&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-778303
]
ASF GitHub Bot logged work on GOBBLIN-1656:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 03/Jun/22 20:48
Start Date: 03/Jun/22 20:48
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #3516:
URL: https://github.com/apache/gobblin/pull/3516#discussion_r889337991
##########
gobblin-service/src/test/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobSchedulerTest.java:
##########
@@ -283,14 +301,61 @@ public boolean apply(Void input) {
Assert.assertEquals(schedulerService.getScheduler().getJobGroupNames().size(),
0);
}
+ @Test
+ public void testJobSchedulerAddFlowQuotaExceeded() throws Exception {
+ File specDir = Files.createTempDir();
+
+ Properties properties = new Properties();
+ properties.setProperty(FLOWSPEC_STORE_DIR_KEY, specDir.getAbsolutePath());
+ FlowCatalog flowCatalog = new
FlowCatalog(ConfigUtils.propertiesToConfig(properties));
+ ServiceBasedAppLauncher serviceLauncher = new
ServiceBasedAppLauncher(properties, "GaaSJobSchedulerTest");
+
+
+ serviceLauncher.addService(flowCatalog);
+ serviceLauncher.start();
+
+ FlowSpec flowSpec0 =
FlowCatalogTest.initFlowSpec(specDir.getAbsolutePath(), URI.create("spec0"),
"flowName0", "group1",
+
ConfigFactory.empty().withValue(ConfigurationKeys.FLOW_RUN_IMMEDIATELY,
ConfigValueFactory.fromAnyRef("true")));
+ FlowSpec flowSpec1 =
FlowCatalogTest.initFlowSpec(specDir.getAbsolutePath(), URI.create("spec1"),
"flowName1", "group1",
+
ConfigFactory.empty().withValue(ConfigurationKeys.FLOW_RUN_IMMEDIATELY,
ConfigValueFactory.fromAnyRef("true")));
+
+ Orchestrator mockOrchestrator = Mockito.mock(Orchestrator.class);
+ SpecCompiler mockSpecCompiler = Mockito.mock(SpecCompiler.class);
+ when(mockOrchestrator.getSpecCompiler()).thenReturn(mockSpecCompiler);
+ Dag<JobExecutionPlan> mockDag0 = this.buildDag(flowSpec0.getConfig(), "0");
+ Dag<JobExecutionPlan> mockDag1 = this.buildDag(flowSpec1.getConfig(), "1");
+ when(mockSpecCompiler.compileFlow(flowSpec0)).thenReturn(mockDag0);
+ when(mockSpecCompiler.compileFlow(flowSpec1)).thenReturn(mockDag1);
+
+ SchedulerService schedulerService = new SchedulerService(new Properties());
+ // Mock a GaaS scheduler.
+ GobblinServiceJobScheduler scheduler = new
GobblinServiceJobScheduler("testscheduler",
+ ConfigFactory.empty(), Optional.absent(), Optional.of(flowCatalog),
null, mockOrchestrator, schedulerService, Optional.of(new
UserQuotaManager(quotaConfig)), Optional.absent());
+
+ schedulerService.startAsync().awaitRunning();
+ scheduler.startUp();
+ scheduler.setActive(true);
+
+ scheduler.onAddSpec(flowSpec0); //Ignore the response for this request
+ AddSpecResponse<String> response1 = scheduler.onAddSpec(flowSpec1);
+ Assert.assertEquals(scheduler.scheduledFlowSpecs.size(), 1);
+
+ Assert.assertEquals(response1.getValue(),
"org.apache.gobblin.exception.QuotaExceededException: Quota exceeded for
flowgroup group1 on executor jobExecutor : quota=1, requests above quota=1\n");
+ // Second flow should not be added to scheduled flows since it was rejected
+ Assert.assertEquals(scheduler.scheduledFlowSpecs.size(), 1);
+ // set scheduler to be inactive and unschedule flows
+ scheduler.setActive(false);
+ Assert.assertEquals(scheduler.scheduledFlowSpecs.size(), 0);
Review Comment:
yeah it's cleanup
Issue Time Tracking
-------------------
Worklog Id: (was: 778303)
Time Spent: 1h 50m (was: 1h 40m)
> Return different Http Status on GaaS if Quota is Exceeded
> ---------------------------------------------------------
>
> Key: GOBBLIN-1656
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1656
> Project: Apache Gobblin
> Issue Type: New Feature
> Components: gobblin-service
> Reporter: William Lo
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> GaaS has quota limits for proxy users and flowGroups.
> When a user wants to create a flow that exceeds their specified quota, the
> flow should
> 1) Not be run
> 2) Return a http status code (i.e. 503) due to exceeding the resource. This
> allows clients to implement some wait and retry functionality
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
