[
https://issues.apache.org/jira/browse/GOBBLIN-1656?focusedWorklogId=778301&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-778301
]
ASF GitHub Bot logged work on GOBBLIN-1656:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 03/Jun/22 20:48
Start Date: 03/Jun/22 20:48
Worklog Time Spent: 10m
Work Description: Will-Lo commented on code in PR #3516:
URL: https://github.com/apache/gobblin/pull/3516#discussion_r889337800
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java:
##########
@@ -360,16 +361,21 @@ public Map<String, AddSpecResponse> put(Spec spec,
boolean triggerListener) {
responseMap.put(entry.getKey().getName(),
entry.getValue().getResult());
}
}
+ AddSpecResponse<String> schedulerResponse =
responseMap.getOrDefault(ServiceConfigKeys.GOBBLIN_SERVICE_JOB_SCHEDULER_LISTENER_CLASS,
new AddSpecResponse<>(null));
- if (isCompileSuccessful(responseMap)) {
+ if (isCompileSuccessful(schedulerResponse.getValue())) {
synchronized (syncObject) {
try {
- if (!flowSpec.isExplain()) {
- long startTime = System.currentTimeMillis();
- specStore.addSpec(spec);
- metrics.updatePutSpecTime(startTime);
+ if
(schedulerResponse.getValue().contains(QuotaExceededException.class.getSimpleName()))
{
+ responseMap.put(ServiceConfigKeys.COMPILATION_SUCCESSFUL, new
AddSpecResponse<>(schedulerResponse.getValue()));
Review Comment:
I believe this occurs if there is no path found due to a number of reasons
i.e. `flow.conf` not matching or some flowgraph not matching problem. This in
turns causes the returned dag to be null (which is bad for users to understand
why their flow is failing) and hence the undetailed response, where compilation
errors are added to a returned flowspec output since the pathfinding algorithm
is BFS, and due to compilation errors there's a wide range of potential edges
that can be matched. The final output of this flag being false leads to this:
```
message.append(" Compilation errors encountered (Sorted by relevance):
");
FlowSpec.CompilationError[] errors =
flowSpec.getCompilationErrors().stream().distinct().toArray(FlowSpec.CompilationError[]::new);
Arrays.sort(errors, Comparator.comparingInt(c ->
((FlowSpec.CompilationError)c).errorPriority));
int errorId = 0;
for (FlowSpec.CompilationError error: errors) {
message.append("\n").append(String.format("ERROR[%s]",
errorId)).append(error.errorMessage);
errorId++;
}
```
I think there is definitely a better set of responses we can add here for
compilation scenarios but this is outside of the scope of the PR. I think we
need a better pathfinding algorithm along with a more structured response
format.
Issue Time Tracking
-------------------
Worklog Id: (was: 778301)
Time Spent: 1h 40m (was: 1.5h)
> Return different Http Status on GaaS if Quota is Exceeded
> ---------------------------------------------------------
>
> Key: GOBBLIN-1656
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1656
> Project: Apache Gobblin
> Issue Type: New Feature
> Components: gobblin-service
> Reporter: William Lo
> Assignee: Abhishek Tiwari
> Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> GaaS has quota limits for proxy users and flowGroups.
> When a user wants to create a flow that exceeds their specified quota, the
> flow should
> 1) Not be run
> 2) Return a http status code (i.e. 503) due to exceeding the resource. This
> allows clients to implement some wait and retry functionality
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
