Will-Lo commented on code in PR #3516:
URL: https://github.com/apache/gobblin/pull/3516#discussion_r889337800
##########
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java:
##########
@@ -360,16 +361,21 @@ public Map<String, AddSpecResponse> put(Spec spec,
boolean triggerListener) {
responseMap.put(entry.getKey().getName(),
entry.getValue().getResult());
}
}
+ AddSpecResponse<String> schedulerResponse =
responseMap.getOrDefault(ServiceConfigKeys.GOBBLIN_SERVICE_JOB_SCHEDULER_LISTENER_CLASS,
new AddSpecResponse<>(null));
- if (isCompileSuccessful(responseMap)) {
+ if (isCompileSuccessful(schedulerResponse.getValue())) {
synchronized (syncObject) {
try {
- if (!flowSpec.isExplain()) {
- long startTime = System.currentTimeMillis();
- specStore.addSpec(spec);
- metrics.updatePutSpecTime(startTime);
+ if
(schedulerResponse.getValue().contains(QuotaExceededException.class.getSimpleName()))
{
+ responseMap.put(ServiceConfigKeys.COMPILATION_SUCCESSFUL, new
AddSpecResponse<>(schedulerResponse.getValue()));
Review Comment:
I believe this occurs if there is no path found due to a number of reasons
i.e. `flow.conf` not matching or some flowgraph not matching problem. This in
turns causes the returned dag to be null (which is bad for users to understand
why their flow is failing) and hence the undetailed response, where compilation
errors are added to a returned flowspec output since the pathfinding algorithm
is BFS, and due to compilation errors there's a wide range of potential edges
that can be matched. The final output of this flag being false leads to this:
```
message.append(" Compilation errors encountered (Sorted by relevance):
");
FlowSpec.CompilationError[] errors =
flowSpec.getCompilationErrors().stream().distinct().toArray(FlowSpec.CompilationError[]::new);
Arrays.sort(errors, Comparator.comparingInt(c ->
((FlowSpec.CompilationError)c).errorPriority));
int errorId = 0;
for (FlowSpec.CompilationError error: errors) {
message.append("\n").append(String.format("ERROR[%s]",
errorId)).append(error.errorMessage);
errorId++;
}
```
I think there is definitely a better set of responses we can add here for
compilation scenarios but this is outside of the scope of the PR. I think we
need a better pathfinding algorithm along with a more structured response
format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
