[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[ https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=439837=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439837 ] ASF GitHub Bot logged work on GOBBLIN-1154: --- Author: ASF GitHub Bot Created on: 02/Jun/20 01:27 Start Date: 02/Jun/20 01:27 Worklog Time Spent: 10m Work Description: asfgit closed pull request #2993: URL: https://github.com/apache/incubator-gobblin/pull/2993 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439837) Time Spent: 3h (was: 2h 50m) > Improve gaas error messages > --- > > Key: GOBBLIN-1154 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1154 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jack Moseley >Priority: Major > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[ https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=439658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-439658 ] ASF GitHub Bot logged work on GOBBLIN-1154: --- Author: ASF GitHub Bot Created on: 01/Jun/20 18:22 Start Date: 01/Jun/20 18:22 Worklog Time Spent: 10m Work Description: jack-moseley commented on pull request #2993: URL: https://github.com/apache/incubator-gobblin/pull/2993#issuecomment-637027075 @sv2000 please merge This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 439658) Time Spent: 2h 50m (was: 2h 40m) > Improve gaas error messages > --- > > Key: GOBBLIN-1154 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1154 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jack Moseley >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[ https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=438993=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438993 ] ASF GitHub Bot logged work on GOBBLIN-1154: --- Author: ASF GitHub Bot Created on: 30/May/20 00:39 Start Date: 30/May/20 00:39 Worklog Time Spent: 10m Work Description: codecov-commenter commented on pull request #2993: URL: https://github.com/apache/incubator-gobblin/pull/2993#issuecomment-636248091 # [Codecov](https://codecov.io/gh/apache/incubator-gobblin/pull/2993?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@72373ee`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0.00%`. [](https://codecov.io/gh/apache/incubator-gobblin/pull/2993?src=pr=tree) ```diff @@ Coverage Diff@@ ## master #2993 +/- ## Coverage ? 7.15% Complexity?1247 Files ?1945 Lines ? 73950 Branches ?8189 Hits ?5294 Misses? 68149 Partials ? 507 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-gobblin/pull/2993?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../org/apache/gobblin/metrics/event/TimingEvent.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2993/diff?src=pr=tree#diff-Z29iYmxpbi1tZXRyaWNzLWxpYnMvZ29iYmxpbi1tZXRyaWNzLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vbWV0cmljcy9ldmVudC9UaW1pbmdFdmVudC5qYXZh) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | | | [...blin/service/FlowConfigV2ResourceLocalHandler.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2993/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93Q29uZmlnVjJSZXNvdXJjZUxvY2FsSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [.../apache/gobblin/service/FlowExecutionResource.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2993/diff?src=pr=tree#diff-Z29iYmxpbi1yZXN0bGkvZ29iYmxpbi1mbG93LWNvbmZpZy1zZXJ2aWNlL2dvYmJsaW4tZmxvdy1jb25maWctc2VydmljZS1zZXJ2ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2dvYmJsaW4vc2VydmljZS9GbG93RXhlY3V0aW9uUmVzb3VyY2UuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [.../apache/gobblin/service/modules/flowgraph/Dag.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2993/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9mbG93Z3JhcGgvRGFnLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...blin/service/modules/orchestration/DagManager.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2993/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL0RhZ01hbmFnZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...service/modules/orchestration/DagManagerUtils.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2993/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL0RhZ01hbmFnZXJVdGlscy5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...in/service/modules/orchestration/Orchestrator.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2993/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9kdWxlcy9vcmNoZXN0cmF0aW9uL09yY2hlc3RyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [.../service/monitoring/KafkaAvroJobStatusMonitor.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2993/diff?src=pr=tree#diff-Z29iYmxpbi1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL3NlcnZpY2UvbW9uaXRvcmluZy9LYWZrYUF2cm9Kb2JTdGF0dXNNb25pdG9yLmphdmE=) | `0.00% <ø> (ø)` | `0.00 <0.00> (?)` | | | [...va/org/apache/gobblin/converter/jdbc/JdbcType.java](https://codecov.io/gh/apache/incubator-gobblin/pull/2993/diff?src=pr=tree#diff-Z29iYmxpbi1tb2R1bGVzL2dvYmJsaW4tc3FsL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9nb2JibGluL2NvbnZlcnRlci9qZGJjL0pkYmNUeXBlLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00% <0.00%> (?%)` | | |
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[ https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=438456=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-438456 ] ASF GitHub Bot logged work on GOBBLIN-1154: --- Author: ASF GitHub Bot Created on: 28/May/20 21:27 Start Date: 28/May/20 21:27 Worklog Time Spent: 10m Work Description: aplex commented on pull request #2993: URL: https://github.com/apache/incubator-gobblin/pull/2993#issuecomment-635617535 LGTM, thanks for improving experience of our users! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 438456) Time Spent: 2.5h (was: 2h 20m) > Improve gaas error messages > --- > > Key: GOBBLIN-1154 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1154 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jack Moseley >Priority: Major > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=437582=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437582
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 27/May/20 04:10
Start Date: 27/May/20 04:10
Worklog Time Spent: 10m
Work Description: jack-moseley commented on a change in pull request
#2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r430694079
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigV2ResourceLocalHandler.java
##
@@ -81,7 +83,7 @@ public CreateKVResponse createFlowConfig(FlowConfig
flowConfig, boolean triggerL
} else if
(Boolean.parseBoolean(responseMap.getOrDefault(ServiceConfigKeys.COMPILATION_SUCCESSFUL,
new AddSpecResponse<>("false")).getValue().toString())) {
httpStatus = HttpStatus.S_201_CREATED;
} else {
- httpStatus = HttpStatus.S_400_BAD_REQUEST;
+ throw new RestLiServiceException(HttpStatus.S_400_BAD_REQUEST, "Flow was
not compiled successfully");
Review comment:
The reason could be a couple things:
1. No path found. This could be due to: the edges don't have the paths they
are using, there are no edges between those nodes, the nodes or edges were not
added on gaas startup due to an error, etc.
2. There is an unresolved config. We do have a field that stores whatever
errors we encountered with unresolved config.
The problems are:
- If it fails due to 1, there isn't really any way to know exactly what went
wrong since we don't know what the intended path was.
- Even if it fails due to 1, we may have encountered some unresolved config
on a separate unintended path. So it would be pretty confusing to include an
error that some config was unresolved when the real reason was that the path.
If anything maybe the best message for now would just be "This could be
because no path is found, or because of X unresolved config"?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437582)
Time Spent: 2h 20m (was: 2h 10m)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=437527=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-437527
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 27/May/20 04:02
Start Date: 27/May/20 04:02
Worklog Time Spent: 10m
Work Description: aplex commented on a change in pull request #2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r430721098
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigV2ResourceLocalHandler.java
##
@@ -81,7 +83,7 @@ public CreateKVResponse createFlowConfig(FlowConfig
flowConfig, boolean triggerL
} else if
(Boolean.parseBoolean(responseMap.getOrDefault(ServiceConfigKeys.COMPILATION_SUCCESSFUL,
new AddSpecResponse<>("false")).getValue().toString())) {
httpStatus = HttpStatus.S_201_CREATED;
} else {
- httpStatus = HttpStatus.S_400_BAD_REQUEST;
+ throw new RestLiServiceException(HttpStatus.S_400_BAD_REQUEST, "Flow was
not compiled successfully");
Review comment:
That can be a good start.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 437527)
Time Spent: 2h 10m (was: 2h)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436600=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436600
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 22/May/20 17:22
Start Date: 22/May/20 17:22
Worklog Time Spent: 10m
Work Description: aplex commented on a change in pull request #2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r429366904
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigV2ResourceLocalHandler.java
##
@@ -81,7 +83,7 @@ public CreateKVResponse createFlowConfig(FlowConfig
flowConfig, boolean triggerL
} else if
(Boolean.parseBoolean(responseMap.getOrDefault(ServiceConfigKeys.COMPILATION_SUCCESSFUL,
new AddSpecResponse<>("false")).getValue().toString())) {
httpStatus = HttpStatus.S_201_CREATED;
} else {
- httpStatus = HttpStatus.S_400_BAD_REQUEST;
+ throw new RestLiServiceException(HttpStatus.S_400_BAD_REQUEST, "Flow was
not compiled successfully");
Review comment:
So this will be primarily used by SREs? I can think of a situation when
they have a 2-page flow and a typo in it. How would they find where the typo
is? Are there any helpful compilation exceptions we can provide? For example:
"problem is in the line 123, variable 'chema' is not found", or "no path
found between 'faro' and 'warr', check the names of the source and the target").
If we can't pinpoint a single error and the problems are all around, we can
send users to the compilation logs, and provide them some unique token so that
they can find the relevant lines easily. For example: "for more information,
check the GaaS logs that have requestId=XXX"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436600)
Time Spent: 2h (was: 1h 50m)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436584=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436584
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 22/May/20 16:57
Start Date: 22/May/20 16:57
Worklog Time Spent: 10m
Work Description: aplex commented on a change in pull request #2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r429356449
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowExecutionResource.java
##
@@ -65,8 +66,11 @@ public FlowExecutionResource() {}
*/
@Override
public FlowExecution get(ComplexResourceKey key) {
-// this returns null to raise a 404 error if flowStatus is null
-return convertFlowStatus(getFlowStatusFromGenerator(key,
this._flowStatusGenerator));
+FlowExecution flowExecution =
convertFlowStatus(getFlowStatusFromGenerator(key, this._flowStatusGenerator));
+if (flowExecution == null) {
+ throw new RestLiServiceException(HttpStatus.S_404_NOT_FOUND, "No flow
execution found for flowStatusId " + key.getKey());
Review comment:
Can you add it to the message? The point here is to provide detailed and
helpful error messages to users, so they don't come back to us with follow up
questions, and can troubleshoot themselves.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436584)
Time Spent: 1h 50m (was: 1h 40m)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[ https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436583=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436583 ] ASF GitHub Bot logged work on GOBBLIN-1154: --- Author: ASF GitHub Bot Created on: 22/May/20 16:54 Start Date: 22/May/20 16:54 Worklog Time Spent: 10m Work Description: aplex commented on a change in pull request #2993: URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r429354910 ## File path: gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/TimingEventUtils.java ## @@ -26,6 +26,7 @@ import org.apache.gobblin.runtime.api.FlowSpec; import org.apache.gobblin.runtime.api.JobSpec; import org.apache.gobblin.runtime.api.SpecExecutor; +import org.apache.gobblin.service.modules.flowgraph.Dag; Review comment: These types of problems should be automatically caught by SpotBugs/CheckStyle. Another example that they should be enabled on all repos. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 436583) Time Spent: 1h 40m (was: 1.5h) > Improve gaas error messages > --- > > Key: GOBBLIN-1154 > URL: https://issues.apache.org/jira/browse/GOBBLIN-1154 > Project: Apache Gobblin > Issue Type: Improvement >Reporter: Jack Moseley >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436226=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436226
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 21/May/20 21:55
Start Date: 21/May/20 21:55
Worklog Time Spent: 10m
Work Description: jack-moseley commented on a change in pull request
#2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r428937502
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowExecutionResource.java
##
@@ -65,8 +66,11 @@ public FlowExecutionResource() {}
*/
@Override
public FlowExecution get(ComplexResourceKey key) {
-// this returns null to raise a 404 error if flowStatus is null
-return convertFlowStatus(getFlowStatusFromGenerator(key,
this._flowStatusGenerator));
+FlowExecution flowExecution =
convertFlowStatus(getFlowStatusFromGenerator(key, this._flowStatusGenerator));
+if (flowExecution == null) {
+ throw new RestLiServiceException(HttpStatus.S_404_NOT_FOUND, "No flow
execution found for flowStatusId " + key.getKey());
Review comment:
It would mean either wrong flow id or the flow status was cleaned up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436226)
Time Spent: 1.5h (was: 1h 20m)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436224
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 21/May/20 21:53
Start Date: 21/May/20 21:53
Worklog Time Spent: 10m
Work Description: jack-moseley commented on a change in pull request
#2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r428936777
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigV2ResourceLocalHandler.java
##
@@ -81,7 +83,7 @@ public CreateKVResponse createFlowConfig(FlowConfig
flowConfig, boolean triggerL
} else if
(Boolean.parseBoolean(responseMap.getOrDefault(ServiceConfigKeys.COMPILATION_SUCCESSFUL,
new AddSpecResponse<>("false")).getValue().toString())) {
httpStatus = HttpStatus.S_201_CREATED;
} else {
- httpStatus = HttpStatus.S_400_BAD_REQUEST;
+ throw new RestLiServiceException(HttpStatus.S_400_BAD_REQUEST, "Flow was
not compiled successfully");
Review comment:
The challenge is that if a flow doesn't compile it's hard to know why
because we don't know what was the intended path. It could be because of
missing edges/nodes, or unresolved config keys. It's something that could be
improved upon still, but most users of gaas self-serve should be using edges
we've already defined so probably shouldn't have this problem anyway.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436224)
Time Spent: 1h 20m (was: 1h 10m)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436223
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 21/May/20 21:49
Start Date: 21/May/20 21:49
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r428935120
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java
##
@@ -259,6 +259,7 @@ public void orchestrate(Spec spec) throws Exception {
// In this case, the current time is used as the flow executionId.
flowMetadata.putIfAbsent(TimingEvent.FlowEventConstants.FLOW_EXECUTION_ID_FIELD,
Long.toString(System.currentTimeMillis()));
+flowMetadata.put(TimingEvent.METADATA_MESSAGE, "Flow compilation
failed");
Review comment:
So the next challenge for us is to find failures in resolving which
flow/job template are the correct errors.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436223)
Time Spent: 1h 10m (was: 1h)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436222
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 21/May/20 21:47
Start Date: 21/May/20 21:47
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r428934311
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java
##
@@ -259,6 +259,7 @@ public void orchestrate(Spec spec) throws Exception {
// In this case, the current time is used as the flow executionId.
flowMetadata.putIfAbsent(TimingEvent.FlowEventConstants.FLOW_EXECUTION_ID_FIELD,
Long.toString(System.currentTimeMillis()));
+flowMetadata.put(TimingEvent.METADATA_MESSAGE, "Flow compilation
failed");
Review comment:
Right.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436222)
Time Spent: 1h (was: 50m)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436214=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436214
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 21/May/20 21:19
Start Date: 21/May/20 21:19
Worklog Time Spent: 10m
Work Description: jack-moseley commented on a change in pull request
#2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r428920315
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java
##
@@ -259,6 +259,7 @@ public void orchestrate(Spec spec) throws Exception {
// In this case, the current time is used as the flow executionId.
flowMetadata.putIfAbsent(TimingEvent.FlowEventConstants.FLOW_EXECUTION_ID_FIELD,
Long.toString(System.currentTimeMillis()));
+flowMetadata.put(TimingEvent.METADATA_MESSAGE, "Flow compilation
failed");
Review comment:
We could, the reason I didn't want to is it would be kind of misleading
right now since sometimes that will have unrelated missing config messages. I
guess for now I could just add it and say "it may be due to these errors".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436214)
Time Spent: 50m (was: 40m)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436212=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436212
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 21/May/20 21:16
Start Date: 21/May/20 21:16
Worklog Time Spent: 10m
Work Description: jack-moseley commented on a change in pull request
#2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r428919270
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java
##
@@ -476,6 +476,9 @@ private void cancelDag(String dagToCancel) throws
ExecutionException, Interrupte
for (DagNode dagNodeToCancel : dagNodesToCancel) {
cancelDagNode(dagNodeToCancel);
}
+
+
this.dags.get(dagToCancel).setFlowEvent(TimingEvent.FlowTimings.FLOW_CANCELLED);
Review comment:
This. is setting the flow event on a dag level, not a dag node level, so
it makes more sense to do it in cancelDag.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436212)
Time Spent: 40m (was: 0.5h)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=436209=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-436209
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 21/May/20 21:12
Start Date: 21/May/20 21:12
Worklog Time Spent: 10m
Work Description: aplex commented on a change in pull request #2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r428915654
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowConfigV2ResourceLocalHandler.java
##
@@ -81,7 +83,7 @@ public CreateKVResponse createFlowConfig(FlowConfig
flowConfig, boolean triggerL
} else if
(Boolean.parseBoolean(responseMap.getOrDefault(ServiceConfigKeys.COMPILATION_SUCCESSFUL,
new AddSpecResponse<>("false")).getValue().toString())) {
httpStatus = HttpStatus.S_201_CREATED;
} else {
- httpStatus = HttpStatus.S_400_BAD_REQUEST;
+ throw new RestLiServiceException(HttpStatus.S_400_BAD_REQUEST, "Flow was
not compiled successfully");
Review comment:
What are the next steps for the user here? Should he check the flow
properties (we can point to the list of allowed properties), should he file a
ticket and ask GaaS engineers to fix the issue? Also, do we have additional
details on why the compilation was unsuccessful and what specific place had a
problem?
##
File path:
gobblin-restli/gobblin-flow-config-service/gobblin-flow-config-service-server/src/main/java/org/apache/gobblin/service/FlowExecutionResource.java
##
@@ -65,8 +66,11 @@ public FlowExecutionResource() {}
*/
@Override
public FlowExecution get(ComplexResourceKey key) {
-// this returns null to raise a 404 error if flowStatus is null
-return convertFlowStatus(getFlowStatusFromGenerator(key,
this._flowStatusGenerator));
+FlowExecution flowExecution =
convertFlowStatus(getFlowStatusFromGenerator(key, this._flowStatusGenerator));
+if (flowExecution == null) {
+ throw new RestLiServiceException(HttpStatus.S_404_NOT_FOUND, "No flow
execution found for flowStatusId " + key.getKey());
Review comment:
When can this happen? Was the wrong flow id provided and user needs to
check it, or the flows can get cleaned up/deleted after some period of time?
Can it be the problem with permissions that he does not see the flow?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 436209)
Time Spent: 0.5h (was: 20m)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=435901=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-435901
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 21/May/20 09:49
Start Date: 21/May/20 09:49
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on a change in pull request
#2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993#discussion_r428528095
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/TimingEventUtils.java
##
@@ -26,6 +26,7 @@
import org.apache.gobblin.runtime.api.FlowSpec;
import org.apache.gobblin.runtime.api.JobSpec;
import org.apache.gobblin.runtime.api.SpecExecutor;
+import org.apache.gobblin.service.modules.flowgraph.Dag;
Review comment:
unused import
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManager.java
##
@@ -476,6 +476,9 @@ private void cancelDag(String dagToCancel) throws
ExecutionException, Interrupte
for (DagNode dagNodeToCancel : dagNodesToCancel) {
cancelDagNode(dagNodeToCancel);
}
+
+
this.dags.get(dagToCancel).setFlowEvent(TimingEvent.FlowTimings.FLOW_CANCELLED);
Review comment:
Should we setFlowEvent() inside cancelDagNode() ?
##
File path:
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/Orchestrator.java
##
@@ -259,6 +259,7 @@ public void orchestrate(Spec spec) throws Exception {
// In this case, the current time is used as the flow executionId.
flowMetadata.putIfAbsent(TimingEvent.FlowEventConstants.FLOW_EXECUTION_ID_FIELD,
Long.toString(System.currentTimeMillis()));
+flowMetadata.put(TimingEvent.METADATA_MESSAGE, "Flow compilation
failed");
Review comment:
Can we add spec.getCompilationErrors() also? We will know which
execution failed with which config exception.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 435901)
Time Spent: 20m (was: 10m)
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (GOBBLIN-1154) Improve gaas error messages
[
https://issues.apache.org/jira/browse/GOBBLIN-1154?focusedWorklogId=435865=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-435865
]
ASF GitHub Bot logged work on GOBBLIN-1154:
---
Author: ASF GitHub Bot
Created on: 21/May/20 07:03
Start Date: 21/May/20 07:03
Worklog Time Spent: 10m
Work Description: jack-moseley opened a new pull request #2993:
URL: https://github.com/apache/incubator-gobblin/pull/2993
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-1154
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Variety of improvements to GaaS error returning:
- When creating flowConfig, in case of error throw it instead of returning
4xx along with flowConfig. Returning the `CreateKVResponse` means that the the
status is only seen if using `--verbose`, and there is no clear error message
- Similarly return a more clear error message when calling get for a
flowExecution that doesn't exist
- Instead of concatenating execution links in the message field of
flowStatus (this is not useful since they are already in the jobStatus list),
use the message field to return a flow level error message
- Added a `FlowCancelled` event so that flows that were SLA killed or
manually killed will have a `CANCELLED` status instead of `FAILED`
### Tests
- [ ] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 435865)
Remaining Estimate: 0h
Time Spent: 10m
> Improve gaas error messages
> ---
>
> Key: GOBBLIN-1154
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1154
> Project: Apache Gobblin
> Issue Type: Improvement
>Reporter: Jack Moseley
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
