[
https://issues.apache.org/jira/browse/GRIFFIN-316?focusedWorklogId=401479&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-401479
]
ASF GitHub Bot logged work on GRIFFIN-316:
------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Mar/20 14:41
Start Date: 11/Mar/20 14:41
Worklog Time Spent: 10m
Work Description: guoyuepeng commented on issue #562: [GRIFFIN-316] Fix
job exception handling
URL: https://github.com/apache/griffin/pull/562#issuecomment-597673956
ok, my local build passed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 401479)
Time Spent: 3h 50m (was: 3h 40m)
> Spark runtime exception cannot be caught while running a dq application
> -----------------------------------------------------------------------
>
> Key: GRIFFIN-316
> URL: https://issues.apache.org/jira/browse/GRIFFIN-316
> Project: Griffin
> Issue Type: Bug
> Affects Versions: 0.4.0, 0.5.0, 0.6.0
> Reporter: Yu LIU
> Priority: Major
> Fix For: 0.6.0
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> If we put an invalid rule for a batch job (as it happens quite often given
> that the rules are evaluated at runtime via spark sql), the exception thrown
> by SparkSession has not been caught and transferred properly to user via
> "Try" instance, but the job actually succeed with a "Success" returned.
> The reason is that we are only wrapping the returned Boolean result by
> applying "Try" at the most outside level for DQApp.run, so the exception
> thrown deeper through the call stack cannot be caught.
>
> Here is an example config file to reproduce the issue:
> {noformat}
> {
> "name": "prof_batch",
> "process.type": "batch",
> "timestamp": 123456,
> "data.sources": [
> {
> "name": "source",
> "connectors": [
> {
> "type": "avro",
> "version": "1.7",
> "dataframe.name" : "this_table",
> "config": {
> "file.name": "src/test/resources/users_info_src.avro"
> },
> "pre.proc": [
> {
> "dsl.type": "spark-sql",
> "rule": "select * from this_table where user_id < 10014"
> }
> ]
> }
> ]
> }
> ],
> "evaluate.rule": {
> "rules": [
> {
> "dsl.type": "griffin-dsl",
> "dq.type": "profiling",
> "out.dataframe.name": "prof",
> "rule": "xxx",
> "out":[
> {
> "type": "metric",
> "name": "prof",
> "flatten": "array"
> }
> ]
> }
> ]
> },
> "sinks": ["CONSOLE"]
> }{noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
