[jira] [Commented] (GRIFFIN-190) Blank Health and DQ Metrics Screen

[Lionel Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/GRIFFIN-190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605153#comment-16605153
 ] 

Lionel Liu commented on GRIFFIN-190:


Hi [~cwoytasik], you might need to check for some information.

1. Assume you're using the default env.json, there should be result persisted 
in hdfs if the measure job succeed, you can find the results in the path: 
hdfs:///griffin/persist//, there will be several directories named as 
the timestamp of job triggered, inside there lists the metrics.
 * If the "_METRICS" file seems good, it means that the job succeed in spark.
 * If the "_METRICS" doesn't exist, we have to find the yarn log of the spark 
application for the job. In that way, we need to find the application id in 
livy log or griffin service log, then fetch yarn log by this: 
yarn logs -applicationId  > app.log
To export the application log into app.log, then you can find the ERROR msg in 
that log.

2. If the results exist in hdfs, we can try to query them from ES like this: 

curl -XGET 
':9200/griffin/accuracy/_search?pretty_path=hits.hits._source' -d 
'\{"query":{"match_all":{}},  "sort": [\{"tmst": {"order": "asc"}}]}'

If it doesn't exist, there might be something wrong when spark application 
submit metrics to ES.

> Blank Health and DQ Metrics Screen
> --
>
> Key: GRIFFIN-190
> URL: https://issues.apache.org/jira/browse/GRIFFIN-190
> Project: Griffin (Incubating)
>  Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Cory Woytasik
>Priority: Major
>
> Griffin is up and running.  We have both an accuracy measure and a profiling 
> measure that is set to run every minute via jobs.  When we click the chart 
> icon next to the job we receive a "no content" message.  When we click on the 
> Health link or DQ Metrics link they think for a second and then display a 
> blank screen.  We are thinking this might be ES related, but aren't 
> completely sure.  Need some help.  We assume it's a path or property setup 
> issue.  Here are the versions we are running:
> Hive - 3.1.0
> Elasticsearch - 5.3.1
> griffin - 0.2.0
> hadoop - 3.1.1
> livy - 0.3.0
> spark - 2.3.1
> Using postgres too



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (GRIFFIN-190) Blank Health and DQ Metrics Screen

[Cory Woytasik (JIRA)]

Cory Woytasik created GRIFFIN-190:
-

 Summary: Blank Health and DQ Metrics Screen
 Key: GRIFFIN-190
 URL: https://issues.apache.org/jira/browse/GRIFFIN-190
 Project: Griffin (Incubating)
  Issue Type: Bug
Affects Versions: 0.2.0-incubating
Reporter: Cory Woytasik


Griffin is up and running.  We have both an accuracy measure and a profiling 
measure that is set to run every minute via jobs.  When we click the chart icon 
next to the job we receive a "no content" message.  When we click on the Health 
link or DQ Metrics link they think for a second and then display a blank 
screen.  We are thinking this might be ES related, but aren't completely sure.  
Need some help.  We assume it's a path or property setup issue.  Here are the 
versions we are running:

Hive - 3.1.0

Elasticsearch - 5.3.1

griffin - 0.2.0

hadoop - 3.1.1

livy - 0.3.0

spark - 2.3.1

Using postgres too



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (GRIFFIN-189) Griffin - Livy error

[Cory Woytasik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/GRIFFIN-189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cory Woytasik closed GRIFFIN-189.
-
Resolution: Fixed

Fixed via by installing an old version of livy

> Griffin - Livy error
> 
>
> Key: GRIFFIN-189
> URL: https://issues.apache.org/jira/browse/GRIFFIN-189
> Project: Griffin (Incubating)
>  Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Cory Woytasik
>Priority: Major
>  Labels: beginner, newbie, usability
> Attachments: livy_server.log, sparkJob.properties, sparkJob.properties
>
>
> We are trying to get griffin set up and after creating measure and jobs and 
> letting them run we have noticed the results are not available via the DQ 
> metrics link or metric link from the job itself.  We have noticed when the 
> job gets submitted the following spark context and error message are 
> generated.  We assume we must have a setting in one of the directories set 
> incorrectly.  Thoughts?
>  
> INFO 20972 --- [ryBean_Worker-2] o.a.g.c.j.SparkSubmitJob : {
>   "measure.type" : "griffin",
>   "id" : 13,
>   "name" : "LineageAccuracy",
>   "owner" : "test",
>   "description" : "AccuracyTest",
>   "organization" : null,
>   "deleted" : false,
>   "timestamp" : 153599832,
>   "dq.type" : "accuracy",
>   "process.type" : "batch",
>   "data.sources" : [ {
>     "id" : 16,
>     "name" : "source",
>     "connectors" : [ {
>   "id" : 17,
>   "name" : "source1535741016027",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineage"
>   }
>     } ]
>   }, {
>     "id" : 18,
>     "name" : "target",
>     "connectors" : [ {
>   "id" : 19,
>   "name" : "target1535741022277",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineageload"
>   }
>     } ]
>   } ],
>   "evaluate.rule" : {
>     "id" : 14,
>     "rules" : [ {
>   "id" : 15,
>   "rule" : "source.asset=target.asset AND source.element=target.element 
> AND source.elementtype=target.elementtype AND source.object=target.object AND 
> source.objecttype=target.objecttype AND source.objectfield=target.objectfield 
> AND source.sourceelement=target.sourceelement AND 
> source.sourceobject=target.sourceobject AND 
> source.sourcefield=target.sourcefield AND 
> source.sourcefieldname=target.sourcefieldname AND 
> source.transformationtext=target.transformationtext AND 
> source.displayindicator=target.displayindicator",
>   "name" : "accuracy",
>   "dsl.type" : "griffin-dsl",
>   "dq.type" : "accuracy"
>     } ]
>   },
>   "measure.type" : "griffin"
> }
> {color:#FF}2018-09-04 13:12:00.752 ERROR 20972 --- [ryBean_Worker-2] 
> o.a.g.c.j.SparkSubmitJob : Post to livy error. 500 Internal 
> Server Error{color}
> [EL Fine]: sql: 2018-09-04 
> 13:12:00.754--ClientSession(787879814)--Connection(1389579691)--UPDATE 
> JOBINSTANCEBEAN SET predicate_job_deleted = ?, STATE = ? WHERE (ID = ?)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GRIFFIN-189) Griffin - Livy error

[Cory Woytasik (JIRA)]

[ 
https://issues.apache.org/jira/browse/GRIFFIN-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604958#comment-16604958
 ] 

Cory Woytasik commented on GRIFFIN-189:
---

I will close this Jira item and open a new item for the new issue.

> Griffin - Livy error
> 
>
> Key: GRIFFIN-189
> URL: https://issues.apache.org/jira/browse/GRIFFIN-189
> Project: Griffin (Incubating)
>  Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Cory Woytasik
>Priority: Major
>  Labels: beginner, newbie, usability
> Attachments: livy_server.log, sparkJob.properties, sparkJob.properties
>
>
> We are trying to get griffin set up and after creating measure and jobs and 
> letting them run we have noticed the results are not available via the DQ 
> metrics link or metric link from the job itself.  We have noticed when the 
> job gets submitted the following spark context and error message are 
> generated.  We assume we must have a setting in one of the directories set 
> incorrectly.  Thoughts?
>  
> INFO 20972 --- [ryBean_Worker-2] o.a.g.c.j.SparkSubmitJob : {
>   "measure.type" : "griffin",
>   "id" : 13,
>   "name" : "LineageAccuracy",
>   "owner" : "test",
>   "description" : "AccuracyTest",
>   "organization" : null,
>   "deleted" : false,
>   "timestamp" : 153599832,
>   "dq.type" : "accuracy",
>   "process.type" : "batch",
>   "data.sources" : [ {
>     "id" : 16,
>     "name" : "source",
>     "connectors" : [ {
>   "id" : 17,
>   "name" : "source1535741016027",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineage"
>   }
>     } ]
>   }, {
>     "id" : 18,
>     "name" : "target",
>     "connectors" : [ {
>   "id" : 19,
>   "name" : "target1535741022277",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineageload"
>   }
>     } ]
>   } ],
>   "evaluate.rule" : {
>     "id" : 14,
>     "rules" : [ {
>   "id" : 15,
>   "rule" : "source.asset=target.asset AND source.element=target.element 
> AND source.elementtype=target.elementtype AND source.object=target.object AND 
> source.objecttype=target.objecttype AND source.objectfield=target.objectfield 
> AND source.sourceelement=target.sourceelement AND 
> source.sourceobject=target.sourceobject AND 
> source.sourcefield=target.sourcefield AND 
> source.sourcefieldname=target.sourcefieldname AND 
> source.transformationtext=target.transformationtext AND 
> source.displayindicator=target.displayindicator",
>   "name" : "accuracy",
>   "dsl.type" : "griffin-dsl",
>   "dq.type" : "accuracy"
>     } ]
>   },
>   "measure.type" : "griffin"
> }
> {color:#FF}2018-09-04 13:12:00.752 ERROR 20972 --- [ryBean_Worker-2] 
> o.a.g.c.j.SparkSubmitJob : Post to livy error. 500 Internal 
> Server Error{color}
> [EL Fine]: sql: 2018-09-04 
> 13:12:00.754--ClientSession(787879814)--Connection(1389579691)--UPDATE 
> JOBINSTANCEBEAN SET predicate_job_deleted = ?, STATE = ? WHERE (ID = ?)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GRIFFIN-189) Griffin - Livy error

[Cory Woytasik (JIRA)]

[ 
https://issues.apache.org/jira/browse/GRIFFIN-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604804#comment-16604804
 ] 

Cory Woytasik commented on GRIFFIN-189:
---

We look to have figured out the livy error message.  We installed an older 
version of livy and it cleared itself.  We are now trying to figure out why 
metrics are not displayed and we receive a "no content" message when trying to 
view the metrics from the job itself.  We also have a blank health screen and a 
blank DQ Metrics screen.  Assuming that is not related to livy.

> Griffin - Livy error
> 
>
> Key: GRIFFIN-189
> URL: https://issues.apache.org/jira/browse/GRIFFIN-189
> Project: Griffin (Incubating)
>  Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Cory Woytasik
>Priority: Major
>  Labels: beginner, newbie, usability
> Attachments: livy_server.log, sparkJob.properties, sparkJob.properties
>
>
> We are trying to get griffin set up and after creating measure and jobs and 
> letting them run we have noticed the results are not available via the DQ 
> metrics link or metric link from the job itself.  We have noticed when the 
> job gets submitted the following spark context and error message are 
> generated.  We assume we must have a setting in one of the directories set 
> incorrectly.  Thoughts?
>  
> INFO 20972 --- [ryBean_Worker-2] o.a.g.c.j.SparkSubmitJob : {
>   "measure.type" : "griffin",
>   "id" : 13,
>   "name" : "LineageAccuracy",
>   "owner" : "test",
>   "description" : "AccuracyTest",
>   "organization" : null,
>   "deleted" : false,
>   "timestamp" : 153599832,
>   "dq.type" : "accuracy",
>   "process.type" : "batch",
>   "data.sources" : [ {
>     "id" : 16,
>     "name" : "source",
>     "connectors" : [ {
>   "id" : 17,
>   "name" : "source1535741016027",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineage"
>   }
>     } ]
>   }, {
>     "id" : 18,
>     "name" : "target",
>     "connectors" : [ {
>   "id" : 19,
>   "name" : "target1535741022277",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineageload"
>   }
>     } ]
>   } ],
>   "evaluate.rule" : {
>     "id" : 14,
>     "rules" : [ {
>   "id" : 15,
>   "rule" : "source.asset=target.asset AND source.element=target.element 
> AND source.elementtype=target.elementtype AND source.object=target.object AND 
> source.objecttype=target.objecttype AND source.objectfield=target.objectfield 
> AND source.sourceelement=target.sourceelement AND 
> source.sourceobject=target.sourceobject AND 
> source.sourcefield=target.sourcefield AND 
> source.sourcefieldname=target.sourcefieldname AND 
> source.transformationtext=target.transformationtext AND 
> source.displayindicator=target.displayindicator",
>   "name" : "accuracy",
>   "dsl.type" : "griffin-dsl",
>   "dq.type" : "accuracy"
>     } ]
>   },
>   "measure.type" : "griffin"
> }
> {color:#FF}2018-09-04 13:12:00.752 ERROR 20972 --- [ryBean_Worker-2] 
> o.a.g.c.j.SparkSubmitJob : Post to livy error. 500 Internal 
> Server Error{color}
> [EL Fine]: sql: 2018-09-04 
> 13:12:00.754--ClientSession(787879814)--Connection(1389579691)--UPDATE 
> JOBINSTANCEBEAN SET predicate_job_deleted = ?, STATE = ? WHERE (ID = ?)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GRIFFIN-189) Griffin - Livy error

[Cory Woytasik (JIRA)]

[ 
https://issues.apache.org/jira/browse/GRIFFIN-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604626#comment-16604626
 ] 

Cory Woytasik commented on GRIFFIN-189:
---

Here is our livy log.  The error looks to be generated when we are bringing 
Griffin up [^livy_server.log]

> Griffin - Livy error
> 
>
> Key: GRIFFIN-189
> URL: https://issues.apache.org/jira/browse/GRIFFIN-189
> Project: Griffin (Incubating)
>  Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Cory Woytasik
>Priority: Major
>  Labels: beginner, newbie, usability
> Attachments: livy_server.log, sparkJob.properties, sparkJob.properties
>
>
> We are trying to get griffin set up and after creating measure and jobs and 
> letting them run we have noticed the results are not available via the DQ 
> metrics link or metric link from the job itself.  We have noticed when the 
> job gets submitted the following spark context and error message are 
> generated.  We assume we must have a setting in one of the directories set 
> incorrectly.  Thoughts?
>  
> INFO 20972 --- [ryBean_Worker-2] o.a.g.c.j.SparkSubmitJob : {
>   "measure.type" : "griffin",
>   "id" : 13,
>   "name" : "LineageAccuracy",
>   "owner" : "test",
>   "description" : "AccuracyTest",
>   "organization" : null,
>   "deleted" : false,
>   "timestamp" : 153599832,
>   "dq.type" : "accuracy",
>   "process.type" : "batch",
>   "data.sources" : [ {
>     "id" : 16,
>     "name" : "source",
>     "connectors" : [ {
>   "id" : 17,
>   "name" : "source1535741016027",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineage"
>   }
>     } ]
>   }, {
>     "id" : 18,
>     "name" : "target",
>     "connectors" : [ {
>   "id" : 19,
>   "name" : "target1535741022277",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineageload"
>   }
>     } ]
>   } ],
>   "evaluate.rule" : {
>     "id" : 14,
>     "rules" : [ {
>   "id" : 15,
>   "rule" : "source.asset=target.asset AND source.element=target.element 
> AND source.elementtype=target.elementtype AND source.object=target.object AND 
> source.objecttype=target.objecttype AND source.objectfield=target.objectfield 
> AND source.sourceelement=target.sourceelement AND 
> source.sourceobject=target.sourceobject AND 
> source.sourcefield=target.sourcefield AND 
> source.sourcefieldname=target.sourcefieldname AND 
> source.transformationtext=target.transformationtext AND 
> source.displayindicator=target.displayindicator",
>   "name" : "accuracy",
>   "dsl.type" : "griffin-dsl",
>   "dq.type" : "accuracy"
>     } ]
>   },
>   "measure.type" : "griffin"
> }
> {color:#FF}2018-09-04 13:12:00.752 ERROR 20972 --- [ryBean_Worker-2] 
> o.a.g.c.j.SparkSubmitJob : Post to livy error. 500 Internal 
> Server Error{color}
> [EL Fine]: sql: 2018-09-04 
> 13:12:00.754--ClientSession(787879814)--Connection(1389579691)--UPDATE 
> JOBINSTANCEBEAN SET predicate_job_deleted = ?, STATE = ? WHERE (ID = ?)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (GRIFFIN-189) Griffin - Livy error

[Cory Woytasik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/GRIFFIN-189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cory Woytasik updated GRIFFIN-189:
--
Attachment: livy_server.log

> Griffin - Livy error
> 
>
> Key: GRIFFIN-189
> URL: https://issues.apache.org/jira/browse/GRIFFIN-189
> Project: Griffin (Incubating)
>  Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Cory Woytasik
>Priority: Major
>  Labels: beginner, newbie, usability
> Attachments: livy_server.log, sparkJob.properties, sparkJob.properties
>
>
> We are trying to get griffin set up and after creating measure and jobs and 
> letting them run we have noticed the results are not available via the DQ 
> metrics link or metric link from the job itself.  We have noticed when the 
> job gets submitted the following spark context and error message are 
> generated.  We assume we must have a setting in one of the directories set 
> incorrectly.  Thoughts?
>  
> INFO 20972 --- [ryBean_Worker-2] o.a.g.c.j.SparkSubmitJob : {
>   "measure.type" : "griffin",
>   "id" : 13,
>   "name" : "LineageAccuracy",
>   "owner" : "test",
>   "description" : "AccuracyTest",
>   "organization" : null,
>   "deleted" : false,
>   "timestamp" : 153599832,
>   "dq.type" : "accuracy",
>   "process.type" : "batch",
>   "data.sources" : [ {
>     "id" : 16,
>     "name" : "source",
>     "connectors" : [ {
>   "id" : 17,
>   "name" : "source1535741016027",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineage"
>   }
>     } ]
>   }, {
>     "id" : 18,
>     "name" : "target",
>     "connectors" : [ {
>   "id" : 19,
>   "name" : "target1535741022277",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineageload"
>   }
>     } ]
>   } ],
>   "evaluate.rule" : {
>     "id" : 14,
>     "rules" : [ {
>   "id" : 15,
>   "rule" : "source.asset=target.asset AND source.element=target.element 
> AND source.elementtype=target.elementtype AND source.object=target.object AND 
> source.objecttype=target.objecttype AND source.objectfield=target.objectfield 
> AND source.sourceelement=target.sourceelement AND 
> source.sourceobject=target.sourceobject AND 
> source.sourcefield=target.sourcefield AND 
> source.sourcefieldname=target.sourcefieldname AND 
> source.transformationtext=target.transformationtext AND 
> source.displayindicator=target.displayindicator",
>   "name" : "accuracy",
>   "dsl.type" : "griffin-dsl",
>   "dq.type" : "accuracy"
>     } ]
>   },
>   "measure.type" : "griffin"
> }
> {color:#FF}2018-09-04 13:12:00.752 ERROR 20972 --- [ryBean_Worker-2] 
> o.a.g.c.j.SparkSubmitJob : Post to livy error. 500 Internal 
> Server Error{color}
> [EL Fine]: sql: 2018-09-04 
> 13:12:00.754--ClientSession(787879814)--Connection(1389579691)--UPDATE 
> JOBINSTANCEBEAN SET predicate_job_deleted = ?, STATE = ? WHERE (ID = ?)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GRIFFIN-189) Griffin - Livy error

[Cory Woytasik (JIRA)]

[ 
https://issues.apache.org/jira/browse/GRIFFIN-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604421#comment-16604421
 ] 

Cory Woytasik commented on GRIFFIN-189:
---

We are not using docker for this implementation. Thanks

> Griffin - Livy error
> 
>
> Key: GRIFFIN-189
> URL: https://issues.apache.org/jira/browse/GRIFFIN-189
> Project: Griffin (Incubating)
>  Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Cory Woytasik
>Priority: Major
>  Labels: beginner, newbie, usability
> Attachments: sparkJob.properties, sparkJob.properties
>
>
> We are trying to get griffin set up and after creating measure and jobs and 
> letting them run we have noticed the results are not available via the DQ 
> metrics link or metric link from the job itself.  We have noticed when the 
> job gets submitted the following spark context and error message are 
> generated.  We assume we must have a setting in one of the directories set 
> incorrectly.  Thoughts?
>  
> INFO 20972 --- [ryBean_Worker-2] o.a.g.c.j.SparkSubmitJob : {
>   "measure.type" : "griffin",
>   "id" : 13,
>   "name" : "LineageAccuracy",
>   "owner" : "test",
>   "description" : "AccuracyTest",
>   "organization" : null,
>   "deleted" : false,
>   "timestamp" : 153599832,
>   "dq.type" : "accuracy",
>   "process.type" : "batch",
>   "data.sources" : [ {
>     "id" : 16,
>     "name" : "source",
>     "connectors" : [ {
>   "id" : 17,
>   "name" : "source1535741016027",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineage"
>   }
>     } ]
>   }, {
>     "id" : 18,
>     "name" : "target",
>     "connectors" : [ {
>   "id" : 19,
>   "name" : "target1535741022277",
>   "type" : "HIVE",
>   "version" : "1.2",
>   "predicates" : [ ],
>   "data.unit" : "1day",
>   "config" : {
>     "database" : "default",
>     "table.name" : "lineageload"
>   }
>     } ]
>   } ],
>   "evaluate.rule" : {
>     "id" : 14,
>     "rules" : [ {
>   "id" : 15,
>   "rule" : "source.asset=target.asset AND source.element=target.element 
> AND source.elementtype=target.elementtype AND source.object=target.object AND 
> source.objecttype=target.objecttype AND source.objectfield=target.objectfield 
> AND source.sourceelement=target.sourceelement AND 
> source.sourceobject=target.sourceobject AND 
> source.sourcefield=target.sourcefield AND 
> source.sourcefieldname=target.sourcefieldname AND 
> source.transformationtext=target.transformationtext AND 
> source.displayindicator=target.displayindicator",
>   "name" : "accuracy",
>   "dsl.type" : "griffin-dsl",
>   "dq.type" : "accuracy"
>     } ]
>   },
>   "measure.type" : "griffin"
> }
> {color:#FF}2018-09-04 13:12:00.752 ERROR 20972 --- [ryBean_Worker-2] 
> o.a.g.c.j.SparkSubmitJob : Post to livy error. 500 Internal 
> Server Error{color}
> [EL Fine]: sql: 2018-09-04 
> 13:12:00.754--ClientSession(787879814)--Connection(1389579691)--UPDATE 
> JOBINSTANCEBEAN SET predicate_job_deleted = ?, STATE = ? WHERE (ID = ?)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (GRIFFIN-102) [Service] Fix bug of fetching metrics of different jobs with the same name

[Lionel Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/GRIFFIN-102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604152#comment-16604152
 ] 

Lionel Liu commented on GRIFFIN-102:


Actually it's not that easy.

Say we have a metric named "test" with some points between 2018-08-09T12:00 and 
2018-08-10T12:00, and the metric is deleted. After 3 days, we create another 
metric also named "test", and there will be metric points after 
2018-08-13T12:00. In ES, we only saved the metric name "test" as the key, when 
we fetch the metrics of "test", ES will return two slices of data. If the two 
metrics are in the same type like "accuracy", it will confuse the latter user. 
If they are in different types, it will lead error.

> [Service] Fix bug of fetching metrics of different jobs with the same name
> --
>
> Key: GRIFFIN-102
> URL: https://issues.apache.org/jira/browse/GRIFFIN-102
> Project: Griffin (Incubating)
>  Issue Type: Bug
>Reporter: Lionel Liu
>Assignee: He Wang
>Priority: Major
>  Labels: SP_5
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> When fetch metrics from ES, if one job has the same name with another job 
> which was deleted, the metrics would be union and confuse.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] incubator-griffin pull request #410: Fix download table bug

[icesmartjuan]

GitHub user icesmartjuan opened a pull request:

https://github.com/apache/incubator-griffin/pull/410

Fix download table bug



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/icesmartjuan/incubator-griffin 
fix_ui_bug_2018905

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-griffin/pull/410.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #410


commit 811b5838d7e274f356c985a2be9a5cb349f75944
Author: Li, Juan 
Date:   2018-09-05T07:58:47Z

Fix download table bug




---

Re: [VOTE] Release of Apache Griffin-0.3.0-incubating [RC1]

[Henry Saputra]

+1 (binding)

LICENSE file looks good
NOTICE file looks good
DISCLAIMER file looks good
NO exes in source artifacts
License header exists
Source compile.

- Henry

On Thu, Aug 30, 2018 at 7:44 AM Lionel Liu  wrote:

> Hi all,
>
> This is a call for a vote on releasing Apache Griffin 0.3.0-incubating,
> release candidate 1.
> Apache Griffin is data quality service for modern data system, it
> defines a standard process to define,measure data quality for well-known
> dimensions.
> With Apache Griffin, users will be able to quickly define their data
> quality requirements and then get the result in near real time in
> systematical approach.
>
>
> ** Highlights **
> * Refactor measure module for better abstraction.
> * Support missing records download for accuracy measurement.
> * Support regular expression detection count in profiling measurement.
> * Fix several bugs on UI.
>
>
> The source tarball, including signatures, digests, etc. can be found
> at:
> *
> https://dist.apache.org/repos/dist/dev/incubator/griffin/0.3.0-incubating/
> <
> https://dist.apache.org/repos/dist/dev/incubator/griffin/0.3.0-incubating/
> >*
>
> The tag to be voted upon is 0.3.0-incubating:
> *
> https://git-wip-us.apache.org/repos/asf?p=incubator-griffin.git;a=shortlog;h=refs/tags/griffin-0.3.0-incubating
> <
> https://git-wip-us.apache.org/repos/asf?p=incubator-griffin.git;a=shortlog;h=refs/tags/griffin-0.3.0-incubating
> >*
>
> The release hash is :
> *
> https://git-wip-us.apache.org/repos/asf?p=incubator-griffin.git;a=commit;h=797cc62c94449e485d3af910bc8557ca9841bb22
> <
> https://git-wip-us.apache.org/repos/asf?p=incubator-griffin.git;a=commit;h=797cc62c94449e485d3af910bc8557ca9841bb22
> >*
>
> The Nexus Staging URL:
> *
> https://repository.apache.org/content/repositories/orgapachegriffin-1018
>  >*
>
>
> Release artifacts are signed with the following key:
> 7F00C3BA90F3ECAEECB843A79BD6EC6C02379561
> KEYS file available:
> https://dist.apache.org/repos/dist/dev/incubator/griffin/KEYS
>
> For information about the contents of this release, see:
> *
> https://dist.apache.org/repos/dist/dev/incubator/griffin/0.3.0-incubating/CHANGES.txt
> <
> https://dist.apache.org/repos/dist/dev/incubator/griffin/0.3.0-incubating/CHANGES.txt
> >*
>
>
> Please vote on releasing this package as Apache Griffin
> 0.3.0-incubating
>
>
> The vote will be open for 72 hours.
>
> [ ] +1 Release this package as Apache Griffin 0.3.0-incubating
> [ ] +0 no opinion
> [ ] -1 Do not release this package because ...
>
>
> You can follow the steps here to verify the release before you vote:
> https://cwiki.apache.org/confluence/display/GRIFFIN/How+to+Verify+
> Release+Package
>
>
> Thanks,
> Lionel
> On behalf of Apache Griffin PPMC
>

Re: [RESULT][VOTE] Release of Apache Griffin 0.3.0-incubating [RC1]

[Henry Saputra]

I missed the VOTE for this one will VOTE in general list instead.

On Sun, Sep 2, 2018 at 11:07 PM Lionel Liu  wrote:

> Hi all,
>
> Vote passed with 5[+1] binding votes and no[-1] votes , Please check
> the following tally.
>
> +1 binding: [5]
>   Lv Alex
>   Eugene Liu
>   Lionel Liu
>   William Guo
>   Kevin Yao
>
> 0 : [0]
>
> -1 : [0]
>
> The vote thread lists here:
> *
> https://lists.apache.org/thread.html/b5184c861d299462f369a87b0a911bfd00905b1c79232ff242e0b78a@%3Cdev.griffin.apache.org%3E
> <
> https://lists.apache.org/thread.html/b5184c861d299462f369a87b0a911bfd00905b1c79232ff242e0b78a@%3Cdev.griffin.apache.org%3E
> >*
>
> Thanks,
> Lionel
> On behalf of Apache Griffin PPMC
>

[GitHub] incubator-griffin issue #409: Support for HDP

[SparshSinghalHM]

Github user SparshSinghalHM commented on the issue:

https://github.com/apache/incubator-griffin/pull/409
  
@bhlx3lyx7  No, not required. We can ask for the ambari username in 
application.properties. Admin is the default user. 


---

[GitHub] incubator-griffin pull request #409: Support for HDP

[SparshSinghalHM]

Github user SparshSinghalHM commented on a diff in the pull request:

https://github.com/apache/incubator-griffin/pull/409#discussion_r215148542
  
--- Diff: 
service/src/main/java/org/apache/griffin/core/job/SparkSubmitJob.java ---
@@ -113,8 +117,13 @@ private void 
updateJobInstanceState(JobExecutionContext context) throws
 private String post2Livy() {
 String result = null;
 try {
-result = restTemplate.postForObject(livyUri, livyConfMap,
-String.class);
+HttpHeaders headers = new HttpHeaders();
+headers.setContentType(MediaType.APPLICATION_JSON);
+headers.set("X-Requested-By","admin");
--- End diff --

Yes. Can do it this way. 


---