[
https://issues.apache.org/jira/browse/GRIFFIN-290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wan kun updated GRIFFIN-290:
----------------------------
Description:
When griffin submit multiple DQ jobs to livy, the http parameter `name` is
always griffin.
So livy will reject them.
job request :
{quote}[owner: null, request: [proxyUser: None, file:
hdfs://nameservice-standby/user/kun.wan/measure-0.6.0-SNAPSHOT.jar,
args: {
"spark" :
Unknown macro: \{ "log.level" }
,
"sinks" : [
Unknown macro: \{ .... }
],
"griffin.checkpoint" : [ ]
},{
"measure.type" : "griffin",
"id" : 5202,
"name" : "spu_null_check",
"owner" : "test",
"description" : "check null value for store and category",
"deleted" : false,
"timestamp" : 1568195100000,
"dq.type" : "PROFILING",
"sinks" : [ "ELASTICSEARCH", "HDFS" ],
"process.type" : "BATCH",
"rule.description" :
,
"data.sources" : [
Unknown macro: \{ .... }
],
"evaluate.rule" :
,
"measure.type" : "griffin"
},raw,raw, driverMemory: 1g, executorMemory: 6g, executorCores: 2,
numExecutors: 6, queue: root.users.kun_dot_wan, {color:#ff0000}*name:
griffin*{color}]]
{quote}
livy Response :
{quote}400 Bad Request
[Date:"Thu, 12 Sep 2019 10:00:00 GMT",
Content-Type:"application/json;charset=utf-8", Content-Length:"47",
Server:"Jetty(9.3.24.v20180605)"]
{"msg":"Duplicate session name: Some(griffin)"}
{quote}
was:
When griffin submit multiple DQ jobs to livy, the http parameter `name` is
always griffin.
So livy will reject them.
job request :
{quote}[owner: null, request: [proxyUser: None, file:
hdfs://nameservice-standby/user/kun.wan/measure-0.6.0-SNAPSHOT.jar,
args: {
"spark" : {
"log.level" : "WARN"
},
"sinks" : [ {
....
} ],
"griffin.checkpoint" : [ ]
},{
"measure.type" : "griffin",
"id" : 5202,
"name" : "spu_null_check",
"owner" : "test",
"description" : "check null value for store and category",
"deleted" : false,
"timestamp" : 1568195100000,
"dq.type" : "PROFILING",
"sinks" : [ "ELASTICSEARCH", "HDFS" ],
"process.type" : "BATCH",
"rule.description" : {
....
},
"data.sources" : [ {
....
} ],
"evaluate.rule" : {
....
},
"measure.type" : "griffin"
},raw,raw, driverMemory: 1g, executorMemory: 6g, executorCores: 2,
numExecutors: 6, queue: root.users.kun_dot_wan, {color:#FF0000}*name:
griffin*{color}]]
{quote}
livy Response :
{quote}400 Bad Request
[Date:"Thu, 12 Sep 2019 10:00:00 GMT",
Content-Type:"application/json;charset=utf-8", Content-Length:"47",
Server:"Jetty(9.3.24.v20180605)"]
Unknown macro: \{"msg"}{quote}
> Fix bug for submitting job to livy
> ----------------------------------
>
> Key: GRIFFIN-290
> URL: https://issues.apache.org/jira/browse/GRIFFIN-290
> Project: Griffin
> Issue Type: Task
> Reporter: wan kun
> Priority: Major
>
> When griffin submit multiple DQ jobs to livy, the http parameter `name` is
> always griffin.
> So livy will reject them.
> job request :
>
> {quote}[owner: null, request: [proxyUser: None, file:
> hdfs://nameservice-standby/user/kun.wan/measure-0.6.0-SNAPSHOT.jar,
> args: {
> "spark" :
> Unknown macro: \{ "log.level" }
> ,
> "sinks" : [
> Unknown macro: \{ .... }
> ],
> "griffin.checkpoint" : [ ]
> },{
> "measure.type" : "griffin",
> "id" : 5202,
> "name" : "spu_null_check",
> "owner" : "test",
> "description" : "check null value for store and category",
> "deleted" : false,
> "timestamp" : 1568195100000,
> "dq.type" : "PROFILING",
> "sinks" : [ "ELASTICSEARCH", "HDFS" ],
> "process.type" : "BATCH",
> "rule.description" :
> ,
> "data.sources" : [
> Unknown macro: \{ .... }
> ],
> "evaluate.rule" :
> ,
> "measure.type" : "griffin"
> },raw,raw, driverMemory: 1g, executorMemory: 6g, executorCores: 2,
> numExecutors: 6, queue: root.users.kun_dot_wan, {color:#ff0000}*name:
> griffin*{color}]]
> {quote}
>
> livy Response :
> {quote}400 Bad Request
> [Date:"Thu, 12 Sep 2019 10:00:00 GMT",
> Content-Type:"application/json;charset=utf-8", Content-Length:"47",
> Server:"Jetty(9.3.24.v20180605)"]
> {"msg":"Duplicate session name: Some(griffin)"}
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
