Hi William, wankun, Actually, I don't think it's a bug for Griffin. `sparkProperties.json` is a config file which can be replaced by user in external location. We have already supported such function with `external.config.location` in `application.properties` file.
Thanks, Kevin On Wed, Sep 18, 2019 at 7:30 AM ASF GitHub Bot (Jira) <j...@apache.org> wrote: > > [ > https://issues.apache.org/jira/browse/GRIFFIN-290?focusedWorklogId=314053&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-314053 > ] > > ASF GitHub Bot logged work on GRIFFIN-290: > ------------------------------------------ > > Author: ASF GitHub Bot > Created on: 17/Sep/19 23:29 > Start Date: 17/Sep/19 23:29 > Worklog Time Spent: 10m > Work Description: asfgit commented on pull request #534: > [GRIFFIN-290]Fix bug for submitting job to livy > URL: https://github.com/apache/griffin/pull/534 > > > > > ---------------------------------------------------------------- > This is an automated message from the Apache Git Service. > To respond to the message, please log on to GitHub and use the > URL above to go to the specific comment. > > For queries about this service, please contact Infrastructure at: > us...@infra.apache.org > > > Issue Time Tracking > ------------------- > > Worklog Id: (was: 314053) > Time Spent: 50m (was: 40m) > > > Fix bug for submitting job to livy > > ---------------------------------- > > > > Key: GRIFFIN-290 > > URL: https://issues.apache.org/jira/browse/GRIFFIN-290 > > Project: Griffin > > Issue Type: Task > > Reporter: wan kun > > Priority: Major > > Fix For: 0.6.0 > > > > Time Spent: 50m > > Remaining Estimate: 0h > > > > When griffin submit multiple DQ jobs to livy, the http parameter `name` > is always griffin. > > So livy will reject them. > > job request : > > > > {quote}[owner: null, request: [proxyUser: None, file: > hdfs://nameservice-standby/user/kun.wan/measure-0.6.0-SNAPSHOT.jar, > > args: { > > "spark" : > > Unknown macro: \{ "log.level" } > > , > > "sinks" : [ > > Unknown macro: \{ .... } > > ], > > "griffin.checkpoint" : [ ] > > },{ > > "measure.type" : "griffin", > > "id" : 5202, > > "name" : "spu_null_check", > > "owner" : "test", > > "description" : "check null value for store and category", > > "deleted" : false, > > "timestamp" : 1568195100000, > > "dq.type" : "PROFILING", > > "sinks" : [ "ELASTICSEARCH", "HDFS" ], > > "process.type" : "BATCH", > > "rule.description" : > > , > > "data.sources" : [ > > Unknown macro: \{ .... } > > ], > > "evaluate.rule" : > > , > > "measure.type" : "griffin" > > },raw,raw, driverMemory: 1g, executorMemory: 6g, executorCores: 2, > numExecutors: 6, queue: root.users.kun_dot_wan, {color:#ff0000}*name: > griffin*{color}]] > > {quote} > > > > livy Response : > > {quote}400 Bad Request > > [Date:"Thu, 12 Sep 2019 10:00:00 GMT", > Content-Type:"application/json;charset=utf-8", Content-Length:"47", > Server:"Jetty(9.3.24.v20180605)"] > > {"msg":"Duplicate session name: Some(griffin)"} > > {quote} > > > > -- > This message was sent by Atlassian Jira > (v8.3.4#803005) >
