[jira] [Commented] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900602#comment-16900602 ] Saisai Shao commented on LIVY-616: -- [~oshevchenko] you can either push an empty commit or reopen the PR to trigger travis test. I'm curious about the scenarios of this proposal. Seems like you don't want to maintain the Livy address in client side, instead asking ZK to get Livy server address. I can see there's some advantages with this proposal, but instead you have to maintain ZK address, what's difference between maintaining either? Typically server discovery mechanism is mainly used in HA scenario, where you can get the active master address from ZK, but current Livy doesn't support HA. So IMHO, I don't see it super useful for now. > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-632) Implement SetClientInfo metadata operation
Yiheng Wang created LIVY-632: Summary: Implement SetClientInfo metadata operation Key: LIVY-632 URL: https://issues.apache.org/jira/browse/LIVY-632 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support SetClientInfo metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-631) Implement GetQueryId metadata operation
Yiheng Wang created LIVY-631: Summary: Implement GetQueryId metadata operation Key: LIVY-631 URL: https://issues.apache.org/jira/browse/LIVY-631 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support GetSchemas metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (LIVY-631) Implement GetQueryId metadata operation
[ https://issues.apache.org/jira/browse/LIVY-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiheng Wang updated LIVY-631: - Description: We should support GetQueryId metadata operation in Livy thrift server. (was: We should support GetSchemas metadata operation in Livy thrift server.) > Implement GetQueryId metadata operation > --- > > Key: LIVY-631 > URL: https://issues.apache.org/jira/browse/LIVY-631 > Project: Livy > Issue Type: Sub-task > Components: Thriftserver >Reporter: Yiheng Wang >Assignee: Yiheng Wang >Priority: Minor > > We should support GetQueryId metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-630) Implement RenewDelegationToken metadata operation
Yiheng Wang created LIVY-630: Summary: Implement RenewDelegationToken metadata operation Key: LIVY-630 URL: https://issues.apache.org/jira/browse/LIVY-630 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support RenewDelegationToken metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-629) Implement CancelDelegationToken metadata operation
Yiheng Wang created LIVY-629: Summary: Implement CancelDelegationToken metadata operation Key: LIVY-629 URL: https://issues.apache.org/jira/browse/LIVY-629 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support CancelDelegationToken metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-628) Implement GetDelegationToken metadata operation
Yiheng Wang created LIVY-628: Summary: Implement GetDelegationToken metadata operation Key: LIVY-628 URL: https://issues.apache.org/jira/browse/LIVY-628 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support GetDelegationToken metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-627) Implement GetCrossReference metadata operation
Yiheng Wang created LIVY-627: Summary: Implement GetCrossReference metadata operation Key: LIVY-627 URL: https://issues.apache.org/jira/browse/LIVY-627 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support GetCrossReference metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-626) Implement GetPrimaryKeys metadata operation
Yiheng Wang created LIVY-626: Summary: Implement GetPrimaryKeys metadata operation Key: LIVY-626 URL: https://issues.apache.org/jira/browse/LIVY-626 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support GetPrimaryKeys metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-624) Implement GetColumns metadata operation
Yiheng Wang created LIVY-624: Summary: Implement GetColumns metadata operation Key: LIVY-624 URL: https://issues.apache.org/jira/browse/LIVY-624 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support GetColumns metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-625) Implement GetFunctions metadata operation
Yiheng Wang created LIVY-625: Summary: Implement GetFunctions metadata operation Key: LIVY-625 URL: https://issues.apache.org/jira/browse/LIVY-625 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support GetFunctions metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-623) Implement GetTables metadata operation
Yiheng Wang created LIVY-623: Summary: Implement GetTables metadata operation Key: LIVY-623 URL: https://issues.apache.org/jira/browse/LIVY-623 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang We should support GetTables metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Assigned] (LIVY-575) Implement missing metadata operations
[ https://issues.apache.org/jira/browse/LIVY-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiheng Wang reassigned LIVY-575: Assignee: Yiheng Wang > Implement missing metadata operations > - > > Key: LIVY-575 > URL: https://issues.apache.org/jira/browse/LIVY-575 > Project: Livy > Issue Type: Improvement > Components: Thriftserver >Reporter: Marco Gaido >Assignee: Yiheng Wang >Priority: Minor > > Many metadata operations (eg. table list retrieval, schema retrieval, ...) > are currently not implemented. We should implement them. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (LIVY-622) Implement GetSchemas metadata operation
[ https://issues.apache.org/jira/browse/LIVY-622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiheng Wang updated LIVY-622: - Description: We should support GetSchemas metadata operation in Livy thrift server. > Implement GetSchemas metadata operation > --- > > Key: LIVY-622 > URL: https://issues.apache.org/jira/browse/LIVY-622 > Project: Livy > Issue Type: Sub-task > Components: Thriftserver >Reporter: Yiheng Wang >Assignee: Yiheng Wang >Priority: Minor > > We should support GetSchemas metadata operation in Livy thrift server. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (LIVY-622) Implement GetSchemas metadata operation
Yiheng Wang created LIVY-622: Summary: Implement GetSchemas metadata operation Key: LIVY-622 URL: https://issues.apache.org/jira/browse/LIVY-622 Project: Livy Issue Type: Sub-task Components: Thriftserver Reporter: Yiheng Wang Assignee: Yiheng Wang -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (LIVY-616) Livy Server discovery
[ https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898761#comment-16898761 ] Oleksandr Shevchenko edited comment on LIVY-616 at 8/5/19 7:25 AM: --- Refactored code to provide a more flexible mechanism for configuration: {noformat} # Livy Server discovery # ZooKeeper quorum URLs, e.g. host1:port1,host2:port2 # livy.zookeeper.url = # Name of base Livy znode. Default livy # livy.zookeeper.namespace = livy # Name of Livy Server znode. Uses livy.zookeeper.namespace as parent. # By default will be /livy/server.uri # livy.server.zookeeper.namespace = server.uri # Number of trials to establish the connection to ZooKeeper quorum # livy.server.zookeeper.connection.max.retries = 3 # Sleep time between connection retries to ZooKeeper quorum # livy.server.zookeeper.connection.retry.interval.ms = 500 {noformat} was (Author: oshevchenko): Refactored configuration to provide a more flexible mechanism for configuration: {noformat} # Livy Server discovery # ZooKeeper quorum URLs, e.g. host1:port1,host2:port2 # livy.zookeeper.url = # Name of base Livy znode. Default livy # livy.zookeeper.namespace = livy # Name of Livy Server znode. Uses livy.zookeeper.namespace as parent. # By default will be /livy/server.uri # livy.server.zookeeper.namespace = server.uri # Number of trials to establish the connection to ZooKeeper quorum # livy.server.zookeeper.connection.max.retries = 3 # Sleep time between connection retries to ZooKeeper quorum # livy.server.zookeeper.connection.retry.interval.ms = 500 {noformat} > Livy Server discovery > - > > Key: LIVY-616 > URL: https://issues.apache.org/jira/browse/LIVY-616 > Project: Livy > Issue Type: Improvement > Components: Server >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: Livy Server discovery spec.pdf > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there isn't a way to get Livy Server URI by the client without > setting Livy address explicitly to livy.conf. A client should set > "livy.server.host" variable and then get it via LivyConf. The same behavior > if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It > very inconvenient when we install Livy packages on couple nodes and don't > know where exactly Livy Server will be started e.g. by Ambari or Cloudera > Manager. Also, in this case, we need to have Livy configuration files on a > node where we want to get Livy address. > It will be very helpful if we will add Livy Server address to Zookeeper and > expose API for clients to get Livy URL to use it in client code for REST > calls. > Livy already supports state saving in Zookeeper but I don't see that we store > Livy server address somewhere. Before starting investigating and > implementation I want to ask here about this. > Please, correct me if I missed something. > Any comments will be highly appreciated! -- This message was sent by Atlassian JIRA (v7.6.14#76016)