[
https://issues.apache.org/jira/browse/LIVY-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900706#comment-16900706
]
Oleksandr Shevchenko edited comment on LIVY-616 at 8/6/19 7:27 AM:
-------------------------------------------------------------------
Hi [~jerryshao].
Thanks for reviewing it.
{quote}but instead you have to maintain ZK address, what's difference between
maintaining either
{quote}
Correct, the reasonable question. I was thinking about this and tried different
configurations.
Using of ZK address more easily since a lot of other components typically use
ZK and we don't need to perform any additional configuration for our cluster
like adding env variables for Livy address and specifying an additional
property in livy.conf. It can provide a better user experience.
The aim here is simplifying configuration. If we already have ZK on our cluster
we can configure only one property to use ZK StateStore, Livy Server discovery,
Livy ThriftServer discovery, Livy HA etc, and reuse the same ZK cluster
configuration (env variables, additional conf files etc) the same way as we do
it for other components.
It's not convenient to use ZK as state store with configured ZK address and
then add an additional variable for Livy Server address.
When I started to use Livy and find that we can use ZK as state store I tried
to find the ability to get Livy Server address as well. I was surprised that
this functionality is missed and we should still to specify Livy address
manually even though we already use ZK. It will be more easily and conventional
to store Livy Server address in ZK as well, isn't it?
I thought that this functionality will be useful for Livy community.
Also, PR for this ticket includes some small refactoring to separate ZK
communication logic from StateStore logic. It might be helpful for subsequent
Livy HA implementation. I also interested in Livy Server HA functionality and
tried to make code and configurations adaptive and reusable for Livy HA.
What do you think?
was (Author: oshevchenko):
Hi [~jerryshao].
Thanks for reviewing it.
{quote}but instead you have to maintain ZK address, what's difference between
maintaining either
{quote}
Correct, the reasonable question. I thought about it and tried different
configurations.
Using of ZK address more easily since a lot of other components typically use
ZK and we don't need to perform any additional configuration for our cluster
like adding env variables for Livy address and specifying an additional
property in livy.conf. It can provide a better user experience.
The aim here is simplifying configuration. If we already have ZK on our cluster
we can configure only one property to use ZK StateStore, Livy Server discovery,
Livy ThriftServer discovery, Livy HA etc, and reuse the same ZK cluster
configuration (env variables, additional conf files etc) the same way as we do
it for other components.
It's not convenient to use ZK as state store with configured ZK address and
then add an additional variable for Livy Server address.
When I started to use Livy and find that we can use ZK as state store I tried
to find the ability to get Livy Server address as well. I was surprised that
this functionality is missed and we should still to specify Livy address
manually even though we already use ZK. It will be more easily and conventional
to store Livy Server address in ZK as well, isn't it?
I thought that this functionality will be useful for Livy community.
Also, PR for this ticket includes some small refactoring to separate ZK
communication logic from StateStore logic. It might be helpful for subsequent
Livy HA implementation. I also interested in Livy Server HA functionality and
tried to make code and configurations adaptive and reusable for Livy HA.
What do you think?
> Livy Server discovery
> ---------------------
>
> Key: LIVY-616
> URL: https://issues.apache.org/jira/browse/LIVY-616
> Project: Livy
> Issue Type: Improvement
> Components: Server
> Reporter: Oleksandr Shevchenko
> Priority: Major
> Attachments: Livy Server discovery spec.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, there isn't a way to get Livy Server URI by the client without
> setting Livy address explicitly to livy.conf. A client should set
> "livy.server.host" variable and then get it via LivyConf. The same behavior
> if you want to use Livy with Zeppelin, we need to set "zeppelin.livy.url". It
> very inconvenient when we install Livy packages on couple nodes and don't
> know where exactly Livy Server will be started e.g. by Ambari or Cloudera
> Manager. Also, in this case, we need to have Livy configuration files on a
> node where we want to get Livy address.
> It will be very helpful if we will add Livy Server address to Zookeeper and
> expose API for clients to get Livy URL to use it in client code for REST
> calls.
> Livy already supports state saving in Zookeeper but I don't see that we store
> Livy server address somewhere. Before starting investigating and
> implementation I want to ask here about this.
> Please, correct me if I missed something.
> Any comments will be highly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
