Hi Vivek, We are running multiple instances of Livy on our clusters. Our users can create and access jobs on any of the Livy instances.
To answer your particular question *1. Is this feature available in the 0.3 release?* No. The feature is not available out of the box–we modified Livy to add this feature. *2. How would I name/number the multiple instances I bring up?* The load balancer knows all the instances. Users only interact with the load balancer. *3. How does one load balance and send requests across the multiple instances?* Livy instances do not know that requests come from a load balancer. Each processes requests it receives and updates a shared *"session store"*, which we implemented on top of ZooKeeper. *4. Does Livy have a heartbeat mechanism to understand which or how many instances are up?* No. There is a heartbeat mechanism in Livy, but we do not using it to detect live/dead Livy instances. Each instance only reacts to the updates to the ZooKeeper *"session store"*, which is shared by all instances of Livy. Thanks, Meisam On Wed, Jul 26, 2017 at 1:36 AM Vivek <[email protected]> wrote: > Ok.so if I start multiple instances how will I know which instance to send > the request to? > And multiple instances would then be only controlled by the port id? > > > Sent from my iPhone > > On 26 Jul 2017, at 4:04 PM, Saisai Shao <[email protected]> wrote: > > Current Livy doesn't support the things you mentioned here. You can start > multiple Livy in the cluster, but each LivyServer is a standalone service > doesn't aware the existence of others. > > On Wed, Jul 26, 2017 at 10:27 AM, Vivek <[email protected]> wrote: > >> Hi, >> >> We are now considering moving into a uat environment using Livy at my >> company. >> >> Has anyone implemented multiple Livy instances on a single cluster with >> load balancing? >> >> A few questions. >> 1. Is this feature available in the 0.3 release? >> 2. How would I name/number the multiple instances I bring up? >> 3. How does one load balance and send requests across the multiple >> instances? >> 4. Does Livy have a heartbeat mechanism to understand which or how many >> instances are up? >> >> Any answers would be appreciated. >> >> Regards >> Vivek >> >> >> Sent from my iPhone >> > >
