yantzu commented on issue #193: [LIVY-621]add dynamic service discovery for thrift server URL: https://github.com/apache/incubator-livy/pull/193#issuecomment-521081313 Hi @mgaido91 @jerryshao, thanks a lot for your comments! If I understand correctly, "cluster" you mentioned should be something that communicate with each other and be able to transfer task from failed livy instance to active livy instance. There are SessionStore code in livy, I am not sure if it is related to "cluster". However I think, no matter if "cluster" is supported, service discovery is necessary. Some considerations : - This PR is to make livy thrift server compatibility with hiveserver2, hiveserver2 has no "cluster" neither, but it works very well. - Hiveserver2 is quite stable, sometimes it may down, but we just rerun failed tasks in another hiveserver. And from architecture overview, livy should be more stable than hiveserver2. - Hiveserver2 jdbc/beeline has a RoundRobin based client rebalance mechanism - Service discovery can hide backend server instances, and can enable HA. We have dozens of hiveserver2 in product deployment, it will help our users to move smoothly from hive to spark with service discovery, because it is almost impossible to let users to know all of these server instances. Please feel free to advice.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
