[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015606#comment-17015606 ] Yiheng Wang commented on LIVY-718: -- I updated the design doc based on the recent discussions in the JIRA. The major changes are: # Refine the solution architecture section # Add a new allocateServer method to allocator interface # Add details for node-session mapping allocation method [~mgaido] # update getAllSession [~mgaido] # Refine the section of comparing client-side routing and server-side routing # Add a new section "Load Balancer", which gives example how to put livy servers behind a load balancer when using client-side routing [~bikassaha] [~meisam] # Add a new section "Session Recover", which describe we recover a session object in a lazy way(when a request for that session arrives, which can be leveraged in multi-designated server solution) [~bikassaha] # Remove the session recover when there's server failover # Add multi-designate server to non-goal and add a new section "Multi-designate Server Solution Extension" to discuss how to continue to implement a multi-designate server solution from the current one [~bikassaha] > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014846#comment-17014846 ] Saisai Shao commented on LIVY-718: -- Active-active HA doesn't only address scalability issue, but also high availability. Personally I don't feel super useful for active-standby HA about Livy. Usually it is because master node has large amount of state to maintain, so it is hard to implement active-active HA with consistency. If this is not the case, then active-active HA is better both for HA and scalability. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013965#comment-17013965 ] Saisai Shao commented on LIVY-718: -- [~bikassaha] The merged two sub-tasks, they're actually required by both solutions (the one you proposed and another [~yihengw] proposed). That's why I merged it beforehand, they're not the key differences for two solutions. Actually the proposal [~yihengw] made is just the mid-term solution compared to stateless Livy Server, the key difference is to: 1. change the time when RSCDriver and Livy Server get reconnection. 2. Refactor the most of the current code to make Livy Server stateless. I'm more concerned about the 2nd point, because it has lots of works to do, and could easily introduce regressions. So IMHO, I think we could move on with the current mid-term proposal. If someone else want to pursue a stateless solution, they could simply continue based on our current solution, that would take less efforts compared to start from scratch. Just my two cents. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013498#comment-17013498 ] Bikas Saha commented on LIVY-718: - Clearly, I am not aligned with the above or else I would not start and push on this discussion :) In my experience, code refactoring and cost is paid once initially and easier to test relative to operational complexity and runtime correctness/reliability. However, if others are onboard with the current proposal then I will not pursue this discussion further. On the voting thread, IIRC the ask had been to add more details to the design doc and align on parts where no conclusion has been reached yet. Is that done? I ask because a couple of PRs are committed already indicating that coding has started. Even if we go with the proposal in the document, having the details water tight and converged is super important for a feature like this which involves distributed state and coordination. These things are notoriously difficult to get right. So the more we can solidify the design up front the safer it will be to implement. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011909#comment-17011909 ] Yiheng Wang commented on LIVY-718: -- Hi [~bikassaha] I agree that allows multiple servers to connect to one spark driver solution is more flexible and ideal. But I think the current one designated server solution is not conflicted with it. The difference is multiple server solution brings in the state consist problem, it needs extra effort to optimize the existing Livy server code. Other parts are compatible. As you comment, the ideal solution is 1. The user sends a request to a Livy cluster 2. The request may be handled by the current server or route to another server (based on configured strategy) 3. The server handles the request. It may take some time to hydrate the state if it's the first hit For step 2, the one designated server solution strategy is always route to one same server. It doesn't allow route to different servers to avoid the state consist problem. For step 3, we can change the time of initializing the session object and RPC connection in the server. We change it from doing it when server fail/new server join to session first hit, which also works in the one designated solution. I did some dig in these days to see how to optimize livy server code to fix the state consist problem. I list some work items to achieve it. ||Component||Object||Effort|| |Yarn Job|SparkYarnApp|SparkYarnApp contains yarn application log. It is stored in the original session create server memory, which may not be access by other servers. I think it is not suitable to put it in state store. Batch session also have this problem so we cannot push it to rsc driver.| |Session|last activity time|Push to driver for interactive session. Push it to state store for batch session as it always be start time in batch session| |Interactive session|operations(statement records)|push to driver| |Interactive session|operationCount|push to driver| |Session|appId|appId is empty when session metadata is persisted to state store frist time, it is changed after a while. We need to make sure the consist across servers| |Interactive Session|Rsc driver url|same with the above| |Session|session state|Session state is updated by SparkYarnApp thread. There's already some inconsistent between livy server and yarn. If there're multiple servers, the inconsistent will be amplified, as the thread check time is different across servers. One solution is query yarn but it will make many query much longer. Another solution is put it in state store. It also add overhead| |Thrift|Fetched row count|push to driver| |Thrift|mapping from thrift session to livy session|put to state store| |Thrift|operation states|push to driver, touch 1/2 thrift code| I think it's quite a lot of effort and touch existing code base which brings many risks. I suggest we make it as a separate goal. Another discussion is session id. I'm very worried the thing that it breaks the API compatible. We may need to upgrade the API version to V2. And it's a lot effort for people to migrate existing code. I strongly suggest to stick on the int session id and make it as a separate goal. Make a summary, my suggestion is: 1. Change the design to make it compatible with multi-server solution 2. Implement the one-designate solution 3. Make multi-servers solution as a separate goal and do it in another JIRA 3.1 make livy server behavior consist in the cluster(see the table) 3.2 add a new route strategy 4. Stick on int session id What do you think? > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing >
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011485#comment-17011485 ] Saisai Shao commented on LIVY-718: -- Hi [~shanyu] what is the main reason that we should have "active-standby" HA? From my understanding, looks like compared to active-active HA, active-standby HA seems not so useful, and current fail recovery could cover most of the scenarios. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17010965#comment-17010965 ] shanyu zhao commented on LIVY-718: -- LIVY-11 aims at active-standby HA, user should be able to choose active-active vs. active-standby HA through configurations. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[
https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17006882#comment-17006882
]
Bikas Saha commented on LIVY-718:
-
{quote}JDBC part maintains lots of results/metadata/information on LivyServer
{quote}
Might be ok to do it in phases where the JDBC support is not present initially
(like its mentioned in the document also).
{quote}there're M server and N session. In designate solution, there're N
connections. In stateless solution, there're M x N connections
{quote}
Yes that would be the worst case where for some reason all servers are
connected to all spark drivers. But with clients and load balancers maintaining
sticky sessions one would typically not expect that to happen. At the end of
the day the number of connections needed (ie not dropped because of inactivity)
would be proportional to the number of clients.
Also, this does not preclude the possibility of allowing for pseudo designated
servers where the clients could be redirected to preferred servers (using
similar consistent hashing). Or if a server is already connected to a driver
then other servers could redirect clients to that server. Or if a server is
overloaded then it could request the client to choose a different server
(dynamic load balancing). These variations can be allowed (over time) while at
the same time having the flexibility to not need it as servers scale up and
down based on actual client load. Requiring a single designated server
increases the complexity while not providing the same flexibility.
> Support multi-active high availability in Livy
> --
>
> Key: LIVY-718
> URL: https://issues.apache.org/jira/browse/LIVY-718
> Project: Livy
> Issue Type: Epic
> Components: RSC, Server
>Reporter: Yiheng Wang
>Priority: Major
>
> In this JIRA we want to discuss how to implement multi-active high
> availability in Livy.
> Currently, Livy only supports single node recovery. This is not sufficient in
> some production environments. In our scenario, the Livy server serves many
> notebook and JDBC services. We want to make Livy service more fault-tolerant
> and scalable.
> There're already some proposals in the community for high availability. But
> they're not so complete or just for active-standby high availability. So we
> propose a multi-active high availability design to achieve the following
> goals:
> # One or more servers will serve the client requests at the same time.
> # Sessions are allocated among different servers.
> # When one node crashes, the affected sessions will be moved to other active
> services.
> Here's our design document, please review and comment:
> https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17006078#comment-17006078 ] Yiheng Wang commented on LIVY-718: -- [~bikassaha] Compared to the designated server solution, I think stateless server solution get more accessibility by sacrificing scalability. In this background, one concern is memory. We observed that when the running session number grows to 400~500, the Livy server process consuming about 2G memory. Another concern is Livy use long connections between server and spark drivers. Say there're M server and N session. In designate solution, there're N connections. In stateless solution, there're M x N connections. I'm afraid this may bring a lot of overhead in RPC communication(e.g. serialization, routing). > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17006069#comment-17006069 ] Yiheng Wang commented on LIVY-718: -- bq. When a server fails, its sessions become unavailable until other servers are designated to handle them. This was not acceptable behavior, at least for clusters that I worked with in my previous job. [~meisam] Currently, Livy only supports single node failure recover. Do you use Livy in that cluster? If so, would you like to share your solution? > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005895#comment-17005895 ] Saisai Shao commented on LIVY-718: -- IIUC, current JDBC part maintains lots of results/metadata/information on LivyServer, I think the key point mentioned by [~bikassaha] is to make Livy Server stateless, that would be an ideal solution, but currently it requires a large amount of works. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005512#comment-17005512 ] Marco Gaido commented on LIVY-718: -- > IIRC JDBC had a REST and an RPC mode. The RPC mode might not be HA without a > fat client but perhaps the REST mode could. Does Hive JDBC support HA on the > Hive Thrift server? Then maybe the hive JDBC client now supports server side > transitions. If not, then we may have the caveat that HA won't work for such > connections. I am not super familiar with the JDBC client. Actually, JDBC has either HTTP or binary mode, but it relates only to the communication between the Livy server and the client and it is unrelated to how the Livy server interacts with Spark (ie. via RPC). > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005439#comment-17005439 ] Bikas Saha commented on LIVY-718: - There could be in-memory state in the livy server but could that be re-created from the state in the Spark driver with an initial sync operation? If not, then what additional metadata could be stored in the Spark drive to make it happen? The ideal situation would be (keeping in mind Meisam's observations) # Any livy client can hit any livy server and continue from where it was. The first time a livy server is hit for a session it may take some time to hydrate the state in case it was not done in the background. ## Note that this can happen even without any livy server failure in cases where a load balancer is running in front of the livy server and sticky sessions are not working or there is too much hot-spotting. # A livy server can (with some extra sync operation if needed) service any session from that sessions Spark driver. The only information it needs is the information of how to connect with the Spark driver. That could be stored in a reliable state store (e.g. even in a YARN application tag for YARN clusters) If we can achieve the above then the system could be much simpler to operate and work with. IIRC JDBC had a REST and an RPC mode. The RPC mode might not be HA without a fat client but perhaps the REST mode could. Does Hive JDBC support HA on the Hive Thrift server? Then maybe the hive JDBC client now supports server side transitions. If not, then we may have the caveat that HA won't work for such connections. I am not super familiar with the JDBC client. Any locking or serialization could be achieved in livy code running inside the Spark driver since its a single point where every request eventually has to come and get serialized. Good point on monitoring - if communicating with YARN is the main overhead (like Meisam observed) then perhaps the number of livy sessions being monitored is not that important. Assuming that a single YARN query can return all livy session related Spark jobs (using tags etc.) which is then iterated to determine status of livy sessions for running jobs. Until monitoring starts consuming significant cycles on the server maybe its not a problem. If the Livy server does not use the integral identifier semantically (and neither do clients) then its long term better IMO to change it now rather than have to live with its limitations (even more with HA) going forward. I am not sure why integral identifiers were chosen in the first place (in case there is some important use case for it that I am missing). It would be a breaking change for clients but its a mechanical change that may be less effort compared to a semantic change. And the benefit is huge - getting HA - which may justify that cost. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005434#comment-17005434 ] Bikas Saha commented on LIVY-718: - There could be in-memory state in the livy server but could that be re-created from the state in the Spark driver with an initial sync operation? If not, then what additional metadata could be stored in the Spark drive to make it happen? The ideal situation would be (keeping in mind Meisam's observations) # Any livy client can hit any livy server and continue from where it was. The first time a livy server is hit for a session it may take some time to hydrate the state in case it was not done in the background. ## Note that this can happen even without any livy server failure in cases where a load balancer is running in front of the livy server and sticky sessions are not working or there is too much hot-spotting. # A livy server can (with some extra sync operation if needed) service any session from that sessions Spark driver. The only information it needs is the information of how to connect with the Spark driver. That could be stored in a reliable state store (e.g. even in a YARN application tag for YARN clusters) If we can achieve the above then the system could be much simpler to operate and work with. IIRC JDBC had a REST and an RPC mode. The RPC mode might not be HA without a fat client but perhaps the REST mode could. Does Hive JDBC support HA on the Hive Thrift server? Then maybe the hive JDBC client now supports server side transitions. If not, then we may have the caveat that HA won't work for such connections. I am not super familiar with the JDBC client. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005203#comment-17005203 ] Yiheng Wang commented on LIVY-718: -- Thanks for your comments [~bikassaha] and [~meisam]. I summary the discussing points and put my comments below(please point it out if I miss something). h4. Designated Server - Is it because there are issues with multiple servers handling multiple clients to the same session? The issue includes: 1. Livy server needs to monitor spark sessions. If we remove the designated server, each server may need to monitor all sessions. It's kind of waste and there may be some inconsistent among servers. 2. Besides service data, Livy server also stores other data like the application log and last active time in memory. Such information has a higher update rate. It's not suitable to store in some state-store backend like zookeeper. 3. If multiple servers serve one session, we need to add some kind of lock mechanism to handle the concurrent state-change request(e.g. stop session) h4. Session id - I would strongly suggest deprecating the integral session id I agree that the incremental integral session ID is not necessary for Livy. For changing it to UUID, my biggest concern is compatible with the earlier API(session-id data type may be needed to change from Int to String). It's a quite big move so we choose a conservative way in the design. h4. Dependency on ZK ZK is introduced to resolve the above two problems(server status change notification and unique id generation). If we don't need the designated server and incremental session-id, I think we can remove zk. h4. Service discovery, Ease of use of the API and the number of ports in the firewall that needs to be opened for Livy HA can become a security concern I think the point here is a single URL for the Livy cluster. It depends on the first questions. If we can remove the designated server, we just need to put a load balancer before all the servers. If we keep the designated server design, we can use a http load balancer which is aware of 307 responses. Currently, we use a 307 response to route the request to the designated server on the client-side. The response can be handled automatically by some load balancer. I think the key point of the discussion is the designated server. Please let me know your suggestions for my list issues and see if we can improve the design proposal. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17005063#comment-17005063 ] Marco Gaido commented on LIVY-718: -- Thanks for your comments [~bikassaha]. I am not sure (actually I think it is not true) that all the metadata information is present in the Spark driver process and there is metadata which is frequently accessed/changed (eg. the ongoing statements for a given session) on the Livy server side (at least for the thrift part). Indeed, there are definitely metadata which are currently kept in the server memory which would need to be saved for HA sake. Hence, I am afraid that at least for the thrift case, the usage of a slow storage like HDFS would at least would require a significant revisit of the thrift part. I agree that active-active is by far the most desirable choice. I see, though, that it is not easy to implement, IMHO, because for the metadata above, it would require a distributed state store being the source of truth for that. Given your negative opinion on ZK, I hardly see any other system which would fit (a relational DB cluster maybe? but not easier to maintain than ZK for sure, I'd say). Hence I am drawn to consider that we would need to trade off things here, unless I am very mistaken on the point above: namely, the REST part has really no significant metadata on Livy server side and we keep the thrift one out of scope here. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17004874#comment-17004874 ] Meisam commented on LIVY-718: - CC [~prabhu1984] [~kpraveen] > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17004873#comment-17004873 ] Meisam commented on LIVY-718: - I also have concerns about the second point that Bikas raised, i.e. "Service availability for a given object". When a server fails, its sessions become unavailable until other servers are designated to handle them. This was not acceptable behavior, at least for clusters that I worked with in my previous job. But I'd like to share the my experience with those clusters, which ran tens of thousands of spark jobs daily. It makes me think that consistent hashing is an over-kill for Livy HA. My first observation was that, even with the the largest and busiest clusters, the whole session metadata easily fits into a Java process with 500MB of RAM and less than 20% CPU usage at pick (Yarn and ZooKeeper connections were the bottlenecks). My second observation was that, ease of use of the API was extremely important for many end users, especially for users with data science background. Ease of use was important for integrating Livy to other tools and services. My third observation was that, the number of ports in the firewall that needs to be opened for Livy HA can become a security concern. It was important that HA Livy handle any request for any session, which allows to open one single port in the Firewall for a load-balancer in front all HA Livy servers. > Support multi-active high availability in Livy > -- > > Key: LIVY-718 > URL: https://issues.apache.org/jira/browse/LIVY-718 > Project: Livy > Issue Type: Epic > Components: RSC, Server >Reporter: Yiheng Wang >Priority: Major > > In this JIRA we want to discuss how to implement multi-active high > availability in Livy. > Currently, Livy only supports single node recovery. This is not sufficient in > some production environments. In our scenario, the Livy server serves many > notebook and JDBC services. We want to make Livy service more fault-tolerant > and scalable. > There're already some proposals in the community for high availability. But > they're not so complete or just for active-standby high availability. So we > propose a multi-active high availability design to achieve the following > goals: > # One or more servers will serve the client requests at the same time. > # Sessions are allocated among different servers. > # When one node crashes, the affected sessions will be moved to other active > services. > Here's our design document, please review and comment: > https://docs.google.com/document/d/1bD3qYZpw14_NuCcSGUOfqQ0pqvSbCQsOLFuZp26Ohjc/edit?usp=sharing > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (LIVY-718) Support multi-active high availability in Livy
[ https://issues.apache.org/jira/browse/LIVY-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17004728#comment-17004728 ] Bikas Saha commented on LIVY-718: - Thank you for the taking this up and providing the detailed design document. After reviewing the document I have a few observations. There are 2 aspects to active active HA. # Higher service availability - The service continues to be available with (little to no) downtime visible to clients. At the minimum this means no loss of operational data that will negatively affect clients correctness or success of operations. Beyond that the service can try to make the disruption as transparently and low impact as possible for clients. # Higher service operations - Multiple active servers can service more client/operations. This can be done even without high service availability (1 above). In order to achieve the above several items need to be covered # High availability of service state data - This enables the work of a crashed service instance to be taken over by a different (passive or active) service instance. This can be done by storage in a reliable store (HDFS like) or by an HA state maintained within the services instances (Paxos-like). ## In Livy's case the actual service data is stored in the Spark process running the Livy session. Hence the minimum (and perhaps only) data that needs to be reliably stored is the Spark driver information (and maybe some other metadata like ACL's, ownership etc.). Any other data can be retrieved from the running Spark driver and in fact should not be duplicated since there should be only 1 source of truth. ## If the above is correct then perhaps the reliable data has a low update rate. An initial data about the session and an update for the spark driver information. This could be provided by many different systems (even slow ones with HDFS). ## To provide choice of storage, we should consider making it a pluggable such that all store operations go through the plugin interface and plugins are loaded at runtime based on configuration (class and jar). Plugins could be in-memory (current), any HDFS compatible FS, ZooKeeper etc. IIRC pluggable reliable storage is already available and we need to just check if the data stored is sufficient and perhaps trim extra data. # Service availability for a given object - This means client can continue to access/operate on a given object despite service instance failure. This in the minimum involves some other service instance being able to load that object metadata and provide operations on it. With active passive this would mean a different single server becomes the designated. OR with active active any other available server can provide the operations. ## In the latter case any server could service the request or a new designated server is chosen. The design document chooses a designated server approach (potentially via consistent hashing). I am curious why. Is it because there are issues with multiple servers handling multiple clients to the same session? If yes then what are those issues? Could these issues be handled instead such that any/multiple servers could service a session? The reason I ask this is because ### Designated server is very similar to active-passive systems and as such brings its own complications like ensuring a truly single active instance (e.g. when a previously active server comes back or was not really crashed) ### With consistent hashing or any other scheme we have to take care of ensure load balancing of objects across servers so that we do not create hot servers (handling majority of the objects) by design. Which could be the case with integer session ids and hashing. ### If we do need a designated server to be chosen then the document needs to clarify how all the cases with leader election and old leader lockout, failures will be handled. Otherwise, I am worried that we might find many issues during deployment. How will load balancing be handled? ## Session id - I would strongly suggest deprecating the integral session id. IIRC we may not have built any super important features that depend on a monotonically increasing number and so there may not be much value in continuing to support it (for the service or the client). It adds an avoidable coordination issue among multiple active servers and any bug in there would have bad consequences with duplicated session ids - which can lead to security issues where one user reads data from a session they are not entitled to. Moving to a UUID scheme removes the potential of any kind of overloading of this id and also removes the need for coordination. Also it might help with better load balancing if we have to use a designated server per object. # Service discovery - How does a client discover a server to operate on a given object. How does it discover a new server when the
