[jira] [Commented] (LENS-1538) Lens HA changes
[ https://issues.apache.org/jira/browse/LENS-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16754826#comment-16754826 ] Hudson commented on LENS-1538: -- SUCCESS: Integrated in Jenkins build Lens-Commit #1494 (See [https://builds.apache.org/job/Lens-Commit/1494/]) LENS-1538 : Lens changes for HA to persist and restore sessions and (amareshwari: rev f64a0f5984216a61f0a2c5ef90510a88390acde0) * (edit) lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java * (edit) lens-server/src/test/java/org/apache/lens/server/query/TestLensDAO.java * (edit) lens-server/src/main/java/org/apache/lens/server/query/QueryServiceResource.java * (edit) lens-server-api/src/main/java/org/apache/lens/server/api/query/QueryExecutionService.java * (edit) lens-server/src/main/java/org/apache/lens/server/query/LensServerDAO.java * (edit) lens-server/src/main/java/org/apache/lens/server/metastore/CubeMetastoreServiceImpl.java * (edit) lens-client/src/test/java/org/apache/lens/server/MockQueryExecutionServiceImpl.java * (edit) lens-server/src/main/java/org/apache/lens/server/session/HiveSessionService.java * (add) lens-server/src/test/java/org/apache/lens/server/TestBaseLensService.java * (edit) lens-server/src/main/java/org/apache/lens/server/BaseLensService.java > Lens HA changes > --- > > Key: LENS-1538 > URL: https://issues.apache.org/jira/browse/LENS-1538 > Project: Apache Lens > Issue Type: Improvement > Components: server >Reporter: Ankit Kailaswar >Assignee: Ankit Kailaswar >Priority: Major > Fix For: 2.8 > > Attachments: Lens-1538_8.diff, Lens-1538_9.diff > > > # > h1. Implementation > # > h2. Session > # We need to persist all client session in database. > # This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > # For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > if > session is present in mysql and not present in lens server’s in memory > map(session manager) then it should create a lens session with same session > id and add it to in memory map > else If > session is not present in mysql and present in server’s in memory map then it > signifies that user have initiated close session from other host and we will > need to remove this session from current host’s in memory map as well > else if > session is not present in mysql and in server’s in memory map then return > “invalid session”. > # Whenever session expired/closed then we need to remove this session from > lens server’s in memory map and mysql. > > # > h2. *Query* > # For sync queries lens server will close connection as soon as it is > stopped or failed with appropriate failure message. Client will be retrying > the query. > # For async-light queries and async-heavy queries lens server takes care of > rescheduling all queries which were in running or queued state at the time of > restart. we can use this to make query ids available with both servers. > # For async queries we need to persist all non finished async user queries > in mysql server in new table current_query. > # If user executes query on host1 and it goes down and nginx points to host2 > then user should be able to poll on query status. Lens server will first > check query id in its inmemory maps if it is not present then it will check > in “finished_query” table in mysql else in “current_query” table in mysql. In > this case we will continue showing old status of query since it is scheduled > by host1. As soon as host1 comes up it will reschedule these queries and will > change the status. Optionally we can have host2 to move these queries in > “allqueries” map of its query service which will take care of recovering, in > this case we don't need to wait for host1 to come up and reschedule. > # While moving query from ”finished_quey” map to “finished_query” table in > mysql we will need to remove it from “current_query” tables as well. > # This flow will remain same for request ids/query ids generated for query > plan/estimate. > > # > h1. Issue addressed > # Lens downtime will be zero in case of deployment and if primary fail. > # User will not have to create a new session whenever switch happens > # User will be able to get status of all async queries seamlessly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1538) Lens HA changes
[ https://issues.apache.org/jira/browse/LENS-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated LENS-1538: -- Resolution: Fixed Fix Version/s: 2.8 Status: Resolved (was: Patch Available) Attached patch from review board here and committed the same. Thanks [~ankitkailaswar] for the contribution ! > Lens HA changes > --- > > Key: LENS-1538 > URL: https://issues.apache.org/jira/browse/LENS-1538 > Project: Apache Lens > Issue Type: Improvement > Components: server >Reporter: Ankit Kailaswar >Assignee: Ankit Kailaswar >Priority: Major > Fix For: 2.8 > > Attachments: Lens-1538_8.diff, Lens-1538_9.diff > > > # > h1. Implementation > # > h2. Session > # We need to persist all client session in database. > # This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > # For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > if > session is present in mysql and not present in lens server’s in memory > map(session manager) then it should create a lens session with same session > id and add it to in memory map > else If > session is not present in mysql and present in server’s in memory map then it > signifies that user have initiated close session from other host and we will > need to remove this session from current host’s in memory map as well > else if > session is not present in mysql and in server’s in memory map then return > “invalid session”. > # Whenever session expired/closed then we need to remove this session from > lens server’s in memory map and mysql. > > # > h2. *Query* > # For sync queries lens server will close connection as soon as it is > stopped or failed with appropriate failure message. Client will be retrying > the query. > # For async-light queries and async-heavy queries lens server takes care of > rescheduling all queries which were in running or queued state at the time of > restart. we can use this to make query ids available with both servers. > # For async queries we need to persist all non finished async user queries > in mysql server in new table current_query. > # If user executes query on host1 and it goes down and nginx points to host2 > then user should be able to poll on query status. Lens server will first > check query id in its inmemory maps if it is not present then it will check > in “finished_query” table in mysql else in “current_query” table in mysql. In > this case we will continue showing old status of query since it is scheduled > by host1. As soon as host1 comes up it will reschedule these queries and will > change the status. Optionally we can have host2 to move these queries in > “allqueries” map of its query service which will take care of recovering, in > this case we don't need to wait for host1 to come up and reschedule. > # While moving query from ”finished_quey” map to “finished_query” table in > mysql we will need to remove it from “current_query” tables as well. > # This flow will remain same for request ids/query ids generated for query > plan/estimate. > > # > h1. Issue addressed > # Lens downtime will be zero in case of deployment and if primary fail. > # User will not have to create a new session whenever switch happens > # User will be able to get status of all async queries seamlessly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (LENS-1538) Lens HA changes
[ https://issues.apache.org/jira/browse/LENS-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated LENS-1538: -- Attachment: Lens-1538_9.diff > Lens HA changes > --- > > Key: LENS-1538 > URL: https://issues.apache.org/jira/browse/LENS-1538 > Project: Apache Lens > Issue Type: Improvement > Components: server >Reporter: Ankit Kailaswar >Assignee: Ankit Kailaswar >Priority: Major > Attachments: Lens-1538_8.diff, Lens-1538_9.diff > > > # > h1. Implementation > # > h2. Session > # We need to persist all client session in database. > # This involve persisting client session’s id and configuration in mysql > server. persisting entire session object is not required. We need to do this > as first thing while creating user session. > # For each request from client lens server should not validate session id > from its in memory map. It should rather validate it from persisted session > in mysql server. > if > session is present in mysql and not present in lens server’s in memory > map(session manager) then it should create a lens session with same session > id and add it to in memory map > else If > session is not present in mysql and present in server’s in memory map then it > signifies that user have initiated close session from other host and we will > need to remove this session from current host’s in memory map as well > else if > session is not present in mysql and in server’s in memory map then return > “invalid session”. > # Whenever session expired/closed then we need to remove this session from > lens server’s in memory map and mysql. > > # > h2. *Query* > # For sync queries lens server will close connection as soon as it is > stopped or failed with appropriate failure message. Client will be retrying > the query. > # For async-light queries and async-heavy queries lens server takes care of > rescheduling all queries which were in running or queued state at the time of > restart. we can use this to make query ids available with both servers. > # For async queries we need to persist all non finished async user queries > in mysql server in new table current_query. > # If user executes query on host1 and it goes down and nginx points to host2 > then user should be able to poll on query status. Lens server will first > check query id in its inmemory maps if it is not present then it will check > in “finished_query” table in mysql else in “current_query” table in mysql. In > this case we will continue showing old status of query since it is scheduled > by host1. As soon as host1 comes up it will reschedule these queries and will > change the status. Optionally we can have host2 to move these queries in > “allqueries” map of its query service which will take care of recovering, in > this case we don't need to wait for host1 to come up and reschedule. > # While moving query from ”finished_quey” map to “finished_query” table in > mysql we will need to remove it from “current_query” tables as well. > # This flow will remain same for request ids/query ids generated for query > plan/estimate. > > # > h1. Issue addressed > # Lens downtime will be zero in case of deployment and if primary fail. > # User will not have to create a new session whenever switch happens > # User will be able to get status of all async queries seamlessly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
