Re: Review Request 69554: Lens HA changes

[Ankit Kailaswar]

> On Jan. 7, 2019, 3:48 a.m., Amareshwari Sriramadasu wrote:
> > No unit tests added for the change introduced. Can you add unit tests?

added test cases for lensserver dao functions.


- Ankit


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211711
---


On Jan. 22, 2019, 11:08 a.m., Ankit Kailaswar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> ---
> 
> (Updated Jan. 22, 2019, 11:08 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
> 
> 
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Implementation
> 
> Session
> 
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql 
> server. persisting entire session object is not required. We need to do this 
> as first thing while creating user session.
> For each request from client lens server should not validate session id 
> from its in memory map. It should rather validate it from persisted session 
> in mysql server.
> 
> if
> 
> session is present in mysql and not present in lens server’s in memory 
> map(session manager) then it should create a lens session with same session 
> id and add it to in memory map
> 
> else If
> 
> session is not present in mysql and present in server’s in memory map then it 
> signifies that user have initiated close session from other host and we will 
> need to remove this session from current host’s in memory map as well
> 
> else if
> 
> session is not present in mysql and in server’s in memory map then return 
> “invalid session”.
> 
> Whenever session expired/closed then we need to remove this session from 
> lens server’s in memory map and mysql.
> 
>  
> 
> Query
> 
> For sync queries lens server will close connection as soon as it is 
> stopped or failed with appropriate failure message. Client will be retrying 
> the query.
> For async-light queries and async-heavy queries lens server takes care of 
> rescheduling all queries which were in running or queued state at the time of 
> restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries 
> in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to 
> host2 then user should be able to poll on query status. Lens server will 
> first check query id in its inmemory maps if it is not present then it will 
> check in “finished_query” table in mysql else in “current_query” table in 
> mysql. In this case we will continue showing old status of query since it is 
> scheduled by host1. As soon as host1 comes up it will reschedule these 
> queries and will change the status. Optionally we can have host2 to move 
> these queries in “allqueries” map of its query service which will take care 
> of recovering, in this case we don't need to wait for host1 to come up and 
> reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in 
> mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query 
> plan/estimate.
> 
>  
> 
> Issue addressed
> 
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
> 
> 
> Build Log :
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Lens Checkstyle Rules . SUCCESS [2.740s]
> [INFO] Lens .. SUCCESS [5.788s]
> [INFO] Lens API .. SUCCESS [48.711s]
> [INFO] Lens API for server and extensions  SUCCESS [33.537s]
> [INFO] Lens Cube . SUCCESS [7:43.672s]
> [INFO] Lens DB storage ... SUCCESS [29.279s]
> [INFO] Lens Query Library  SUCCESS [22.507s]
> [INFO] Lens Hive Driver .. SUCCESS [1:27.344s]
> [INFO] Lens Driver for JDBC .. SUCCESS [1:24.416s]
> [INFO] Lens Elastic Search Driver  SUCCESS [44.067s]
> [INFO] Lens Server ... SUCCESS 
> [12:19.915s]
> [INFO] Lens client ... SUCCESS [1:19.368s]
> 

Re: Review Request 69554: Lens HA changes

[Ankit Kailaswar]

> On Jan. 7, 2019, 3:47 a.m., Amareshwari Sriramadasu wrote:
> > lens-server/src/main/java/org/apache/lens/server/query/QueryExecutionServiceImpl.java
> > Lines 1176 (patched)
> > 
> >
> > no counter for failures?

added


- Ankit


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/#review211710
---


On Jan. 22, 2019, 11:08 a.m., Ankit Kailaswar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69554/
> ---
> 
> (Updated Jan. 22, 2019, 11:08 a.m.)
> 
> 
> Review request for lens, Amareshwari Sriramadasu and Rajitha R.
> 
> 
> Bugs: LENS-1538
> https://issues.apache.org/jira/browse/LENS-1538
> 
> 
> Repository: lens
> 
> 
> Description
> ---
> 
> Implementation
> 
> Session
> 
> We need to persist all client session in database.
> This involve persisting client session’s id and configuration in mysql 
> server. persisting entire session object is not required. We need to do this 
> as first thing while creating user session.
> For each request from client lens server should not validate session id 
> from its in memory map. It should rather validate it from persisted session 
> in mysql server.
> 
> if
> 
> session is present in mysql and not present in lens server’s in memory 
> map(session manager) then it should create a lens session with same session 
> id and add it to in memory map
> 
> else If
> 
> session is not present in mysql and present in server’s in memory map then it 
> signifies that user have initiated close session from other host and we will 
> need to remove this session from current host’s in memory map as well
> 
> else if
> 
> session is not present in mysql and in server’s in memory map then return 
> “invalid session”.
> 
> Whenever session expired/closed then we need to remove this session from 
> lens server’s in memory map and mysql.
> 
>  
> 
> Query
> 
> For sync queries lens server will close connection as soon as it is 
> stopped or failed with appropriate failure message. Client will be retrying 
> the query.
> For async-light queries and async-heavy queries lens server takes care of 
> rescheduling all queries which were in running or queued state at the time of 
> restart. we can use this to make query ids available with both servers.
> For async queries we need to persist all non finished async user queries 
> in mysql server in new table current_query.
> If user executes query on host1 and it goes down and nginx points to 
> host2 then user should be able to poll on query status. Lens server will 
> first check query id in its inmemory maps if it is not present then it will 
> check in “finished_query” table in mysql else in “current_query” table in 
> mysql. In this case we will continue showing old status of query since it is 
> scheduled by host1. As soon as host1 comes up it will reschedule these 
> queries and will change the status. Optionally we can have host2 to move 
> these queries in “allqueries” map of its query service which will take care 
> of recovering, in this case we don't need to wait for host1 to come up and 
> reschedule.
> While moving query from ”finished_quey” map to “finished_query” table in 
> mysql we will need to remove it from “current_query” tables as well.
> This flow will remain same for request ids/query ids generated for query 
> plan/estimate.
> 
>  
> 
> Issue addressed
> 
> Lens downtime will be zero in case of deployment and if primary fail.
> User will not have to create a new session whenever switch happens
> User will be able to get status of all async queries seamlessly.
> 
> 
> Build Log :
> [INFO] 
> 
> [INFO] Reactor Summary:
> [INFO] 
> [INFO] Lens Checkstyle Rules . SUCCESS [2.740s]
> [INFO] Lens .. SUCCESS [5.788s]
> [INFO] Lens API .. SUCCESS [48.711s]
> [INFO] Lens API for server and extensions  SUCCESS [33.537s]
> [INFO] Lens Cube . SUCCESS [7:43.672s]
> [INFO] Lens DB storage ... SUCCESS [29.279s]
> [INFO] Lens Query Library  SUCCESS [22.507s]
> [INFO] Lens Hive Driver .. SUCCESS [1:27.344s]
> [INFO] Lens Driver for JDBC .. SUCCESS [1:24.416s]
> [INFO] Lens Elastic Search Driver  SUCCESS [44.067s]
> [INFO] Lens Server 

Re: Review Request 69554: Lens HA changes

[Ankit Kailaswar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69554/
---

(Updated Jan. 22, 2019, 11:08 a.m.)


Review request for lens, Amareshwari Sriramadasu and Rajitha R.


Bugs: LENS-1538
https://issues.apache.org/jira/browse/LENS-1538


Repository: lens


Description (updated)
---

Implementation

Session

We need to persist all client session in database.
This involve persisting client session’s id and configuration in mysql 
server. persisting entire session object is not required. We need to do this as 
first thing while creating user session.
For each request from client lens server should not validate session id 
from its in memory map. It should rather validate it from persisted session in 
mysql server.

if

session is present in mysql and not present in lens server’s in memory 
map(session manager) then it should create a lens session with same session id 
and add it to in memory map

else If

session is not present in mysql and present in server’s in memory map then it 
signifies that user have initiated close session from other host and we will 
need to remove this session from current host’s in memory map as well

else if

session is not present in mysql and in server’s in memory map then return 
“invalid session”.

Whenever session expired/closed then we need to remove this session from 
lens server’s in memory map and mysql.

 

Query

For sync queries lens server will close connection as soon as it is stopped 
or failed with appropriate failure message. Client will be retrying the query.
For async-light queries and async-heavy queries lens server takes care of 
rescheduling all queries which were in running or queued state at the time of 
restart. we can use this to make query ids available with both servers.
For async queries we need to persist all non finished async user queries in 
mysql server in new table current_query.
If user executes query on host1 and it goes down and nginx points to host2 
then user should be able to poll on query status. Lens server will first check 
query id in its inmemory maps if it is not present then it will check in 
“finished_query” table in mysql else in “current_query” table in mysql. In this 
case we will continue showing old status of query since it is scheduled by 
host1. As soon as host1 comes up it will reschedule these queries and will 
change the status. Optionally we can have host2 to move these queries in 
“allqueries” map of its query service which will take care of recovering, in 
this case we don't need to wait for host1 to come up and reschedule.
While moving query from ”finished_quey” map to “finished_query” table in 
mysql we will need to remove it from “current_query” tables as well.
This flow will remain same for request ids/query ids generated for query 
plan/estimate.

 

Issue addressed

Lens downtime will be zero in case of deployment and if primary fail.
User will not have to create a new session whenever switch happens
User will be able to get status of all async queries seamlessly.


Build Log :
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Lens Checkstyle Rules . SUCCESS [2.740s]
[INFO] Lens .. SUCCESS [5.788s]
[INFO] Lens API .. SUCCESS [48.711s]
[INFO] Lens API for server and extensions  SUCCESS [33.537s]
[INFO] Lens Cube . SUCCESS [7:43.672s]
[INFO] Lens DB storage ... SUCCESS [29.279s]
[INFO] Lens Query Library  SUCCESS [22.507s]
[INFO] Lens Hive Driver .. SUCCESS [1:27.344s]
[INFO] Lens Driver for JDBC .. SUCCESS [1:24.416s]
[INFO] Lens Elastic Search Driver  SUCCESS [44.067s]
[INFO] Lens Server ... SUCCESS [12:19.915s]
[INFO] Lens client ... SUCCESS [1:19.368s]
[INFO] Lens CLI .. SUCCESS [1:19.672s]
[INFO] Lens Examples . SUCCESS [7.287s]
[INFO] Lens Ship Jars to Distributed Cache ... SUCCESS [0.661s]
[INFO] Lens Distribution . SUCCESS [10.625s]
[INFO] Lens ML Lib ... SUCCESS [53.247s]
[INFO] Lens ML Ext Distribution .. SUCCESS [1.418s]
[INFO] Lens Regression ... SUCCESS [9.296s]
[INFO] Lens UI ... SUCCESS [28.750s]
[INFO] Lens Contrib .. SUCCESS [0.229s]
[INFO] 

Re: [VOTE] Lens - Code Commit using Pull Requests

[Raju Bairishetti]

+1

On Tue, 22 Jan 2019, 14:01 Sushil Mohanty  +1
>
> On Tue, Jan 22, 2019 at 1:15 PM Raghavendra Singh <
> raghavendra.si...@inmobi.com> wrote:
>
> > +1
> >
> > On Tue, Jan 22, 2019, 9:17 AM Puneet Gupta  wrote:
> >
> > > +1.
> > >
> > > On Tue, 22 Jan, 2019, 9:08 AM Amareshwari Sriramdasu <
> > > amareshw...@apache.org
> > > wrote:
> > >
> > > > Rajitha,
> > > >
> > > > Can we make sure emails for the new pull requests and the comments on
> > > pull
> > > > requests come to dev@lens.apache.org as part of this change?
> > > >
> > > > Thanks
> > > >
> > > > On Sun, Jan 20, 2019 at 10:29 AM Amareshwari Sriramdasu <
> > > > amareshw...@apache.org> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Sat, Jan 19, 2019 at 2:54 PM Rajat Khandelwal <
> pro...@apache.org>
> > > > > wrote:
> > > > >
> > > > >> +1
> > > > >>
> > > > >
> > > >
> > >
> >
> > --
> > _
> > The
> > information contained in this communication is intended solely for the
> use
> > of the individual or entity to whom it is addressed and others authorized
> > to receive it. It may contain confidential or legally privileged
> > information. If you are not the intended recipient you are hereby
> notified
> > that any disclosure, copying, distribution or taking any action in
> > reliance
> > on the contents of this information is strictly prohibited and may be
> > unlawful. If you have received this communication in error, please notify
> > us immediately by responding to this email and then delete it from your
> > system. The firm is neither liable for the proper and complete
> > transmission
> > of the information contained in this communication nor for any delay in
> > its
> > receipt.
> >
>
>
> --
> -Sushil
>