Hi Rajat, The number of connections created is usually small (less than 10? probably), I run the same query multiple times, and it only happen twice when Lens server created too many connections. I wonder if it is something unexpected.
Thanks, On Fri, Aug 5, 2016 at 9:30 PM, Puneet Gupta <[email protected]> wrote: > Hi Tao > > these connections can be created various operations that interact with > hive server like launching a query, pulling its status, etc (There is a > Status Poller thread which polls the status of hive query at regular > intervals.) > The hive connections are cached per user with an configurable expiry time > (which is 10 mins by default). You can configure it differently in hive > driver config (lens.driver.hive.hs2.connection.expiry.delay)if needed. > > I see from the logs attached by you, the connections are created for two > different users (test_user and anonymous) > > Do let me know if you need more details . > > Thanks, > Puneet Gupta > > On Fri, Aug 5, 2016 at 5:24 AM, Tao Yan <[email protected]> wrote: > >> Hi Lens Developers, >> >> I am running Lens queries on example data. And I realized sometimes Lens >> server will keep trying to connect to HiveServer2 during the query >> execution, and Lens log shows: >> >> *...* >> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >> [grizzly-http-server-28] INFO org.apache.lens.driver.hive.HiveDriver - >> New thrift connection class >> org.apache.lens.driver.hive.RemoteThriftConnection >> for thread: 112 for user: test_user *connection ID=89* on >> driver:hive/hive1 >> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >> [grizzly-http-server-28] INFO >> org.apache.lens.driver.hive.RemoteThriftConnection >> - HiveDriver connecting to HiveServer @ *******.com:10000 >> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >> [grizzly-http-server-28] INFO org.apache.hive.service.cli.t >> hrift.RetryingThriftCLIServiceClient - Connecting to *******.com:10000 >> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >> [grizzly-http-server-28] INFO org.apache.hive.service.cli.t >> hrift.RetryingThriftCLIServiceClient - Connected! >> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >> [grizzly-http-server-28] INFO >> org.apache.lens.driver.hive.RemoteThriftConnection >> - HiveDriver connected to HiveServer @ *******.com:10000 >> 04 Aug 2016 23:39:47 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [HiveDriver-ConnectionExpiryThread] INFO >> org.apache.lens.driver.hive.HiveDriver >> - Closed connection: 30 >> 04 Aug 2016 23:39:47 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >> [grizzly-http-server-28] INFO org.apache.lens.server.api.query.QueryContext >> - Updating status of 44c647ae-83dc-4ac7-a61f-040d2be07573 from Status : >> RUNNING >> Message : Query is running in HiveServer! >> Progress : 0.0 >> Progress Message : [{"taskId":"Stage-1","type":"M >> APRED","externalHandle":"job_1469684228847_467645","taskStat >> e":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE"," >> externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}] >> to Status : RUNNING >> Message : Query is running in HiveServer! >> Progress : 0.0 >> Progress Message : [{"taskId":"Stage-1","type":"M >> APRED","externalHandle":"job_1469684228847_467645","taskStat >> e":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE"," >> externalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}] >> >> *...* >> >> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [QueryPurger] INFO org.apache.lens.driver.hive.HiveDriver - New thrift >> connection class org.apache.lens.driver.hive.RemoteThriftConnection for >> thread: 25 for user: anonymous *connection ID=106* on driver:hive/hive1 >> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [QueryPurger] INFO org.apache.lens.driver.hive.RemoteThriftConnection - >> HiveDriver connecting to HiveServer @ *******.com:10000 >> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [QueryPurger] INFO org.apache.hive.service.cli.t >> hrift.RetryingThriftCLIServiceClient - Connecting to *******.com:10000 >> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [QueryPurger] INFO org.apache.hive.service.cli.t >> hrift.RetryingThriftCLIServiceClient - Connected! >> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [QueryPurger] INFO org.apache.lens.driver.hive.RemoteThriftConnection - >> HiveDriver connected to HiveServer @ *******.com:10000 >> >> >> It created 106 connections in total for the following example query: >> lens-shell>query execute cube select product_id, store_sales from sales >> where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') >> >> And it even created 2 connection when the query was successful. Then, it >> started to close those connections: >> >> *....* >> 04 Aug 2016 23:49:22 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [HiveDriver-ConnectionExpiryThread] INFO >> org.apache.lens.driver.hive.HiveDriver >> - Closed connection: 87 >> 04 Aug 2016 23:49:32 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [HiveDriver-ConnectionExpiryThread] INFO >> org.apache.lens.driver.hive.HiveDriver >> - Closed connection: 88 >> 04 Aug 2016 23:49:42 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [HiveDriver-ConnectionExpiryThread] INFO >> org.apache.lens.driver.hive.HiveDriver >> - Closed connection: 89 >> 04 Aug 2016 23:49:52 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >> [HiveDriver-ConnectionExpiryThread] INFO >> org.apache.lens.driver.hive.HiveDriver >> - Closed connection: 90 >> *...* >> >> Do we know why did Lens created so many connections for such a simple >> query? And it really takes time to create those queries. >> >> Thanks, >> -- >> >> *Tao Yan* >> Software Engineer >> Data Analytics Infrastructure Tools and Services >> >> >> >> 206.250.5345 >> [email protected] >> https://www.linkedin.com/in/taousc >> > > > _____________________________________________________________ > The information contained in this communication is intended solely for the > use of the individual or entity to whom it is addressed and others > authorized to receive it. It may contain confidential or legally privileged > information. If you are not the intended recipient you are hereby notified > that any disclosure, copying, distribution or taking any action in reliance > on the contents of this information is strictly prohibited and may be > unlawful. If you have received this communication in error, please notify > us immediately by responding to this email and then delete it from your > system. The firm is neither liable for the proper and complete transmission > of the information contained in this communication nor for any delay in its > receipt. -- *Tao Yan* Software Engineer Data Analytics Infrastructure Tools and Services 206.250.5345 [email protected] https://www.linkedin.com/in/taousc
