Hi Tao, Please attach the complete log if it happens again. We will open a JIRA for this .
I checked the production setup, I am not seeing too many connections. Thanks, Puneet Gupta On Tue, Aug 9, 2016 at 1:49 AM, Tao Yan <[email protected]> wrote: > Hi Rajat, > > The number of connections created is usually small (less than 10? > probably), I run the same query multiple times, and it only happen twice > when Lens server created too many connections. I wonder if it is something > unexpected. > > Thanks, > > On Fri, Aug 5, 2016 at 9:30 PM, Puneet Gupta <[email protected]> > wrote: > >> Hi Tao >> >> these connections can be created various operations that interact with >> hive server like launching a query, pulling its status, etc (There is a >> Status Poller thread which polls the status of hive query at regular >> intervals.) >> The hive connections are cached per user with an configurable expiry time >> (which is 10 mins by default). You can configure it differently in hive >> driver config (lens.driver.hive.hs2.connection.expiry.delay)if needed. >> >> I see from the logs attached by you, the connections are created for two >> different users (test_user and anonymous) >> >> Do let me know if you need more details . >> >> Thanks, >> Puneet Gupta >> >> On Fri, Aug 5, 2016 at 5:24 AM, Tao Yan <[email protected]> wrote: >> >>> Hi Lens Developers, >>> >>> I am running Lens queries on example data. And I realized sometimes Lens >>> server will keep trying to connect to HiveServer2 during the query >>> execution, and Lens log shows: >>> >>> *...* >>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >>> [grizzly-http-server-28] INFO org.apache.lens.driver.hive.HiveDriver - >>> New thrift connection class >>> org.apache.lens.driver.hive.RemoteThriftConnection >>> for thread: 112 for user: test_user *connection ID=89* on >>> driver:hive/hive1 >>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >>> [grizzly-http-server-28] INFO >>> org.apache.lens.driver.hive.RemoteThriftConnection >>> - HiveDriver connecting to HiveServer @ *******.com:10000 >>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >>> [grizzly-http-server-28] INFO org.apache.hive.service.cli.t >>> hrift.RetryingThriftCLIServiceClient - Connecting to *******.com:10000 >>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >>> [grizzly-http-server-28] INFO org.apache.hive.service.cli.t >>> hrift.RetryingThriftCLIServiceClient - Connected! >>> 04 Aug 2016 23:39:42 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >>> [grizzly-http-server-28] INFO >>> org.apache.lens.driver.hive.RemoteThriftConnection >>> - HiveDriver connected to HiveServer @ *******.com:10000 >>> 04 Aug 2016 23:39:47 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [HiveDriver-ConnectionExpiryThread] INFO >>> org.apache.lens.driver.hive.HiveDriver >>> - Closed connection: 30 >>> 04 Aug 2016 23:39:47 [1b0ba53f-e063-4dd0-a2f1-1519ce940839] >>> [grizzly-http-server-28] INFO org.apache.lens.server.api.query.QueryContext >>> - Updating status of 44c647ae-83dc-4ac7-a61f-040d2be07573 from Status >>> : RUNNING >>> Message : Query is running in HiveServer! >>> Progress : 0.0 >>> Progress Message : [{"taskId":"Stage-1","type":"M >>> APRED","externalHandle":"job_1469684228847_467645","taskStat >>> e":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","exter >>> nalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}] >>> to Status : RUNNING >>> Message : Query is running in HiveServer! >>> Progress : 0.0 >>> Progress Message : [{"taskId":"Stage-1","type":"M >>> APRED","externalHandle":"job_1469684228847_467645","taskStat >>> e":"RUNNING_STATE"},{"taskId":"Stage-0","type":"MOVE","exter >>> nalHandle":"Stage-0","taskState":"UNKNOWN_STATE"}] >>> >>> *...* >>> >>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [QueryPurger] INFO org.apache.lens.driver.hive.HiveDriver - New thrift >>> connection class org.apache.lens.driver.hive.RemoteThriftConnection for >>> thread: 25 for user: anonymous *connection ID=106* on driver:hive/hive1 >>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [QueryPurger] INFO org.apache.lens.driver.hive.RemoteThriftConnection >>> - HiveDriver connecting to HiveServer @ *******.com:10000 >>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [QueryPurger] INFO org.apache.hive.service.cli.t >>> hrift.RetryingThriftCLIServiceClient - Connecting to *******.com:10000 >>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [QueryPurger] INFO org.apache.hive.service.cli.t >>> hrift.RetryingThriftCLIServiceClient - Connected! >>> 04 Aug 2016 23:42:23 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [QueryPurger] INFO org.apache.lens.driver.hive.RemoteThriftConnection >>> - HiveDriver connected to HiveServer @ *******.com:10000 >>> >>> >>> It created 106 connections in total for the following example query: >>> lens-shell>query execute cube select product_id, store_sales from sales >>> where time_range_in(order_time, '2015-04-11-00', '2015-04-13-01') >>> >>> And it even created 2 connection when the query was successful. Then, it >>> started to close those connections: >>> >>> *....* >>> 04 Aug 2016 23:49:22 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [HiveDriver-ConnectionExpiryThread] INFO >>> org.apache.lens.driver.hive.HiveDriver >>> - Closed connection: 87 >>> 04 Aug 2016 23:49:32 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [HiveDriver-ConnectionExpiryThread] INFO >>> org.apache.lens.driver.hive.HiveDriver >>> - Closed connection: 88 >>> 04 Aug 2016 23:49:42 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [HiveDriver-ConnectionExpiryThread] INFO >>> org.apache.lens.driver.hive.HiveDriver >>> - Closed connection: 89 >>> 04 Aug 2016 23:49:52 [9b8ae2cc-89a0-4670-af1b-4481da5aa9f2] >>> [HiveDriver-ConnectionExpiryThread] INFO >>> org.apache.lens.driver.hive.HiveDriver >>> - Closed connection: 90 >>> *...* >>> >>> Do we know why did Lens created so many connections for such a simple >>> query? And it really takes time to create those queries. >>> >>> Thanks, >>> -- >>> >>> *Tao Yan* >>> Software Engineer >>> Data Analytics Infrastructure Tools and Services >>> >>> >>> >>> 206.250.5345 >>> [email protected] >>> https://www.linkedin.com/in/taousc >>> >> >> >> _____________________________________________________________ >> The information contained in this communication is intended solely for >> the use of the individual or entity to whom it is addressed and others >> authorized to receive it. It may contain confidential or legally privileged >> information. If you are not the intended recipient you are hereby notified >> that any disclosure, copying, distribution or taking any action in reliance >> on the contents of this information is strictly prohibited and may be >> unlawful. If you have received this communication in error, please notify >> us immediately by responding to this email and then delete it from your >> system. The firm is neither liable for the proper and complete transmission >> of the information contained in this communication nor for any delay in its >> receipt. > > > > > -- > > *Tao Yan* > Software Engineer > Data Analytics Infrastructure Tools and Services > > > > 206.250.5345 > [email protected] > https://www.linkedin.com/in/taousc > -- _____________________________________________________________ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
