[jira] [Commented] (HIVE-4132) Connection to Zookeeper is created for each client of HS2

2013-06-19 Thread Philippe Marseille (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687958#comment-13687958
 ] 

Philippe Marseille commented on HIVE-4132:
--

Is this related to https://issues.apache.org/jira/browse/HIVE-4398 ?

 Connection to Zookeeper is created for each client of HS2
 -

 Key: HIVE-4132
 URL: https://issues.apache.org/jira/browse/HIVE-4132
 Project: Hive
  Issue Type: Sub-task
  Components: Server Infrastructure
Reporter: Brock Noland
  Labels: HiveServer2

 Currently when concurrency is enabled and HS2 is used each client will have a 
 connection to Zookeeper. Zookeeper has a property, maxClientCnxns, in the 
 zoo.cfg which limits the number of connections *per* host to a low number 
 like 50. The stack trace of the threads accessing zookeeper is below for 
 reference.
 {noformat}
 - 
 org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.lock(java.util.List,
  boolean) @bci=94, line=220 (Interpreted frame)
  - org.apache.hadoop.hive.ql.Driver.acquireReadWriteLocks() @bci=428, 
 line=834 (Interpreted frame)
  - org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=453, line=944 
 (Interpreted frame)
  - org.apache.hive.service.cli.operation.SQLOperation.run() @bci=126, line=95 
 (Interpreted frame)
  - 
 org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(java.lang.String,
  java.util.Map) @bci=19, line=168 (Interpreted frame)
  - 
 org.apache.hive.service.cli.CLIService.executeStatement(org.apache.hive.service.cli.SessionHandle,
  java.lang.String, java.util.Map) @bci=10, line=153 (Interpreted frame)
  - 
 org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(org.apache.hive.service.cli.thrift.TExecuteStatementReq)
  @bci=41, line=190 (Interpreted frame)
  - 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(org.apache.hive.service.cli.thrift.TCLIService$Iface,
  org.apache.hive.service.cli.thrift.TCLIService$ExecuteStatement_args) 
 @bci=14, line=1133 (Interpreted frame)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4132) Connection to Zookeeper is created for each client of HS2

2013-06-19 Thread Brock Noland (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13687981#comment-13687981
 ] 

Brock Noland commented on HIVE-4132:


It's possible but I believe this one was more around the fact that there is no 
pool of zookeeper connections. For example an HS2 with 500 concurrent users 
will have 500 connections to zookeeper even though zk connections are used for 
fairly short periods.

 Connection to Zookeeper is created for each client of HS2
 -

 Key: HIVE-4132
 URL: https://issues.apache.org/jira/browse/HIVE-4132
 Project: Hive
  Issue Type: Sub-task
  Components: Server Infrastructure
Reporter: Brock Noland
  Labels: HiveServer2

 Currently when concurrency is enabled and HS2 is used each client will have a 
 connection to Zookeeper. Zookeeper has a property, maxClientCnxns, in the 
 zoo.cfg which limits the number of connections *per* host to a low number 
 like 50. The stack trace of the threads accessing zookeeper is below for 
 reference.
 {noformat}
 - 
 org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.lock(java.util.List,
  boolean) @bci=94, line=220 (Interpreted frame)
  - org.apache.hadoop.hive.ql.Driver.acquireReadWriteLocks() @bci=428, 
 line=834 (Interpreted frame)
  - org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=453, line=944 
 (Interpreted frame)
  - org.apache.hive.service.cli.operation.SQLOperation.run() @bci=126, line=95 
 (Interpreted frame)
  - 
 org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(java.lang.String,
  java.util.Map) @bci=19, line=168 (Interpreted frame)
  - 
 org.apache.hive.service.cli.CLIService.executeStatement(org.apache.hive.service.cli.SessionHandle,
  java.lang.String, java.util.Map) @bci=10, line=153 (Interpreted frame)
  - 
 org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(org.apache.hive.service.cli.thrift.TExecuteStatementReq)
  @bci=41, line=190 (Interpreted frame)
  - 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(org.apache.hive.service.cli.thrift.TCLIService$Iface,
  org.apache.hive.service.cli.thrift.TCLIService$ExecuteStatement_args) 
 @bci=14, line=1133 (Interpreted frame)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HIVE-4132) Connection to Zookeeper is created for each client of HS2

2013-03-06 Thread Brock Noland (JIRA) 

[ 
https://issues.apache.org/jira/browse/HIVE-4132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595042#comment-13595042
 ] 

Brock Noland commented on HIVE-4132:


Setting maxClientCnxns to a high number in zoo.cfg works around this issue. 
However, since the Zookeeper object itself is thread safe (except for a few 
methods noted here 
http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html) 
it might make sense to use a single object or a pool of objects.

 Connection to Zookeeper is created for each client of HS2
 -

 Key: HIVE-4132
 URL: https://issues.apache.org/jira/browse/HIVE-4132
 Project: Hive
  Issue Type: Sub-task
  Components: Server Infrastructure
Reporter: Brock Noland
  Labels: HiveServer2

 Currently when concurrency is enabled and HS2 is used each client will have a 
 connection to Zookeeper. Zookeeper has a property, maxClientCnxns, in the 
 zoo.cfg which limits the number of connections *per* host to a low number 
 like 50. The stack trace of the threads accessing zookeeper is below for 
 reference.
 {noformat}
 - 
 org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager.lock(java.util.List,
  boolean) @bci=94, line=220 (Interpreted frame)
  - org.apache.hadoop.hive.ql.Driver.acquireReadWriteLocks() @bci=428, 
 line=834 (Interpreted frame)
  - org.apache.hadoop.hive.ql.Driver.run(java.lang.String) @bci=453, line=944 
 (Interpreted frame)
  - org.apache.hive.service.cli.operation.SQLOperation.run() @bci=126, line=95 
 (Interpreted frame)
  - 
 org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(java.lang.String,
  java.util.Map) @bci=19, line=168 (Interpreted frame)
  - 
 org.apache.hive.service.cli.CLIService.executeStatement(org.apache.hive.service.cli.SessionHandle,
  java.lang.String, java.util.Map) @bci=10, line=153 (Interpreted frame)
  - 
 org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(org.apache.hive.service.cli.thrift.TExecuteStatementReq)
  @bci=41, line=190 (Interpreted frame)
  - 
 org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(org.apache.hive.service.cli.thrift.TCLIService$Iface,
  org.apache.hive.service.cli.thrift.TCLIService$ExecuteStatement_args) 
 @bci=14, line=1133 (Interpreted frame)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira