[jira] [Commented] (HIVE-6866) Hive server2 jdbc driver connection leak with namenode
[
https://issues.apache.org/jira/browse/HIVE-6866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14127451#comment-14127451
]
Ankita Bakshi commented on HIVE-6866:
-
We are facing same issue in production. We are using CDH4.4 with Apache Hive
0.12. Is there a workaround for this issue other than restarting hiveserver2?
Hive server2 jdbc driver connection leak with namenode
--
Key: HIVE-6866
URL: https://issues.apache.org/jira/browse/HIVE-6866
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Shengjun Xin
1. Set 'ipc.client.connection.maxidletime' to 360 in core-site.xml and
start hive-server2.
2. Connect hive server2 repetitively in a while true loop.
3. The tcp connection number will increase until out of memory, it seems that
hive server2 will not close the connection until the time out, the error
message is as the following:
{code}
2014-03-18 23:30:36,873 ERROR ql.Driver (SessionState.java:printError(386)) -
FAILED: RuntimeException java.io.IOException: Failed on local exception:
java.io.IOException: Couldn't set up IO streams; Host Details : local host
is: hdm1.hadoop.local/192.168.2.101; destination host is:
hdm1.hadoop.local:8020;
java.lang.RuntimeException: java.io.IOException: Failed on local exception:
java.io.IOException: Couldn't set up IO streams; Host Details : local host
is: hdm1.hadoop.local/192.168.2.101; destination host is:
hdm1.hadoop.local:8020;
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:190)
at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)
at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8676)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:278)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:433)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:95)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:181)
at
org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:148)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:203)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:40)
at
org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:37)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:524)
at
org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:37)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Failed on local exception:
java.io.IOException: Couldn't set up IO streams; Host Details : local host
is: hdm1.hadoop.local/192.168.2.101; destination host is:
hdm1.hadoop.local:8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761)
at org.apache.hadoop.ipc.Client.call(Client.java:1239)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
[jira] [Commented] (HIVE-6866) Hive server2 jdbc driver connection leak with namenode
[
https://issues.apache.org/jira/browse/HIVE-6866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979479#comment-13979479
]
Zilvinas Saltys commented on HIVE-6866:
---
Got the same issue on CDH5. It seems that when the open files limit is set to
low for hive and these connections are not closed it will start throwing this
error.
Hive server2 jdbc driver connection leak with namenode
--
Key: HIVE-6866
URL: https://issues.apache.org/jira/browse/HIVE-6866
Project: Hive
Issue Type: Bug
Affects Versions: 0.11.0
Reporter: Shengjun Xin
1. Set 'ipc.client.connection.maxidletime' to 360 in core-site.xml and
start hive-server2.
2. Connect hive server2 repetitively in a while true loop.
3. The tcp connection number will increase until out of memory, it seems that
hive server2 will not close the connection until the time out, the error
message is as the following:
{code}
2014-03-18 23:30:36,873 ERROR ql.Driver (SessionState.java:printError(386)) -
FAILED: RuntimeException java.io.IOException: Failed on local exception:
java.io.IOException: Couldn't set up IO streams; Host Details : local host
is: hdm1.hadoop.local/192.168.2.101; destination host is:
hdm1.hadoop.local:8020;
java.lang.RuntimeException: java.io.IOException: Failed on local exception:
java.io.IOException: Couldn't set up IO streams; Host Details : local host
is: hdm1.hadoop.local/192.168.2.101; destination host is:
hdm1.hadoop.local:8020;
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:190)
at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:231)
at org.apache.hadoop.hive.ql.Context.getMRTmpFileURI(Context.java:288)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1274)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1059)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8676)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:278)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:433)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
at
org.apache.hive.service.cli.operation.SQLOperation.run(SQLOperation.java:95)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(HiveSessionImpl.java:181)
at
org.apache.hive.service.cli.CLIService.executeStatement(CLIService.java:148)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:203)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1133)
at
org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1118)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:40)
at
org.apache.hive.service.auth.TUGIContainingProcessor$1.run(TUGIContainingProcessor.java:37)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1478)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure.doAs(HadoopShimsSecure.java:524)
at
org.apache.hive.service.auth.TUGIContainingProcessor.process(TUGIContainingProcessor.java:37)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.IOException: Failed on local exception:
java.io.IOException: Couldn't set up IO streams; Host Details : local host
is: hdm1.hadoop.local/192.168.2.101; destination host is:
hdm1.hadoop.local:8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761)
at org.apache.hadoop.ipc.Client.call(Client.java:1239)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)
at com.sun.proxy.$Proxy11.mkdirs(Unknown Source)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
