[
https://issues.apache.org/jira/browse/HIVE-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15172928#comment-15172928
]
Siddharth Seth commented on HIVE-13176:
---------------------------------------
There's several issues here, and two cases being looked at.
In the first case, there's a lot of failed queries. Driver instances are
retained by the Shutdownhook - 35K of them. Looking at the code, I couldn't see
any direct leaks - and can only attribute this to failed queries / connections
- where the client may have failed to close the connection.
In the second case, there's two main culprits. DataNucleus PluginManager seems
to be retaining a large chunk of the heap. The second is a single FileSystem
object which is over 100MB. This is caused by retention of FSStatistics objects
- and has been fixed in a Hadoop jira - we should look at adding a workaround
for this, i.e. consume the statistics instance as part of the close. The second
reason is a large number of entities registered in the deleteOnExit tracker
within FileSystem. That, I believe, is something that Hive will need to look at.
> OutOfMemoryError : GC overhead limit exceeded
> ----------------------------------------------
>
> Key: HIVE-13176
> URL: https://issues.apache.org/jira/browse/HIVE-13176
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Reporter: Kavan Suresh
> Assignee: Siddharth Seth
>
> Detected leaks while testing hiveserver2 concurrency setup with LLAP.
> 2016-02-26T12:50:58,131 ERROR [HiveServer2-Background-Pool: Thread-311030]:
> operation.Operation (SQLOperation.java:run(230)) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.StatsTask. GC overhead limit exceeded
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:333)
> ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:177)
> ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36]
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:73)
> ~[hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36]
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:227)
> [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_45]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_45]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.1.2.3.5.1-36.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:239)
> [hive-jdbc-2.0.0.2.3.5.1-36-standalone.jar:2.0.0.2.3.5.1-36]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_45]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [?:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_45]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_45]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)