Re: Review Request 14298: Memory leak when using JDBC connections.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14298/ --- (Updated Sept. 27, 2013, 12:12 a.m.) Review request for hive. Bugs: HIVE-5296 https://issues.apache.org/jira/browse/HIVE-5296 Repository: hive-git Description --- Hiveserver2 will occur memory leak caused by increasing Hashtable$Entry at least 2 situation as follows. 1. When Exceptions are thrown during executing commmand or query, operation handle will not release. 2. Hiveserver2 calls FileSystem#get method and never call FileSystem#close or FileSystem.closeAll so FileSystem$Cache will continue to increase. I've modified HiveSessionImpl and HiveStatement not to lose operation handle. Operation handle is needed by OperationManager to remove from handleToOpration. Also, I've modified HiveSessionImpl to close FileSystem object at the end of session. Diffs (updated) - service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 Diff: https://reviews.apache.org/r/14298/diff/ Testing --- I confirmed only not increasing Hashtable$Entry by jmap. Thanks, Kousuke Saruta
Re: Review Request 14298: Memory leak when using JDBC connections.
On Sept. 26, 2013, 6:36 p.m., Vaibhav Gumashta wrote: jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java, line 204 https://reviews.apache.org/r/14298/diff/4/?file=356951#file356951line204 Similar to my other comment (on removing opHandle entry from handleToOperation map in OperationManager), do you see any use of keeping the stmtHandle on an exception if we ensure that the operationHandle on the server is removed from the handleToOperation map in case of an error? Do you mean we should not put off calling closeClientOperation until next HiveStatement#execute or until HiveStatement#close will be called but call closeClientOperation immediately on an exception right? I think it's reasonable exactly. On Sept. 26, 2013, 6:36 p.m., Vaibhav Gumashta wrote: service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java, line 197 https://reviews.apache.org/r/14298/diff/4/?file=356952#file356952line197 In the current (patch) implementation, when an operation throws an exception, we're making sure that the opHandle gets added to the opHandleSet in HiveSessionImpl for later cleanup by HiveSessionImpl#close. Wouldn't a better to just remove the opHandle entry from handleToOperation map in OperationManager but pass on the error to the client? In the new patch, I modified HiveSessionImpl to remove opHandle from handleToOperation immediately on an error. - Kousuke --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14298/#review26414 --- On Sept. 27, 2013, 12:12 a.m., Kousuke Saruta wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14298/ --- (Updated Sept. 27, 2013, 12:12 a.m.) Review request for hive. Bugs: HIVE-5296 https://issues.apache.org/jira/browse/HIVE-5296 Repository: hive-git Description --- Hiveserver2 will occur memory leak caused by increasing Hashtable$Entry at least 2 situation as follows. 1. When Exceptions are thrown during executing commmand or query, operation handle will not release. 2. Hiveserver2 calls FileSystem#get method and never call FileSystem#close or FileSystem.closeAll so FileSystem$Cache will continue to increase. I've modified HiveSessionImpl and HiveStatement not to lose operation handle. Operation handle is needed by OperationManager to remove from handleToOpration. Also, I've modified HiveSessionImpl to close FileSystem object at the end of session. Diffs - service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 Diff: https://reviews.apache.org/r/14298/diff/ Testing --- I confirmed only not increasing Hashtable$Entry by jmap. Thanks, Kousuke Saruta
Re: Review Request 14298: Memory leak when using JDBC connections.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14298/ --- (Updated Sept. 25, 2013, 9:09 a.m.) Review request for hive. Changes --- I found using FileSystem.closeAll is a bad idea and FIleSystem$Cache problem will be addressed HIVE-4501 so I try to address another problem that opHandle will not be released when Exception occurred during executing query or command. Bugs: HIVE-5296 https://issues.apache.org/jira/browse/HIVE-5296 Repository: hive-git Description --- Hiveserver2 will occur memory leak caused by increasing Hashtable$Entry at least 2 situation as follows. 1. When Exceptions are thrown during executing commmand or query, operation handle will not release. 2. Hiveserver2 calls FileSystem#get method and never call FileSystem#close or FileSystem.closeAll so FileSystem$Cache will continue to increase. I've modified HiveSessionImpl and HiveStatement not to lose operation handle. Operation handle is needed by OperationManager to remove from handleToOpration. Also, I've modified HiveSessionImpl to close FileSystem object at the end of session. Diffs (updated) - jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2912ece service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 Diff: https://reviews.apache.org/r/14298/diff/ Testing --- I confirmed only not increasing Hashtable$Entry by jmap. Thanks, Kousuke Saruta
Re: Review Request 14298: Memory leak when using JDBC connections.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14298/ --- (Updated Sept. 24, 2013, 11:48 p.m.) Review request for hive. Bugs: HIVE-5296 https://issues.apache.org/jira/browse/HIVE-5296 Repository: hive-git Description --- Hiveserver2 will occur memory leak caused by increasing Hashtable$Entry at least 2 situation as follows. 1. When Exceptions are thrown during executing commmand or query, operation handle will not release. 2. Hiveserver2 calls FileSystem#get method and never call FileSystem#close or FileSystem.closeAll so FileSystem$Cache will continue to increase. I've modified HiveSessionImpl and HiveStatement not to lose operation handle. Operation handle is needed by OperationManager to remove from handleToOpration. Also, I've modified HiveSessionImpl to close FileSystem object at the end of session. Diffs (updated) - jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2912ece service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 Diff: https://reviews.apache.org/r/14298/diff/ Testing --- I confirmed only not increasing Hashtable$Entry by jmap. Thanks, Kousuke Saruta
Review Request 14298: Memory leak when using JDBC connections.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14298/ --- Review request for hive. Bugs: HIVE-5296 https://issues.apache.org/jira/browse/HIVE-5296 Repository: hive-git Description --- Hiveserver2 will occur memory leak caused by increasing Hashtable$Entry at least 2 situation as follows. 1. When Exceptions are thrown during executing commmand or query, operation handle will not release. 2. Hiveserver2 calls FileSystem#get method and never call FileSystem#close or FileSystem.closeAll so FileSystem$Cache will continue to increase. I've modified HiveSessionImpl and HiveStatement not to lose operation handle. Operation handle is needed by OperationManager to remove from handleToOpration. Also, I've modified HiveSessionImpl to close FileSystem object at the end of session. Diffs - jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 478fa57 service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 Diff: https://reviews.apache.org/r/14298/diff/ Testing --- I confirmed only not increasing Hashtable$Entry by jmap. Thanks, Kousuke Saruta
Re: Review Request 14298: Memory leak when using JDBC connections.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14298/ --- (Updated Sept. 24, 2013, 5:27 a.m.) Review request for hive. Changes --- I've re-created a patch for HEAD. Bugs: HIVE-5296 https://issues.apache.org/jira/browse/HIVE-5296 Repository: hive-git Description --- Hiveserver2 will occur memory leak caused by increasing Hashtable$Entry at least 2 situation as follows. 1. When Exceptions are thrown during executing commmand or query, operation handle will not release. 2. Hiveserver2 calls FileSystem#get method and never call FileSystem#close or FileSystem.closeAll so FileSystem$Cache will continue to increase. I've modified HiveSessionImpl and HiveStatement not to lose operation handle. Operation handle is needed by OperationManager to remove from handleToOpration. Also, I've modified HiveSessionImpl to close FileSystem object at the end of session. Diffs (updated) - jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 2912ece service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 11c96b2 Diff: https://reviews.apache.org/r/14298/diff/ Testing --- I confirmed only not increasing Hashtable$Entry by jmap. Thanks, Kousuke Saruta
