[ https://issues.apache.org/jira/browse/HBASE-4633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162538#comment-13162538 ]
gaojinchao commented on HBASE-4633: ----------------------------------- Hbase version is 0.90.4 + patch. Cluseter number is 10 One HBase client process includes 50 threads, So the max threads connect to the RS is (50 * RS number). I have noticed some memory leak problems in my HBase client. RES has increased to 27g PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 12676 root 20 0 30.8g 27g 5092 S 2 57.5 587:57.76 /opt/java/jre/bin/java -Djava.library.path=lib/. But I am not sure the leak comes from HBase Client jar itself or just our client code. This is some parameters of jvm. :-Xms15g -Xmn12g -Xmx15g -XX:PermSize=64m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=65 -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=1 -XX:+CMSParallelRemarkEnabled > Potential memory leak in client RPC timeout mechanism > ----------------------------------------------------- > > Key: HBASE-4633 > URL: https://issues.apache.org/jira/browse/HBASE-4633 > Project: HBase > Issue Type: Bug > Components: client > Affects Versions: 0.90.3 > Environment: HBase version: 0.90.3 + Patches , Hadoop version: CDH3u0 > Reporter: Shrijeet Paliwal > Attachments: HBaseclientstack.png > > > Relevant Jiras: https://issues.apache.org/jira/browse/HBASE-2937, > https://issues.apache.org/jira/browse/HBASE-4003 > We have been using the 'hbase.client.operation.timeout' knob > introduced in 2937 for quite some time now. It helps us enforce SLA. > We have two HBase clusters and two HBase client clusters. One of them > is much busier than the other. > We have seen a deterministic behavior of clients running in busy > cluster. Their (client's) memory footprint increases consistently > after they have been up for roughly 24 hours. > This memory footprint almost doubles from its usual value (usual case > == RPC timeout disabled). After much investigation nothing concrete > came out and we had to put a hack > which keep heap size in control even when RPC timeout is enabled. Also > note , the same behavior is not observed in 'not so busy > cluster. > The patch is here : https://gist.github.com/1288023 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira