Hi, list I’m new to solr. Recently I encounter a “memory leak” problem with solrcloud.
I have two 64GB servers running a solrcloud cluster. In the solrcloud, I have one collection with about 400k docs. The index size of the collection is about 500MB. Memory for solr is 16GB. Following is "ps aux | grep solr” : /usr/java/jdk1.7.0_67-cloudera/bin/java -Djava.util.logging.config.file=/var/lib/solr/tomcat-deployment/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.net.preferIPv4Stack=true -Dsolr.hdfs.blockcache.enabled=true -Dsolr.hdfs.blockcache.direct.memory.allocation=true -Dsolr.hdfs.blockcache.blocksperbank=16384 -Dsolr.hdfs.blockcache.slab.count=1 -Xms16608395264 -Xmx16608395264 -XX:MaxDirectMemorySize=21590179840 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -Xloggc:/var/log/solr/gc.log -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -DzkHost=bjzw-datacenter-hadoop-160.d.yourmall.cc:2181,bjzw-datacenter-hadoop-163.d.yourmall.cc:2181,bjzw-datacenter-hadoop-164.d.yourmall.cc:2181/solr -Dsolr.solrxml.location=zookeeper -Dsolr.hdfs.home=hdfs://datacenter/solr -Dsolr.hdfs.confdir=/var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/hadoop-conf -Dsolr.authentication.simple.anonymous.allowed=true -Dsolr.security.proxyuser.hue.hosts=* -Dsolr.security.proxyuser.hue.groups=* -Dhost=bjzw-datacenter-solr-15.d.yourmall.cc -Djetty.port=8983 -Dsolr.host=bjzw-datacenter-solr-15.d.yourmall.cc -Dsolr.port=8983 -Dlog4j.configuration=file:///var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/log4j.properties -Dsolr.log=/var/log/solr -Dsolr.admin.port=8984 -Dsolr.max.connector.thread=10000 -Dsolr.solr.home=/var/lib/solr -Djava.net.preferIPv4Stack=true -Dsolr.hdfs.blockcache.enabled=true -Dsolr.hdfs.blockcache.direct.memory.allocation=true -Dsolr.hdfs.blockcache.blocksperbank=16384 -Dsolr.hdfs.blockcache.slab.count=1 -Xms16608395264 -Xmx16608395264 -XX:MaxDirectMemorySize=21590179840 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:-CMSConcurrentMTEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC -Xloggc:/var/log/solr/gc.log -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -DzkHost=bjzw-datacenter-hadoop-160.d.yourmall.cc:2181,bjzw-datacenter-hadoop-163.d.yourmall.cc:2181,bjzw-datacenter-hadoop-164.d.yourmall.cc:2181/solr -Dsolr.solrxml.location=zookeeper -Dsolr.hdfs.home=hdfs://datacenter/solr -Dsolr.hdfs.confdir=/var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/hadoop-conf -Dsolr.authentication.simple.anonymous.allowed=true -Dsolr.security.proxyuser.hue.hosts=* -Dsolr.security.proxyuser.hue.groups=* -Dhost=bjzw-datacenter-solr-15.d.yourmall.cc -Djetty.port=8983 -Dsolr.host=bjzw-datacenter-solr-15.d.yourmall.cc -Dsolr.port=8983 -Dlog4j.configuration=file:///var/run/cloudera-scm-agent/process/6288-solr-SOLR_SERVER/log4j.properties -Dsolr.log=/var/log/solr -Dsolr.admin.port=8984 -Dsolr.max.connector.thread=10000 -Dsolr.solr.home=/var/lib/solr -Djava.endorsed.dirs=/usr/lib/bigtop-tomcat/endorsed -classpath /usr/lib/bigtop-tomcat/bin/bootstrap.jar -Dcatalina.base=/var/lib/solr/tomcat-deployment -Dcatalina.home=/usr/lib/bigtop-tomcat -Djava.io.tmpdir=/var/lib/solr/ org.apache.catalina.startup.Bootstrap start solr version is solr4.4.0-cdh5.3.0 jdk version is 1.7.0_67 Soft commit time is 1.5s. And we have real time indexing/partialupdating rate about 100 docs per second. When fresh started, Solr will use about 500M memory(the memory show in solr ui panel). After several days running, Solr will meet with long time gc problems, and no response to user query. During solr running, the memory used by solr is keep increasing until some large value, and decrease to a low level(because of gc), and keep increasing until a larger value again, then decrease to a low level again … and keep increasing to an more larger value … until solr has no response and i restart it. I don’t know how to solve this problem. Can you give me some advices? Thanks.
