On Thu, Apr 19, 2012 at 3:12 PM, Sami Siren <ssi...@gmail.com> wrote: > I have a simple solrcloud setup from trunk with default configs; 1 > shard with one replica. As few other people have reported there seems > to be some kind of leak somewhere that causes the number of open files > to grow over time when doing indexing. > > One thing that correlates with the open file count that the jvm > reports is the count of deleted files that solr still keeps open (not > sure if the problem is this or something else). The deleted but not > closed files are all ending with "nrm.cfs", for example > > <solr_home>/solr/data/index/_jwk_nrm.cfs (deleted) > > Any ideas about what could be the cause for this? I don't even know > where to start looking...
I think the problem is not just the (deleted) files that counts towards the max open files the user is allowed to have. There's also the mmapped files that are kept open and count towards the limit configured in /proc/sys/vm/max_map_count and eventually lead to exceptions like java.io.IOException: Map failed there's plenty of those too visible to lsof (all kinds of lucene index files), for example: java 32624 sam DEL REG 8,3 6425004 /home/sam/example/solr/data/index/_4.fdx java 32624 sam DEL REG 8,3 6424981 /home/sam/example/solr/data/index/_4.fdt java 32624 sam DEL REG 8,3 6425019 /home/sam/example/solr/data/index/_4_nrm.cfs java 32624 sam DEL REG 8,3 6425016 /home/sam/example/solr/data/index/_4_0.tim java 32624 sam DEL REG 8,3 6425015 /home/sam/example/solr/data/index/_4_0.prx java 32624 sam DEL REG 8,3 6425014 /home/sam/example/solr/data/index/_4_0.frq Why are all these files, regular and mmapped kept open even if merging has occurred in the background and the files are deleted? - Sami Siren
