Hi Chris, Thanks for the response. Eventualy I want to install Solr on a machine with a maximum memory of 4GB. I tried to index the data on that machine before, but it resulted in index locks and memory errors. Is 4GB not enough to index 100,000 documents in a row? How much should it be? Is there a way to tune this?
Regards, Marc 2011/8/30 Chris Hostetter <hossman_luc...@fucit.org> > > : The current system I'm using has 150GB of memory and while I'm indexing > the > : memoryconsumption is growing and growing (eventually more then 50GB). > : In the attached graph (http://postimage.org/image/acyv7kec/) I indexed > about > : 70k of office-documents (pdf,doc,xls etc) and between 1 and 2 percent > throws > > Unless i'm missunderstanding sometihng about your graph, only ~12GB of > memory is used by applications on that machine. About 60GB is in use by > the "filesystem cache". > > The Filesystem cache is not memory being used by Solr, it's memory that is > free and not in use by an application, so your OS is (wisely) using it to > cache files from disk that you've recently accessed in case you need them > again. This is handy, and for max efficients (when keeping your index on > disk) it's useful to make sure you allocate resources so that you have > enough extra memory on your server that the entire index can be kept in > the filesystem cache -- but the OS will happily free up that space for > other apps that need it if they ask for more memory. > > : After indexing the memoryconsumption isn't dropping. Even after an > optimize > : command it's still there. > > as for why your "Used" memory grows to ~12GB and doesn't decrease even > after an optimize: that's the way the Java memory model works. whe nyou > run the JVM you specificy (either explicitly or implicitly via defaults) a > min & max heap size for hte JVM to allocate for itself. it starts out > asking the OS for the min, and as it needs more it asks for more up to the > max. but (most JVM implementations i know of) don't "give back" ram to > the OS if they don't need it anymore -- they keep it as free space in the > heap for future object allocation. > > > > -Hoss >
