: The current system I'm using has 150GB of memory and while I'm indexing the : memoryconsumption is growing and growing (eventually more then 50GB). : In the attached graph (http://postimage.org/image/acyv7kec/) I indexed about : 70k of office-documents (pdf,doc,xls etc) and between 1 and 2 percent throws
Unless i'm missunderstanding sometihng about your graph, only ~12GB of memory is used by applications on that machine. About 60GB is in use by the "filesystem cache". The Filesystem cache is not memory being used by Solr, it's memory that is free and not in use by an application, so your OS is (wisely) using it to cache files from disk that you've recently accessed in case you need them again. This is handy, and for max efficients (when keeping your index on disk) it's useful to make sure you allocate resources so that you have enough extra memory on your server that the entire index can be kept in the filesystem cache -- but the OS will happily free up that space for other apps that need it if they ask for more memory. : After indexing the memoryconsumption isn't dropping. Even after an optimize : command it's still there. as for why your "Used" memory grows to ~12GB and doesn't decrease even after an optimize: that's the way the Java memory model works. whe nyou run the JVM you specificy (either explicitly or implicitly via defaults) a min & max heap size for hte JVM to allocate for itself. it starts out asking the OS for the min, and as it needs more it asks for more up to the max. but (most JVM implementations i know of) don't "give back" ram to the OS if they don't need it anymore -- they keep it as free space in the heap for future object allocation. -Hoss
