And let's not forget the interesting bug in MMapDirectory: http://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/api/all/org/apache/lucene/store/MMapDirectory.html
"NOTE: memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space, e.g. by using a 64 bit JRE, or a 32 bit JRE with indexes that are guaranteed to fit within the address space. On 32 bit platforms also consult setMaxChunkSize(int) if you have problems with mmap failing because of fragmented address space. If you get an OutOfMemoryException, it is recommended to reduce the chunk size, until it works. Due to this bug in Sun's JRE, MMapDirectory's IndexInput.close() is unable to close the underlying OS file handle. Only when GC finally collects the underlying objects, which could be quite some time later, will the file handle be closed. This will consume additional transient disk usage: on Windows, attempts to delete or overwrite the files will result in an exception; on other platforms, which typically have a "delete on last close" semantics, while such operations will succeed, the bytes are still consuming space on disk. For many applications this limitation is not a problem (e.g. if you have plenty of disk space, and you don't rely on overwriting files on Windows) but it's still an important limitation to be aware of." If you're measuring by directory size (and not explicitly by the viewable files) you may very well be seeing this. Jason On Jun 16, 2013, at 4:53 AM, Erick Erickson <erickerick...@gmail.com> wrote: > Optimzing will _temporarily_ double the index size, > but it shouldn't be permanent. Is it possible that > you have inadvertently told Solr to keep an extra > snapshot? I think it's "numberToKeep" in your > replication handler, but I'm going from memory here. > > Best > Erick > > > On Fri, Jun 14, 2013 at 2:15 AM, Montu v Boda > <montu.b...@highqsolutions.com> wrote: >> Hi, >> >> i have replicate my index from 1.4 to 3.5 and after replication i try >> optimize the index in 3.5 with below URL. >> http://localhost:9002/solr35/collection1/update?optimize=true&commit=true >> >> when i optimize the index in 3.5, it's increase the index size to double. >> >> in 1.4 the size of index is 428GB and after optimization in 3.5 it becomes >> 791 GB. >> >> Thanks & Rrgards >> Montu v Boda >> >> >> >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/in-Solr-3-5-optimization-increase-the-index-size-to-double-tp4070433.html >> Sent from the Solr - User mailing list archive at Nabble.com.
