> I'm curious if anyone has thought about (or even tried) caching the low-level > index data in Java, rather than > in the OS. For example, at the IndexInput level there could be an LRU cache > of byte[] blocks, similar to > how a RDBMS caches index pages. (Conveniently, BufferedIndexInput already > reads in 1k chunks.) You > would reverse the advice above and instead make your JVM heap as large as > possible (or at least large > enough to achieve a desired speed/space tradeoff). I did something along these lines. It sucks. Having big Java heaps ends you up with insane GC times. Loading GB-sized files into a bunch of byte[1024] also wastes memory. Best bet by now is to rely on mmap/file cache.
> I think swappiness is exactly the configuration that tells Linux just > how happily it should swapp out application memory for IO cache vs > other IO cache for new IO cache. swappiness is roughly the percentage of free memory after which OS starts searching for pages suitable for paging out. If set to low values, OS wakes up in near-OOM conditions. If set to high values, as soon as OS decides (according to some heuristics) that page is eligible for page-out, it goes to disk. -- Kirill Zakharenko/Кирилл Захаренко (ear...@gmail.com) Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423 ICQ: 104465785 --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
