Thanks, that's good feedback! Chris
On Mon, Aug 3, 2009 at 8:50 AM, Earwin Burrfoot <ear...@gmail.com> wrote: > > I'm curious if anyone has thought about (or even tried) caching the > low-level index data in Java, rather than > > in the OS. For example, at the IndexInput level there could be an LRU > cache of byte[] blocks, similar to > > how a RDBMS caches index pages. (Conveniently, BufferedIndexInput > already reads in 1k chunks.) You > > would reverse the advice above and instead make your JVM heap as large as > possible (or at least large > > enough to achieve a desired speed/space tradeoff). > I did something along these lines. It sucks. Having big Java heaps > ends you up with insane GC times. Loading GB-sized files into a bunch > of byte[1024] also wastes memory. Best bet by now is to rely on > mmap/file cache. > > > I think swappiness is exactly the configuration that tells Linux just > > how happily it should swapp out application memory for IO cache vs > > other IO cache for new IO cache. > swappiness is roughly the percentage of free memory after which OS > starts searching for pages suitable for paging out. If set to low > values, OS wakes up in near-OOM conditions. If set to high values, as > soon as OS decides (according to some heuristics) that page is > eligible for page-out, it goes to disk. > >
