On Thu, Jun 11, 2009 at 4:30 PM, Jason Rutherglen<jason.rutherg...@gmail.com> wrote: >> Yes please post feature requests to Sun ;) > > I signed up for > http://mail.openjdk.java.net/mailman/listinfo/nio-discuss
Looks like a fun list ;) >> But I think in the short term Lucene will have to drop to > native code to tell OS not to cache bytes read by segment > merging... > > LUCENE-1121 uses transferTo which presumably doesn't run bytes > through the IO cache? Granted it's slower on most platforms, but > could this be fixed in future Java releases? I fear transferTo may in fact burn through the IO cache... or, at least, when I tested it in the past, on multi-GB files, I saw vmem used by the process grow enormously... I think OS was choosing to use massive chunks of RAM as the intermediate buffer (slurping lots of bytes in, and then writing lots of bytes out), to minimize seeking. We got mixed results on the performance gains on that issue, and some (Windows Server 2003) were hideously slower. I think the OS may have been evicting dirty pages, or swapping app core out, or something, in order to free up RAM. We really need better control on what the OS does w/ IO and our RAM, but we lack that from Java. EG if I had the option I'd like to provide the option to Lucene users to pin certain data structures (like the terms index) to ensure the OS never swaps it out. But we just can't do that from Java... Mike --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
