On Fri, Jul 19, 2013 at 9:27 AM, Shai Erera <ser...@gmail.com> wrote:
> Fair enough, let's resolve it in a way that makes everyone happy. Not > being able to use DV on-disk "officially" seems like a drawback to me. > > I still prefer that we support both in-memory and on-disk and even default > to in-memory, because Lucene should have great performance out of the box, > and these days RAM is not so much an issue. Perhaps we should explore > writing DirectDVFormat which keeps everything in memory outside the heap > (this is for a separate issue). > > I get what you're saying about supporting formats, but I think that if > on-disk vs in-memory (for any format) means "same representation on disk, > different behavior at runtime", we should be able to support both variants? > That way, in-memory is just an optimized implementation detail, and is not > strictly a "Format". > But its not: they use different datastructures and so on. Different codecs. I dont think we should default to loading everything in heap. This is really trappy and its not interesting to me that fieldcache used to do this, its not relevant at all to what we are doing now. We should leave this to the OS like we do with postings lists and so on. I think diskdv is "correct" in that it loads up the minimal stuff it should need into heap to keep things fast (the monotonic blockpacked readers for addressing, which are typically very small).
