Re: external file stored field codec

2013-10-18 Thread Michael Sokolov
On 10/18/2013 1:08 AM, Shai Erera wrote: The codec intercepts merges in order to clean up files that are no longer referenced What happens if a document is deleted while there's a reader open on the index, and the segments are merged? Maybe I misunderstand what you meant by this statement, but

Re: external file stored field codec

2013-10-17 Thread Shai Erera
> > The codec intercepts merges in order to clean up files that are no longer > referenced > What happens if a document is deleted while there's a reader open on the index, and the segments are merged? Maybe I misunderstand what you meant by this statement, but if the external file is deleted, sin

Re: external file stored field codec

2013-10-17 Thread Michael Sokolov
On 10/13/13 8:09 PM, Michael Sokolov wrote: On 10/13/2013 1:52 PM, Adrien Grand wrote: Hi Michael, I'm not aware enough of operating system internals to know what exactly happens when a file is open but it sounds to be like having separate files per document or field adds levels of indirection

Re: external file stored field codec

2013-10-13 Thread Michael Sokolov
On 10/13/2013 1:52 PM, Adrien Grand wrote: Hi Michael, I'm not aware enough of operating system internals to know what exactly happens when a file is open but it sounds to be like having separate files per document or field adds levels of indirection when loading stored fields, so I would be sur

Re: external file stored field codec

2013-10-13 Thread Adrien Grand
Hi Michael, I'm not aware enough of operating system internals to know what exactly happens when a file is open but it sounds to be like having separate files per document or field adds levels of indirection when loading stored fields, so I would be surprised it it actually proved to be more effic

Re: external file stored field codec

2013-10-11 Thread Michael Sokolov
On 10/11/2013 03:19 PM, Michael Sokolov wrote: On 10/11/2013 03:04 PM, Adrien Grand wrote: On Fri, Oct 11, 2013 at 7:03 PM, Michael Sokolov wrote: I've been running some tests comparing storing large fields (documents, say 100K .. 10M) as files vs. storing them in Lucene as stored fields. In

Re: external file stored field codec

2013-10-11 Thread Michael Sokolov
On 10/11/2013 03:04 PM, Adrien Grand wrote: On Fri, Oct 11, 2013 at 7:03 PM, Michael Sokolov wrote: I've been running some tests comparing storing large fields (documents, say 100K .. 10M) as files vs. storing them in Lucene as stored fields. Initial results seem to indicate storing them exter

Re: external file stored field codec

2013-10-11 Thread Adrien Grand
On Fri, Oct 11, 2013 at 7:03 PM, Michael Sokolov wrote: > I've been running some tests comparing storing large fields (documents, say > 100K .. 10M) as files vs. storing them in Lucene as stored fields. Initial > results seem to indicate storing them externally is a win (at least for > binary doc

external file stored field codec

2013-10-11 Thread Michael Sokolov
I've been running some tests comparing storing large fields (documents, say 100K .. 10M) as files vs. storing them in Lucene as stored fields. Initial results seem to indicate storing them externally is a win (at least for binary docs which don't compress, and presumably we can compress the ex