Yonik Seeley wrote:
On 8/18/06, Michael McCandless <[EMAIL PROTECTED]> wrote:
> One would also have to worry about partially deleted segments on
> Windows... while removing a segment, some of the files might fail to
> delete (due to still being open) and some might succeed.
Yes, I think this case is handled correctly. Once all searchers using
those old segments are closed, then the next commit that runs will
remove those files (just like it does today).
Unix systems don't have to worry about this.
Windows systems use "deletable" to track what they should try to delete
later.
How are you handling it? Get a full directory listing and try to
remove any older segment files?
I agree with the benefits of not requiring locking to open an
IndexReader, but I wonder at the performance cost of the directory
listing needed to "garbage collect" old segment files and to find the
newest "segments.xxx"
Agreed, it looks like the big tradeoff here is possible performance
loss (due to directory listings & IndexReader having to retry) vs.
better robustness with a lock-less design.
I'm planning on doing benchmarks to measure net system throughput
(#docs indexed, #queries run) across different OS's, remote/local
filesystems, and across "high frequency" to "low frequency" of
re-opening the IndexSearchers.
The good news (in my testing so far) is that the lock-less design is
functionally correct.
On deletable: yes, I'm currently GC'ing unused segments by doing a
full directory listing. But, this is actually a separable change:
reading/writing the deletable file only requires the write lock, so I
could keep the current approach. I was wanting to avoid the
File.renameTo deletable.new -> deletable which has hit "Access Denied"
on Windows in the past.
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
