Re: MergePolicy interface and SegmentInfos

2007-11-03 Thread Michael McCandless
"Chris Hostetter" <[EMAIL PROTECTED]> wrote: > I haven't really delved into the MergePolicy work that's been done, but a > recent Jira comment going me poking arround the javadocs -- MergePolicy is > a public interface, which suggests clients are allowed to impliment it, > leading me wonder a

DocumentsWriter questions

2007-11-03 Thread Leon
Hi,Michael McCandless Glad to know that you contributed the new optimize method to DocumentsWriter. i am a new one to lucene. However i think the class DocumentsWriter is not well designed. The first coming out question is 1: Why not extract the code of ThreadState management to a new

Re: DocumentsWriter questions

2007-11-03 Thread Michael McCandless
"Leon" <[EMAIL PROTECTED]> wrote: > 1: Why not extract the code of ThreadState management to a new > internal class such as ThreadStatePool. > > At present, there are lots of threadstate management code occured > everywhere in the DocumentsWriter. I'm not sure what you mean by "ThreadSta

Managing Growing Lucene Index size

2007-11-03 Thread Sandeep Mahendru
Hi , We have been developing an enterprise logging service at the Wachovia bank. The logs (Busines, application, error) for all the bank related applications are consolidated at one single location in an Oracle 10g Database. In our second phase, we are now building a high perforinmg report vie

Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown

2007-11-03 Thread Mark Miller
Is the new autocommit=false code atomic (the new check point is successfully made and moved to or its not)? If not I imagine it could be made to be without too much work right? Hoss Man (JIRA) wrote: [ https://issues.apache.org/jira/browse/LUCENE-1044?page=com.atlassian.jira.plugin.syste

Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown

2007-11-03 Thread Chris Hostetter
: Is the new autocommit=false code atomic (the new check point is successfully : made and moved to or its not)? If not I imagine it could be made to be without : too much work right? No matter what work we do in Java code to try and garuntee atomicity, the JVM can't garuntee that File IO buffers

Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown

2007-11-03 Thread Mark Miller
Derby does not guarantee atomicity? Other java DB's? I thought they did, but perhaps not. You cannot rig a simple system with some sort of rename call? If a hard power shutdown interrupted, on the next start the system could ideally be left in a condition that uses the previous commit point, o

Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown

2007-11-03 Thread robert engels
This is simple not true. See FileDescriptor.sync(). There are several options, but normally it is used so that when close completes, all data must be on disk. This is a much slower way to write data. It is very common in database systems when committing the log file. By using this, and pr

Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown

2007-11-03 Thread robert engels
See my previous email. I have written several 'atomic' ACID systems in java. It is VERY EASY TO DO. Read any introductory (academic) book on database systems. On Nov 3, 2007, at 7:08 PM, Mark Miller wrote: Derby does not guarantee atomicity? Other java DB's? I thought they did, but perhaps

Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown

2007-11-03 Thread Chris Hostetter
: This is simple not true. See FileDescriptor.sync(). : : There are several options, but normally it is used so that when close : completes, all data must be on disk. This is a much slower way to write data. : It is very common in database systems when committing the log file. Ok. I'll certainl

Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown

2007-11-03 Thread Mark Miller
I think it would be great if it where an option. Perhaps even a sandbox implementation that could wrap or replace a few classes. Maybe that complicates things too much, but that way you could just not use the transaction system if you where on NFS (if that ends up being a problem) or you didn't

Re: [jira] Resolved: (LUCENE-1044) Behavior on hard power shutdown

2007-11-03 Thread robert engels
I think using FD.sync() might have enabled the proper operation on NFS in a much simpler way... I argued quite a bit that the approach people were taking was probably not correct, and/or a bug in NFS (which it seems there was at least one that was corrected in a later release). IMHO the w