On 6/5/06, Marvin Humphrey <[EMAIL PROTECTED]> wrote: > > On Jun 3, 2006, at 8:42 PM, David Balmain wrote: > > > Now that the Lucy[1] project has Apache approval and is about to > > begin, the onus is no longer on Ferret to strive for Lucene > > compatability. (We'll be doing that in Lucy). > > We'll take this up more aggressively once some under-appreciated > volunteers at Apache create mailing lists and other infrastructure > for Lucy, but I doubt we'll want to have Lucy's API mirror Lucene's > 100%. Do we really want separate Hits and HitIterator classes, for > instance? Other, more substantial issues are on the table too as far > as I'm concerned, such as whether deletions should handled by the > IndexReader rather than the IndexWriter. APIs are really hard to > change once defined, and not taking hard-won lessons from Lucene into > account would be a crime.
Thanks for pointing that out. I couldn't agree with you more. What I meant was that Lucy would be striving to maintain "index file format" compatibility (which I believe was the plan). I didn't make this very clear, though, as I was talking about changes to the API but as I was writing this I was thinking about what changes to the index file format would allow. > Lucy will definitely need a define-fields-once interface, so although > you're proposing stuff specifically for Ferret here, I'm studying it > with an eye towards using it with Lucy. My inclination is to start > with define-fields-once, then add dynamic field definitions later if > we have to. This sounds good to me. > > Of course, this raises the question, why do we need to specify that > > field :name is tokenized every time we create a :name field? Isn't it > > always going to be the same? > > The primary argument for allowing dynamic field definitions I've > seen, is not that the definition might change, but that each document > might contain previously undefined fields which are unknowable in > advance. The CNET/Solr folks, Yonik and Hoss, really, really care > about that. > > I think the idea of dynamic field definitions is weird. (A database > that allows you to change the table definition with each INSERT? > Huh?) I'm sure that CNet could have been done another way if dynamic > field definitions hadn't been available, but they're committed now. :( Actually, I fall into the category of people who like dynamic field definitions. I agree that they are not necessary but it certainly makes some things easy. For instance, in a rails application you can add models to an index and you get to specify within the model itself which of its fields will be added to the index. The index itself doesn't need to know which models will be indexed or how they will be indexed it just needs to know to store the id field and the model name field and index everything else. It's all about keeping it DRY. The part I don't like about lucene is *sometimes* being able to change a fields properties. > > What if we use a different value the next > > time we and a :name field? Well the answer to this last question is a > > specific set of rules; > > > > 1. Once you choose to index a field, that field is > > always indexed from that point forward. > > 2. Once you store term vectors, always store term > > vectors > > 3. Once you store positions, always store positions > > 4. Once you store offsets, always store offsets > > 5. Once you store norms, always store norms > > It's actually messier than that, isn't it? Just because you've > started marking a field as indexed doesn't mean that Lucene goes back > to all the documents that you've already processed and indexes that > field. Same deal with TermVectors, etc. > > At least in SQL, when you add a field to a table it goes and adds a > default value for every row. > > > The problem is that earlier documents will > > have been added without storing term vectors. Now I don't know the > > exact thinking behind these rules but it seems to me that it would be > > better to just keep whatever rule you used when you first added the > > document. If you want to add term vectors later, then re-index. > > 'Zactly! > > > So here's my radical api change proposal. You set a fields properties > > when you create the index and Document becomes (almost) a simple Hash > > object. > > KinoSearch thinks of documents like hashes, too. Lucene, however, > thinks of documents like arrays. > > > Actually, you may not have realized this, but you can almost > > do this currently in Ferret. Once you add the first instance of a > > field, that field's properties are set. From then on you and just add > > documents as Hash objects and each field will have the same properties > > as in that first document that was added. (This isn't true of the > > Store or boost properties. These are set on a per document basis.) > > Why not set Store once and for all per-field? And heck, why not > start with a default boost, but allow it to be overridden? My plan exactly. In my experimental version of Ferret I have a fields file along with the segments file. The fields file stores all the field metadata such as store, index, term-vector and field boosts. That way there is no need to maintain a separate FieldInfos file per segment. (This will make merging a lot more difficult but I'm still thinking about that one.) > > So here is a possible way example of the way I'd implement this; > > > > # the following might even look better in a YAML file. > > Ooo, nifty idea! How about a class whose sole purpose is to define > fields and generate the YAML file? Or, if we're thinking future > Lucene 2.1 file format, some Lucene-readable index definition file? Now this idea I like. Perhaps even a simple question/answer app to generate the index definition file. I'd guess that Lucene will probably end up going with XML rather than YAML. Cheers, Dave _______________________________________________ Ferret-talk mailing list [email protected] http://rubyforge.org/mailman/listinfo/ferret-talk
