On Jun 3, 2006, at 8:42 PM, David Balmain wrote:
> Now that the Lucy[1] project has Apache approval and is about to
> begin, the onus is no longer on Ferret to strive for Lucene
> compatability. (We'll be doing that in Lucy).
We'll take this up more aggressively once some under-appreciated
volunteers at Apache create mailing lists and other infrastructure
for Lucy, but I doubt we'll want to have Lucy's API mirror Lucene's
100%. Do we really want separate Hits and HitIterator classes, for
instance? Other, more substantial issues are on the table too as far
as I'm concerned, such as whether deletions should handled by the
IndexReader rather than the IndexWriter. APIs are really hard to
change once defined, and not taking hard-won lessons from Lucene into
account would be a crime.
Lucy will definitely need a define-fields-once interface, so although
you're proposing stuff specifically for Ferret here, I'm studying it
with an eye towards using it with Lucy. My inclination is to start
with define-fields-once, then add dynamic field definitions later if
we have to.
> Of course, this raises the question, why do we need to specify that
> field :name is tokenized every time we create a :name field? Isn't it
> always going to be the same?
The primary argument for allowing dynamic field definitions I've
seen, is not that the definition might change, but that each document
might contain previously undefined fields which are unknowable in
advance. The CNET/Solr folks, Yonik and Hoss, really, really care
about that.
I think the idea of dynamic field definitions is weird. (A database
that allows you to change the table definition with each INSERT?
Huh?) I'm sure that CNet could have been done another way if dynamic
field definitions hadn't been available, but they're committed now. :(
> What if we use a different value the next
> time we and a :name field? Well the answer to this last question is a
> specific set of rules;
>
> 1. Once you choose to index a field, that field is
> always indexed from that point forward.
> 2. Once you store term vectors, always store term
> vectors
> 3. Once you store positions, always store positions
> 4. Once you store offsets, always store offsets
> 5. Once you store norms, always store norms
It's actually messier than that, isn't it? Just because you've
started marking a field as indexed doesn't mean that Lucene goes back
to all the documents that you've already processed and indexes that
field. Same deal with TermVectors, etc.
At least in SQL, when you add a field to a table it goes and adds a
default value for every row.
> The problem is that earlier documents will
> have been added without storing term vectors. Now I don't know the
> exact thinking behind these rules but it seems to me that it would be
> better to just keep whatever rule you used when you first added the
> document. If you want to add term vectors later, then re-index.
'Zactly!
> So here's my radical api change proposal. You set a fields properties
> when you create the index and Document becomes (almost) a simple Hash
> object.
KinoSearch thinks of documents like hashes, too. Lucene, however,
thinks of documents like arrays.
> Actually, you may not have realized this, but you can almost
> do this currently in Ferret. Once you add the first instance of a
> field, that field's properties are set. From then on you and just add
> documents as Hash objects and each field will have the same properties
> as in that first document that was added. (This isn't true of the
> Store or boost properties. These are set on a per document basis.)
Why not set Store once and for all per-field? And heck, why not
start with a default boost, but allow it to be overridden?
> So here is a possible way example of the way I'd implement this;
>
> # the following might even look better in a YAML file.
Ooo, nifty idea! How about a class whose sole purpose is to define
fields and generate the YAML file? Or, if we're thinking future
Lucene 2.1 file format, some Lucene-readable index definition file?
> field_props = {
> :default => {:store => :no, :index
> => :tokenized, :term_vector => :no},
> :fields => {
> :id => {:store => :yes, :index => :no},
> :title => {:store => :yes, :term_vector
> => :with_positions_offsets},
> [:created_on, :updated_on] => {:store => :yes, :index =>
> :untokenized}
> }
> }
> index = Index.new(:field_properties => field_props)
This is nice and dense, but maybe a tad complicated.
KinoSearch's take on doing field defs has some problems too. It was
a mistake to make spec_field() a method of InvIndexer (KinoSearch's
index writer/modifier class). The index writer and reader classes
suffer from serious bloat no matter what, so anything that can be
shunted somewhere else should be.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
_______________________________________________
Ferret-talk mailing list
[email protected]
http://rubyforge.org/mailman/listinfo/ferret-talk
