On Wednesday 14 August 2002 06:03 pm, Guido van Rossum wrote:
> > Fix for issue #505
> > ZCTextIndex is now associated by path to its lexicon. After replacing a 
lexicon used by an index, clear the index to make it use the new lexicon.
> So the semantics are that when you replace the lexicon, the index is
> reset to empty, right?  Why not create a new index instead?  Then the
> lexicon could be internal to the index.  Sharing lexicons doesn't
> sound like a probable use case, the more I think about it.
> --Guido van Rossum (home page: http://www.python.org/~guido/)

I don't disagree. This was a conceptual holdover from the previous generation 
TextIndex. I'm switching this over to zope-dev for wider discussion:

The current implementation of ZCTextIndex is like the old TextIndex in that 
you can create one Lexicon (the sucessor to Vocabularies) shared by multiple 

I imagine the thought was that there are only a finite number of words and 
that sharing the lexicon would save space and possibly index time, since a 
given word would only need to be inserted once into the lexicon regardless of 
the number of indexes it occurred in. More significant might be the (cache) 
memory savings of only having to keep one copy of the words in memory across 
several indexes. Plus fewer loads and stores to the database overall by 
sharing the word list.

On the other hand I think query speeds may be compromised since one large 
lexicon would take longer to search for a given word (or words) then several 
smaller ones. This would be especially true for small indexes sharing a 
lexicon with a much larger one.

The other downside (as illustrated by issue #505) is the complication of 
linking index to lexicon and making the link manageable so that you can tweak 
the indexing system easily. My fix is not entirely complete because a hard 
ref to the lexicon is still stored in the low-level index (to which the 
ZCTextIndex class delegates). In order to fix this effectively without 
introducing Zope dependancies at the low level (which we have looked to 
avoid) I would need to create some sort of Lexicon proxy that can access the 
correct lexicon on demand by a path efficiently. This proxy would be 
referenced by the low level index in place of the actual lexicon.

Of course the other solution, which is much simpler is to dispense with this 
notion of sharing lexicons entirely and as Guido suggests, just make the 
lexicon part of the index.

Without hard use cases to the contrary, I lean toward that simpler design. 
However I would like to perform some additional testing on large corpuses 
with many indexes to assess the memory/performance differences between these 
two approaches. We have already ascertained that with the new ZODB cache code 
in 2.6, the cache setting can have a profound affect on query performance 
(like a factor of 10), so I think testing would be helpful.

Anyone care to weigh in with use cases for shared lexicons?


Zope-Dev maillist  -  [EMAIL PROTECTED]
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope )

Reply via email to