Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-05 Thread Chris Withers
Wolfram Kerber wrote: Could you explain the problems that should be solved by this a little? The idea is to provide a flexible, scalable and powerful indexing solution that works out of the box for Python, rather than Zope. I find it rather hard to contribute anything useful without

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-05 Thread Chris Withers
Casey Duncan wrote: I posted a few references I found around the web on info retrieval and indexing to the s'forge doc area. SF understands HTML, not structured text ;-) I think Chris'll need to approve them first I think before they are publically accessible. Yeah, you can do this now

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-05 Thread Wolfram Kerber
Wolfram Kerber wrote: Could you explain the problems that should be solved by this a little? The idea is to provide a flexible, scalable and powerful indexing solution that works out of the box for Python, rather than Zope. I find it rather hard to contribute anything useful without

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-05 Thread Chris Withers
Wolfram Kerber wrote: Ok, i've put it here : http://www.gallileus.info/gallileus/members/m_wolf/publications/100756611705 /10075665840/protoCat.zip I should put together some doc about the changes as well ... Indeed :-) I shall have a look though... Chris

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-04 Thread Chris Withers
Matt Hamilton wrote: if index1 is in SQL and index2 is in ZODB, for example, how would you go about efficiently combining results? Is there not a set datatype in python that could be used? There is, but what would happen if index1 returned 25,000 results and index2 returned 250 and you

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-04 Thread Matt Hamilton
On Tue, 4 Dec 2001, Chris Withers wrote: (it can do boolean searches too, but the book is mainly about ranking). Please god tell me they cover phrase matching :-S No they don't really (if I remember right). I think they do talk about storing the position of the word in the document, so

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-04 Thread Casey Duncan
Can you all recommend any other books on information retrieval? I was looking at Amazon last night, and I found a few that looked interesting, I'm just the type of guy that has to leaf through before I buy the damn thing. I guess I just need to find a good tech book store around here... I think

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-04 Thread Casey Duncan
Chris, hows about adding me to this project, my s'forge username is cduncan. Thanks. -Casey --- Chris Withers [EMAIL PROTECTED] wrote: Matt Hamilton wrote: if index1 is in SQL and index2 is in ZODB, for example, how would you go about efficiently combining results? Is there not

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-04 Thread Matt Hamilton
On Tue, 4 Dec 2001, Casey Duncan wrote: Can you all recommend any other books on information retrieval? I was looking at Amazon last night, and I found a few that looked interesting, I'm just the type of guy that has to leaf through before I buy the damn thing. I guess I just need to find a

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-04 Thread Chris Withers
Casey Duncan wrote: I think my first real proposal of any significance will be to replace the catalog with a truely industrial strength indexing bohemoth, that can be plugged into this whole component arch. thingamawhammy. *cough* --- http://sourceforge.net/projects/pythonindexer Care

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-04 Thread Chris Withers
Casey Duncan wrote: Chris, hows about adding me to this project, my s'forge username is cduncan. done... lemme know if ya need anything else :-) Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev **

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-12-04 Thread Wolfram Kerber
[EMAIL PROTECTED] Cc: Matt Hamilton [EMAIL PROTECTED]; Casey Duncan [EMAIL PROTECTED]; Steve Alexander [EMAIL PROTECTED]; Wolfram Kerber [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Tuesday, December 04, 2001 10:36 PM Subject: Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB Casey Duncan wrote

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-11-29 Thread Chris Withers
Casey Duncan wrote: I'm not sure I want to store the indexes in the ZODB, just index ZODB data at a low level. Ah, okay, and yes, in that case, I am in complete agreement ;-) (the level I'm aiming at is just to be able to index python objects, I'll leave plugging that into the ZODB

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-11-29 Thread Matt Hamilton
On Thu, 29 Nov 2001, Chris Withers wrote: I would rather avoid having to use a relational database unless I have to. Perhaps the index pluggability could be made to support different backends (like FileStorage et al does). Yeah, unfortunately, the difficult bit is combining queries:

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-11-29 Thread Jim Fulton
Chris Withers wrote: Toby Dickenson wrote: FileStorage is 'damn fast', so almost anything is going to be slower. Indeed, until it runs out of RAM for its indexes ;-) I wish you would finish testing the change I made for you. It should reduce the memory consumption by an order of

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-11-29 Thread Chris Withers
Jim Fulton wrote: Chris Withers wrote: Toby Dickenson wrote: FileStorage is 'damn fast', so almost anything is going to be slower. Indeed, until it runs out of RAM for its indexes ;-) I wish you would finish testing the change I made for you. Sorry, to be clear, my comment

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-11-28 Thread Chris Withers
Casey Duncan wrote: I would be willing to help both in coding and getting the code put into the Zope core. raises hand me too! Me three! :-) Just to put my take on all of this... As some of you may know, I've been looking at indexign for a while now in one way or another... I'm

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-11-28 Thread Andreas Jung
- Original Message - From: Chris Withers [EMAIL PROTECTED] To: Casey Duncan [EMAIL PROTECTED] Cc: Steve Alexander [EMAIL PROTECTED]; Wolfram Kerber [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Wednesday, November 28, 2001 09:37 Subject: Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

Re: [Zope-dev] Searching/Indexing/ZODB/SQL/BerkleyDB

2001-11-28 Thread Barry A. Warsaw
CW == Chris Withers [EMAIL PROTECTED] writes: How much slower was is? Did you measure ratios (between the two storages) of time per indexing operation, or ratios of disk blocks transferred per indexing operation? CW In my tests, Barry agreed with me that Berkley was turning