Re: [Zope-dev] ZCatalog FieldIndex Bug (Fix?)
Chris McDonough wrote: > > > Chris McDonough wrote: > > > > > > You can run them by doing "python testCatalog.py -p"... some will fail > due > > > to not having various files on the filesystem, but most should pass. > > > > Ah... this doesn't work if you have Python 2.0 installed as your default > python > > and a Zope binary release ;-) > > Zope needs python 1.5.2 ;-) > > Thanks, professor. ;-) :-P > Will do! Thanks very much for tracking this down and sorry to not be more > helpful in the process. Pleasure, I wish I could have provided a unit test, but such is life ;-) cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex Bug (Fix?)
> Chris McDonough wrote: > > > > You can run them by doing "python testCatalog.py -p"... some will fail due > > to not having various files on the filesystem, but most should pass. > > Ah... this doesn't work if you have Python 2.0 installed as your default python > and a Zope binary release ;-) > Zope needs python 1.5.2 ;-) Thanks, professor. ;-) > I got 8 fails before and 8 fails after applying the tests. Sounds good to me!! > > I haven't had time to actually even look at the bug in the bugreport. > > Thanks for the possible fix to the bug that I don't understand yet, however! > > ;-) > > Well, import the .zexp I attached earlier in this thread, and run the 'dotest' > Python Script. Will do! Thanks very much for tracking this down and sorry to not be more helpful in the process. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex Bug (Fix?)
Chris McDonough wrote: > > You can run them by doing "python testCatalog.py -p"... some will fail due > to not having various files on the filesystem, but most should pass. Ah... this doesn't work if you have Python 2.0 installed as your default python and a Zope binary release ;-) Zope needs python 1.5.2 ;-) I got 8 fails before and 8 fails after applying the tests. > I haven't had time to actually even look at the bug in the bugreport. > Thanks for the possible fix to the bug that I don't understand yet, however! > ;-) Well, import the .zexp I attached earlier in this thread, and run the 'dotest' Python Script. cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex Bug (Fix?)
You can run them by doing "python testCatalog.py -p"... some will fail due to not having various files on the filesystem, but most should pass. I haven't had time to actually even look at the bug in the bugreport. Thanks for the possible fix to the bug that I don't understand yet, however! ;-) - Original Message - From: "Chris Withers" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]>; "Chris McDonough" <[EMAIL PROTECTED]> Sent: Thursday, April 05, 2001 6:41 PM Subject: [Zope-dev] ZCatalog FieldIndex Bug (Fix?) > Chris Withers wrote: > > Correct me if I'm wrong, but if I query a field index with an empty string, I > > expect to get returned all catalogued objects where that attribute's value _is_ > > the empty string. > > > > Anyway, this happens most of the time and I wish it happened all the time. > > Basically, the only situation where this _doesn't_ happen is when _none_ of the > > objects catalogued in the field index have a value of the empty string, which > > seems really bug and nasty to me ;-) > > I think I've got a solution, but I dunno if it's right. The patch is at the end. > How do I get the unit tests in ZCatalog/tests to run so I can check I haven't > screwed anything else up? > > cheers, > > Chris > > E:\Zope\2.3.1\lib\python\SearchIndex>diff -u unindex.py.old unindex.py > --- unindex.py.old Thu Apr 05 23:35:40 2001 > +++ unindex.py Thu Apr 05 23:37:46 2001 > @@ -407,9 +407,9 @@ > > if r is None: > if anyTrue: > -r=IISet() > -else: > return None > +else: > +r=IISet() > > return r, (id,) > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog
Marco Nova wrote: > > I was trying to customize the ZDiscussion product with file upload, editing > of previous posts and deletes of old posts Why not just use Squishdot with a 'Plain' demo? :-) Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog
> So if a a method raise an exception all the things done (that involve zodb) > in the method will be rolled back ? Yes. Actually, I need to clarify that. ZODB doesn't handle this for you. Zope does. When a request in Zope starts, a transaction is started. If the response is sent back without an error, the transaction is committed. If anything prevents the response from being sent back, the transaction is aborted. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ZCatalog
> > I'm a newbie about python and zope (about a week and a half > till now), so > > probably most of what I say now is probably wrong. > > > > I was trying to customize the ZDiscussion product with file upload, > editing > > of previous posts and deletes of old posts and I guess I've > find a bug in > > the ZCatalog.uncatalogObject method. > > Thanks for the report! > > This is already fixed in CVS. you're welcome. > > Or if we want to handle situations where the unindexs are > unsuccessfull, > we > > have to rewrite better the method with the use of transactions. > > We needn't worry about manually managing transactions here > because the ZODB > takes care of it for us. So if a a method raise an exception all the things done (that involve zodb) in the method will be rolled back ? - mn ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog
Hi Marco, > I'm a newbie about python and zope (about a week and a half till now), so > probably most of what I say now is probably wrong. > > I was trying to customize the ZDiscussion product with file upload, editing > of previous posts and deletes of old posts and I guess I've find a bug in > the ZCatalog.uncatalogObject method. Thanks for the report! This is already fixed in CVS. > Or if we want to handle situations where the unindexs are unsuccessfull, we > have to rewrite better the method with the use of transactions. We needn't worry about manually managing transactions here because the ZODB takes care of it for us. Thanks! - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog phrase indexing revisited
Oren Yosifon writes: > ... phrase searches ... > Can anyone help me ? URL:http://www.dieter.handshake.de/pyprojects/zope/near.pat is a patch that made near searches work ages ago for me. The patch is old (against Zope 2.1.6). As todays ZCatalog is quite different, it will not be easy to apply the patch to modern code. You only have a chance, when you are quite familiar with Python. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog phrase indexing revisited
Quoted phrase searching doesn't really work. Well, it does, but not the way you'd expect. Zope versions below 2.3.1b2 (all the way down to 2.2.1 AFAICT) used to choke and error out on quoted-phrase searching. But quoted-phrase searching in Zopes past 2.3.1b2 is essentially turned into an "AND" query instead of erroring out. So if you do "foo bar", it's roughly equivalent to "foo AND bar". This of course isn't what most people expect, but the machinery for NEAR searching in the catalog was used by the quoting operator (e.g. "foo bar" would become "foo NEAR bar"). Unfortunately, the NEAR searching machinery *never* worked (and still doesnt), so we had to turn the NEAR into an AND to get a reasonable, if misleading, result. Hopefully we can graft on real NEAR searching in the future. For now, I think "foo AND bar" is about as close as you're going to get to phrase searching without post-filtering results. - Original Message - From: "Oren Yosifon" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Thursday, March 29, 2001 9:35 AM Subject: [Zope-dev] ZCatalog phrase indexing revisited Hi all,I'm asking this again ,in case anything has changed since the last time.I'm trying to search for a phrase in ZCatalog, but, it doesn't seem towork.Let's say I want to search for "foo bar", if i ask ZCatalog to find it,it treats it as if I wrote "foo OR bar" .Last time I asked this, SteveA kindly replied: You should use quotes to group the phrase. ['"foo bar"'] Take a look in the source code: lib/python/SearchIndex/UnTextIndex.py Follow the flow of code through from line 550: def query(... however, this doesn't seem to work. You can try it even on the zope.orgwebsite using their search interface. Using the above suggested syntax(incl. the brackets) returns an error message.Can anyone help me ?Thx.oren.___Zope-Dev maillist - [EMAIL PROTECTED]http://lists.zope.org/mailman/listinfo/zope-dev** No cross posts or HTML encoding! **(Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog bug in 2.3.1b2
I cannot reproduce this, Andy, although I'm sure something is up. If it
continues to fail under b3, can you possibly send me a Data.fs that exhibits
the problem?
Sorry,
- C
- Original Message -
From: "Andy Dawkins" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Tuesday, March 20, 2001 11:02 AM
Subject: [Zope-dev] ZCatalog bug in 2.3.1b2
> I posted this before but didn't get a reply.
>
> Has any one seen this bug, Or do they know a fix?
>
> ATM this is stopping me using 2.3.1b2 which is a bad thing(tm)
>
> ---
> Zope Errorr
>
> Zope has encountered an error while publishing this resource.
>
> Error Type: TypeError
> Error Value: len() of unsized object
>
> With this traceback:
> Traceback (innermost last):
> File
> /usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/ZPublisher/Publish.py,
> line 223, in publish_module
> File
> /usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/ZPublisher/Publish.py,
> line 187, in publish
> File
> /usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/Zope/__init__.py,
> line 221, in zpublisher_exception_hook
> (Object: Traversable)
> File
> /usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/ZPublisher/Publish.py,
> line 171, in publish
> File
> /usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/ZPublisher/mapply.py,
> line
> 160, in mapply
> (Object: executeSearch)
> File
> /usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/ZPublisher/Publish.py,
> line 112, in call_object
> (Object: executeSearch)
> File
> /usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/OFS/DTMLMethod.py,
> line 189, in __call__
> (Object: executeSearch)
> File
>
/usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/DocumentTemplate/DT_String
.py,
> line 538, in __call__
> (Object: executeSearch)
> File
>
/usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/DocumentTemplate/DT_Let.py
,
>
> line 146, in render
> (Object: Results="Catalog(REQUEST=REQUEST)")
> File
>
/usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/DocumentTemplate/DT_Util.p
y,
> line 334, in eval
> (Object: Catalog(REQUEST=REQUEST))
> (Info: REQUEST)
> File , line 0, in ?
> File
>
/usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/Products/ZCatalog/ZCatalog
.py,
> line 530, in searchResults
> (Object: Traversable)
> File
>
>
/usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/Products/ZCatalog/Catalog.
py,
> line 654, in searchResults
> File
>
/usr/local/src/Zope-2.3.1b2-linux2-x86/lib/python/Products/ZCatalog/Catalog.
py,
> line 591, in _indexedSearch
> TypeError: (see above)
>
>
> Here is the line which calls the search:
>
> sr = self.searchResults({'Type' : type, 'sort_on':'lowercase_title_'})
>
> fyi. type = 'TypeTest'
>
> The indexes are as follows:
> Status = Text Index
> Type = Field Index
> Identifier = Field Index
> lowercase_title_ = Field Index
> parent_identifiers_ = Keyword Index
>
> The meta types are like this:
> Title
> id
> identifier
>
> There are only 13 records in the catalog.
> And there are some results to be returned.
>
> Cheers
> -Andy
>
> ___
> Zope-Dev maillist - [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope-dev
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://lists.zope.org/mailman/listinfo/zope-announce
> http://lists.zope.org/mailman/listinfo/zope )
>
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Bug?
On Fri, 16 Mar 2001 08:48:09 -0500, Edmund Goppelt <[EMAIL PROTECTED]> wrote: >Could the author have meant to write 'Exception' here instead of 'Except'? > >I am running Zope 2.3.0 stable. The current cvs has this changed to 'except AttributeError' Toby Dickenson [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
R. David Murray writes: > ... > The issue is that parts of the database get updated periodically from an > external source. That is, the author and book tables get replaced > wholesale. But the Book and Author objects are cataloged using > ZCatalog, doing a full text index on a combination of fields from > the external database and the html chunks. As far as I can see > at the moment, this means that every object has to get uncataloged > and recataloged, meaning every time the database update happens > the zodb inflates considerably, even though very little data has > actually changed. Maybe, you can determine whether something relevant changed and call "catalog_object" only in this case. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
> Will the new data be 'made ready for indexing' before it is compared to the > existing data? That is, will ZCatalog have to compute the data in some > way before it compares it to what is already stored? Yes. > I'm wondering because it would be significant overhead to 'make a data field > of 100MBs into an index-like value' and then compare it to what already > exists in the ZCatalog. Yes, it's potentially slower on update but causes less database bloat. Most importantly the update scheme causes fewer ConflictErrors on heavy-write sites because we're writing to the ZODB less often. That said, some of the operations of comparison in the indexing code are moving from Python to C, which should offset this penalty a little. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
> Cool. Now, in the examples I've seen for interfacing ZCatalog and > ZPatterns, the 'object updated' code does an unindex of the object > and then an index of the object. I copied that pattern for my > "the tables have been updated, reindex everything" code. > So what I should do instead is just do an index of the objects? Yup. Don't call uncatalog_object on the object before calling catalog_object. > This leaves me with a different problem, though. Sometimes when the tables > are updated objects dissapear (ie: were deleted from the external > database). I have to figure out how to delete those from the catalog. > A pain, but it shouldn't be too hard. This is a problem that ZCatalog doesn't address anyway (unless you're using "CatalogAware", blech). > > Note that the algoritm is simple - for each index, compare the what exists > > in the index to what is to be put in. If they're the same, do nothing. If > > they're different, reindex. I wasn't able to understand completely from > > your description whether the object method your're attempting to index via a > > TextIndex actually returns different data or not when you recatalog it. > > Does it? > > Yeah, it should be returning exactly the same data. I can stand the > update hit when the data actually changes (even though the change > will typically be only one or two words out of hundreds). We're working on integrating new BTree data structures into the indexes which will make the time/space hit very low when updating documents in which nothing has changed. I have no hard numbers, however, for the old or the new stuff. > > There's a huge possibility that the merge stuff isn't working completely > > because until now there have been no deterministic tests. > > Well, I don't know yet, since I was doing an unindex/reindex cycle. Well, give it a shot and tell me what happens! ;-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
On Sat, 3 Mar 2001, Chris McDonough wrote: > Has the physical path of the object changed? If not, the newer (2.3.0 + ) Nope. > catalog stuff should be smart enough to figure out whether anything inside > the object has changed during catalog_object. If nothing has changed, none > of the indexes or metadata columns should be updated. We're scrambling > right now to write tests for this kind of thing. :-( Cool. Now, in the examples I've seen for interfacing ZCatalog and ZPatterns, the 'object updated' code does an unindex of the object and then an index of the object. I copied that pattern for my "the tables have been updated, reindex everything" code. So what I should do instead is just do an index of the objects? This leaves me with a different problem, though. Sometimes when the tables are updated objects dissapear (ie: were deleted from the external database). I have to figure out how to delete those from the catalog. A pain, but it shouldn't be too hard. > Note that the algoritm is simple - for each index, compare the what exists > in the index to what is to be put in. If they're the same, do nothing. If > they're different, reindex. I wasn't able to understand completely from > your description whether the object method your're attempting to index via a > TextIndex actually returns different data or not when you recatalog it. > Does it? Yeah, it should be returning exactly the same data. I can stand the update hit when the data actually changes (even though the change will typically be only one or two words out of hundreds). > There's a huge possibility that the merge stuff isn't working completely > because until now there have been no deterministic tests. Well, I don't know yet, since I was doing an unindex/reindex cycle. --RDM ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
[Chris McDonough] | Note that the algoritm is simple - for each index, compare the what exists | in the index to what is to be put in. If they're the same, do nothing. If | they're different, reindex. I wasn't able to understand completely from | your description whether the object method your're attempting to index via a | TextIndex actually returns different data or not when you recatalog it. | Does it? Will the new data be 'made ready for indexing' before it is compared to the existing data? That is, will ZCatalog have to compute the data in some way before it compares it to what is already stored? I'm wondering because it would be significant overhead to 'make a data field of 100MBs into an index-like value' and then compare it to what already exists in the ZCatalog. Thanks, Morten ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
Has the physical path of the object changed? If not, the newer (2.3.0 + ) catalog stuff should be smart enough to figure out whether anything inside the object has changed during catalog_object. If nothing has changed, none of the indexes or metadata columns should be updated. We're scrambling right now to write tests for this kind of thing. :-( Note that the algoritm is simple - for each index, compare the what exists in the index to what is to be put in. If they're the same, do nothing. If they're different, reindex. I wasn't able to understand completely from your description whether the object method your're attempting to index via a TextIndex actually returns different data or not when you recatalog it. Does it? There's a huge possibility that the merge stuff isn't working completely because until now there have been no deterministic tests. - Original Message - From: "R. David Murray " <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Saturday, March 03, 2001 2:29 PM Subject: Re: [Zope-dev] ZCatalog hackery > I've got an analogous but different problem with ZCatalog udpates. I'd > like to ask for ideas about the best way to handle this. > > I've got a ZPatterns based ap that is pulling data from a postgresql > database. But associated with the object created out of the database > are signficant chunks of html (Author bios, book descriptions). I've > stored these html chunks in the database as well, though I don't think > that matters to the problem or possible solution strategies. > > The issue is that parts of the database get updated periodically from an > external source. That is, the author and book tables get replaced > wholesale. But the Book and Author objects are cataloged using > ZCatalog, doing a full text index on a combination of fields from > the external database and the html chunks. As far as I can see > at the moment, this means that every object has to get uncataloged > and recataloged, meaning every time the database update happens > the zodb inflates considerably, even though very little data has > actually changed. > > So far I can think of two simple solutions to this problem: (1) pack > often (2) put the Catalog into a mounted storage backed by a > non-undoable storage. I'd rather not do either one of these > (the first for obvious reasons, the second simply because I don't > want to take the time to learn how to set up a non-undoable storage). > Am I missing some other obvious options? It seems like there *ought* > to be a way to avoid the overhead of updating the catalog for objects > that haven't really changed. > > --RDM > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
I've got an analogous but different problem with ZCatalog udpates. I'd like to ask for ideas about the best way to handle this. I've got a ZPatterns based ap that is pulling data from a postgresql database. But associated with the object created out of the database are signficant chunks of html (Author bios, book descriptions). I've stored these html chunks in the database as well, though I don't think that matters to the problem or possible solution strategies. The issue is that parts of the database get updated periodically from an external source. That is, the author and book tables get replaced wholesale. But the Book and Author objects are cataloged using ZCatalog, doing a full text index on a combination of fields from the external database and the html chunks. As far as I can see at the moment, this means that every object has to get uncataloged and recataloged, meaning every time the database update happens the zodb inflates considerably, even though very little data has actually changed. So far I can think of two simple solutions to this problem: (1) pack often (2) put the Catalog into a mounted storage backed by a non-undoable storage. I'd rather not do either one of these (the first for obvious reasons, the second simply because I don't want to take the time to learn how to set up a non-undoable storage). Am I missing some other obvious options? It seems like there *ought* to be a way to avoid the overhead of updating the catalog for objects that haven't really changed. --RDM ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
Casey Duncan wrote: > > "Morten W. Petersen" wrote: > > > > Hi guys, > > > > I've got a problem with ZCatalog. I've got plenty of large > > objects, ranging from 100KB to 100MB in size. Needless to > > say, these take up a lot of processor time when indexed by > > the ZCatalog. > > > > Now, these object have to be moved from time to time, only > > moved, so that one or two of the related columns and indexes > > are affected (the path and the parent container id); is > > there a way to go around this so that those two, the > > one column and the index are updated, and not, let's say > > the indexed body (which is 100KB - 100MB in size) ? > > > > Thanks, > > > > Morten > > > > There is no built-in way. however I see no reason that the Catalog could > not be extended to do this with a bit of python coding. > > A catalog contains a collection called indexes that contains the indexes > themselves (what else?). An External method could be written like so > (not tested): > > def partialReindexObject(catalog, indexes, uid, object): > rid = catalog.uids.get[uid] > > for idx_key in indexes: > idx = catalog.indexes[idx_key].__of__(catalog) > idx.unindex_object(rid) > idx.index_object(rid, object, None) > > I think that will do it. Pass the names of the indexes as a sequence in > indexes, and data_record_id_ as the uid. > > The above does not update any meta-data in the catalog. > Actually what I wrote assumes you are passing a Catalog not a ZCatalog. So you will need to change it for a ZCatalog to: def partialReindexObject(zcatalog, indexes, uid, object): catalog = zcatalog._catalog rid = catalog.uids.get[uid] for idx_key in indexes: idx = catalog.indexes[idx_key].__of__(catalog) idx.unindex_object(rid) idx.index_object(rid, object, None) Sorry about that. -- | Casey Duncan | Kaivo, Inc. | [EMAIL PROTECTED] `--> ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
[Casey Duncan] | Actually what I wrote assumes you are passing a Catalog not a ZCatalog. | So you will need to change it for a ZCatalog to: I figured that out. :-) There is one problem, the uids stored in the Catalog are based on the path of the object, so I guess I'll have to make a copy of the records, and then paste it in under the new uid (path). -Morten ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog hackery
"Morten W. Petersen" wrote: > > Hi guys, > > I've got a problem with ZCatalog. I've got plenty of large > objects, ranging from 100KB to 100MB in size. Needless to > say, these take up a lot of processor time when indexed by > the ZCatalog. > > Now, these object have to be moved from time to time, only > moved, so that one or two of the related columns and indexes > are affected (the path and the parent container id); is > there a way to go around this so that those two, the > one column and the index are updated, and not, let's say > the indexed body (which is 100KB - 100MB in size) ? > > Thanks, > > Morten > There is no built-in way. however I see no reason that the Catalog could not be extended to do this with a bit of python coding. A catalog contains a collection called indexes that contains the indexes themselves (what else?). An External method could be written like so (not tested): def partialReindexObject(catalog, indexes, uid, object): rid = catalog.uids.get[uid] for idx_key in indexes: idx = catalog.indexes[idx_key].__of__(catalog) idx.unindex_object(rid) idx.index_object(rid, object, None) I think that will do it. Pass the names of the indexes as a sequence in indexes, and data_record_id_ as the uid. The above does not update any meta-data in the catalog. -- | Casey Duncan | Kaivo, Inc. | [EMAIL PROTECTED] `--> ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
"R. David Murray" wrote: > > On Thu, 22 Feb 2001, Chris Withers wrote: > > Apart from that, any ideas how I'd search for a record where the indexed > > attribute is blank or the secondary sort stuff? > > Somebody contributed a patch for the secondary sort stuff a while back. > Whether it still works with 2.3.0 is a good question. Any idea where I'd find that? I may need to give it a go... > I also remember a *long* time ago (back when I was first starting with > zope about a year ago) someone posting a method for picking up all > the records in a catalog with a blank value for a field. How you'd > find it in the mail archives, though, I have no clue, nor do I > remember what the method involved. Likewise with this. The only solution I can think of at the moment is to have a seperate 'x_is_blank' index that set to 'not x', but that seems terribly inefficient to me :-S Any ideas gratefully recieved :-) Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog problems
Neat!
- Original Message -
From: "Steve Alexander" <[EMAIL PROTECTED]>
To: "Morten W. Petersen" <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Monday, February 26, 2001 1:31 PM
Subject: Re: [Zope-dev] ZCatalog problems
> Morten W. Petersen wrote:
>
> > Hi guys,
> >
> > I'm having trouble making ZCatalog work. The problem is that there
> > are 29 objects of a given meta type, with the same booleans that
> > should be returned for an iteration; but only 20 are. Is this a
> > result of caching perhaps? Or lazy results?
>
> Could be that your objects have different ideas of what are proper
> boolean values.
>
> I assume you're using a FieldIndex. Try calling
> uniqueValuesFor(index_name) to see what values are stored. You may find
> that you have 0, 1, '1' and '', for example.
>
> In situations like that, I get the unique values, then construct a query
> for the catalog based on the true or false values.
>
> To get the true values, something like
>
>Catalog(boolean_index=
> filter(None, Catalog.uniqueValuesFor('boolean_index'))
>
> To get the false ones:
>
>Catalog(boolean_index=
> filter(lambda x: not x,
> Catalog.uniqueValuesFor('boolean_index'))
>
> --
> Steve Alexander
> Software Engineer
> Cat-Box limited
>
>
> ___
> Zope-Dev maillist - [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope-dev
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://lists.zope.org/mailman/listinfo/zope-announce
> http://lists.zope.org/mailman/listinfo/zope )
>
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog problems
Morten W. Petersen wrote:
> Hi guys,
>
> I'm having trouble making ZCatalog work. The problem is that there
> are 29 objects of a given meta type, with the same booleans that
> should be returned for an iteration; but only 20 are. Is this a
> result of caching perhaps? Or lazy results?
Could be that your objects have different ideas of what are proper
boolean values.
I assume you're using a FieldIndex. Try calling
uniqueValuesFor(index_name) to see what values are stored. You may find
that you have 0, 1, '1' and '', for example.
In situations like that, I get the unique values, then construct a query
for the catalog based on the true or false values.
To get the true values, something like
Catalog(boolean_index=
filter(None, Catalog.uniqueValuesFor('boolean_index'))
To get the false ones:
Catalog(boolean_index=
filter(lambda x: not x,
Catalog.uniqueValuesFor('boolean_index'))
--
Steve Alexander
Software Engineer
Cat-Box limited
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
If you're using getobject to retrieve bog-standard indexed DTML methods, Python Scripts, etc., you should never get an unauthorized from getobject (although you *will* later get an unauthorized if youy try to do anything with that object if you don't have an adequate set of roles). It's only if you're using getobject to retrieve custom instances that it could conceivably happen in getobject itself. Or at least that's what my Zope here tells me. :-) These problems suck. - Original Message - From: "Erik Enge" <[EMAIL PROTECTED]> To: "Chris McDonough" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Saturday, February 24, 2001 3:18 PM Subject: Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.) > On Sat, 24 Feb 2001, Chris McDonough wrote: > > > I can't replicate this behavior with normal objects. :-( > > Riiight... > > *thinking very hard* > > I just realized how it all started. Me and a collegue was indexing > objects, searching, doing regular development (probably changing classes, > attributes etc.) and suddenly we were asked for the (then) SuperUser > password. This was under 2.2.5. > > I remember thinking: "hm. that's odd...". But I had more important > things on my mind (and this wasn't really my field of expertice) so I > didn't dwell much on it. > > I'm trying to remember what we did when it happend, but I'm sorry to say I > can't (it's been two, three weeks now). But, the good point - I guess - > is that it hasn't got anything to do with Zope 2.3.1b1 in perticular. > > The sad thing, is that the problem is probably due to some of Python > classes begin set up correctly, or accordingly to the Product API. That's > what I'm guessing anyways. > > I'll run some more tests to make sure these assumptions are correct. More > feedback, and hopefully a solution, to come this week. > > Sorry for not realizing this sooner, it would've saved us all lots of > time. > > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
On Sat, 24 Feb 2001, Chris McDonough wrote: > I can't replicate this behavior with normal objects. :-( Riiight... *thinking very hard* I just realized how it all started. Me and a collegue was indexing objects, searching, doing regular development (probably changing classes, attributes etc.) and suddenly we were asked for the (then) SuperUser password. This was under 2.2.5. I remember thinking: "hm. that's odd...". But I had more important things on my mind (and this wasn't really my field of expertice) so I didn't dwell much on it. I'm trying to remember what we did when it happend, but I'm sorry to say I can't (it's been two, three weeks now). But, the good point - I guess - is that it hasn't got anything to do with Zope 2.3.1b1 in perticular. The sad thing, is that the problem is probably due to some of Python classes begin set up correctly, or accordingly to the Product API. That's what I'm guessing anyways. I'll run some more tests to make sure these assumptions are correct. More feedback, and hopefully a solution, to come this week. Sorry for not realizing this sooner, it would've saved us all lots of time. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
I can't replicate this behavior with normal objects. :-( - Original Message - From: "Erik Enge" <[EMAIL PROTECTED]> To: "Chris McDonough" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Saturday, February 24, 2001 4:58 AM Subject: Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.) > On Fri, 23 Feb 2001, Erik Enge wrote: > > > Feedback and debug information coming your way as soon as possible :) > > Ok, I index DTML Methods, Python objects, and all different kind of > things. Then I did a search, meta_type set to 'DTML Method' and it gave > me an unauthorized. Strangeness. > > I've installed the latest Hotfix, but that didn't make any difference > either. > > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
On Fri, 23 Feb 2001, Erik Enge wrote: > Feedback and debug information coming your way as soon as possible :) Ok, I index DTML Methods, Python objects, and all different kind of things. Then I did a search, meta_type set to 'DTML Method' and it gave me an unauthorized. Strangeness. I've installed the latest Hotfix, but that didn't make any difference either. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
On Fri, 23 Feb 2001, Chris McDonough wrote: > I meant to narrow down the problem domain in cases where you do call > getobject... cases where you aren't calling getobject are not relevant. Ok. I see. > It would be helpful to find out for which objects getobject fails and for > which it succeeds under your setup. I suspect that it's a __roles__ problem > with a type of object being retrieved. Interessting... > Narrowing the problem down to a set of objects for which getobject *always* > fails is probably the first step. Then maybe I can reproduce it here if you > can give me the code for these objects. I'll see if I don't have the time to do this bright and early in the morning, but - as I've said - I suspect it is true for all kinds of objects and meta_types. Feedback and debug information coming your way as soon as possible :) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
> > And any access to getobject with any data_record_id_ returns unauthorized > > for any user besides emergency user? > > Hm... No, not entirely correct. If I don't get any hits, I don't get the > unauthorized, but that is probably because I don't even try the > getobject. The point I'm making is that I might be able to call getobject > with a bogus data_record_id_ (although I highly doubt it). It wouldn't return unauthorized if you passed it a bogus data_record_id_, it would fail differently. I meant to narrow down the problem domain in cases where you do call getobject... cases where you aren't calling getobject are not relevant. It would be helpful to find out for which objects getobject fails and for which it succeeds under your setup. I suspect that it's a __roles__ problem with a type of object being retrieved. These kind of problems are notoriously difficult to debug. :-( > Steve Alexander suggested that I try the ZCatalog from CVS. The problem > is fixed there, isn't it? You can try it, but I don't think there's any significant difference between the ZCatalog in CVS and the one in Zope 2.3b1 if you see "unrestrictedTraverse" in your 2.3b1's ZCatalog.py's getobject method. Narrowing the problem down to a set of objects for which getobject *always* fails is probably the first step. Then maybe I can reproduce it here if you can give me the code for these objects. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
On Fri, 23 Feb 2001, Chris McDonough wrote: > And any access to getobject with any data_record_id_ returns unauthorized > for any user besides emergency user? Hm... No, not entirely correct. If I don't get any hits, I don't get the unauthorized, but that is probably because I don't even try the getobject. The point I'm making is that I might be able to call getobject with a bogus data_record_id_ (although I highly doubt it). Steve Alexander suggested that I try the ZCatalog from CVS. The problem is fixed there, isn't it? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
And any access to getobject with any data_record_id_ returns unauthorized for any user besides emergency user? Or is it a specific set of objects? A specific set of users? - Original Message - From: "Erik Enge" <[EMAIL PROTECTED]> To: "Chris McDonough" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Friday, February 23, 2001 2:01 PM Subject: Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.) > On Fri, 23 Feb 2001, Chris McDonough wrote: > > > different python classes. Do they inherit from a common base class? > > Yes. A homemade one. > > > Did you make these objects or are they from another Product or are > > they standard Zope objects (like DTML methods, etc.)? > > It's a mix. Most of them I created (probably around 30-50 classes) and > the remaining (5-10) are variants of DTML Method, Image, File and others. > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
On Fri, 23 Feb 2001, Chris McDonough wrote: > different python classes. Do they inherit from a common base class? Yes. A homemade one. > Did you make these objects or are they from another Product or are > they standard Zope objects (like DTML methods, etc.)? It's a mix. Most of them I created (probably around 30-50 classes) and the remaining (5-10) are variants of DTML Method, Image, File and others. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
On Fri, 23 Feb 2001, Chris McDonough wrote: > > It happens with all kinds of objects I have in my index. All my objects > > in the index are from Python-based products. > > Are they all of one type? The same type (as in, they are all Python based), but with different meta_types. > Can you replicate the behavior across many different kinds of > objects, or is this problem only with a single type of object? Different kinds of meta_types or ZClass/Python classes? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
different python classes. Do they inherit from a common base class? Did you make these objects or are they from another Product or are they standard Zope objects (like DTML methods, etc.)? - Original Message - From: "Erik Enge" <[EMAIL PROTECTED]> To: "Chris McDonough" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Friday, February 23, 2001 1:55 PM Subject: Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.) > On Fri, 23 Feb 2001, Chris McDonough wrote: > > > > It happens with all kinds of objects I have in my index. All my objects > > > in the index are from Python-based products. > > > > Are they all of one type? > > The same type (as in, they are all Python based), but with different > meta_types. > > > Can you replicate the behavior across many different kinds of > > objects, or is this problem only with a single type of object? > > Different kinds of meta_types or ZClass/Python classes? > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
> > "No" to which question? No to the question "And is there a way to actuall give access to restrictedTraverse (or, probably more corretly, to let it traverse)?" > > what are you getobjecting? Does it happen only with certain kinds of > > objects? Are they ZClass objects or Product-based objects? > > It happens with all kinds of objects I have in my index. All my objects > in the index are from Python-based products. Are they all of one type? unrestrictedTraverse uses the security policy code to determine user access to an object, which is at this point (unfortunately) different than the way that publishing traversal security works. Can you replicate the behavior across many different kinds of objects, or is this problem only with a single type of object? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
Hi Chris, RDM.. There's also Zieve for this sort of thing... it seems to work well enough for me... -steve > "RDM" == R David Murray <" <[EMAIL PROTECTED]>> writes: RDM> On Thu, 22 Feb 2001, Chris Withers wrote: >> Apart from that, any ideas how I'd search for a record where >> the indexed attribute is blank or the secondary sort stuff? RDM> Somebody contributed a patch for the secondary sort stuff a RDM> while back. Whether it still works with 2.3.0 is a good RDM> question. RDM> I also remember a *long* time ago (back when I was first RDM> starting with zope about a year ago) someone posting a method RDM> for picking up all the records in a catalog with a blank RDM> value for a field. How you'd find it in the mail archives, RDM> though, I have no clue, nor do I remember what the method RDM> involved. RDM> --RDM RDM> ___ Zope-Dev RDM> maillist - [EMAIL PROTECTED] RDM> http://lists.zope.org/mailman/listinfo/zope-dev ** No cross RDM> posts or HTML encoding! ** (Related lists - RDM> http://lists.zope.org/mailman/listinfo/zope-announce RDM> http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
On Thu, 22 Feb 2001, Chris Withers wrote: > Apart from that, any ideas how I'd search for a record where the indexed > attribute is blank or the secondary sort stuff? Somebody contributed a patch for the secondary sort stuff a while back. Whether it still works with 2.3.0 is a good question. I also remember a *long* time ago (back when I was first starting with zope about a year ago) someone posting a method for picking up all the records in a catalog with a blank value for a field. How you'd find it in the mail archives, though, I have no clue, nor do I remember what the method involved. --RDM ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog & distinct in SQL
--- Valérie Aulnette <[EMAIL PROTECTED]> wrote: > Hi, > > I am using PTK and it does work with ZCatalog. > I need to do the list of values that are in the > catalog but I need do do like a "select distinct" > in > SQL to have each value only once. > Do you know how to do that in ZCatalog ? > > Thanks. > Valerie. > > For fieldindexes there is a catalog method uniqueValuesFor(name) which when passed the name of the index returns the unique values within it. hth, = | Casey Duncan | Kaivo, Inc. | [EMAIL PROTECTED] `-> __ Do You Yahoo!? Yahoo! Auctions - Buy the things you want at great prices! http://auctions.yahoo.com/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
On Fri, 23 Feb 2001, Chris McDonough wrote: > No... "No" to which question? > what are you getobjecting? Does it happen only with certain kinds of > objects? Are they ZClass objects or Product-based objects? It happens with all kinds of objects I have in my index. All my objects in the index are from Python-based products. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
No... what are you getobjecting? Does it happen only with certain kinds of objects? Are they ZClass objects or Product-based objects? - Original Message - From: "Erik Enge" <[EMAIL PROTECTED]> To: "Chris McDonough" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Friday, February 23, 2001 9:23 AM Subject: Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.) > On Thu, 22 Feb 2001, Chris McDonough wrote: > > > I'm not sure why this isn't in 2.3.1b1, but yes, the code in getobject was > > changed to use unrestrictedTraverse for this very reason. > > On closer inspection, I can see that it is actually changed in > 2.3.1b1. It does say unrestrictedTraverse (line 457, I belive). I > grepped for restrictedTraverse, but couldn't find any occurances > (lib/python/Products/ZCatalog). Why am I still being asked to log in as > emergencyuser to be able to do this, if I'm not using > restrictedTraverse. And is there a way to actuall give access to > restrictedTraverse (or, probably more corretly, to let it traverse)? > > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
On Thu, 22 Feb 2001, Chris McDonough wrote: > I'm not sure why this isn't in 2.3.1b1, but yes, the code in getobject was > changed to use unrestrictedTraverse for this very reason. On closer inspection, I can see that it is actually changed in 2.3.1b1. It does say unrestrictedTraverse (line 457, I belive). I grepped for restrictedTraverse, but couldn't find any occurances (lib/python/Products/ZCatalog). Why am I still being asked to log in as emergencyuser to be able to do this, if I'm not using restrictedTraverse. And is there a way to actuall give access to restrictedTraverse (or, probably more corretly, to let it traverse)? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Lazy Results
Tres Seaver writes: > 'getObject' forces you to "wake up" the catalogued object; > in a large enough result set, the price is pretty punitive, > compared to serving the results as cached metadata. You can batch large result sets: "getObject" would just be applied to the objects in the batch. The objects are cached, too (ZODB cache). Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
You can get the object, but you can't do anything with it. - Original Message - From: "Steve Alexander" <[EMAIL PROTECTED]> To: "Chris McDonough" <[EMAIL PROTECTED]> Cc: "Erik Enge" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Thursday, February 22, 2001 12:38 PM Subject: Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.) > Chris McDonough wrote: > > > I'm not sure why this isn't in 2.3.1b1, but yes, the code in getobject was > > changed to use unrestrictedTraverse for this very reason. > > Does that open up a security hole? > > Can I get to an object via the getobject method of ZCatalog that I can't get to otherwise? > > I thought that was the reason it was changed to restrictedTraverse in the first place. > > > -- > Steve Alexander > Software Engineer > Cat-Box limited > http://www.cat-box.net > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
Chris McDonough wrote: > I'm not sure why this isn't in 2.3.1b1, but yes, the code in getobject was > changed to use unrestrictedTraverse for this very reason. Does that open up a security hole? Can I get to an object via the getobject method of ZCatalog that I can't get to otherwise? I thought that was the reason it was changed to restrictedTraverse in the first place. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
> Well hell, what's good enough for Perl is good enough > for... *slaps self silly* > > C'mon, are you really seriously gonna back up > reasoning for Zope features by quoting Larry Wall??? > 8^) Why not ? (sometimes) he makes sense ;) -- Andy McKay. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.)
I'm not sure why this isn't in 2.3.1b1, but yes, the code in getobject was changed to use unrestrictedTraverse for this very reason. - Original Message - From: "Erik Enge" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Thursday, February 22, 2001 10:33 AM Subject: [Zope-dev] ZCatalog madness. (Must log in as emergencyuser.) > Right. > > This is the traceback I get, after doing a search (searchResults) - which > goes fine by teh way - and then trying to do an > getobject(x.data_record_id_) as a non-emergencyuser user get up an login > box and press escape: > > Unauthorized > > Sorry, a Zope error occurred. > > Traceback (innermost last): > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/Publish.py, line > 222, in publish_module > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/Publish.py, line > 187, in publish > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/Publish.py, line > 171, in publish > File /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/mapply.py, > line 160, in mapply > (Object: candidate_search) > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/Publish.py, line > 112, in call_object > (Object: candidate_search) > File /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/OFS/DTMLMethod.py, > line 189, in __call__ > (Object: candidate_search) > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/DocumentTemplate/DT_String.py, > line 538, in __call__ > (Object: candidate_search) > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/DocumentTemplate/DT_In.py, > line 702, in renderwob > (Object: candidateEngine(REQUEST)) > Unauthorized: 0 > > Right. Your average unauthorized I should think. Next, if I log in as > the emergencyuser, this is what I get: > > Traceback (innermost last): > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/Publish.py, line > 222, in publish_module > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/Publish.py, line > 187, in publish > File /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/Zope/__init__.py, > line 221, in zpublisher_exception_hook > (Object: Traversable) > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/Publish.py, line > 171, in publish > File /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/mapply.py, > line 160, in mapply > (Object: candidate_search) > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/ZPublisher/Publish.py, line > 112, in call_object > (Object: candidate_search) > File /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/OFS/DTMLMethod.py, > line 189, in __call__ > (Object: candidate_search) > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/DocumentTemplate/DT_String.py, > line 538, in __call__ > (Object: candidate_search) > File > /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/DocumentTemplate/DT_In.py, > line 711, in renderwob > (Object: candidateEngine(REQUEST)) > File /usr/local/Zope-2.3.1b1-linux2-x86/lib/python/OFS/Traversable.py, > line 107, in absolute_url > (Object: CatalogAware) > AttributeError: (see above) > > Error Type: AttributeError > Error Value: 'string' object has no attribute 'get' > > > The get thing is probably from me trying to call absolute_url on the > objects it returns. Calling atributes works, though. > > Strange. Doing getobject as a non-emergencyuser doesn't owrk, and when I > have logged in the objects I get seems to be somewhat fubar. > > Help is greatly appretiated. > > (PS. I seem to recall a checkin in hte CVS about unrestrictedTraverse in > one of the files belonging to the Catalog, could this has something to do > with it?) > > Zope 2.3.1b1 on Linux. > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
--- Andy McKay <[EMAIL PROTECTED]> wrote: > > I disagree but it's not worth arguing about. > However, the people who pay > my > > salary feel that way which means I care ;-) > > Ive been taking the Larry Wall approach to this when > people tell me the > correct way to do things: > > "A Perl script is "correct" if it gets the job done > before your boss fires > you." > [snip] Well hell, what's good enough for Perl is good enough for... *slaps self silly* C'mon, are you really seriously gonna back up reasoning for Zope features by quoting Larry Wall??? 8^) = | Casey Duncan | Kaivo, Inc. | [EMAIL PROTECTED] `-> __ Do You Yahoo!? Yahoo! Auctions - Buy the things you want at great prices! http://auctions.yahoo.com/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
> I disagree but it's not worth arguing about. However, the people who pay my > salary feel that way which means I care ;-) Ive been taking the Larry Wall approach to this when people tell me the correct way to do things: "A Perl script is "correct" if it gets the job done before your boss fires you." The simple fact is many non-technical users dont give a hoot about the underlying python reasons or computer science, its just common to see a case-insensitive sort. And yes it can be done in ZCatalog and dtml-in with a little fiddling. -- Andy McKay. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
On Thu, 22 Feb 2001, Chris Withers wrote: > Tres Seaver wrote: > > > > Chris, this necrohippophilic sadism has to stop! > > Erm? I think my education is lacking... what does that mean? :-S "Beating a dead horse". :) > > "most humeans" > > does *not* map to "English speakers who prefer case-insensitive > > collation." > > I disagree but it's not worth arguing about. However, the > people who pay my salary feel that way which means I care ;-) > > > case-insensitive search / sort, then create a method on your > > object which does the approprate case-flattening, and index that > > instead of the "real" attribute. > > Thanks, I always forget the flexibility of ZCatalog :-) That > said, how hard would it be to alter ZCatalog so you could > override the comparison function it uses? It feels like a > ,cmp=cmp argument to the added search index consructor, then a > self._cmp = cmp in the constructor, followed by a query and > replace on cmp with self._cmp in the search index source. But I > might be completely wrong :-( The comparison function is built into the structure the of the index, and is not swappable "on-the-fly". When the catalog allows "drop-in" indexes, then you could define a CaseInsensitifeUnIndex class which implemented the semantics you prefer (but I still think it should be done at the application level, as I described above). > Apart from that, any ideas how I'd search for a record where > the indexed attribute is blank or the secondary sort stuff? "Trust the source, Luke!" is the best answer I've got. Tres. -- === Tres Seaver[EMAIL PROTECTED] Digital Creations "Zope Dealers" http://www.zope.org ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
Chris Withers wrote: > Tres Seaver wrote: > >> Chris, this necrohippophilic sadism has to stop! > > > Erm? I think my education is lacking... what does that mean? :-S necrohippo philic sadism dead-horse-loving act of deliberate harm upon -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
Tres Seaver wrote: > > Chris, this necrohippophilic sadism has to stop! Erm? I think my education is lacking... what does that mean? :-S > "most humeans" > does *not* map to "English speakers who prefer case-insensitive > collation." I disagree but it's not worth arguing about. However, the people who pay my salary feel that way which means I care ;-) > case-insensitive search / sort, then create a method on your > object which does the approprate case-flattening, and index that > instead of the "real" attribute. Thanks, I always forget the flexibility of ZCatalog :-) That said, how hard would it be to alter ZCatalog so you could override the comparison function it uses? It feels like a ,cmp=cmp argument to the added search index consructor, then a self._cmp = cmp in the constructor, followed by a query and replace on cmp with self._cmp in the search index source. But I might be completely wrong :-( Apart from that, any ideas how I'd search for a record where the indexed attribute is blank or the secondary sort stuff? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog stuff
On Thu, 22 Feb 2001, Chris Withers wrote: > I think this is _exactly_ what I've been looking for for ages, > especially, I'm guessing, that if you don't specify 'used' > then: No, 'used' has *no* effect on the processing of the search; it exists only to allow keeping track of the indices used in searches across catalogs. > Okay while I'm here, one final one. > > We have the sort_on and sort_order parameters, they'll return > stuff ordered like: > > Aardvark > Badger > anteater > bird > > Whereas most humans would think the following would be more > readable: > > Aardvark > anteater > Badger > bird Chris, this necrohippophilic sadism has to stop! "most humeans" does *not* map to "English speakers who prefer case-insensitive collation." Case-flattening is a lossy operation; there is *no way* you are going to get it by default. If you want the catalog to do a case-insensitive search / sort, then create a method on your object which does the approprate case-flattening, and index that instead of the "real" attribute. Tres. -- === Tres Seaver[EMAIL PROTECTED] Digital Creations "Zope Dealers" http://www.zope.org ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Lazy Results
On Thu, 22 Feb 2001, Johan Carlsson wrote:
> > Casey Duncan <[EMAIL PROTECTED]> wrote:
> > >
> > > Yes, you always get all the meta-data. However, does
> > > anyone know how the "used" argument of
> > > ZCatalog.searchResults works, or why you would use it?
> >
> > 'used' tells you the names of the indexes which actually
> > participated in the search (as distinct from those you
> > asked for).
>
> What does this meen?
>
> Example, What's the different between:
>
> searchResults(used=(index1,index2), index1='kalle', index2='kaviar')
>
> and;
>
> searchResults(index1='kalle', index2='kaviar')
There is no practical difference, unless you are "federating"
queries across multiple catalogs; the 'used' parameter is there
solely to accumulate a record of the indices used in the search.
> (Are used supposed to be a tuple?)
It should be something supporting the mapping interface. Here is
how it is used (from 'lib/python/Products/ZCatalog/Catalog.py'):
if used is None: used={}
for i in self.indexes.keys():
try:
index = self.indexes[i].__of__(self)
if hasattr(index,'_apply_index'):
r=index._apply_index(args)
if r is not None:
r, u = r
for name in u:
used[name]=1
In other words, "Nothing to see here; move along." :)
Tres.
--
===
Tres Seaver[EMAIL PROTECTED]
Digital Creations "Zope Dealers" http://www.zope.org
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Lazy Results
> Casey Duncan <[EMAIL PROTECTED]> wrote: > > > > Yes, you always get all the meta-data. However, does > > anyone know how the "used" argument of > > ZCatalog.searchResults works, or why you would use it? > > 'used' tells you the names of the indexes which actually > participated in the search (as distinct from those you > asked for). What does this meen? Example, What's the different between: searchResults(used=(index1,index2), index1='kalle', index2='kaviar') and; searchResults(index1='kalle', index2='kaviar') (Are used supposed to be a tuple?) TIA, Johan Carlsson ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Lazy Results
Casey Duncan <[EMAIL PROTECTED]> wrote:
> --- Johan Carlsson <[EMAIL PROTECTED]> wrote:
> > Hi,
> > I need to check an assumption I have made on the
> > ZCatalog Brains and Lazy Results from reading the
> > source.
> >
> > 1. The Brains gets its schema from the ZCatalog and
> > this
> > schema is basicly fixed. E.g. a search result
> > always has
> > the attributes defined by the Meta Data in the
> > ZCatalog?
>
> Yes, you always get all the meta-data. However, does
> anyone know how the "used" argument of
> ZCatalog.searchResults works, or why you would use it?
'used' tells you the names of the indexes which actually
participated in the search (as distinct from those you
asked for).
> > I would rather be able to construct a Brain Schema
> > at search time
> > equivalent to the way theSELECT statement sets up
> > the result attributes
> > in SQL.
> > Example:
> > resultset =
> >
> Catalog.searchResults(query,schema=('id','title','keywords'))
>
> Why is this important? I would suggest not putting big
> stuff in the meta-data that might warrant this. Just
> use the nifty new (v2.3) getObject method of the brain
> to access whatever properties/methods of the actual
> object you might need.
'getObject' forces you to "wake up" the catalogued object;
in a large enough result set, the price is pretty punitive,
compared to serving the results as cached metadata.
Tres.
--
===
Tres Seaver[EMAIL PROTECTED]
Digital Creations "Zope Dealers" http://www.zope.org
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Lazy Results
> How many is "a lot" of meta datas?
At least 20-30.
Johan
> Catalog.searchResults(query,schema=('id','title','keywords'))
> > >
> > > Why is this important? I would suggest not putting
> > big
> > > stuff in the meta-data that might warrant this.
> > Just
> > > use the nifty new (v2.3) getObject method of the
> > brain
> > > to access whatever properties/methods of the
> > actual
> > > object you might need.
> >
> > If I have allot of MetaDatas and do allot of
> > searches,
> > one for each page hit, I assume it would use up
> > allot of
> > memory and do allot of copying of attributes?
> > I want to use ZCatalog for dynamic linking.
> >
> >
> > > > 2. The laziness of the ZCatalog search result
> > means
> > > > it's batched?
> > > Lazy sequences work by not loading the result
> > items in
> > > memory until they are actually accessed.
> >
> > Ok, suppose I should have known that. Thanks.
> > Johan
> >
> >
h¥àÞ¾f¢X¬µ)x7¯Î^¢¸!¶ÚþX¬¶Ìè¥ê+ùYùb²Ø§~ó¢zóhrº,²,¶Ê+ 3 zw(v)àEéZµçeËlÛiÿùb²Û3¢¨®æj)fjåËbú?Î^jyèºw Ûiÿùb²Û3¢¨®æj)fjåËbú?Î^
Re: [Zope-dev] ZCatalog Lazy Results
> >
> > I would rather be able to construct a Brain Schema
> > at search time
> > equivalent to the way theSELECT statement sets up
> > the result attributes
> > in SQL.
> > Example:
> > resultset =
> > Catalog.searchResults(query,schema=('id','title','keywords'))
>
> Why is this important? I would suggest not putting big
> stuff in the meta-data that might warrant this. Just
> use the nifty new (v2.3) getObject method of the brain
> to access whatever properties/methods of the actual
> object you might need.
If I have allot of MetaDatas and do allot of searches,
one for each page hit, I assume it would use up allot of
memory and do allot of copying of attributes?
I want to use ZCatalog for dynamic linking.
> > 2. The laziness of the ZCatalog search result means
> > it's batched?
> Lazy sequences work by not loading the result items in
> memory until they are actually accessed.
Ok, suppose I should have known that. Thanks.
Johan
¢zùYb²Öh¥àÞ¿:)zàÛiÿùb²Û3¢¨®æj)fjåËbú?Î^uëÍ¡Êè²Êh²Û(¬tÌ-éܡا¥j×+-²m§ÿåËlÎ^¢¸?¨¥©ÿ+-wèÿ:)y©ç¢éÜzm§ÿåËlÎ^¢¸?¨¥©ÿ+-wèÿ:)
Re: [Zope-dev] ZCatalog Lazy Results
--- Johan Carlsson <[EMAIL PROTECTED]> wrote:
> Hi,
> I need to check an assumption I have made on the
> ZCatalog Brains and Lazy Results from reading the
> source.
>
> 1. The Brains gets its schema from the ZCatalog and
> this
> schema is basicly fixed. E.g. a search result
> always has
> the attributes defined by the Meta Data in the
> ZCatalog?
Yes, you always get all the meta-data. However, does
anyone know how the "used" argument of
ZCatalog.searchResults works, or why you would use it?
>
> I would rather be able to construct a Brain Schema
> at search time
> equivalent to the way theSELECT statement sets up
> the result attributes
> in SQL.
> Example:
> resultset =
>
Catalog.searchResults(query,schema=('id','title','keywords'))
Why is this important? I would suggest not putting big
stuff in the meta-data that might warrant this. Just
use the nifty new (v2.3) getObject method of the brain
to access whatever properties/methods of the actual
object you might need.
>
> 2. The laziness of the ZCatalog search result means
> it's batched?
>
Lazy sequences work by not loading the result items in
memory until they are actually accessed.
> Regards,
> Johan Carlsson
=
| Casey Duncan
| Kaivo, Inc.
| [EMAIL PROTECTED]
`->
__
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail - only $35
a year! http://personal.mail.yahoo.com/
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Erik Enge wrote: > > [Michael R. Bernstein] > > | What I'm looking for is any indication that object creation time > | and/or indexing time goes up with the number of objects already in > | the ZODB. > > Well, one thing I've already learned - which you all probably know - > is that you do _not_ want to put index_object() in your class' > __init__() method. That's because the CatalogAwarness class you > subclass does this for you in the manage_afterAdd() method. For my 'archive' applications, I'm using a SkinScript to index the objects as they're added instead of subclassing from CatalogAware. > | Will you be populating the ZODB in batches (say 100,000 objects or > | so)? > > I'll do that as a secondary solution, if doing it in one batch is too > ineffective. I should mention that even spliting this up into three batches of 9 million records would *probably* give me the indication I'm looking for, as to whether there was any progressive performance degradation with the number of objects. Thanks again, Erik, and good luck! Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
[Michael R. Bernstein] | What I'm looking for is any indication that object creation time | and/or indexing time goes up with the number of objects already in | the ZODB. Well, one thing I've already learned - which you all probably know - is that you do _not_ want to put index_object() in your class' __init__() method. That's because the CatalogAwarness class you subclass does this for you in the manage_afterAdd() method. If you put index_object() in __init__() you might as well go on holliday before it's finished. A long one. | Will you be populating the ZODB in batches (say 100,000 objects or | so)? I'll do that as a secondary solution, if doing it in one batch is too ineffective. | If so, can you benchmark each batch, so we can see if the batch | proccessing time goes up as you progress through the 270 batches? Yes. ¡No problemo, senor! :) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Erik Enge wrote: > > [Michael R. Bernstein] > > | Erik Enge wrote: > | > > | > I'll let you know how it goes. (And please, do poke at me if it takes > | > too long.) > | > | Ok, I'm poking :-). > > Thanks. Keep doing it till you get what you need, I truly don't > mind. :-) > > | How did it go? > > Thanks to the speed of delivery at Royal Mail in the UK, I haven't > recieved the data yet (!). They promised to have it too me by the > coming friday (the 23rd, I believe). I'll try to process all 27 > million records and give feeback to the community. > > (I did a premature calculation of how long it would take to populate > the Zope instance, and I'm guessing somewhere between 80 to 100 hours, > meaning the feedback cannot come any sooner than Sunday, at best.) What I'm looking for is any indication that object creation time and/or indexing time goes up with the number of objects already in the ZODB. Will you be populating the ZODB in batches (say 100,000 objects or so)? If so, can you benchmark each batch, so we can see if the batch proccessing time goes up as you progress through the 270 batches? Thanks, Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
[Michael R. Bernstein] | Erik Enge wrote: | > | > I'll let you know how it goes. (And please, do poke at me if it takes | > too long.) | | Ok, I'm poking :-). Thanks. Keep doing it till you get what you need, I truly don't mind. :-) | How did it go? Thanks to the speed of delivery at Royal Mail in the UK, I haven't recieved the data yet (!). They promised to have it too me by the coming friday (the 23rd, I believe). I'll try to process all 27 million records and give feeback to the community. (I did a premature calculation of how long it would take to populate the Zope instance, and I'm guessing somewhere between 80 to 100 hours, meaning the feedback cannot come any sooner than Sunday, at best.) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Erik Enge wrote: > > [Michael Bernstein] > > | I need to know how far the ZCatalog will scale using this indexing > | and search strategy. Does anyone have anectodal or benchmark data to > | suggest if (and when) I will hit a 'wall' regarding the number of > | objects being indexed and searched? > > I'm going to try to stuff 27 million objects into ZODB sometime in the > next week or the week after that (all post addresses in England). I > haven't got a clue as to whether this will work or just... well not > work. I haven't come up with a strategy for segmenting the data, but > that shouldn't be a problem at all. This isn't actually much data, so > I don't expect the Data.fs file to more than 500 MB. > > I'm quite confident that ZODB, ZCatalog and BTree will scale very > nicely for this. I have a plan ;). > > I'll let you know how it goes. (And please, do poke at me if it takes > too long.) Ok, I'm poking :-). How did it go? Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog/TextIndex/OR - keyword search
Arno Gross wrote: > Steve, thanks for your hints. > At moment I'm working with version 2.2.3 and could't bring it to work. Should > it work? PythonScripts won't work. You should be able to do it with an External Method. I don't know about PythonMethods. > So I switched now tho 2.3 and after some struggles it works. Great! > Actually I have ZClass objects. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog/TextIndex/OR - keyword search
Steve, thanks for your hints. At moment I'm working with version 2.2.3 and could't bring it to work. Should it work? So I switched now tho 2.3 and after some struggles it works. Actually I have ZClass objects. Best regards Arno Gross, [EMAIL PROTECTED] On Thu, 08 Feb 2001, Steve Alexander wrote: > Arno Gross wrote: > > > Hi Steve, > > maybe I'm missing one step. If I create a TextIndex for search_text (or > > PrincipiaSearchSource) > > I have to tell somewhere how this index is composite? Becaus I try to > > list PrincipiaSearchSource in my result list it's empty. > > You need to provide a method or property or attribute that gives the catalog the >concatenated search text. > > You haven't said what kind of objects you are cataloging: are they ZClass instances, >Python class instances, DTML Documents... ? > > You may be able to use a PythonScript in the acquisition path of your objects to do >this work. > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog/TextIndex/OR - keyword search
Arno Gross wrote: > Hi Steve, > maybe I'm missing one step. If I create a TextIndex for search_text (or > PrincipiaSearchSource) > I have to tell somewhere how this index is composite? Becaus I try to > list PrincipiaSearchSource in my result list it's empty. You need to provide a method or property or attribute that gives the catalog the concatenated search text. You haven't said what kind of objects you are cataloging: are they ZClass instances, Python class instances, DTML Documents... ? You may be able to use a PythonScript in the acquisition path of your objects to do this work. > On Thu, 08 Feb 2001, Steve Alexander wrote: > >> Arno Gross wrote: >> >> >>> I understand that the search terms for a field with TextIndex are ORed >>> and the fields in a search form are ANDed. >>> But assume the following: >>> * I have three fields (eg. abstract, content, title) with TextIndex >>> * I have one field (eg. category) with FieldIndex >>> >>> When searching I want just give a keyword for the three TextIndex fields >>> and a search term for category. >>> The applying expression should look like this: >>> >>> (abstract=keyword OR content=keyword OR title=keyword) AND >>> category=searchTerm >>> >>> Any hints? >> >> Create a method or computed attribute called "search_text" that returns the >> concatenation of abstract, content and title. >> >> In ZPatterns I'd use the following SkinScript: >> >> WITH SELF COMPUTE >> search_text='%s %s %s' % (abstract, content, title) >> >> An equivalent PythonScript as a method of your ZClass would be: >> >> name: search_text >> return '%s %s %s' % (container.abstract, container.content, container.title) >> >> Use a similar method in a python class, replacing "container" with "self". >> >> Set up search_text as a TextIndex in your ZCatalog. >> >> Then, query the catalog with something like: >> >> Catalog(search_text=keyword_for_query, category=searchTerm) >> >> The standard name for the search_text method or attribute is >> PrincipiaSearchSource. That's why PrincipiaSearchSource is a default >> TextIndex in a new ZCatalog. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog/TextIndex/OR - keyword search
Hi Steve, maybe I'm missing one step. If I create a TextIndex for search_text (or PrincipiaSearchSource) I have to tell somewhere how this index is composite? Becaus I try to list PrincipiaSearchSource in my result list it's empty. Thanks Arno Gross, email: [EMAIL PROTECTED] On Thu, 08 Feb 2001, Steve Alexander wrote: > Arno Gross wrote: > > > I understand that the search terms for a field with TextIndex are ORed > > and the fields in a search form are ANDed. > > But assume the following: > > * I have three fields (eg. abstract, content, title) with TextIndex > > * I have one field (eg. category) with FieldIndex > > > > When searching I want just give a keyword for the three TextIndex fields > > and a search term for category. > > The applying expression should look like this: > > > > (abstract=keyword OR content=keyword OR title=keyword) AND > > category=searchTerm > > > > Any hints? > > Create a method or computed attribute called "search_text" that returns the > concatenation of abstract, content and title. > > In ZPatterns I'd use the following SkinScript: > > WITH SELF COMPUTE > search_text='%s %s %s' % (abstract, content, title) > > An equivalent PythonScript as a method of your ZClass would be: > > name: search_text > return '%s %s %s' % (container.abstract, container.content, container.title) > > Use a similar method in a python class, replacing "container" with "self". > > Set up search_text as a TextIndex in your ZCatalog. > > Then, query the catalog with something like: > > Catalog(search_text=keyword_for_query, category=searchTerm) > > The standard name for the search_text method or attribute is > PrincipiaSearchSource. That's why PrincipiaSearchSource is a default > TextIndex in a new ZCatalog. > > -- > Steve Alexander > Software Engineer > Cat-Box limited > http://www.cat-box.net > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog/TextIndex/OR - keyword search
Arno Gross wrote: > I understand that the search terms for a field with TextIndex are ORed > and the fields in a search form are ANDed. > But assume the following: > * I have three fields (eg. abstract, content, title) with TextIndex > * I have one field (eg. category) with FieldIndex > > When searching I want just give a keyword for the three TextIndex fields > and a search term for category. > The applying expression should look like this: > > (abstract=keyword OR content=keyword OR title=keyword) AND > category=searchTerm > > Any hints? Create a method or computed attribute called "search_text" that returns the concatenation of abstract, content and title. In ZPatterns I'd use the following SkinScript: WITH SELF COMPUTE search_text='%s %s %s' % (abstract, content, title) An equivalent PythonScript as a method of your ZClass would be: name: search_text return '%s %s %s' % (container.abstract, container.content, container.title) Use a similar method in a python class, replacing "container" with "self". Set up search_text as a TextIndex in your ZCatalog. Then, query the catalog with something like: Catalog(search_text=keyword_for_query, category=searchTerm) The standard name for the search_text method or attribute is PrincipiaSearchSource. That's why PrincipiaSearchSource is a default TextIndex in a new ZCatalog. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Bug Fix II
> Chris Withers wrote:
> >
> > Okay, tracked down what's happening, can't understand why :-(
Now I can... convoluted but with a simple answer ;-)
> As to why the object is on the top of the namespace, I dunno whether it's
> dtml-in or ZCatalog that's responsible...
The reason for this, was because the object was being acquired. The naming
similarity was pure chance. I had an index called TestCase in the catalog,
the same name as the DTML method for the form. I _didn't_ have a column
called TestCase in the ZCatalog MetaData, so when the result-displaying DTML
method tried to dtml-var it, it was acquired, and hence the method was
plopped on top of the namespace.
The actual problem is that ZCatalog was using the indexes instead of the
metadata when telling Z Search was results form to build. The following
patch solves that, I'm sure I put it in the collector when I moved Squishdot
to use ZCatalog too :-S
cheers,
Chris
--- zcatalog.py.old2 Sun Jan 28 20:56:06 2001
+++ zcatalog.py Sun Jan 28 21:00:09 2001
@@ -486,7 +486,7 @@
def _searchable_result_columns(self):
r = []
-for name in self._catalog.indexes.keys():
+for name in self._catalog.schema.keys():
i = {}
i['name'] = name
i['type'] = 's'
PS: I won't bother with the collector on this one, since both relevent
people are already being CC'ed ;-)
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3
> > I'm guessing the DTML method TestCase is acquired from it, before the > > namespace > > is searched for the meta_data, which is a little screwy :-( > > Odd, I thought you were reffering to another bug that I fixed, thanks to > your report :-) hehehe, apparantly that wasn't the actual bug you were > interested in Should I put the other one back? :-) Other one? not sure what you mean there... The bug with FieldIndexes was the most important, one of our VIP(roject)s crucially relies on this ;-) The ZSearch problem is weird. I'd need to fix ZDebug to work with 2.3 first though as, AFAIK, a normal TemplateDict won't tell you what mappings/InstanceDicts it has in it and what order they're in :-S Thanks for the quick fix though :-) Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3
On 1/26/01 11:56 AM, "Chris Withers" <[EMAIL PROTECTED]> wrote: > Chris Withers wrote: >> >> Okay, tracked down what's happening, can't understand why :-( > > Some data points before I rush off... > > - ZDebug doesn't like 2.3 very much :-S > - Python Scripts rock ;-) > > However, from what ZDebug did show, it seems that the object the catalog has > found ends up on the top of the namespace stack. Because this is a DTML > method, > I'm guessing the DTML method TestCase is acquired from it, before the > namespace > is searched for the meta_data, which is a little screwy :-( > > As to why the object is on the top of the namespace, I dunno whether it's > dtml-in or ZCatalog that's responsible... Odd, I thought you were reffering to another bug that I fixed, thanks to your report :-) hehehe, apparantly that wasn't the actual bug you were interested in Should I put the other one back? :-) Chris -- | Christopher Petrilli | [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3
On 1/26/01 11:41 AM, "Chris McDonough" <[EMAIL PROTECTED]> wrote: > I've put the fieldindex bug in the collector. This is now fixed... It was a bizarre logic problem, but I could reproduce it easily :-) Thanks for finding this. Chris -- | Christopher Petrilli | [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3
Chris Withers wrote: > > Okay, tracked down what's happening, can't understand why :-( Some data points before I rush off... - ZDebug doesn't like 2.3 very much :-S - Python Scripts rock ;-) However, from what ZDebug did show, it seems that the object the catalog has found ends up on the top of the namespace stack. Because this is a DTML method, I'm guessing the DTML method TestCase is acquired from it, before the namespace is searched for the meta_data, which is a little screwy :-( As to why the object is on the top of the namespace, I dunno whether it's dtml-in or ZCatalog that's responsible... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3
This is fixed in CVS. ChrisP works fast. :-) - Original Message - From: "Chris Withers" <[EMAIL PROTECTED]> To: "Chris McDonough" <[EMAIL PROTECTED]> Sent: Friday, January 26, 2001 11:23 AM Subject: Re: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3 > Chris McDonough wrote: > > > > I've put the fieldindex bug in the collector. > > Thanks... wasn't sure whether it was collector material or not :-) > > Chris > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3
Brian Lloyd wrote: > > > PS: Tabular reporting ZSearch Interfaces generated from Catalogs > > have quite > > badly b0rKen results forms, in that, for some reason, the search > > form ends up in > > a column of the results table, very confusing... > > Huh? I cannot reproduce this at all (or even see how it could > happen, looking at the tabular report dtml, which is pretty > straightforward). Okay, tracked down what's happening, can't understand why :-( Right, one of the metadata columns has the same name as the DTML method which shows the search form 'TestCase' in this situation. For some reason, the DTML method is being rendered instead of the meta data. I have no idea why, but I'll take a look... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3
I've put the fieldindex bug in the collector. - Original Message - From: "Chris Withers" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Friday, January 26, 2001 10:38 AM Subject: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3 > Hi, > > FieldIndex's don't work quite right in 2.3. > > Try this: > - create a Catalog with a FieldIndex called 'test' > - create a DTML document with a property 'test', value 'fish' > - catalog the document > - now change the value of the property to 'finger' and re-catalog. > > If you now search for test='finger' it'll return the document. > However, if you search for test='fish', it will also return the document, which > is wrong. > > cheers, > > Chris > > PS: Tabular reporting ZSearch Interfaces generated from Catalogs have quite > badly b0rKen results forms, in that, for some reason, the search form ends up in > a column of the results table, very confusing... > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ZCatalog FieldIndex bug in Zope 2.3.0b3
> PS: Tabular reporting ZSearch Interfaces generated from Catalogs > have quite > badly b0rKen results forms, in that, for some reason, the search > form ends up in > a column of the results table, very confusing... Huh? I cannot reproduce this at all (or even see how it could happen, looking at the tabular report dtml, which is pretty straightforward). You are seeing this on b3 with a totally freshly generated ZSearch interface that you have not altered (and looking at the generated dtml directly, not through a site-specific page that includes it)? Brian Lloyd[EMAIL PROTECTED] Software Engineer 540.371.6909 Digital Creations http://www.digicool.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
[Michael Bernstein] | Erik Enge wrote: | > | > I don't believe that B-Tree folders have those kinds of limitations by | > general design. I'm more conserned that somewhere along the lines, | > doing operations on a huge BTree Folder (Yes, in Zope) will be slow. | | What sort of 'operations' do you mean? copying and pasting | the whole thing? My point was that anything will be slow at some stage. This is valid for BTree Folders as well, and doing opertions on the objects (copying, deleting, modifying), will be slower than adding them, I guess. I'm not quite sure here Michael, someone else has probably more experience and knowledge to answer more correctly. | I had separated the storage issue into a different thread Oops, I forgot that, sorry. | Anyway Eric, I hope that when you report your results, you're able | to separate indexing, searching, storage, and retreival results, so | that the appropriate factor can be identified as the bottleneck. Or | at least into indexing/searching and storage/retreival. Yes, I hope I'll manage to do that. And till then, I guess we just have to wait to see how things work in practice, which might be different to the theory. ;) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Erik Enge wrote: > > [Chris Withers] > > | ...and is that specifically for BTree folders, or Zope BTree's in general? > > I don't believe that B-Tree folders have those kinds of limitations by > general design. I'm more conserned that somewhere along the lines, > doing operations on a huge BTree Folder (Yes, in Zope) will be slow. What sort of 'operations' do you mean? copying and pasting the whole thing? > Hm, more over, if you actually need to stuff that many objects into > one Folder, you are probably trying to use the wrong tool for the job. > > I do expect that stuffing 27 million objects into one BTree Folder > will be slow, and I don't want to segment the data. I do expect that > I'll have to resort to a relational database, and I have no problem > with that. Object databases aren't always the right tool for the job, > and when they aren't, Zope let's me talk with the «other» ones nicely, > so no problemo señor ;). Eric, I had separated the storage issue into a different thread (Specialist/Rack Scalability), and received a reply from Phillip Eby: > Just to expand a little on the abov... Racks should scale at least as > well, if not larger than a ZCatalog, given the same storage backing for > the ZODB. This is because ZCatalog has to manage a minimum of one > forward and reverse BTree for *each* index, plus another few BTrees > for overall storage and housekeeping. Also, keyword and full text > indexes store multiple BTree entries per object, so that's a factor as > well. So the question I was asking is: "if we ignore the issue of storage and consider indexing and searching the ZCatalog alone, and assuming that wildcard searches are disallowed, how far will a single ZCatalog with a text index (on a computed attribute that concatenates several properties) and a keyword index (for creating ZTopic heirarchies) scale?" While I'm perfectly willing to split up the storage of the data as neccessary, I am far less enamoured by the prospect of divvying up the indexing and searching to multiple ZCatalogs. In any case, according to Phillip, if I don't have to split the ZCatalog, I shouldn't have to split the storage (in Racks, anyway, but probably BTree Folders too), either. Anyway Eric, I hope that when you report your results, you're able to separate indexing, searching, storage, and retreival results, so that the appropriate factor can be identified as the bottleneck. Or at least into indexing/searching and storage/retreival. Thanks, Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
[Chris Withers] | ...and is that specifically for BTree folders, or Zope BTree's in general? I don't believe that B-Tree folders have those kinds of limitations by general design. I'm more conserned that somewhere along the lines, doing operations on a huge BTree Folder (Yes, in Zope) will be slow. However, this is more gut-feeling than anything else. Hm, more over, if you actually need to stuff that many objects into one Folder, you are probably trying to use the wrong tool for the job. I do expect that stuffing 27 million objects into one BTree Folder will be slow, and I don't want to segment the data. I do expect that I'll have to resort to a relational database, and I have no problem with that. Object databases aren't always the right tool for the job, and when they aren't, Zope let's me talk with the «other» ones nicely, so no problemo señor ;). ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Erik Enge wrote: > > I don't know. But there is one on BTree folders, right? Really? Anyone got a figure for that? ...and is that specifically for BTree folders, or Zope BTree's in general? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
[Michael Bernstein] | We seem to have disposed of the wildcard issue [snipped out | below], and I'm looking forward to Eric's results, but does | anyone else have any information about whether there is a | practical upper limit on how many objects can be indexed and | searched in a ZCatalog? I don't know. But there is one on BTree folders, right? And as soon as those get «full», you need to start segment your data (which you probably would have done in the first place anyway). Then you can use several ZCatalogs in different locations (so that they don't carry so many objects each). Then create a nice little method that finds out which (or all) of the ZCatalogs to ask, when users make queries. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
We seem to have disposed of the wildcard issue [snipped out below], and I'm looking forward to Eric's results, but does anyone else have any information about whether there is a practical upper limit on how many objects can be indexed and searched in a ZCatalog? Michael Bernstein wrote: > > After comsidering the feedback I got from the previous > 'Massive scalability' thread, I decided to split my queries > into two areas: Rack scalability and ZCatalog scalability. > This email deals with the latter. > > [snip] > > What I am interested in for my application are two things: > > - ZTopics populated using one or more keyword indexes > > - Full text search on a single computed attribute that > concatenates several fields including the aforementioned > keyword index fields and a few simple string attributes > (title, caption, description, etc.) > > I need to know how far the ZCatalog will scale using this > indexing and search strategy. Does anyone have anectodal or > benchmark data to suggest if (and when) I will hit a 'wall' > regarding the number of objects being indexed and searched? > Some anectodal data suggests that single field indexing will > scale easily to 60,000 objects, but what about hundreds of > thousands or even millions of objects? > > [snip] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Steve Alexander writes: > Michael Bernstein wrote: > > > > Also, is there a way to disable wildcards in full text > > searches? > > Do not allow direct queries to search the catalog. Instead, make > searches go through an external method (or a PythonScript with Proxy > permissions) that uses string.replace to change '*' and '?' to ''. Simply do not use a globbing vocabulary is another alternative. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Chris Withers wrote: > > Wouldn't using a normal vocabulary as opposed to a globbing vocabulary > prevent this as well? That would stop globbing searches for everyone. While I might want to stop users of a site making wildcard searches, I still want to keep that facility for myself :-) -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
> Michael Bernstein wrote: > > > > Also, is there a way to disable wildcards in full text > > searches? > > Do not allow direct queries to search the catalog. Instead, make > searches go through an external method (or a PythonScript with Proxy > permissions) that uses string.replace to change '*' and '?' to ''. Wouldn't using a normal vocabulary as opposed to a globbing vocabulary prevent this as well? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
[Michael Bernstein] | I need to know how far the ZCatalog will scale using this indexing | and search strategy. Does anyone have anectodal or benchmark data to | suggest if (and when) I will hit a 'wall' regarding the number of | objects being indexed and searched? I'm going to try to stuff 27 million objects into ZODB sometime in the next week or the week after that (all post addresses in England). I haven't got a clue as to whether this will work or just... well not work. I haven't come up with a strategy for segmenting the data, but that shouldn't be a problem at all. This isn't actually much data, so I don't expect the Data.fs file to more than 500 MB. I'm quite confident that ZODB, ZCatalog and BTree will scale very nicely for this. I have a plan ;). I'll let you know how it goes. (And please, do poke at me if it takes too long.) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Erik Enge wrote: > > [Michael Bernstein] > > | I need to know how far the ZCatalog will scale using this indexing > | and search strategy. Does anyone have anectodal or benchmark data to > | suggest if (and when) I will hit a 'wall' regarding the number of > | objects being indexed and searched? > > I'm going to try to stuff 27 million objects into ZODB sometime in the > next week or the week after that (all post addresses in England). I > haven't got a clue as to whether this will work or just... well not > work. I haven't come up with a strategy for segmenting the data, but > that shouldn't be a problem at all. This isn't actually much data, so > I don't expect the Data.fs file to more than 500 MB. > > I'm quite confident that ZODB, ZCatalog and BTree will scale very > nicely for this. I have a plan ;). > > I'll let you know how it goes. (And please, do poke at me if it takes > too long. Will do, Thanks! Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Steve Alexander wrote: > > Michael Bernstein wrote: > > > Also, is there a way to disable wildcards in full text > > searches? > > Do not allow direct queries to search the catalog. Instead, make > searches go through an external method (or a PythonScript with Proxy > permissions) that uses string.replace to change '*' and '?' to ''. A *very* handy suggestion. You might want to add that as a Tip to Zope.org. Thanks, Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog scalability
Michael Bernstein wrote: > Also, is there a way to disable wildcards in full text > searches? Do not allow direct queries to search the catalog. Instead, make searches go through an external method (or a PythonScript with Proxy permissions) that uses string.replace to change '*' and '?' to ''. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog doesnt find Items
There could be problems depending on how a product handles dealing with attributes, but the exception should be caught. Note that the traceback is *not* in Catalog, but is in the Transaction manager. Can you talk about what other products you have installed? And hwat your object database looks like? Chris -- | Christopher Petrilli | [EMAIL PROTECTED] - Original Message - From: "Andre Schubert" <[EMAIL PROTECTED]> To: "Zope Development Maillist" <[EMAIL PROTECTED]> Sent: Friday, January 19, 2001 5:30 AM Subject: [Zope-dev] ZCatalog doesnt find Items > Hi all, > > I have a problem with my ZCatalog object. > If i have a Folder named Test and inside i create a ZCatalog object and > some DTML-Objects/Methods the ZCatalog finds all Items. > But when i install the ZCatalog in the root folder and start find Items > a got the following error: > > Traceback (innermost last): > File > /usr/share/zope_2.2/Zope-2.2.0-src/lib/python/ZPublisher/Publish.py, > line 222, in publish_module > File > /usr/share/zope_2.2/Zope-2.2.0-src/lib/python/ZPublisher/Publish.py, > line 187, in publish > File /usr/share/zope_2.2/Zope-2.2.0-src/lib/python/Zope/__init__.py, > line 221, in zpublisher_exception_hook > (Object: Traversable) > File > /usr/share/zope_2.2/Zope-2.2.0-src/lib/python/ZPublisher/Publish.py, > line 175, in publish > File /usr/share/zope_2.2/Zope-2.2.0-src/lib/python/Zope/__init__.py, > line 235, in commit > File > /usr/share/zope_2.2/Zope-2.2.0-src/lib/python/ZODB/Transaction.py, line > 290, in commit > AttributeError: (see above) > > Are there some known conflicts with other Products?? > > I use Zope 2.2.2 and python 1.5.2 on Linux(i386) > > Who can help me > > as > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] ZCatalog with ZClass
Hi, I tried it with a new product following the HowTo, and it works perfectly. Thanks a lot. But actually I have already an exsisted product with a Container ZClass 'KTracker'(inherit from ObjectManager) and an Item ZClass called 'TrackerIssue'. And it's been used for a while with the instance 'tracker' and hundreds of issues inside. Now I've succeeded in changing the base class for 'KTracker'to subclass ObjectManager and ZCatalog, and changing 'TrackerIssue' to CatalogAware. As 'tracker' instance was created before, although the base class is changed now, the constructor of the new base class seems not running. So I created an External Method to run the constructor of ZCatalog as follows: from Products.ZCatalog.ZCatalog import ZCatalog def initContainer(self): ZCatalog.__init__(self, self.id) Then I ran this method through IE browser: http://hostname/.../tracker/initContainer. It seems working, as a Vocabulary folder is created. And I could make settings to index and so on. But there's one thing incorrect: When using manage_catalogFind to catalog the object of type 'TrackerIssue', nothing was returned. If changing the finding type to 'All types', all the 'TrackerIssue' objects and other types of objects are cataloged together. The auto-catalogAware works fine for 'TrackerIssue' objects though. Do you have any idea why this happens? Does it have anything to do with the order of base class, etc.? If just the manage_catalogFind doesn't work well, will it affect the searching stuff much, if CatalogAware works fine? Thanks a lot. Lily -Original Message- From: Michael Bernstein [mailto:[EMAIL PROTECTED]] Sent: Wednesday, January 17, 2001 12:11 AM To: Lily Li Cc: '[EMAIL PROTECTED]' Subject: Re: [Zope-dev] ZCatalog with ZClass Lily Li wrote: > > Hello, everyone, > > It seems that many of you have used ZCatalog pretty well. > > My problem is that I'd like a ZClass (of a Product) to subclass > CatalogAware, and implement all the index and search inside the Product. > That means I don't want to do any DTML method stuff outside of the Product. What you probably want is to create a repository that subclasses ZCatalog as is described in this HowTo: http://www.zope.org/Members/tseaver/inherit_ZCatalog And then add your CatalogAware ZClasses into it. Then you can set up your ZCatalog/ObjectManager with search methods. HTH, Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog with ZClass
Lily Li wrote: > > Hello, everyone, > > It seems that many of you have used ZCatalog pretty well. > > My problem is that I'd like a ZClass (of a Product) to subclass > CatalogAware, and implement all the index and search inside the Product. > That means I don't want to do any DTML method stuff outside of the Product. What you probably want is to create a repository that subclasses ZCatalog as is described in this HowTo: http://www.zope.org/Members/tseaver/inherit_ZCatalog And then add your CatalogAware ZClasses into it. Then you can set up your ZCatalog/ObjectManager with search methods. HTH, Michael Bernstein. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and 'fuzzy logic'
Morten W. Petersen writes: > It seems I misunderstood the term fuzzy logic myself. Fuzzy logic means > if I search for a word, for example 'programmer', it will return matches > to the words 'program', 'programming','programmable' etc. This, usually, is called "stemming". Though, your examples indicate quite a strong form of it. If you have some tool, maybe LinguistX, that map from a word to its stem and then from the stem to all words with this as stem (or directly give the stem equivalence class of a word), then it is quite easy to incorporate that in Zope's catalog. However, to do that cleanly, you will need good algorithms and/or large dictionaries. This, usually, is not free of charge. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and 'fuzzy logic'
--- "Morten W. Petersen" <[EMAIL PROTECTED]> wrote: [snip] > > It seems I misunderstood the term fuzzy logic > myself. Fuzzy logic means > if I search for a word, for example 'programmer', it > will return matches > to the words 'program', 'programming','programmable' > etc. > > I.e., it will somewhat intelligently return words > that are similar in > what they mean, using grammar rules (chopping off > endings of words and > making them match others). > > Hmm. > > Cheers, > > Morten > ZCatalog TextIndexes support this type of "wildcard" searching. I posted a message a couple of weeks ago that describes the query syntax. Search the mailing list archives for it. = | Casey Duncan | Kaivo, Inc. | [EMAIL PROTECTED] `-> __ Do You Yahoo!? Yahoo! Photos - Share your holiday photos online! http://photos.yahoo.com/ ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and 'fuzzy logic'
On Wed, 10 Jan 2001, Morten W. Petersen wrote: > > I do not think that "fuzzy logic" is strongly related to "regexp-like". > > Anyway. > > > > Fuzzy searching often means "finding matches with characters omitted, > > replaced or inserted". > > It seems I misunderstood the term fuzzy logic myself. Fuzzy logic means > if I search for a word, for example 'programmer', it will return matches > to the words 'program', 'programming','programmable' etc. I think your talking about something else. Last i checked, "fuzzy logic" was a logical algebra based on the existence of intermediate truth states, between "true" and "false". It has little or nothing to do with aproximate searching, though i guess you could use it to make assertions about the aproximations. I think what you all are talking about is "fuzzy matching". > I.e., it will somewhat intelligently return words that are similar in > what they mean, using grammar rules (chopping off endings of words and > making them match others). There are also matching mechanisms like soundex, that account for misspelling by translating words to phonetic-equivalent normalized codes, and comparing on that basis. Ken [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and 'fuzzy logic'
> I do not think that "fuzzy logic" is strongly related to "regexp-like". > Anyway. > > Fuzzy searching often means "finding matches with characters omitted, > replaced or inserted". It seems I misunderstood the term fuzzy logic myself. Fuzzy logic means if I search for a word, for example 'programmer', it will return matches to the words 'program', 'programming','programmable' etc. I.e., it will somewhat intelligently return words that are similar in what they mean, using grammar rules (chopping off endings of words and making them match others). Hmm. Cheers, Morten ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog and 'fuzzy logic'
Morten W. Petersen writes: > Is there anyone who could try to give an estimate of how long it would > take to add fuzzy logic (regexp-like) searching capability to the > ZCatalog? I do not think that "fuzzy logic" is strongly related to "regexp-like". Anyway. Fuzzy searching often means "finding matches with characters omitted, replaced or inserted". Zope's globbing vocabularies support wildcards '*' and '?'. To implement wildcard based searches efficiently, they index words under their two letter consitutents. When you now get a pattern, you derive from the pattern what two letter constituents the matching words must have and retrieve them. This defines a candidate word set. Then you check, whether the retrieved word really match the expression. You can extend this algorithm to get fuzzy searches. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
