Re: [Zope] ZCatalog queries performance

[Sebastian Tänzer]

Thanks, this answered all of my questions :-)

Best, Sebastian 

> Am 19.05.2015 um 12:50 schrieb Tres Seaver :
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
>> On 05/19/2015 04:54 AM, Sebastian Tänzer wrote:
>> 
>> Let’s say I want to display an accordion with all events of a month 
>> displaying quite a lot of data about the event (30-50 events on one 
>> page, collapsible jQuery accordion). At the moment I get the results
>> and put the objects into the result set via unrestrictedTraverse
>> using getPath(). Would it be better to only get the path and get the
>> object in the template? Does this make any difference performance
>> wise?
> 
> Nope.  Doing it in trusted code (an actual method, and ExternalMethod, or
> a view) would be faster than either a script or a template.  Measure it
> to decide whether to switch, however.
> 
>> Which data of the event objects would I keep as ZCatalog meta columns 
>> and which not? Documentation is not really helpful on this.
> 
> It is super application-dependent.  The costs for maintaining "extra"
> columns of metaata are relatively small (bloating the Record object, and
> therefore its pickle, mostly).
> 
>> The tricky part is that events are linked with different locations 
>> (event location, selling points) that are not children of the event 
>> object but objects in a different folder. The event object itself
>> only keeps the links to these location objects and I get them when
>> iterating over the result set in the template.
> 
> If you can define a URL-addressible method of the event which returns or
> redirects to those objects, you can get away with hard-wiring that in
> your template, using 'brain.getPath()' as a prefix.
> 
>> Any advice what would be the best way for this performance-wise? Is
>> it okay to get the object itself via traversal? Is there a faster way
>> to get the object data?
> 
> Fetching the object is fast if it is already in the ZODB cache, but slow
> if not, nd may cause other, more "popular" / important objects to be
> evicted.  Where possible, design for catalog queries which don't need to
> fetch the object.
> 
> 
> 
> Tres.
> - -- 
> ===
> Tres Seaver  +1 540-429-0999  tsea...@palladion.com
> Palladion Software   "Excellence by Design"http://palladion.com
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> 
> iQIcBAEBAgAGBQJVWxWLAAoJEPKpaDSJE9HY0aQP/1mC0Ll5//c24XQPcvMwS18Y
> /gZfOe/IeL61CRrI2Sv+RMQ6N6gXpL8oM/KsWU2wphfPVYC96VxiD28ubhmWVk+f
> GFbmN4YchYSNHND2kzcjxyHd1+U6+c/NhkcfgL9GtLZBsSNtMZut2pV7sN7VvjhJ
> 0F1Kwu6HxZ5yvEjZOk0tzDfrhh0I4Y3elFTwJkHXq/Z/3xN+NTVNpYXRAw5AWGJC
> A0Y/ll/cx1+JIKlMbjhMNqG+k61JGD8MdZXIvBwJ7jij1VIGYfYG5/7wZieMjCTL
> /SBMkOnODchSzSk2wV1B0ZgpbgTf+DYuZRzKCrYc7QOktfQhqEBOS5ljIjpWkDcq
> wqsxPBD0uyu/K3+dd4egL3Fcvg5DDfm9//Qo2B8JAkLWO5XeJ4UiyZHLYWZXSpXI
> SdjL2zFnhTZhp9pxIv+V4W/XrzMEDAr3UOFlrikiGd4oxTBffFuxuSr2mvpqshTQ
> TaS8Vgty9SmqMaPuvMZPc7FoPExsC4k4sNO9XgSXKT/5xo3FErLTWVCL1/qVHN3I
> 6wHvSrh8n1Z/m60P6OaUzIHF/97sjbxOO75d0RZCnaTFWt2TizrrsrwaBZ2p6Yyo
> 83l5QxPLDge6EZNb1MMewZ9vW8EamDhkR2pKTsQTEVDj2p4gVhmh+J5pjN6BKj50
> D98HPv4JrWhf965/Pay4
> =Sv38
> -END PGP SIGNATURE-
> 
> ___
> Zope maillist  -  Zope@zope.org
> https://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
> https://mail.zope.org/mailman/listinfo/zope-announce
> https://mail.zope.org/mailman/listinfo/zope-dev )

___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog queries performance

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 05/19/2015 04:54 AM, Sebastian Tänzer wrote:

> Let’s say I want to display an accordion with all events of a month 
> displaying quite a lot of data about the event (30-50 events on one 
> page, collapsible jQuery accordion). At the moment I get the results
> and put the objects into the result set via unrestrictedTraverse
> using getPath(). Would it be better to only get the path and get the
> object in the template? Does this make any difference performance
> wise?

Nope.  Doing it in trusted code (an actual method, and ExternalMethod, or
a view) would be faster than either a script or a template.  Measure it
to decide whether to switch, however.

> Which data of the event objects would I keep as ZCatalog meta columns 
> and which not? Documentation is not really helpful on this.

It is super application-dependent.  The costs for maintaining "extra"
columns of metaata are relatively small (bloating the Record object, and
therefore its pickle, mostly).

> The tricky part is that events are linked with different locations 
> (event location, selling points) that are not children of the event 
> object but objects in a different folder. The event object itself
> only keeps the links to these location objects and I get them when
> iterating over the result set in the template.

If you can define a URL-addressible method of the event which returns or
redirects to those objects, you can get away with hard-wiring that in
your template, using 'brain.getPath()' as a prefix.

> Any advice what would be the best way for this performance-wise? Is
> it okay to get the object itself via traversal? Is there a faster way
> to get the object data?

Fetching the object is fast if it is already in the ZODB cache, but slow
if not, nd may cause other, more "popular" / important objects to be
evicted.  Where possible, design for catalog queries which don't need to
fetch the object.



Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJVWxWLAAoJEPKpaDSJE9HY0aQP/1mC0Ll5//c24XQPcvMwS18Y
/gZfOe/IeL61CRrI2Sv+RMQ6N6gXpL8oM/KsWU2wphfPVYC96VxiD28ubhmWVk+f
GFbmN4YchYSNHND2kzcjxyHd1+U6+c/NhkcfgL9GtLZBsSNtMZut2pV7sN7VvjhJ
0F1Kwu6HxZ5yvEjZOk0tzDfrhh0I4Y3elFTwJkHXq/Z/3xN+NTVNpYXRAw5AWGJC
A0Y/ll/cx1+JIKlMbjhMNqG+k61JGD8MdZXIvBwJ7jij1VIGYfYG5/7wZieMjCTL
/SBMkOnODchSzSk2wV1B0ZgpbgTf+DYuZRzKCrYc7QOktfQhqEBOS5ljIjpWkDcq
wqsxPBD0uyu/K3+dd4egL3Fcvg5DDfm9//Qo2B8JAkLWO5XeJ4UiyZHLYWZXSpXI
SdjL2zFnhTZhp9pxIv+V4W/XrzMEDAr3UOFlrikiGd4oxTBffFuxuSr2mvpqshTQ
TaS8Vgty9SmqMaPuvMZPc7FoPExsC4k4sNO9XgSXKT/5xo3FErLTWVCL1/qVHN3I
6wHvSrh8n1Z/m60P6OaUzIHF/97sjbxOO75d0RZCnaTFWt2TizrrsrwaBZ2p6Yyo
83l5QxPLDge6EZNb1MMewZ9vW8EamDhkR2pKTsQTEVDj2p4gVhmh+J5pjN6BKj50
D98HPv4JrWhf965/Pay4
=Sv38
-END PGP SIGNATURE-

___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog queries performance

[Sebastian Tänzer]

Thanks for the feedback and help, Tres. 

Let’s say I want to display an accordion with all events of a month displaying 
quite a lot of data about the event (30-50 events on one page, collapsible 
jQuery accordion). At the moment I get the results and put the objects into the 
result set via unrestrictedTraverse using getPath(). Would it be better to only 
get the path and get the object in the template? Does this make any difference 
performance wise?

Which data of the event objects would I keep as ZCatalog meta columns and which 
not? Documentation is not really helpful on this.

The tricky part is that events are linked with different locations (event 
location, selling points) that are not children of the event object but objects 
in a different folder. The event object itself only keeps the links to these 
location objects and I get them when iterating over the result set in the 
template.

Any advice what would be the best way for this performance-wise?
Is it okay to get the object itself via traversal? Is there a faster way to get 
the object data?

> Am 17.05.2015 um 19:37 schrieb Tres Seaver :
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> On 05/17/2015 11:09 AM, Sebastian Tänzer wrote:
>> What are the alternatives?
> 
> The most performant implementation would be to generate the URL for the
> image using a fixed suffix (relative to the result object).  E.g., ::
> 
>  
>
>  
>   
>   TITLE
>  
>
>  
> 
> You could achieve that fixed suffix either by mandating a fixed object
> ID for the image, or by adding a method / view to the event which
> redirected to the actual image URL.
> 
> 
> 
> Tres.
> - -- 
> ===
> Tres Seaver  +1 540-429-0999  tsea...@palladion.com
> Palladion Software   "Excellence by Design"http://palladion.com
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.11 (GNU/Linux)
> 
> iQIcBAEBAgAGBQJVWNHiAAoJEPKpaDSJE9HY/nYQALbHx3WvyjEk/6rsgomcWJhN
> va4ZgWEjQtFuSlzHOf30CGKIEwNX+EAYWw6juMznQo/4Yq5b2g4Ur8cA9q1gYgFB
> 2ZwW4/W4wKsaAJxQ6sckinAKhlWGJugXdpcxnbbU3V0ePoUtgPsO0GVIlPoDV+WF
> /qEnWYWTu+7xHf/DV5iiHKTqYM0Oqv1aNTuZd9s8/emHoDuU8RpzVLscZuJEYk+B
> 5mdFFSfNJ9wer79fxQH8yM+8Vz6IkVx4XCCTa9/9QM9qHTVLqvv+AllbtolQ4gms
> ASkisuW7p/XvJgK5bMbUFvEkDZQQDMFAuRKe2lnVT6rAFXiR04fHk6GKKtW3WyEb
> Cdq4GYcKz4ZYdF4l2QV8h+FOZRAf6qWW9CCiRsuihaiYB57Uk8hI/GPk00TXL78z
> rmOWtZ86d7hlCarP8k+T+1b5b+RQUB8danop21ys3SeH+FMiEPxjIuhY0i43n6xQ
> dg6HJ8ry7KihYipaRJrC0tqVFluToWFdLUaM1qRSJHpcKi3TpHKiKXGzNSHzeI1y
> GD7zb3HF+lYqsy74wPWIp/flSmJxJMg5givfHt96/sFvaaNstvjIY6KiKUUW0JMy
> 8b/fF4ufHu2ZGoT8TSMCZRa4KpNMcat9Edo/MsgOeir65a3PnFj9Br8DmpZJwTbV
> 9KZEKY4Aq+EJq+RPmH2D
> =U40Y
> -END PGP SIGNATURE-
> 
> ___
> Zope maillist  -  Zope@zope.org
> https://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
> https://mail.zope.org/mailman/listinfo/zope-announce
> https://mail.zope.org/mailman/listinfo/zope-dev )


___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog queries performance

[Tres Seaver]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 05/17/2015 11:09 AM, Sebastian Tänzer wrote:
> What are the alternatives?

The most performant implementation would be to generate the URL for the
image using a fixed suffix (relative to the result object).  E.g., ::

  

  
   
   TITLE
  

  

You could achieve that fixed suffix either by mandating a fixed object
ID for the image, or by adding a method / view to the event which
redirected to the actual image URL.



Tres.
- -- 
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iQIcBAEBAgAGBQJVWNHiAAoJEPKpaDSJE9HY/nYQALbHx3WvyjEk/6rsgomcWJhN
va4ZgWEjQtFuSlzHOf30CGKIEwNX+EAYWw6juMznQo/4Yq5b2g4Ur8cA9q1gYgFB
2ZwW4/W4wKsaAJxQ6sckinAKhlWGJugXdpcxnbbU3V0ePoUtgPsO0GVIlPoDV+WF
/qEnWYWTu+7xHf/DV5iiHKTqYM0Oqv1aNTuZd9s8/emHoDuU8RpzVLscZuJEYk+B
5mdFFSfNJ9wer79fxQH8yM+8Vz6IkVx4XCCTa9/9QM9qHTVLqvv+AllbtolQ4gms
ASkisuW7p/XvJgK5bMbUFvEkDZQQDMFAuRKe2lnVT6rAFXiR04fHk6GKKtW3WyEb
Cdq4GYcKz4ZYdF4l2QV8h+FOZRAf6qWW9CCiRsuihaiYB57Uk8hI/GPk00TXL78z
rmOWtZ86d7hlCarP8k+T+1b5b+RQUB8danop21ys3SeH+FMiEPxjIuhY0i43n6xQ
dg6HJ8ry7KihYipaRJrC0tqVFluToWFdLUaM1qRSJHpcKi3TpHKiKXGzNSHzeI1y
GD7zb3HF+lYqsy74wPWIp/flSmJxJMg5givfHt96/sFvaaNstvjIY6KiKUUW0JMy
8b/fF4ufHu2ZGoT8TSMCZRa4KpNMcat9Edo/MsgOeir65a3PnFj9Br8DmpZJwTbV
9KZEKY4Aq+EJq+RPmH2D
=U40Y
-END PGP SIGNATURE-

___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog : Searching Multiple Indexes : Returning One Result

[Tom Von Lahndorff]

If I understand correctly, this seems like searching for multiple values in
one index. What I'm looking to do is search for one value in multiple
indexes... and get back one result even if there are True hits in more than
one index.

On Mon, Jul 7, 2008 at 4:00 PM, Jonathan <[EMAIL PROTECTED]> wrote:

>
> - Original Message - From: "Tom Von Lahndorff" <[EMAIL PROTECTED]>
> To: "zope list user" 
> Sent: Monday, July 07, 2008 3:42 PM
> Subject: [Zope] ZCatalog : Searching Multiple Indexes : Returning One
> Result
>
>
>
>  Hi, I have a catalog called "catalog" that contains a bunch of cataloged
>> items. I have indexed 3 properties for each item, "name", "body" and
>> "description". I'd like to be able to search the catalog in the following
>> way:
>>
>> if the word "text" shows up in any index, "name", "body" or "description",
>> return that result back once.
>>
>> The problem is I have is that if a do a search for "text" and say there's
>> results that have the word "text" in one or more of those indexes:
>>
>> catalog({name:query}) or catalog({body:query}) or
>> catalog({description:query})
>> only gives back limited results (leaves out some 'True' hits) and:
>> catalog({name:query}) + catalog({body:query}) +
>> catalog({description:query})
>> gives me back the same result 3 times.
>>
>> any ideas? Thanks.
>>
>
> Something I saved from a long time ago post (warning untested) which may
> give you some ideas:
>
> 
> context.Catalog({'keywordindex': ['Value1', 'Value2']})
> Catalog returns me an OR search instead of an AND one
>
> Use "context.Catalog(keywordindex = {'query':[...], operator='and'})"
> when you want to use the "and" operator.
> 
>
> hth
>
> Jonathan
>
>
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [zope] ZCatalog question

[Jonathan]

- Original Message - 
From: "Jonathan Salazar Santos" <[EMAIL PROTECTED]>

To: 
Sent: Tuesday, July 08, 2008 5:17 PM
Subject: [zope] ZCatalog question



Hi everybody


I have a method that creates a Files and DTML Documents but I need to
Index those news items to a ZCatalog when they are created.

Do you know if there are some dtml calls (like manage_catalogACTION)
to call the action: Index, Reindex, Delete indexed objects with dtml
or Python Scrips.

How can I do to get indexed my new items in a zcatalog automatically?

Some help will be appreciated.

Regards,
--
Jonathan Salazar


Basic ZCatalog info can be found in the "Searching and Categorizing Content" 
chapter of the Zope Book (http://www.plope.com/Books/2_7Edition)


Automatic object cataloging was accomplished by creating 'ZCatalog aware' 
object instances - i can't remember if this is a feature of ZClasses only 
(stay away from ZClasses) or if you can create your own product that 
subclasses the 'awareness' features (google is your friend).


You can manually add/remove items using the methods "catalog_object" and 
"uncatalog_object" (Zope Book and google for more info).


hth

Jonathan (another one) 


___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog : Searching Multiple Indexes : Returning One Result

[Jonathan]

- Original Message - 
From: "Tom Von Lahndorff" <[EMAIL PROTECTED]>

To: "zope list user" 
Sent: Monday, July 07, 2008 3:42 PM
Subject: [Zope] ZCatalog : Searching Multiple Indexes : Returning One Result



Hi, I have a catalog called "catalog" that contains a bunch of cataloged
items. I have indexed 3 properties for each item, "name", "body" and
"description". I'd like to be able to search the catalog in the following
way:

if the word "text" shows up in any index, "name", "body" or "description",
return that result back once.

The problem is I have is that if a do a search for "text" and say there's
results that have the word "text" in one or more of those indexes:

catalog({name:query}) or catalog({body:query}) or
catalog({description:query})
only gives back limited results (leaves out some 'True' hits) and:
catalog({name:query}) + catalog({body:query}) + 
catalog({description:query})

gives me back the same result 3 times.

any ideas? Thanks.


Something I saved from a long time ago post (warning untested) which may 
give you some ideas:



context.Catalog({'keywordindex': ['Value1', 'Value2']})
Catalog returns me an OR search instead of an AND one

Use "context.Catalog(keywordindex = {'query':[...], operator='and'})"
when you want to use the "and" operator.


hth

Jonathan

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog, brains & keywordIndex

[Andreas Jung]

--On 25. Februar 2008 16:45:51 +0100 Stefano Guglia <[EMAIL PROTECTED]> 
wrote:






That's not the primary supported usecase of ZCatalog. Perhaps



will help you any further.

-aj


anyway, I do have persistent objects in my plone product, no matter on
how I originated them.. the problem is that I have no idea on how to
index the values from a python list as a keywordindex...



It works exactly as documented here:



If your index is  then your data must be exposed as attribute 
of the related object or returned from a method call to ().

-aj

pgpHFOIBd635u.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog, brains & keywordIndex

[Stefano Guglia]

> 
> That's not the primary supported usecase of ZCatalog. Perhaps
> 
> 
> 
> will help you any further.
> 
> -aj

anyway, I do have persistent objects in my plone product, no matter on
how I originated them.. the problem is that I have no idea on how to
index the values from a python list as a keywordindex...

If anybody there listening, please help! :-)

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog, brains & keywordIndex

[Andreas Jung]

--On 25. Februar 2008 12:00:24 +0100 Stefano Guglia <[EMAIL PROTECTED]> 
wrote:






You are trying to *index* rows returned from a ZSQL operation.
This make no sense. As said: ZCatalog is designed to deal with
persistent objects (the things you see within your ZODB hierarchy within
the ZMI). And why do you want to index stuff within Zope if you can
perform  the search directly within your database!? This makes
absolutely no sense  what you're trying to do. At least the
implementation approach is broken by  design.

-aj


because I need data to be searched thru standard zope/plone search tools
(zcatalog - quick search form etc.) by users and search engines and this
can be done only by 'translating' queries into persistent objects. If I
understand well, brains are there just for this purpose. I'm just a
beginner in zope, so please let me know if there are more effective ways
to do that.



That's not the primary supported usecase of ZCatalog. Perhaps



will help you any further.

-aj

pgpuBhPtXZ70l.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog, brains & keywordIndex

[Stefano Guglia]

> 
> You are trying to *index* rows returned from a ZSQL operation.
> This make no sense. As said: ZCatalog is designed to deal with
> persistent objects (the things you see within your ZODB hierarchy within 
> the ZMI). And why do you want to index stuff within Zope if you can perform 
> the search directly within your database!? This makes absolutely no sense 
> what you're trying to do. At least the implementation approach is broken by 
> design.
> 
> -aj

because I need data to be searched thru standard zope/plone search tools
(zcatalog - quick search form etc.) by users and search engines and this
can be done only by 'translating' queries into persistent objects. If I
understand well, brains are there just for this purpose. I'm just a
beginner in zope, so please let me know if there are more effective ways
to do that.

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog, brains & keywordIndex

[Andreas Jung]

--On 25. Februar 2008 10:09:00 +0100 Stefano Guglia <[EMAIL PROTECTED]> 
wrote:



Hello everybody..

I have to index some data from a mysql query recordset and all works OK
but..I added a keywordIndex to store (with the same url) multiple values
related to the above recordset.
It seems I cannot use brains (query results are strings).

Probably a newbie question, but I really do not know how to update  a
keywordIndex starting from a list..Any help (+sample code pls :-)) ?



Unless you don't provide the related code, it is hard to guess what you're 
really doing. In general: ZCatalog/indexes are designed in the first place 
to deal with  Zope objects, not with foreign data without a persistent 
object.


-aj

pgpona4OmIrXi.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog len problem (resolved)

[Jonathan]

- Original Message - 
From: "Andreas Jung" <[EMAIL PROTECTED]>

To: "Jonathan" <[EMAIL PROTECTED]>; 
Sent: Saturday, March 24, 2007 10:43 AM
Subject: Re: [Zope] ZCatalog len problem

--On 24. März 2007 10:16:46 -0400 Jonathan <[EMAIL PROTECTED]> wrote:


I have a ZCatalog and in the ZMI 'Catalog' tab it reports:

Returns:

len(sRes)=  1820
cnt=  1842



The result remains the same after reindexing the whole catalog?


Yes.  I have an application that has 5 different ZCatalogs. The same 
problem occurs in 3 of them. It seems that a ZCatalog with less than ~500 
records is ok, but the len problem increases with the number of records in 
the ZCatalog:


actuallendiff

108   108   0
322   322   0
613   607   6
1842 1820 22
3628 3590 38




It looks like the __len__ method in Catalog.py is not returning the
correct value - it returns self._length() which is set by the following
method (also in Catalog.py):

def migrate__len__(self):
   """ migration of old __len__ magic for Zope 2.8 """
if not hasattr(self, '_length'):
n = self.__dict__['__len__']()
del self.__dict__['__len__']
self._length = BTrees.Length.Length(n)



Is this migration code for pre-2.8 catalogs. This change was necessary
because the new ExtensionClass implementation did not longer work
with the scary implementation of the __len__ functionality of the Catalog
class.

If the result remains the same after reindexing the catalog there might 
be

an issue with the increment/decrement calls of the _length counter
with catalogObject/uncatalogObject...however no idea why...


Clearing the catalog and then running 'Find Objects' returns the same 
incorrect number every time (at least the error is consistent!). I have 
also loaded records into the ZCatalogs via a python script/external method 
and the record counts are exactly the same as with the 'Find Objects'.


I wrote a bunch of unittests and could not recreate the problem, so i 
reinstalled zope and rebuilt the application infrastructure (ZCatalogs, etc) 
and reloaded all the data from scratch and the problem went away!


Good idea of using the unittests to track down the problem, thanks Andreas.

Jonathan

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog len problem

[Andreas Jung]

--On 24. März 2007 11:03:58 -0400 Jonathan <[EMAIL PROTECTED]> wrote:


I don't think it is an indexing problem as the indexes are reporting the
correct number of entries.  It is just the _length() method of the
catalog that seems to be the problem (but I have n idea how to fix
it!)



Of course it has to do with indexing since the catalog triggers the
related index_object() calls of the indexes :-)
You might track this down by checking the self._length.change()
calls that increment/decrement the counter. A unittest demonstrating
the error might help a lot *wink* :-)

Andreas

pgprZH6xGQ1Hd.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog len problem

[Jonathan]

- Original Message - 
From: "Andreas Jung" <[EMAIL PROTECTED]>

To: "Jonathan" <[EMAIL PROTECTED]>; 
Sent: Saturday, March 24, 2007 10:43 AM
Subject: Re: [Zope] ZCatalog len problem

--On 24. März 2007 10:16:46 -0400 Jonathan <[EMAIL PROTECTED]> wrote:


I have a ZCatalog and in the ZMI 'Catalog' tab it reports:

Returns:

len(sRes)=  1820
cnt=  1842



The result remains the same after reindexing the whole catalog?


Yes.  I have an application that has 5 different ZCatalogs. The same problem 
occurs in 3 of them. It seems that a ZCatalog with less than ~500 records is 
ok, but the len problem increases with the number of records in the 
ZCatalog:


actuallendiff

108   108   0
322   322   0
613   607   6
1842 1820 22
3628 3590 38




It looks like the __len__ method in Catalog.py is not returning the
correct value - it returns self._length() which is set by the following
method (also in Catalog.py):

def migrate__len__(self):
   """ migration of old __len__ magic for Zope 2.8 """
if not hasattr(self, '_length'):
n = self.__dict__['__len__']()
del self.__dict__['__len__']
self._length = BTrees.Length.Length(n)



Is this migration code for pre-2.8 catalogs. This change was necessary
because the new ExtensionClass implementation did not longer work
with the scary implementation of the __len__ functionality of the Catalog
class.

If the result remains the same after reindexing the catalog there might be
an issue with the increment/decrement calls of the _length counter
with catalogObject/uncatalogObject...however no idea why...


Clearing the catalog and then running 'Find Objects' returns the same 
incorrect number every time (at least the error is consistent!). I have also 
loaded records into the ZCatalogs via a python script/external method and 
the record counts are exactly the same as with the 'Find Objects'.


I don't think it is an indexing problem as the indexes are reporting the 
correct number of entries.  It is just the _length() method of the catalog 
that seems to be the problem (but I have n idea how to fix it!)


Should I file a bug report on this?

Jonathan


___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog len problem

[Andreas Jung]

--On 24. März 2007 10:16:46 -0400 Jonathan <[EMAIL PROTECTED]> wrote:


I have a ZCatalog and in the ZMI 'Catalog' tab it reports:

Returns:

len(sRes)=  1820
cnt=  1842



The result remains the same after reindexing the whole catalog?



It looks like the __len__ method in Catalog.py is not returning the
correct value - it returns self._length() which is set by the following
method (also in Catalog.py):

def migrate__len__(self):
""" migration of old __len__ magic for Zope 2.8 """
if not hasattr(self, '_length'):
n = self.__dict__['__len__']()
del self.__dict__['__len__']
self._length = BTrees.Length.Length(n)



Is this migration code for pre-2.8 catalogs. This change was necessary 
because the new ExtensionClass implementation did not longer work

with the scary implementation of the __len__ functionality of the Catalog
class.

If the result remains the same after reindexing the catalog there might be
an issue with the increment/decrement calls of the _length counter
with catalogObject/uncatalogObject...however no idea why...

-aj




pgpdKpUr57W7h.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog strange behavior

[bard stéphane]

thank's dieter !
In fact i lose the tip that the Zcatalog should
have the id "Catalog" so the object could
be automatically cataloged at creation.

thank's a lot

Dieter Maurer a écrit :

bard stéphane wrote at 2006-6-24 19:38 +0200:
  

Today i try to create another zclass on the same zope server (2.6)
zclass with zobject,zcatalogaware inheritance
and the zcatalogawareness don't work any more.
And I don't know why !!

So each time a property is modified i should



"ZcatalogAware" never triggered reindexing when a property
was changed -- only indexing when the object was created and
unindexing when it was deleted.

Call "indexObject" (or "reindexObject") after you have make
changes.



  


___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog strange behavior

[Dieter Maurer]

bard stéphane wrote at 2006-6-24 19:38 +0200:
>Today i try to create another zclass on the same zope server (2.6)
>zclass with zobject,zcatalogaware inheritance
>and the zcatalogawareness don't work any more.
>And I don't know why !!
>
>So each time a property is modified i should

"ZcatalogAware" never triggered reindexing when a property
was changed -- only indexing when the object was created and
unindexing when it was deleted.

Call "indexObject" (or "reindexObject") after you have make
changes.



-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog Strategy

[Dieter Maurer]

Mark Gibson wrote at 2006-3-22 19:28 -0700:
>I'm struggling to weigh the cost of getObject() vs. the cost of adding 
>more metadata to the catalog.  I'll explain my situation.
>
>I have 10,000 widgets cataloged.  I do a path and date query that 
>returns me maybe 12 of these.  Then I have a choice of calling 
>getObject().getData() on each of these, or I could add getData to the 
>catalog metadata.
>
>Does the cost of calling getObject() for a dozen objects justify 
>creating a new metadata field?
>
>More generally how does a large amount of metadata in the catalog affect 
>performance of queries?

This is very difficult to say.


  We have used the standard metadata set for a long time.
  It contains "description". However, in our cases
  "description" was usually several kb large.

  We found that all catalog operations (lookup, indexing)
  were very slow and modifying catalog operations caused
  huge transaktions.

  The "IOBBuckets" containing the metadata records have
  been the culprit. Usually, they contain 45 metadata records.
  If you need one record, you in fact handle usually 45 of them.
  In our case, the buckets usually have been several 100 kB big -- much
  larger then usual container objects...


On the other hand, if you have small metadata records (say, containing
a few integers), then loading them may be much faster
than loading all intermediate objects to your final object -- especially
for applications such as determining statistics data (when you
are processing a lot of objects).


-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog Strategy

[Chris Withers]

Mark Gibson wrote:
Does the cost of calling getObject() for a dozen objects justify 
creating a new metadata field?


No.

More generally how does a large amount of metadata in the catalog affect 
performance of queries?


Badly ;-)


The wisdom of those more knowledgeable than me would be appreciated.


As a rule of thumb, if you need a catalog search to return more than 
10-20 objects and you need to do something with an attribute of all the 
objects returned, then whack it in the metadata. If you only need it for 
the 10-20 objects, then do getObject and get the attribute from the object.


cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
   - http://www.simplistix.co.uk

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog Strategy

[Jonathan]

One factor is the amount of ram that you have.  If you have enough ram to 
fit the entire catalog (with indexes)  then your search time is very fast. 
I would err on the side of less meta-data.  We currently run about 1 million 
entries in a zcatalog with very little meta-data and then use 
restrictedTraverse with the object ids returned by the zcatalog search to 
get the fields we need.  We typically have sub 1 second searches using this 
approach (but we have very fast scsi disks in striped raid configuration).


With only 10k records in the catalog you could try both configurations (very 
easy to clear and then reload the catalog) and time them to see which gives 
you better performance.


hth & good luck!

Jonathan


- Original Message - 
From: "Mark Gibson" <[EMAIL PROTECTED]>

To: 
Sent: Wednesday, March 22, 2006 9:28 PM
Subject: [Zope] ZCatalog Strategy


I'm struggling to weigh the cost of getObject() vs. the cost of adding 
more metadata to the catalog.  I'll explain my situation.


I have 10,000 widgets cataloged.  I do a path and date query that returns 
me maybe 12 of these.  Then I have a choice of calling 
getObject().getData() on each of these, or I could add getData to the 
catalog metadata.


Does the cost of calling getObject() for a dozen objects justify creating 
a new metadata field?


More generally how does a large amount of metadata in the catalog affect 
performance of queries?


The wisdom of those more knowledgeable than me would be appreciated.

Mark
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )



___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: searching where on index == another index

[Dieter Maurer]

Etienne Labuschagne wrote at 2005-10-21 13:00 +0200:
>Is there a way to search for all the records where one indexed field
>equals another indexed field?
>
>Equivalent SQL query would be:
>
>SELECT * FROM table WHERE field1 = field2

If the index has only a few different values,
enumerating them may be feasible.

With "AdvancedQuery", this could look like:

 from Products.AdvancedQuery import Or, Eq

 query = Or(*[Eq(I1,v) & Eq(I2,v) for v in I1.unique(Values)])

-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: searching where on index == another index

[Andreas Jung]

--On 21. Oktober 2005 13:00:35 +0200 Etienne Labuschagne 
<[EMAIL PROTECTED]> wrote:



Hi all,

Is there a way to search for all the records where one indexed field
equals another indexed field?




no.

-aj

pgpT5FL2z86QW.pgp
Description: PGP signature
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and custom Product

[Chris Withers]

Milos Prudek wrote:
The above code catalogs the instance into Catalog_A. Okay, but
self.manage_editCataloger("Catalog_A")
self.index_object()
self.manage_editCataloger("Catalog_B")
self.index_object()
self.manage_editCataloger("Catalog_C")
self.index_object()
... the above code catalogs the instance into Catalog_C, but not into 
Catalog_A and not into Catalog_B. Does anyone know why Catalog_A and 
Catalog_B are ignored?
Using CatalogPathAware with multiple catalogs is asking for trouble, 
since it's onyl designed to support one catalog ;-)

I'd suggest not using it in your case. Just write your own index_object 
method that does what you really want it to...

cheers,
Chris
--
Simplistix - Content Management, Zope & Python Consulting
   - http://www.simplistix.co.uk
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and custom Product

[Milos Prudek]

Milos Prudek wrote:
> Hi,
>
> I need to convert a ZClass cataloged in multiple ZCatalogs into a
> Product. I know how to create a Product. But I have trouble
> understanding how to create a proper ZCatalog and manage it in a Product.
To answer myself, simply follow Garito's advice of 10.6.2004 in mailing 
list archive.

Now the only problem is multiple catalogs. I need to catalog my class 
into Catalog_A, Catalog_B and Catalog_C upon instantiation.

class MyClass(CatalogPathAware):
  def __init__():
self.manage_editCataloger("Catalog_A")
self.index_object()
The above code catalogs the instance into Catalog_A. Okay, but
self.manage_editCataloger("Catalog_A")
self.index_object()
self.manage_editCataloger("Catalog_B")
self.index_object()
self.manage_editCataloger("Catalog_C")
self.index_object()
... the above code catalogs the instance into Catalog_C, but not into 
Catalog_A and not into Catalog_B. Does anyone know why Catalog_A and 
Catalog_B are ignored?

--
Milos Prudek
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog uniqueValuesFor

[Dieter Maurer]

aZaZel writes:
 >  ... "uniqueValuesFor" not updated after modification ...
Almost surely a small bug in ZCatalog.

A field index is implemented as a mapping from values to
lists of documents.
"uniqueValuesFor" returns this mapping's "keys()".

When the field value of a document changes, this
document is removed from the corresponding list.
Obviously, a test is missing whether the result
is the empty list. If it is, the key should be deleted.

Probably, you can fix this yourself.
The code is in "SearchIndex.UnIndex".

If you fixed it, file a bug with patch report into the Collector.


Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] Zcatalog with Word Documents

[Tim Cook]

Nuno Goncalves wrote:
> 
> Hi there !
> Is it possible to catalog the content of a word document with Zcatalog ?
> If so i would apreciate that someone could tell me how !!

The list archives are a great resource:
http://zope.nipltd.com/public/lists/zope-archive.nsf/Main?SearchView=++&Query=MSWord

-- Tim Cook, President --
Free Practice Management,Inc. | http://www.FreePM.com Office:
(901) 884-4126
"When the government fears the People, that is Liberty. When the
People fear
the Government, that is tyranny." ~~ Thomas Jefferson

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

RE: [Zope] ZCatalog and 'fuzzy logic'

[sean . upton]

My understanding is that part of the problem is that sorting a dataset based
upon regular expression (doesn't matter what algorithm you use) creates a
rather large tree of possibilities that is largely impractical to sort, at
least not without a lot of overhead.  

My compromise for the searches that I'm planning on using is to allow regex
searches of "search results" from the catalog.  This requires a bit more
patience, as it is an unindexed search...

Sean

-Original Message-
From: Morten W. Petersen [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, January 09, 2001 7:46 AM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: [Zope] ZCatalog and 'fuzzy logic'


Is there anyone who could try to give an estimate of how long it would
take to add fuzzy logic (regexp-like) searching capability to the
ZCatalog?

And reasoning as to why would be appreciated. ;)

-Morten


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog raises index out of range exception error

[Aleksander Salwa]

On Tue, 19 Dec 2000, Marcin Kasperski wrote:
> Partial solution: at something like 2:00 AM delete your SiteRoot rule,
> reindex with ZCatalog and create this rule again. Searching works
> (although I needed to edit the paths found a bit in the form responsilbe
> for search results presentation).

You don't need to delete your SiteRoot. You can use one of techniques
described in SiteRoot docs to suppress it within particular request.
I use trick with phantom folder, call it 'Z'.
So my AccessRule looks like:

Is there a path, and does it start with 'Z'?

 
  Get rid of 'Z': 
  Put it back logically: 
  
 

Is the first-level domain 'hotsite.com'? Ignore sub-domains and port
number.

 Add physical root: 
 
 


 



In my SiteRoot, I have all properties left blank.

When I access my server using URL like www.myserver.com/Z/something, then
SiteRoot is turned off.

With this solution, your site may be still accessible to customers, while
you are working without SiteRoot (i.e. cataloging objects).

[EMAIL PROTECTED]

/--\
| `long long long' is too long for GCC |
\--/





___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog raises index out of range exception error

[Marcin Kasperski]

> > I believe I saw this error when
> > trying to use the Catalog in a SiteRoot-ed folder...
> 
> Me too :-(
> 

And me too. The most irritating Zope bug for me. 

> > No solution, sorry ;)
> 
> :-(
> 

Partial solution: at something like 2:00 AM delete your SiteRoot rule,
reindex with ZCatalog and create this rule again. Searching works
(although I needed to edit the paths found a bit in the form responsilbe
for search results presentation).

I can imagine writing the script (XMLRPC?) which would do it from the
cron job - although I have not yet found spare time (I still think
someone will do something with this nasty bug).

--
http://www.mk.w.pl /
 Marcin.Kasperski | Dokumentację kodu warto wygenerować:  
   @softax.com.pl |   http://www.mk.w.pl/narzedzia/narzedzia_gendoc   
 @bigfoot.com  \

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog raises index out of range exception error

[Stephane Bortzmeyer]

On Saturday 16 December 2000, at 19 h 21, the keyboard of "Stefan H. Holek" 
<[EMAIL PROTECTED]> wrote:

> Do you have a SiteRoot-ed environment? 

I do.

> I believe I saw this error when
> trying to use the Catalog in a SiteRoot-ed folder...

Me too :-(

> No solution, sorry ;)

:-(



___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog raises index out of range exception error

[Dieter Maurer]

Stefan H. Holek writes:
 > Do you have a SiteRoot-ed environment? I believe I saw this error when
 > trying to use the Catalog in a SiteRoot-ed folder...
 > 
 > No solution, sorry ;)
We use the Catalog together with a SiteRoot without problems.

  I remove the host part of the URL before I index
  the object.

Dieter

PS: Sorry!
After writing, I recognized that we make
only restricted use of SiteRoots:

   we just change the host part but not the
   path part.

   As a result, the URL's only differ in
   the host part. The locally (wrt. the host)
   absolute URL's of an object are equal
   for all virtual domains.
   Therefore, removing the host part
   works out restricted use.

The general case, where the path part is
different for each virtual host,
is more difficult.

   I would probably go for a separate catalog
   for each virtual host and ensure
   that the URL's with which the objects
   are indexed are the URL's for this
   virtual host.

   This may require, that indexing is
   done in this same virtual host environment.

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog raises index out of range exception error

[Stefan H. Holek]

On Fri, 15 Dec 2000, Martin Winkler wrote:

> On a client's server I created a ZCatalog, but when using "Find Items to 
> ZCatalog" and clicking on "Find" in the form, Zope raises the "list index 
> out of range" error shown in detail at the end of this e-mail. Then I took 
> the whole Data.fs, put it on one of our servers, where everything worked as 
> usual.

Do you have a SiteRoot-ed environment? I believe I saw this error when
trying to use the Catalog in a SiteRoot-ed folder...

No solution, sorry ;)
Stefan



___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: Query Syntax

[Chris Withers]

Chris McDonough wrote:
> 
> The place to start might be to actually help write some documentation.
> There is a docs process description at
> http://dev.zope.org/Wikis/DevSite/Projects/DocumentationProcess/Documentatio
> nGuide .  This should at least give you an idea of how to proceed.

Doug,

If you make a start with this, I'll see if I can help (although the
problem is that while I use the catalog, I don't actually seem to know a
lot about how it works, because so little of it is documented). I'd love
to have a decent, working Catalog, but I'm under more than a little time
pressure with other things right now :-S

Chris

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: Query Syntax

[Chris McDonough]

The place to start might be to actually help write some documentation.
There is a docs process description at
http://dev.zope.org/Wikis/DevSite/Projects/DocumentationProcess/Documentatio
nGuide .  This should at least give you an idea of how to proceed.

- Original Message -
From: "Doug Morse" <[EMAIL PROTECTED]>
To: "Chris Withers" <[EMAIL PROTECTED]>
Cc: "Andy McKay" <[EMAIL PROTECTED]>; "Dieter Maurer"
<[EMAIL PROTECTED]>; <[EMAIL PROTECTED]>
Sent: Wednesday, November 29, 2000 9:38 PM
Subject: Re: [Zope] ZCatalog: Query Syntax


> Chris,
>
> I also had great difficulty in finding any relevant documentation... (re:
> specifics regarding text and field searches against a ZCatalog)...
>
> So, it sounds like we should (a) make a feature request and (b) make a
> documentation request.  I'm more than happy to write up and submit such
> requests.  Heck, I'd been even happy to tackle coding the improvements,
but
> I'd need someone very familiar with the ZCatalog implementation to give me
> some direction to get me going (i.e., how he or she would recommend the
> change be implemented).
>
> But, I'm quite new to the Zope world, and have no idea where one goes to
> request a feature, or request documentation, or even where one goes to
offer
> help coding.  Anyone want to point me in the right direction? ;-)
>
> Cheers,
> Doug Morse
> [EMAIL PROTECTED]
>
>
> On Wed, Nov 29, 2000 at 02:38:30PM +, Chris Withers wrote:
> > Andy McKay wrote:
> > >
> > > > I am curious: Do others think this feature might be worth
> > > > adding to the ZCatalog?  If so, perhaps it should work its
> > > > way into a proper feature request...
> > >
> > > Yes definitely.
> >
> > Well, where is the query syntax for ZCatalog actually documented?
> >
> > It'd be great if it was more flexible, applied to indexes other than
> > TextIndexes and was actually documented...
> >
> > Any ideas?
> >
> > Chris
>
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>
>


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: Query Syntax

[Doug Morse]

Chris,

I also had great difficulty in finding any relevant documentation... (re:
specifics regarding text and field searches against a ZCatalog)...

So, it sounds like we should (a) make a feature request and (b) make a
documentation request.  I'm more than happy to write up and submit such
requests.  Heck, I'd been even happy to tackle coding the improvements, but
I'd need someone very familiar with the ZCatalog implementation to give me
some direction to get me going (i.e., how he or she would recommend the
change be implemented).

But, I'm quite new to the Zope world, and have no idea where one goes to
request a feature, or request documentation, or even where one goes to offer
help coding.  Anyone want to point me in the right direction? ;-)

Cheers,
Doug Morse
[EMAIL PROTECTED]


On Wed, Nov 29, 2000 at 02:38:30PM +, Chris Withers wrote:
> Andy McKay wrote:
> > 
> > > I am curious: Do others think this feature might be worth
> > > adding to the ZCatalog?  If so, perhaps it should work its
> > > way into a proper feature request...
> > 
> > Yes definitely.
> 
> Well, where is the query syntax for ZCatalog actually documented?
> 
> It'd be great if it was more flexible, applied to indexes other than
> TextIndexes and was actually documented...
> 
> Any ideas?
> 
> Chris

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and dates

[Dieter Maurer]

James Sintz writes:
 > I have a ZCatalog set up to index a date field and a title field.
 > 
 > If I do a search for both meta_type of "news" a date of 2000/11/16 in the
 > goLiveDate field the results page returns "There was no data matching this
 > Catalog query"
Zope uses "DateTime" object to hold date and time.

When you render such an object, you see just the date portion.
But there is a time portion as well. It is quite difficult
to make a query, where the dates match exactly.

You will probably need range queries to search for dates
(--> ZopeBook, Advances ZCatalog HowTo).


Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: Query Syntax

[Chris Withers]

Andy McKay wrote:
> 
> > I am curious: Do others think this feature might be worth
> > adding to the ZCatalog?  If so, perhaps it should work its
> > way into a proper feature request...
> 
> Yes definitely.

Well, where is the query syntax for ZCatalog actually documented?

It'd be great if it was more flexible, applied to indexes other than
TextIndexes and was actually documented...

Any ideas?

Chris

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: How to request boolean NOT searches?

[Andy McKay]

> I am curious: Do others think this feature might be worth
> adding to the ZCatalog?  If so, perhaps it should work its
> way into a proper feature request...

Yes definitely.


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: How to request boolean NOT searches?

[Doug Morse]

Dieter,

> Doug Morse writes:
>  > ... negating a ZCatalog index lookup ...
> 
> Only ZCatalog text indexes support a restricted form of negation.
> Keyword and field indexes do not.
> Text indexes support only "and not":
> e.g. "Zope and not databases".
> Free standing "not" or combined with "or" (rather than "and")
> is not supported.

Super, thanks Dieter!  Not the answer I was hoping to hear,
but it least it's definitive. ;-)  

I am curious: Do others think this feature might be worth
adding to the ZCatalog?  If so, perhaps it should work its
way into a proper feature request...

Cheers!
Doug Morse
[EMAIL PROTECTED]

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and LoginManager - be aware.

[Oliver Bleutgen]

> The problem is that one of your databases doesn't define the commit_sub()
> and abort_sub() methods on the DB class it exposes.  You don't say which
> SQL
> database you're using, but this happened to me on a consulting project
> with
> ZOracleDA.  A solution is to add the following methods to the database
> adapter's DB class inside DB.py:

> def commit_sub(self, *arg, **kw):
>pass

> def abort_sub(self, *arg, **kw):
>pass

> This isn't optimal, because it sort of breaks transactional integrity, but
> it does make it work.


Thanks for your reply. I use ZPoPyDA (0.6.4 says the readme).

I think I will not apply that patch (thanks nonetheless), 
because I have no requirements for indexing big amounts
of text. And as I understand it, _indexing_ (as opposed to
normal searches) is the only action where subtransactions
help.
My main motivation for posting was to get that problem
into the public, so that others may find it while doing 
desperate searches in this mailing list ;-).
And perhaps - if I dare - to suggest to disable 
subtransactions in ZCatalog by default.

cheers,
oliver

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and LoginManager - be aware.

[Chris McDonough]

The problem is that one of your databases doesn't define the commit_sub()
and abort_sub() methods on the DB class it exposes.  You don't say which SQL
database you're using, but this happened to me on a consulting project with
ZOracleDA.  A solution is to add the following methods to the database
adapter's DB class inside DB.py:

def commit_sub(self, *arg, **kw):
   pass

def abort_sub(self, *arg, **kw):
   pass

This isn't optimal, because it sort of breaks transactional integrity, but
it does make it work.

- Original Message -
From: "Michel Pelletier" <[EMAIL PROTECTED]>
To: "Oliver Bleutgen" <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Tuesday, November 28, 2000 6:12 PM
Subject: Re: [Zope] ZCatalog and LoginManager - be aware.


> Oliver Bleutgen wrote:
> >
> > Hi,
> >
> > I just wanted to mention a IMO bad pitfall when using ZCatalog, for
> > the sake of sparing others some time.
> >
> > I'm running Zope 2.2.2 with LoginManager with SQL in a subfolder and was
getting
> > the following error  when trying to find dtml-documents into ZCatalog:
> >
> > Error Type: AttributeError
> > Error Value: commit_sub
> >
> > Traceback (innermost last):
> >   File
/home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/ZPublisher/Publish.
py, line 222, in publish_module
> >   File
/home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/ZPublisher/Publish.
py, line 187, in publish
> >   File
/home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/Zope/__init__.py,
line 221, in zpublisher_exception_hook
> > (Object: Traversable)
> >   File
/home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/ZPublisher/Publish.
py, line 175, in publish
> >   File
/home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/Zope/__init__.py,
line 235, in commit
> >   File
/home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/ZODB/Transaction.py
, line 261, in commit
> > AttributeError: (see above)
> >
> > All other object-types (incuding dtml-methods) seem to work. I have put
no
> > restrictions on which files I want found, but restricting something to
> > reduce the number of found objects helps.
> >
> > I fiddled around somewhat and discovered that disabling
> > subtransactions in ZCatalog does help (required for using
> > ZSQL Methods and ZCatalog in the same transaction).
>
> Yeah, that sux.  I think this is a known limitation.  I'm not sure.
>
> > Now I use SQL-based LoginManager, but why on earth did it only
> > fail on DTML-Documents? Is it because I have much more of them so
> > only then ZCatalog's subtransactions come into play?
>
> Probably because sub transactions are only commited after a certain
> number of words or values are indexed, and your DTML documents were the
> only things that held words.
>
> -Michel
>
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>
>


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: How to request boolean NOT searches?

[Dieter Maurer]

Doug Morse writes:
 > ... negating a ZCatalog index lookup ...

Only ZCatalog text indexes support a restricted form of negation.
Keyword and field indexes do not.
Text indexes support only "and not":
e.g. "Zope and not databases".
Free standing "not" or combined with "or" (rather than "and")
is not supported.


Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and LoginManager - be aware.

[Oliver Bleutgen]

> Oliver Bleutgen wrote:
>> I fiddled around somewhat and discovered that disabling
>> subtransactions in ZCatalog does help (required for using
>> ZSQL Methods and ZCatalog in the same transaction).

> Yeah, that sux.  I think this is a known limitation.  I'm not sure.

It is, unfortunately this is only mentioned under "status" in
the ZCatalog interface. Well, I just got there by chance after 
several hours of digging around. I first thought perhaps some
DTML-Document was "bad-behaving". (not that I had an idea how
it might do that.)
With a (assumed) growing attractiveness of things like sql-based
UserFolders, perhaps a big warning sign either on the main page
of these products or ZCatalog (better IMO, since this is a default
product) would be in order, I guess. Or perhaps disabling 
subtransactions by default.
All in all this combination seems not uncommon.

>> Now I use SQL-based LoginManager, but why on earth did it only
>> fail on DTML-Documents? Is it because I have much more of them so
>> only then ZCatalog's subtransactions come into play?

> Probably because sub transactions are only commited after a certain
> number of words or values are indexed, and your DTML documents were the
> only things that held words.


> -Michel

cheers,
oliver

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and LoginManager - be aware.

[Michel Pelletier]

Oliver Bleutgen wrote:
> 
> Hi,
> 
> I just wanted to mention a IMO bad pitfall when using ZCatalog, for
> the sake of sparing others some time.
> 
> I'm running Zope 2.2.2 with LoginManager with SQL in a subfolder and was getting
> the following error  when trying to find dtml-documents into ZCatalog:
> 
> Error Type: AttributeError
> Error Value: commit_sub
> 
> Traceback (innermost last):
>   File 
>/home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/ZPublisher/Publish.py, line 
>222, in publish_module
>   File 
>/home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/ZPublisher/Publish.py, line 
>187, in publish
>   File /home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/Zope/__init__.py, 
>line 221, in zpublisher_exception_hook
> (Object: Traversable)
>   File 
>/home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/ZPublisher/Publish.py, line 
>175, in publish
>   File /home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/Zope/__init__.py, 
>line 235, in commit
>   File /home/local-extranet/web/zope3/Zope-2.2.2-src/lib/python/ZODB/Transaction.py, 
>line 261, in commit
> AttributeError: (see above)
> 
> All other object-types (incuding dtml-methods) seem to work. I have put no
> restrictions on which files I want found, but restricting something to
> reduce the number of found objects helps.
> 
> I fiddled around somewhat and discovered that disabling
> subtransactions in ZCatalog does help (required for using
> ZSQL Methods and ZCatalog in the same transaction).

Yeah, that sux.  I think this is a known limitation.  I'm not sure.
 
> Now I use SQL-based LoginManager, but why on earth did it only
> fail on DTML-Documents? Is it because I have much more of them so
> only then ZCatalog's subtransactions come into play?

Probably because sub transactions are only commited after a certain
number of words or values are indexed, and your DTML documents were the
only things that held words.

-Michel

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and dates

[Uros Midic]

> If I do a search for both meta_type of "news" a date of 2000/11/16 in the
> goLiveDate field the results page returns "There was no data matching this
> Catalog query". Yet when I do a search for the meta_type of "news" alone I
> see that indeed the Catalog contains 2 entries with 2000/11/16 in the
> goLiveDate field.
>
> Why doesn't it find any entries when I search by date?

If goLiveDate is of DateTime type and is being indexed as a field index, try
using a range, for instance:





 someDate is _.DateTime('2000/11/16 00:00') or something like that.

p.s. It's very hard to catch a particular moment in the time continuum, but
it is easier with a time interval. :-)



___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

RE: [Zope] ZCatalog and dates

[James Sintz]

They are a date type.

The date index or goLiveDate index is now a keyword index instead of a text
index. However, when I try to find items of the meta_type "news" to catalog
I get an AttributeError with an error value of  __getitem__



What am I doning wrong??

Jamey

> -Original Message-
> From: Andy McKay [SMTP:[EMAIL PROTECTED]]
> Sent: Tuesday, November 28, 2000 3:10 PM
> To:   James Sintz; [EMAIL PROTECTED]
> Subject:  Re: [Zope] ZCatalog and dates
> 
> Are your dates actually of type date or string? If they are dates make
> sure
> your date index is a keyword index and that the input is also of type
> date.
> --
>   Andy McKay, Developer.
>   ActiveState.
> 
> - Original Message -
> From: "James Sintz" <[EMAIL PROTECTED]>
> To: <[EMAIL PROTECTED]>
> Sent: Tuesday, November 28, 2000 11:37 AM
> Subject: [Zope] ZCatalog and dates
> 
> 
> >
> > I have a ZCatalog set up to index a date field and a title field.
> >
> > When I do a search for meta_type "news" I get a tabular list back of all
> 18
> > cataloged items showing, among other things the date field (goLiveDate)
> and
> > the title field.
> >
> > If I do a search for both meta_type of "news" a date of 2000/11/16 in
> the
> > goLiveDate field the results page returns "There was no data matching
> this
> > Catalog query". Yet when I do a search for the meta_type of "news" alone
> I
> > see that indeed the Catalog contains 2 entries with 2000/11/16 in the
> > goLiveDate field.
> >
> > Why doesn't it find any entries when I search by date?
> >
> >
> > Jamey
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > ___
> > Zope maillist  -  [EMAIL PROTECTED]
> > http://lists.zope.org/mailman/listinfo/zope
> > **   No cross posts or HTML encoding!  **
> > (Related lists -
> >  http://lists.zope.org/mailman/listinfo/zope-announce
> >  http://lists.zope.org/mailman/listinfo/zope-dev )
> >

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and dates

[Andy McKay]

Are your dates actually of type date or string? If they are dates make sure
your date index is a keyword index and that the input is also of type date.
--
  Andy McKay, Developer.
  ActiveState.

- Original Message -
From: "James Sintz" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Tuesday, November 28, 2000 11:37 AM
Subject: [Zope] ZCatalog and dates


>
> I have a ZCatalog set up to index a date field and a title field.
>
> When I do a search for meta_type "news" I get a tabular list back of all
18
> cataloged items showing, among other things the date field (goLiveDate)
and
> the title field.
>
> If I do a search for both meta_type of "news" a date of 2000/11/16 in the
> goLiveDate field the results page returns "There was no data matching this
> Catalog query". Yet when I do a search for the meta_type of "news" alone I
> see that indeed the Catalog contains 2 entries with 2000/11/16 in the
> goLiveDate field.
>
> Why doesn't it find any entries when I search by date?
>
>
> Jamey
>
>
>
>
>
>
>
>
>
>
>
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: setting default to AND (not OR).

[Andy McKay]

Either a) fiddle with ZCatalog so it does or
b) add in an "and" in between search terms before the catalog a crude way
could be

_.string.join(_.string.split(_.str(query),' '), ' and ' )

And thanks for the holiday wishes but we had Thanksgiving a month ago in
Canada :)

--
  Andy McKay, Developer.
  ActiveState.

- Original Message -
From: "The Doctor What" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Wednesday, November 22, 2000 8:12 PM
Subject: Re: [Zope] ZCatalog: setting default to AND (not OR).


> I'm sorry.  I seem to have not explained myself very well.
>
> I want ZCatalog to use AND by default.  I think the extended
> behaviour, being able to AND and OR keywords together is fine.  I'm
> only concerned with the default behaviour (with *no* OR or AND
> keywords).
>
> When I first used zope.org's search engine, I couldn't figure out
> why if searched for "zcatalog siteaccess" it gave me (seemingly)
> every article on the site except the one(s) I wanted.
>
> It's because I expected AND to be the default (the union) of the two
> terms, not an OR list.  Frankly, I don't think OR is useful at all
> by it self.  It's usually with an AND that it's most useful.
>
> But that's just my humble opinion.  Oh wait, I don't do humbleso
> that's just my opinion. ;-)
>
> I wish everyone a happy holiday, whether it's Thanksgiving or
> "Laborday/Thanksgiving" (as NPR said Japan was having).
>
> Ciao!
>
> > > All good usability books (I'm going to just cite one: "Designing Web
> > > Usability: The Practice of Simplicity" by Jakob Nielsen
> > > http://www.useit.com/jakob/webusability/ ) say searches should be
> > > ANDed by default.
> > >
> > > ZCatalog is ORed by default.  How do I change this behavior?
> > >
> > > Ciao!
> > >
> > > --
> > > Room Service? Send up a larger room.
> > > -- Groucho Marx
> > >
> > > The Doctor What: Guru to the Gods
http://docwhat.gerf.org/
> > > [EMAIL PROTECTED]
KF6VNC
> > >
>
> --
> Now is the time for all good men to come to.
> -- Walt Kelly
>
> The Doctor What:  http://docwhat.gerf.org/
> [EMAIL PROTECTED]   KF6VNC
>
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: setting default to AND (not OR).

[The Doctor What]

I'm sorry.  I seem to have not explained myself very well.

I want ZCatalog to use AND by default.  I think the extended
behaviour, being able to AND and OR keywords together is fine.  I'm
only concerned with the default behaviour (with *no* OR or AND
keywords).

When I first used zope.org's search engine, I couldn't figure out
why if searched for "zcatalog siteaccess" it gave me (seemingly)
every article on the site except the one(s) I wanted.

It's because I expected AND to be the default (the union) of the two
terms, not an OR list.  Frankly, I don't think OR is useful at all
by it self.  It's usually with an AND that it's most useful.

But that's just my humble opinion.  Oh wait, I don't do humbleso
that's just my opinion. ;-)

I wish everyone a happy holiday, whether it's Thanksgiving or
"Laborday/Thanksgiving" (as NPR said Japan was having).

Ciao!

> > All good usability books (I'm going to just cite one: "Designing Web
> > Usability: The Practice of Simplicity" by Jakob Nielsen
> > http://www.useit.com/jakob/webusability/ ) say searches should be
> > ANDed by default.
> > 
> > ZCatalog is ORed by default.  How do I change this behavior?
> > 
> > Ciao!
> > 
> > -- 
> > Room Service? Send up a larger room.
> > -- Groucho Marx
> > 
> > The Doctor What: Guru to the Godshttp://docwhat.gerf.org/
> > [EMAIL PROTECTED]   KF6VNC
> > 

-- 
Now is the time for all good men to come to.
 -- Walt Kelly

The Doctor What:  http://docwhat.gerf.org/
[EMAIL PROTECTED]   KF6VNC

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog: setting default to AND (not OR).

[Andy McKay]

add in an and between terms eg: foo and bar
--
  Andy McKay, Developer.
  ActiveState.
- Original Message - 
From: "The Doctor What" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Wednesday, November 22, 2000 2:36 PM
Subject: [Zope] ZCatalog: setting default to AND (not OR).


> All good usability books (I'm going to just cite one: "Designing Web
> Usability: The Practice of Simplicity" by Jakob Nielsen
> http://www.useit.com/jakob/webusability/ ) say searches should be
> ANDed by default.
> 
> ZCatalog is ORed by default.  How do I change this behavior?
> 
> Ciao!
> 
> -- 
> Room Service? Send up a larger room.
> -- Groucho Marx
> 
> The Doctor What: Guru to the Godshttp://docwhat.gerf.org/
> [EMAIL PROTECTED]   KF6VNC
> 
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
> 


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and random key error

[Chris McDonough]

You will still see random key errors until Chris Petrilli puts his ZCatalog
fixes into the 2.2 branch (hopefully 2.2.3).

- Original Message -
From: "Bak @ kedai" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Thursday, November 09, 2000 11:56 PM
Subject: [Zope] ZCatalog and random key error


> hello all
> i'm about to venture into building a ZClass that will make use of
ZCatalog.
> is it 'safe' now?  i see that my zope2.2.2 now haven't encountered any
random
> key error yet.  what about others?
>
> also, will iterating through catalog index faster than iterating through
> ..say.. objectItems('meta-type')?
>
> looking for insight, advice, holes to avoid, etc
>
> thanks
>
> --
>
> http://www.kedai.com.my/kk
> http://www.kedai.com.my/eZine
>
> use Zope?  then you got HOPE!
>
>
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>
>


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and random key error

[Erik Enge]

On Fri, 10 Nov 2000, Bak @ kedai wrote:

> also, will iterating through catalog index faster than iterating through 
> ..say.. objectItems('meta-type')?

Much faster.  Much, much faster.  I tried an objectValues() on 2000 items
(in 3 different folder-type objects) and did various calculations on each
object, it took me 20 sec. to render the page.  On the same box the
Catalog return a search (much wider than the one on 2000 objects) on 15000
items (indexd, natuarally) within a second or two.

Hm.  I might be getting out on thin ice now, but I think it depends on
what you want to do with the objects you get back from your
iteration.  If you just are looking for simple attributes, the Catalog
should suffice, but it (the Catalog) might be slower if you are doing
something with the objects, since it has to fetch them first - and the
objectValues() returns the object.  I'm not sure about this, though, the
source code should help you a lot (I think objectValues() and
objectItems() and those alike are in lib/python/ObjectManager.py -
around line 350).

Come to think of it.  You did say objectItems(), and I've described the
objectValues().  I don't think you can compare the Catalog and
objectItems(), since objectItems() returns a list of (id,
subobject) tuples and thus gives you context - something the Catalog
cannot do (unless you hack it.. yeah Catalog wizards?).

I might be wrong though :)




___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog scalability?

[Wolfgang Strobl]

On 8 Nov 2000, 15:54  Júlio Dinis Silva wrote:

> Does anyone has a Catalog with more than 1500 items catalogued?
> If yes does anyone can do searches to this Catalog in a
> resonable time without freezing ZServer until it ends?

I did some experiments over the weekend, for example cataloging 
about ten thousand news messages into a catalog in one step.

Searching isn't the problem (this is quite fast) - cataloging is. This 
takes a lot of time and a lot of memory.

-- 
Wolfgang Strobl


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog scalability?

[Andy McKay]

Yes. Some queries especially ones with wildcards can take a lot more juice
than others. Try and keep your metadata small and your queries simple.

--
  Andy McKay, Developer.
  ActiveState.

- Original Message -
From: "Júlio Dinis Silva" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Wednesday, November 08, 2000 7:54 AM
Subject: [Zope] ZCatalog scalability?


> Does anyone has a Catalog with more than 1500 items catalogued?
> If yes does anyone can do searches to this Catalog in a
> resonable time without freezing ZServer until it ends?
>
> Thanks,
> Júlio Silva
> _
> Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.
>
> Share information about yourself, create your own public profile at
> http://profiles.msn.com.
>
>
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog question. boolean and wildcards.

[Dieter Maurer]

Hannes Grund writes:
 > When mixing boolean operators and wildcards '*' in queries within a Textindex
 > the 'AND' operator is replaced by 'OR'. 
 > Also, the possibility of constructing nested terms using brackets seems 
 > to be not supported (contrary to the chapter 'Searching and indexing' from the Zope 
 >book).
 > The setting is straightforward, several indexes (text-index, keyword-indexes)
 > are used indexing a about 5000 Zclasses.
 > Are there any patches to apply or what else can i do ?
Sorry, the patch I posted yesterday was not complete.

There have been too more problems:

  * if the wildcard expansion could not find matching words,
my patch created an empty subquery.
Such queries, unfortunately, can not be handled by
"UnTextIndex.evaluate".

The new patch, instead creates a new search term,
'no_expansion_for_'. Hopefully, this term is not
indexed and thus gives the same results as
the empty subquery.

Of cause, this is a hack.
The ZCatalog maintainer should find a better solution.

  * The ZCatalog, more precisely, "UnTextIndex.__getitem__"
still raises an "IndexError" exception, when one searches
for a term that is not indexed.
The result is "no data matches your query", even for
an "or" query.

I had already put a patch for this error in the collector
for Zope 2.2.1, but apparently, DC decided not to
look at it.


Please discard my patch from yesterday.
You may give the new patch (appended) a try.


Dieter

--- lib/python/SearchIndex/:GlobbingLexicon.py  Mon Aug 28 19:30:06 2000
+++ lib/python/SearchIndex/GlobbingLexicon.py   Sat Oct 14 11:54:24 2000
@@ -230,10 +230,22 @@
 if ( (self.multi_wc in w) or
 (self.single_wc in w) ):
 wids = self.get(w)
-for wid in wids:
-if words:
-words.append(Or)
-words.append(self._inverseLex[wid])
+#for wid in wids:
+#if words:
+#words.append(Or)
+#words.append(self._inverseLex[wid])
+   if wids:
+   words.append(map(self._inverseLex.get,wids))
+   else:
+   words.append('no_expansion_for_' + w)
+   # We do this here, as "UnTextIndex.evaluate"
+   #can not handle empty subqueries
+   # We use as expansion some expression that
+   #is likely not in the catalog but
+   #gives some indication to the problem
+   #in case "ResultList" does something
+   #with its "src" argument.
+   #This is a hack, though!
 else:
 words.append(w)
 
--- lib/python/SearchIndex/:UnTextIndex.py  Fri Oct 13 23:29:00 2000
+++ lib/python/SearchIndex/UnTextIndex.py   Sat Oct 14 11:57:38 2000
@@ -347,8 +347,10 @@
 if len(src) == 1:
 src=src[0]
 if src[:1]=='"' and src[-1:]=='"': return self[src]
-r = self._index.get(self.getLexicon(self._lexicon).get(src)[0],
-None)
+   codes= self.getLexicon(self._lexicon).get(src)
+   if codes:
+r = self._index.get(codes[0], None)
+   else: r= None # maybe do something, if len(codes) > 1
 if r is None: r = {}
 return ResultList(r, (src,), self)
 
@@ -490,11 +492,11 @@
 except IndexError: raise QueryError, "Malformed query"
 
 t=type(left)
-if t is ListType: left = evaluate(left, self)
+if t is ListType: left = self.evaluate(left)
 elif t is StringType: left=self[left]
 
 t=type(right)
-if t is ListType: right = evaluate(right, self)
+if t is ListType: right = self.evaluate(right)
 elif t is StringType: right=self[right]
 
 return (left, right)

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog problem!

[Kapil Thangavelu]

Kevin Howe wrote:
> 
> Hi, I have a Folderish class (FClass) and I am trying to write it so that
> when an FClass object is added in ZOPE, it will automatically create a
> ZCatalog object inside itself. I attempted to do this in the manner I had
> successfully done this before with other object types:
> 
> # handle represents the new object to be added with self,_setObject(id,
> handle)
> handle.manage_addZCatalog('CatName', 'CatTitle, None)

generally i reserve this type of stuff into the __init__ method of the
object... hence i don't clutter up my manage_add method with code thats
not directly relevant to its function.

anyways, such is mainly a style matter.. 

to answer your question...
here is a method i call from my __init__  method of a python product.


def setup_catalog(self):
''' setups catalog for searching news channels '''

id = 'news_catalog'
catalog = Products.ZCatalog.ZCatalog.ZCatalog(id)
self._setObject(id, catalog)

# setup indexes and such
self.news_catalog.manage_addColumn('description')

self.news_catalog.manage_addColumn('Date')
self.news_catalog.manage_addIndex('Date', 'FieldIndex')

self.news_catalog.manage_addColumn('ChannelName')
self.news_catalog.manage_addIndex('ChannelName', 'FieldIndex')

self.news_catalog.manage_addColumn('link')
self.news_catalog.manage_addIndex('link', 'FieldIndex')

self.news_catalog.manage_addIndex('SearchableText', 'TextIndex')

# get rid of defaults
self.news_catalog.manage_delColumns(['bobobase_modification_time',
'summary'])
   
self.news_catalog.manage_delIndexes(['bobobase_modification_time',   
  'PrincipiaSearchSource'])

return 1


you can also do it from a dtml-method.

i'd don't have the exact syntax handy...

but i'll take a stab at it


> But this doesn't work. I then did some testing with DTML using the following
> method inside a Folder:
> 
> 
> 
> 
> 
> The new Folder and new Image were both added with no problems. This was what
> I expected since the "manage_addFolder" and "manage_addZCatalog" methods are
> inherited from OFS.Folder.Folder. The method "manage_addZCatalog" however,
> doesn't seem to have been inherited for some reason.

the Folder, Image, Document, Method manage_adds are globally available
methods... to add a product you need

see the howto on adding zclasses programamtically for examples/more
info.

if i remember correctly the syntax is something like





but i would check that...

Cheers

Kapil

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog in 2.2.x

[Sebastien Yapo]

And how did you fix it ? I had the same kind of problem with ZCatalog an 
CatalogAware ...

Terry Kerr wrote:
any word...however, i fixed the problem thanks. to Dieter Maurer

Chris McDonough wrote:

 > What is the actual word you're looking for? 'blah'?
 >
 > Terry Kerr wrote:
 > >
 > > Hi,
 > >
 > > I can rebuild indexes and recatalog etc, it doesn't change anything. I
 > > don't see how it could change anything...it is a brand new zcatalog
 > > instance. Simply, the ZCatalog is created, the indexes are created, and
 > > the zcatalog is initially empty. Then I create one CatalogAware object,
 > > and it catalogs itself. I can see this object in the catalog, and I can
 > > see the textindexed words in the vocabulary. But when I search for a word
 > > in the vocabulary using a textindex (which obviously belongs to the only
 > > object in the catalog), I get an empty result. The only way I can get a
 > > non empty result it to search for the id of the object or one other
 > > property I have which are indexed as field indexes.
 > >
 > > The product I am working on is Etailer. If anyone is kind enough to 
take a
 > > look, you can download it at http://e-tailer.adriot.net/download.html. 
The
 > > function that does the search is searchFor(), at line 419 of Etailer.py.
 > > Creating an instance of etailer in 2.1.6 works fine, but creating an
 > > instance of etailer in 2.2.x and the catalog doesn't work.
 > >
 > > i am very confused ;-).
 > >
 > > Chris McDonough wrote:
 > >
 > > > This is not right, obviously. Can you rebuild your indexes (by deleting
 > > > and reentering them), recatalog, and see if the problem goes away? We
 > > > haven't had any colluding reports of something like this, and I'm using
 > > > the 2.2.1 catalog with textindexes successfully on a customer project.
 > > >
 > > > > -Original Message-
 > > > > From: Terry Kerr [mailto:[EMAIL PROTECTED]]
 > > > > Sent: Wednesday, August 30, 2000 3:27 AM
 > > > > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
 > > > > Subject: Re: [Zope] ZCatalog in 2.2.x
 > > > >
 > > > >
 > > > > I am not searching from a form in that sense...I am talking
 > > > > about at the python
 > > > > level.
 > > > >
 > > > > I have self.searchResults({'description':'blah'}) where
 > > > > description is meta
 > > > > data in the catalog and a textindex and 'blah' is in the
 > > > > Vocabulary. This
 > > > > previously worked in zope2.1.6. Now in 2.2.0 and 2.2.1 it
 > > > > doesn't work. The
 > > > > only thing way I can get searchResults() to return a non
 > > > > empty list is to search
 > > > > using a Field index and a valid string to match with.
 > > > >
 > > > >
 > > > > [EMAIL PROTECTED] wrote:
 > > > >
 > > > > > Terry Kerr <[EMAIL PROTECTED]> said:
 > > > > >
 > > > > > > Hi,
 > > > > > >
 > > > > > > I have a python product which uses a ZCatalog and CatalogAware
 > > > > > > products. The catalog has a number of Field and
 > > > > TextIndex indexes.
 > > > > > > Search the ZCatalog has worked fine when my product is
 > > > > instantiated in
 > > > > > > 2.1.6. But now when I instantiate my product in 2.2.0 or 2.2.1,
 > > > > > > searching only works when I search using a field index.
 > > > > Searching using
 > > > > > > a textindex always results in an empty result, even tho I
 > > > > can see the
 > > > > > > words I am searching for in the Vocabulary! Any
 > > > > suggestions? I have
 > > > > > > tried reindexing, completey recreating the Zcatalog,
 > > > > recreating the
 > > > > > > Vobulary and reindexing, and nothing has changed the
 > > > > behaviour. Are
 > > > > > > textindexes simply broken?
 > > > > > >
 > > > > > >
 > > > > > > ZCatalog seems very volatile to me! Every time I have
 > > > > upgraded way back
 > > > > > > from zope2.1.3, something in zcatalog has changed and
 > > > > boken my product.
 > > > > > > So far I have managed to find fixes, but this one has got
 > > > > me baffled!!
 > > > > > >
 > &

Re: [Zope] ZCatalog in 2.2.x

[Terry Kerr]

any word...however, i fixed the problem thanks. to Dieter Maurer

Chris McDonough wrote:

> What is the actual word you're looking for?  'blah'?
>
> Terry Kerr wrote:
> >
> > Hi,
> >
> > I can rebuild indexes and recatalog etc, it doesn't change anything.  I
> > don't see how it could change anything...it is a brand new zcatalog
> > instance.  Simply, the ZCatalog is created, the indexes are created, and
> > the zcatalog is initially empty.  Then I create one CatalogAware object,
> > and it catalogs itself.  I can see this object in the catalog, and I can
> > see the textindexed words in the vocabulary.  But when I search for a word
> > in the vocabulary using a textindex (which obviously belongs to the only
> > object in the catalog), I get an empty result.  The only way I can get a
> > non empty result it to search for the id of the object or one other
> > property I have which are indexed as field indexes.
> >
> > The product I am working on is Etailer.  If anyone is kind enough to take a
> > look, you can download it at http://e-tailer.adriot.net/download.html.  The
> > function that does the search is searchFor(), at line 419 of Etailer.py.
> > Creating an instance of etailer in 2.1.6 works fine, but creating an
> > instance of etailer in 2.2.x and the catalog doesn't work.
> >
> > i am very confused ;-).
> >
> > Chris McDonough wrote:
> >
> > > This is not right, obviously.  Can you rebuild your indexes (by deleting
> > > and reentering them), recatalog, and see if the problem goes away?  We
> > > haven't had any colluding reports of something like this, and I'm using
> > > the 2.2.1 catalog with textindexes successfully on a customer project.
> > >
> > > > -Original Message-
> > > > From: Terry Kerr [mailto:[EMAIL PROTECTED]]
> > > > Sent: Wednesday, August 30, 2000 3:27 AM
> > > > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> > > > Subject: Re: [Zope] ZCatalog in 2.2.x
> > > >
> > > >
> > > > I am not searching from a form in that sense...I am talking
> > > > about at the python
> > > > level.
> > > >
> > > > I have self.searchResults({'description':'blah'})  where
> > > > description is meta
> > > > data in the catalog and a textindex and 'blah' is in the
> > > > Vocabulary.  This
> > > > previously worked in zope2.1.6.  Now in 2.2.0 and 2.2.1 it
> > > > doesn't work.  The
> > > > only thing way I can get searchResults() to return a non
> > > > empty list is to search
> > > > using a Field index and a valid string to match with.
> > > >
> > > >
> > > > [EMAIL PROTECTED] wrote:
> > > >
> > > > > Terry Kerr <[EMAIL PROTECTED]> said:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have a python product which uses a ZCatalog and CatalogAware
> > > > > > products.  The catalog has a number of Field and
> > > > TextIndex indexes.
> > > > > > Search the ZCatalog has worked fine when my product is
> > > > instantiated in
> > > > > > 2.1.6.  But now when I instantiate my product in 2.2.0 or 2.2.1,
> > > > > > searching only works when I search using a field index.
> > > > Searching using
> > > > > > a textindex always results in an empty result, even tho I
> > > > can see the
> > > > > > words I am searching for in the Vocabulary!  Any
> > > > suggestions?  I have
> > > > > > tried reindexing, completey recreating the Zcatalog,
> > > > recreating the
> > > > > > Vobulary and reindexing, and nothing has changed the
> > > > behaviour.  Are
> > > > > > textindexes simply broken?
> > > > > >
> > > > > >
> > > > > > ZCatalog seems very volatile to me!  Every time I have
> > > > upgraded way back
> > > > > > from zope2.1.3, something in zcatalog has changed and
> > > > boken my product.
> > > > > > So far I have managed to find fixes, but this one has got
> > > > me baffled!!
> > > > > >
> > > > >
> > > > > My experience has been that if field indexes are
> > > > unspecified, the catalog
> > > > > will only match whe

Re: [Zope] ZCatalog in 2.2.x

[Chris McDonough]

What is the actual word you're looking for?  'blah'?

Terry Kerr wrote:
> 
> Hi,
> 
> I can rebuild indexes and recatalog etc, it doesn't change anything.  I
> don't see how it could change anything...it is a brand new zcatalog
> instance.  Simply, the ZCatalog is created, the indexes are created, and
> the zcatalog is initially empty.  Then I create one CatalogAware object,
> and it catalogs itself.  I can see this object in the catalog, and I can
> see the textindexed words in the vocabulary.  But when I search for a word
> in the vocabulary using a textindex (which obviously belongs to the only
> object in the catalog), I get an empty result.  The only way I can get a
> non empty result it to search for the id of the object or one other
> property I have which are indexed as field indexes.
> 
> The product I am working on is Etailer.  If anyone is kind enough to take a
> look, you can download it at http://e-tailer.adriot.net/download.html.  The
> function that does the search is searchFor(), at line 419 of Etailer.py.
> Creating an instance of etailer in 2.1.6 works fine, but creating an
> instance of etailer in 2.2.x and the catalog doesn't work.
> 
> i am very confused ;-).
> 
> Chris McDonough wrote:
> 
> > This is not right, obviously.  Can you rebuild your indexes (by deleting
> > and reentering them), recatalog, and see if the problem goes away?  We
> > haven't had any colluding reports of something like this, and I'm using
> > the 2.2.1 catalog with textindexes successfully on a customer project.
> >
> > > -Original Message-
> > > From: Terry Kerr [mailto:[EMAIL PROTECTED]]
> > > Sent: Wednesday, August 30, 2000 3:27 AM
> > > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> > > Subject: Re: [Zope] ZCatalog in 2.2.x
> > >
> > >
> > > I am not searching from a form in that sense...I am talking
> > > about at the python
> > > level.
> > >
> > > I have self.searchResults({'description':'blah'})  where
> > > description is meta
> > > data in the catalog and a textindex and 'blah' is in the
> > > Vocabulary.  This
> > > previously worked in zope2.1.6.  Now in 2.2.0 and 2.2.1 it
> > > doesn't work.  The
> > > only thing way I can get searchResults() to return a non
> > > empty list is to search
> > > using a Field index and a valid string to match with.
> > >
> > >
> > > [EMAIL PROTECTED] wrote:
> > >
> > > > Terry Kerr <[EMAIL PROTECTED]> said:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have a python product which uses a ZCatalog and CatalogAware
> > > > > products.  The catalog has a number of Field and
> > > TextIndex indexes.
> > > > > Search the ZCatalog has worked fine when my product is
> > > instantiated in
> > > > > 2.1.6.  But now when I instantiate my product in 2.2.0 or 2.2.1,
> > > > > searching only works when I search using a field index.
> > > Searching using
> > > > > a textindex always results in an empty result, even tho I
> > > can see the
> > > > > words I am searching for in the Vocabulary!  Any
> > > suggestions?  I have
> > > > > tried reindexing, completey recreating the Zcatalog,
> > > recreating the
> > > > > Vobulary and reindexing, and nothing has changed the
> > > behaviour.  Are
> > > > > textindexes simply broken?
> > > > >
> > > > >
> > > > > ZCatalog seems very volatile to me!  Every time I have
> > > upgraded way back
> > > > > from zope2.1.3, something in zcatalog has changed and
> > > boken my product.
> > > > > So far I have managed to find fixes, but this one has got
> > > me baffled!!
> > > > >
> > > >
> > > > My experience has been that if field indexes are
> > > unspecified, the catalog
> > > > will only match where those fields are empty.  In other
> > > words, by not
> > > > specifing anything for the field indexes, you are actually
> > > specifying that
> > > > for a match, they need to be empty.
> > > > To get around this, I put another method between the form
> > > for entering what
> > > > you wish to search for, and the method that actually
> > > searches the form.  This
> > > > method looks at the field indexes, and if nothing has b

RE: [Zope] ZCatalog Queries...

[Albert Langer]

[Kapil]
I think that zcatalog is great for simple property indexing but it has
some signifigant drawbacks for its most common use, which is mass text
indexing, and searching. As object an catalog, its fine, But i think as
a mass text indexing/searching machinery it bites. I think Zope could
really use either a companion to zcatalog for text searching or a
replacement. If i'd like to index 50M i'd like not to have my search
take unreasonable long, give erroneous results, thrash my machine
because i choose to index it at once, yet these are all things that i've
experienced with zcatalog since i started using zope. for sure its
gotten much better, but it still can't handle my use cases. So i've
accepted that ZCatalog isn't a site wide nor a scalable tool.

[Albert]
Render up to an RDBMS or ORDBMS that which requires indexing and searching.
Render up to ZODB that which requires pointer structures.

Substring searches can be quite efficient with an RDBMS. Use for words in
conjunction
with pointers to sentence/paragraph/section/document/category structure
maintained in ZODB.

See:
http://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=Ec&topic_id=11&to
pic=OpenACS


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog in 2.2.x

[Dieter Maurer]

Terry Kerr writes:
 >  only field searches yield non empty query results 
 > The product I am working on is Etailer.
I downloaded Etailer and looked into the problem.

Indeed, I could reproduce your observation.

I analysed the problem and now do understand what went
wrong here in my installation:

  I have 2 vocabularies, both named 'Vocabulary'.
  One is in my root, the other is in the Etailer "Catalog".

  When objects are indexed, this happens in the context
  of "Etailer/Catalog". Therefore, "Etailer/Catalog/Vocabulary"
  is used for this purpose.

  When objects are searched for, this happens in the context
  of "Etailer". Therefore, "/Vocabulary" is used for
  word lookup. It does know nothing about the words
  used in my Etailer products.

Maybe, in your installation things are similar.
At least, Etailer is very volatile when there is
another object called 'Vocabulary' above it.
This results from line 159 of "Etailer.py":
searchResults = ZCatalog.searchResults

You should provide a true, redirecting definition for
"searchResults":

def searchResults(self,REQUEST=None, used=None, **kw):
  " "
  return apply(self.Catalog.searchResults,(REQUEST,used),kw)

to make it more robust.



Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog in 2.2.x

[Terry Kerr]

Hi,

I can rebuild indexes and recatalog etc, it doesn't change anything.  I
don't see how it could change anything...it is a brand new zcatalog
instance.  Simply, the ZCatalog is created, the indexes are created, and
the zcatalog is initially empty.  Then I create one CatalogAware object,
and it catalogs itself.  I can see this object in the catalog, and I can
see the textindexed words in the vocabulary.  But when I search for a word
in the vocabulary using a textindex (which obviously belongs to the only
object in the catalog), I get an empty result.  The only way I can get a
non empty result it to search for the id of the object or one other
property I have which are indexed as field indexes.

The product I am working on is Etailer.  If anyone is kind enough to take a
look, you can download it at http://e-tailer.adriot.net/download.html.  The
function that does the search is searchFor(), at line 419 of Etailer.py.
Creating an instance of etailer in 2.1.6 works fine, but creating an
instance of etailer in 2.2.x and the catalog doesn't work.


i am very confused ;-).


Chris McDonough wrote:

> This is not right, obviously.  Can you rebuild your indexes (by deleting
> and reentering them), recatalog, and see if the problem goes away?  We
> haven't had any colluding reports of something like this, and I'm using
> the 2.2.1 catalog with textindexes successfully on a customer project.
>
> > -Original Message-
> > From: Terry Kerr [mailto:[EMAIL PROTECTED]]
> > Sent: Wednesday, August 30, 2000 3:27 AM
> > To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> > Subject: Re: [Zope] ZCatalog in 2.2.x
> >
> >
> > I am not searching from a form in that sense...I am talking
> > about at the python
> > level.
> >
> > I have self.searchResults({'description':'blah'})  where
> > description is meta
> > data in the catalog and a textindex and 'blah' is in the
> > Vocabulary.  This
> > previously worked in zope2.1.6.  Now in 2.2.0 and 2.2.1 it
> > doesn't work.  The
> > only thing way I can get searchResults() to return a non
> > empty list is to search
> > using a Field index and a valid string to match with.
> >
> >
> > [EMAIL PROTECTED] wrote:
> >
> > > Terry Kerr <[EMAIL PROTECTED]> said:
> > >
> > > > Hi,
> > > >
> > > > I have a python product which uses a ZCatalog and CatalogAware
> > > > products.  The catalog has a number of Field and
> > TextIndex indexes.
> > > > Search the ZCatalog has worked fine when my product is
> > instantiated in
> > > > 2.1.6.  But now when I instantiate my product in 2.2.0 or 2.2.1,
> > > > searching only works when I search using a field index.
> > Searching using
> > > > a textindex always results in an empty result, even tho I
> > can see the
> > > > words I am searching for in the Vocabulary!  Any
> > suggestions?  I have
> > > > tried reindexing, completey recreating the Zcatalog,
> > recreating the
> > > > Vobulary and reindexing, and nothing has changed the
> > behaviour.  Are
> > > > textindexes simply broken?
> > > >
> > > >
> > > > ZCatalog seems very volatile to me!  Every time I have
> > upgraded way back
> > > > from zope2.1.3, something in zcatalog has changed and
> > boken my product.
> > > > So far I have managed to find fixes, but this one has got
> > me baffled!!
> > > >
> > >
> > > My experience has been that if field indexes are
> > unspecified, the catalog
> > > will only match where those fields are empty.  In other
> > words, by not
> > > specifing anything for the field indexes, you are actually
> > specifying that
> > > for a match, they need to be empty.
> > > To get around this, I put another method between the form
> > for entering what
> > > you wish to search for, and the method that actually
> > searches the form.  This
> > > method looks at the field indexes, and if nothing has been
> > specified, it
> > > doesn't pass them on to the method that does the actual search.
> > > An alternative to this, is to use names for the field
> > indexes that aren't in
> > > the catalog index, and then at the start of the search
> > method, check those
> > > names, and if they have a value associated with them, add
> > to REQUEST what the
> > > names in the index are with the appropriate values.
> > >
> > > There may be a better way, I'm still pr

Re: [Zope] ZCatalog and foreign characters

[Aleksander Salwa]

> 
> For now, I made hack in Spliter.c - I wrote my own replacements for
> isalpha and isalnum functions.
> 

Oops, my solution doesn't work. Probably it needs more hacking in python
code
So, how to do it under FreeBSD ?

[EMAIL PROTECTED]

/--\
| `long long long' is too long for GCC |
\--/


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and foreign characters

[Aleksander Salwa]

On Wed, 30 Aug 2000, Radim Gelner wrote:

> On Tue, Aug 29, 2000 at 11:57:08PM +0200, Dieter Maurer wrote:
> > 
> > If you work with a fixed locale, you can use the "-L" switch
> > to inform Zope about your locale. Then the splitter should
> > work correctly (for your locale).
> 
> Yes, that helped. I've called Zope with -L cs_CZ switch, rebuilt the
> catalog and now searching works as it should.
> 

Great, but what about FreeBSD ? I work on Linux at home, but few days ago
someone decided that our production server will be FreeBSD. What is the
right locale string format on FreeBSD (version 3.5) ?
I've tried "pl_PL", "iso-8859-2", "pl_PL.iso-8859-2" - no success... :(

For now, I made hack in Spliter.c - I wrote my own replacements for
isalpha and isalnum functions.

[EMAIL PROTECTED]

/--\
| `long long long' is too long for GCC |
\--/


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

RE: [Zope] ZCatalog Queries...

[Andy Dawkins]

>
>  > > > It's be nice if ZCatalog had a good general purpose
> > > > interface, and was a
> > > > bit more robust.
> > > > (the BTree implementation which has been mentioned a few
> > times springs
> > > > to mind here ;-)
> > >
> > > Can you be more specific?
> >
> > Andy can fill you in on the specifics.
>
> OK...

Specifics, hmmm OK

Machine 1: (2gb free hard drive space, 64Mb Memory, 128Mb swap, AMD Athlon
600, Linux Red Hat 6.1)

I created a new Zope Instance and installed ZMailIn porduct on to it.
(Catalog Aware)
I created a ZCatalog.
I sent 30,000 mail messages from [EMAIL PROTECTED] to the ZMailIn product.
These messages were catalog aware so they indexed themselves, one-by-one in
to the catalog.  Indexing the entire message body.

I went home.  When I came back the next day the machine had successfully
completed 5,000 messages, eaten all the memory, eaten 99% of the swap and
was chugging away a 1 mail message per 5 seconds.
Additionally the Data.fs was 1.5Gb

I decided this was going to take forever and ended it all.
(FYI: After packing the database it strank to 30Mb)

I started again from nothing.
This time i didn't create a ZCatalog.
I ran the import routines and 30,000 documents were successfully imported in
to the ZODB through ZMailIn in about 2 hours.
I created a catalog and manually added all items of type "ZMailMessage".
After about an hour of serious crunching.  All memory eaten, all swap eaten.
It reports that we are out of hard drive space.
I tried again after with various sub transactions sizes (ie. 1, 5000,
1000, 200) but no change, except on the amount of time before it blows up.

After a couple of messages to [EMAIL PROTECTED] and [EMAIL PROTECTED] we decide
that it may be worth another try without transactions, i.e. Replacing the
ZODB3 with The Berkeley Database.

Machine 2: (4.5gb free hard drive space, 256Mb Memory, 256Mb swap, Dual
PentiumIII 450, Linux Red Hat 6.2)

I created a new Zope Instance and installed ZMailIn porduct on to it.
(Catalog Aware)
I plugged in the BerkeleyDB
I imported all 30,000 mail messages.
I selected all ZMailMessage instance for cataloging and let it go.
About half a hour later the thing crashes with the error "file too large"
and corupts my Berkeley Database.
I try several sub transaction sizes (1, 5000, 1000, 200) to no avail.

The only time I ever got it to work is if I removed the body of the mail
message from the catalogs indexes.  Which is the most important thing to be
indexed.

After tearing my hair out, i have put it aside and am working on something
else.

(I hope that is what Chris meant by Specifics)

>
> >
> > >  What's insufficient about the current
> > > implementation?
> >
> > It doesn't scale well, especially for things where you have
> > lots of new
> > data arriving (this is the BTree problem, I think...)
>
> Yes, we're still working on a "broadtree" implementation that may allay
> some of these problems, although I don't have an ETA.
>
> >
> > It has no published and well defined query syntax (there's
> > patches here,
> > bits there, but no definitive document on how to use it, how to batch
> > with it, how to perform complex and structured queries, particularly
> > with TextIndex'es)
>
> Hopefully, the Zope book will make it more clear from a user
> perspective, and sometime in the very distant future I will be writing a
> chapter in the developer's guide about the catalog.  I agree that the
> ZCatalog wrapper should probably wrap more of the underlying catalog's
> methods, but these need to be rationalized, defined and then documented
> in the API docs.  This is something for dev.zope.org, probably.
>
> >
> > Don't get me wrong, it is very cool, but only kindof 70% there :S
> > (and I get the impression that doing the remaining 30% properly would
> > require a rewrite...)
> >
> > As an example, we've been trying to do Zope-based versions of the
> > mailing list archives for a coupla months now and the Catalog keeps
> > exploding in different ways (huge resource consumption, even for only
> > 30K messages or so, no matter what storage is used)
>
> Yes.  Tweaking and the broadtree stuff should make this a little better.
>
> >
> > Then there's the ubiquitous 'KeyError's and other associated
> > weirdness,
> > all of which leaves me feeling a lot less than totally
> > confident in the
> > Catalog ;-)
>
> These are independent of the coupling problem in the btree, and I'm
> trying to vanquish them now.  This is why the annoying logging code was
> added to the catalog.  It seems to be related to the TextIndex
> implementation, but I'm still trying to pin it down.
>


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog Queries...

[Kapil Thangavelu]

I saw this email with some interest since i had an offlist conversation
with some of the people at nipltd regarding a mailing list project, but
i chose to do it outside of zope because of lingering doubts about
zope's suitability(more ZCatalog) for the task.


Chris Withers wrote:
> 
> Chris McDonough wrote:
> >
> > > It's be nice if ZCatalog had a good general purpose
> > > interface, and was a
> > > bit more robust.
> > > (the BTree implementation which has been mentioned a few times springs
> > > to mind here ;-)
> >
> > Can you be more specific?
> 
> Andy can fill you in on the specifics.
> 
> >  What's insufficient about the current
> > implementation?
> 
> It doesn't scale well, especially for things where you have lots of new
> data arriving (this is the BTree problem, I think...)
> 
> It has no published and well defined query syntax (there's patches here,
> bits there, but no definitive document on how to use it, how to batch
> with it, how to perform complex and structured queries, particularly
> with TextIndex'es)
> 
> Don't get me wrong, it is very cool, but only kindof 70% there :S
> (and I get the impression that doing the remaining 30% properly would
> require a rewrite...)

more of the same...
 
I think that zcatalog is great for simple property indexing but it has
some signifigant drawbacks for its most common use, which is mass text
indexing, and searching. As object an catalog, its fine, But i think as
a mass text indexing/searching machinery it bites. I think Zope could
really use either a companion to zcatalog for text searching or a
replacement. If i'd like to index 50M i'd like not to have my search
take unreasonable long, give erroneous results, thrash my machine
because i choose to index it at once, yet these are all things that i've
experienced with zcatalog since i started using zope. for sure its
gotten much better, but it still can't handle my use cases. So i've
accepted that ZCatalog isn't a site wide nor a scalable tool.

ok so the question(at least to me) than becomes what to implement. i'm
not totally sure... i've taken a look/leaning towards Evolution's text
indexing (libibex), 

i consulted some of the evolution developers and this is the response i
got 8/5/00(us)

>>
You probably dont want to use libibex as it is now - all indexes
are stored in memory for example, and it uses a lot of memory
as well (uses gtree's).  It will handle 100MB of mail fine enough
though on modern hardware, if you ahve the spare memory.

At some point we are going to change the backend to use a disk
based storage, using the search engine used in nautilus (can't
remember what its called), although we will probably make a wrapper
so the api should remain.
>>

which takes back to square one. i am convinced of the  need for
full-text reliable search on a site or a part of site. but i'm unsure of
what the best way to do it would be.
 
my2cents

kapil

> As an example, we've been trying to do Zope-based versions of the
> mailing list archives for a coupla months now and the Catalog keeps
> exploding in different ways (huge resource consumption, even for only
> 30K messages or so, no matter what storage is used)
> 
> Then there's the ubiquitous 'KeyError's and other associated weirdness,
> all of which leaves me feeling a lot less than totally confident in the
> Catalog ;-)
> 
> comments very welcome,
> 
> Chris
> 
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog in 2.2.x

[Dieter Maurer]

Terry Kerr writes:
 >  But now when I instantiate my product in 2.2.0 or 2.2.1,
 > searching only works when I search using a field index.  Searching using
 > a textindex always results in an empty result 
I use ZCatalog text indexes under 2.2.1b1 without problems
(with a globbing lexicon!).

I had to fix some glitches (search for "ZCatalog glitches"
and "Globbing Lexicon bug" in the Collector).
However, none of my fixes had to do with your problem
description.


Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

RE: [Zope] ZCatalog Queries...

[Chris McDonough]

 > > > It's be nice if ZCatalog had a good general purpose
> > > interface, and was a
> > > bit more robust.
> > > (the BTree implementation which has been mentioned a few 
> times springs
> > > to mind here ;-)
> > 
> > Can you be more specific?
> 
> Andy can fill you in on the specifics.

OK...

> 
> >  What's insufficient about the current
> > implementation?
> 
> It doesn't scale well, especially for things where you have 
> lots of new
> data arriving (this is the BTree problem, I think...)

Yes, we're still working on a "broadtree" implementation that may allay
some of these problems, although I don't have an ETA.

> 
> It has no published and well defined query syntax (there's 
> patches here,
> bits there, but no definitive document on how to use it, how to batch
> with it, how to perform complex and structured queries, particularly
> with TextIndex'es)

Hopefully, the Zope book will make it more clear from a user
perspective, and sometime in the very distant future I will be writing a
chapter in the developer's guide about the catalog.  I agree that the
ZCatalog wrapper should probably wrap more of the underlying catalog's
methods, but these need to be rationalized, defined and then documented
in the API docs.  This is something for dev.zope.org, probably.

> 
> Don't get me wrong, it is very cool, but only kindof 70% there :S
> (and I get the impression that doing the remaining 30% properly would
> require a rewrite...)
> 
> As an example, we've been trying to do Zope-based versions of the
> mailing list archives for a coupla months now and the Catalog keeps
> exploding in different ways (huge resource consumption, even for only
> 30K messages or so, no matter what storage is used)

Yes.  Tweaking and the broadtree stuff should make this a little better.

> 
> Then there's the ubiquitous 'KeyError's and other associated 
> weirdness,
> all of which leaves me feeling a lot less than totally 
> confident in the
> Catalog ;-)

These are independent of the coupling problem in the btree, and I'm
trying to vanquish them now.  This is why the annoying logging code was
added to the catalog.  It seems to be related to the TextIndex
implementation, but I'm still trying to pin it down.

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

RE: [Zope] ZCatalog in 2.2.x

[Chris McDonough]

> It's be nice if ZCatalog had a good general purpose 
> interface, and was a
> bit more robust.
> (the BTree implementation which has been mentioned a few times springs
> to mind here ;-)

Can you be more specific?  What's insufficient about the current
implementation?

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

RE: [Zope] ZCatalog in 2.2.x

[Chris McDonough]

This is not right, obviously.  Can you rebuild your indexes (by deleting
and reentering them), recatalog, and see if the problem goes away?  We
haven't had any colluding reports of something like this, and I'm using
the 2.2.1 catalog with textindexes successfully on a customer project.  

> -Original Message-
> From: Terry Kerr [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, August 30, 2000 3:27 AM
> To: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> Subject: Re: [Zope] ZCatalog in 2.2.x
> 
> 
> I am not searching from a form in that sense...I am talking 
> about at the python
> level.
> 
> I have self.searchResults({'description':'blah'})  where 
> description is meta
> data in the catalog and a textindex and 'blah' is in the 
> Vocabulary.  This
> previously worked in zope2.1.6.  Now in 2.2.0 and 2.2.1 it 
> doesn't work.  The
> only thing way I can get searchResults() to return a non 
> empty list is to search
> using a Field index and a valid string to match with.
> 
> 
> [EMAIL PROTECTED] wrote:
> 
> > Terry Kerr <[EMAIL PROTECTED]> said:
> >
> > > Hi,
> > >
> > > I have a python product which uses a ZCatalog and CatalogAware
> > > products.  The catalog has a number of Field and 
> TextIndex indexes.
> > > Search the ZCatalog has worked fine when my product is 
> instantiated in
> > > 2.1.6.  But now when I instantiate my product in 2.2.0 or 2.2.1,
> > > searching only works when I search using a field index.  
> Searching using
> > > a textindex always results in an empty result, even tho I 
> can see the
> > > words I am searching for in the Vocabulary!  Any 
> suggestions?  I have
> > > tried reindexing, completey recreating the Zcatalog, 
> recreating the
> > > Vobulary and reindexing, and nothing has changed the 
> behaviour.  Are
> > > textindexes simply broken?
> > >
> > >
> > > ZCatalog seems very volatile to me!  Every time I have 
> upgraded way back
> > > from zope2.1.3, something in zcatalog has changed and 
> boken my product.
> > > So far I have managed to find fixes, but this one has got 
> me baffled!!
> > >
> >
> > My experience has been that if field indexes are 
> unspecified, the catalog
> > will only match where those fields are empty.  In other 
> words, by not
> > specifing anything for the field indexes, you are actually 
> specifying that
> > for a match, they need to be empty.
> > To get around this, I put another method between the form 
> for entering what
> > you wish to search for, and the method that actually 
> searches the form.  This
> > method looks at the field indexes, and if nothing has been 
> specified, it
> > doesn't pass them on to the method that does the actual search.
> > An alternative to this, is to use names for the field 
> indexes that aren't in
> > the catalog index, and then at the start of the search 
> method, check those
> > names, and if they have a value associated with them, add 
> to REQUEST what the
> > names in the index are with the appropriate values.
> >
> > There may be a better way, I'm still pretty new to Zope and 
> DTML sort of
> > stuff.
> >
> > Peter Armstrong
> 
> --
> Terry Kerr ([EMAIL PROTECTED])
> Adroit Internet Solutions Pty Ltd (www.adroit.net)
> Phone:   +613 9563 4461
> Fax: +613 9563 3856
> Mobile:  +61 414 938 124
> ICQ: 79303381
> 
> 
> 
> 
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
> 

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog in 2.2.x

[Chris Withers]

Terry Kerr wrote:
> ZCatalog seems very volatile to me!  Every time I have upgraded way back
> from zope2.1.3, something in zcatalog has changed and boken my product.
> So far I have managed to find fixes, but this one has got me baffled!!

Can I just say a big "me too!" here.

At the EuroZope meeting in Paris, quite a few people mentioned this as
well.

It's be nice if ZCatalog had a good general purpose interface, and was a
bit more robust.
(the BTree implementation which has been mentioned a few times springs
to mind here ;-)

cheers,

Chris

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog in 2.2.x

[Terry Kerr]

I am not searching from a form in that sense...I am talking about at the python
level.

I have self.searchResults({'description':'blah'})  where description is meta
data in the catalog and a textindex and 'blah' is in the Vocabulary.  This
previously worked in zope2.1.6.  Now in 2.2.0 and 2.2.1 it doesn't work.  The
only thing way I can get searchResults() to return a non empty list is to search
using a Field index and a valid string to match with.


[EMAIL PROTECTED] wrote:

> Terry Kerr <[EMAIL PROTECTED]> said:
>
> > Hi,
> >
> > I have a python product which uses a ZCatalog and CatalogAware
> > products.  The catalog has a number of Field and TextIndex indexes.
> > Search the ZCatalog has worked fine when my product is instantiated in
> > 2.1.6.  But now when I instantiate my product in 2.2.0 or 2.2.1,
> > searching only works when I search using a field index.  Searching using
> > a textindex always results in an empty result, even tho I can see the
> > words I am searching for in the Vocabulary!  Any suggestions?  I have
> > tried reindexing, completey recreating the Zcatalog, recreating the
> > Vobulary and reindexing, and nothing has changed the behaviour.  Are
> > textindexes simply broken?
> >
> >
> > ZCatalog seems very volatile to me!  Every time I have upgraded way back
> > from zope2.1.3, something in zcatalog has changed and boken my product.
> > So far I have managed to find fixes, but this one has got me baffled!!
> >
>
> My experience has been that if field indexes are unspecified, the catalog
> will only match where those fields are empty.  In other words, by not
> specifing anything for the field indexes, you are actually specifying that
> for a match, they need to be empty.
> To get around this, I put another method between the form for entering what
> you wish to search for, and the method that actually searches the form.  This
> method looks at the field indexes, and if nothing has been specified, it
> doesn't pass them on to the method that does the actual search.
> An alternative to this, is to use names for the field indexes that aren't in
> the catalog index, and then at the start of the search method, check those
> names, and if they have a value associated with them, add to REQUEST what the
> names in the index are with the appropriate values.
>
> There may be a better way, I'm still pretty new to Zope and DTML sort of
> stuff.
>
> Peter Armstrong

--
Terry Kerr ([EMAIL PROTECTED])
Adroit Internet Solutions Pty Ltd (www.adroit.net)
Phone:   +613 9563 4461
Fax: +613 9563 3856
Mobile:  +61 414 938 124
ICQ: 79303381




___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and foreign characters

[Radim Gelner]

On Tue, Aug 29, 2000 at 11:57:08PM +0200, Dieter Maurer wrote:
> 
> If you work with a fixed locale, you can use the "-L" switch
> to inform Zope about your locale. Then the splitter should
> work correctly (for your locale).

Yes, that helped. I've called Zope with -L cs_CZ switch, rebuilt the
catalog and now searching works as it should.

Thanks.

Radim

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog in 2.2.x

[pja]

Terry Kerr <[EMAIL PROTECTED]> said: 

> Hi,
> 
> I have a python product which uses a ZCatalog and CatalogAware
> products.  The catalog has a number of Field and TextIndex indexes.
> Search the ZCatalog has worked fine when my product is instantiated in
> 2.1.6.  But now when I instantiate my product in 2.2.0 or 2.2.1,
> searching only works when I search using a field index.  Searching using
> a textindex always results in an empty result, even tho I can see the
> words I am searching for in the Vocabulary!  Any suggestions?  I have
> tried reindexing, completey recreating the Zcatalog, recreating the
> Vobulary and reindexing, and nothing has changed the behaviour.  Are
> textindexes simply broken?
> 
> 
> ZCatalog seems very volatile to me!  Every time I have upgraded way back
> from zope2.1.3, something in zcatalog has changed and boken my product.
> So far I have managed to find fixes, but this one has got me baffled!!
> 

My experience has been that if field indexes are unspecified, the catalog 
will only match where those fields are empty.  In other words, by not 
specifing anything for the field indexes, you are actually specifying that 
for a match, they need to be empty.
To get around this, I put another method between the form for entering what 
you wish to search for, and the method that actually searches the form.  This 
method looks at the field indexes, and if nothing has been specified, it 
doesn't pass them on to the method that does the actual search.
An alternative to this, is to use names for the field indexes that aren't in 
the catalog index, and then at the start of the search method, check those 
names, and if they have a value associated with them, add to REQUEST what the 
names in the index are with the appropriate values.

There may be a better way, I'm still pretty new to Zope and DTML sort of 
stuff.

Peter Armstrong

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and foreign characters

[Dieter Maurer]

Radim Gelner writes:
 > is it possible to make ZCatalog work correctly with words containing
 > characters other then those given in ISO-8859-1.
 > 
 > Now, it reports "no found" for all such queries even when these words
 > are present inside the documents on site.
I have made a very crude patch to "splitter.c" which lets it
treat every non-ascii character as a letter.

Obviously, this is not correct. It may include punctution into
words. This will lead to not find the words unless searched for
with the exact same punctuation.
Furthermore, non-ascii letters are not translated to lowercase.

Up to now, it gives acceptable results for us.
However, we did not yet make stress tests.

For a correct solution, splitter must be informed about
the encoding and a unicode letter classification and
case transformation must be applied.


If you work with a fixed locale, you can use the "-L" switch
to inform Zope about your locale. Then the splitter should
work correctly (for your locale).



Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog and foreign characters

[Aleksander Salwa]

On Tue, 29 Aug 2000, Radim Gelner wrote:

> is it possible to make ZCatalog work correctly with words containing
> characters other then those given in ISO-8859-1.
> 

Yes, I did it very simply.
You have to patch file lib/python/SearchIndex/Splitter.c
very slightly.

Just add line:
#include 
somewhere beside include lines (~ line 86)

and line:
setlocale(LC_ALL, "pl_PL");
in function initSplitter, line no. ~493.

It works for polish characters, and all other characters in iso-8859-2
set, so it should satisfy you :)

P.S. You need to compile it, of course:
 gcc --shared -O2 Splitter.c -I/usr/include/python1.5 -o Splitter
 (include path for RH Linux)
 Probably you will need some include files from source distribution.


[EMAIL PROTECTED]

/--\
| `long long long' is too long for GCC |
\--/


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog - OR'ing on different colums (indices)

[R. David Murray]

On Fri, 25 Aug 2000, Aleksander Salwa wrote:
> Is there any better method of OR'ing on differend indices in quering
> ZCatalog, then the method described in 'Advanced ZCatalog Searching'

Ah, my solution obviously only works if all your fields are text
indexes...

--RDM


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog - OR'ing on different colums (indices)

[R. David Murray]

On Fri, 25 Aug 2000, Aleksander Salwa wrote:
> Is there any better method of OR'ing on differend indices in quering
> ZCatalog, then the method described in 'Advanced ZCatalog Searching'
> How-To:

One way that works well but only works if you always do the same
type of search (which it sounds like is true in your case) is to
create a method for all of the objects that returns the concatenation
of all of the data fields of intrest, and index that in the Catalog.
You then search on the field, and return the relevant documents.
Since the Catalog only stores word indexes and document indexes and not
the text itself, this is not as inefficient as it sounds.  It's
also a little more execution-time efficient (which is good for a
web ap) since you are effectively pre-computing the between-fields
OR portion of the search.

--RDM


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zCatalog and subobjects

[Chris McDonough]

ZCatalog doesn't do well on hierarchical searches like this.  The way
you're doing it (without some sort of change to your objects'
structures) is pretty much the only way.  There's a "ZTopics" product
floating around that is supposed to provide functionality like this, but
1) I don't know if it's released, 2) I've never used it.

You *could* add a method to your "superobjects" that returned the
meta_type of all of its subobjects and add that to the Catalog index,
e.g.

def getChildMetaTypes(self):
   return map(lambda x: getattr(x, 'meta_type'), self.objectValues())

then add getChildMetaTypes as a keyword index...

then...

for x in Catalog.searchResults(getChildMetaTypes="My Kind Of Object"):
 ...





[EMAIL PROTECTED] wrote:
> 
> (Post #2)
> I'm trying to search a zCatalog to find all the objects containing a certain
> type of sub-object.
> 
> Right now, I use a dtml-if inside the result loop, but the result count is
> obviously wrong.
> 
> There must be a better way to do this, right?
> 
> Jean-François Ménard
> Intranet DPAS
> Pratiques d'affaires et orientations
> *(514) 840-3000  poste 3939
> *  (514) 840-5585
> *  [EMAIL PROTECTED]
> * 855 Ste-Catherine est, 6e étage
>   Montréal, Qué. H2L 4P5
> 
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )

-- 
Chris McDonough
Digital Creations
Publishers of Zope - http://www.zope.org

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog key error

[Chris McDonough]

Yeah, it's running 2.2.1b1 and it's still hosed.  :-(  When Ethan gets
back from LinuxWorld, I'm going to try to have him update to the latest
CVS and see if we can pin the problem down.

I added logging code to the catalog that was *supposed* to make it
easier to track down why these keyerrors were happening.  However, I
muffed it and the current Zope 2.2.1b1 release logs errors on all
fieldindex uncatalog_object calls (no matter if there's a problem or
not).  That particular problem is fixed in CVS.

We need to get a chance to examine the error output from index-unindex
functions to see exactly what's going on with the catalog.  I
desperately need to fix this problem for reasons unrelated to zope.org. 
If anyone is running a current CVS checkout of the Zope 2.2 branch, and
they're having KeyErrors, if you can send me relevant sections of your
stupid_log_file, I'd be grateful.  Maybe I can establish a pattern.  But
please don't send me log files with hundreds of unindex_object calls ala
Zope 2.2.1b1, it needs to be the latest CVS checkout on the 2.2 branch.

"Bak @ kedai" wrote:
> 
> is Zope.org running 2.2.1b1 or CVS?  cause i think key error still occurs.
> check out 'What's New' at Zope.org, and you'll get the error.
> 
> i might be wrong though
> thanks
> 
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )

-- 
Chris McDonough
Digital Creations
Publishers of Zope - http://www.zope.org

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog -- returning context of hits on fulltext

[R. David Murray]

On Mon, 14 Aug 2000, Jimmie Houchin wrote:
> I may be clueless and out of my league here and I haven't read the
> sources so I don't know... Well enough of a disclaimer. :)

I *have* read the ZCatalog/SearchIndex sources, but I don't understand
this part of it yet (or really that much of it at all!).  I think
we're getting into zope-dev terratory here...

> Is there anything in there which can provide the seek or byte position
> of the hit within text object? If so, it shouldn't be too difficult to
> read X bytes before and after the position and thereby provide what your
> looking for.

The standard TextIndex implementation records a notion of "position" for
each occurence of each word indexed.  I *think* this position is a word
count position, but I'm not sure.  Part of the code references a
'row', but it isn't at all clear that that has any relationship to
a source record.  If it is a word count, the other thing you'd need to
check would be whether it is a word count before or after splitter
activity.  I think it's the latter, which makes things more complicated.
Or just means you have to use more fuzz in your context .

> This would be nice to have out of the box.

The TextIndex 'position' information is intended to be used for
the 'near' operator (...) (so you can search on multiple words
"close" to each other for some definition of close).  You could
also use it to enforce word order (Maybe the "" operator does
that?).  Currently I think the result of applying the near operator
is used to adjust the "weight" of the index match, which affects
the order of the results returned.  (I haven't tested to see if
any of this works!)

So, the basic information you are looking for is there in some sense
to establish the position, but you'd still have to retrieve the
original sentences from the object itself, or from a full-text
metadata field.  Both of these are going to be memory intensive
operations.  If you index based on, say, individual lines, you'd
loose some of the the benefits of the near operator, though.  So
I'd say indexing based on paragraphs would probably be your best
approach.  This would also help mask position errors introduced if
the word count is indeed post-splitter.  Of course, you'll have to
decend to python to get access to the methods that will return the
actual position information.  But at least the code to record it
is already there.

Take a look at lib/python/SearchIndex/TextIndex.py for source
enlightenment.

--RDM


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog -- returning context of hits on fulltext

[Jimmie Houchin]

Hello,

I may be clueless and out of my league here and I haven't read the
sources so I don't know... Well enough of a disclaimer. :)

Is there anything in there which can provide the seek or byte position
of the hit within text object? If so, it shouldn't be too difficult to
read X bytes before and after the position and thereby provide what your
looking for.

This would be nice to have out of the box.

Just a thought.

Jimmie Houchin


Jean Jordaan wrote:
> 
> Hi Geir
> 
> > make a pythonmethod that returns the first 200 letters or
> > something  of the text ,
> 
> I've already got a pretty structured-text "Abstract" field
> that tells about the document, but I'd like to *see* the
> sentence on page 67 or wherever in a document where my
> term matches, so I know whether it's mentioned in passing
> or really important ..
> 
> --
> jean
> 
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog -- returning context of hits on fulltext

[Jimmie Houchin]

Hello,

I may be clueless and out of my league here and I haven't read the
sources so I don't know... Well enough of a disclaimer. :)

Is there anything in there which can provide the seek or byte position
of the hit within text object? If so, it shouldn't be too difficult to
read X bytes before and after the position and thereby provide what your
looking for.

This would be nice to have out of the box.

Just a thought.

Jimmie Houchin


Jean Jordaan wrote:
> 
> Hi Geir
> 
> > make a pythonmethod that returns the first 200 letters or
> > something  of the text ,
> 
> I've already got a pretty structured-text "Abstract" field
> that tells about the document, but I'd like to *see* the
> sentence on page 67 or wherever in a document where my
> term matches, so I know whether it's mentioned in passing
> or really important ..
> 
> --
> jean
> 
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

RE: [Zope] zcatalog -- returning context of hits on fulltext

[Jean Jordaan]

Hi Toby

> If you really do have a 67 page document, 

For the sake of the argument, that was page 67 of a 149-page
document .. 

> it would be better to store each page in its own ZODB 
> object, and index each page individually.

Well, the number of pages depends on the formatting .. but it
might be an idea -- chunking the input, say a paragraph, or, 
better, a sentence at a time.

I want to spend the absolute minimum time doing anything like 
that manually though. Content managers should be able to add a 
document without jumping through any hoops. So that chunking 
should be automatic. 

-- 
jean

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog -- returning context of hits on fulltext

[Toby Dickenson]

On Mon, 14 Aug 2000 13:04:49 +0100, Chris Withers <[EMAIL PROTECTED]>
wrote:

>Jean Jordaan wrote:
>> I've already got a pretty structured-text "Abstract" field
>> that tells about the document, but I'd like to *see* the
>> sentence on page 67 or wherever in a document where my
>> term matches, so I know whether it's mentioned in passing
>> or really important ..
>
>erk... that's a little harder :S
>
>I don't know if Catalog can do it, but at the very least you'd need a
>reference to your object to search the whole text, which means you loose
>the 'cool' metadata feature of not sucking a lot fo resource for search
>results.


If you really do have a 67 page document, it would be better to store
each page in its own ZODB object, and index each page individually.

With that scheme your search results page only has to load a few
pages, rather than a few documents.


Toby Dickenson
[EMAIL PROTECTED]

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog -- returning context of hits on fulltext

[Chris Withers]

Jean Jordaan wrote:
> 
> Dunno if Catalog can do it either .. not even if I do include the
> fulltext in the MetaData Table. 'Cause the hit will have come
> from the indexed text, so it has no way of knowing *which* hit in
> the original fulltext it was .. right?

Yeah, but there may be some undocumented way of getting where the hit
came from in the attribute out of the catalog.

Your guess is as good as mine :S

UTSL ;-)

Chris

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

RE: [Zope] zcatalog -- returning context of hits on fulltext

[Jean Jordaan]

Dunno if Catalog can do it either .. not even if I do include the 
fulltext in the MetaData Table. 'Cause the hit will have come
from the indexed text, so it has no way of knowing *which* hit in
the original fulltext it was .. right?

-- 
jean

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog -- returning context of hits on fulltext

[Chris Withers]

Jean Jordaan wrote:
> I've already got a pretty structured-text "Abstract" field
> that tells about the document, but I'd like to *see* the
> sentence on page 67 or wherever in a document where my
> term matches, so I know whether it's mentioned in passing
> or really important ..

erk... that's a little harder :S

I don't know if Catalog can do it, but at the very least you'd need a
reference to your object to search the whole text, which means you loose
the 'cool' metadata feature of not sucking a lot fo resource for search
results.

cheers,

Chris

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

RE: [Zope] zcatalog -- returning context of hits on fulltext

[Jean Jordaan]

Hi Geir

> make a pythonmethod that returns the first 200 letters or
> something  of the text , 

I've already got a pretty structured-text "Abstract" field
that tells about the document, but I'd like to *see* the 
sentence on page 67 or wherever in a document where my 
term matches, so I know whether it's mentioned in passing
or really important .. 

-- 
jean

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog -- returning context of hits on fulltext

[Geir Bækholt]

on Monday, August 14, 2000 Jean Jordaan wrote :

JJ> This works great, but returns only the
JJ> found objects and the cached properties specified in the 
JJ> MetaDate Table.

JJ> Wouldn't it be great to be able to give just a line or two
JJ> of context from the full text, the way Google does? I have
JJ> a feeling I'm asking for pie in the sky, but if anyone has
JJ> a brilliant (or a functional) solution I'd be delighted.

make a pythonmethod that returns the first 200 letters or
something  of the text , and add a metadata field to your
catalog referencing this pythonmethod..

-- Should give you what you want..

:-)

--
Geir Bækholt
web-developer/designer
[EMAIL PROTECTED]
http://www.funcom.com



___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog attachments?

[jan]

> > Simon Coles writes:
> >  > We have binary files stored in Zope, for example Word documents (but
> >  > could be any of a variety of document types).
> >  >
> >  > We would like to be able to index and search the contents of these
> >  > files using ZCatalog. So if a Word file contains the word "Fred",
> >  > then any search for "Fred" would include that file in the list of
> >  > documents returned.
> > Someone else already told you, that you must create a parameterless
> > method (it need not necessary be named "PrincipiaSearchSource")
> > that returns the files content.
> >
> > You may not need to keep the rendered version around but
> > may be able to extract the plain text on demand.
> > I think, there is a "word.dll" that provides access to
> > MS Word from applications. Alternatively, you could
> > control Word via COM.

Ther is a Perl (I know, I know...) script to convert Word DOC
files into HTML. That should work well enough to make the stuff
searchable (I would use doc2html.pl | lynx -d to get a pure ASCII
version, though).
It is probably fast enough to just render on the fly (i.e., upon
indexing).

HTH,
Jan

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog attachments?

[Phil Harris]

All,

Doing the text pulling from COM is *SLOW* to say the least,  You'd probabnly
be better converting them to RTF and then using something like OmniMark to
convert to XML.

That way you'd have the best of both worlds, including something you can
render to HTML when zDOM/zXSLT becomes a reality.

I already do this and it's fast enough at the conversion, a 500k doc takes
about 2 seconds.

hth

Phil
[EMAIL PROTECTED]
- Original Message -
From: Dieter Maurer <[EMAIL PROTECTED]>
To: Simon Coles <[EMAIL PROTECTED]>
Cc: <[EMAIL PROTECTED]>
Sent: Friday, August 04, 2000 9:06 PM
Subject: Re: [Zope] ZCatalog attachments?


> Simon Coles writes:
>  > We have binary files stored in Zope, for example Word documents (but
>  > could be any of a variety of document types).
>  >
>  > We would like to be able to index and search the contents of these
>  > files using ZCatalog. So if a Word file contains the word "Fred",
>  > then any search for "Fred" would include that file in the list of
>  > documents returned.
> Someone else already told you, that you must create a parameterless
> method (it need not necessary be named "PrincipiaSearchSource")
> that returns the files content.
>
> You may not need to keep the rendered version around but
> may be able to extract the plain text on demand.
> I think, there is a "word.dll" that provides access to
> MS Word from applications. Alternatively, you could
> control Word via COM.
>
>
> Dieter
>
> ___
> Zope maillist  -  [EMAIL PROTECTED]
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog attachments?

[Dieter Maurer]

Simon Coles writes:
 > We have binary files stored in Zope, for example Word documents (but 
 > could be any of a variety of document types).
 > 
 > We would like to be able to index and search the contents of these 
 > files using ZCatalog. So if a Word file contains the word "Fred", 
 > then any search for "Fred" would include that file in the list of 
 > documents returned.
Someone else already told you, that you must create a parameterless
method (it need not necessary be named "PrincipiaSearchSource")
that returns the files content.

You may not need to keep the rendered version around but
may be able to extract the plain text on demand.
I think, there is a "word.dll" that provides access to
MS Word from applications. Alternatively, you could
control Word via COM.


Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog attachments?

[Aleksander Salwa]

On Fri, 4 Aug 2000, Simon Coles wrote:

> We have binary files stored in Zope, for example Word documents (but 
> could be any of a variety of document types).
> 
> We would like to be able to index and search the contents of these 
> files using ZCatalog. So if a Word file contains the word "Fred", 
> then any search for "Fred" would include that file in the list of 
> documents returned.
> 
> Is anyone doing something like this? If so, how?
> 

Simple search in binary data of course won't do it, because of complex
format of Word documents. So:
Try to keep beside every document its 'rendered' version - converted to
plain text (created by saving them with Word in plain text format).
Then create class representing your document. This class should provide
parameterless method 'PrincipiaSearchSource' returning rendered version of
document. However, it's untested - but seems to be a step in right
direction ;)


[EMAIL PROTECTED]

/--\
| `long long long' is too long for GCC |
\--/



___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog updating

[R. David Murray]

On Mon, 31 Jul 2000, Aaron Williamson wrote:
> I am convinced this is a stupid question that has been answered a
> hundred times already, but after scouring zope.org and the archives, I
> still don't know the answer, so...
> 
> How can I automatically update my ZCatalog when items have been
> added/modified/changed?

Any objects that you want to be automatically updated in the Catalog
must have CatalogAware as one of their base classes.  And then you
have to add a reindex_object call to any forms that add or update
them.  (I fail to understand why one has to explictly call
reindex_object when properties change, even though I think someone
tried to explain it to me once).

I suppose you could do something similar with non-catalog aware
objects, where you made sure the add/update forms called the Catalog
to index them, but they wouldn't get deleted from the catalog automatically,
or reindexed automaticaly when moved.  If you are writing your own
management screens for those operations, though, you could make
it all work, I think.

You could also use ZSchedular to perioducally run the commands to
reindex your site, if instaneous reindexing is not a requirement.

--RDM


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog updating

[Chris Withers]

Aaron Williamson wrote:
> How can I automatically update my ZCatalog when items have been
> added/modified/changed?

CatalogAware

Chris

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog and squishdot newbie question

[Chris Withers]

George Donnelly wrote:
> you shouldn't need to create a Catlog and then put your Squishdot site in
> it. The main squishdot site *is* a Catalog, or has Catalog-like qualities.
> (i think).

Well, it is a subclass of ZCatalog as of Squishdot 0.4.0

> if you want to be able to search the Squishdot site *and* other stuff you
> should be able to select the Squishdot site as well as whatever other
> Catalogs you have when you add a Z Search Interface.

Sounds liek great advice :-)

cheers,

Chris

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] zcatalog and squishdot newbie question

[George Donnelly]

i'm a bit of a newbie myself so take my advice with a grain of salt, but...

you shouldn't need to create a Catlog and then put your Squishdot site in
it. The main squishdot site *is* a Catalog, or has Catalog-like qualities.
(i think).

if you want to be able to search the Squishdot site *and* other stuff you
should be able to select the Squishdot site as well as whatever other
Catalogs you have when you add a Z Search Interface.

hth

Regards,
GEORGE DONNELLY
[EMAIL PROTECTED]
http://cyklotron.com/
The man who follows the crowd will usually get no further than the crowd.
The man who walks alone is likely to find himself in places no one has ever
been. --Alan Ashley-Pitt

> I have created a catalog and added my squishdot site to it.  However,
> when I go to look at the catalog contents there are no squishdot
> articles listed (the suishdot site object is listed though).  When I run
> a search for a keyword that I know is in the squishdot object, I get no
> matches (but do when I search for other non-squishdot keywords).  I
> thought this may have something to do with running squishdot 3.1 so I
> upgraded to 4.1, reindexed and still nothing.  What could be the
> problem?  I heard somone say something about your catalog cataloging
> itself (mine does)- should it do this?  I tried removing it but nothing.


___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )

Re: [Zope] ZCatalog & dynamic sites

[Dieter Maurer]

 > I have a Zope site that uses MySQL to dynamically create some pages.
 > What I want is the ability to search throughout the site, INCLUDING
 > the pages that are database-driven. It seems to me that ZCatalog only=20
 > finds items that exist *in zope*.
 > 
 > Is it something ZCatalog can't do or am I missing anything obvious ?
ZCatalog catalogues objects.

Objects must help ZCatalog by providing attributes and/or
parameterless methods. When the object is indexed, ZCatalog
accesses the attributes/methods to obtain the terms the
object should be indexed for.
And the objects must be URL addressable,
because they are itentified inside the catalog by their URL.

ZCatalog can index any object that satisfies these
requirements.

You probably must work that your dynamic objects with database
contents do that.


Dieter

___
Zope maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )