Re: [Zope-dev] PathIndex unindex fix
If you have write access to the Zope CVS, go ahead. Thanks, Andreas --On Donnerstag, 21. November 2002 17:35 + seb bacon [EMAIL PROTECTED] wrote: I've already fixed it; should I commit it or do you want the patches? Andreas Jung wrote: Please file a collector issue and assign it to me. -aj --On Donnerstag, 21. November 2002 13:24 + seb bacon [EMAIL PROTECTED] wrote: Anyone object to me changing PathIndex so it swallows exceptions and logs them when unindexing content? On upgrading to Zope 2.6 some if the unindex paths seem to have got hosed, and I don't want this breaking my site. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) - -Andreas Jung http://www.andreas-jung.com - - EMail: andreas at andreas-jung.com - -Life is too short to (re)write parsers - - -- [] j a m k i t seb bacon T: 020 7549 0520 F: 020 7490 1152 M: 07968 301 336 W: www.jamkit.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) - -Andreas Jung http://www.andreas-jung.com - - EMail: andreas at andreas-jung.com - -Life is too short to (re)write parsers - - ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex unindex fix
Leonardo Rochael Almeida wrote: If you are to ignore errors, please ignore specific errors that are known to happen. Please don't use a generic except. ...or re-raise only ConflictErrors. Hmmm... this issue is perhaps deeper than it first appeared. Are conflict errors swallowed by the other indexes? Should they be? I wonder if the discussion about swallowing ConflictErrors (on zodb-dev IIRC) was ever resolved? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex unindex fix
+1 from me... most other PluginIndexes already fail gracefully when something to be unindexed has disappeared. IMHO all indexes should behave that way. jens On Thursday, Nov 21, 2002, at 08:24 US/Eastern, seb bacon wrote: Anyone object to me changing PathIndex so it swallows exceptions and logs them when unindexing content? On upgrading to Zope 2.6 some if the unindex paths seem to have got hosed, and I don't want this breaking my site. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex unindex fix
On Thu, Nov 21, 2002 at 08:52:41AM -0500, Jens Vagelpohl wrote: | +1 from me... most other PluginIndexes already fail gracefully when | something to be unindexed has disappeared. IMHO all indexes should | behave that way. Now that you mentioned, I got an error while trying to index NZO using ZCTextIndex, so I guess it is suffering from the same issue. Anyone can confirm? []'s -- Sidnei da Silva (dreamcatcher) [EMAIL PROTECTED] X3ng Web Technology http://www.x3ng.com.br GNU/Linux user 257852 Debian GNU/Linux 3.0 (Sid) 2.4.18 ppc This file will self-destruct in five minutes. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex unindex fix
Please file a collector issue and assign it to me. -aj --On Donnerstag, 21. November 2002 13:24 + seb bacon [EMAIL PROTECTED] wrote: Anyone object to me changing PathIndex so it swallows exceptions and logs them when unindexing content? On upgrading to Zope 2.6 some if the unindex paths seem to have got hosed, and I don't want this breaking my site. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) - -Andreas Jung http://www.andreas-jung.com - - EMail: andreas at andreas-jung.com - -Life is too short to (re)write parsers - - ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex unindex fix
Sidnei da Silva wrote: Now that you mentioned, I got an error while trying to index NZO using ZCTextIndex, so I guess it is suffering from the same issue. Anyone can confirm? No, no, failure during _indexing_ is a good thing (tm). Failure during _unindexing_ is a bad thing (r). cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex unindex fix
On Thu, 2002-11-21 at 11:24, seb bacon wrote: Anyone object to me changing PathIndex so it swallows exceptions and logs them when unindexing content? This sounds REALLY LOUD ALARMS in my head. There is a class of errors (subclasses of ConflictError) that should never be ingored because they're automatically handled by Zope itself (in ZServer, I think). I can think of situations where a changed content causes a Conflict error, and the standard Zope retry would be enough to make the PathIndex be correctly unindexed. Actually, ZCatalogs are a conflict error magnet, I believe to ignore conflict errors is to beg for problems. If you are to ignore errors, please ignore specific errors that are known to happen. Please don't use a generic except. -- Ideas don't stay in some minds very long because they don't like solitary confinement. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex unindex fix
I've already fixed it; should I commit it or do you want the patches? Andreas Jung wrote: Please file a collector issue and assign it to me. -aj --On Donnerstag, 21. November 2002 13:24 + seb bacon [EMAIL PROTECTED] wrote: Anyone object to me changing PathIndex so it swallows exceptions and logs them when unindexing content? On upgrading to Zope 2.6 some if the unindex paths seem to have got hosed, and I don't want this breaking my site. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) - -Andreas Jung http://www.andreas-jung.com - - EMail: andreas at andreas-jung.com - -Life is too short to (re)write parsers - - -- [] j a m k i t seb bacon T: 020 7549 0520 F: 020 7490 1152 M: 07968 301 336 W: www.jamkit.com ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] PathIndex doesn't index last part of path
I'm all for putting together a proposal to do this, soliciting comments, and
contributing to the effort. What would be the right venue for that?
On the other hand, AFAIK, the addition of a depth query feature to PathIndex
as it is wouldn't likely break existing indexes. And given that Andy's need
(get metadata for a Cataloged object, given some cached referecne, in this
case a path, to it) could be filled by the Catalog itself (or changes to
it), I'm not sure what other features would be needed out of PathIndex?
In brief, what I would like:
results = Catalog.searchResults(path={'query':myContainersPath,
'depth':0, 'level':0})
Where depth means:
-1 In parent of path specified (up one)
0 Contained within path specified
1 Contained in path specified or directly in one of its
children
2 Same as above, but also in child container's child
No depth specified means behavior is like current functionality in path
indexes. A brief glance at docs and code gives me the impression that this
could peacefuly co-exist with level.
Sean
-Original Message-
From: Andreas Jung [mailto:[EMAIL PROTECTED]]
Sent: Monday, August 19, 2002 11:14 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]
Subject: Re: [Zope-dev] PathIndex doesn't index last part of path
- Original Message -
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Tuesday, August 20, 2002 00:58
Subject: RE: [Zope-dev] PathIndex doesn't index last part of path
One thing that seems to be feature-broken about PathIndexes, though, is
the
lack of ability to specify a depth (not level, but depth of a match, for
example, to specify items). Above and beyond support for a specific place
within a hierarchy, there needs to be a simple mechanism to support query
for an absolute path, and only objects directly matching that path, or
directly contained within that path.
Following this thread I appears to me that there is a need for several
new functionalities inside the PathIndex (maybe a new PathIndexNG). In
this case I suggest to write a proposal to collect all requirements.
I volunteer to help on the implementation.
Andreas
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
I think you should publish it as fishbowl proposal on dev.zope.org
and propose the implementation to go into Zope 2.7.
Andreas
- Original Message -
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]
Sent: Wednesday, August 21, 2002 17:49
Subject: RE: [Zope-dev] PathIndex doesn't index last part of path
I'm all for putting together a proposal to do this, soliciting comments,
and
contributing to the effort. What would be the right venue for that?
On the other hand, AFAIK, the addition of a depth query feature to
PathIndex
as it is wouldn't likely break existing indexes. And given that Andy's
need
(get metadata for a Cataloged object, given some cached referecne, in this
case a path, to it) could be filled by the Catalog itself (or changes to
it), I'm not sure what other features would be needed out of PathIndex?
In brief, what I would like:
results = Catalog.searchResults(path={'query':myContainersPath,
'depth':0, 'level':0})
Where depth means:
-1 In parent of path specified (up one)
0 Contained within path specified
1 Contained in path specified or directly in one of its
children
2 Same as above, but also in child container's child
No depth specified means behavior is like current functionality in path
indexes. A brief glance at docs and code gives me the impression that
this
could peacefuly co-exist with level.
Sean
-Original Message-
From: Andreas Jung [mailto:[EMAIL PROTECTED]]
Sent: Monday, August 19, 2002 11:14 PM
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]
Subject: Re: [Zope-dev] PathIndex doesn't index last part of path
- Original Message -
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Tuesday, August 20, 2002 00:58
Subject: RE: [Zope-dev] PathIndex doesn't index last part of path
One thing that seems to be feature-broken about PathIndexes, though, is
the
lack of ability to specify a depth (not level, but depth of a match, for
example, to specify items). Above and beyond support for a specific
place
within a hierarchy, there needs to be a simple mechanism to support
query
for an absolute path, and only objects directly matching that path, or
directly contained within that path.
Following this thread I appears to me that there is a need for several
new functionalities inside the PathIndex (maybe a new PathIndexNG). In
this case I suggest to write a proposal to collect all requirements.
I volunteer to help on the implementation.
Andreas
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] PathIndex doesn't index last part of path
If the question is begged, why use the catalog: for folders with thousands of objects, applications like CMF skins can be amazingly slow using ObjectManager methods and CMF-wrapped equivalents; they also do not provide sorting. For applications like this, both containment and flexible query support (sorting) are important. Use-case: thousands of objects in a Portal Folder. CMF Skins like Plone use ObjectManager based methods to get a list of siblings in a sidebar. When you have 1000+ items in the folder, the fact that this takes 2 minutes on an Athlon 2000+/1GB server, isn't batchable, and isn't sortable is sort of sad. Use of the Catalog makes much more sense here (I can sort, and the performance penalty is low, so I can take a slice of the resulting sequence of brains, and do getObject() over each one; even if a container could be improved performance wise to have a quick objectIds() method, ObjectManager interfaces have no mechanism to specify sorting based on metadata like you could do with a Catalog query). One thing that seems to be feature-broken about PathIndexes, though, is the lack of ability to specify a depth (not level, but depth of a match, for example, to specify items). Above and beyond support for a specific place within a hierarchy, there needs to be a simple mechanism to support query for an absolute path, and only objects directly matching that path, or directly contained within that path. The workaround for this is to add a custom FieldIndex that indexes the string value of the path of the items container, and this requires putting a python script in the root of a CMF portal, for example, that is acquired by all indexed content. It would be nice if the PathIndex machinery could do this out of the box. http://lists.zope.org/pipermail/zope-cmf/2002-August/014167.html http://lists.zope.org/pipermail/zope-cmf/2002-August/014204.html Sean -Original Message- From: Casey Duncan [mailto:[EMAIL PROTECTED]] Sent: Friday, August 16, 2002 4:55 PM To: Andy McKay; [EMAIL PROTECTED] Subject: Re: [Zope-dev] PathIndex doesn't index last part of path A PathIndex is designed to make it more efficient to aggregate objects at various levels of containment. Their primary use case AFAIK is to allow to to limit queries to particular places within a hierarchy. The idea is to eliminate recursive searching of leaf level folders when you want all objects under a higher level and its child levels. Also, by not indexing the nodes themselves, the index is an order of magnitude smaller and searches are therefore faster and it takes less room and is faster to update. In fact there is no need to index the entire path of an object in the catalog. Even with no Indexes defined, ZCatalog already does this for you. The uid of every entry in the catalog is the full path to the object (as a string). Unfortunately, ZCatalog does not expose this to the surface but you can write a trivial external method to do it. And I might entertain adding a ZCatalog API to do so if I had a good use case. Right now you can only access entries by RID. Now that begs the question, If you already know the path to the object you are looking for, why are you using the Catalog in the first place? I highly doubt doing what you describe below is faster than just directly accessing the object. In fact I'd be willing to be its slower, especially since you are searching two indexes to get it. Unless of course these are dynamically generated objects of some kind (no stored in Zope). As for making RIDs more permanent, that would basically require a rewrite of the Catalog, and make certain operations much more expensive. As it stands, your application should only assume that RIDs are valid within a single transaction. You should use the path to uniquely identify objects, or some application defined uid that gets cataloged otherwise. -Casey - Original Message - From: Andy McKay [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Saturday, August 17, 2002 6:22 PM Subject: [Zope-dev] PathIndex doesn't index last part of path This is mostly a question for AJ, but any input would be great. This bug bit me today and is documented here: http://collector.zope.org/Zope/449/ISSUE_TRANSCRIPT/view I dont understand the brief argument against this one, it would make sense to me to able to pull an object out of the catalog based on its path. For example if I want /foo/bar/blammo, currently this means there is only one way of pulling the an object of the catalog given this path. Thats to send (path='/foo/bar', id='blammo'), rather than (path='/foo/bar/blammo'). Why wouldnt we want it this way? One thing I have done is store a whole bunch of references to objects as selected by the user. These are essentially random objects and the quickest way is to pull them back out of the catalog. Of course I cant do more than one object per query (unless Im missing some other way) Id love to do (path=['/foo/bar
Re: [Zope-dev] PathIndex doesn't index last part of path
- Original Message - From: [EMAIL PROTECTED] To: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Tuesday, August 20, 2002 00:58 Subject: RE: [Zope-dev] PathIndex doesn't index last part of path One thing that seems to be feature-broken about PathIndexes, though, is the lack of ability to specify a depth (not level, but depth of a match, for example, to specify items). Above and beyond support for a specific place within a hierarchy, there needs to be a simple mechanism to support query for an absolute path, and only objects directly matching that path, or directly contained within that path. Following this thread I appears to me that there is a need for several new functionalities inside the PathIndex (maybe a new PathIndexNG). In this case I suggest to write a proposal to collect all requirements. I volunteer to help on the implementation. Andreas ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
If you call getObject, that actually does traversal anyway. Right, in that situation it would be pointless... You rock, Casey, thanks. I was thinking more about adding: def getMetadataFromPath(self, path): get metadata for an object using its path rid = self._catalog.uids[path] return self._catalog.getMetadataForRID(rid) def getIndexFromPath(self, path): get index for an object using its path rid = self._catalog.uids[path] return self._catalog.getIndexDataForRID(rid) Since this uses the same terminology and returns the same data as getIndexDataForRID and getMetadataDataForRID. Is there any reason why I couldn't checked these in? -- Andy McKay Agmweb Consulting http://www.agmweb.ca ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
Your code looks fine, I think it meshes better with the underlying catalog code too. I don't have a problem with this getting checked in, just make sure you update IZCatalog.py, help/Catalog.py and add unit tests (that pass ;^). -Casey - Original Message - From: Andy McKay [EMAIL PROTECTED] To: Casey Duncan [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Sunday, August 18, 2002 2:54 AM Subject: Re: [Zope-dev] PathIndex doesn't index last part of path If you call getObject, that actually does traversal anyway. Right, in that situation it would be pointless... You rock, Casey, thanks. I was thinking more about adding: def getMetadataFromPath(self, path): get metadata for an object using its path rid = self._catalog.uids[path] return self._catalog.getMetadataForRID(rid) def getIndexFromPath(self, path): get index for an object using its path rid = self._catalog.uids[path] return self._catalog.getIndexDataForRID(rid) Since this uses the same terminology and returns the same data as getIndexDataForRID and getMetadataDataForRID. Is there any reason why I couldn't checked these in? -- Andy McKay Agmweb Consulting http://www.agmweb.ca ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
Woohoo my first check in ;) -- Andy McKay Agmweb Consulting http://www.agmweb.ca - Original Message - From: Casey Duncan [EMAIL PROTECTED] To: Andy McKay [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Saturday, August 17, 2002 12:14 PM Subject: Re: [Zope-dev] PathIndex doesn't index last part of path Your code looks fine, I think it meshes better with the underlying catalog code too. I don't have a problem with this getting checked in, just make sure you update IZCatalog.py, help/Catalog.py and add unit tests (that pass ;^). -Casey - Original Message - From: Andy McKay [EMAIL PROTECTED] To: Casey Duncan [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Sunday, August 18, 2002 2:54 AM Subject: Re: [Zope-dev] PathIndex doesn't index last part of path If you call getObject, that actually does traversal anyway. Right, in that situation it would be pointless... You rock, Casey, thanks. I was thinking more about adding: def getMetadataFromPath(self, path): get metadata for an object using its path rid = self._catalog.uids[path] return self._catalog.getMetadataForRID(rid) def getIndexFromPath(self, path): get index for an object using its path rid = self._catalog.uids[path] return self._catalog.getIndexDataForRID(rid) Since this uses the same terminology and returns the same data as getIndexDataForRID and getMetadataDataForRID. Is there any reason why I couldn't checked these in? -- Andy McKay Agmweb Consulting http://www.agmweb.ca ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
Casey Duncan writes: A PathIndex is designed to make it more efficient to aggregate objects at various levels of containment. Their primary use case AFAIK is to allow to to limit queries to particular places within a hierarchy. The idea is to eliminate recursive searching of leaf level folders when you want all objects under a higher level and its child levels. Also, by not indexing the nodes themselves, the index is an order of magnitude smaller and searches are therefore faster and it takes less room and is faster to update. This property need to be well documented. At least, it is not intuitive. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
Hi, Sorry, maybe I'm too late ;) but I have a note here. uid in Cataloc can be path but it can be any arbitrary unigue identifier. In system I'm working with it is ID generated in the time of object creation and it persists and it is uid in Catalog. Nothing wrong with the methods themselves but about their name. As far as I understood they are getRecordFromPath, getMetadataFromPath, getIndexFromPath. Generally it will be path a uid of Catalolog, but naming in other cases will be confusing. Maybe it is just a question of documentation? Regards, m. Casey Duncan wrote: Your code looks fine, I think it meshes better with the underlying catalog code too. I don't have a problem with this getting checked in, just make sure you update IZCatalog.py, help/Catalog.py and add unit tests (that pass ;^). -Casey - Original Message - From: Andy McKay [EMAIL PROTECTED] To: Casey Duncan [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Sunday, August 18, 2002 2:54 AM Subject: Re: [Zope-dev] PathIndex doesn't index last part of path If you call getObject, that actually does traversal anyway. Right, in that situation it would be pointless... You rock, Casey, thanks. I was thinking more about adding: def getMetadataFromPath(self, path): get metadata for an object using its path rid = self._catalog.uids[path] return self._catalog.getMetadataForRID(rid) def getIndexFromPath(self, path): get index for an object using its path rid = self._catalog.uids[path] return self._catalog.getIndexDataForRID(rid) Since this uses the same terminology and returns the same data as getIndexDataForRID and getMetadataDataForRID. Is there any reason why I couldn't checked these in? -- Andy McKay Agmweb Consulting http://www.agmweb.ca -- Myroslav Opyr zope.net.ua http://zope.net.ua/ ° Ukrainian Zope Hosting e-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
Yes I agree, I think it would be better if the apis were getRecordForUid, getIndexForUid since the uids can be something other than paths.Thanks for the input on that. -Casey - Original Message - From: Myroslav Opyr [EMAIL PROTECTED] To: Casey Duncan [EMAIL PROTECTED] Cc: Andy McKay [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Sunday, August 18, 2002 6:55 PM Subject: Re: [Zope-dev] PathIndex doesn't index last part of path Hi, Sorry, maybe I'm too late ;) but I have a note here. uid in Cataloc can be path but it can be any arbitrary unigue identifier. In system I'm working with it is ID generated in the time of object creation and it persists and it is uid in Catalog. Nothing wrong with the methods themselves but about their name. As far as I understood they are getRecordFromPath, getMetadataFromPath, getIndexFromPath. Generally it will be path a uid of Catalolog, but naming in other cases will be confusing. Maybe it is just a question of documentation? Regards, m. Casey Duncan wrote: Your code looks fine, I think it meshes better with the underlying catalog code too. I don't have a problem with this getting checked in, just make sure you update IZCatalog.py, help/Catalog.py and add unit tests (that pass ;^). -Casey - Original Message - From: Andy McKay [EMAIL PROTECTED] To: Casey Duncan [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Sunday, August 18, 2002 2:54 AM Subject: Re: [Zope-dev] PathIndex doesn't index last part of path If you call getObject, that actually does traversal anyway. Right, in that situation it would be pointless... You rock, Casey, thanks. I was thinking more about adding: def getMetadataFromPath(self, path): get metadata for an object using its path rid = self._catalog.uids[path] return self._catalog.getMetadataForRID(rid) def getIndexFromPath(self, path): get index for an object using its path rid = self._catalog.uids[path] return self._catalog.getIndexDataForRID(rid) Since this uses the same terminology and returns the same data as getIndexDataForRID and getMetadataDataForRID. Is there any reason why I couldn't checked these in? -- Andy McKay Agmweb Consulting http://www.agmweb.ca -- Myroslav Opyr zope.net.ua http://zope.net.ua/ ° Ukrainian Zope Hosting e-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
A PathIndex is designed to make it more efficient to aggregate objects at various levels of containment. Their primary use case AFAIK is to allow to to limit queries to particular places within a hierarchy. The idea is to eliminate recursive searching of leaf level folders when you want all objects under a higher level and its child levels. Also, by not indexing the nodes themselves, the index is an order of magnitude smaller and searches are therefore faster and it takes less room and is faster to update. In fact there is no need to index the entire path of an object in the catalog. Even with no Indexes defined, ZCatalog already does this for you. The uid of every entry in the catalog is the full path to the object (as a string). Unfortunately, ZCatalog does not expose this to the surface but you can write a trivial external method to do it. And I might entertain adding a ZCatalog API to do so if I had a good use case. Right now you can only access entries by RID. Now that begs the question, If you already know the path to the object you are looking for, why are you using the Catalog in the first place? I highly doubt doing what you describe below is faster than just directly accessing the object. In fact I'd be willing to be its slower, especially since you are searching two indexes to get it. Unless of course these are dynamically generated objects of some kind (no stored in Zope). As for making RIDs more permanent, that would basically require a rewrite of the Catalog, and make certain operations much more expensive. As it stands, your application should only assume that RIDs are valid within a single transaction. You should use the path to uniquely identify objects, or some application defined uid that gets cataloged otherwise. -Casey - Original Message - From: Andy McKay [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Saturday, August 17, 2002 6:22 PM Subject: [Zope-dev] PathIndex doesn't index last part of path This is mostly a question for AJ, but any input would be great. This bug bit me today and is documented here: http://collector.zope.org/Zope/449/ISSUE_TRANSCRIPT/view I dont understand the brief argument against this one, it would make sense to me to able to pull an object out of the catalog based on its path. For example if I want /foo/bar/blammo, currently this means there is only one way of pulling the an object of the catalog given this path. Thats to send (path='/foo/bar', id='blammo'), rather than (path='/foo/bar/blammo'). Why wouldnt we want it this way? One thing I have done is store a whole bunch of references to objects as selected by the user. These are essentially random objects and the quickest way is to pull them back out of the catalog. Of course I cant do more than one object per query (unless Im missing some other way) Id love to do (path=['/foo/bar/blammo', '/foo/bar/blammoz']) and get these 2 objects... I think that would be neat. It would seem data_record_id_ is not guaranteed to permanent after a reindex_object (which CatalogAwareness uses), since this uncatalog and then recatalogs the object. If this did work it would be cool and I could undo all the changes to my app back again. - The patch is already there, so Im curious why do we have what seems to be a more limited design? - Would a halfway option such as path_match='final' be a choice that wont break any code but would confuse everyone and not make into the documentation? - Is it just a matter of fixing reindex_object as was suggested on #zope so that data_record_id_ is more permanent? Cheers -- Andy McKay Agmweb Consulting http://www.agmweb.ca ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
A PathIndex is designed to make it more efficient to aggregate objects at various levels of containment. Their primary use case AFAIK is to allow to to limit queries to particular places within a hierarchy. The idea is to eliminate recursive searching of leaf level folders when you want all objects under a higher level and its child levels. Also, by not indexing the nodes themselves, the index is an order of magnitude smaller and searches are therefore faster and it takes less room and is faster to update. In fact there is no need to index the entire path of an object in the catalog. Even with no Indexes defined, ZCatalog already does this for you. The uid of every entry in the catalog is the full path to the object (as a string). Unfortunately, ZCatalog does not expose this to the surface but you can write a trivial external method to do it. And I might entertain adding a ZCatalog API to do so if I had a good use case. Right now you can only access entries by RID. Now that begs the question, If you already know the path to the object you are looking for, why are you using the Catalog in the first place? I highly doubt doing what you describe below is faster than just directly accessing the object. In fact I'd be willing to be its slower, especially since you are searching two indexes to get it. Unless of course these are dynamically generated objects of some kind (no stored in Zope). As for making RIDs more permanent, that would basically require a rewrite of the Catalog, and make certain operations much more expensive. As it stands, your application should only assume that RIDs are valid within a single transaction. You should use the path to uniquely identify objects, or some application defined uid that gets cataloged otherwise. -Casey - Original Message - From: Andy McKay [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Saturday, August 17, 2002 6:22 PM Subject: [Zope-dev] PathIndex doesn't index last part of path This is mostly a question for AJ, but any input would be great. This bug bit me today and is documented here: http://collector.zope.org/Zope/449/ISSUE_TRANSCRIPT/view I dont understand the brief argument against this one, it would make sense to me to able to pull an object out of the catalog based on its path. For example if I want /foo/bar/blammo, currently this means there is only one way of pulling the an object of the catalog given this path. Thats to send (path='/foo/bar', id='blammo'), rather than (path='/foo/bar/blammo'). Why wouldnt we want it this way? One thing I have done is store a whole bunch of references to objects as selected by the user. These are essentially random objects and the quickest way is to pull them back out of the catalog. Of course I cant do more than one object per query (unless Im missing some other way) Id love to do (path=['/foo/bar/blammo', '/foo/bar/blammoz']) and get these 2 objects... I think that would be neat. It would seem data_record_id_ is not guaranteed to permanent after a reindex_object (which CatalogAwareness uses), since this uncatalog and then recatalogs the object. If this did work it would be cool and I could undo all the changes to my app back again. - The patch is already there, so Im curious why do we have what seems to be a more limited design? - Would a halfway option such as path_match='final' be a choice that wont break any code but would confuse everyone and not make into the documentation? - Is it just a matter of fixing reindex_object as was suggested on #zope so that data_record_id_ is more permanent? Cheers -- Andy McKay Agmweb Consulting http://www.agmweb.ca ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex doesn't index last part of path
[snip] Hmm ok, I can see those reasons. Unfortunately, ZCatalog does not expose this to the surface but you can write a trivial external method to do it. And I might entertain adding a ZCatalog API to do so if I had a good use case. Ah... I think this might be the best idea, I'll add that in to mine and, see if anyone else wants it. Now that begs the question, If you already know the path to the object you are looking for, why are you using the Catalog in the first place? I highly doubt doing what you describe below is faster than just directly accessing the object. In fact I'd be willing to be its slower, especially since you are searching two indexes to get it. Okay so lets assume there is only index I need to search, the path index. Wouldn't it be faster to pull that out of the Catalog then do a traverse over to the sub object, wake up a bunch of objects to do that and get the object? It would be interesting to test that... perhaps Im just leaning on the old crutch that its faster to get stuff from the catalog than wake many objects up. Suppose I have 100 such objects. I would have thought one catalog query on one index (even though its a big union) would be faster than 100 traversals. Anyway since I cant efficiently go and get an individual object from the catalog, this is what Im doing now... -- Andy McKay Agmweb Consulting http://www.agmweb.ca ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex
Let me know what you think You should submit this to the Collector: http://collector.zope.org I have already had a good experience with it ;-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] PathIndex query options
- Original Message - From: Jeffrey P Shell [EMAIL PROTECTED] To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Monday, November 19, 2001 21:36 Subject: [Zope-dev] PathIndex query options I've looked hard at the two pages returned for PathIndex on Zope.org, and cannot make sense out of how to use 'level'. From the example, which is full of paths like '/aa/bb/cc' and '/cc/bb/cc' - all of the same size and using the same three letters -, I cannot tell if I can do a path query to find objects ONLY in a certain path, and not above or below it. I have a situation where I have a hierarchy of Task content objects, and in one case I only want to display the immediate child tasks at certain paths (basically to top-level tasks), using the catalog query except for (hopefully) one changed parameter. http://www.zope.org/Members/michel/ZB/SearchingZCatalog.dtml http://www.zope.org//Wikis/DevSite/Proposals/PathIndexDocumentation The level parameter specifies where the search starts. E.g. when you index an object with the URL http://www.zope.org//Wikis/DevSite/Proposals/PathIndexDocumentation then the PathIndex uses its physical path for indexing. In this example: '/Wikis/DevSite/Proposals/PathIndexDocumentation' Searching for DevSite and level=1 would give you this document as search result. The first element of the physical path has level 0, the second level 1 and so on. Hope this helps, Andreas ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
