Re: [Zope-dev] ZCatalog Key Error question
[EMAIL PROTECTED] wrote at 2006-9-3 18:06 -0400: Thank you so much for following-up with me, here. I think you're right about this having to do with the indexes on the ZCatalog. The catalog has: 1. Seven FieldIndex indexes. 2. A DateIndex index 3. Three ZCTextIndex indexes I have not seen reports that these indexes cause inconsistencies of the type you have described. A long time ago, there have been problems in the BTrees conflict resolution which caused catalog inconsistencies (of a different kind). They affected document lists of length one in the index. When the last remaining document was removed and a new one concurrently inserted, the insertion could be lost. But this was fixed a long time ago. Apart from bugs, inconsistencies are often caused by try: ... except: ... clauses. If the try clause modifies the persistent state and then is aborted by an exception, it is vital that the exception reaches the ZPublisher such that it aborts the transaction. If the except prevents this, then the ZPublisher will commit and you are likely to get inconsistencies. In older Zope versions (before Zope 2.8.1), such a dangerous try: ... except: ... has been in OFS.ObjectManager.ObjectManager.manage_beforeDelete. In more modern versions, the code has been slightly cleaned up but it still is error prone. Skim your log files whether you see manage_beforeDelete () threw ERROR log entries. You can get more hints towards the problem by examining the query that results in your exception: If this query just consists of a subquery to a single index, then you know the problematic index. If it combines several indexes (by an and), then all of these indexes would need to be inconsistent (which is possible but far less likely than a single index). In this case, one would expect that not the index but the catalog primary data got corrupted. In the latter case, the problem would need to happen during initial indexing or final unindexing. Studying the code may exclude some possibilities. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
[EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400: Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): ... - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 This does not look like a catalog corruption. Instead, it looks as if the catalog changed since the search was done. Are you caching your search results? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Dieter Maurer wrote at 2006-9-3 19:22 +0200: [EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400: Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): ... - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 This does not look like a catalog corruption. Instead, it looks as if the catalog changed since the search was done. Are you caching your search results? Sorry! I was wrong! It has nothing to do with a catalog modification after the search. Looks indeed as if an index reported a document which is no longer indexed. Looks like an index bug. Which (types of) indexes are involved in your failing queries? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Thank you so much for following-up with me, here. I think you're right
about this having to do with the indexes on the ZCatalog.
The catalog has:
1. Seven FieldIndex indexes.
2. A DateIndex index
3. Three ZCTextIndex indexes
I actually create these from python in a function that accepts the
catalog as a parameter during the Product Initialization process.
I have a function catalog_add that creates and initializes the a
ZCatalog when this product is created and initialized, so let me
actually post the sourcecode here for that function and the functions it
calls. It's fairly short. There are two functions of note
catalog_add and addIndexes. I'd actually be grateful if you would
take a look and let me know if you see anything. In particular, am I
using an index that you know to have problems, or that I'm using
improperly.
class Newsdesk(Folder,Persistent,Implicit):
meta_type = Newsdesk
# ...
def catalog_add(self):
setup the catalog
self.manage_addProduct[ZCatalog].manage_addZCatalog(Catalog,
Default Catalog)
catalog = self._getOb('Catalog')
self.addLexicon(catalog)
self.addIndexes(catalog)
self.addMetas(catalog)
catalog = self._getOb('Catalog')
return catalog
def addIndexes(self, cat):
cat.addIndex('date_found', 'FieldIndex', 'date_found')
cat.addIndex('deleted_p', 'FieldIndex', 'deleted_p')
cat.addIndex('uri', 'FieldIndex', 'uri')
cat.addIndex('categories_field', 'FieldIndex',
'categories')
cat.addIndex('categorized_p', 'FieldIndex',
'categorized_p')
cat.addIndex('approved_p', 'FieldIndex', 'approved_p')
cat.addIndex('search_id', 'FieldIndex', 'search_id')
extras = Empty()
extras.doc_attr = 'title'
extras.index_type = 'Okapi BM25 Rank'
extras.lexicon_id = 'Lexicon'
cat.addIndex('title', 'ZCTextIndex', extras)
extras2 = Empty()
extras2.doc_attr = 'source_name'
extras2.index_type = 'Okapi BM25 Rank'
extras2.lexicon_id = 'Lexicon'
cat.addIndex('source_name', 'ZCTextIndex', extras2)
extras3 = Empty()
extras3.doc_attr = 'categories'
extras3.index_type = 'Okapi BM25 Rank'
extras3.lexicon_id = 'Lexicon'
cat.addIndex('categories', 'ZCTextIndex', extras3)
cat.addIndex('bobobase_modification_time', 'DateIndex')
def addMetas(self, cat):
create the metadata columns
cat.manage_addColumn(categorized_p)
cat.manage_addColumn(deleted_p)
cat.manage_addColumn(search_id)
cat.manage_addColumn(categories)
cat.manage_addColumn(date_found)
cat.manage_addColumn(source_name)
cat.manage_addColumn(title)
cat.manage_addColumn(uri)
def addLexicon(self, cat):
elem = []
wordSplitter = Empty()
wordSplitter.group = 'Word Splitter'
wordSplitter.name = 'HTML aware splitter'
caseNormalizer = Empty()
caseNormalizer.group = 'Case Normalizer'
caseNormalizer.name = 'Case Normalizer'
stopWords = Empty()
stopWords.group = 'Stop Words'
stopWords.name = 'Remove listed and single char words'
elem.append(wordSplitter)
elem.append(caseNormalizer)
elem.append(stopWords)
cat.manage_addProduct['ZCTextIndex'].manage_addLexicon('Lexicon',
'Default Lexicon', elem)
On Sun, 3 Sep 2006 20:11:33 +0200, Dieter Maurer [EMAIL PROTECTED]
said:
Dieter Maurer wrote at 2006-9-3 19:22 +0200:
[EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400:
Traceback for my Key Error (Dieter, thank you for alerting me on where
to find this):
Traceback (innermost last):
...
- Physical Path: /nephron_links/news_dtml
Module DocumentTemplate.DT_String, line 476, in __call__
Module DocumentTemplate.DT_In, line 465, in renderwb
Module DocumentTemplate.DT_In, line 747, in sort_sequence
Module Products.ZCatalog.Lazy, line 158, in __getitem__
Module Products.ZCatalog.Catalog, line 520, in getScoredResult
KeyError: -464571725
This does not look like a catalog corruption.
Instead, it looks as if the catalog changed since the search was
done. Are you caching your search results?
Sorry! I was wrong!
It has nothing to do with a catalog modification after the search.
Looks indeed as if an index reported a document which is no longer
indexed. Looks like an index bug. Which
[Zope-dev] ZCatalog Key Error question
I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Here are the clues: 1. When I re-index the catalog, everything works fine. Unfortunately, I cannot re-index the catalog very frequently because it takes a long time 2. The objects that are indexed are Citation objects, a class I defined. 3. The Citation class extends CatalogAware (as well as Item,Persistent,Implicit, PropertyManager) 4. Every time I change a citation, I call reindex_object() (on the citation) 5. When I add a citation, I call reindex_object() on the citation immediately. 6. I'm using Zope 2.8.4-final, built from source. 7. I'm using python 2.3.5 Thank you in advance. I can provide more information upon request. Best, Brian Rosenthal Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
--On 1. September 2006 02:54:26 -0400 [EMAIL PROTECTED] wrote: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Read or Write conflict errors? Are the objects modified at the same time by other threads? -aj pgpRrhqcwgEtE.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Read or Write conflict errors? I'm getting the Key Error traceback on Reads, but my guess is that the actual error happens sometime before the Key Error, when the ZCatalog itself is corrupted. I am not sure what that is, because it does not generate an error. However, I see the error when I call ZCatalogInstance.Search(...), and only when the Search includes objects that are corrupted in the ZCatalog. Are the objects modified at the same time by other threads? I'm not sure about this. I'm running ZEO, and in general, I think I can cause the error when I am the only user on the system, only issuing one request. However, I'm getting another ConflictError when I try to set an attribute value of a ZODB object during the execution of a script, and I'm told that may be because there is a problem accessing the same object from multiple threads. Is there a good article that you know of that I could find out more about thread-safe programming in ZEO? Best, Brian On Fri, 01 Sep 2006 09:27:41 +0200, Andreas Jung [EMAIL PROTECTED] said: --On 1. September 2006 02:54:26 -0400 [EMAIL PROTECTED] wrote: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Read or Write conflict errors? Are the objects modified at the same time by other threads? -aj Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
[EMAIL PROTECTED] wrote at 2006-9-1 02:54 -0400: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Here are the clues: You forget to provide us with an essential glue: The complete error information (error type, error value and traceback). You find it in your error_log object (in Zope's Root Folder). -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog Key Error question
Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): Module ZPublisher.Publish, line 113, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 40, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.PythonScripts.PythonScript, line 323, in _exec Module None, line 11, in to_javascript - PythonScript at /nephron_links/to_javascript used for /nephron_links/news_dtml - Line 11 Module OFS.DTMLDocument, line 128, in __call__ - DTMLDocument at /nephron_links/news_dtml - URL: http://nephron.org/news_dtml/manage_main - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 On Fri, 1 Sep 2006 19:11:12 +0200, Dieter Maurer [EMAIL PROTECTED] said: [EMAIL PROTECTED] wrote at 2006-9-1 02:54 -0400: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Here are the clues: You forget to provide us with an essential glue: The complete error information (error type, error value and traceback). You find it in your error_log object (in Zope's Root Folder). -- Dieter Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
