Re: [Zope-dev] ZCatalog "Key Error" question
[EMAIL PROTECTED] wrote at 2006-9-3 18:06 -0400: > >Thank you so much for following-up with me, here. I think you're right >about this having to do with the indexes on the ZCatalog. > >The catalog has: >1. Seven "FieldIndex" indexes. >2. A "DateIndex" index >3. Three "ZCTextIndex" indexes I have not seen reports that these indexes cause inconsistencies of the type you have described. A long time ago, there have been problems in the "BTrees" conflict resolution which caused catalog inconsistencies (of a different kind). They affected document lists of length one in the index. When the last remaining document was removed and a new one concurrently inserted, the insertion could be lost. But this was fixed a long time ago. Apart from bugs, inconsistencies are often caused by "try: ... except: ..." clauses. If the "try" clause modifies the persistent state and then is aborted by an exception, it is vital that the exception reaches the ZPublisher such that it aborts the transaction. If the "except" prevents this, then the ZPublisher will commit and you are likely to get inconsistencies. In older Zope versions (before Zope 2.8.1), such a dangerous "try: ... except: ..." has been in "OFS.ObjectManager.ObjectManager.manage_beforeDelete". In more modern versions, the code has been slightly cleaned up but it still is error prone. Skim your log files whether you see "manage_beforeDelete () threw" ERROR log entries. You can get more hints towards the problem by examining the query that results in your exception: If this query just consists of a subquery to a single index, then you know the problematic index. If it combines several indexes (by an "and"), then all of these indexes would need to be inconsistent (which is possible but far less likely than a single index). In this case, one would expect that not the index but the catalog primary data got corrupted. In the latter case, the problem would need to happen during initial indexing or final unindexing. Studying the code may exclude some possibilities. -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog "Key Error" question
Thank you so much for following-up with me, here. I think you're right
about this having to do with the indexes on the ZCatalog.
The catalog has:
1. Seven "FieldIndex" indexes.
2. A "DateIndex" index
3. Three "ZCTextIndex" indexes
I actually create these from python in a function that accepts the
"catalog" as a parameter during the Product Initialization process.
I have a function "catalog_add" that creates and initializes the a
ZCatalog when this product is created and initialized, so let me
actually post the sourcecode here for that function and the functions it
calls. It's fairly short. There are two functions of note
"catalog_add" and "addIndexes". I'd actually be grateful if you would
take a look and let me know if you see anything. In particular, am I
using an index that you know to have problems, or that I'm using
improperly.
class Newsdesk(Folder,Persistent,Implicit):
meta_type = "Newsdesk"
# ...
def catalog_add(self):
"setup the catalog"
self.manage_addProduct["ZCatalog"].manage_addZCatalog("Catalog",
"Default Catalog")
catalog = self._getOb('Catalog')
self.addLexicon(catalog)
self.addIndexes(catalog)
self.addMetas(catalog)
catalog = self._getOb('Catalog')
return catalog
def addIndexes(self, cat):
cat.addIndex('date_found', 'FieldIndex', 'date_found')
cat.addIndex('deleted_p', 'FieldIndex', 'deleted_p')
cat.addIndex('uri', 'FieldIndex', 'uri')
cat.addIndex('categories_field', 'FieldIndex',
'categories')
cat.addIndex('categorized_p', 'FieldIndex',
'categorized_p')
cat.addIndex('approved_p', 'FieldIndex', 'approved_p')
cat.addIndex('search_id', 'FieldIndex', 'search_id')
extras = Empty()
extras.doc_attr = 'title'
extras.index_type = 'Okapi BM25 Rank'
extras.lexicon_id = 'Lexicon'
cat.addIndex('title', 'ZCTextIndex', extras)
extras2 = Empty()
extras2.doc_attr = 'source_name'
extras2.index_type = 'Okapi BM25 Rank'
extras2.lexicon_id = 'Lexicon'
cat.addIndex('source_name', 'ZCTextIndex', extras2)
extras3 = Empty()
extras3.doc_attr = 'categories'
extras3.index_type = 'Okapi BM25 Rank'
extras3.lexicon_id = 'Lexicon'
cat.addIndex('categories', 'ZCTextIndex', extras3)
cat.addIndex('bobobase_modification_time', 'DateIndex')
def addMetas(self, cat):
"create the metadata columns"
cat.manage_addColumn("categorized_p")
cat.manage_addColumn("deleted_p")
cat.manage_addColumn("search_id")
cat.manage_addColumn("categories")
cat.manage_addColumn("date_found")
cat.manage_addColumn("source_name")
cat.manage_addColumn("title")
cat.manage_addColumn("uri")
def addLexicon(self, cat):
elem = []
wordSplitter = Empty()
wordSplitter.group = 'Word Splitter'
wordSplitter.name = 'HTML aware splitter'
caseNormalizer = Empty()
caseNormalizer.group = 'Case Normalizer'
caseNormalizer.name = 'Case Normalizer'
stopWords = Empty()
stopWords.group = 'Stop Words'
stopWords.name = 'Remove listed and single char words'
elem.append(wordSplitter)
elem.append(caseNormalizer)
elem.append(stopWords)
cat.manage_addProduct['ZCTextIndex'].manage_addLexicon('Lexicon',
'Default Lexicon', elem)
On Sun, 3 Sep 2006 20:11:33 +0200, "Dieter Maurer" <[EMAIL PROTECTED]>
said:
> Dieter Maurer wrote at 2006-9-3 19:22 +0200:
> >[EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400:
> >>
> >>Traceback for my Key Error (Dieter, thank you for alerting me on where
> >>to find this):
> >>
> >>Traceback (innermost last):
> >> ...
> >> - Physical Path: /nephron_links/news_dtml
> >> Module DocumentTemplate.DT_String, line 476, in __call__
> >> Module DocumentTemplate.DT_In, line 465, in renderwb
> >> Module DocumentTemplate.DT_In, line 747, in sort_sequence
> >> Module Products.ZCatalog.Lazy, line 158, in __getitem__
> >> Module Products.ZCatalog.Catalog, line 520, in getScoredResult
> >>KeyError: -464571725
> >
> >This does not look like a catalog corruption.
> >
> >Instead, it looks as if the catalog changed since the search was
> >done. Are you caching your search results?
>
>
> Sorry! I was wrong!
>
> It has nothing to do with a catalog modification after the search.
>
> Look
Re: [Zope-dev] ZCatalog "Key Error" question
Dieter Maurer wrote at 2006-9-3 19:22 +0200: >[EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400: >> >>Traceback for my Key Error (Dieter, thank you for alerting me on where >>to find this): >> >>Traceback (innermost last): >> ... >> - Physical Path: /nephron_links/news_dtml >> Module DocumentTemplate.DT_String, line 476, in __call__ >> Module DocumentTemplate.DT_In, line 465, in renderwb >> Module DocumentTemplate.DT_In, line 747, in sort_sequence >> Module Products.ZCatalog.Lazy, line 158, in __getitem__ >> Module Products.ZCatalog.Catalog, line 520, in getScoredResult >>KeyError: -464571725 > >This does not look like a catalog corruption. > >Instead, it looks as if the catalog changed since the search was >done. Are you caching your search results? Sorry! I was wrong! It has nothing to do with a catalog modification after the search. Looks indeed as if an index reported a document which is no longer indexed. Looks like an index bug. Which (types of) indexes are involved in your failing queries? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog "Key Error" question
[EMAIL PROTECTED] wrote at 2006-9-1 13:29 -0400: > >Traceback for my Key Error (Dieter, thank you for alerting me on where >to find this): > >Traceback (innermost last): > ... > - Physical Path: /nephron_links/news_dtml > Module DocumentTemplate.DT_String, line 476, in __call__ > Module DocumentTemplate.DT_In, line 465, in renderwb > Module DocumentTemplate.DT_In, line 747, in sort_sequence > Module Products.ZCatalog.Lazy, line 158, in __getitem__ > Module Products.ZCatalog.Catalog, line 520, in getScoredResult >KeyError: -464571725 This does not look like a catalog corruption. Instead, it looks as if the catalog changed since the search was done. Are you caching your search results? -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog "Key Error" question
Traceback for my Key Error (Dieter, thank you for alerting me on where to find this): Traceback (innermost last): Module ZPublisher.Publish, line 113, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 40, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.PythonScripts.PythonScript, line 323, in _exec Module None, line 11, in to_javascript - - Line 11 Module OFS.DTMLDocument, line 128, in __call__ - - URL: http://nephron.org/news_dtml/manage_main - Physical Path: /nephron_links/news_dtml Module DocumentTemplate.DT_String, line 476, in __call__ Module DocumentTemplate.DT_In, line 465, in renderwb Module DocumentTemplate.DT_In, line 747, in sort_sequence Module Products.ZCatalog.Lazy, line 158, in __getitem__ Module Products.ZCatalog.Catalog, line 520, in getScoredResult KeyError: -464571725 On Fri, 1 Sep 2006 19:11:12 +0200, "Dieter Maurer" <[EMAIL PROTECTED]> said: > [EMAIL PROTECTED] wrote at 2006-9-1 02:54 -0400: > >I have a Zope product that relies heavily on a ZCatalog to search for > >objects within a folder matching certain criteria. The ZCatalog > >frequently becomes corrupt, and on searching the catalog, the catalog > >gives me key errors. I'm trying to figure out what causes the > >corruption so I can stop doing whatever I am doing to corrupt the > >ZCatalog. > > > >Here are the clues: > > You forget to provide us with an essential glue: > > The complete error information (error type, error value > and traceback). You find it in your "error_log" object > (in Zope's "Root Folder"). > > > > -- > Dieter Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog "Key Error" question
[EMAIL PROTECTED] wrote at 2006-9-1 02:54 -0400: >I have a Zope product that relies heavily on a ZCatalog to search for >objects within a folder matching certain criteria. The ZCatalog >frequently becomes corrupt, and on searching the catalog, the catalog >gives me key errors. I'm trying to figure out what causes the >corruption so I can stop doing whatever I am doing to corrupt the >ZCatalog. > >Here are the clues: You forget to provide us with an essential glue: The complete error information (error type, error value and traceback). You find it in your "error_log" object (in Zope's "Root Folder"). -- Dieter ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog "Key Error" question
> Read or Write conflict errors? I'm getting the "Key Error" traceback on Reads, but my guess is that the actual "error" happens sometime before the "Key Error", when the ZCatalog itself is corrupted. I am not sure what that is, because it does not generate an error. However, I see the error when I call ZCatalogInstance.Search(...), and only when the Search includes objects that are corrupted in the ZCatalog. > Are the objects modified at the same time by other threads? I'm not sure about this. I'm running ZEO, and in general, I think I can cause the error when I am the only user on the system, only issuing one request. However, I'm getting another ConflictError when I try to set an attribute value of a ZODB object during the execution of a script, and I'm told that may be because there is a problem accessing the same object from multiple threads. Is there a good article that you know of that I could find out more about thread-safe programming in ZEO? Best, Brian On Fri, 01 Sep 2006 09:27:41 +0200, "Andreas Jung" <[EMAIL PROTECTED]> said: > > > --On 1. September 2006 02:54:26 -0400 [EMAIL PROTECTED] > wrote: > > > > > I have a Zope product that relies heavily on a ZCatalog to search for > > objects within a folder matching certain criteria. The ZCatalog > > frequently becomes corrupt, and on searching the catalog, the catalog > > gives me key errors. I'm trying to figure out what causes the > > corruption so I can stop doing whatever I am doing to corrupt the > > ZCatalog. > > > > > > Read or Write conflict errors? Are the objects modified at the same time > by > other threads? > > -aj Brian Rosenthal General Partner, RoboCommerce, LLC [EMAIL PROTECTED] http://www.robocommerce.com 800-644-7626 ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog "Key Error" question
--On 1. September 2006 02:54:26 -0400 [EMAIL PROTECTED] wrote: I have a Zope product that relies heavily on a ZCatalog to search for objects within a folder matching certain criteria. The ZCatalog frequently becomes corrupt, and on searching the catalog, the catalog gives me key errors. I'm trying to figure out what causes the corruption so I can stop doing whatever I am doing to corrupt the ZCatalog. Read or Write conflict errors? Are the objects modified at the same time by other threads? -aj pgpRrhqcwgEtE.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
