Hi Simon, progress! Yes, go ahead and re-create and delete documents with those IDs and see if you can successfully compact. Regards,
Adam On Aug 12, 2011, at 6:00 AM, Simon Eisenmann wrote: > Hi Adam, > > all right, looks like i am able to find the trouble makers when also > looking at the deleted documents: > > ~# python repaircompact.py http://localhost:5984/gangstercluster_1 > Changes feed is > http://localhost:5984/gangstercluster_1/_changes?since=0. > Fetched feed into /tmp/tmpLjRkZY. > Not found error: missing!=deleted > (/gangstercluster_1/alive_1dcc4efdb2a411e0acd3003048679e10) > Not found error: missing!=deleted > (/gangstercluster_1/alive_1dcc523bb2a411e0acd4003048679e10) > Not found error: missing!=deleted > (/gangstercluster_1/alive_41ed63c1b2a411e0acd4003048679e10) > Not found error: missing!=deleted > (/gangstercluster_1/alive_41ed678bb2a411e0acd5003048679e10) > Processed 8225 entries (last sequence: 5972567) > Document count: 8222 (count: 74, deleted: 8148) > Document commited sequence is now: 5972569 > > > Though its hard to say if the number of changes feed rows matches the > number of the database infos as its constantly changing. Though its very > close so probably it would match if nothing changes. > > So now that i have found the missing documents. Should i just create an > empty document with that ID, and delete it again? > > Thank you and best regards > Simon > > ps: updated script attached > > > Am Donnerstag, den 11.08.2011, 12:30 -0400 schrieb Adam Kocoloski: >> Hi Simon, I wouldn't skip the deleted documents, a deleted doc could just as >> easily be the one missing in the ID index. When you lookup a deleted >> document you should see "reason":"deleted" instead of the "reason":"missing" >> that you get if the ID is not in the index at all. Then again, if you see >> that only ever see deleted docs after retrieving the full _changes then a >> deleted doc is probably not the source of your troubles. >> >> Can you confirm that the number of rows in the changes feed is equal to >> doc_count + doc_del_count from db.info()? Best, >> >> Adam >> >> On Aug 11, 2011, at 12:08 PM, Simon Eisenmann wrote: >> >>> Hi Adam, >>> >>> i wrote a short python script, which loads the complete changes feed and >>> requests all of the documents not marked "deleted" using HTTP HEAD >>> requests. Though i only find documents which have been deleted after i >>> have retrieved the complete changes view (takes a while for large >>> databases), and never the same. >>> >>> So again no luck here in finding the problems. >>> >>> Any more suggestions? Is there a way to rebuild the complete database >>> file (maybe offline?). >>> >>> Thank you and best regards >>> Simon >>> >>> >>> ps: The script i have been using is attached. >>> >>> >>> >>> Am Dienstag, den 09.08.2011, 16:10 -0400 schrieb Adam Kocoloski: >>>> Hi Simon, CouchDB 1.1.0 includes a recent optimization to >>>> _changes?include_docs=true which allows it to skip a lookup in the id tree >>>> and instead load the document body from the pointer in the sequence tree. >>>> In that case you wouldn't notice any missing entry in the id tree. You >>>> would notice it, however, if you did direct lookups for each document. >>>> Apologies for the outdated instructions. Can you try looking up the >>>> documents in a separate request and see if the results change? >>>> >>>> Adam >>>> >>>> On Aug 9, 2011, at 5:49 AM, Simon Eisenmann wrote: >>>> >>>>> Hi Adam, >>>>> >>>>> i just checked the whole _changes feed (since=0) and could not find any >>>>> document "missing" when using "include_docs=true". >>>>> >>>>> The database in itself has only around 30 documents, so should be quite >>>>> small. Though there are lots of creations and deletions happening all >>>>> the time. Thus it its daily purged and compacted. >>>>> >>>>> So - the changes feed is of no help. Any other idea? >>>>> >>>>> Thank you and best regards >>>>> Simon >>>>> >>>>> Am Montag, den 08.08.2011, 13:25 -0400 schrieb Adam Kocoloski: >>>>>> Hi Simon, I think my amended instructions to Mike are still a sensible >>>>>> way to debug/workaround the problem. Reiterating (96282148 was the last >>>>>> seq Mike observed in the Futon status for the compaction): >>>>>> >>>>>>> 1) What you really want are the last 1000 Ids in the seq_tree prior to >>>>>>> the compactor crash. So maybe something like >>>>>>> >>>>>>> GET /iris/_changes?descending=true&limit=1000&since=96282148 >>>>>> >>>>>>> 2) Figure out which of those entries are missing from the id tree, e.g. >>>>>>> lookup the document and see if the response is {"not_found":"missing"}. >>>>>>> You could also try using include_docs=true on the _changes feed to >>>>>>> accomplish the same. >>>>>> >>>>>>> 3) Once you've identified the problematic IDs, try creating them again. >>>>>>> You might end up introducing duplicates in the _changes feed, but if >>>>>>> you do there's a procedure to fix that. >>>>>> >>>>>> Regards, Adam >>>>>> >>>>>> On Aug 8, 2011, at 12:31 PM, Simon Eisenmann wrote: >>>>>> >>>>>>> Hi Guys, >>>>>>> >>>>>>> i have a couple of CouchDB instances which started to be come >>>>>>> unpackable. It shows this error: >>>>>>> >>>>>>> [Mon, 08 Aug 2011 16:16:27 GMT] [info] [<0.10808.123>] Starting >>>>>>> compaction for db "database1" >>>>>>> [Mon, 08 Aug 2011 16:16:45 GMT] [error] [<0.10808.123>] ** Generic >>>>>>> server <0.10808.123> terminating >>>>>>> ** Last message in was {'EXIT',<0.30396.143>, >>>>>>> {function_clause, >>>>>>> [{couch_db_updater,'-copy_docs/4-fun-2-', >>>>>>> [not_found, >>>>>>> {db,<0.10807.123>,<0.10808.123>,nil, >>>>>>> <<"1312767347007568">>,<0.10805.123>, >>>>>>> <0.10809.123>, >>>>>>> {db_header,5,6340261,0, >>>>>>> {7198006895,{65952,10145}}, >>>>>>> {7198010315,76813}, >>>>>>> {7198051016,[]}, >>>>>>> 364,7050915618,nil,1000}, >>>>>>> 6340261, >>>>>>> {btree,<0.10805.123>, >>>>>>> {7198006895,{65952,10145}}, >>>>>>> #Fun<couch_db_updater.10.19222179>, >>>>>>> #Fun<couch_db_updater.11.21515767>, >>>>>>> #Fun<couch_btree.5.124754102>, >>>>>>> #Fun<couch_db_updater.12.93888648>}, >>>>>>> {btree,<0.10805.123>, >>>>>>> {7198010315,76813}, >>>>>>> #Fun<couch_db_updater.13.40165027>, >>>>>>> #Fun<couch_db_updater.14.82810239>, >>>>>>> #Fun<couch_btree.5.124754102>, >>>>>>> #Fun<couch_db_updater.15.104121193>}, >>>>>>> {btree,<0.10805.123>, >>>>>>> {7198051016,[]}, >>>>>>> #Fun<couch_btree.0.83553141>, >>>>>>> #Fun<couch_btree.1.30790806>, >>>>>>> #Fun<couch_btree.2.124754102>,nil}, >>>>>>> 6340261,<<"spreedcom_accounts_1">>, >>>>>>> "/var/lib/couchdb/database1.couch",[], >>>>>>> [],nil, >>>>>>> {user_ctx,null,[],undefined}, >>>>>>> nil,1000, >>>>>>> [before_header,after_header,on_file_open], >>>>>>> false}, >>>>>>> <0.30397.143>]}, >>>>>>> {lists,map,2}, >>>>>>> {lists,map,2}, >>>>>>> {couch_db_updater,copy_docs,4}, >>>>>>> {couch_db_updater,'-copy_compact/3-fun-0-',6}, >>>>>>> {couch_btree,stream_kv_node2,8}, >>>>>>> {couch_btree,stream_kp_node,7}, >>>>>>> {couch_btree,fold,4}]}} >>>>>>> >>>>>>> >>>>>>> ... lots of more similar errors following. >>>>>>> >>>>>>> In this mailing list i have found a similar issue from the beginning of >>>>>>> this year (Fri, 31 Dec 2010 12:38:18), though without a solution. >>>>>>> >>>>>>> This database got purged a lot and has constant changes. So pretty >>>>>>> similar from the older topic. So i assume there is something wrong in >>>>>>> the database file related to previous purges. So looks like some bug >>>>>>> here. >>>>>>> >>>>>>> So - any hints how to fix this? The database is getting pretty large and >>>>>>> has to be packed from time to time. >>>>>>> >>>>>>> CouchDB is running with Version 1.1.0 on Linux 64bit. The database has >>>>>>> initially been created with CouchDB 1.0.1 - though the issue appeared a >>>>>>> couple of weeks ago (packing has been working with 1.1.0 before. >>>>>>> >>>>>>> >>>>>>> Thank you and best regards >>>>>>> Simon >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Simon Eisenmann >>>>>>> >>>>>>> [ mailto:[email protected] ] >>>>>>> >>>>>>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ] >>>>>>> [ T. +49.711.896656.0 | F.+49.711.89665610 ] >>>>>>> [ http://www.struktur.de | mailto:[email protected] ] >>>>>> >>>>> >>>>> -- >>>>> Simon Eisenmann >>>>> >>>>> [ mailto:[email protected] ] >>>>> >>>>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ] >>>>> [ T. +49.711.896656.68 | F.+49.711.89665610 ] >>>>> [ http://www.struktur.de | mailto:[email protected] ] >>>> >>> >>> -- >>> Simon Eisenmann >>> >>> [ mailto:[email protected] ] >>> >>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ] >>> [ T. +49.711.896656.68 | F.+49.711.89665610 ] >>> [ http://www.struktur.de | mailto:[email protected] ] >>> <repaircompact.py> >> > > -- > Simon Eisenmann > > [ mailto:[email protected] ] > > [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ] > [ T. +49.711.896656.68 | F.+49.711.89665610 ] > [ http://www.struktur.de | mailto:[email protected] ] > <repaircompact.py>
