Hi Adam, i wrote a short python script, which loads the complete changes feed and requests all of the documents not marked "deleted" using HTTP HEAD requests. Though i only find documents which have been deleted after i have retrieved the complete changes view (takes a while for large databases), and never the same.
So again no luck here in finding the problems.
Any more suggestions? Is there a way to rebuild the complete database
file (maybe offline?).
Thank you and best regards
Simon
ps: The script i have been using is attached.
Am Dienstag, den 09.08.2011, 16:10 -0400 schrieb Adam Kocoloski:
> Hi Simon, CouchDB 1.1.0 includes a recent optimization to
> _changes?include_docs=true which allows it to skip a lookup in the id tree
> and instead load the document body from the pointer in the sequence tree. In
> that case you wouldn't notice any missing entry in the id tree. You would
> notice it, however, if you did direct lookups for each document. Apologies
> for the outdated instructions. Can you try looking up the documents in a
> separate request and see if the results change?
>
> Adam
>
> On Aug 9, 2011, at 5:49 AM, Simon Eisenmann wrote:
>
> > Hi Adam,
> >
> > i just checked the whole _changes feed (since=0) and could not find any
> > document "missing" when using "include_docs=true".
> >
> > The database in itself has only around 30 documents, so should be quite
> > small. Though there are lots of creations and deletions happening all
> > the time. Thus it its daily purged and compacted.
> >
> > So - the changes feed is of no help. Any other idea?
> >
> > Thank you and best regards
> > Simon
> >
> > Am Montag, den 08.08.2011, 13:25 -0400 schrieb Adam Kocoloski:
> >> Hi Simon, I think my amended instructions to Mike are still a sensible way
> >> to debug/workaround the problem. Reiterating (96282148 was the last seq
> >> Mike observed in the Futon status for the compaction):
> >>
> >>> 1) What you really want are the last 1000 Ids in the seq_tree prior to
> >>> the compactor crash. So maybe something like
> >>>
> >>> GET /iris/_changes?descending=true&limit=1000&since=96282148
> >>
> >>> 2) Figure out which of those entries are missing from the id tree, e.g.
> >>> lookup the document and see if the response is {"not_found":"missing"}.
> >>> You could also try using include_docs=true on the _changes feed to
> >>> accomplish the same.
> >>
> >>> 3) Once you've identified the problematic IDs, try creating them again.
> >>> You might end up introducing duplicates in the _changes feed, but if you
> >>> do there's a procedure to fix that.
> >>
> >> Regards, Adam
> >>
> >> On Aug 8, 2011, at 12:31 PM, Simon Eisenmann wrote:
> >>
> >>> Hi Guys,
> >>>
> >>> i have a couple of CouchDB instances which started to be come
> >>> unpackable. It shows this error:
> >>>
> >>> [Mon, 08 Aug 2011 16:16:27 GMT] [info] [<0.10808.123>] Starting
> >>> compaction for db "database1"
> >>> [Mon, 08 Aug 2011 16:16:45 GMT] [error] [<0.10808.123>] ** Generic
> >>> server <0.10808.123> terminating
> >>> ** Last message in was {'EXIT',<0.30396.143>,
> >>> {function_clause,
> >>> [{couch_db_updater,'-copy_docs/4-fun-2-',
> >>> [not_found,
> >>> {db,<0.10807.123>,<0.10808.123>,nil,
> >>> <<"1312767347007568">>,<0.10805.123>,
> >>> <0.10809.123>,
> >>> {db_header,5,6340261,0,
> >>> {7198006895,{65952,10145}},
> >>> {7198010315,76813},
> >>> {7198051016,[]},
> >>> 364,7050915618,nil,1000},
> >>> 6340261,
> >>> {btree,<0.10805.123>,
> >>> {7198006895,{65952,10145}},
> >>> #Fun<couch_db_updater.10.19222179>,
> >>> #Fun<couch_db_updater.11.21515767>,
> >>> #Fun<couch_btree.5.124754102>,
> >>> #Fun<couch_db_updater.12.93888648>},
> >>> {btree,<0.10805.123>,
> >>> {7198010315,76813},
> >>> #Fun<couch_db_updater.13.40165027>,
> >>> #Fun<couch_db_updater.14.82810239>,
> >>> #Fun<couch_btree.5.124754102>,
> >>> #Fun<couch_db_updater.15.104121193>},
> >>> {btree,<0.10805.123>,
> >>> {7198051016,[]},
> >>> #Fun<couch_btree.0.83553141>,
> >>> #Fun<couch_btree.1.30790806>,
> >>> #Fun<couch_btree.2.124754102>,nil},
> >>> 6340261,<<"spreedcom_accounts_1">>,
> >>> "/var/lib/couchdb/database1.couch",[],
> >>> [],nil,
> >>> {user_ctx,null,[],undefined},
> >>> nil,1000,
> >>> [before_header,after_header,on_file_open],
> >>> false},
> >>> <0.30397.143>]},
> >>> {lists,map,2},
> >>> {lists,map,2},
> >>> {couch_db_updater,copy_docs,4},
> >>> {couch_db_updater,'-copy_compact/3-fun-0-',6},
> >>> {couch_btree,stream_kv_node2,8},
> >>> {couch_btree,stream_kp_node,7},
> >>> {couch_btree,fold,4}]}}
> >>>
> >>>
> >>> ... lots of more similar errors following.
> >>>
> >>> In this mailing list i have found a similar issue from the beginning of
> >>> this year (Fri, 31 Dec 2010 12:38:18), though without a solution.
> >>>
> >>> This database got purged a lot and has constant changes. So pretty
> >>> similar from the older topic. So i assume there is something wrong in
> >>> the database file related to previous purges. So looks like some bug
> >>> here.
> >>>
> >>> So - any hints how to fix this? The database is getting pretty large and
> >>> has to be packed from time to time.
> >>>
> >>> CouchDB is running with Version 1.1.0 on Linux 64bit. The database has
> >>> initially been created with CouchDB 1.0.1 - though the issue appeared a
> >>> couple of weeks ago (packing has been working with 1.1.0 before.
> >>>
> >>>
> >>> Thank you and best regards
> >>> Simon
> >>>
> >>>
> >>> --
> >>> Simon Eisenmann
> >>>
> >>> [ mailto:[email protected] ]
> >>>
> >>> [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> >>> [ T. +49.711.896656.0 | F.+49.711.89665610 ]
> >>> [ http://www.struktur.de | mailto:[email protected] ]
> >>
> >
> > --
> > Simon Eisenmann
> >
> > [ mailto:[email protected] ]
> >
> > [ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
> > [ T. +49.711.896656.68 | F.+49.711.89665610 ]
> > [ http://www.struktur.de | mailto:[email protected] ]
>
--
Simon Eisenmann
[ mailto:[email protected] ]
[ struktur AG | Kronenstraße 22a | D-70173 Stuttgart ]
[ T. +49.711.896656.68 | F.+49.711.89665610 ]
[ http://www.struktur.de | mailto:[email protected] ]
#!/usr/bin/python """ Simple script to load all documents returned by a CouchDB _changes feed with (since=0) to find if some of the documents are missing (checks HTTP status). (c)2011 Simon Eisenmann - mailto:[email protected] """ import sys import cjson import urllib import tempfile import urlparse import httplib import urllib class Repairer(object): def __init__(self, database_url, batchsize=1000): if database_url.endswith("/"): database_url = database_url[:-1] self.database_url = database_url self.database_url_parts = urlparse.urlparse(self.database_url) self.database_host, self.database_port = self.database_url_parts[1].split(":", 1) self.batchsize = batchsize def repair(self): url = "%s/_changes?since=0" % self.database_url print "Changes feed is %s." % url changes, headers = urllib.urlretrieve(url, tempfile.mktemp()) print "Fetched feed into %s." % changes fp = file(changes, "rb") fp.readline() line = fp.readline().strip() count = 0 buf = [] failed = {} seq = None while line: if line.endswith(","): line = line[:-1] if line.strip() == "]": break buf.append(cjson.decode(line)) if len(buf) >= self.batchsize: count, seq = self._check(buf, count=count, failed=failed) buf = [] line = fp.readline().strip() fp.close() if len(buf): count, seq = self._check(buf, count=count, failed=failed) buf = [] print "Processed %d entries (last sequence: %s)" % (count, seq) def _check(self, results, count=0, failed=None): if failed is None: failed = {} seq = None conn = httplib.HTTPConnection(self.database_host, int(self.database_port)) for c in results: seq = c.get("seq", 0) deleted = c.get("deleted", None) if deleted != True: i = c.get("id") quoted_i = urllib.quote_plus(i) if quoted_i.startswith("_design%2F"): # FIXME(longsleep) waah what a cheap hack quoted_i = "_design/%s" % quoted_i[10:] path = "%s/%s" % (self.database_url_parts[2], quoted_i) conn.request("HEAD", path) res = conn.getresponse() if res.status not in (200,): print "i", i, count, path failed[i]=True print res.status, res.reason, deleted data = res.read() count = count + 1 return count, seq if __name__ == "__main__": args = sys.argv[1:]; if not len(args) == 1: print "Usage: %s database_url" % sys.argv[0] sys.exit(1) repairer = Repairer(args[0]) repairer.repair()
smime.p7s
Description: S/MIME cryptographic signature
