Okay, I think I've confirmed this theory. Thanks to your guidance I managed to create a python script that can reproduce the problem on my macbook: http://saucelabs.com/couch-bug.py
(To run it, you need the latest trunk version of couchdb-python.) Looking at our logs, we started seeing this problem just after we crossed the threshold to >100 dbs. And very frequent reads to the problematic database fixed the issue, because they made it always more recently active than most of the others. - Steve On Mon, Apr 27, 2009 at 9:40 AM, Steven Hazel <[email protected]> wrote: > I think we do have a lot of active databases -- we have a view that is run > frequently against each of our 137 databases. > > - Steve > > > On Mon, Apr 27, 2009 at 6:20 AM, Damien Katz <[email protected]> wrote: > >> It looks like the db server instance is being restarted before it commits >> the header, but it's odd you aren't seeing any errors on the console. If >> it's a crash it should dump an error to the console. >> >> One thing that occurs to me, do you have a lot of active databases (> 100) >> ? I think it's possible when you hit the max dbs open, CouchDB will shutdown >> an idle instance with pending deferred commit, which is a bug. >> >> -Damien >> >> >> >> On Apr 27, 2009, at 12:52 AM, Steven Hazel wrote: >> >> Hi, >>> >>> Last week we discovered that records in one of our CouchDB databases were >>> occasionally being reverted to the previous revision. We believe this is >>> due >>> to a bug in CouchDB 0.9 related to asynchronous writes. Turning off >>> asynchronous writes with the X-Couch-Full-Commit header fixes the problem >>> for us. I've written up the problem in more detail here: >>> http://saucelabs.com/couch-bug.html >>> >>> We haven't yet reproduced this problem outside our production >>> environment, >>> but can trigger it there on-demand. Although we have a work-around that >>> solves the immediate problem for us, we're happy to help out in tracking >>> down the underlying issue. Any advice on how to proceed would be >>> appreciated. >>> >>> -- >>> Steven Hazel >>> Sauce Labs >>> Cofounder and Director, Development >>> http://twitter.com/sahazel >>> >> >> > > > -- > Steven Hazel > Sauce Labs > Cofounder and Director, Development > http://twitter.com/sahazel > -- Steven Hazel Sauce Labs Cofounder and Director, Development http://twitter.com/sahazel
