The compactor is written to flush batches of docs every 5K bytes and
then write a header out ever 5M bytes (assuming default batch sizes).
Its important to remember that this judged against #doc_info{} records
which don't contain a full doc body. For documents with relatively few
revisions we're looking at (rough guess) ~100 bytes per record, which
is going to give us 50K docs per header commit. Seeing as the OP
mentions lots of attachments this could give us a relatively large gap
in the file to search for a header.On Mon, Sep 24, 2012 at 11:17 AM, Tim Tisdall <[email protected]> wrote: > Since this is the result of a compaction, shouldn't the header be at > the beginning of the file? (just testing my knowledge on how all this > works...) > > -Tim > > On Mon, Sep 24, 2012 at 12:09 PM, Robert Newson <[email protected]> wrote: >> That does imply that the last valid header is a long way back up the >> file, though. >> >> On 24 September 2012 17:00, Paul Davis <[email protected]> wrote: >>> I'd ignore the snappy error for now. There's no way this thing ran for >>> an hour and then suddenly hit an error in that code. If this is like a >>> bug I've seen before the reason that this runs out of RAM is due to >>> the code that's searching for a header not releasing binary ref counts >>> as it should be. >>> >>> The quickest way to fix this would probably be to go back and update >>> recover-couchdb to recognize the new disk format. Although that gets >>> harder now that snappy compression is involved. >>> >>> On Mon, Sep 24, 2012 at 10:32 AM, Dave Cottlehuber <[email protected]> >>> wrote: >>>> On 24 September 2012 15:02, Robert Newson <[email protected]> wrote: >>>>> {badmatch,{error,snappy_nif_not_loaded} makes me wonder if this 1.2 >>>>> installation is even right. >>>>> >>>>> Can someone enlighten me? Is it possible to get this error spuriously? >>>> >>>> No. I'd be keen to see a bit of logfiles to understand what's not working. >>>> >>>>> Does running out of RAM cause erlang to unload NIF's? >>>> >>>> I don't think so on Windows. >>>> >>>> There's an R15B01 based build here: >>>> https://www.dropbox.com/sh/jeifcxpbtpo78ak/GG9fjWOyDt/Snapshots/20120524 >>>> that has a fix for a more recent version of Windows server than I have >>>> to address one NIF loading error, although there are a number of >>>> possible causes. >>>> >>>> @Rudi can you give this a go & report back? >>>> >>>> A+ >>>> Dave
