Were there any view indexes on that DB? My suspicion is that couch_view_group
misses a step somewhere in its dance and keeps a reference to an old #db{}
Adam
On Sep 14, 2010, at 4:46 PM, [mRg] wrote:
> It was just on a DB with a lot of writes (documents having lots of
> revisions, rather that lots of new docs), the database grows as expected but
> after just doing a simple compact on the db the old files still seem to be
> on disk as reported previously.
>
> It seems its been around a while now as (as far as we can see) 0.11 / 1.0
> and 1.0.1 all have the same issue, when we replicated the same data over to
> the newer versions the same identical problem happened.
>
> On 14 September 2010 14:49, Robert Newson <[email protected]> wrote:
>
>> This was mentioned at CouchCamp, several developers suspect there's a
>> subtle bug in the code where file descriptors are not being closed
>> under some conditions.
>>
>> Did the OP identify a particular sequence that led here?
>>
>> B.
>>
>> On Mon, Sep 13, 2010 at 6:25 PM, Juan Jose Comellas <[email protected]>
>> wrote:
>>> I'm pretty certain it's a bug in CouchDB. I haven't checked the code, but
>>> what is happening is that CouchDB is keeping file descriptors
>> corresponding
>>> to deleted files open, probably after compacting a database. I'm
>> inferring
>>> this based on the fact that with lsof I saw several cases of two files
>> with
>>> the same name (one deleted and one not) opened by CouchDB.
>>>
>>>
>>> On Mon, Sep 13, 2010 at 7:28 PM, [mRg] <[email protected]> wrote:
>>>
>>>> Is this an issue with the OS or with Couch/Beam itself ?
>>>>
>>>> On 10 September 2010 16:57, Juan Jose Comellas <[email protected]>
>>>> wrote:
>>>>
>>>>> It also happened to me with Ubuntu 8.04 and CouchDB 0.10.x. The only
>> way
>>>> I
>>>>> found to solve the problem was by restarting CouchDB.
>>>>>
>>>>>
>>>>> On Thu, Sep 9, 2010 at 10:25 AM, [mRg] <[email protected]> wrote:
>>>>>
>>>>>> [CouchDB Version : 0.11]
>>>>>> [OS: RHEL 5.5]
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> We are seeing some odd behaviour with CouchDb not releasing files
>> after
>>>>> it
>>>>>> has compacted them meaning we are seeing disk space continue to
>>>> increase
>>>>>> until the service is restarted even though we have regular
>> compaction /
>>>>>> cleanup.
>>>>>>
>>>>>> Typical scenario : File system alert for disk space ..
>>>>>>
>>>>>> df says 5.5.GB used ..
>>>>>>
>>>>>> df -h | grep -P 'Filesystem|db'
>>>>>>
>>>>>> Filesystem Size Used Avail Use% Mounted on
>>>>>>
>>>>>> 8.0G 5.5G 2.2G 72% /var/lib/db
>>>>>>
>>>>>>
>>>>>> .. but the files on the disk (and in Futon) say only 793MB is being
>>>> used
>>>>> ..
>>>>>>
>>>>>>
>>>>>> du -sh /var/lib/db/*
>>>>>>
>>>>>> 793M /var/lib/db/couchdb
>>>>>>
>>>>>>
>>>>>> .. running lsof shows the files marked as deleted but still there
>> (I'm
>>>>>> assuming beam still has some kind of handle still open to them) ..
>>>>>>
>>>>>>
>>>>>> lsof | grep -P 'COMMAND|/var/lib/db/couchdb/'
>>>>>>
>>>>>> COMMAND PID USER FD TYPE DEVICE
>> SIZE
>>>>>> NODE NAME
>>>>>>
>>>>>> beam.smp 21352 couchdb 12u REG 253,10
>> 4183
>>>>>> 1210374
>>>>>>
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.active_context_design/8efe9584a944f4b24b97f134f541d80f.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 16u REG 253,10
>> 612039080
>>>>>> 1204227 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 20u REG 253,10
>> 180699242
>>>>>> 1204237 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 21u REG 253,10
>> 3133540
>>>>>> 2023429 /var/lib/db/couchdb/thesaurus.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 25u REG 253,10
>> 71422052
>>>>>> 2023428 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 26u REG 253,10
>> 71372900
>>>>>> 2056193 /var/lib/db/couchdb/active_context_schedule.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 28u REG 253,10
>> 621252717
>>>>>> 1204238 /var/lib/db/couchdb/active_context.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 29u REG 253,10
>> 1740903
>>>>>> 90113 /var/lib/db/couchdb/mapping_cache.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 30u REG 253,10
>> 3133540
>>>>>> 1204253 /var/lib/db/couchdb/thesaurus.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 31u REG 253,10
>> 2318439
>>>>>> 1204240 /var/lib/db/couchdb/mappings.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 32u REG 253,10
>> 983150
>>>>>> 2004994
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.mappings_design/f9964c50b95c167c8ac3d8d3b4c76a4b.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 33u REG 253,10
>> 3133540
>>>>>> 2056194 /var/lib/db/couchdb/thesaurus.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 34u REG 253,10
>> 70742116
>>>>>> 1204231 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 44u REG 253,10
>> 1740903
>>>>>> 2023427 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 45u REG 253,10
>> 65396836
>>>>>> 1994756 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 47u REG 253,10
>> 71229540
>>>>>> 1204243 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 48u REG 253,10
>> 240254078
>>>>>> 1204226 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 49u REG 253,10
>> 2232423
>>>>>> 1204241 /var/lib/db/couchdb/mappings.couch
>>>>>>
>>>>>> beam.smp 21352 couchdb 51u REG 253,10
>> 96903353
>>>>>> 1210371
>>>>>>
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.active_context_design/b406f7bb6d01b6d50324f14cde588324.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 52u REG 253,10
>> 1192034
>>>>>> 1249282
>>>>>>
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.mapping_cache_design/8bbf80a6e1ca8f2c71ebc90c730e7190.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 53u REG 253,10
>> 1622135
>>>>>> 2004993
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.mappings_design/1ff69004b575837aae9d4147162197d4.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 54u REG 253,10
>> 4172
>>>>>> 1208327
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.thesaurus_design/73fe79b20ff06e341257fa78231ca87e.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 55u REG 253,10
>> 13496483
>>>>>> 1208325
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.thesaurus_design/fcc08d6e41c54c0d470c5984ea9eb6e3.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 56u REG 253,10
>> 12959902
>>>>>> 1208326
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.thesaurus_design/c06195da19f34ced37fa1ab91e663c73.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 57u REG 253,10
>> 70254692
>>>>>> 1204236 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 58u REG 253,10
>> 581730
>>>>>> 1210373
>>>>>>
>>>>>>
>>>>>
>>>>
>> /var/lib/db/couchdb/.active_context_schedule_design/c64c047dcff74ea2122356efb1042759.view
>>>>>>
>>>>>> beam.smp 21352 couchdb 67u REG 253,10
>> 1818727
>>>>>> 1204247 /var/lib/db/couchdb/mapping_cache.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 78u REG 253,10
>> 609905146
>>>>>> 1204234 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 80u REG 253,10
>> 545919085
>>>>>> 1994753 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 81u REG 253,10
>> 71032932
>>>>>> 1204249 /var/lib/db/couchdb/active_context_schedule.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 84u REG 253,10
>> 548866111
>>>>>> 1204239 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 85u REG 253,10
>> 564612402
>>>>>> 1204233 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 87u REG 253,10
>> 615996179
>>>>>> 1204228 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>> beam.smp 21352 couchdb 102u REG 253,10
>> 561111149
>>>>>> 2023425 /var/lib/db/couchdb/active_context.couch (deleted)
>>>>>>
>>>>>>
>>>>>> Has anyone seen this behaviour before or knows of a way around this
>> ?
>>>>>> Currently we have to restart the couchdb service in order to restore
>>>> the
>>>>>> disk space.
>>>>>>
>>>>>> Any help/advice is much appreciated.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Stephen
>>>>>>
>>>>>
>>>>
>>>
>>
