Mix of all those things. Storing and retrieving attachments scales as O(log N) with the number of documents in the database, so no worries there. The throughput could be a bit better but I think it was fine for NPM (and it’s quite simple to cache heavily-used attachments). Compaction on the other hand needs to copy the current version(s) of each attachment to the new file, so that scales as O(N) and you need effectively 2x the disk space of all the attachments you store in a DB for it to complete successfully. Not great.
The other thing that really caused NPM trouble was attachment replication. That process exercises some unusual code paths and there have been fairly subtle bugs in there over the years which made the replication pretty finicky. Adam > On Sep 25, 2015, at 1:46 PM, Michael Power <[email protected]> wrote: > > Has anyone looked into the performance of attachments as the DB increases in > size? From my reading, npm needed to move away from couchdb attachments. > Its not clear to me if that was due to performance issues, replication > issues, or administrative issues with the very large db file. > > > > > On 9/22/15, 4:32 PM, "Adam Kocoloski" <[email protected]> wrote: > >> Yes, actually we do have something in the works related to COUCHDB-769 to >> offload attachments to an external object storage system. We preserve the >> CouchDB API so an end user can’t tell if the offloading is happening. I’m >> overdue to get the code posted, and it’s only a prototype so it’ll need some >> work, but glad to see there’s interest here. I’ll look to get this posted on >> Wednesday. Cheers, >> >> Adam
