-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Roger Binns wrote: > As a guess should I perhaps also order my documents so that the value > emitted in a view (in most cases one value always from a key named "name") > is also in sorted order? With some SQL magic during document generation > should be able to do that.
So I did that. The resulting view file is 15GB (was 27GB) and took 43 minutes to generate (was 75 minutes). Disk utilization started at 3% and climbed to 10% at the end (was 30%). couchjs remained at ~25% cpu consumption and CouchDB was typically at 100% of one core (was around 110%). The obvious conclusion is the append only file format for views is a really bad thing. As a format it is very good for data integrity but not efficient for performance or size. I do care about integrity of my documents, but the thing I care most about for my views is performance. (The major reason for a view is that it is more performant than visiting every document.) Losing data from a view is no big deal - it can be regenerated (assuming generation doesn't remain as slow as it is today :-). Consequently I'd suggest using a different file format for views that is space and performance oriented, and the only data integrity feature being the ability to tell if some or all of it is inconsistent (eg if there was abrupt shutdown). Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAktMJkYACgkQmOOfHg372QTokwCfZgm1LZnNGKO2nCNGa8C6uwoR OpMAmwRTlTOrPm5Cjrkj8fyWR4xc4k7w =F59/ -----END PGP SIGNATURE-----
