On Jan 4, 2009, at 2:59 PM, Damien Katz wrote:
What you say is possible, but doesn't seem too likely as I don't
ever recall any report of a corrupt db on Linux, but before the
F_FILESYNC change in Erlang we would see reports of corruptions on
OS X due to power loss. Since the change, we've had no reports of
corrupt databases on OS X or Linux.
Have you seen the correlation between the # of pirates and global
warming? :)
There may be correlation here between the Erlang update and
corruption, but I don't see the causality - fsync() on linux should
only get as far as the on-device buffering if it does write buffering,
whereas if the OS X docs are truthful, the F_FILESYNC fcntl() pushs
the bits to the rust on the platter. Maybe they fixed something else
in that erlang update, or maybe the universe is smiling upon CouchDB
users or ....
I don't think anyone else sees update performance that slow (I'm
assuming these are fairly small documents?), I'd guess there is
something particular about your setups. Since it involves update
speed it wouldn't be the javascript engine. It's possible it's the
openssl interface, I've seen instances of wacked openssl installs
that slowed down Erlang but didn't crash it.
I have an idea.... try it. I bet you can't get better than that (6-8
inserts per second) on Mac OS X unless you have stupid-fast disks
(like maybe an SSD).
Even get a "back of the envelope" approx using time and curl - that
came in at the same rate as Ruby, JS and Java test programs.
Everyone who has tried my experiment on IRC has reported the same
numbers. I can get 6/sec on a laptop (2.5GHz core2 dual core MBP w/
4GB ram and a 7200rpm disk), and 8/sec on my desktop (a quad-core, 8
GB w/ raided 7200rpm disks, IIRC)
geir
-Damien
On Jan 4, 2009, at 1:35 PM, Geir Magnusson Jr. wrote:
I'm suspicious.
On an m1.small instance running 32-bit ubuntu 8 on ec2 running
trunk, I can get 85 inserts per second with the client running on
same slice, so no network hop. it's a single core instance, but
I'm not sure that would matter too much (i'll try later).
Now, this makes me wonder... the fsync() in linux isn't doing what
F_FILESYNC does in OS X presumably.... in which case what you
think is being done for you to work around the lack of an "off
switch" in the erlang VM isn't actually being done, and since no
one seems to have noticed, does it really matter? :)
To be fair, I'm going to try a better experiment on linux w/ more
cores and more clients...
geir
On Jan 4, 2009, at 12:45 PM, Jan Lehnardt wrote:
On 4 Jan 2009, at 14:09, Geir Magnusson Jr. wrote:
2) Is anyone using CouchDB in a manner that really requires this
level of data security? I appreciate having the *option* to turn
on a mode like this, but I don't think I need it all the time. I
can use RAID systems that give me battery-backed cache, or I can
make the design decision that I am happy to lose X seconds of
data in a tradeoff (e.g. do the deep fnctl() every X seconds.....)
Everybody. CouchDB doesn't have an `off`-switch. You just terminate
the Erlang VM. Without this design, this wouldn't be possible. There
has been no discussion about optionally making different trade-offs
for speed and against data integrity, but this is not off-the-map.
Cheers
Jan
--
