moved this to https://issues.apache.org/jira/browse/COUCHDB-271
so it doesn't fall between the cracks.
Cheers
Jan
--
On 27 Feb 2009, at 19:32, Damien Katz wrote:
Oh yeah, one more option that is kind of crazy is to spawn a small
external child process for file io. It would be a very small simple
process that opens a file and responds to read/write commands from
the erlang server. Then we can implement exactly the low level apis
and caching behavior desired. The cost is extra IPC, but that should
be small compare the the cost of a blown file cache.
-Damien
On Feb 27, 2009, at 1:23 PM, Damien Katz wrote:
The problem is we don't get access to the low level apis or flags
passed in to the OS unless Erlang chooses to expose it. We have
similar problems with compaction on windows because we need special
flags to give us unix file semantics.
To fix this, we'll either need the Erlang VM changed or use our own
Erlang file driver interface.
-Damien
On Feb 26, 2009, at 8:25 PM, Adam Kocoloski wrote:
Hi, I've noticed that compacting large DBs pretty much kills any
filesystem caching benefits for CouchDB. I believe the problem is
that the OS (Linux 2.6.21 kernel in my case) is caching blocks
from the .compact file, even though those blocks won't be read
again until compaction has finished. In the meantime, the portion
of the cache dedicated to the old DB file shrinks and performance
really suffers.
I think a better mode of operation would be to advise/instruct the
OS not to cache any portion of the .compact file until we're ready
to replace the main DB. On Linux, specifying the
POSIX_FADV_DONTNEED option to posix_fadvise() seems like the way
to go:
http://linux.die.net/man/2/posix_fadvise
This link has a little more detail and a usage example:
http://insights.oetiker.ch/linux/fadvise.html
Of course, POSIX_FADV_DONTNEED isn't really available from inside
the Erlang VM. Perhaps the simplest approach would be to have a
helper process that we can spawn which calls that function (or its
equivalent on a non-Linux OS) periodically during compaction? I'm
not really sure, but I wanted to get this out on the list for
discussion. Best,
Adam
