On Oct 21, 2009, at 10:46 AM, Rajkumar S <[email protected]> wrote:
Hello,
I am using couch db 0.9.0 for storing logs from my mail server.
Logs are sent from mail servers to a RabbitMQ queue server. Log
insertions into couchdb is done by a python program, after fetching it
from RabbitMQ and converting to Json, using couchdb module (from
couchdb import *). I have a single document storing entire history of
the email transactions. I also have multiple RabbitMQ clients each
pulling from same queue and updating the same coudhdb. This means I
have to update the same document from different clients several times
during the life time of an email message.
To do this I use the message id of each mail transaction as it's key.
(this appears in every log entry) When a first log entry arrives I
check if a doc with that key is present in db, if not I create a new
doc with that key. When second log arrives I extract the doc, convert
it to a hash table in my program, merge the new log entry with the
hash table and update the doc with the updated hash table's json. If a
conflict occurs, the program retries, fetching the doc and updating it
and storing again till conflict is resolved.
This means for every write there is a corresponding read.
Currently I am running it as a pilot and just have a single server
logging to couchdb. I have about 0.75 GB per day right now, with
GET/PUT happening almost continuously (say 1 - 2 per second).
Previously I had a test server running and I tested couple of map
reduce using that DB (about 5 mb)
Now after logging from a single production machine I am not able to
run a single view so far. I get the following error if I wait long
enough:
Error: case_clause
{{bad_return_value,{os_process_error,"OS process timed out."}},
{gen_server,call,
[<0.436.0>,
{prompt,[<<"rereduce">>,
[<<"function(keys, values)\n{\n return
values;\n}">>],.....
I suspect "return values;" is the culprit here. That's pretty much
guranteed to cause you problems as the data grows recursing up the
tree. 0.10.0 even detects this and throws an error to avoid precisely
this issue.
It's been discussed a couple times on the list but my stomach is
rumbling.
I have changed os_process_timeout to 50000, removed the reduce part
but even after about 6 hours my map is not yet finished. Currently the
db size is 3.6G
The map function I am using is:
function(doc) {
if ("msgtype" in doc){
if (doc.msgtype == "allow"){
if ((doc.event == "action_allowed_ip") || (doc.event ==
"action_allow_new")){
result = {};
ip = doc.parameters.client_address;
result["helo"] = doc.parameters.helo_name;
result["event"] = doc.event;
result["timestamp"] = doc.timestamp;
result["id"] = doc._id;
result["from"] = doc.parameters.sender;
result["to"] = doc.parameters.recipient;
emit (ip,result);
}
}
}
}
Top shows that couchjs is most active process and it shows the
following line right now,
11410 root 20 0 90752 27m 752 R 76 0.7 1235:05 couchjs
My hardware is Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz, 4Gig RAM
and one SATA hard disk. I do not think this is the expected
performance of couchdb, so is there some thing I am doing wrong? Any
tips to enhance the performance to acceptable levels?
thanks and much regards,
raj
