I was trying nodeload but could not generate the load I need to trigger the problem. I attached the script. Can you tell me how to change the script to get to the load I need to trigger the problem?
The attached script was making about 150 request per minute. Thanks, Marco On 11 May 2012 14:26, Robert Newson <rnew...@apache.org> wrote: > Can you reproduce this behavior with other benchmarking tools? ab, > nodeload, etc? > > B. > > On 11 May 2012 14:18, Marco Monteiro <ma...@textovirtual.com> wrote: > > Each node.js process had multiple concurrent requests. I just tried with > > sequential requests and the problem persists. > > > > So, now I have 8 node.js processes each sending one write request only > > after the previous when is done. And the problem remains. > > > > The machine is not under any kind of huge load. Both top and iostat > report > > less than 10% machine use. The machines have 8 Core Xeon with 4 > > 10000 rpm hard disks in raid 10 and 16 Gb.of RAM. > > > > Note that I'm testing with less than 500 requests per second, at the > > moment. > > > > One more thing: when the problem happens, it's not that the database > becomes > > slow. It just drops the requests. And reads also fail. For example, > trying > > to > > use Futon I get a "connection was reset" message from firefox. > > > > This is on CouchDB 1.2. I'm going to try 1.1.1 next. > > > > Thanks, > > Marco > > > > On 11 May 2012 13:56, Robert Newson <rnew...@apache.org> wrote: > > > >> Perhaps CouchDB on this particular hardware isn't fast enough to cope > >> with 4,000 writes per second? > >> > >> Does your node.js test send every update asynchronously or is it > >> carefully controlling qps? For what it's worth, I've benchmarked > >> successfully using a node.js library called nodeload > >> (https://github.com/benschmaus/nodeload). It's been a while since I > >> last used it, and node has changed a few dozen times since then, but > >> it was pretty solid and sane when I was using it. > >> > >> B. > >> > >> On 11 May 2012 13:48, Marco Monteiro <ma...@textovirtual.com> wrote: > >> > Thanks, Robert. > >> > > >> > Disabling delayed commits did make the problem start later, but it is > >> still > >> > there. > >> > > >> > It's funny that the first think that I checked when I first saw this > >> > problem was to > >> > make sure that delayed commits where enabled. > >> > > >> > Thanks, > >> > Marco > >> > > >> > On 11 May 2012 13:20, Robert Newson <rnew...@apache.org> wrote: > >> > > >> >> The first thing is to ensure you have disabled delayed commits; > >> >> > >> >> curl -XPUT -d '"false" localhost:5984/_config/couchdb/delayed_commits > >> >> > >> >> This is the production setting anyway (though not the default because > >> >> of complaints from incompetent benchmarkers). This will ensure an > >> >> fsync for each write and, as a consequence, will greatly smooth your > >> >> insert performance. Since you said you were inserting concurrently, > >> >> you should not experience a slowdown either. > >> >> > >> >> B. > >> >> > >> >> On 11 May 2012 02:42, Marco Monteiro <ma...@textovirtual.com> wrote: > >> >> > Hello! > >> >> > > >> >> > I'm running a load test on CouchDB. I have a cluster of 8 node.js > >> servers > >> >> > writing to > >> >> > CouchDB. They write about 30000 documents per minute (500 per > second). > >> >> > There are > >> >> > multiple concurrent requests form each server. There are no > updates: > >> >> > documents are > >> >> > created and not modified. > >> >> > > >> >> > I first tried CouchDB 1.1.1 from Debian 6.4 apt repo. After a few > >> >> minutes, > >> >> > CouchDB > >> >> > starts freezing for a period of 1 to 3 seconds about every 10 > >> seconds. It > >> >> > keeps this > >> >> > behaviour for some time and eventually it starts freezing more > >> frequently > >> >> > and for longer > >> >> > periods. When the database has about 1.5 million documents, > couchdb is > >> >> > freezing for > >> >> > more than 5 seconds each time. > >> >> > > >> >> > I then tried CouchDB 1.2, from build-couch. The freezes happen > with it > >> >> > also, but the > >> >> > behavior is much worse: in less than one minute it's freezing for 5 > >> >> seconds > >> >> > or more, > >> >> > and it spends more time not doing anything than working. > >> >> > > >> >> > When testing with 1.1.1 I was writing only to one database. With > 1.2 I > >> >> > tried with one database > >> >> > and then with multiple databases but the problem was exactly the > same. > >> >> > > >> >> > The documents have about 10 properties, only numbers or string and > the > >> >> > strings are small > >> >> > (about 20 chars each). The document IDs are generated in the app > and > >> have > >> >> > the format > >> >> > > >> >> > <milliseconds since epoch>-<random 16 chars string> > >> >> > > >> >> > When CouchDB freezes, it's processor use (from top) goes to zero. > It > >> does > >> >> > not reply to read or write > >> >> > requests. The disk does not seem to be the problem as iostat > reports > >> >> near > >> >> > 0% utilization. > >> >> > CPU is mostly idle, and from the 16 GB of RAM, some of it is free > and > >> is > >> >> > not even used to > >> >> > cache disk. > >> >> > > >> >> > There are no error message in Couchdb log. > >> >> > > >> >> > I tried this in two different machines and the problem is the same > in > >> >> both. > >> >> > > >> >> > I did not change anything in the configuration files expect > changing > >> the > >> >> > database dir to use > >> >> > a RAID partition. > >> >> > > >> >> > Anyone has any idea of what the problem could be? > >> >> > > >> >> > Any help solving this issue is greatly appreciated. > >> >> > > >> >> > Thanks, > >> >> > Marco > >> >> > >> >
