"1. Try to replicate the database in another CouchDB." I have done this to a couchdb 1.2 database successfully. FYI The Source DB is a couchdb 1.1.1.
I haven't done the other tests, but have tested replicating from the couchdb 1.2 database to the bigcouch install and got the same issue. Mike -----Original Message----- From: CGS [mailto:[email protected]] Sent: 13 April 2012 15:01 To: [email protected] Subject: Re: BigCouch - Replication failing with Cannot Allocate memory If you say so, Robert, I won't argue with you on that. I meant no offense, so, please, accept my apologies if I crossed the line. It's all your's from now on. Mike, please, ignore my suggestion. Sorry for interfering. Good luck! CGS On Fri, Apr 13, 2012 at 3:19 PM, Robert Newson <[email protected]> wrote: > I think you should point out that "My idea behind these tests is that > it may be that your database may be > corrupted (or seen as corrupted by BigCouch at the second test) and what > you get is just garbage at a certain document. " is based on no > evidence. Nor, if it were true, would it necessarily explain the > observed behavior either. > > It would be useful if we could all stick to asserting only things we > know to be true or have reasonable grounds to believe are true. > Unfounded speculation, though offered sincerely, is not helpful on a > mailing list intended to provide assistance. > > Thanks, > B. > > On 13 April 2012 13:55, CGS <[email protected]> wrote: > > Hi Mike, > > > > I haven't used BigCouch by now and that's why I haven't said anything by > > now. Still, giving a thought of what may occur there, I propose few tests > > if you have time: > > 1. Try to replicate the database in another CouchDB. > > 2. If 1 passes, try to replicate to only one node at the time. > > 3. If 2 passes, increase the pool of nodes with 1 and repeat the > > replication (for sure it will fail at all 3 nodes at the time). > > > > My idea behind these tests is that it may be that your database may be > > corrupted (or seen as corrupted by BigCouch at the second test) and what > > you get is just garbage at a certain document. That's why I proposed the > > first test. The second test is to see if any of the nodes has a problem > in > > configuration (or if there is any incompatibility in between your CouchDB > > and BigCouch in manipulating your docs). Finally, the third test is to > see > > if server/node resources limit the number of replications (and at how > many > > it starts to fail). > > > > Can you also check the size of the shards at tests 2 and 3? > > > > If you consider that these tests are irrelevant, please, ignore my > > suggestion. > > > > CGS > > > > > > > > On Fri, Apr 13, 2012 at 1:27 PM, Mike Kimber <[email protected]> wrote: > > > >> I upped the memory to 6GB on each of the nodes and got exactly the same > >> issue in the same time frame i.e. the increased RAM did not seem to by > me > >> any additional time. > >> > >> Mike > >> > >> -----Original Message----- > >> From: Robert Newson [mailto:[email protected]] > >> Sent: 12 April 2012 19:34 > >> To: [email protected] > >> Subject: Re: BigCouch - Replication failing with Cannot Allocate memory > >> > >> 2GB total ram does sound tight. I can only compare to high volume > >> production clusters which have much more ram than this. Given that > >> beam.smp wanted 1.4 gb and you have 2gb total, do you know where the > >> rest one? To couchjs processes, by chance? If so, you can reduce the > >> maximum size of that pool in config, I think the default is 50. > >> > >> On 12 April 2012 18:32, Mike Kimber <[email protected]> wrote: > >> > Ok, I have 3 nodes all load balanced with HAproxy: > >> > > >> > Centos 5.8 (Virtualised) > >> > 2 Cores > >> > 2GB RAM > >> > > >> > I'm trying to replicate about 75K documents which total 6GB when > >> compacted (0n Couchdb 1.2 which has compression turned on). I'm told > they > >> are fairly large documents. > >> > > >> > When it goes pear shaped Vsmstat starts using a lot of memory: > >> > > >> > procs -----------memory---------- ---swap-- -----io---- --system-- > >> -----cpu------ > >> > r b swpd free buff cache si so bi bo in cs us > sy > >> id wa st > >> > 1 2 570576 8808 140 7208 2998 2249 3154 2249 1234 569 1 > 6 > >> 2 91 0 > >> > 0 2 569656 9156 156 7504 2330 1899 2405 1904 1246 595 1 > 5 > >> 9 85 0 > >> > 1 1 575412 9516 236 14928 1549 2261 3242 2261 1237 593 1 > 7 > >> 1 91 0 > >> > 0 2 607092 13220 168 8156 3772 9012 3871 9017 1284 714 1 > 10 > >> 4 85 0 > >> > 1 0 444336 857004 220 10212 5781 0 6202 0 1574 1010 13 > 7 > >> 33 47 0 > >> > 1 0 442176 870684 428 11052 2049 0 2208 140 2561 1541 17 > 8 > >> 49 26 0 > >> > 0 0 442176 813140 460 11968 170 0 348 0 2672 1565 25 > 9 > >> 61 4 0 > >> > 0 1 442176 744972 484 12224 5440 0 5493 7 2432 900 8 > 4 > >> 49 40 0 > >> > 0 1 442176 714048 484 12296 4547 0 4547 0 1799 827 4 > 2 > >> 50 44 0 > >> > 0 1 442176 686304 496 12688 5128 0 5222 0 1696 999 9 > 2 > >> 50 40 0 > >> > 0 3 444000 8712 444 12876 299 368 331 380 1294 188 22 > 20 > >> 36 23 0 > >> > 0 3 469340 10040 116 7336 29 5087 74 5090 1232 268 3 > 22 > >> 0 75 0 > >> > 1 2 584356 10220 124 6744 11367 28722 11370 28722 1643 1300 5 > >> 19 17 59 0 > >> > 0 1 624908 10640 132 7036 6518 12879 6590 12884 1296 717 3 > 10 > >> 29 58 0 > >> > 0 2 652556 10948 252 14776 3799 9494 5459 9494 1294 646 2 > 9 > >> 32 57 0 > >> > 0 2 677784 10648 244 14528 3819 8196 3819 8201 1274 588 2 > 7 > >> 30 61 0 > >> > 0 2 688460 9512 212 8224 3013 4522 3125 4522 1379 519 2 > 7 > >> 6 84 0 > >> > 0 3 699164 9888 208 8468 2192 4014 2228 4014 1302 495 1 > 6 > >> 11 83 0 > >> > 2 0 713104 9004 144 9192 2606 4490 2848 4490 1350 487 1 > 8 > >> 16 75 0 > >> > > >> > It only ever takes out one node at a time and the other nodes seem to > be > >> doing very little while the one node is running out of memory. > >> > > >> > If I kick it off again it processed some more and then spikes the > memory > >> and fails > >> > > >> > Thanks > >> > > >> > Mike > >> > > >> > PS: hope you enjoyed you couchdb get together! > >> > > >> > -----Original Message----- > >> > From: Robert Newson [mailto:[email protected]] > >> > Sent: 12 April 2012 17:28 > >> > To: [email protected] > >> > Subject: Re: BigCouch - Replication failing with Cannot Allocate > memory > >> > > >> > What kind of load were you putting the machine on? > >> > > >> > On 12 April 2012 17:24, Robert Newson <[email protected]> wrote: > >> >> Could you show your vm.args file? > >> >> > >> >> On 12 April 2012 17:23, Robert Newson <[email protected]> wrote: > >> >>> Unfortunately your request for help coincided with the two day > CouchDB > >> >>> Summit. #cloudant and the Issues tab on cloudant/bigcouch are other > >> >>> ways to get bigcouch support, but we happily answer queries here > too, > >> >>> when not at the Model UN of CouchDB. :D > >> >>> > >> >>> B. > >> >>> > >> >>> On 12 April 2012 17:10, Mike Kimber <[email protected]> wrote: > >> >>>> Looks like this isn't the right place based on the responses so > far. > >> Shame I hoped this was going to help solve our index/view rebuild times > etc. > >> >>>> > >> >>>> Mike > >> >>>> > >> >>>> -----Original Message----- > >> >>>> From: Mike Kimber [mailto:[email protected]] > >> >>>> Sent: 10 April 2012 09:20 > >> >>>> To: [email protected] > >> >>>> Subject: BigCouch - Replication failing with Cannot Allocate memory > >> >>>> > >> >>>> I'm not sure if this is the correct place to raise an issue I am > >> having with replicating a standalone couchdb 1.1.1 to a 3 node BigCouch > >> cluster? If this is not the correct place please point me in the right > >> direction if it is then any one have any ideas why I keep getting the > >> following error message when I kick of a replication; > >> >>>> > >> >>>> eheap_alloc: Cannot allocate 1459620480 bytes of memory (of type > >> "heap"). > >> >>>> > >> >>>> My set-up is: > >> >>>> > >> >>>> Standalone couchdb 1.1.1 running on Centos 5.7 > >> >>>> > >> >>>> 3 Node BigCouch cluster running on Centos 5.8 with the following > >> local.ini overrides pulling from the Standalone couchdb (78K documents) > >> >>>> > >> >>>> [httpd] > >> >>>> bind_address = XXX.XX.X.XX > >> >>>> > >> >>>> [cluster] > >> >>>> ; number of shards for a new database > >> >>>> q = 9 > >> >>>> ; number of copies of each shard > >> >>>> n = 1 > >> >>>> > >> >>>> [couchdb] > >> >>>> database_dir = /other/bigcouch/database > >> >>>> view_index_dir = /other/bigcouch/view > >> >>>> > >> >>>> The error is always generate on the third node in the cluster and > the > >> server basically max's out on memory before hand. The other nodes seem > to > >> be doing very little, but are getting data i.e. the shard sizes are > >> growing. I've put the copies per shard down to 1 as currently I'm not > >> interested in resilience. > >> >>>> > >> >>>> Any help would be greatly appreciated. > >> >>>> > >> >>>> Mike > >> >>>> > >> >
