Hi
I'm running a couchbase 2.5 cluster with 5 machines on aws
I have several times sees issues with rebalancing of the cluster while the
cluster is loaded. Normally I add more nodes to handle more load
What typically happens is that the cluster gets unstable, the rebalance
fails.
Then a node just goes down and a little while after it comes up again,
Normally I get messages like
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{{{badmatch,undefined},
[{ns_storage_conf,
in the log
The node just keeps on being unstable. It comes up again, bucket gets
loaded and then a little while after this just happens again and again.
This behaviour continues even if I remove all the load from the cluster and
don't try new rebalances. So basically I have a cluster without handling
any requests, but where a node just goes down, continuously
What also seems odd to me was that almost all memory was reported as used
in the ui > 95%, I tried removing a bucket, but memory usage just creeps
upto > 95% again
I have done a cbcollect on the nodes, and the files are here
https://s3-eu-west-1.amazonaws.com/tmp-idtargeting-public/couchbase/ip-10-33-38-143.collect.cb.zip
https://s3-eu-west-1.amazonaws.com/tmp-idtargeting-public/couchbase/ip-10-74-20-120.collect.cb.zip
https://s3-eu-west-1.amazonaws.com/tmp-idtargeting-public/couchbase/ip-10-74-23-89.collect.cb.zip
https://s3-eu-west-1.amazonaws.com/tmp-idtargeting-public/couchbase/ip-10-77-0-111.collect.cb.zip
https://s3-eu-west-1.amazonaws.com/tmp-idtargeting-public/couchbase/ip-10-78-1-25.collect.cb.zip
The node
https://s3-eu-west-1.amazonaws.com/tmp-idtargeting-public/couchbase/ip-10-33-38-143.collect.cb.zip
is the node that just kept on going down and I had to remove it from the
cluster
I have seen previous post on this mailing list that mentions that large
documents could cause malloc problems and which again could cause the
behaviour I have seen. But problem is I don't know what to look after in
the above files.
Any suggestion
Thanks
Niels
--
BinaryConstructors ApS
Vestergade 10a, 4th
1456 Kbh K
Denmark
phone: +4529722259
web: http://www.binaryconstructors.dk
mail: [email protected]
skype: nielsboldt
--
You received this message because you are subscribed to the Google Groups
"Couchbase" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
