Re: Cassandra HAProxy

2010-08-30 Thread Dave Viner
FWIW - we've been using HAProxy in front of a cassandra cluster in production and haven't run into any problems yet. It sounds like our cluster is tiny in comparison to Anthony M's cluster. But I just wanted to mentioned that others out there are doing the same. One thing in this thread that I

Re: Cassandra HAProxy

2010-08-30 Thread Edward Capriolo
On Mon, Aug 30, 2010 at 12:40 PM, Dave Viner davevi...@pobox.com wrote: FWIW - we've been using HAProxy in front of a cassandra cluster in production and haven't run into any problems yet.  It sounds like our cluster is tiny in comparison to Anthony M's cluster.  But I just wanted to mentioned

Re: Cassandra HAProxy

2010-08-29 Thread Benjamin Black
On Sun, Aug 29, 2010 at 11:04 AM, Anthony Molinaro antho...@alumni.caltech.edu wrote: I don't know it seems to tax our setup of 39 extra large ec2 nodes, its also closer to 24000 reqs/sec at peak since there are different tables (2 tables for each read and 2 for each write) Could you

Re: Cassandra HAProxy

2010-08-29 Thread Ming Fang
Sent from my iPhone On Aug 29, 2010, at 3:20 PM, Benjamin Black b...@b3k.us wrote: On Sun, Aug 29, 2010 at 11:04 AM, Anthony Molinaro antho...@alumni.caltech.edu wrote: I don't know it seems to tax our setup of 39 extra large ec2 nodes, its also closer to 24000 reqs/sec at peak since

Re: Cassandra HAProxy

2010-08-29 Thread Anthony Molinaro
On Sun, Aug 29, 2010 at 12:20:10PM -0700, Benjamin Black wrote: On Sun, Aug 29, 2010 at 11:04 AM, Anthony Molinaro antho...@alumni.caltech.edu wrote: I don't know it seems to tax our setup of 39 extra large ec2 nodes, its also closer to 24000 reqs/sec at peak since there are different

Cassandra HAProxy

2010-08-28 Thread Mark
I will be loadbalancing between nodes using HAProxy. Is this recommended? Also is there a some sort of ping/health check uri available? Thanks

Re: Cassandra HAProxy

2010-08-28 Thread Mark
On 8/28/10 11:20 AM, Benjamin Black wrote: no and no. On Sat, Aug 28, 2010 at 10:28 AM, Markstatic.void@gmail.com wrote: I will be loadbalancing between nodes using HAProxy. Is this recommended? Also is there a some sort of ping/health check uri available? Thanks Also, what would

Re: Cassandra HAProxy

2010-08-28 Thread Benjamin Black
Because you create a bottleneck at the HAProxy and because the presence of the proxy precludes clients properly backing off from nodes returning errors. The proper approach is to have clients maintain connection pools with connections to multiple nodes in the cluster, and then to spread requests

Re: Cassandra HAProxy

2010-08-28 Thread Anthony Molinaro
I think maybe he thought you meant put a layer between cassandra internal communication. There's no problem balancing client connections with haproxy, we've been pushing several billion requests per month through haproxy to cassandra. we use mode tcp balance leastconn server local

Re: Cassandra HAProxy

2010-08-28 Thread Benjamin Black
munin is the simplest thing. There are numerous JMX stats of interest. As a symmetric distributed system, you should not expect to monitor Cassandra like you would a web server. Intelligent clients use connection pools and react to current node behavior in making choices of where to send

Re: Cassandra HAProxy

2010-08-28 Thread Benjamin Black
On Sat, Aug 28, 2010 at 2:34 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: I think maybe he thought you meant put a layer between cassandra internal communication. No, I took the question to be about client connections. There's no problem balancing client connections with haproxy,

Re: Cassandra HAProxy

2010-08-28 Thread Joe Stump
On Aug 28, 2010, at 12:29 PM, Mark wrote: Also, what would be a good way of monitoring the health of the cluster? We use Ganglia. I believe failover is usually built into clients. Not sure why using HAProxy or LVS wouldn't be a good option though. I used to use it with MySQL slaves with much

Re: Cassandra HAProxy

2010-08-28 Thread Mark
On 8/28/10 2:44 PM, Benjamin Black wrote: On Sat, Aug 28, 2010 at 2:34 PM, Anthony Molinaro antho...@alumni.caltech.edu wrote: I think maybe he thought you meant put a layer between cassandra internal communication. No, I took the question to be about client connections. There's no problem