Re: Recommended OS
I haven't run Cassandra in production myself, but for other high load Java based servers I've had really good scaling success with OpenSolaris. In particular I've used Joyent's SmartOS which has the additional advantage of bursting to cover brief periods of exceptional load. On Tue, Feb 11, 2014 at 9:25 AM, Brust, Corwin [Hollander] corwin.br...@hollanderparts.com wrote: This /is/ our first cluster. We've upgraded one-over-one since 2.0.2 (our initially deployed version), doing rolling updates across the rings. No especial resource tuning save turning off SELinux and the usual (disabling swap, separate disk for commit logs, data and the OS). *From:* Keith Wright [mailto:kwri...@nanigans.com] *Sent:* Monday, February 10, 2014 4:35 PM *To:* user@cassandra.apache.org *Cc:* Don Jackson; Dave Carroll *Subject:* RE: Recommended OS Is this your first cluster? Have you run older versions of Cassandra? Any specific resource tuning? Thanks all. We are unable to bootstrap nodes and are considering creating a fresh cluster in hopes this is some how data related. On Feb 10, 2014 5:33 PM, Brust, Corwin [Hollander] corwin.br...@hollanderparts.com wrote: We're running C* 2.0.5 under CentOS 6.5 and have not noticed anything like you describe. We have just a couple of pre-production rings (Dev and Test) meaning nothing we have has received particularly intense utilization. Corwin *From:* Keith Wright [mailto:kwri...@nanigans.com kwri...@nanigans.com] *Sent:* Monday, February 10, 2014 2:09 PM *To:* user@cassandra.apache.org *Cc:* Don Jackson; Dave Carroll *Subject:* Re: Recommended OS We are running on CentOS 6.4 but an upgrade to 6.5 caused packets to backup on the net queue causing HUGE load spikes and cluster meltdown. Ultimately we reverted. Have others seen this? Are others running CentOS 6.4/6.5? Thanks *From: *Sholes, Joshua joshua_sho...@cable.comcast.com *Reply-To: *user@cassandra.apache.org user@cassandra.apache.org *Date: *Monday, February 10, 2014 at 1:56 PM *To: *user@cassandra.apache.org user@cassandra.apache.org *Cc: *Don Jackson djack...@nanigans.com, Dave Carroll dcarr...@nanigans.com *Subject: *Re: Recommended OS What issues are you running into with CentOS 6.4/5? I'm running 1.2.8 on CentOS 6.3 and Java 1.7.0-25, and about to test with 1.7.latest. -- Josh Sholes *From: *Keith Wright kwri...@nanigans.com *Reply-To: *user@cassandra.apache.org user@cassandra.apache.org *Date: *Monday, February 10, 2014 at 1:50 PM *To: *user@cassandra.apache.org user@cassandra.apache.org *Cc: *Don Jackson djack...@nanigans.com, Dave Carroll dcarr...@nanigans.com *Subject: *Recommended OS Hi all, I was wondering what operating systems and versions people are running with success in production environments? We are using C* 1.2.13 and have had issues using CentOS 6.4/6.5. Are others using that OS? What would people recommend? What about Java 6 vs 7 (specific versions?!)? Thanks!!! -- PRIVILEGE AND CONFIDENTIALITY NOTICE The information in this electronic mail (and any associated attachments) is intended for the named recipient(s) only and may contain privileged and confidential information. If you have received this message in error, you are hereby notified that any use, disclosure, copying or alteration of this message is strictly prohibited. If you are not the intended recipient(s), please contact the sender by reply email and destroy all copies of the original message. Thank you. -- -- It's always darkest just before you are eaten by a grue.
Re: Recommended OS
On Wed, Feb 12, 2014 at 8:55 AM, Jeffrey Kesselman jef...@gmail.com wrote: I haven't run Cassandra in production myself, but for other high load Java based servers I've had really good scaling success with OpenSolaris. In particular I've used Joyent's SmartOS which has the additional advantage of bursting to cover brief periods of exceptional load. There are a significant number of Linux only optimizations in Cassandra. Very few people operate production clusters on anything but Linux. The most obvious optimization that comes to mind is the use of direct i/o to avoid blowing out the page cache under various circumstances. My approach towards running Cassandra on anything but Linux would be to try to directly compare performance to the same hardware running Linux. =Rob
Re: Recommended OS
Its quite possible its well tricked out for Linux. My major issue with Linux has been that its TCP/IP stack is nowhere near as scalable as Solaris' for massive numbers of simultaneous connections. But thats probably less of an issue with a Cassandra node then it has been with the game servers I've built. On Wed, Feb 12, 2014 at 1:52 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Feb 12, 2014 at 8:55 AM, Jeffrey Kesselman jef...@gmail.comwrote: I haven't run Cassandra in production myself, but for other high load Java based servers I've had really good scaling success with OpenSolaris. In particular I've used Joyent's SmartOS which has the additional advantage of bursting to cover brief periods of exceptional load. There are a significant number of Linux only optimizations in Cassandra. Very few people operate production clusters on anything but Linux. The most obvious optimization that comes to mind is the use of direct i/o to avoid blowing out the page cache under various circumstances. My approach towards running Cassandra on anything but Linux would be to try to directly compare performance to the same hardware running Linux. =Rob -- It's always darkest just before you are eaten by a grue.
Re: Recommended OS
We are currently trialling SmartOS with Cassandra and have seen some pretty good results (and the mmap stuff appears to work). As Rob said, if this is production cluster, run with linux… there will be far less pain. If you are super keen on running on something different from linux in production (after all the warnings), run most of your cluster on linux, then run a single node or a separate DC with SmartOS, Solaris, BeOS, OS/2, Minix, Windows 3.1 or whatever it is that you choose and let us know how it all goes! Cheers Ben Bromhead Instaclustr | www.instaclustr.com | @instaclustr | +61 415 936 359 On 13/02/2014, at 6:32 AM, Jeffrey Kesselman jef...@gmail.com wrote: Its quite possible its well tricked out for Linux. My major issue with Linux has been that its TCP/IP stack is nowhere near as scalable as Solaris' for massive numbers of simultaneous connections. But thats probably less of an issue with a Cassandra node then it has been with the game servers I've built. On Wed, Feb 12, 2014 at 1:52 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Feb 12, 2014 at 8:55 AM, Jeffrey Kesselman jef...@gmail.com wrote: I haven't run Cassandra in production myself, but for other high load Java based servers I've had really good scaling success with OpenSolaris. In particular I've used Joyent's SmartOS which has the additional advantage of bursting to cover brief periods of exceptional load. There are a significant number of Linux only optimizations in Cassandra. Very few people operate production clusters on anything but Linux. The most obvious optimization that comes to mind is the use of direct i/o to avoid blowing out the page cache under various circumstances. My approach towards running Cassandra on anything but Linux would be to try to directly compare performance to the same hardware running Linux. =Rob -- It's always darkest just before you are eaten by a grue.
Re: Recommended OS
On Wed, Feb 12, 2014 at 1:25 PM, Ben Bromhead b...@instaclustr.com wrote: If you are super keen on running on something different from linux in production (after all the warnings), run most of your cluster on linux, then run a single node or a separate DC with SmartOS, Solaris, BeOS, OS/2, Minix, Windows 3.1 or whatever it is that you choose and let us know how it all goes! My understanding is that running a mixed OS cluster is not officially supported. I could be wrong, but don't think I am. :) =Rob
Re: Recommended OS
I just would advise against it because it's going to be difficult to narrow down what's causing problems. For instance, if you have Node A which is performing GC, it will affect query times on Node B which is trying to satisfy a quorum read. Node B might actually have very low load, and it will be difficult to understand why it's queries are responding slowly. Meanwhile, Node A, during the GC pause, will have no disk activity, and most of the CPUs will not be fully utilized. I'm not saying it's impossible to do this, but I will say you better have a really great understanding of every single OS in your cluster. It's generally hard to find people who are experts in Linux, Windows, and BeOS. Of course, if you want to ride that train, you'd probably have a great blog post. My guess is it'll end in our recommendation is 'don't do this' On Wed, Feb 12, 2014 at 2:36 PM, Robert Coli rc...@eventbrite.com wrote: On Wed, Feb 12, 2014 at 1:25 PM, Ben Bromhead b...@instaclustr.com wrote: If you are super keen on running on something different from linux in production (after all the warnings), run most of your cluster on linux, then run a single node or a separate DC with SmartOS, Solaris, BeOS, OS/2, Minix, Windows 3.1 or whatever it is that you choose and let us know how it all goes! My understanding is that running a mixed OS cluster is not officially supported. I could be wrong, but don't think I am. :) =Rob -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
RE: Recommended OS
This /is/ our first cluster. We've upgraded one-over-one since 2.0.2 (our initially deployed version), doing rolling updates across the rings. No especial resource tuning save turning off SELinux and the usual (disabling swap, separate disk for commit logs, data and the OS). From: Keith Wright [mailto:kwri...@nanigans.com] Sent: Monday, February 10, 2014 4:35 PM To: user@cassandra.apache.org Cc: Don Jackson; Dave Carroll Subject: RE: Recommended OS Is this your first cluster? Have you run older versions of Cassandra? Any specific resource tuning? Thanks all. We are unable to bootstrap nodes and are considering creating a fresh cluster in hopes this is some how data related. On Feb 10, 2014 5:33 PM, Brust, Corwin [Hollander] corwin.br...@hollanderparts.commailto:corwin.br...@hollanderparts.com wrote: We're running C* 2.0.5 under CentOS 6.5 and have not noticed anything like you describe. We have just a couple of pre-production rings (Dev and Test) meaning nothing we have has received particularly intense utilization. Corwin From: Keith Wright [mailto:kwri...@nanigans.com] Sent: Monday, February 10, 2014 2:09 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Cc: Don Jackson; Dave Carroll Subject: Re: Recommended OS We are running on CentOS 6.4 but an upgrade to 6.5 caused packets to backup on the net queue causing HUGE load spikes and cluster meltdown. Ultimately we reverted. Have others seen this? Are others running CentOS 6.4/6.5? Thanks From: Sholes, Joshua joshua_sho...@cable.comcast.commailto:joshua_sho...@cable.comcast.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, February 10, 2014 at 1:56 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Cc: Don Jackson djack...@nanigans.commailto:djack...@nanigans.com, Dave Carroll dcarr...@nanigans.commailto:dcarr...@nanigans.com Subject: Re: Recommended OS What issues are you running into with CentOS 6.4/5? I'm running 1.2.8 on CentOS 6.3 and Java 1.7.0-25, and about to test with 1.7.latest. -- Josh Sholes From: Keith Wright kwri...@nanigans.commailto:kwri...@nanigans.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, February 10, 2014 at 1:50 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Cc: Don Jackson djack...@nanigans.commailto:djack...@nanigans.com, Dave Carroll dcarr...@nanigans.commailto:dcarr...@nanigans.com Subject: Recommended OS Hi all, I was wondering what operating systems and versions people are running with success in production environments? We are using C* 1.2.13 and have had issues using CentOS 6.4/6.5. Are others using that OS? What would people recommend? What about Java 6 vs 7 (specific versions?!)? Thanks!!! PRIVILEGE AND CONFIDENTIALITY NOTICE The information in this electronic mail (and any associated attachments) is intended for the named recipient(s) only and may contain privileged and confidential information. If you have received this message in error, you are hereby notified that any use, disclosure, copying or alteration of this message is strictly prohibited. If you are not the intended recipient(s), please contact the sender by reply email and destroy all copies of the original message. Thank you.
RE: Recommended OS
Is this your first cluster? Have you run older versions of Cassandra? Any specific resource tuning? Thanks all. We are unable to bootstrap nodes and are considering creating a fresh cluster in hopes this is some how data related. On Feb 10, 2014 5:33 PM, Brust, Corwin [Hollander] corwin.br...@hollanderparts.com wrote: We’re running C* 2.0.5 under CentOS 6.5 and have not noticed anything like you describe. We have just a couple of pre-production rings (Dev and Test) meaning nothing we have has received particularly intense utilization. Corwin From: Keith Wright [mailto:kwri...@nanigans.com] Sent: Monday, February 10, 2014 2:09 PM To: user@cassandra.apache.org Cc: Don Jackson; Dave Carroll Subject: Re: Recommended OS We are running on CentOS 6.4 but an upgrade to 6.5 caused packets to backup on the net queue causing HUGE load spikes and cluster meltdown. Ultimately we reverted. Have others seen this? Are others running CentOS 6.4/6.5? Thanks From: Sholes, Joshua joshua_sho...@cable.comcast.commailto:joshua_sho...@cable.comcast.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, February 10, 2014 at 1:56 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Cc: Don Jackson djack...@nanigans.commailto:djack...@nanigans.com, Dave Carroll dcarr...@nanigans.commailto:dcarr...@nanigans.com Subject: Re: Recommended OS What issues are you running into with CentOS 6.4/5? I’m running 1.2.8 on CentOS 6.3 and Java 1.7.0-25, and about to test with 1.7.latest. -- Josh Sholes From: Keith Wright kwri...@nanigans.commailto:kwri...@nanigans.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Monday, February 10, 2014 at 1:50 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Cc: Don Jackson djack...@nanigans.commailto:djack...@nanigans.com, Dave Carroll dcarr...@nanigans.commailto:dcarr...@nanigans.com Subject: Recommended OS Hi all, I was wondering what operating systems and versions people are running with success in production environments? We are using C* 1.2.13 and have had issues using CentOS 6.4/6.5. Are others using that OS? What would people recommend? What about Java 6 vs 7 (specific versions?!)? Thanks!!! PRIVILEGE AND CONFIDENTIALITY NOTICE The information in this electronic mail (and any associated attachments) is intended for the named recipient(s) only and may contain privileged and confidential information. If you have received this message in error, you are hereby notified that any use, disclosure, copying or alteration of this message is strictly prohibited. If you are not the intended recipient(s), please contact the sender by reply email and destroy all copies of the original message. Thank you.
