Output from FailureDetector MBean changed in 2.1.10

[Jared Biel]

Hello,

We use the output from the AllEndpointStates attribute of the
org.apache.cassandra.net:type=FailureDetector MBean via MX4J for basic
monitoring/sanity checking. I've found that the output changed between
2.1.9 and 2.1.10; there is now additional info for some of the keys (DC,
RELEASE_VERSION, STATUS, etc.)

Is this maybe due to CASSANDRA-10330
? I perused the
changelog, JIRA, and the git repo and didn't see anything else that really
stuck out at me. Is the new output documented somewhere?

*Example 2.1.9 output:*
["DC:us-west-2",
 "HOST_ID:ed17c00e-2015-4b0f-9ee1-fa96e6c2ff8c",
 "LOAD:3525337.0",
 "NET_VERSION:8",
 "RACK:2a",
 "RELEASE_VERSION:2.1.9",
 "RPC_ADDRESS:1.2.3.4",
 "SCHEMA:b3a65c2a-8955-30f3-b403-54db3260f777",
 "SEVERITY:0.0",
 "STATUS:NORMAL,-1012622047045121026",
 "generation:1441736600",
 "heartbeat:7847815"]

*Example 2.1.10 output:*
["DC:6:us-east",
 "HOST_ID:2:8c8ed6f6-84d9-494d-a6c3-a9701cefe347",
 "LOAD:537977:1.45221778E8",
 "NET_VERSION:1:8",
 "RACK:8:1b",
 "RELEASE_VERSION:4:2.1.10",
 "RPC_ADDRESS:3:1.2.3.4",
 "SCHEMA:10:d58b9de5-9e25-3128-b07e-dfa0b2568ae8",
 "SEVERITY:538103:0.0",
 "STATUS:14:NORMAL,-1022178689192707266",
 "TOKENS:13:",
 "generation:1444146690",
 "heartbeat:538104"]

Thanks!

Re: Not able to cqlsh on 2.1.9 on Ubuntu 14.04

[Jared Biel]

I assume "@ Of node" is ethX's IP address? Has cassandra been restarted
since changes were made to cassandra.yaml? The netstat output that you
posted doesn't look right; we use settings similar to what you've posted.
Here's what it looks like on one of our nodes.


-Jared

On 14 September 2015 at 10:34, Ahmed Eljami  wrote:

> In cassanrda.yaml:
> listen_address:@ Of node
> rpc_address:0.0.0.0
>
> brodcast_rpc_address:@ Of node
>
> 2015-09-14 11:31 GMT+01:00 Neha Dave :
>
>> Try
>> >cqlsh 
>>
>> regards
>> Neha
>>
>> On Mon, Sep 14, 2015 at 3:53 PM, Ajay Garg 
>> wrote:
>>
>>> Hi All.
>>>
>>> We have setup a Ubuntu-14.04 server, and followed the steps exactly as
>>> per http://wiki.apache.org/cassandra/DebianPackaging
>>>
>>> Installation completes fine, Cassandra starts fine, however cqlsh does
>>> not work.
>>> We get the error ::
>>>
>>>
>>> ###
>>> ajay@comp:~$ cqlsh
>>> Connection error: ('Unable to connect to any servers', {'127.0.0.1':
>>> error(None, "Tried connecting to [('127.0.0.1', 9042)]. Last error:
>>> None")})
>>>
>>> ###
>>>
>>>
>>>
>>> Version-Info ::
>>>
>>>
>>> ###
>>> ajay@comp:~$ dpkg -l | grep cassandra
>>> ii  cassandra   2.1.9
>>>  all  distributed storage system for structured data
>>>
>>> ###
>>>
>>>
>>>
>>> The port "seems" to be opened fine.
>>>
>>>
>>> ###
>>> ajay@comp:~$ netstat -an | grep 9042
>>> tcp6   0  0 127.0.0.1:9042  :::*
>>> LISTEN
>>>
>>> ###
>>>
>>>
>>>
>>> Firewall-filters ::
>>>
>>>
>>> ###
>>> ajay@comp:~$ sudo iptables -L
>>> [sudo] password for ajay:
>>> Chain INPUT (policy ACCEPT)
>>> target prot opt source   destination
>>> ACCEPT all  --  anywhere anywhere state
>>> RELATED,ESTABLISHED
>>> ACCEPT tcp  --  anywhere anywhere tcp dpt:ssh
>>> DROP   all  --  anywhere anywhere
>>>
>>> Chain FORWARD (policy ACCEPT)
>>> target prot opt source   destination
>>>
>>> Chain OUTPUT (policy ACCEPT)
>>> target prot opt source   destination
>>>
>>> ###
>>>
>>>
>>>
>>> Even telnet fails :(
>>>
>>>
>>> ###
>>> ajay@comp:~$ telnet localhost 9042
>>> Trying 127.0.0.1...
>>>
>>> ###
>>>
>>>
>>>
>>> Any ideas please?? We have been stuck on this for a good 3 hours now :(
>>>
>>>
>>>
>>> Thanks and Regards,
>>> Ajay
>>>
>>
>>
>
>
> --
> Cordialement;
>
> Ahmed ELJAMI
>

Re: Not able to cqlsh on 2.1.9 on Ubuntu 14.04

[Jared Biel]

Is there a reason that you're setting listen_address and rpc_address to
localhost?

listen_address doc: "the Right Thing is to use the address associated with
the hostname". So, set the IP address of this to eth0 for example. I
believe if it is set to localhost then you won't be able to form a cluster
with other nodes.

rpc_address: this is the address to which clients will connect. I recommend
0.0.0.0 here so clients can connect to IP address of the server as well as
localhost if they happen to reside on the same instance.


Here are all of the address settings from our config file. 192.168.1.10 is
the IP address of eth0 and broadcast_address is commented out.

listen_address: 192.168.1.10
# broadcast_address: 1.2.3.4
rpc_address: 0.0.0.0
broadcast_rpc_address: 192.168.1.10

Follow these directions to get up and running with the first node
(destructive process):

1. Stop cassandra
2. Remove data from cassandra var directory (rm -rf /var/lib/cassandra/*)
3. Make above changes to config file. Also set seeds to the eth0 IP address
4. Start cassandra
5. Set seeds in config file back to "" after cassandra is up and running.

After following that process, you'll be able to connect to the node from
any host that can reach Cassandra's ports on that node ("cqlsh" command
will work.) To join more nodes to the cluster, follow the steps same steps
as above, except the seeds value to the IP address of an already running
node.

Regarding the empty "seeds" config entry: our configs are automated with
configuration management. During the node bootstrap process a script
performs the above. The reason that we set seeds back to empty is that we
don't want nodes coming up/down to cause the config file to change and thus
cassandra to restart needlessly. So far we haven't had any issues with
seeds being set to empty after a node has joined the cluster, but this may
not be the recommended way of doing things.

-Jared

On 14 September 2015 at 16:46, Ajay Garg <ajaygargn...@gmail.com> wrote:

> Hi All.
>
> Thanks for your replies.
>
> a)
> cqlsh  does not work either :(
>
>
> b)
> Following are the parameters as asked ::
>
> listen_address: localhost
> rpc_address: localhost
>
> broadcast_rpc_address is not set.
> According to the yaml file ::
>
> # RPC address to broadcast to drivers and other Cassandra nodes. This
> cannot
> # be set to 0.0.0.0. If left blank, this will be set to the value of
> # rpc_address. If rpc_address is set to 0.0.0.0, broadcast_rpc_address must
> # be set.
> # broadcast_rpc_address: 1.2.3.4
>
>
> c)
> Following is the netstat-output, with process information ::
>
>
> ###
> ajay@comp:~$ sudo netstat -apn | grep 9042
> [sudo] password for admin:
> tcp6   0  0 127.0.0.1:9042  :::*
> LISTEN  10169/java
>
> ###
>
>
> Kindly let me know what else we can try .. it is really driving us nuttsss
> :(
>
> On Mon, Sep 14, 2015 at 9:40 PM, Jared Biel
> <jared.b...@bolderthinking.com> wrote:
> > Whoops, I accidentally pressed a hotkey and sent my message prematurely.
> > Here's what netstat should look like with those settings:
> >
> > sudo netstat -apn | grep 9042
> > tcp6   0  0 0.0.0.0:9042:::*
> LISTEN
> > 21248/java
> >
> > -Jared
> >
> > On 14 September 2015 at 16:09, Jared Biel <jared.b...@bolderthinking.com
> >
> > wrote:
> >>
> >> I assume "@ Of node" is ethX's IP address? Has cassandra been restarted
> >> since changes were made to cassandra.yaml? The netstat output that you
> >> posted doesn't look right; we use settings similar to what you've
> posted.
> >> Here's what it looks like on one of our nodes.
> >>
> >>
> >> -Jared
> >>
> >> On 14 September 2015 at 10:34, Ahmed Eljami <ahmed.elj...@gmail.com>
> >> wrote:
> >>>
> >>> In cassanrda.yaml:
> >>> listen_address:@ Of node
> >>> rpc_address:0.0.0.0
> >>>
> >>> brodcast_rpc_address:@ Of node
> >>>
> >>> 2015-09-14 11:31 GMT+01:00 Neha Dave <nehajtriv...@gmail.com>:
> >>>>
> >>>> Try
> >>>> >cqlsh 
> >>>>
> >>>> regards
> >>>> Neha
> >>>>
> >>>> On Mon, Sep 14, 2015 at 3:53 PM, Ajay Garg <ajaygargn...@gmail.com>
> >>>> wrote:
> >

Re: Cassandra leap second

[Jared Biel]

Here is the full Datastax article:
http://www.datastax.com/dev/blog/preparing-for-the-leap-second

We had issues with C* and MySQL last time around (2012) but no issues this
time thanks to upgraded systems.

On 1 July 2015 at 14:39, Alain RODRIGUEZ arodr...@gmail.com wrote:

 I think it is more a kernel / java version issue than a Cassandra version
 related issue.

 From Datastax:

 Leap Second on June 30, 2015

 Required Actions:

 Ensure that you are running on kernel version 3.4 or higher and using JDK
 version 7u60 or higher. This should protect you from the livelock problems
 users experienced in 2012.

 We had recent versions on both things and ran smoothly this night.

 C*heers



 2015-07-01 16:29 GMT+02:00 Narendra Sharma narendra.sha...@gmail.com:

 We also experienced same, i.e. high cpu on Cassandra 1.1.4 node running
 in AWS. Restarting the vm worked.

 On Wed, Jul 1, 2015 at 4:58 AM, Jason Wee peich...@gmail.com wrote:

 same here too, on branch 1.1 and have not seen any high cpu usage.

 On Wed, Jul 1, 2015 at 2:52 PM, John Wong gokoproj...@gmail.com wrote:

 Which version are you running and what's your kernel version? We are
 still running on 1.2 branch but we have not seen any high cpu usage yet...

 On Tue, Jun 30, 2015 at 11:10 PM, snair123 . nair...@outlook.com
 wrote:

 reboot of the machine worked

 --
 From: nair...@outlook.com
 To: user@cassandra.apache.org
 Subject: Cassandra leap second
 Date: Wed, 1 Jul 2015 02:54:53 +

 Is it ok to run this


 https://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/

 Seeing high cpu consumption for cassandra process





 --
 Sent from Jeff Dean's printf() mobile console





 --
 Narendra Sharma
 Software Engineer
 *http://www.aeris.com http://www.aeris.com*
 *http://narendrasharma.blogspot.com/
 http://narendrasharma.blogspot.com/*

Re: EC2snitch in AWS

[Jared Biel]

It's pretty straightforward - there is documentation here
http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html.
Simply set endpoint_snitch: Ec2Snitch in cassandra.yaml when creating the
cluster and create keyspaces with the necessary NetworkTopologyStrategy.
The only real gotcha that I've run into is that the datacenter
(cassandra) and regions (AWS) match up 1:1 except for us-east-1 (it's
just us-east  in the Ec2Snitch.)

Below are some examples that specify that data for that keyspace will be
present on 3 nodes. See consistency documentation
http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html
for
how to tune your clients to meet certain consistency guarantees.

CREATE KEYSPACE ExampleKeyspace WITH REPLICATION = { 'class' :
'NetworkTopologyStrategy', 'us-east' : 3 };

CREATE KEYSPACE ExampleKeyspace WITH REPLICATION = { 'class' :
'NetworkTopologyStrategy', 'us-west-2' : 3 };


Jared Biel | Bolder Thinking | 701-205-3153 | jared.b...@bolderthinking.com

On 27 May 2015 at 16:31, Kaushal Shriyan kaushalshri...@gmail.com wrote:

 Hi,

 Can somebody please share me details about setting up of EC2snitch in AWS
 single region which has availability zone 1a and 1b? I am using Cassandra
 version 1.2.19 in the setup.

 I would appreciate your help.

 Regards,

 Kaushal

Re: Multi Datacenter / MultiRegion on AWS Best practice ?

[Jared Biel]

If you decide to go the iptables route, you could try neti
https://github.com/Instagram/neti (blog post here
http://instagram-engineering.tumblr.com/post/100758229719/migrating-from-aws-to-aws
.)


On 27 October 2014 16:44, Juho Mäkinen juho.maki...@gmail.com wrote:

 Hi!


 2014-10-23 11:16 GMT+02:00 Alain RODRIGUEZ arodr...@gmail.com:

 We are currently wondering about the best way to configure network
 architecture to have a Cassandra cluster multi DC.

 On solution 2, we would need to open IPs one by one on 3 ports (7000,
 9042, 9160) at least. 100 entries in a security group would allow us to
 have a maximum of ~30 nodes


 You can also allow those ports from everywhere and then use local iptables
 to limit the access to only those IPs which you are actually using. You'll
 most certainly need some kind of configuration management system for this
 (Chef, puppet, salt-stack etc).

Re: Looking for a cassandra web interface

[Jared Biel]

Very nice, I had forgotten about Cyclop. We used to use Cassandra Cluster
Admin, but it doesn't support CQL so it's not very useful anymore. If
you're looking for a developer UI, I recommend trying DataStax DevCenter (
http://www.datastax.com/what-we-offer/products-services/devcenter). It's
completely free.


On 20 October 2014 18:05, DuyHai Doan doanduy...@gmail.com wrote:

 I remember someone mentioning Cyclop, a web interface based on Apache
 Wicket for Cassandra: https://github.com/maciejmiklas/cyclop

 If you want a module which exposes Cassandra operations as REST services,
 look at Virgil https://github.com/hmsonline/virgil

 On Mon, Oct 20, 2014 at 7:40 PM, Vishanth Balasubramaniam 
 vishan...@wso2.com wrote:

 Hi,

 I am very new to cassandra. I have started cassandra inside an instance
 in my VM and I want to expose a cassandra web interface. What is the most
 stable web interface for Cassandra with a proper guide to set up?

 Thanks and Regards,
 Vishanth

Re: FileNotFoundException during reboot (Cassandra-2.1.1)

[Jared Biel]

Maybe you're running out of file descriptors?

On 29 September 2014 01:31, Jay Patel pateljay3...@gmail.com wrote:

 Hey there,

 We've trying out Cassandra-2.1.1 and facing the below exception
 frequently, mainly during the node reboot. File is there with proper
 permissions (as below), not sure why C* is not able to find it.

 

 ERROR [OptionalTasks:1] 2014-09-27 16:00:20,888 CassandraDaemon.java:167 -
 Exception in thread Thread[OptionalTasks:1,5,main]
 java.lang.RuntimeException: java.io.FileNotFoundException:
 /var/lib/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-12756-Data.db
 at
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1881)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.io.sstable.SSTableScanner.init(SSTableScanner.java:67)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1646)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:67)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1920)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:2035)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1969)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:513)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:93)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:174)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:75)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 [na:1.7.0_65]
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
 [na:1.7.0_65]
 at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 [na:1.7.0_65]
 at
 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
 [na:1.7.0_65]
 at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 [na:1.7.0_65]
 at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
 [na:1.7.0_65]
 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
 Caused by: java.io.FileNotFoundException:
 /var/lib/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-12756-Data.db
 at
 org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:66)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 at
 org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)
 ~[apache-cassandra-2.1.1-SNAPSHOT.jar:2.1.1-SNAPSHOT]
 ... 18 common frames omitted
 ^C



 $ ll
 /var/lib/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-12756-Data.db
 -rw-r--r-- 1 cassandra root 5305 Sep 27 00:57
 /var/lib/cassandra/data/system/hints-2666e20573ef38b390fefecf96e8f0c7/la-12756-Data.db

Re: Experience with multihoming cassandra?

[Jared Biel]

Doing this seems counter-productive to Cassandra's design/use-cases. It's
best at home running on a large number of smaller servers rather than a
small number of large servers. Also, as you said, you won't get any of the
high availability benefits that it offers if you run multiple copies of
Cassandra on the same box.


On 25 September 2014 16:58, Donald Smith donald.sm...@audiencescience.com
wrote:

  We have large boxes with 256G of RAM and SSDs.  From iostat, top,
 and sar we think the system has excess capacity.  Anyone have
 recommendations about multihoming
 http://en.wikipedia.org/wiki/Multihoming cassandra on such a node
 (connecting it to multiple IPs and running multiple cassandras
 simultaneously)?  I’m skeptical, since Cassandra already has built-in
 multi-threading and since if the node went down multiple nodes would
 disappear.  We’re using C* version 2.0.9.



 A google/bing search for  multihoming cassandra doesn’t turn much up.



 *Donald A. Smith* | Senior Software Engineer
 P: 425.201.3900 x 3866
 C: (206) 819-5965
 F: (646) 443-2333
 dona...@audiencescience.com


 [image: AudienceScience]

Re: Question about EC2 and SSDs

[Jared Biel]

EBS performance was a major concern a few years ago, but it's gotten better
in the last few years with things like Provisioned IOPS and SSDs. However,
it's still not recommended (reasons outlined here
http://www.datastax.com/documentation/cassandra/2.0/cassandra/architecture/architecturePlanningEC2_c.html
.)

I personally don't care for EBS because of the reliability factor. Do a web
search for EBS outage and you'll see that EBS has caused much pain for lots
of organizations in the past. Ephemeral disks (which exist on the host of
which your instance is running) will always be more reliable and generally
are more performant. The trend seems to be that they're pushing people
towards EBS (t2 instances are EBS only, other current gen instances are SSD
only) and it's a shame. It's good to have choices.

-Jared

On 5 September 2014 13:51, William Oberman ober...@civicscience.com wrote:

 Theory aside, I switched from RAID of ephemerals for data, and root
 volume for write log to single EBS-based SSD without any noticeable impact
 on performance.

 will

 On Thu, Sep 4, 2014 at 9:35 PM, Steve Robenalt sroben...@highwire.org
 wrote:

 Yes, I am aware there are no heads on an SSD. I also have seen plenty of
 examples where compatibility issues force awkward engineering tradeoffs,
 even as technology advances so I am jaded enough to be wary of making
 assumptions, which is why I asked the question.

 Steve
 On Sep 4, 2014 5:50 PM, Robert Coli rc...@eventbrite.com wrote:

 On Thu, Sep 4, 2014 at 5:44 PM, Steve Robenalt sroben...@highwire.org
 wrote:

 Thanks Robert! I am assuming that you meant that it's possible with a
 single SSD, right?


 Yes, no matter how many SSDs you have you are unlikely to be able to
 convince one of them to physically seek a drive head across its plater,
 because they don't have heads or platters.

 =Rob

Re: abusing cassandra's multi DC abilities

[Jared Biel]

Have you heard of this https://github.com/Comcast/cmb? Maybe it's along the
path of what you're looking for.


On 22 February 2014 22:33, Jonathan Haddad j...@jonhaddad.com wrote:

 Upfront TLDR: We want to do stuff (reindex documents, bust cache) when
 changed data from DC1 shows up in DC2.

 Full Story:
 We're planning on adding data centers throughout the US.  Our platform is
 used for business communications.  Each DC currently utilizes elastic
 search and redis.  A message can be sent from one user to another, and the
 intent is that it would be seen in near-real-time.  This means that 2
 people may be using different data centers, and the messages need to
 propagate from one to the other.

 On the plus side, we know we get this with Cassandra (fist pump) but the
 other pieces, not so much.  Even if they did work, there's all sorts of
 race conditions that could pop up from having different pieces of our
 architecture communicating over different channels.  From this, we've
 arrived at the idea that since Cassandra is the authoritative data source,
 we might be able to trigger events in DC2 based on activity coming through
 either the commit log or some other means.  One idea was to use a CF with a
 low gc time as a means of transporting messages between DCs, and watching
 the commit logs for deletes to that CF in order to know when we need to do
 things like reindex a document (or a new document), bust cache, etc.
  Facebook did something similar with their modifications to MySQL to
 include cache keys in the replication log.

 Assuming this is sane, I'd want to avoid having the same event register on
 3 servers, thus registering 3 items in the queue when only one should be
 there.  So, for any piece of data replicated from the other DC, I'd need a
 way to determine if it was supposed to actually trigger the event or not.
  (Maybe it looks at the token and determines if the current server falls in
 the token range?)  Or is there a better way?

 So, my questions to all ye Cassandra users:

 1. Is this is even sane?
 2. Is anyone doing it?

 --
 Jon Haddad
 http://www.rustyrazorblade.com
 skype: rustyrazorblade

Re: MUTATION messages dropped

[Jared Biel]

I can't comment on your specific issue, but I don't know if running 2.0.0
in production is a good idea. At the very least I'd try upgrading to the
latest 2.0.X (currently 2.0.3.)

 https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/

On 20 December 2013 06:08, Alexander Shutyaev shuty...@gmail.com wrote:

 Thanks for you answers.

 *srmore*,

 We are using v2.0.0. As for GC I guess it does not correlate in our case,
 because we had cassandra running 9 days under production load and no
 dropped messages and I guess that during this time there were a lot of GCs.

 *Ken*,

 I've checked the values you indicated. Here they are:

 node1 6498
 node2 6476
 node3 6642

 I guess this is not good :) What can we do to fix this problem?


 2013/12/19 Ken Hancock ken.hanc...@schange.com

 We had issues where the number of CF families that were being flushed
 would align and then block writes for a very brief period. If that happened
 when a bunch of writes came in, we'd see a spike in Mutation drops.

 Check nodetool tpstats for FlushWriter all time blocked.


 On Thu, Dec 19, 2013 at 7:12 AM, Alexander Shutyaev 
 shuty...@gmail.comwrote:

 Hi all!

 We've had a problem with cassandra recently. We had 2 one-minute periods
 when we got a lot of timeouts on the client side (the only timeouts during
 9 days we are using cassandra in production). In the logs we've found
 corresponding messages saying something about MUTATION messages dropped.

 Now, the official faq [1] says that this is an indicator that the load
 is too high. We've checked our monitoring and found out that 1-minute
 average cpu load had a local peak at the time of the problem, but it was
 like 0.8 against 0.2 usual which I guess is nothing for a 2 core virtual
 machine. We've also checked java threads - there was no peak there and
 their count was reasonable ~240-250.

 Can anyone give us a hint - what should we monitor to see this high
 load and what should we tune to make it acceptable?

 Thanks in advance,
 Alexander

 [1] http://wiki.apache.org/cassandra/FAQ#dropped_messages




 --
  *Ken Hancock *| System Architect, Advanced Advertising
 SeaChange International
 50 Nagog Park
 Acton, Massachusetts 01720
 ken.hanc...@schange.com | www.schange.com | 
 NASDAQ:SEAChttp://www.schange.com/en-US/Company/InvestorRelations.aspx

 Office: +1 (978) 889-3329 | [image: Google Talk:] ken.hanc...@schange.com
  | [image: Skype:]hancockks | [image: Yahoo IM:]hancockks [image:
 LinkedIn] http://www.linkedin.com/in/kenhancock

 [image: SeaChange International]
  http://www.schange.com/This e-mail and any attachments may contain
 information which is SeaChange International confidential. The information
 enclosed is intended only for the addressees herein and may not be copied
 or forwarded without permission from SeaChange International.

Re: mmaped files and swap

[Jared Biel]

See http://en.wikipedia.org/wiki/Swappiness


On 13 March 2013 19:56, Fredrik Stigbäck
fredrik.l.stigb...@sitevision.sewrote:

 Well, we've seen a Cassandra process swap out 500 MB on a Linux OS
 with plenty of RAM, so I was just curious as why the OS thinks it
 should use the swap at all.

 2013/3/13 karim duran karim.du...@gmail.com:
  I agree with Edward Capriolo,
  Even when swap is enabled on your system, swaping rarely occurs on OS
  today...(except for very loaded systems).
 
  But, take care that some 32 bits system kernels allows only 2^32 bits
 memory
  mapped file length ( ~ 2 Go ).
  It could be a limitation for NoSQL databases. It's the case for MongoDB
 on
  32 bits OS.
 
  I don't know how to avoid swaping if Cassandra exceeds these limitation
 when
  this case occurs.
 
 
  2013/3/13 Edward Capriolo edlinuxg...@gmail.com
 
  You really can not control what the OS-swaps out. java has other memory
  usage outside the heap, and native memory. best to turn swap off. Swap
 is
  kinda old school anyway at this point. It made sense when machines had
 32MB
  RAM.
 
  Keeping your read 95th percentile low is mostly about removing
 deviations
  that cause requests to slow down, swap is one of the things that cause
  fluctuation becuase it is not predictable.
 
  On Wed, Mar 13, 2013 at 10:39 AM, Fredrik
  fredrik.l.stigb...@sitevision.se wrote:
 
  I've got a question regarding understanding the recomendation to
 disable
  swap.
  Since Cassandra uses mlockall to lock the heap in RAM what is the
 reason
  for disabling swap?
  My guess is that is has to do with memory mapped files but as of my
  understanding, accessing pages of
  memory mapped files, those pages are never put in swap since they're
  backed by files on disk and the OS
  writes those pages to the memory mapped file instead of swap.
  We've seen on Cassandra installations on Linux with swap enabled that
  parts of the java process is swaped out and increasing.
  So what's swaped out?
 
  Regards
  /Fredrik
 
 
 
 
 
 



 --
 Fredrik Larsson Stigbäck
 SiteVision AB Vasagatan 10, 107 10 Örebro
 019-17 30 30

Re: Cassandra with SAN

[Jared Biel]

 As a counter argument though, anyone running a C* cluster on the Amazon
cloud is going to be using SAN storage (or some kind of proprietary storage
array) at the lowest  layers...Amazon isn't going to have a bunch of JBOD
running their cloud infrastructure.  However, they've invested in the
infrastructure to do it right.

This is certainly true when using EBS, however it's generally not
recommended to use EBS when running Cassandra. EBS has proven to be
unreliable in the past and it's a bit of a SPOF. Instead, it's recommended
to use the instance store disks that come with most instances (handy
chart here: http://www.ec2instances.info/). These are the rough equivalent
of local disks (probably host level RAID 10 storage if I'd have to guess.)

-Jared

On 22 February 2013 00:40, Michael Morris michael.m.mor...@gmail.comwrote:

 I'm running a 27 node cassandra cluster on SAN without issue.  I will be
 perfectly clear though, the hosts are multi-homed to different
 switches/fabrics in the SAN, we have an _expensive_ EMC array, and other
 than a datacenter-wide power outage, there's no SPOF for the SAN.  We use
 it because it's there, and it's already a sunk cost.

 I certainly would not go out of my way to purchase SAN infrastructure for
 a C* cluster, it just doesn't make sense (for all the reasons others have
 mentioned).  Any more, you can load up a single 2U server with multi-TB
 worth of disk, so the aggregate storage capacity of your C* cluster could
 potentially be as much as a SAN you would purchase (and a lot less hassle
 too).

 As a counter argument though, anyone running a C* cluster on the Amazon
 cloud is going to be using SAN storage (or some kind of proprietary storage
 array) at the lowest layers...Amazon isn't going to have a bunch of JBOD
 running their cloud infrastructure.  However, they've invested in the
 infrastructure to do it right.

 - Mike


 On Thu, Feb 21, 2013 at 6:08 PM, P. Taylor Goetz ptgo...@gmail.comwrote:

 I shouldn't have used the word spinning... SSDs are a great option as
 well.

 I also agree with all the expensive SPOF points others have made.

 Sent from my iPhone

 On Feb 21, 2013, at 6:56 PM, P. Taylor Goetz ptgo...@gmail.com wrote:

 Cassandra is designed to write and read data in a way that is optimized
 for physical spinning disks.

 Running C* on a SAN introduces a layer of abstraction that, at best
 negates those optimizations, and at worst introduces additional overhead.

 Sent from my iPhone

 On Feb 21, 2013, at 6:42 PM, Kanwar Sangha kan...@mavenir.com wrote:

  Ok. What would be the drawbacks J

 ** **

 *From:* Michael Kjellman 
 [mailto:mkjell...@barracuda.commkjell...@barracuda.com]

 *Sent:* 21 February 2013 17:12
 *To:* user@cassandra.apache.org
 *Subject:* Re: Cassandra with SAN

 ** **

 No, this is a really really bad idea and C* was not designed for this, in
 fact, it was designed so you don't need to have a large expensive SAN.***
 *

 ** **

 Don't be tempted by the shiny expensive SAN. :)

 ** **

 If money is no object instead throw SSD's in your nodes and run 10G
 between racks

 ** **

 *From: *Kanwar Sangha kan...@mavenir.com
 *Reply-To: *user@cassandra.apache.org user@cassandra.apache.org
 *Date: *Thursday, February 21, 2013 2:56 PM
 *To: *user@cassandra.apache.org user@cassandra.apache.org
 *Subject: *Cassandra with SAN

 ** **

 Hi – Is it a good idea to use Cassandra with SAN ?  Say a SAN which
 provides me 8 Petabytes of storage. Would I not be I/O bound irrespective
 of the no of Cassandra machines and scaling by adding 

 machines won’t help ?

  

 Thanks

 Kanwar

 ** **

 --
 Copy, by Barracuda, helps you store, protect, and share all your amazing
 things. Start today: www.copy.com http://www.copy.com?a=em_footer. 

   ­­  

Re: Nodetool doesn't shows two nodes

[Jared Biel]

This is something that I found while using the multi-region snitch -
it uses public IPs for communication. See the original ticket here:
https://issues.apache.org/jira/browse/CASSANDRA-2452. It'd be nice if
it used the private IPs to communicate with nodes that are in the same
region as itself, but I do not believe this is the case. Be aware that
you will be charged for external data transfer even for nodes in the
same region because the traffic will not fall under their free (for
same AZ) or reduced (for intra-AZ) tiers.

If you continue using this snitch in the mean time, it is not
necessary (or recommended) to have those ports open to 0.0.0.0/0.
You'll simply need to add the public IPs of your C* servers to the
correct security group(s) to allow access.

There's something else that's a little strange about the EC2 snitches:
us-east-1 is (incorrectly) represented as the datacenter us-east.
Other regions are recognized and named properly (us-west-2 for
example) This is kind-of covered in the ticket here:
https://issues.apache.org/jira/browse/CASSANDRA-4026 I wish it could
be fixed properly.

Good luck!


On 17 February 2013 16:16, Boris Solovyov boris.solov...@gmail.com wrote:
 OK. I got it. I realized that storage_port wasn't actually open between the
 nodes, because it is using the public IP. (I did find this information in
 the docs, after looking more... it is in section on Types of snitches. It
 explains everything I found by try and error.)

 After opening this port 7000 to all IP addresses, the cluster boots OK and
 the two nodes see each other. Now I have the happy result. But my nodes are
 wide open to the entire internet on port 7000. This is a serious problem.
 This obviously can't be put into production.

 I definitely need cross-continent deployment. Single AZ or single region
 deployment is not going to be enough. How do people solve this in practice?

Re: Cassandra at Amazon AWS

[Jared Biel]

We use a replication factor such that if any one instance dies the
cluster would remain alive. If a node dies, we simply replace it and
move on. As far as disaster recovery, it's easy to store snapshots in
S3, although glacier is looking interesting.
Jared Biel
System Administrator
Bolder Thinking
www.bolderthinking.com
Office: 701.205.3153
jared.b...@bolderthinking.com


On 17 January 2013 13:44, Adam Venturella aventure...@gmail.com wrote:
 Jared, how do you guys handle data backups for your ephemeral based cluster?

 I'm trying to move to ephemeral drives myself, and that was my last sticking
 point; asking how others in the community deal with backup in case the VM
 explodes.



 On Wed, Jan 16, 2013 at 1:21 PM, Jared Biel jared.b...@bolderthinking.com
 wrote:

 We're currently using Cassandra on EC2 at very low scale (a 2 node
 cluster on m1.large instances in two regions.) I don't believe that
 EBS is recommended for performance reasons. Also, it's proven to be
 very unreliable in the past (most of the big/notable AWS outages were
 due to EBS issues.) We've moved 99% of our instances off of EBS.

 As other have said, if you require more space in the future it's easy
 to add more nodes to the cluster. I've found this page
 (http://www.ec2instances.info/) very useful in determining the amount
 of space each instance type has. Note that by default only one
 ephemeral drive is attached and you must specify all ephemeral drives
 that you want to use at launch time. Also, you can create a RAID 0 of
 all local disks to provide maximum speed and space.


 On 16 January 2013 20:42, Marcelo Elias Del Valle mvall...@gmail.com
 wrote:
  Hello,
 
 I am currently using hadoop + cassandra at amazon AWS. Cassandra runs
  on
  EC2 and my hadoop process runs at EMR. For cassandra storage, I am using
  local EC2 EBS disks.
 My system is running fine for my tests, but to me it's not a good
  setup
  for production. I need my system to perform well for specially for
  writes on
  cassandra, but the amount of data could grow really big, taking several
  Tb
  of total storage.
  My first guess was using S3 as a storage and I saw this can be done
  by
  using Cloudian package, but I wouldn't like to become dependent on a
  pre-package solution and I found it's kind of expensive for more than
  100Tb:
  http://www.cloudian.com/pricing.html
  I saw some discussion at internet about using EBS or ephemeral disks
  for
  storage at Amazon too.
 
  My question is: does someone on this list have the same problem as
  me?
  What are you using as solution to Cassandra's storage when running it at
  Amazon AWS?
 
  Any thoughts would be highly appreciatted.
 
  Best regards,
  --
  Marcelo Elias Del Valle
  http://mvalle.com - @mvallebr

Re: Cassandra at Amazon AWS

[Jared Biel]

We're currently using Cassandra on EC2 at very low scale (a 2 node
cluster on m1.large instances in two regions.) I don't believe that
EBS is recommended for performance reasons. Also, it's proven to be
very unreliable in the past (most of the big/notable AWS outages were
due to EBS issues.) We've moved 99% of our instances off of EBS.

As other have said, if you require more space in the future it's easy
to add more nodes to the cluster. I've found this page
(http://www.ec2instances.info/) very useful in determining the amount
of space each instance type has. Note that by default only one
ephemeral drive is attached and you must specify all ephemeral drives
that you want to use at launch time. Also, you can create a RAID 0 of
all local disks to provide maximum speed and space.


On 16 January 2013 20:42, Marcelo Elias Del Valle mvall...@gmail.com wrote:
 Hello,

I am currently using hadoop + cassandra at amazon AWS. Cassandra runs on
 EC2 and my hadoop process runs at EMR. For cassandra storage, I am using
 local EC2 EBS disks.
My system is running fine for my tests, but to me it's not a good setup
 for production. I need my system to perform well for specially for writes on
 cassandra, but the amount of data could grow really big, taking several Tb
 of total storage.
 My first guess was using S3 as a storage and I saw this can be done by
 using Cloudian package, but I wouldn't like to become dependent on a
 pre-package solution and I found it's kind of expensive for more than 100Tb:
 http://www.cloudian.com/pricing.html
 I saw some discussion at internet about using EBS or ephemeral disks for
 storage at Amazon too.

 My question is: does someone on this list have the same problem as me?
 What are you using as solution to Cassandra's storage when running it at
 Amazon AWS?

 Any thoughts would be highly appreciatted.

 Best regards,
 --
 Marcelo Elias Del Valle
 http://mvalle.com - @mvallebr