OpsCenter detecting decommissioning node

[Carlos Alonso]

While decommissioning nodes...Why doesn't OpsCenter show it as an activity
or something in the status of the node?

That would be helpful to monitor the progress, as when upgrading sstables
or compacting for example.

Regards

Carlos Alonso | Software Engineer | @calonso 

Secondary indices on boolean type columns

[anuja jain]

I have two questions,
1. Does creating secondary index on low cardinality columns like of boolean
type helps in read performance any ways? Because there will be only two
values( true and false ) for that column in index table.

2. Should secondary indexes be created on clustering columns even if these
columns are not frequently used in where clause of a query?

Stale node decommission?

[Carlos Alonso]

Hi guys, I’m decommissioning a node and the nodetool netstats bytes count
doesn’t move, have you seen it before?


Mode: LEAVING

Unbootstrap 6eeec0c0-6d9a-11e5-addd-8bc1b1cfbf0a

/XXX.XXX.XXX.XXX

Sending 0 files, 2773603294 bytes total. Already sent 0 files, 0
bytes total

/XXX.XXX.XXX.XXX

Sending 0 files, 2994447775 bytes total. Already sent 0 files, 0
bytes total

/XXX.XXX.XXX.XXX

Sending 0 files, 2765944684 bytes total. Already sent 0 files, 0
bytes total

Read Repair Statistics:

Attempted: 2041

Mismatch (Blocking): 0

Mismatch (Background): 0

Pool NameActive   Pending  Completed

Commandsn/a 0   6917

Responses   n/a 01279658


read repair values keep growing, but the files/bytes sent count is stale
since the very beginning

as an aside, the node being decommisioned is in a different version to the
others


Any ideas?


Carlos Alonso | Software Engineer | @calonso 

Re: Why can't nodetool status include a hostname?

[Paulo Motta]

Have you tried using the -r or --resolve-ip option?

2015-10-07 19:59 GMT-07:00 Kevin Burton :

> I find it really frustrating that nodetool status doesn't include a
> hostname
>
> Makes it harder to track down problems.
>
> I realize it PRIMARILY uses the IP but perhaps cassandra.yml can include
> an optional 'hostname' parameter that can be set by the user.  OR have the
> box itself include the hostname in gossip when it starts up.
>
> I realize that hostname wouldn't be authoritative and that the IP must
> still be shown but we could add another column for the hostname.
>
> --
>
> We’re hiring if you know of any awesome Java Devops or Linux Operations
> Engineers!
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> 
>
>

Re: Secondary indices on boolean type columns

[Jack Krupansky]

Secondary indexes are not a recommended best practice. Better choices:

1. The new 3.0 materialized views.
2. Manual query tables
3. DSE Search/Solr
4a. Stratio Lucene indexing
4b. TupleJump Stargate Lucene indexing

In any case, low cardinality is not recommended for indexes. The goal is
that a query on the index will return a modest to moderate number of rows
from a single node. Very high cardinality is also not recommended since
indexes are local, so fanout is required if the coordinator node cannot
supply the desired row limit.

-- Jack Krupansky

On Thu, Oct 8, 2015 at 5:37 AM, anuja jain  wrote:

> I have two questions,
> 1. Does creating secondary index on low cardinality columns like of
> boolean type helps in read performance any ways? Because there will be only
> two values( true and false ) for that column in index table.
>
> 2. Should secondary indexes be created on clustering columns even if these
> columns are not frequently used in where clause of a query?
>

Cassandra 1.x upgrade to Cassandra 2.x

[Caraballo, Rafael]

Hi,

When planning an upgrade on a 20 node multi DC cluster from Cassandra 1.x to 
Cassandra 2.x version, does the order of running upgradesstables matter?

What is the difference between:
(a) upgrade Cassandra and run upgradesstables one node at a time;
-versus-
(b) upgrade every Cassandra node, then start the upgradesstables one node at a 
time?

Thanks,
Rafael




This E-mail and any of its attachments may contain Time Warner Cable 
proprietary information, which is privileged, confidential, or subject to 
copyright belonging to Time Warner Cable. This E-mail is intended solely for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient of this E-mail, you are hereby notified that any 
dissemination, distribution, copying, or action taken in relation to the 
contents of and attachments to this E-mail is strictly prohibited and may be 
unlawful. If you have received this E-mail in error, please notify the sender 
immediately and permanently delete the original and any copy of this E-mail and 
any printout.

Node won't go away

[Robert Wille]

We had some problems with a node, so we decided to rebootstrap it. My IT guy 
screwed up, and when he added -Dcassandra.replace_address to cassandra-env.sh, 
he forgot the closing quote. The node bootstrapped, and then refused to join 
the cluster. We shut it down, and then noticed that nodetool status no longer 
showed that node, and the “Owns” column had increased from ~10% per node to 
~11% (we originally had 10 nodes). I don’t know why Cassandra decided to 
automatically remove the node from the cluster, but it did. We figured it would 
be best to make sure the node was completely forgotten, and then add it back 
into the cluster as a new node. Problem is, it won’t completely go away.

nodetool status doesn’t list it, but its still in system.peers, and OpsCenter 
still shows it. When I run nodetool removenode, it says that it can’t find the 
node.

How do I completely get rid of it?

Thanks in advance

Robert

Re: Secondary indices on boolean type columns

[Robert Coli]

On Thu, Oct 8, 2015 at 2:37 AM, anuja jain  wrote:

> 1. Does creating secondary index on low cardinality columns like of
> boolean type helps in read performance any ways?
>
No.


> 2. Should secondary indexes be created on clustering columns even if these
> columns are not frequently used in where clause of a query?
>
Secondary Indexes should generally not be used, IMO.

=Rob

RE: Cassandra 1.x upgrade to Cassandra 2.x

[Caraballo, Rafael]

Is there any reason to wait for all the Cassandra nodes to be upgrades from 1.x 
to 2.x before running the first upgradesstables?


From: Robert Coli [mailto:rc...@eventbrite.com]
Sent: Thursday, October 08, 2015 11:30 AM
To: user@cassandra.apache.org
Subject: Re: Cassandra 1.x upgrade to Cassandra 2.x

On Thu, Oct 8, 2015 at 7:44 AM, Caraballo, Rafael 
> wrote:
When planning an upgrade on a 20 node multi DC cluster from Cassandra 1.x to 
Cassandra 2.x version, does the order of running upgradesstables matter?

What is the difference between:
(a) upgrade Cassandra and run upgradesstables one node at a time;
-versus-
(b) upgrade every Cassandra node, then start the upgradesstables one node at a 
time?

This is becoming a FAQ, I should write a blog post...

Briefly, until upgradesstables is done, one must assume that streaming 
operations will fail. Streaming is used in bootstrap, decommission and repair. 
Bootstrap is required to replace or add nodes, decommission is required to 
remove nodes, and repair is required once every gc_grace_seconds.

One must balance the overhead of running upgradesstables simultaneously on 
multiple nodes with the duration of exposure to inability-to-use streaming. 
Often this means running upgradesstables in parallel on all hosts 
simultaneously.

=Rob




This E-mail and any of its attachments may contain Time Warner Cable 
proprietary information, which is privileged, confidential, or subject to 
copyright belonging to Time Warner Cable. This E-mail is intended solely for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient of this E-mail, you are hereby notified that any 
dissemination, distribution, copying, or action taken in relation to the 
contents of and attachments to this E-mail is strictly prohibited and may be 
unlawful. If you have received this E-mail in error, please notify the sender 
immediately and permanently delete the original and any copy of this E-mail and 
any printout.

Re: Does failing to run "nodetool cleanup" end up causing more data to be transferred during bootstrapping?

[Robert Coli]

On Wed, Oct 7, 2015 at 9:06 PM, Kevin Burton  wrote:

> Let's say I have 10 nodes, I add 5 more, if I fail to run nodetool
> cleanup, is excessive data transferred when I add the 6th node?  IE do the
> existing nodes send more data to the 6th node?
>

No. Streaming only streams ranges which are currently owned by the source
and will be owned by the target.

https://issues.apache.org/jira/browse/CASSANDRA-7764

Has some details on the type of edge cases one is exposed to if one does
not run cleanup; mostly they involve moving a range away from a node and
then back onto it.


> but I also think that the excessive data will be removed in the next major
> compaction and that nodetool cleanup just triggers a major compaction.
>

Nothing* removes data-which-doesn't-belong-on-the-node-it's-on but cleanup
compaction (*or scrub).

=Rob

Re: Why can't nodetool status include a hostname?

[Gene]

Yeah, -r or --resolve-ip is what you're looking for.

Cassandra's nodetool command is kind of wonky.  Inconsistent across
functions (e.g. sometimes 'keyspace.columnfamily' other times 'keyspace
columnfamily', pay attention to the character between the items), doesn't
resolve IPs by default (while standard linux commands require you to pass
something like -n to not resolve names), so on and so forth.

When in doubt run nodetool without specifying a command and it'll list all
of the available options (another example of wonkiness, the 'help' argument
is not listed in this output)

-Gene

On Thu, Oct 8, 2015 at 7:01 AM, Paulo Motta 
wrote:

> Have you tried using the -r or --resolve-ip option?
>
> 2015-10-07 19:59 GMT-07:00 Kevin Burton :
>
>> I find it really frustrating that nodetool status doesn't include a
>> hostname
>>
>> Makes it harder to track down problems.
>>
>> I realize it PRIMARILY uses the IP but perhaps cassandra.yml can include
>> an optional 'hostname' parameter that can be set by the user.  OR have the
>> box itself include the hostname in gossip when it starts up.
>>
>> I realize that hostname wouldn't be authoritative and that the IP must
>> still be shown but we could add another column for the hostname.
>>
>> --
>>
>> We’re hiring if you know of any awesome Java Devops or Linux Operations
>> Engineers!
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> 
>>
>>
>

Re: Cassandra 1.x upgrade to Cassandra 2.x

[Robert Coli]

On Thu, Oct 8, 2015 at 7:44 AM, Caraballo, Rafael <
rafael.caraba...@twcable.com> wrote:

> When planning an upgrade on a 20 node multi DC cluster from Cassandra 1.x
> to Cassandra 2.x version, does the order of running upgradesstables matter?
>
>
>
> What is the difference between:
>
> (a) upgrade Cassandra and run upgradesstables one node at a time;
>
> -versus-
>
> (b) upgrade every Cassandra node, then start the upgradesstables one node
> at a time?
>

This is becoming a FAQ, I should write a blog post...

Briefly, until upgradesstables is done, one must assume that streaming
operations will fail. Streaming is used in bootstrap, decommission and
repair. Bootstrap is required to replace or add nodes, decommission is
required to remove nodes, and repair is required once every
gc_grace_seconds.

One must balance the overhead of running upgradesstables simultaneously on
multiple nodes with the duration of exposure to inability-to-use streaming.
Often this means running upgradesstables in parallel on all hosts
simultaneously.

=Rob

Re: Stale node decommission?

[Robert Coli]

On Thu, Oct 8, 2015 at 5:20 AM, Carlos Alonso  wrote:

> Hi guys, I’m decommissioning a node and the nodetool netstats bytes count
> doesn’t move, have you seen it before?
>

Your decom stream is hung, if you don't have a version with the new
non-infinite timeout for streams, you will have to restart the node and
restart the decommission.

=Rob

[cassandra 2.1.3] Missing host ID

[Eduardo Cusa]

Hi Guys, I have a custer with 12 nodes.

when I restart one of them I receive the error "Missing host ID":



WARN  [SharedPool-Worker-1] 2015-10-08 13:15:33,882
AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
Thread[SharedPool-Worker-1,5,main]: {}
java.lang.AssertionError: Missing host ID for 63.251.156.141
at
org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:978)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:950)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2235)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_60]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.3.jar:2.1.3]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
[apache-cassandra-2.1.3.jar:2.1.3]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]




If I made nodetool status, the problematic node has ID:

UN  10.10.10.12  1.3 TB 1   ?
4d5c8fd2-a909-4f09-a23c-4cd6040f338a  rack3




Any idea what could be happening?


Regards
Eduardo

Re: Node won't go away

[Michael Shuler]

On 10/08/2015 10:27 AM, Robert Wille wrote:

nodetool status doesn’t list it, but its still in system.peers, and
OpsCenter still shows it. When I run nodetool removenode, it says
that it can’t find the node.

How do I completely get rid of it?


https://twitter.com/faltering/status/559845791741657088

"Don't forget the hierarchy of node removal in #cassandra: decommission, 
removenode, removenode force, assassinate.  Escalate in that order."


--
Michael

Re: Stale node decommission?

[Carlos Alonso]

I ended up killing the node (nodetool removenode) because the streaming was
not working (probably because of the different sstables versions)

These guys were appearing in logs:

ERROR [STREAM-IN-/XXX.XXX.XXX.XXX] 2015-10-08 12:52:43,202
 StreamSession.java:505 - [Stream #75fd3ab0-6dbb-11e5-a275-8bc1b1cfbf0a]
Streaming error occurred

java.io.IOException: Connection reset by peer

at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_75]

at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
~[na:1.7.0_75]

at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
~[na:1.7.0_75]

at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.7.0_75]

at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
~[na:1.7.0_75]

at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
~[cassandra-all-2.1.9.791.jar:2.1.9.791]

at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
~[cassandra-all-2.1.9.791.jar:2.1.9.791]

at java.lang.Thread.run(Thread.java:745) [na:1.7.0_75]

Carlos Alonso | Software Engineer | @calonso 

On 8 October 2015 at 17:31, Robert Coli  wrote:

> On Thu, Oct 8, 2015 at 5:20 AM, Carlos Alonso  wrote:
>
>> Hi guys, I’m decommissioning a node and the nodetool netstats bytes
>> count doesn’t move, have you seen it before?
>>
>
> Your decom stream is hung, if you don't have a version with the new
> non-infinite timeout for streams, you will have to restart the node and
> restart the decommission.
>
> =Rob
>

Output from FailureDetector MBean changed in 2.1.10

[Jared Biel]

Hello,

We use the output from the AllEndpointStates attribute of the
org.apache.cassandra.net:type=FailureDetector MBean via MX4J for basic
monitoring/sanity checking. I've found that the output changed between
2.1.9 and 2.1.10; there is now additional info for some of the keys (DC,
RELEASE_VERSION, STATUS, etc.)

Is this maybe due to CASSANDRA-10330
? I perused the
changelog, JIRA, and the git repo and didn't see anything else that really
stuck out at me. Is the new output documented somewhere?

*Example 2.1.9 output:*
["DC:us-west-2",
 "HOST_ID:ed17c00e-2015-4b0f-9ee1-fa96e6c2ff8c",
 "LOAD:3525337.0",
 "NET_VERSION:8",
 "RACK:2a",
 "RELEASE_VERSION:2.1.9",
 "RPC_ADDRESS:1.2.3.4",
 "SCHEMA:b3a65c2a-8955-30f3-b403-54db3260f777",
 "SEVERITY:0.0",
 "STATUS:NORMAL,-1012622047045121026",
 "generation:1441736600",
 "heartbeat:7847815"]

*Example 2.1.10 output:*
["DC:6:us-east",
 "HOST_ID:2:8c8ed6f6-84d9-494d-a6c3-a9701cefe347",
 "LOAD:537977:1.45221778E8",
 "NET_VERSION:1:8",
 "RACK:8:1b",
 "RELEASE_VERSION:4:2.1.10",
 "RPC_ADDRESS:3:1.2.3.4",
 "SCHEMA:10:d58b9de5-9e25-3128-b07e-dfa0b2568ae8",
 "SEVERITY:538103:0.0",
 "STATUS:14:NORMAL,-1022178689192707266",
 "TOKENS:13:",
 "generation:1444146690",
 "heartbeat:538104"]

Thanks!

Re: [cassandra 2.1.3] Missing host ID

[Eduardo Cusa]

Hello Paulo, this issue started today and always happened in the same node.

Run the following command seems to solve the problem :

$ nodetool truncatehints


Now the node is up.

Regards

Eduardo




2015-10-08 15:41 GMT-03:00 Paulo Motta :

> Hello Eduardo,
>
> Your node is trying to write a hint to another node (after a timed out
> write), but because of some race condition it does not have its token table
> updated soon after startup, so it cannot locate the node with that ID. You
> should not be worried, as the only consequence is that one hint was lost
> and data consistency can be fixed with a simple repair (or during read
> repairs).
>
> Some other people have reported a similar condition so I opened a JIRA
> ticket: https://issues.apache.org/jira/browse/CASSANDRA-10485
>
> Some questions to help troubleshooting:
>
> - Does it happen always with the same node or any node that you restart?
> - Was that node ever replaced or upgraded?
> - With what frequency does it happen?
>
> Thanks,
>
> Paulo
>
> 2015-10-08 10:45 GMT-07:00 Eduardo Cusa :
>
>> Hi Guys, I have a custer with 12 nodes.
>>
>> when I restart one of them I receive the error "Missing host ID":
>>
>>
>>
>> WARN  [SharedPool-Worker-1] 2015-10-08 13:15:33,882
>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>> Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.AssertionError: Missing host ID for 63.251.156.141
>> at
>> org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:978)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:950)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2235)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[na:1.8.0_60]
>> at
>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>> at
>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> [apache-cassandra-2.1.3.jar:2.1.3]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>
>>
>>
>>
>> If I made nodetool status, the problematic node has ID:
>>
>> UN  10.10.10.12  1.3 TB 1   ?
>> 4d5c8fd2-a909-4f09-a23c-4cd6040f338a  rack3
>>
>>
>>
>>
>> Any idea what could be happening?
>>
>>
>> Regards
>> Eduardo
>>
>>
>>
>

Re: Duplicate records returned

[Robert Wille]

If anyone is following this, I also logged the bug at 
https://datastax-oss.atlassian.net/browse/JAVA-943. I suspect that its a driver 
bug, so I anticipate CASSANDRA-10442 being closed, and hopefully the folks at 
datastax can get this fixed. This bug must affect a whole lot of people.

On Oct 3, 2015, at 2:33 PM, Robert Wille 
> wrote:

It's a paging bug. I ALWAYS get a duplicated record every fetchSize records. 
Easily duplicated 100% of the time.

I’ve logged a bug: https://issues.apache.org/jira/browse/CASSANDRA-10442

Robert

On Oct 3, 2015, at 10:59 AM, Robert Wille 
> wrote:

Oops, I was trimming out some irrelevant stuff, and trimmed out too much. The 
second snippet should be this:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;

for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

if (imageId == lastImageId)
{
logger.warn("Cassandra duplicated " + imageId);
continue;
}

total++;
lastImageId = imageId;
}


On Oct 3, 2015, at 10:54 AM, Robert Wille 
> wrote:

I don’t think its an application problem. The following simple snippets produce 
different totals:

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
for (Row row : rs)
{
total++;
}

-

ResultSet rs = cassandraUtil.executeNamedQuery(Q_LIST_IMAGES, rollId);

int total = 0;
long lastImageId = -1;
for (Row row : rs)
{
long imageId = row.getLong(PARAM_IMAGE_ID);

total++;
lastImageId = imageId;
}

This doesn’t happen for all partitions. In fact most don’t have this problem 
(maybe 20% do this). But the ones that do repeat records, do so 
deterministically. I see this problem in multiple tables.

I’m only retrieving the clustering key, so it has nothing to do with the data 
field.

I suspect this is a paging problem, and might be a driver issue.

Robert

On Oct 3, 2015, at 9:29 AM, Eric Stevens 
> wrote:

Can you give us an example of the duplicate records that comes back?  How 
reliable is it (i.e. is it every record, is it one record per read, etc)?  By 
any chance is it just the `data` field that duplicates while the other fields 
change per row?

> I don’t see duplicates in cqlsh.

I've never seen this, and I can't think of a failure mode which would cause it 
to happen.  Not to say it's impossible, but Cassandra's standard read path 
involves collapsing duplicate or otherwise overlapping answers from multiple 
replicas; such a thing would be a pretty substantial deviation.  Especially 
since you don't see the duplicates in cqlsh, I have a hunch this is an 
application bug.


On Fri, Oct 2, 2015 at 4:58 PM Robert Wille 
> wrote:
When I run the query "SELECT image FROM roll WHERE roll = :roll“ against this 
table

CREATE TABLE roll (
roll INT,
image BIGINT,
data VARCHAR static,
mid VARCHAR,
imp_st VARCHAR,
PRIMARY KEY ((roll), image)
) WITH gc_grace_seconds = 3456000 AND compaction = { 'class' : 
'LeveledCompactionStrategy', 'sstable_size_in_mb' : 160 };

I often get duplicate records back. Seems like a very simple query to botch. 
I’m running 2.0.16 with RF=3 and CL=QUORUM and Java client 2.0.10.1. I don’t 
see duplicates in cqlsh. Any thoughts?

Thanks

Robert

Re: Cassandra query degradation with high frequency updated tables.

[Tyler Hobbs]

Upgrade to 2.2.2.  Your sstables are probably not compacting due to
CASSANDRA-10270 ,
which was fixed in 2.2.2.

Additionally, you may want to look into using leveled compaction (
http://www.datastax.com/dev/blog/when-to-use-leveled-compaction).

On Thu, Oct 8, 2015 at 4:27 PM, Nazario Parsacala 
wrote:

>
> Hi,
>
> so we are developing a system that computes profile of things that it
> observes. The observation comes in form of events. Each thing that it
> observe has an id and each thing has a set of subthings in it which has
> measurement of some kind. Roughly there are about 500 subthings within each
> thing. We receive events containing measurements of these 500 subthings
> every 10 seconds or so.
>
> So as we receive events, we  read the old profile value, calculate the new
> profile based on the new value and save it back. We use the following
> schema to hold the profile.
>
> CREATE TABLE myprofile (
> id text,
> month text,
> day text,
> hour text,
> subthings text,
> lastvalue double,
> count int,
> stddev double,
>  PRIMARY KEY ((id, month, day, hour), subthings)
> ) WITH CLUSTERING ORDER BY (subthings ASC) );
>
>
> This profile will then be use for certain analytics that can use in the
> context of the ‘thing’ or in the context of specific thing and subthing.
>
> A profile can be defined as monthly, daily, hourly. So in case of monthly
> the month will be set to the current month (i.e. ‘Oct’) and the day and
> hour will be set to empty ‘’ string.
>
>
> The problem that we have observed is that over time (actually in just a
> matter of hours) we will see a huge degradation of query response  for the
> monthly profile. At the start it will be respinding in 10-100 ms and after
> a couple of hours it will go to 2000-3000 ms . If you leave it for a couple
> of days you will start experiencing readtimeouts . The query is basically
> just :
>
> select * from myprofile where id=‘1’ and month=‘Oct’ and day=‘’ and hour=‘'
>
> This will have only about 500 rows or so.
>
>
> I believe that this is cause by the fact there are multiple updates done
> to this specific partition. So what do we think can be done to resolve this
> ?
>
> BTW, I am using Cassandra 2.2.1 . And since this is a test , this is just
> running on a single node.
>
>
>
>
>


-- 
Tyler Hobbs
DataStax 

Realtime data and (C)AP

[Renato Perini]

How the two things can fit together?
Cassandra endorses the AP side of the CAP theorem. So how Cassandra can 
deliver realtime consistent data?
AFAIK, choosing a consistency level equals to ALL can be a huge 
performance hit for C*, so, please, explain me why I should choose C* 
for realtime data

that should be consistent across reads after the data has been written.

Thank you.

Cassandra query degradation with high frequency updated tables.

[Nazario Parsacala]

Hi,

so we are developing a system that computes profile of things that it observes. 
The observation comes in form of events. Each thing that it observe has an id 
and each thing has a set of subthings in it which has measurement of some kind. 
Roughly there are about 500 subthings within each thing. We receive events 
containing measurements of these 500 subthings every 10 seconds or so.

So as we receive events, we  read the old profile value, calculate the new 
profile based on the new value and save it back. We use the following schema to 
hold the profile. 

CREATE TABLE myprofile (
id text,
month text,
day text,
hour text,
subthings text,
lastvalue double,
count int,
stddev double,
 PRIMARY KEY ((id, month, day, hour), subthings)
) WITH CLUSTERING ORDER BY (subthings ASC) );


This profile will then be use for certain analytics that can use in the context 
of the ‘thing’ or in the context of specific thing and subthing. 

A profile can be defined as monthly, daily, hourly. So in case of monthly the 
month will be set to the current month (i.e. ‘Oct’) and the day and hour will 
be set to empty ‘’ string.


The problem that we have observed is that over time (actually in just a matter 
of hours) we will see a huge degradation of query response  for the monthly 
profile. At the start it will be respinding in 10-100 ms and after a couple of 
hours it will go to 2000-3000 ms . If you leave it for a couple of days you 
will start experiencing readtimeouts . The query is basically just :

select * from myprofile where id=‘1’ and month=‘Oct’ and day=‘’ and hour=‘'

This will have only about 500 rows or so.


I believe that this is cause by the fact there are multiple updates done to 
this specific partition. So what do we think can be done to resolve this ? 

BTW, I am using Cassandra 2.2.1 . And since this is a test , this is just 
running on a single node.

CQL error when adding multiple conditional update statements in the same batch

[Peddi, Praveen]

Hi,

I am trying to understand this error message that CQL is throwing when I try to 
update 2 different rows with different values on same conditional columns. 
Doesn't CQL support that? I am wondering why CQL has this restriction (since 
condition applies to each row independently, why does CQL even care if the 
values of the condition is same or different).

BEGIN BATCH
UPDATE activities SET state='CLAIMED',version=11 WHERE key='Key1' IF 
version=10;
UPDATE activities SET state='ALLOCATED',version=2 WHERE key='Key2' IF 
version=1;
APPLY BATCH;


gives the following error

Bad Request: Duplicate and incompatible conditions for column version

Is there anyway to update more than 1 row with different conditional value for 
each row (other than executing these statements individually)?

-Praveen

Re: CQL error when adding multiple conditional update statements in the same batch

[sai krishnam raju potturi]

could you also provide the columnfamily schema.

On Thu, Oct 8, 2015 at 4:13 PM, Peddi, Praveen  wrote:

> Hi,
>
> I am trying to understand this error message that CQL is throwing when I
> try to update 2 different rows with different values on same conditional
> columns. Doesn't CQL support that? I am wondering why CQL has this
> restriction (since condition applies to each row independently, why does
> CQL even care if the values of the condition is same or different).
>
> BEGIN BATCH
> UPDATE activities SET state='CLAIMED',version=11 WHERE key='Key1' IF 
> version=10;
> UPDATE activities SET state='ALLOCATED',version=2 WHERE key='Key2' IF 
> version=1;
> APPLY BATCH;
>
> gives the following error
>
> Bad Request: Duplicate and incompatible conditions for column version
>
> Is there anyway to update more than 1 row with different conditional value
> for each row (other than executing these statements individually)?
> -Praveen
>
>

Re: Realtime data and (C)AP

[Jonathan Haddad]

Your options are

1. Read & write at quorum
2. Recognize that, in general, if you've got a real need for Cassandra,
your data is out of date almost immediately after you've read it no matter
what guarantee your DB gives you, so you might as well just forget about
ever getting the "right" answer because you probably can't even define what
that is.

On Thu, Oct 8, 2015 at 4:17 PM Renato Perini 
wrote:

> How the two things can fit together?
> Cassandra endorses the AP side of the CAP theorem. So how Cassandra can
> deliver realtime consistent data?
> AFAIK, choosing a consistency level equals to ALL can be a huge
> performance hit for C*, so, please, explain me why I should choose C*
> for realtime data
> that should be consistent across reads after the data has been written.
>
> Thank you.
>
>

Re: Realtime data and (C)AP

[Sebastian Estevez]

Renato, please watch this netflix video on consistency:

http://www.planetcassandra.org/blog/a-netflix-experiment-eventual-consistency-hopeful-consistency-by-christos-kalantzis/

All the best,


[image: datastax_logo.png] 

Sebastián Estévez

Solutions Architect | 954 905 8615 | sebastian.este...@datastax.com

[image: linkedin.png]  [image:
facebook.png]  [image: twitter.png]
 [image: g+.png]





DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.

On Thu, Oct 8, 2015 at 7:34 PM, Jonathan Haddad  wrote:

> Your options are
>
> 1. Read & write at quorum
> 2. Recognize that, in general, if you've got a real need for Cassandra,
> your data is out of date almost immediately after you've read it no matter
> what guarantee your DB gives you, so you might as well just forget about
> ever getting the "right" answer because you probably can't even define what
> that is.
>
> On Thu, Oct 8, 2015 at 4:17 PM Renato Perini 
> wrote:
>
>> How the two things can fit together?
>> Cassandra endorses the AP side of the CAP theorem. So how Cassandra can
>> deliver realtime consistent data?
>> AFAIK, choosing a consistency level equals to ALL can be a huge
>> performance hit for C*, so, please, explain me why I should choose C*
>> for realtime data
>> that should be consistent across reads after the data has been written.
>>
>> Thank you.
>>
>>

Re: CQL error when adding multiple conditional update statements in the same batch

[Tyler Hobbs]

I assume you're running Cassandra 2.0?

In 2.1.1 the check for "incompatible" conditions was removed (see this
comment

for details).  I wouldn't be surprised if that check didn't work properly
for batch statements in 2.0.

On Thu, Oct 8, 2015 at 3:22 PM, sai krishnam raju potturi <
pskraj...@gmail.com> wrote:

> could you also provide the columnfamily schema.
>
> On Thu, Oct 8, 2015 at 4:13 PM, Peddi, Praveen  wrote:
>
>> Hi,
>>
>> I am trying to understand this error message that CQL is throwing when I
>> try to update 2 different rows with different values on same conditional
>> columns. Doesn't CQL support that? I am wondering why CQL has this
>> restriction (since condition applies to each row independently, why does
>> CQL even care if the values of the condition is same or different).
>>
>> BEGIN BATCH
>> UPDATE activities SET state='CLAIMED',version=11 WHERE key='Key1' IF 
>> version=10;
>> UPDATE activities SET state='ALLOCATED',version=2 WHERE key='Key2' IF 
>> version=1;
>> APPLY BATCH;
>>
>> gives the following error
>>
>> Bad Request: Duplicate and incompatible conditions for column version
>>
>> Is there anyway to update more than 1 row with different conditional
>> value for each row (other than executing these statements individually)?
>> -Praveen
>>
>>
>


-- 
Tyler Hobbs
DataStax 

Re: Re : Nodetool Cleanup on multiple nodes in parallel

[Jonathan Haddad]

Unless you're close to running out of disk space, what's the harm in it
taking a while?  How big is your DC?  At 45 min per node, you can do 32
nodes a day.  Diverting traffic away from a DC just to run cleanup feels
like overkill to me.



On Thu, Oct 8, 2015 at 2:39 PM sai krishnam raju potturi <
pskraj...@gmail.com> wrote:

> hi;
>our cassandra cluster currently uses DSE 4.6. The underlying cassandra
> version is 2.0.14.
>
> We are planning on adding multiple nodes to one of our datacenters. This
> requires "nodetool cleanup". The "nodetool cleanup" operation takes
> around 45 mins for each node.
>
> Datastax documentation recommends running "nodetool cleanup" for one node
> at a time. That would be really long, owing to the size of our
> datacenter.
>
> If we were to divert the read and write traffic away from a particular
> datacenter, could we run "cleanup" on multiple nodes in parallel for that
> datacenter??
>
>
> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html
>
>
> thanks
> Sai
>

Re: Re : Nodetool Cleanup on multiple nodes in parallel

[sai krishnam raju potturi]

We plan to do it during non-peak hours when customer traffic is less. That
sums up to 10 nodes a day, which is concerning as we have other data
centers to be expanded eventually.

Since cleanup is similar to compaction, which is CPU intensive and will
effect reads  if this data center were to serve traffic. Is running cleanup
in parallel advisable??

On Thu, Oct 8, 2015, 17:53 Jonathan Haddad  wrote:

> Unless you're close to running out of disk space, what's the harm in it
> taking a while?  How big is your DC?  At 45 min per node, you can do 32
> nodes a day.  Diverting traffic away from a DC just to run cleanup feels
> like overkill to me.
>
>
>
> On Thu, Oct 8, 2015 at 2:39 PM sai krishnam raju potturi <
> pskraj...@gmail.com> wrote:
>
>> hi;
>>our cassandra cluster currently uses DSE 4.6. The underlying cassandra
>> version is 2.0.14.
>>
>> We are planning on adding multiple nodes to one of our datacenters. This
>> requires "nodetool cleanup". The "nodetool cleanup" operation takes
>> around 45 mins for each node.
>>
>> Datastax documentation recommends running "nodetool cleanup" for one
>> node at a time. That would be really long, owing to the size of our
>> datacenter.
>>
>> If we were to divert the read and write traffic away from a particular
>> datacenter, could we run "cleanup" on multiple nodes in parallel for
>> that datacenter??
>>
>>
>> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html
>>
>>
>> thanks
>> Sai
>>
>

Re: CQL error when adding multiple conditional update statements in the same batch

[Peddi, Praveen]

Sorry Cassandra 2.0.9 (I was using 2.0.8 but there was another bug in 2.0.8 so 
upgraded to 2.0.9)

On Oct 8, 2015, at 4:43 PM, Peddi, Praveen 
> wrote:

Cassandra 2.0.8.





From: Tyler Hobbs >
Reply-To: "user@cassandra.apache.org" 
>
Date: Thursday, October 8, 2015 at 4:41 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: CQL error when adding multiple conditional update statements in 
the same batch

I assume you're running Cassandra 2.0?

In 2.1.1 the check for "incompatible" conditions was removed (see this 
comment
 for details).  I wouldn't be surprised if that check didn't work properly for 
batch statements in 2.0.

On Thu, Oct 8, 2015 at 3:22 PM, sai krishnam raju potturi 
> wrote:
could you also provide the columnfamily schema.

On Thu, Oct 8, 2015 at 4:13 PM, Peddi, Praveen 
> wrote:

Hi,

I am trying to understand this error message that CQL is throwing when I try to 
update 2 different rows with different values on same conditional columns. 
Doesn't CQL support that? I am wondering why CQL has this restriction (since 
condition applies to each row independently, why does CQL even care if the 
values of the condition is same or different).

BEGIN BATCH
UPDATE activities SET state='CLAIMED',version=11 WHERE key='Key1' IF 
version=10;
UPDATE activities SET state='ALLOCATED',version=2 WHERE key='Key2' IF 
version=1;
APPLY BATCH;


gives the following error

Bad Request: Duplicate and incompatible conditions for column version

Is there anyway to update more than 1 row with different conditional value for 
each row (other than executing these statements individually)?

-Praveen








--
Tyler Hobbs
DataStax

Re: Re : Nodetool Cleanup on multiple nodes in parallel

[Jonathan Haddad]

My hunch is the bigger your cluster the less impact it will have, as each
node takes part in smaller and smaller % of total queries.  Considering
that compaction is always happening, I'd wager if you've got a big cluster
(as you say you do) you'll probably be ok running several cleanups at a
time.

I'd say start one, see how your perf is impacted (if at all) and go from
there.

If you're running a proper snitch you could probably do an entire rack / AZ
at a time.


On Thu, Oct 8, 2015 at 3:08 PM sai krishnam raju potturi <
pskraj...@gmail.com> wrote:

> We plan to do it during non-peak hours when customer traffic is less. That
> sums up to 10 nodes a day, which is concerning as we have other data
> centers to be expanded eventually.
>
> Since cleanup is similar to compaction, which is CPU intensive and will
> effect reads  if this data center were to serve traffic. Is running cleanup
> in parallel advisable??
>
> On Thu, Oct 8, 2015, 17:53 Jonathan Haddad  wrote:
>
>> Unless you're close to running out of disk space, what's the harm in it
>> taking a while?  How big is your DC?  At 45 min per node, you can do 32
>> nodes a day.  Diverting traffic away from a DC just to run cleanup feels
>> like overkill to me.
>>
>>
>>
>> On Thu, Oct 8, 2015 at 2:39 PM sai krishnam raju potturi <
>> pskraj...@gmail.com> wrote:
>>
>>> hi;
>>>our cassandra cluster currently uses DSE 4.6. The underlying
>>> cassandra version is 2.0.14.
>>>
>>> We are planning on adding multiple nodes to one of our datacenters. This
>>> requires "nodetool cleanup". The "nodetool cleanup" operation takes
>>> around 45 mins for each node.
>>>
>>> Datastax documentation recommends running "nodetool cleanup" for one
>>> node at a time. That would be really long, owing to the size of our
>>> datacenter.
>>>
>>> If we were to divert the read and write traffic away from a particular
>>> datacenter, could we run "cleanup" on multiple nodes in parallel for
>>> that datacenter??
>>>
>>>
>>> http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html
>>>
>>>
>>> thanks
>>> Sai
>>>
>>

Re: CQL error when adding multiple conditional update statements in the same batch

[Peddi, Praveen]

Cassandra 2.0.8.





From: Tyler Hobbs >
Reply-To: "user@cassandra.apache.org" 
>
Date: Thursday, October 8, 2015 at 4:41 PM
To: "user@cassandra.apache.org" 
>
Subject: Re: CQL error when adding multiple conditional update statements in 
the same batch

I assume you're running Cassandra 2.0?

In 2.1.1 the check for "incompatible" conditions was removed (see this 
comment
 for details).  I wouldn't be surprised if that check didn't work properly for 
batch statements in 2.0.

On Thu, Oct 8, 2015 at 3:22 PM, sai krishnam raju potturi 
> wrote:
could you also provide the columnfamily schema.

On Thu, Oct 8, 2015 at 4:13 PM, Peddi, Praveen 
> wrote:

Hi,

I am trying to understand this error message that CQL is throwing when I try to 
update 2 different rows with different values on same conditional columns. 
Doesn't CQL support that? I am wondering why CQL has this restriction (since 
condition applies to each row independently, why does CQL even care if the 
values of the condition is same or different).

BEGIN BATCH
UPDATE activities SET state='CLAIMED',version=11 WHERE key='Key1' IF 
version=10;
UPDATE activities SET state='ALLOCATED',version=2 WHERE key='Key2' IF 
version=1;
APPLY BATCH;


gives the following error

Bad Request: Duplicate and incompatible conditions for column version

Is there anyway to update more than 1 row with different conditional value for 
each row (other than executing these statements individually)?

-Praveen








--
Tyler Hobbs
DataStax

Re : Nodetool Cleanup on multiple nodes in parallel

[sai krishnam raju potturi]

hi;
   our cassandra cluster currently uses DSE 4.6. The underlying cassandra
version is 2.0.14.

We are planning on adding multiple nodes to one of our datacenters. This
requires "nodetool cleanup". The "nodetool cleanup" operation takes around
45 mins for each node.

Datastax documentation recommends running "nodetool cleanup" for one node
at a time. That would be really long, owing to the size of our datacenter.

If we were to divert the read and write traffic away from a particular
datacenter, could we run "cleanup" on multiple nodes in parallel for that
datacenter??

http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html


thanks
Sai

Re: Realtime data and (C)AP

[Renato Perini]

I'm asking because the DataStax DS-201 course states that C* is an ideal 
fit for messaging applications.

What I'm not understanding? :-)
Messaging applications generally must be totally consistent, expecially 
real-time ones.


Il 09/10/2015 01:34, Jonathan Haddad ha scritto:

Your options are

1. Read & write at quorum
2. Recognize that, in general, if you've got a real need for 
Cassandra, your data is out of date almost immediately after you've 
read it no matter what guarantee your DB gives you, so you might as 
well just forget about ever getting the "right" answer because you 
probably can't even define what that is.


On Thu, Oct 8, 2015 at 4:17 PM Renato Perini > wrote:


How the two things can fit together?
Cassandra endorses the AP side of the CAP theorem. So how
Cassandra can
deliver realtime consistent data?
AFAIK, choosing a consistency level equals to ALL can be a huge
performance hit for C*, so, please, explain me why I should choose C*
for realtime data
that should be consistent across reads after the data has been
written.

Thank you.

Re: Realtime data and (C)AP

[Steve Robenalt]

Hi Renato,

Your original statement is not entirely correct, and the video Sebastian is
suggesting can probably help make it clearer. It's not that Cassandra
favors the AP side of CAP. It more that Cassandra allows you to tune what
level of consistency you require. Cassandra does force you to reckon with
the impact of the tunablility - it's not known to be possible to beat the
CAP theorem yet, despite some occasional rumblings to the contrary.

If you want to optimize for consistency, you can use CL_ALL on both write
and read, but you'll sacrifice availability when a node fails which has a
required replica of your data. If you want to optimize for performance and
fault tolerance, you can read and write at CL_ONE, in which case you won't
always have consistency.

In general, if you write at QUORUM and read at ONE (or LOCAL variants
thereof if you have multiple data centers), your apps will work well
despite the theoretical consistency issues.

If you need better consistency guarantees, there's always the world of SQL
and distributed transactions...

Steve




On Thu, Oct 8, 2015 at 5:04 PM, Renato Perini 
wrote:

> I'm asking because the DataStax DS-201 course states that C* is an ideal
> fit for messaging applications.
> What I'm not understanding? :-)
> Messaging applications generally must be totally consistent, expecially
> real-time ones.
>
>
> Il 09/10/2015 01:34, Jonathan Haddad ha scritto:
>
> Your options are
>
> 1. Read & write at quorum
> 2. Recognize that, in general, if you've got a real need for Cassandra,
> your data is out of date almost immediately after you've read it no matter
> what guarantee your DB gives you, so you might as well just forget about
> ever getting the "right" answer because you probably can't even define what
> that is.
>
> On Thu, Oct 8, 2015 at 4:17 PM Renato Perini < 
> renato.per...@gmail.com> wrote:
>
>> How the two things can fit together?
>> Cassandra endorses the AP side of the CAP theorem. So how Cassandra can
>> deliver realtime consistent data?
>> AFAIK, choosing a consistency level equals to ALL can be a huge
>> performance hit for C*, so, please, explain me why I should choose C*
>> for realtime data
>> that should be consistent across reads after the data has been written.
>>
>> Thank you.
>>
>>
>


-- 
Steve Robenalt
Software Architect
sroben...@highwire.org 
(office/cell): 916-505-1785

HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org

Technology for Scholarly Communication

Re: Is replication possible with already existing data?

[Ajay Garg]

On Thu, Oct 8, 2015 at 9:47 AM, Ajay Garg  wrote:
> Thanks Eric for the reply.
>
>
> On Thu, Oct 8, 2015 at 1:44 AM, Eric Stevens  wrote:
>> If you're at 1 node (N=1) and RF=1 now, and you want to go N=3 RF=3, you
>> ought to be able to increase RF to 3 before bootstrapping your new nodes,
>> with no downtime and no loss of data (even temporary).  Effective RF is
>> min-bounded by N, so temporarily having RF > N ought to behave as RF = N.
>>
>> If you're starting at N > RF and you want to increase RF, things get
>> harrier
>> if you can't afford temporary consistency issues.
>>
>
> We are ok with temporary consistency issues.
>
> Also, I was going through the following articles
> https://10kloc.wordpress.com/2012/12/27/cassandra-chapter-5-data-replication-strategies/
>
> and following doubts came up in my mind ::
>
>
> a)
> Let's say at site-1, Application-Server (APP1) uses the two
> Cassandra-instances (CAS11 and CAS12), and APP1 generally uses CAS11 for all
> its needs (of course, whatever happens on CAS11, the same is replicated to
> CAS12 at Cassandra-level).
>
> Now, if CAS11 goes down, will it be the responsibility of APP1 to "detect"
> this and pick up CAS12 for its needs?
> Or some automatic Cassandra-magic will happen?
>
>
> b)
> In the same above scenario, let's say before CAS11 goes down, the amount of
> data in both CAS11 and CAS12 was "x".
>
> After CAS11 goes down, the data is being put in CAS12 only.
> After some time, CAS11 comes back up.
>
> Now, data in CAS11 is still "x", while data in CAS12 is "y" (obviously, "y"
>> "x").
>
> Now, will the additional ("y" - "x") data be automatically
> put/replicated/whatever back in CAS11 through Cassandra?
> Or it has to be done manually?
>

Any pointers, please ???

>
> If there are easy recommended solutions to above, I am beginning to think
> that a 2*2 (2 nodes each at 2 data-centres) will be the ideal setup
> (allowing failures of entire site, or a few nodes on the same site).
>
> I am sorry for asking such newbie questions, and I will be grateful if these
> silly questions could be answered by the experts :)
>
>
> Thanks and Regards,
> Ajay



-- 
Regards,
Ajay

Re: Realtime data and (C)AP

[Jonathan Haddad]

Most of the things that people thing are "totally consistent" or instant
are in reality, not.  Examples include banks (inconsistent by days), the
time it takes for light to travel around the earth (~150ms), and messaging
(email can take quite minutes).

Try to imagine being perfectly consistent between 2 data centers that are
separated by a large body of water, like NY to London, where you've got a
25ms ping, roughly.  Perfectly consistent, at scale, without downtime, when
a shark bites the undersea cable you're using?  No, not happening.

On Thu, Oct 8, 2015 at 5:05 PM Renato Perini 
wrote:

> I'm asking because the DataStax DS-201 course states that C* is an ideal
> fit for messaging applications.
> What I'm not understanding? :-)
> Messaging applications generally must be totally consistent, expecially
> real-time ones.
>
>
> Il 09/10/2015 01:34, Jonathan Haddad ha scritto:
>
> Your options are
>
> 1. Read & write at quorum
> 2. Recognize that, in general, if you've got a real need for Cassandra,
> your data is out of date almost immediately after you've read it no matter
> what guarantee your DB gives you, so you might as well just forget about
> ever getting the "right" answer because you probably can't even define what
> that is.
>
> On Thu, Oct 8, 2015 at 4:17 PM Renato Perini 
> wrote:
>
>> How the two things can fit together?
>> Cassandra endorses the AP side of the CAP theorem. So how Cassandra can
>> deliver realtime consistent data?
>> AFAIK, choosing a consistency level equals to ALL can be a huge
>> performance hit for C*, so, please, explain me why I should choose C*
>> for realtime data
>> that should be consistent across reads after the data has been written.
>>
>> Thank you.
>>
>>
>

Re: Node won't go away

[sai krishnam raju potturi]

the below solution should work.

For each node in the cluster :
 a : Stop cassandra service on the node.
 b : manually delete data under $data_directory/system/peers/  directory.
 c : In cassandra-env.sh file, add the line JVM_OPTS="$JVM_OPTS
-Dcassandra.load_ring_state=false".
 d : Restart service on the node.
 e : delete the added line in cassandra-env.sh  JVM_OPTS="$JVM_OPTS
-Dcassandra.load_ring_state=false".

thanks
Sai Potturi



On Thu, Oct 8, 2015 at 11:27 AM, Robert Wille  wrote:

> We had some problems with a node, so we decided to rebootstrap it. My IT
> guy screwed up, and when he added -Dcassandra.replace_address to
> cassandra-env.sh, he forgot the closing quote. The node bootstrapped, and
> then refused to join the cluster. We shut it down, and then noticed that
> nodetool status no longer showed that node, and the “Owns” column had
> increased from ~10% per node to ~11% (we originally had 10 nodes). I don’t
> know why Cassandra decided to automatically remove the node from the
> cluster, but it did. We figured it would be best to make sure the node was
> completely forgotten, and then add it back into the cluster as a new node.
> Problem is, it won’t completely go away.
>
> nodetool status doesn’t list it, but its still in system.peers, and
> OpsCenter still shows it. When I run nodetool removenode, it says that it
> can’t find the node.
>
> How do I completely get rid of it?
>
> Thanks in advance
>
> Robert
>
>

Re: [cassandra 2.1.3] Missing host ID

[Paulo Motta]

Hello Eduardo,

Your node is trying to write a hint to another node (after a timed out
write), but because of some race condition it does not have its token table
updated soon after startup, so it cannot locate the node with that ID. You
should not be worried, as the only consequence is that one hint was lost
and data consistency can be fixed with a simple repair (or during read
repairs).

Some other people have reported a similar condition so I opened a JIRA
ticket: https://issues.apache.org/jira/browse/CASSANDRA-10485

Some questions to help troubleshooting:

- Does it happen always with the same node or any node that you restart?
- Was that node ever replaced or upgraded?
- With what frequency does it happen?

Thanks,

Paulo

2015-10-08 10:45 GMT-07:00 Eduardo Cusa :

> Hi Guys, I have a custer with 12 nodes.
>
> when I restart one of them I receive the error "Missing host ID":
>
>
>
> WARN  [SharedPool-Worker-1] 2015-10-08 13:15:33,882
> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[SharedPool-Worker-1,5,main]: {}
> java.lang.AssertionError: Missing host ID for 63.251.156.141
> at
> org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:978)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:950)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2235)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]
> at
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> ~[apache-cassandra-2.1.3.jar:2.1.3]
> at
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> [apache-cassandra-2.1.3.jar:2.1.3]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>
>
>
>
> If I made nodetool status, the problematic node has ID:
>
> UN  10.10.10.12  1.3 TB 1   ?
> 4d5c8fd2-a909-4f09-a23c-4cd6040f338a  rack3
>
>
>
>
> Any idea what could be happening?
>
>
> Regards
> Eduardo
>
>
>

Re: Cassandra 1.x upgrade to Cassandra 2.x

[Robert Coli]

On Thu, Oct 8, 2015 at 9:40 AM, Caraballo, Rafael <
rafael.caraba...@twcable.com> wrote:

> Is there any reason to wait for all the Cassandra nodes to be upgrades
> from 1.x to 2.x before running the first upgradesstables?
>

Not really, because the new version nodes will start to flush new version
sstables at which point you can't stream from them anyway.

=Rob