Hanging pending compactions
Hi, After executing `nodetool cleanup` on some nodes they are all showing lots (123, 97, 64) of pending compaction tasks, but not a single active task. I'm running Cassandra 2.0.14 with Leveled Compaction Strategy on most of our tables. Anyone experienced this before? Also, is there any way for me to extract debugging information to file a bug report before restarting the nodes? Cheers, Jens -- Jens Rantil Backend engineer Tink AB Email: jens.ran...@tink.se Phone: +46 708 84 18 32 Web: www.tink.se Facebook https://www.facebook.com/#!/tink.se Linkedin http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_phototrkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary Twitter https://twitter.com/tink
Re: Consistent reads and first write wins
Hi John,
The general answer: Each cell in a CQL table has a corresponding timestamp
which is taken from the clock on the Cassandra node that orchestrates the
write. When you are reading from a Cassandra cluster the node that
coordinates the read will compare the timestamps of the values it fetches.
Last write(=highest timestamp) wins and will be returned to the client.
As you may now understand, the above is why it is crucial you NTP sync your
Cassandra nodes.
If time_uuid_1 comes before time_uuid_2 and if both clients follow up the
writes with quorum reads, then will both clients see the value 'bar' for
prop1?
As you might have understood by now, the values of your timeuuid aren't
really relevant here - the timestamp transparently taken from the clock of
the coordinating node is. This is because you could supply your own
timeuuid from the client, which might have a differing clock. However, it
will basically correspond to the timestamp if you use the helper function
`now()` in CQL.
Anyway, if you make a quorum write (that succeeds) and then make a
successful quorum read, you can be 100% that you will get the latest value.
Are there situations in which clients might see different values?
I can see three scenarios where that could happen:
1. If you write with a weaker consistency such as ONE and read quorum.
2. If you write with quorum and read with a weaker consistency such as
ONE.
3. If you make a quorum write that fails. That write might still have
been applied to some node. Cassandra does not guarantee atomic writes (that
is, either applied or not at all). In other words a failed write will not
roll back partially applied writes in any way.
Cheers,
Jens
On Wed, Jul 8, 2015 at 3:35 AM, John Sanda john.sa...@gmail.com wrote:
Suppose I have the following schema,
CREATE TABLE foo (
id text,
time timeuuid,
prop1 text,
PRIMARY KEY (id, time)
)
WITHCLUSTERING ORDER BY (time ASC);
And I have two clients who execute quorum writes, e.g.,
// client 1
INSERT INTO FOO (id, time, prop1) VALUES ('test', time_uuid_1, 'bar');
// client 2
INSERT INTO FOO (id, time, prop1) VALUES ('test', time_uuid_2, 'bam');
If time_uuid_1 comes before time_uuid_2 and if both clients follow up the
writes with quorum reads, then will both clients see the value 'bar' for
prop1? Are there situations in which clients might see different values?
--
- John
--
Jens Rantil
Backend engineer
Tink AB
Email: jens.ran...@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook https://www.facebook.com/#!/tink.se Linkedin
http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_phototrkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary
Twitter https://twitter.com/tink
Datastax agent - Connection issue
Hi there, Im having some trouble here trying to execute the datastax agent on my cassandra nodes on ubuntu 14.04. I have a cassandra 2.0.15 cluster running on docker containers , so i guess my main problem must be related to the container port exposure. The cluster is ok. The nodetool status show all the nodes UN. My opscenter server have some access to the nodes, because the panel identifies the cluster name and the node quantity is also showing ok, but the opscenter keep showing the box Agent status with the message The DataStax Agent must be connected and running on each node in order for OpsCenter functionality to work properly. Looking into the agent.log on each instance i notice the following error: INFO [StompConnection receiver] 2015-07-08 13:27:39,738 New JMX connection (127.0.0.1:7199) ERROR [StompConnection receiver] 2015-07-08 13:27:39,742 Error connecting via JMX: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root e xception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketException: Connection reset] INFO [StompConnection receiver] 2015-07-08 13:27:39,742 Couldn't get broadcast address, will retry in five seconds. I've found an old issue saying that the random RMI port problem was solved on releases newer than 2.0 ( https://issues.apache.org/jira/browse/CASSANDRA-7087). So i've configured the rmi port to be the same as jmx: JVM_OPTS=$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT The JMX port seens open: telnet localhost 7199 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Connection closed by foreign host. Has anyone here had this kind of problem? Thanks, Francisco Andrade
Re: Read Repair
On Wed, Jul 8, 2015 at 2:07 PM, Saladi Naidu naidusp2...@yahoo.com wrote: Suppose I have a row of existing data with set of values for attributes I call this State1, and issue an update to some columns with Quorum consistency. If the write is succeeded in one node, Node1 and and failed on remaining nodes. As there is no Rollback, Node1 row attributes will remain new state, State2 and rest of the nodes row will have old state, State1. If I do a Read and Cassandra detects state difference, it will issue a Read repair which will result in new state, State2 being propagated to other nodes. But from a application point of view the update never happened because it received an exception. Yes, that's correct. This is the property I call coalescing to an un-acknowledged-write, which is extremely unusual to find in non-distributed, non-log-structured databases. How to handle this kind of a situation? Design your application in such a way that operations are idempotent and therefore retryable on failure. =Rob
Read Repair
Suppose I have a row of existing data with set of values for attributes I call this State1, and issue an update to some columns with Quorum consistency. If the write is succeeded in one node, Node1 and and failed on remaining nodes. As there is no Rollback, Node1 row attributes will remain new state, State2 and rest of the nodes row will have old state, State1. If I do a Read and Cassandra detects state difference, it will issue a Read repair which will result in new state, State2 being propagated to other nodes. But from a application point of view the update never happened because it received an exception. How to handle this kind of a situation? Naidu Saladi
Re: Is there a way to remove a node with Opscenter?
When you do a nodetool command and you don’t specify a hostname, it sends the requests via JMX to the localhost node. If that node is down then the command will not succeed. In your case you are probably doing the command from a machine which has not cassandra running, in that case you need to specify a node with the switch -h. So for your that would be: nodetool -h a-node-ip-address removenode Host ID where a-node-ip-address is the address of a server which has cassandra daemon running. Cheers Jean On 08 Jul 2015, at 01:39 , Sid Tantia sid.tan...@baseboxsoftware.commailto:sid.tan...@baseboxsoftware.com wrote: I tried both `nodetool remove node Host ID` and `nodetool decommission` and they both give the error: nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused’. Here is what I have tried to fix this: 1) Uncommented JVM_OPTS=”$JVM_OPTS -Djava.rmi.server.hostname=public name” 2) Changed rpc_address to 0.0.0.0 3) Restarted cassandra 4) Restarted datastax-agent (Note that I installed my cluster using opscenter so that may have something to do with it? ) On Tue, Jul 7, 2015 at 2:08 PM, Surbhi Gupta surbhi.gupt...@gmail.commailto:surbhi.gupt...@gmail.com wrote: If node is down use : nodetool removenode Host ID We have to run the below command when the node is down if the cluster does not use vnodes, before running the nodetool removenode command, adjust the tokens. If the node is up, then the command would be “nodetool decommission” to remove the node. Remove the node from the “seed list” within the configuration cassandra.yaml. On 7 July 2015 at 12:56, Sid Tantia sid.tan...@baseboxsoftware.commailto:sid.tan...@baseboxsoftware.com wrote: Thanks for the response. I’m trying to remove a node that’s already down for some reason so its not allowing me to decommission it, is there some other way to do this? On Tue, Jul 7, 2015 at 12:45 PM, Kiran mk coolkiran2...@gmail.commailto:coolkiran2...@gmail.com wrote: Yes, if your intension is to decommission a node. You can do that by clicking on the node and decommission. Best Regards, Kiran.M.K. On Jul 8, 2015 1:00 AM, Sid Tantia sid.tan...@baseboxsoftware.commailto:sid.tan...@baseboxsoftware.com wrote: I know you can use `nodetool removenode` from the command line but is there a way to remove a node from a cluster using OpsCenter?
Re: Example Data Modelling
If going by Month as partition key then you need to duplicate the data. I dont think going with name as partition key is good datamodel practice as it will create a hotspot. Also I believe your queries will be mostly by employee not by month. You can create employee id as partition key and month as clustering and keep employee details as static columns so they wont be repeated Naidu Saladi From: Srinivasa T N seen...@gmail.com To: user@cassandra.apache.org user@cassandra.apache.org Sent: Tuesday, July 7, 2015 3:07 AM Subject: Re: Example Data Modelling Thanks for the inputs. Now my question is how should the app populate the duplicate data, i.e., if I have an employee record (along with his FN, LN,..) for the month of Apr and later I am populating the same record for the month of may (with salary changed), should my application first read/fetch the corresponding data for apr and re-insert with modification for month of may? Regards, Seenu. On Tue, Jul 7, 2015 at 11:32 AM, Peer, Oded oded.p...@rsa.com wrote: The data model suggested isn’t optimal for the “end of month” query you want to run since you are not querying by partition key.The query would look like “select EmpID, FN, LN, basic from salaries where month = 1” which requires filtering and has unpredictable performance. For this type of query to be fast you can use the “month” column as the partition key and the “EmpID” and the clustering column.This approach also has drawbacks:1. This data model creates a wide row. Depending on the number of employees this partition might be very large. You should limit partition sizes to 25MB2. Distributing data according to month means that only a small number of nodes will hold all of the salary data for a specific month which might cause hotspots on those nodes. Choose the approach that works best for you. From: Carlos Alonso [mailto:i...@mrcalonso.com] Sent: Monday, July 06, 2015 7:04 PM To: user@cassandra.apache.org Subject: Re: Example Data Modelling Hi Srinivasa, I think you're right, In Cassandra you should favor denormalisation when in RDBMS you find a relationship like this. I'd suggest a cf like thisCREATE TABLE salaries ( EmpID varchar, FN varchar, LN varchar, Phone varchar, Address varchar, month integer, basic integer, flexible_allowance float, PRIMARY KEY(EmpID, month)) That way the salaries will be partitioned by EmpID and clustered by month, which I guess is the natural sorting you want. Hope it helps,Cheers! Carlos Alonso | Software Engineer | @calonso On 6 July 2015 at 13:01, Srinivasa T N seen...@gmail.com wrote:Hi, I have basic doubt: I have an RDBMS with the following two tables: Emp - EmpID, FN, LN, Phone, Address Sal - Month, Empid, Basic, Flexible Allowance My use case is to print the Salary slip at the end of each month and the slip contains emp name and his other details. Now, if I want to have the same in cassandra, I will have a single cf with emp personal details and his salary details. Is this the right approach? Should we have the employee personal details duplicated each month? Regards, Seenu.
RE: Read Repair
One thing to note is that the exception you get... in this case, you'll get a timeout, not a failure. i.e. as far as Cassandra is concerned, the write is still ongoing - it hasn't failed; but from the client's perspective, it's timed out. In this case (i.e. timeout), the application would usually retry - which would likely get a failure instead of timeout as by then, the coordinator would know of down nodes. In this case, saving with reduced consistency level is usually good enough, as that data will get replicated as nodes rejoin / new nodes are added. Date: Wed, 8 Jul 2015 15:06:46 -0700 Subject: Re: Read Repair From: rc...@eventbrite.com To: user@cassandra.apache.org; naidusp2...@yahoo.com On Wed, Jul 8, 2015 at 2:07 PM, Saladi Naidu naidusp2...@yahoo.com wrote: Suppose I have a row of existing data with set of values for attributes I call this State1, and issue an update to some columns with Quorum consistency. If the write is succeeded in one node, Node1 and and failed on remaining nodes. As there is no Rollback, Node1 row attributes will remain new state, State2 and rest of the nodes row will have old state, State1. If I do a Read and Cassandra detects state difference, it will issue a Read repair which will result in new state, State2 being propagated to other nodes. But from a application point of view the update never happened because it received an exception. Yes, that's correct. This is the property I call coalescing to an un-acknowledged-write, which is extremely unusual to find in non-distributed, non-log-structured databases. How to handle this kind of a situation? Design your application in such a way that operations are idempotent and therefore retryable on failure. =Rob
