Re: AssertionError in hintedhandoff - 1.0.5
Do you mind opening a ticket on https://issues.apache.org/jira/browse/CASSANDRA ? Thanks -- Sylvain On Tue, Dec 6, 2011 at 12:52 AM, Ramesh Natarajan rames...@gmail.com wrote: Hi, We are running a 8 node cassandra cluster running cassandra 1.0.5. All our CF use leveled compaction. We ran a test where we did a lot of inserts for 3 days. After that we started to run tests where some of the reads could ask for information that was inserted a while back. In this scenario we are seeing this assertion error in HintedHandoff. Is this a known issue? ERROR [HintedHandoff:3] 2011-12-05 15:42:04,324 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[HintedHandoff:3,1,main] java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:330) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:326) ... 6 more Caused by: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:158) at org.apache.cassandra.db.compaction.CompactionManager$6.call(CompactionManager.java:275) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more ERROR [HintedHandoff:3] 2011-12-05 15:42:04,333 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[HintedHandoff:3,1,main] java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:330) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:326) ... 6 more Caused by: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at
Re: Re: Re: Cassandra DataModeling recommendations
Thanks to both of you for these very useful and interesting links/proposals.
Best Regards
Hi
Thanks for the answer, as I read the book on Cassandra, I was not aware at
that time on Composite Key which I recently discovered.
Composite Type's are useful for handling data-versions.
You mentioned a TTL and let the database remove the date for me. I never read
about that. Is it possible without an external batch ?
Yes, TTL if set on column, auto delete column for you.
I will try to rephrase in any case my goal:
Storage:
- I would like to store for a user (identified by its id) several carts (BLOB).
- Associated to these carts, I would like to attach metadata like expiration
date and possibly others.
Queries/tasks:
- I would like to be able to retrieve all the carts of a given userId.
I would use timeline with TTL for carts as separate CF. And cart_Id to reverse
index in userId CF with TTL set on columns.
- I would like to have a mean to remove expired carts.
set TTL on each column.
1.
cartCF{
cart1_uuidkey:{
metadata_column:ttl
}
cart2_uuidkey:{
metadata_column:ttl
}
.
.
.cartN_uuidkey:{
metadata_column:ttl
}
}
2.
userIdCF:{
user1:{
id:user1 //hack : to prevent unwanted behavior one column with no ttl.
cart1:cart1_uuidkey:ttl
cart2:ttl
cart3:ttl
}
user2:{
id:user2
cart1:cartX_uuidkey:ttl
cart2:cart4:ttl
cart3:cartMttl
}
}
/Samal
Cassandra not suitable?
Hi, I'm quite desperate about Cassandra's performance in our production cluster. We have 8 real-HW nodes, 32core CPU, 32GB memory, 4 disks in raid10, cassandra 0.8.8, RF=3 and Hadoop. We four keyspaces, one is the large one, it has 2 CFs, one is kind of index, the other holds data. There are about 7milinon rows, mean row size is 7kB. We run several mapreduce tasks, most of them just reads from cassandra and writes to hdfs, but one fetch rows from cassnadra, compute something and write it back, for each row we compute three new json values, about 1kB each (they get overwritten next round). We got lots and lots of Timeout exceptions, LiveSSTablesCount over 100. Reapir doesn't finish even in 24hours, reading from the other keyspaces timeouts as well. We set compaction_throughput_mb_per_sec: 0 but it didn't help. Did we choose wrong DB for our usecase? Regards, Patrik This is from one node: INFO 10:28:40,035 Pool NameActive Pending Blocked INFO 10:28:40,036 ReadStage96 695 0 INFO 10:28:40,037 RequestResponseStage 0 0 0 INFO 10:28:40,037 ReadRepairStage 0 0 0 INFO 10:28:40,037 MutationStage 1 1 0 INFO 10:28:40,038 ReplicateOnWriteStage 0 0 0 INFO 10:28:40,038 GossipStage 0 0 0 INFO 10:28:40,038 AntiEntropyStage 0 0 0 INFO 10:28:40,039 MigrationStage0 0 0 INFO 10:28:40,039 StreamStage 0 0 0 INFO 10:28:40,040 MemtablePostFlusher 0 0 0 INFO 10:28:40,040 FlushWriter 0 0 0 INFO 10:28:40,040 MiscStage 0 0 0 INFO 10:28:40,041 FlushSorter 0 0 0 INFO 10:28:40,041 InternalResponseStage 0 0 0 INFO 10:28:40,041 HintedHandoff 1 5 0 INFO 10:28:40,042 CompactionManager n/a27 INFO 10:28:40,042 MessagingServicen/a 0,16559 And here is the nodetool ring output: 10.2.54.91 NG RAC1Up Normal 118.04 GB 12.50% 0 10.2.54.92 NG RAC1Up Normal 102.74 GB 12.50% 21267647932558653966460912964485513216 10.2.54.93 NG RAC1Up Normal 76.95 GB 12.50% 42535295865117307932921825928971026432 10.2.54.94 NG RAC1Up Normal 56.97 GB 12.50% 63802943797675961899382738893456539648 10.2.54.95 NG RAC1Up Normal 75.55 GB 12.50% 85070591730234615865843651857942052864 10.2.54.96 NG RAC1Up Normal 102.57 GB 12.50% 106338239662793269832304564822427566080 10.2.54.97 NG RAC1Up Normal 68.03 GB 12.50% 127605887595351923798765477786913079296 10.2.54.98 NG RAC1Up Normal 194.6 GB 12.50% 148873535527910577765226390751398592512
Re: CQL Install for 0.8.X?
Thanks Eric! On Mon, Dec 5, 2011 at 10:38 PM, Eric Evans eev...@acunu.com wrote: On Mon, Dec 5, 2011 at 1:40 PM, Joe Stein crypt...@gmail.com wrote: Hey, trying to grab cqlsh for a 0.8.6 cluster but all the online docs I am finding are pointing to http://www.apache.org/dist/cassandra/drivers/pyor say it moved to the source and checked there in the 0.8 branch and nothing either... Right, the drivers were moved and the site wasn't updated (until just now). The Python driver is now hosted on Apache Extras, here: http://code.google.com/a/apache-extras.org/hosting/search?q=label:cql But, that driver probably won't work right with 0.8.6, and the shell has been moved out of the driver and into Cassandra trunk anyway. What you probably want is 1.0.3 from here: http://archive.apache.org/dist/cassandra/drivers/py Sorry, I know, it's a confusing mess. also saw something about about 2.0 not being compatible with 0.8.X so not sure where to go from here. The language incompatibilities are pretty minor, but there were some changes to the results format that will prevent a new driver (for 1.x) from working on an older Cassandra (0.8.x). Let me know, want/need to jump into a bunch of CQL stuff and want to-do it on the cqlsh first if I can. -- Eric Evans Acunu | http://www.acunu.com | @acunu -- /* Joe Stein http://www.linkedin.com/in/charmalloc Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop */
Re: Second Cassandra users survey
It took some time to gather our requirements and to check what are our most important needs. However, here they are: * Column position range queries: We would like to access columns not by their name, but by their position in the row. Example: row(A:v1, B:v2, C:v3, D:v4); ; ordered by UTF8TYPE Example query: Give all elements with position range 1..3 would return (B:v2, C:v3, D:v4) * Arbitrary position range queries: We would like to access arbitrary colums by their position in the row: Example query: Give all elements with positions (0, 3, 1) would return (B:v2, D:v4, A:v1) * Security for client-server communication (thrift): A big benefit for all users of cassandra which deploy the cluster into untrusted environments (Amazon EC2 etc.) would be the possibility to secure the client-server communication with SSL. This has already been implemented in Thrift (see https://issues.apache.org/jira/browse/THRIFT-106) and must probably be added to CassandraDaemon.ThriftServer. Kind regards Arne and Matthias On 11/01/2011 11:59 PM, Jonathan Ellis wrote: Hi all, Two years ago I asked for Cassandra use cases and feature requests. [1] The results [2] have been extremely useful in setting and prioritizing goals for Cassandra development. But with the release of 1.0 we've accomplished basically everything from our original wish list. [3] I'd love to hear from modern Cassandra users again, especially if you're usually a quiet lurker. What does Cassandra do well? What are your pain points? What's your feature wish list? As before, if you're in stealth mode or don't want to say anything in public, feel free to reply to me privately and I will keep it off the record. [1] http://www.mail-archive.com/cassandra-dev@incubator.apache.org/msg01148.html [2] http://www.mail-archive.com/cassandra-user@incubator.apache.org/msg01446.html [3] http://www.mail-archive.com/dev@cassandra.apache.org/msg01524.html
Re: best practices for simulating transactions in Cassandra
Hi John, I had exactly the same reflexions. I'm using zookeeper and cage to lock et isolate. but how to rollback? It's impossible so try replay! the idea is explained in this presentation http://www.slideshare.net/mattdennis/cassandra-data-modeling (starting from slide 24) - insert your whole data into one column - make the job - remove (or expire) your column. if there is a problem during making the job, you keep the possibility to replay and replay and replay (synchronously or in a batch). Regards Jérémy 2011/12/5 John Laban j...@pagerduty.com Hello, I'm building a system using Cassandra as a datastore and I have a few places where I am need of transactions. I'm using ZooKeeper to provide locking when I'm in need of some concurrency control or isolation, so that solves that half of the puzzle. What I need now is to sometimes be able to get atomicity across multiple writes by simulating the begin/rollback/commit abilities of a relational DB. In other words, there are places where I need to perform multiple updates/inserts, and if I fail partway through, I would ideally be able to rollback the partially-applied updates. Now, I *know* this isn't possible with Cassandra. What I'm looking for are all the best practices, or at least tips and tricks, so that I can get around this limitation in Cassandra and still maintain a consistent datastore. (I am using quorum reads/writes so that eventual consistency doesn't kick my ass here as well.) Below are some ideas I've been able to dig up. Please let me know if any of them don't make sense, or if there are better approaches: 1) Updates to a row in a column family are atomic. So try to model your data so that you would only ever need to update a single row in a single CF at once. Essentially, you model your data around transactions. This is tricky but can certainly be done in some situations. 2) If you are only dealing with multiple row *inserts* (and not updates), have one of the rows act as a 'commit' by essentially validating the presence of the other rows. For example, say you were performing an operation where you wanted to create an Account row and 5 User rows all at once (this is an unlikely example, but bear with me). You could insert 5 rows into the Users CF, and then the 1 row into the Accounts CF, which acts as the commit. If something went wrong before the Account could be created, any Users that had been created so far would be orphaned and unusable, as your business logic can ensure that they can't exist without an Account. You could also have an offline cleanup process that swept away orphans. 3) Try to model your updates as idempotent column inserts instead. How do you model updates as inserts? Instead of munging the value directly, you could insert a column containing the operation you want to perform (like +5). It would work kind of like the Consistent Vote Counting implementation: ( https://gist.github.com/41 ). How do you make the inserts idempotent? Make sure the column names correspond to a request ID or some other identifier that would be identical across re-drives of a given (perhaps originally failed) request. This could leave your datastore in a temporarily inconsistent state, but would eventually become consistent after a successful re-drive of the original request. 4) You could take an approach like Dominic Williams proposed with Cages: http://ria101.wordpress.com/2010/05/12/locking-and-transactions-over-cassandra-using-cages/ The gist is that you snapshot all the original values that you're about to munge somewhere else (in his case, ZooKeeper), make your updates, and then delete the snapshot (and that delete needs to be atomic). If the snapshot data was never deleted, then subsequent accessors (even readers) of the data rows need to do the rollback of the previous transaction themselves before they can read/write this data. They do the rollback by just overwriting the current values with what is in the snapshot. It offloads the work of the rollback to the next worker that accesses the data. This approach probably needs an generic/high-level programming layer to handle all of the details and complexity, and it doesn't seem like it was ever added to Cages. Are there other approaches or best practices that I missed? I would be very interested in hearing any opinions from those who have tackled these problems before. Thanks! John -- Jérémy
Re: AssertionError in hintedhandoff - 1.0.5
https://issues.apache.org/jira/browse/CASSANDRA-3579 thanks Ramesh On Tue, Dec 6, 2011 at 2:16 AM, Sylvain Lebresne sylv...@datastax.com wrote: Do you mind opening a ticket on https://issues.apache.org/jira/browse/CASSANDRA ? Thanks -- Sylvain On Tue, Dec 6, 2011 at 12:52 AM, Ramesh Natarajan rames...@gmail.com wrote: Hi, We are running a 8 node cassandra cluster running cassandra 1.0.5. All our CF use leveled compaction. We ran a test where we did a lot of inserts for 3 days. After that we started to run tests where some of the reads could ask for information that was inserted a while back. In this scenario we are seeing this assertion error in HintedHandoff. Is this a known issue? ERROR [HintedHandoff:3] 2011-12-05 15:42:04,324 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[HintedHandoff:3,1,main] java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:330) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:326) ... 6 more Caused by: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:160) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:158) at org.apache.cassandra.db.compaction.CompactionManager$6.call(CompactionManager.java:275) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) ... 3 more ERROR [HintedHandoff:3] 2011-12-05 15:42:04,333 AbstractCassandraDaemon.java (line 133) Fatal exception in thread Thread[HintedHandoff:3,1,main] java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:330) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) ... 3 more Caused by: java.util.concurrent.ExecutionException: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:326) ... 6 more Caused by: java.lang.AssertionError: originally calculated column size of 470937164 but now it is 470294247 at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:124) at
Re: OutOfMemory Exception during bootstrap
You are right Samal, configuring cassandra-env.sh a little bit and let it calculate heap size solved the problem! thank you! On Mon, Dec 5, 2011 at 11:23, Harald Falzberger h.falzber...@gmail.com wrote: I don't think that this is the problem because I'm testing on a supercomputer with 16T RAM. Is it possible, that cassandra runs into this bug of the jvm? http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6399443 On Mon, Dec 5, 2011 at 06:29, samal samalgo...@gmail.com wrote: Lower your heap size, if you are testing multiple instance with single node. https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L64 On Sun, Dec 4, 2011 at 11:08 PM, Harald Falzberger h.falzber...@gmail.com wrote: Hi, I'm trying to set up a test environment with 2 nodes on one physical machine with two ips. I configured both as adviced in the documentation: cluster_name: 'MyDemoCluster' initial_token: 0 seed_provider: - seeds: IP1 listen_address: IP1 rpc_address: IP1 cluster_name: 'MyDemoCluster' initial_token: 85070591730234615865843651857942052864 seed_provider: - seeds: IP1 listen_address: IP2 rpc_address: IP2 Node1 uses 7199 as JMX port, Node2 7198 because JMX by default is listening on all interfaces. When I bootstrap node2, on node1 following exception is thrown and node1 terminates. the same error occurs again if I try to restart node1 and node2 is still running. Does anyone of you have an idea why this happens? I'm starting each cassandra instance with 16GB RAM and my database is empty. Exception on Node1 java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:597) at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:703) at java.util.concurrent.ThreadPoolExecutor.prestartAllCoreThreads(ThreadPoolExecutor.java:1384) at org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.init(JMXEnabledThreadPoolExecutor.java:77) at org.apache.cassandra.concurrent.JMXEnabledThreadPoolExecutor.init(JMXEnabledThreadPoolExecutor.java:65) at org.apache.cassandra.concurrent.StageManager.multiThreadedStage(StageManager.java:58) at org.apache.cassandra.concurrent.StageManager.clinit(StageManager.java:44) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:512) at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:159)
Re: best practices for simulating transactions in Cassandra
Ah, neat. It is similar to what was proposed in (4) above with adding transactions to Cages, but instead of snapshotting the data to be rolled back (the before data), you snapshot the data to be replayed (the after data). And then later, if you find that the transaction didn't complete, you just keep replaying the transaction until it takes. The part I don't understand with this approach though: how do you ensure that someone else didn't change the data between your initial failed transaction and the later replaying of the transaction? You could get lost writes in that situation. Dominic (in the Cages blog post) explained a workaround with that for his rollback proposal: all subsequent readers or writers of that data would have to check for abandoned transactions and roll them back themselves before they could read the data. I don't think this is possible with the XACT_LOG replay approach in these slides though, based on how the data is indexed (cassandra node token + timeUUID). PS: How are you liking Cages? 2011/12/6 Jérémy SEVELLEC jsevel...@gmail.com Hi John, I had exactly the same reflexions. I'm using zookeeper and cage to lock et isolate. but how to rollback? It's impossible so try replay! the idea is explained in this presentation http://www.slideshare.net/mattdennis/cassandra-data-modeling (starting from slide 24) - insert your whole data into one column - make the job - remove (or expire) your column. if there is a problem during making the job, you keep the possibility to replay and replay and replay (synchronously or in a batch). Regards Jérémy 2011/12/5 John Laban j...@pagerduty.com Hello, I'm building a system using Cassandra as a datastore and I have a few places where I am need of transactions. I'm using ZooKeeper to provide locking when I'm in need of some concurrency control or isolation, so that solves that half of the puzzle. What I need now is to sometimes be able to get atomicity across multiple writes by simulating the begin/rollback/commit abilities of a relational DB. In other words, there are places where I need to perform multiple updates/inserts, and if I fail partway through, I would ideally be able to rollback the partially-applied updates. Now, I *know* this isn't possible with Cassandra. What I'm looking for are all the best practices, or at least tips and tricks, so that I can get around this limitation in Cassandra and still maintain a consistent datastore. (I am using quorum reads/writes so that eventual consistency doesn't kick my ass here as well.) Below are some ideas I've been able to dig up. Please let me know if any of them don't make sense, or if there are better approaches: 1) Updates to a row in a column family are atomic. So try to model your data so that you would only ever need to update a single row in a single CF at once. Essentially, you model your data around transactions. This is tricky but can certainly be done in some situations. 2) If you are only dealing with multiple row *inserts* (and not updates), have one of the rows act as a 'commit' by essentially validating the presence of the other rows. For example, say you were performing an operation where you wanted to create an Account row and 5 User rows all at once (this is an unlikely example, but bear with me). You could insert 5 rows into the Users CF, and then the 1 row into the Accounts CF, which acts as the commit. If something went wrong before the Account could be created, any Users that had been created so far would be orphaned and unusable, as your business logic can ensure that they can't exist without an Account. You could also have an offline cleanup process that swept away orphans. 3) Try to model your updates as idempotent column inserts instead. How do you model updates as inserts? Instead of munging the value directly, you could insert a column containing the operation you want to perform (like +5). It would work kind of like the Consistent Vote Counting implementation: ( https://gist.github.com/41 ). How do you make the inserts idempotent? Make sure the column names correspond to a request ID or some other identifier that would be identical across re-drives of a given (perhaps originally failed) request. This could leave your datastore in a temporarily inconsistent state, but would eventually become consistent after a successful re-drive of the original request. 4) You could take an approach like Dominic Williams proposed with Cages: http://ria101.wordpress.com/2010/05/12/locking-and-transactions-over-cassandra-using-cages/ The gist is that you snapshot all the original values that you're about to munge somewhere else (in his case, ZooKeeper), make your updates, and then delete the snapshot (and that delete needs to be atomic). If the snapshot data was never deleted, then subsequent accessors (even readers) of the data rows need to do the rollback of the
cqlsh not returning the column name of the first column when reversed
I am running Cassandra 1.0.0. I am using cqlsh for inspecting my data
(very useful tool, thank you whoever wrote it). I notice that when I
query for the FIRST N REVERSED column, it is omitting the column name
on the first column. For example,
cqlsh SELECT FIRST 1 REVERSED * FROM netflow_raw;
'{body:{src_as:0,srcport:0,dstport:148,nexthop:0.0.0.0,First:607502600,prot:2,src_mask:24,tos:-64,input:9721,Last:607502600,dOctets:40,tcp_flags:16,dst_as:0,dstaddr:224.0.0.22,dPkts:1,output:0,dst_mask:0,srcaddr:172.30.3.5},header:{Sequence
Number:75569,RouterIP:172.29.1.3,UNIX
Seconds:1323201095,Source ID:0,System
Uptime:607525604},version:9}' | 'KEY','172.29.1.3,10311323198000'
cqlsh SELECT FIRST 2 REVERSED * FROM netflow_raw;
Notice that the KEY is given but not the column name of the single column.
When I ask for more than one column, only one of them has the column
name omitted:
cqlsh SELECT FIRST 2 REVERSED * FROM netflow_raw;
'{body:{src_as:0,srcport:68,dstport:67,nexthop:0.0.0.0,First:607497540,prot:17,src_mask:24,tos:16,input:9721,Last:607554532,dOctets:1968,tcp_flags:16,dst_as:0,dstaddr:255.255.255.255,dPkts:6,output:0,dst_mask:0,srcaddr:172.30.3.5},header:{Sequence
Number:75571,RouterIP:172.29.1.3,UNIX
Seconds:1323201132,Source ID:0,System
Uptime:607562604},version:9}' |
'1323201132,0.0.0.0,11','{body:{src_as:0,srcport:0,dstport:148,nexthop:0.0.0.0,First:607540516,prot:2,src_mask:0,tos:-64,input:2,Last:607540516,dOctets:32,tcp_flags:16,dst_as:0,dstaddr:224.0.0.1,dPkts:1,output:0,dst_mask:0,srcaddr:0.0.0.0},header:{Sequence
Number:75571,RouterIP:172.29.1.3,UNIX
Seconds:1323201132,Source ID:0,System
Uptime:607562604},version:9}' | 'KEY','172.29.1.3,10311323198000'
This is not an issue when we do not do reversed queries:
cqlsh SELECT FIRST 1 * FROM netflow_raw;
'172.29.1.3,10311323198000' |
'1323201033,172.30.3.2,02','{body:{src_as:0,srcport:0,dstport:148,nexthop:0.0.0.0,First:607445536,prot:2,src_mask:24,tos:-64,input:9721,Last:607445536,dOctets:32,tcp_flags:16,dst_as:0,dstaddr:224.0.0.251,dPkts:1,output:0,dst_mask:0,srcaddr:172.30.3.2},header:{Sequence
Number:75564,RouterIP:172.29.1.3,UNIX
Seconds:1323201033,Source ID:0,System
Uptime:607463584},version:9}'
cqlsh SELECT FIRST 2 * FROM netflow_raw;
'172.29.1.3,10311323198000' |
'1323201033,172.30.3.2,02','{body:{src_as:0,srcport:0,dstport:148,nexthop:0.0.0.0,First:607445536,prot:2,src_mask:24,tos:-64,input:9721,Last:607445536,dOctets:32,tcp_flags:16,dst_as:0,dstaddr:224.0.0.251,dPkts:1,output:0,dst_mask:0,srcaddr:172.30.3.2},header:{Sequence
Number:75564,RouterIP:172.29.1.3,UNIX
Seconds:1323201033,Source ID:0,System
Uptime:607463584},version:9}' |
'1323201033,172.30.3.3,01','{body:{src_as:0,srcport:0,dstport:148,nexthop:0.0.0.0,First:607445000,prot:2,src_mask:24,tos:-64,input:9721,Last:607445000,dOctets:32,tcp_flags:16,dst_as:0,dstaddr:224.0.0.251,dPkts:1,output:0,dst_mask:0,srcaddr:172.30.3.3},header:{Sequence
Number:75564,RouterIP:172.29.1.3,UNIX
Seconds:1323201033,Source ID:0,System
Uptime:607463584},version:9}'
I am not using cql for anything other than debugging purposes so I do
not care when this bug is fixed, I just wanted to make people aware of
it.
Is this a limitation of CQL?
I was doing some testing with CQL and noticed something. I'm guessing it's probably user error on my part, but just in case it isn't. If I execute a query like this, I get the following error Caused by: InvalidRequestException(why:No indexed columns present in by-columns clause with equals operator) select * from user_columnfamily where effectiveTime = 1323203100182 The relevant parts of the exception thrown: me.prettyprint.hector.api.exceptions.HInvalidRequestException: InvalidRequestException(why:No indexed columns present in by-columns clause with equals operator) at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:50) at me.prettyprint.cassandra.model.CqlQuery$1.execute(CqlQuery.java:127) at me.prettyprint.cassandra.model.CqlQuery$1.execute(CqlQuery.java:94) at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101) at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:232) at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97) at me.prettyprint.cassandra.model.CqlQuery.execute(CqlQuery.java:93) If I change the query and add another filter with equal operator, it works fine. select * from user_columnfamily where effectiveTime = 1323203100182 and firstName = 'bob' I'm assuming this is as designed in 0.8.x release. thanks peter
Re: Is this a limitation of CQL?
thanks for confirming that. I was guessing that was the rationale and it makes sense. Didn't want to make a stupid assumption. On Tue, Dec 6, 2011 at 4:14 PM, Jeremiah Jordan jeremiah.jor...@morningstar.com wrote: As designed in all versions. The search queries have to have an indexed column which is keyed by =. C* is still a key/value store, so you have to have a key to get anywhere... Once the = is used to pull in rows, the other parts of the query are then used to filter the results before they are returned. On 12/06/2011 02:54 PM, Peter Lin wrote: I was doing some testing with CQL and noticed something. I'm guessing it's probably user error on my part, but just in case it isn't. If I execute a query like this, I get the following error Caused by: InvalidRequestException(why:No indexed columns present in by-columns clause with equals operator) select * from user_columnfamily where effectiveTime= 1323203100182 The relevant parts of the exception thrown: me.prettyprint.hector.api.exceptions.HInvalidRequestException: InvalidRequestException(why:No indexed columns present in by-columns clause with equals operator) at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:50) at me.prettyprint.cassandra.model.CqlQuery$1.execute(CqlQuery.java:127) at me.prettyprint.cassandra.model.CqlQuery$1.execute(CqlQuery.java:94) at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101) at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:232) at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97) at me.prettyprint.cassandra.model.CqlQuery.execute(CqlQuery.java:93) If I change the query and add another filter with equal operator, it works fine. select * from user_columnfamily where effectiveTime= 1323203100182 and firstName = 'bob' I'm assuming this is as designed in 0.8.x release. thanks peter
cassandra most stable version ?
Hello, Recent problems with Cassandra 1.0.x versions seems to tell it is still not ready for prime time. We are currently using version 0.8.5 on our development cluster - although we have not seen much problems with this one, maybe recent versions of 0.8.x might be safer to use. So what version are you running in production ? What kinds of problems do you encounter if any ? Thanks, - Pierre
Re: cassandra most stable version ?
We are running 0.8.7. No big issues so far. Thanks, Jahangir. On Tue, Dec 6, 2011 at 5:05 PM, Pierre Chalamet pie...@chalamet.net wrote: Hello, ** ** Recent problems with Cassandra 1.0.x versions seems to tell it is still not ready for prime time. ** ** We are currently using version 0.8.5 on our development cluster – although we have not seen much problems with this one, maybe recent versions of 0.8.x might be safer to use. ** ** So what version are you running in production ? What kinds of problems do you encounter if any ? ** ** Thanks, - Pierre
Re: Upgrade from 0.6 to 1.0
Found my problem: I was using a very old cassandra (0.6.1). So I upgraded to 0.6.13 first then I could start 1.0.5. And so by trial, I can then answer my other questions: - no, config-convert is not necessary and I can just edit the default cassandra.yaml - no, I don't need schematool, can just re-create the keyspace with cassandra-cli and cassandra will reload my old data. Jehan On 2011-12-06 12:12, Jehan Bing wrote: Hi, I've seen recent posts saying it was possible to upgrade directly from 0.6 to 1.0. But how? I ran nodetool drain on all my nodes and shut them down. However, there is not config-convert tool anymore. Since I basically using the default config, is it important? Or is it OK to just use the default one and change the few settings I need? Also, there is no schematool anymore either. So how do I load the schema? Can I just create one using cassandra-cli? Will cassandra then load the existing data? Last, I tried to start cassandra 1.0.5, I get the following error in cassandra.log: ERROR 11:41:19,399 Exception encountered during startup java.lang.AssertionError at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:295) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:150) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:337) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107) java.lang.AssertionError at org.apache.cassandra.db.SystemTable.checkHealth(SystemTable.java:295) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:150) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:337) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:107) Exception encountered during startup: null Then cassandra exits so I can't run nodetool repair (or create the schema if that's the problem). So how should I proceed? Or maybe I misread the previous post and I should actually do 0.6-0.7-1.0? Thanks, Jehan
Re: building a new email-like inbox service with cassandra
Hi, Just updating this thread: We've pushed initial version to github today. You can find sources, binary package and some information here: https://github.com/elasticinbox/elasticinbox/wiki Your feedback is most welcome. We can discuss it further on elasticin...@googlegroups.com mail list. Regards, Rustam. On 18/11/2011 13:08, Dotan N. wrote: Thanks!! -- Dotan, @jondot http://twitter.com/jondot On Fri, Nov 18, 2011 at 2:48 PM, Rustam Aliyev rus...@code.az mailto:rus...@code.az wrote: It's pleasing to see interest out there. We'll try to do some cleanups and push it to github this weekend. You can follow us on twitter: @elasticinbox Regards, Rustam. On Fri Nov 18 01:42:01 2011 tel:2011, Andrey V. Panov wrote: I'm also interesting in your project and will be glad to follow you on twitter if I can. On 18 November 2011 tel:2011 00:37, Rustam Aliyev rus...@code.az mailto:rus...@code.az mailto:rus...@code.az mailto:rus...@code.az wrote: Hi Dotan, We have already built something similar and were planning to open source it. It will be available under http://www.elasticinbox.com/. We haven't followed exactly IBM's paper, we believe our Cassandra model design is more robust. It's written in Java and provides LMTP and REST interfaces. ElasticInbox also stores original messages outside of the Cassandra, in the blob store. Let me know if you are interested, I will need some time to do cleanup. Regards, Rustam. On 17/11/2011 14:17, Dotan N. wrote: Hi all, New to cassandra, i'm about to embrak on building a scalable user inbox service on top of cassandra. I've done the preliminary googling and got some more info on bluerunner (IBM's project on the subject), and now looking for more information in this specific topic. If anyone can point me to researches/articles that would nudge me in the right direction i'd be tremendously thankful! Thanks! -- Dotan, @jondot http://twitter.com/jondot
sstable count=0, why nodetool ring is not 0
hi,all We are using Cassandra 1.0.2. I am testing the TTL with loading 400G. When all the data are expired, I waited for some hours. Later, the nodetool ring is still have 90GB. So I made a major compaction. Then there are 30GB from the nodetool ring. After I saw the file system,I found there are zero sstable. I don't know where comes the 30GB? Best Regards
Re: Cassandra not suitable?
I'm quite desperate about Cassandra's performance in our production cluster. We have 8 real-HW nodes, 32core CPU, 32GB memory, 4 disks in raid10, cassandra 0.8.8, RF=3 and Hadoop. We four keyspaces, one is the large one, it has 2 CFs, one is kind of index, the other holds data. There are about 7milinon rows, mean row size is 7kB. We run several mapreduce tasks, most of them just reads from cassandra and writes to hdfs, but one fetch rows from cassnadra, compute something and write it back, for each row we compute three new json values, about 1kB each (they get overwritten next round). We got lots and lots of Timeout exceptions, LiveSSTablesCount over 100. Reapir doesn't finish even in 24hours, reading from the other keyspaces timeouts as well. We set compaction_throughput_mb_per_sec: 0 but it didn't help. Exactly why you're seeing timeouts would depend on quite a few factors. In general however, my observation is that you have ~ 100 gig per node on nodes with 32 gigs of memory, *AND* you say you're running map reduce jobs. In general, I would expect that any performance problems you have are probably due to cache misses and simply bottlenecking on disk I/O. What to do about it depends very much on the situation and it's difficult to give a concrete suggestion without more context. Some things that might mitigate effects include using row cache for the hot data set (if you have a very small hot data set that should work well since the row cache is unaffected by e.g. mapreduce jobs), selecting a different compaction strategy (leveled can be better, depending), running map reduce on a separate DC that takes writes but is separated from the live cluster that takes reads (unless you're only doing batch request). But those are just some random things thrown in the air; do not take that as concrete suggestions for your particular case. The key is understanding the access pattern, what the bottlenecks are, in combination with how the database works - and figure out what the most cost-effective solution is. Note that if you're bottlenecking on disk I/O, it's not surprising at all that repairing ~ 100 gigs of data takes more than 24 hours. -- / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
