Re: CQL 3.0 composite keys and secondary indexes
On Tue, May 8, 2012 at 12:08 AM, Roland Mechler rmech...@sencha.com wrote: It seems as though secondary indexes are not supported in tables (column families) that have composite keys. Is that true? It is. If so, are there plans to suport that combination in the future? There is: https://issues.apache.org/jira/browse/CASSANDRA-3680 -- Sylvain
Re: cassandra1.1 can't start
8G, by default the jvm was taking 2G, and I had this error even with 1G, I had the error, finally 500M made it work (4 Intel Atoms, OS: ubuntu 10.04 ) 2012/5/8 Watanabe Maki watanabe.m...@gmail.com How much memory do you have on the box? It seems you need more memory. maki On 2012/05/08, at 1:29, cyril auburtin cyril.aubur...@gmail.com wrote: well I uncommented lines9697 in cassandra-env.sh, with lower values MAX_HEAP_SIZE=500M HEAP_NEWSIZE=100M seems to fix, it 2012/5/7 cyril auburtin cyril.aubur...@gmail.com The xassandra lauch command worked the first time then now I keep getting INFO 18:18:39,354 Starting up server gossip ERROR 18:18:39,357 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Map failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogAllocator$3.run(CommitLogAllocator.java:191) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:803) at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:119) ... 4 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:800) ... 5 more ERROR 18:18:39,361 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.gms.Gossiper.stop(Gossiper.java:1113) at org.apache.cassandra.service.StorageService$2.runMayThrow(StorageService.java:478) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:636) I have tried rebooting, starting it alone, clearing all /var/lib/cassandra dir but keep getting this error any idea?
Re: strange gossip messages after node reboot with different ip
On 05/01/2012 04:16 AM, aaron morton wrote: Gossip information about a node can stay in the cluster for up to 3 days. How long has this been going on for ? This has been going for over a week already without any signs of slow down, all nodes that have changed ip popup as UP/DEAD endlessly. Any ideas? Thanks I'm unsure if this is expected behaviour. But it sounds like Gossip is kicking out the phantom node correctly. Can you use nodetool gossipinfo on the nodes to capture some artefacts while it is still running? How come the old ip 10.63.14.214 still popup as UP and then declared as DEAD again, an so on and on? I think this is gossip bouncing information about the node around. Once it has been observed as dead for 3 days it should be purged. Another question, if node is recognised as new (due to ip change) but with same token - will other nodes stream the hinted handoffs to it? Hints are stored against the token, not the end point address. When a node comes up the process is reversed and the end point is mapped to it's (new) token. And is there way to tell cassandra also use names and if ip changes but node name is the same and resolves to the new ip then the cluster treat it as old node? Not that I am aware of. It's designed to handle IP addresses changing. AFAIK the log messages are not indicative of a fault. Instead they indicate something odd happening with Gossip that is being correctly handled. Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 1/05/2012, at 3:09 AM, Piavlo wrote: Hi, We have a cassandra cluster in ec2. If i stop a node and start it - as a result the node ip changes. The node is recognised as NEW node and is declared as replacing the previous node with same token.(But this is the same node of course) In this specific case the node ip before stop/start was 10.63.14.214 and new ip is 10.54.81.14. And even that the cluster and node seems to be working fine for more than a day after the stop/start of this node, I see the following loop of messages ~ once every minute. INFO [GossipStage:1] 2012-04-30 14:18:57,089 Gossiper.java (line 838) Node /10.63.14.214 is now part of the cluster INFO [GossipStage:1] 2012-04-30 14:18:57,089 Gossiper.java (line 804) InetAddress /10.63.14.214 is now UP INFO [GossipStage:1] 2012-04-30 14:18:57,090 StorageService.java (line 1017) Nodes /10.63.14.214 and cassa1a.internal/10.54.81.14 have the same token 0. Ignoring /10.63.14.214 INFO [GossipTasks:1] 2012-04-30 14:19:11,834 Gossiper.java (line 818) InetAddress /10.63.14.214 is now dead. INFO [GossipTasks:1] 2012-04-30 14:19:27,896 Gossiper.java (line 632) FatClient /10.63.14.214 has been silent for 3ms, removing from gossip INFO [GossipStage:1] 2012-04-30 14:20:30,803 Gossiper.java (line 838) Node /10.63.14.214 is now part of the cluster ... How come the old ip 10.63.14.214 still popup as UP and then declared as DEAD again, an so on and on? I know since this is ec2 other node with same ip can come UP, but i've verified and there is no such node and it certainly does not run cassandra :) I stop/started another node and observe similar behaviour. This is version 1.0.8 Another question, if node is recognised as new (due to ip change) but with same token - will other nodes stream the hinted handoffs to it? And is there way to tell cassandra also use names and if ip changes but node name is the same and resolves to the new ip then the cluster treat it as old node? Thanks Alex
RE: cassandra1.1 can't start
This looks like CASSANDRA-4201 where map() was failing with oom under 32 bits jvm. Jonathan provided a patch for that. You can apply it on top of 1.1. - Pierre From: cyril auburtin [mailto:cyril.aubur...@gmail.com] Sent: mardi 8 mai 2012 08:56 To: user@cassandra.apache.org Subject: Re: cassandra1.1 can't start 8G, by default the jvm was taking 2G, and I had this error even with 1G, I had the error, finally 500M made it work (4 Intel Atoms, OS: ubuntu 10.04 ) 2012/5/8 Watanabe Maki watanabe.m...@gmail.com How much memory do you have on the box? It seems you need more memory. maki On 2012/05/08, at 1:29, cyril auburtin cyril.aubur...@gmail.com wrote: well I uncommented lines9697 in cassandra-env.sh, with lower values MAX_HEAP_SIZE=500M HEAP_NEWSIZE=100M seems to fix, it 2012/5/7 cyril auburtin cyril.aubur...@gmail.com The xassandra lauch command worked the first time then now I keep getting INFO 18:18:39,354 Starting up server gossip ERROR 18:18:39,357 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Map failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.j ava:127) at org.apache.cassandra.db.commitlog.CommitLogAllocator$3.run(CommitLogAllocato r.java:191) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLog Allocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:803) at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.j ava:119) ... 4 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:800) ... 5 more ERROR 18:18:39,361 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.gms.Gossiper.stop(Gossiper.java:1113) at org.apache.cassandra.service.StorageService$2.runMayThrow(StorageService.jav a:478) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:636) I have tried rebooting, starting it alone, clearing all /var/lib/cassandra dir but keep getting this error any idea?
Re: cassandra1.1 can't start
ah thx yes that's true I forgot to say we are using 32bits Java, we should use 64 now that everything is stable with 64bit thx 2012/5/8 Pierre Chalamet pie...@chalamet.net This looks like CASSANDRA-4201 where map() was failing with oom under 32 bits jvm. Jonathan provided a patch for that. You can apply it on top of 1.1. ** ** - Pierre ** ** *From:* cyril auburtin [mailto:cyril.aubur...@gmail.com] *Sent:* mardi 8 mai 2012 08:56 *To:* user@cassandra.apache.org *Subject:* Re: cassandra1.1 can't start ** ** 8G, by default the jvm was taking 2G, and I had this error even with 1G, I had the error, finally 500M made it work (4 Intel Atoms, OS: ubuntu 10.04 ) 2012/5/8 Watanabe Maki watanabe.m...@gmail.com How much memory do you have on the box? It seems you need more memory. ** ** maki ** ** On 2012/05/08, at 1:29, cyril auburtin cyril.aubur...@gmail.com wrote:** ** well I uncommented lines9697 in cassandra-env.sh, with lower values ** ** MAX_HEAP_SIZE=500M HEAP_NEWSIZE=100M ** ** seems to fix, it ** ** 2012/5/7 cyril auburtin cyril.aubur...@gmail.com The xassandra lauch command worked the first time ** ** then now I keep getting ** ** INFO 18:18:39,354 Starting up server gossip ERROR 18:18:39,357 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException: Map failed at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:127) at org.apache.cassandra.db.commitlog.CommitLogAllocator$3.run(CommitLogAllocator.java:191) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:636) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:803) at org.apache.cassandra.db.commitlog.CommitLogSegment.init(CommitLogSegment.java:119) ... 4 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:800) ... 5 more ERROR 18:18:39,361 Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException at org.apache.cassandra.gms.Gossiper.stop(Gossiper.java:1113) at org.apache.cassandra.service.StorageService$2.runMayThrow(StorageService.java:478) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:636) ** ** I have tried rebooting, starting it alone, clearing all /var/lib/cassandra dir but keep getting this error ** ** any idea? ** ** ** **
[RELEASE] Apache Cassandra 1.0.10 released
The Cassandra team is pleased to announce the release of Apache Cassandra version 1.0.10. Cassandra is a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. You can read more here: http://cassandra.apache.org/ Downloads of source and binary distributions are listed in our download section: http://cassandra.apache.org/download/ This version is maintenance/bug fix release[1]. As always, please pay attention to the release notes[2] and Let us know[3] if you were to encounter any problem. Have fun! [1]: http://goo.gl/u8gIO (CHANGES.txt) [2]: http://goo.gl/mAHbY (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA
Re: [RELEASE] Apache Cassandra 1.0.10 released
Hi, Can someone give some more details about the CASSANDRA-4116 bug fixed in this release? Could this cause resurrection of deleted data for example? https://issues.apache.org/jira/browse/CASSANDRA-4116 / Jonas On 2012-05-08 11:04 , Sylvain Lebresne wrote: The Cassandra team is pleased to announce the release of Apache Cassandra version 1.0.10. Cassandra is a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. You can read more here: http://cassandra.apache.org/ Downloads of source and binary distributions are listed in our download section: http://cassandra.apache.org/download/ This version is maintenance/bug fix release[1]. As always, please pay attention to the release notes[2] and Let us know[3] if you were to encounter any problem. Have fun! [1]: http://goo.gl/u8gIO (CHANGES.txt) [2]: http://goo.gl/mAHbY (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA signature.asc Description: OpenPGP digital signature
Re: count after truncate NOT zero
Some 'feature' for future implementation, maybe? imho truncation working as a meta data operation is the correct approach. It's generally used in testing and development. It deletes the data and removes the SSTables, giving you a clean state. A CF level tombstone would mean that reads had to examine every SSTable to see if they had columns with a higher timestamp. And the data would not be removed intil gc_grace_seconds had passed. It seems odd if you expect the same behaviour of delete from usertable (in SQL, not yet in CQL, I presume), especially because the truncate is synced over all nodes before it returns to client, so a truncate may rightfully discard its handoffs, right? It is analogous to http://en.wikipedia.org/wiki/Truncate_(SQL) A CF level tomstone would be analogous to delete * from foo I wonder though that you don't know YCSB, what do you use to do performance testing? wrote your own or use another tool? In the latter I would like to know what you use :) If I want to stress test a normal sized cluster I use the java stress tool included in the source distribution. Previously where I had to benchmark a system for a proof of concept and capacity planning. I cobbled together a system with python, redis and flat files to replay real life requests using multiple clients. That allowed us to adjust the read / write mix and the adjust the rate. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 7/05/2012, at 11:51 PM, Peter Dijkshoorn wrote: Check, I understand. Thanks! The cluster certainly was overloaded and I did not realize that truncate does not tombstone or have a timestamp. Some 'feature' for future implementation, maybe? It seems odd if you expect the same behaviour of delete from usertable (in SQL, not yet in CQL, I presume), especially because the truncate is synced over all nodes before it returns to client, so a truncate may rightfully discard its handoffs, right? btw, it was very hard to replicate this behaviour, seems to be a rare occurrence... I wonder though that you don't know YCSB, what do you use to do performance testing? wrote your own or use another tool? In the latter I would like to know what you use :) Ciao Peter Dijkshoorn Adyen - Payments Made Easy www.adyen.com Visiting address: Mail Address: Simon Carmiggeltstraat 6-50 P.O. Box 10095 1011 DJ Amsterdam 1001 EB Amsterdam The Netherlands The Netherlands Office +31.20.240.1240 Email peter.dijksho...@adyen.com On 05/07/2012 12:59 PM, aaron morton wrote: I don't know the YCSB code, but one theory would be… 1) The cluster is overloaded by the test. 2) A write at CL ALL fails because a node does not respond in time. 3) The coordinator stores the hint and returns failure to the client. 4) The client gets an UnavailableException and retries the operation. Did the nodes show any dropped messages ? Either in nodetool tpstats or in the logs? Truncate is meta data operation, unlike deleting columns or rows. When a column is deleted a Tombstone column is written, when row is deleted information is associated with the key, in the context of the CF. Truncate snapshots and then deletes the SSTables on disk, it does not write to the SSTables. So it is possible for a write to be stored with a lower timestamp than the truncate, because truncate does not have a timestamp. cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 4/05/2012, at 1:28 AM, Peter Dijkshoorn wrote: Hi guys, I got a weird thingy popping up twice today, I run a test where I insert a milion records via YCSB and edited it to allow me to adjust the consistency level: the write operations are done with ConsistencyLevel.ALL. This is send to a 4 (virtual) node cluster with a keyspace 'test' set up with replication factor 3. Now I expect that because of the ConsistencyLevel.ALL there is no hinted handoff active, since writes are to be accepted by all nodes before the operation returns to the client. The client gets only OK back, none fails. After the test I run a truncate, and a count which reveals still active records, time does not matter, I have to re-invoke the truncate to remove the records. [cqlsh 2.0.0 | Cassandra 1.0.8 | CQL spec 2.0.0 | Thrift protocol 19.20.0] cqlsh use test; cqlsh:test truncate usertable; cqlsh:test select count(*) from usertable ; count --- 3 On the cassandra output (-f) I can see that there is some handoff-ing active, which I did not expect. Has anyone an idea why the handoff is active while issuing opperations with ConsistencyLevel.ALL? Why is the truncate not correctly put in sync and allows subsequent handoff's delivered of records originally set before the truncate? Thanks if you can clarify these thing, I
enforcing ordering
Hi, I'm wondering if there is a common 'pattern' to address a scenario we will have to deal with. We will be storing a set of Column/Value pairs per Key where the Column/Values are read from a set of files that we download regularly. We need the loading to be resilient and we can receive corrections for some of the Column/Values that can only be loaded after the initial data has been inserted. The challenge we have is that we have a strong preference for active/active loading of data and can't see how to achieve this without some form of serialisation (which Cassandra doesn't support - correct ?) thanks -- *Franc Carter* | Systems architect | Sirca Ltd marc.zianideferra...@sirca.org.au franc.car...@sirca.org.au | www.sirca.org.au Tel: +61 2 9236 9118 Level 9, 80 Clarence St, Sydney NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215
Re: using the proxy on the cli or configHelper to connect to cassandra server
There is no support in the cli for using a socks proxy. You would need to add it. Take a look in CliMain.java Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/05/2012, at 10:00 AM, Shawna Qian wrote: Hello: In our cassandra settings, we need to specify the proxy to access the cassandra: if using the java code, it will be like this: Proxy proxy = new Proxy(Proxy.Type.SOCKS, new InetSocketAddress(socks.corp.yahoo.com, 1080)); Socket socket = new Socket (proxy); socket.connect(new InetSocketAddress(cassieHostName, cassiePort)); TSocket tsokect = new TSocket(socket); TTransport tr = new TFramedTransport(tsokect); But I am not sure how to specify this in the cassandra-cli. Also if I use configHelper in hadoop jobs, how can I specify the proxy information? Thx Shawna
Re: getting status of long running repair
When you look in the logs please let me know if you see this error… https://issues.apache.org/jira/browse/CASSANDRA-4223 I look at nodetool compactionstats (for the Merkle tree phase), nodetool netstats for the streaming, and this to check for streaming progress: while true; do date; diff (nodetool -h localhost netstats) (sleep 5 nodetool -h localhost netstats); done Or use Data Stax Ops Centre where possible http://www.datastax.com/products/opscenter Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/05/2012, at 2:15 PM, Ben Coverston wrote: Check the log files for warnings or errors. They may indicate why your repair failed. On Mon, May 7, 2012 at 10:09 AM, Bill Au bill.w...@gmail.com wrote: I restarted the nodes and then restarted the repair. It is still hanging like before. Do I keep repeating until the repair actually finish? Bill On Fri, May 4, 2012 at 2:18 PM, Rob Coli rc...@palominodb.com wrote: On Fri, May 4, 2012 at 10:30 AM, Bill Au bill.w...@gmail.com wrote: I know repair may take a long time to run. I am running repair on a node with about 15 GB of data and it is taking more than 24 hours. Is that normal? Is there any way to get status of the repair? tpstats does show 2 active and 2 pending AntiEntropySessions. But netstats and compactionstats show no activity. As indicated by various recent threads to this effect, many versions of cassandra (including current 1.0.x release) contain bugs which sometimes prevent repair from completing. The other threads suggest that some of these bugs result in the state you are in now, where you do not see anything that looks like appropriate activity. Unfortunately the only solution offered on these other threads is the one I will now offer, which is to restart the participating nodes and re-start the repair. I am unaware of any JIRA tickets tracking these bugs (which doesn't mean they don't exist, of course) so you might want to file one. :) =Rob -- =Robert Coli AIMGTALK - rc...@palominodb.com YAHOO - rcoli.palominob SKYPE - rcoli_palominodb -- Ben Coverston DataStax -- The Apache Cassandra Company
Re: enforcing ordering
Can you store the corrections in a separate CF? When the client reads the key, reads from the original the corrects CF at the same time. Apply the correction only on the client side. When you have confirmed the ingest has completed, run a background jobs to apply the corrections, store the updated values and delete the correction data. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/05/2012, at 9:35 PM, Franc Carter wrote: Hi, I'm wondering if there is a common 'pattern' to address a scenario we will have to deal with. We will be storing a set of Column/Value pairs per Key where the Column/Values are read from a set of files that we download regularly. We need the loading to be resilient and we can receive corrections for some of the Column/Values that can only be loaded after the initial data has been inserted. The challenge we have is that we have a strong preference for active/active loading of data and can't see how to achieve this without some form of serialisation (which Cassandra doesn't support - correct ?) thanks -- Franc Carter | Systems architect | Sirca Ltd franc.car...@sirca.org.au | www.sirca.org.au Tel: +61 2 9236 9118 Level 9, 80 Clarence St, Sydney NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215
Re: enforcing ordering
On Tue, May 8, 2012 at 8:09 PM, aaron morton aa...@thelastpickle.comwrote: Can you store the corrections in a separate CF? Yes, I thought of that, but that turns on read in to two ;-( When the client reads the key, reads from the original the corrects CF at the same time. Apply the correction only on the client side. When you have confirmed the ingest has completed, run a background jobs to apply the corrections, store the updated values and delete the correction data. I was thinking down this path, but I ended up chasing the rabbit down a deep hole of race conditions . . . cheers Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/05/2012, at 9:35 PM, Franc Carter wrote: Hi, I'm wondering if there is a common 'pattern' to address a scenario we will have to deal with. We will be storing a set of Column/Value pairs per Key where the Column/Values are read from a set of files that we download regularly. We need the loading to be resilient and we can receive corrections for some of the Column/Values that can only be loaded after the initial data has been inserted. The challenge we have is that we have a strong preference for active/active loading of data and can't see how to achieve this without some form of serialisation (which Cassandra doesn't support - correct ?) thanks -- *Franc Carter* | Systems architect | Sirca Ltd marc.zianideferra...@sirca.org.au franc.car...@sirca.org.au | www.sirca.org.au Tel: +61 2 9236 9118 Level 9, 80 Clarence St, Sydney NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215 -- *Franc Carter* | Systems architect | Sirca Ltd marc.zianideferra...@sirca.org.au franc.car...@sirca.org.au | www.sirca.org.au Tel: +61 2 9236 9118 Level 9, 80 Clarence St, Sydney NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215
Use-case: multi-instance webshop
Hi there, I'm working on a datamodel for a multi-website, multi-customer system. Things we would like to do: - search products (lucene / solr / solandra) - multi-filter (e.g. categories) - reviews - voting I can't really see how to do the filtering of the products by categories and even things like price (ranges would be possible with C*). Is Cassandra a fit for this use-case or should we just stick with the oldskool MySQL and put things like votes, reviews etc in our C* store? -- With kind regards, Robin Verlangen www.robinverlangen.nl
RE: sstableloader 1.1 won't stream
I've updated all nodes to 1.1 but I keep getting the same problem... Any other thoughts about this? Kind regards, Pieter -Original Message- From: Benoit Perroud [mailto:ben...@noisette.ch] Sent: maandag 7 mei 2012 22:21 To: user@cassandra.apache.org Subject: Re: sstableloader 1.1 won't stream You may want to upgrade all your nodes to 1.1. The streaming process connect to every living nodes of the cluster (you can explicitely diable some nodes), so all nodes need to speak 1.1. 2012/5/7 Pieter Callewaert pieter.callewa...@be-mobile.be: Hi, I’m trying to upgrade our bulk load process in our testing env. We use the SSTableSimpleUnsortedWriter to write tables, and use sstableloader to stream it into our cluster. I’ve changed the writer program to fit to the 1.1 api, but now I’m having troubles to load them to our cluster. The cluster exists out of one 1.1 node and two 1.0.9 nodes. I’ve enabled debug as parameter and in the log4j conf. [root@bms-app1 ~]# ./apache-cassandra/bin/sstableloader --debug -d 10.10.10.100 /tmp/201205071234/MapData024/HOS/ INFO 16:25:40,735 Opening /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1 (1588949 bytes) INFO 16:25:40,755 JNA not found. Native methods will be disabled. DEBUG 16:25:41,060 INDEX LOAD TIME for /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1: 327 ms. Streaming revelant part of /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to [/10.10.10.102, /10.10.10.100, /10.10.10.101] INFO 16:25:41,083 Stream context metadata [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6557280 - 0%], 1 sstables. DEBUG 16:25:41,084 Adding file /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed. INFO 16:25:41,087 Streaming to /10.10.10.102 DEBUG 16:25:41,092 Files are /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6557280 - 0% INFO 16:25:41,099 Stream context metadata [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6551840 - 0%], 1 sstables. DEBUG 16:25:41,100 Adding file /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed. INFO 16:25:41,100 Streaming to /10.10.10.100 DEBUG 16:25:41,100 Files are /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6551840 - 0% INFO 16:25:41,102 Stream context metadata [/tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=2 progress=0/6566400 - 0%], 1 sstables. DEBUG 16:25:41,102 Adding file /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db to be streamed. INFO 16:25:41,102 Streaming to /10.10.10.101 DEBUG 16:25:41,102 Files are /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=2 progress=0/6566400 - 0% progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 16:25:41,107 Failed attempt 1 to connect to /10.10.10.101 to stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=2 progress=0/6566400 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address) WARN 16:25:41,108 Failed attempt 1 to connect to /10.10.10.102 to stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6557280 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address) WARN 16:25:41,108 Failed attempt 1 to connect to /10.10.10.100 to stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6551840 - 0%. Retrying in 4000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address) progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 16:25:45,109 Failed attempt 2 to connect to /10.10.10.101 to stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=2 progress=0/6566400 - 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address) WARN 16:25:45,110 Failed attempt 2 to connect to /10.10.10.102 to stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6557280 - 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address) WARN 16:25:45,110 Failed attempt 2 to connect to /10.10.10.100 to stream /tmp/201205071234/MapData024/HOS/MapData024-HOS-hc-1-Data.db sections=1 progress=0/6551840 - 0%. Retrying in 8000 ms. (java.net.SocketException: Invalid argument or cannot assign requested address) progress: [/10.10.10.102 0/1 (0)] [/10.10.10.100 0/1 (0)] [/10.10.10.101 0/1 (0)] [total: 0 - 0MB/s (avg: 0MB/s)] WARN 16:25:53,113 Failed attempt 3 to connect to /10.10.10.101 to stream
Re: [RELEASE] Apache Cassandra 1.0.10 released
Hi Jonas, the bug that was fixed in 4116 meant that the max timestamp recorded for an sstable didn't consider any tombstones from row deletions. This meant that from some queries, some sstables were not being read when they should have been. I couldn't say categorically that this would cause the deleted data to reappear in read results, but I can see how it could do. Cheers, Sam On 8 May 2012 10:15, Jonas Borgström jo...@borgstrom.se wrote: Hi, Can someone give some more details about the CASSANDRA-4116 bug fixed in this release? Could this cause resurrection of deleted data for example? https://issues.apache.org/jira/browse/CASSANDRA-4116 / Jonas On 2012-05-08 11:04 , Sylvain Lebresne wrote: The Cassandra team is pleased to announce the release of Apache Cassandra version 1.0.10. Cassandra is a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. You can read more here: http://cassandra.apache.org/ Downloads of source and binary distributions are listed in our download section: http://cassandra.apache.org/download/ This version is maintenance/bug fix release[1]. As always, please pay attention to the release notes[2] and Let us know[3] if you were to encounter any problem. Have fun! [1]: http://goo.gl/u8gIO (CHANGES.txt) [2]: http://goo.gl/mAHbY (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA
Re: Timeout Exception in get_slice
Maybe one of the problems is that I am reading the columns in a row and the rows themselves in batches, using the count attribute in the SliceRange and by changing the start column or the corresponding for rows with the KeyRange. According to your blog post, using start key to read for millions of rows/columns has a lot of latency, but how else can I read an entire row that does not fit into memory? I'll have to run some tests again and check the tpstats. Still, do you think that adding more machines to the cluster will help a lot? I say this, because I started with a 3 node cluster and have scaled to a 5 node cluster with little improvement... Thanks anyway. On May 8, 2012, at 9:54 AM, aaron morton wrote: If I was rebuilding my power after spending the first thousand years of the Third Age as a shapeless evil I would cast my Eye of Fire in the direction of the filthy little multi_gets. A node can fail to respond to a query with rpc_timeout for two reasons: either the command did not run or the command started but did not complete. The former is much more likely. If it is happening you will see large pending counts and dropped messages in nodetool tpstats, you will also see log entries about dropped messages. When you send a multi_get each row you request becomes a message in the read thread pool. If you request 100 rows you will put 100 messages in the pool, which by default has 32 threads. If some clients are sending large multi get (or batch mutations) you can overload nodes and starve other clients. for background, some metrics here for selecting from 10million columns http://thelastpickle.com/2011/07/04/Cassandra-Query-Plans/ Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 6/05/2012, at 7:14 AM, Luís Ferreira wrote: Hi, I'm doing get_slice on huge rows (3 million columns) and even though I am doing it iteratively I keep getting TimeoutExceptions. I've tried to change the number of columns fetched but it did not work. I have a 5 machine cluster, each with 4GB of which 3 are dedicated to cassandra's heap, but still the all consume all of the memory and get huge IO wait due to the amout of reads. I am running tests with 100 clients all performing multiple operations mostly get_slice, get and multi_get, but the timeouts only occur in the get_slice. Does this have anything to do with cassandra's ability (or lack thereof) to keep the rows in memory? Or am I doing anything wrong? Any tips? Cumpliments, Luís Ferreira Cumprimentos, Luís Ferreira
Re: Failing to delete commitlog at startup/shutdown (Windows)
Hi Steve, Thanks for your reply, sorry for the delay in getting back to you. We're actually doing something very similar already, using Hector's EmbeddedServerHelper (it's basically the same, maybe it came from the same code). Unfortunately whilst writing this our internet went down and I sometimes need to develop offline anyway, so using an external Cassandra instance isn't really an option. I've had a try using the maven-cassandra-plugin and don't seem to be having the problem any more, plus it's a neater solution anyway. Conan On 23 April 2012 15:51, Steve Neely sne...@rallydev.com wrote: We used a modified version of Ran's embedded Cassandra for a while: http://prettyprint.me/2010/02/14/running-cassandra-as-an-embedded-service/which worked well for us. You have way more control over that. Recently, we switched to having a single Cassandra installation that runs all the time. Kind of like you'd treat a regular relational DB. Just fire up Cassandra, leave it running and point your tests at that instance. Seems like starting up your data store every time you execute integration tests will slow them down and isn't really helpful. BTW, you may want to scrub the test data out of Cassandra when you're test suite finishes. -- Steve On Mon, Apr 23, 2012 at 8:41 AM, Conan Cook conan.c...@amee.com wrote: Hi, I'm experiencing a problem running a suite of integration tests on Windows 7, using Cassandra 1.0.9 and Java 1.6.0_31. A new cassandra instance is spun up for each test class and shut down afterwards, using the Maven Failsafe plugin. The problem is that the Commitlog file seems to be kept open, and so subsequent test classes fail to delete it. Here is the stack trace: java.io.IOException: Failed to delete D:\amee.realtime.api\server\engine\tmp\var\lib\cassandra\commitlog\CommitLog-1335190398587.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:220) at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:216) ... I've tried to delete the file when shutting down Cassandra and before firing up a new one. I've tried setting the failsafe plugin's forkMode to both once and always, so that it fires up a new JVM for each test or a single JVM for all tests; the results are similar. Debugging through the code takes me right down to the native method call in the windows filesystem class in the JVM, and an access denied error is returned; I'm also unable to delete it manually through Windows Explorer or a terminal window at that point (with the JVM suspended), and running Process Explorer indicates that a Java process has a handle open to that file. I've read a number of posts and mails mentioning this problem and there is a JIRA saying a similar problem is fixed ( https://issues.apache.org/jira/browse/CASSANDRA-1348). I've tried a number of things to clean up the Commitlog file after each test is complete, and have followed the recommendations made here (I'm also using Hector's EmbeddedServerHelper to start/stop Cassandra): http://stackoverflow.com/questions/7944287/how-to-cleanup-embedded-cassandra-after-unittest Does anyone have any ideas on how to avoid this issue? I don't have any way of knowing what it is that's holding onto this file other than a Java process. Thanks! Conan
Re: getting status of long running repair
There are no error message in my log. I ended up restarting all the nodes in my cluster. After that I was able to run repair successfully on one of the node. It took about 40 minutes. Feeling lucky I ran repair on another node and it is stuck again. tpstats show 1 active and 1 pending AntiEntropySessions. netstats and compactionstats show no activity. I took a close look at the log file, it shows that the node requested merkle tree from 4 nodes (including itself). It actually received 3 of those merkle trees. It looks like it is stuck waiting for that last one. I checked the node where the request was sent to, there isn't anything in the log on repair. So it looks like the merkle tree request has gotten lost some how. It has been 8 hours since the repair was issue and it is still stuck. I am going to let it run a bit longer to see if it will eventually finish. I have observed that if I restart all the nodes, I would be able to run repair successfully on a single node. I have done that twice already. But after that all repairs will hang. Since we are supposed to run repair periodically, having to restart all nodes before running repair on each node isn't really viable for us. Bill On Tue, May 8, 2012 at 6:04 AM, aaron morton aa...@thelastpickle.comwrote: When you look in the logs please let me know if you see this error… https://issues.apache.org/jira/browse/CASSANDRA-4223 I look at nodetool compactionstats (for the Merkle tree phase), nodetool netstats for the streaming, and this to check for streaming progress: while true; do date; diff (nodetool -h localhost netstats) (sleep 5 nodetool -h localhost netstats); done Or use Data Stax Ops Centre where possible http://www.datastax.com/products/opscenter Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 8/05/2012, at 2:15 PM, Ben Coverston wrote: Check the log files for warnings or errors. They may indicate why your repair failed. On Mon, May 7, 2012 at 10:09 AM, Bill Au bill.w...@gmail.com wrote: I restarted the nodes and then restarted the repair. It is still hanging like before. Do I keep repeating until the repair actually finish? Bill On Fri, May 4, 2012 at 2:18 PM, Rob Coli rc...@palominodb.com wrote: On Fri, May 4, 2012 at 10:30 AM, Bill Au bill.w...@gmail.com wrote: I know repair may take a long time to run. I am running repair on a node with about 15 GB of data and it is taking more than 24 hours. Is that normal? Is there any way to get status of the repair? tpstats does show 2 active and 2 pending AntiEntropySessions. But netstats and compactionstats show no activity. As indicated by various recent threads to this effect, many versions of cassandra (including current 1.0.x release) contain bugs which sometimes prevent repair from completing. The other threads suggest that some of these bugs result in the state you are in now, where you do not see anything that looks like appropriate activity. Unfortunately the only solution offered on these other threads is the one I will now offer, which is to restart the participating nodes and re-start the repair. I am unaware of any JIRA tickets tracking these bugs (which doesn't mean they don't exist, of course) so you might want to file one. :) =Rob -- =Robert Coli AIMGTALK - rc...@palominodb.com YAHOO - rcoli.palominob SKYPE - rcoli_palominodb -- Ben Coverston DataStax -- The Apache Cassandra Company
