Re: SnappyCompressor and Cassandra 1.1.1
Yeah, we tried to catch when Snappy couldn't load (and deactivate it when that happens) but apparently we've missed a few of the exceptions that can be thrown in that case. I've created https://issues.apache.org/jira/browse/CASSANDRA-4400 to fix that. If you could try the patch on that issue and check it works for you that would be awesome since I don't have a Raspberry Pi myself to test. -- Sylvain On Sun, Jul 1, 2012 at 3:26 PM, Andy Cobley acob...@computing.dundee.ac.uk wrote: I'm running Cassandra on Raspberry Pi (for educational reason) and have been successfully running 1.1.0 for some time. However there is no native build of SnappyCompressor for the platform (I'm currently working n rectifying that if I can) so that compression is unavailable. When I try and start 1.1.1 on the platform I'm getting the following error which looks to me like 1.1.1 is trying to load snappy compressor at startup and falls over when to can't find it. Thats not been the case with 1.1.0: INFO 14:22:07,600 Global memtable threshold is enabled at 35MB java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317) at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219) at org.xerial.snappy.Snappy.clinit(Snappy.java:44) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompressor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.clinit(SnappyCompressor.java:37) at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:76) at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:439) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106) Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1681) at java.lang.Runtime.loadLibrary0(Runtime.java:840) at java.lang.System.loadLibrary(System.java:1047) at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52) ... 17 more ERROR 14:22:09,934 Exception encountered during startup org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) at org.xerial.snappy.Snappy.clinit(Snappy.java:44) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompressor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.clinit(SnappyCompressor.java:37) at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:76) at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:439) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106) org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) at org.xerial.snappy.Snappy.clinit(Snappy.java:44) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor.java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompressor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.clinit(SnappyCompressor.java:37) at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:76) at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:439) at
Re: Bootstrap code path
On Sun, Jul 1, 2012 at 3:31 PM, Bill Hastings bllhasti...@gmail.com wrote: Could someone please tell me where I should start looking at code to understand how cassandra bootstrap process works? The bootstrap phase pretty much start in the bootstrap() method in StorageService.java, though a good part of the code is in Bootstrapper.java. Also is my understanding correct that the new nodes that are added are not joining the ring till the bootstrap process is complete i.e do not receive any read or write requests from outside? It is true for reads but not for writes. Writes are send early in the bootstrap process otherwise the node wouldn't be up to date once the bootstrap is completed. -- Sylvain
Re: BulkLoading SSTables and compression
On Mon, Jul 2, 2012 at 12:14 AM, jmodha jmo...@gmail.com wrote: Sure, before I create a ticket, is there a way I can confirm that the sstables are indeed not compressed other than running the rebuildsstables nodetool command (and observing the live size go down)? If the sstable is compressed it will have a -CompressionInfo.db component (i.e. one of the file composing the sstable with end with --CompressionInfo.db). If there is no such file then it'ss not compressed. -- Sylvain Thanks. -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/BulkLoading-SSTables-and-compression-tp7580849p7580922.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Memtable tuning in 1.0 and higher
I'm afraid not. It's too much change for an oldstable release series, and the bulk of the change is to AtomicSortedColumns which doesn't exist in 1.0, so even if we wanted to take a maybe it's okay if we release it first in 1.1.3 and then backport approach it wouldn't improve our safety margin since you'd basically need to rewrite the patch. On Sun, Jul 1, 2012 at 6:40 AM, Joost Van De Wijgerd jwijg...@gmail.com wrote: Hi Jonathan, Looks good, any chance of porting this fix to the 1.0 branch? Kind regards Joost Sent from my iPhone On 1 jul. 2012, at 09:25, Jonathan Ellis jbel...@gmail.com wrote: On Thu, Jun 28, 2012 at 1:39 PM, Joost van de Wijgerd jwijg...@gmail.com wrote: the currentThoughput is increased even before the data is merged into the memtable so it is actually measuring the throughput afaik. You're right. I've attached a patch to https://issues.apache.org/jira/browse/CASSANDRA-4399 to fix this. -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: SnappyCompressor and Cassandra 1.1.1
Thanks , Will give it a go now. Andy On 02/07/2012 08:13, Sylvain Lebresne sylv...@datastax.com wrote: Yeah, we tried to catch when Snappy couldn't load (and deactivate it when that happens) but apparently we've missed a few of the exceptions that can be thrown in that case. I've created https://issues.apache.org/jira/browse/CASSANDRA-4400 to fix that. If you could try the patch on that issue and check it works for you that would be awesome since I don't have a Raspberry Pi myself to test. -- Sylvain On Sun, Jul 1, 2012 at 3:26 PM, Andy Cobley acob...@computing.dundee.ac.uk wrote: I'm running Cassandra on Raspberry Pi (for educational reason) and have been successfully running 1.1.0 for some time. However there is no native build of SnappyCompressor for the platform (I'm currently working n rectifying that if I can) so that compression is unavailable. When I try and start 1.1.1 on the platform I'm getting the following error which looks to me like 1.1.1 is trying to load snappy compressor at startup and falls over when to can't find it. Thats not been the case with 1.1.0: INFO 14:22:07,600 Global memtable threshold is enabled at 35MB java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java :57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI mpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317) at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219) at org.xerial.snappy.Snappy.clinit(Snappy.java:44) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor .java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompr essor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.clinit(SnappyCompress or.java:37) at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:76) at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescripto r.java:439) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescripto r.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassan draDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCas sandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106 ) Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1681) at java.lang.Runtime.loadLibrary0(Runtime.java:840) at java.lang.System.loadLibrary(System.java:1047) at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java: 52) ... 17 more ERROR 14:22:09,934 Exception encountered during startup org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) at org.xerial.snappy.Snappy.clinit(Snappy.java:44) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor .java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompr essor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.clinit(SnappyCompress or.java:37) at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:76) at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescripto r.java:439) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescripto r.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassan draDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCas sandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106 ) org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) at org.xerial.snappy.Snappy.clinit(Snappy.java:44) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor .java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompr essor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.clinit(SnappyCompress or.java:37) at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:76) at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79) at
Re: ClassCastException during Cassandra server startup
Hello Yuki, Could you give me hints about where to find these files. I have a look in the installation folder of Cassandra and in the /var/lib/cassandra folder? Thanks very much for your help. Thierry That was bug in 1.1.1 and fixed in https://issues.apache.org/jira/browse/CASSANDRA-4331. Workaround is deleting the key cache files for your index CFs should fix this. Yuki
Nodes marked dead…. leap second?
Hi, we had some really weird issues during the weekend, with our cassandra nodes starting marking as dead other (working) nodes in the cluster. That happened all Sunday, and it's still happening. Node are marked dead and up all the time…. Some example logs: INFO [GossipTasks:1] 2012-07-02 06:55:01,804 Gossiper.java (line 818) InetAddress /xx.xx.xx.233 is now dead. INFO [GossipTasks:1] 2012-07-02 06:55:01,805 Gossiper.java (line 818) InetAddress /xx.xx.xx.235 is now dead. INFO [GossipStage:1] 2012-07-02 06:55:21,748 Gossiper.java (line 804) InetAddress /xx.xx.xx.233 is now UP INFO [GossipStage:1] 2012-07-02 06:55:21,893 Gossiper.java (line 804) InetAddress /xx.xx.xx.235 is now UP INFO [GossipTasks:1] 2012-07-02 06:56:03,877 Gossiper.java (line 818) InetAddress /xx.xx.xx.235 is now dead. INFO [GossipTasks:1] 2012-07-02 06:57:58,537 Gossiper.java (line 818) InetAddress /xx.xx.xx.233 is now dead. INFO [GossipStage:1] 2012-07-02 06:59:06,444 Gossiper.java (line 804) InetAddress /xx.xx.xx.233 is now UP I couldn't find any real exception in the logs, but I noticed that the first error occurred at INFO [GossipTasks:1] 2012-07-01 02:00:31,169 Gossiper.java (line 818) InetAddress /xx.xx.xx.234 is now dead. 2012-07-01 02:00:31,169, in the German timezone were the machine is hosted, is June 30th 23:59:60 UTC, the leap second that caused quite a few issues this weekend. Can it be the cause of the cluster failure? Has anybody noticed similar issues? ( also see https://twitter.com/redditstatus/status/219244389044731904 ) I'm running Ubuntu 10.04.3 LTS. Many thanks, -- Filippo Diotalevi
cql 3.0.0
Hi All,I am using cassandra 1.0.8. How can i use cql 3.0.0 in this? -- Abhijit Chanda Analyst VeHere Interactive Pvt. Ltd. +91-974395
Re: BulkLoading SSTables and compression
Thanks Sylvain.
I had a look at a node where we streamed data to and I do indeed see the
..-CompressionInfo.db files..
However, prior to running the upgradesstables command, the total size of
all the SSTables was 27GB and afterwards its 12GB.
So even though the CompressionInfo files were there immediately after bulk
loading the data, it wasn't really compressed..?
Can you think of anything else I can try to confirm this is indeed a bug?
Out of interest, we're not specifying a specific chunk size on the schema
(in the hope that it would just use the default of 64kb), so it reads
something like:
create column family test
with column_type = 'Standard'
and comparator = 'BytesType'
and default_validation_class = 'UTF8Type'
and key_validation_class = 'BytesType'
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and compression_options = {'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
Would this cause any issues?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/BulkLoading-SSTables-and-compression-tp7580849p7580933.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
Re: cql 3.0.0
On Mon, Jul 2, 2012 at 1:09 PM, Abhijit Chanda abhijit.chan...@gmail.com wrote: Hi All, I am using cassandra 1.0.8. How can i use cql 3.0.0 in this? You can't, CQL 3 is only in Cassandra 1.1 onwards. -- Sylvain -- Abhijit Chanda Analyst VeHere Interactive Pvt. Ltd. +91-974395
Re: BulkLoading SSTables and compression
On Mon, Jul 2, 2012 at 1:24 PM, jmodha jmo...@gmail.com wrote:
Out of interest, we're not specifying a specific chunk size on the schema
(in the hope that it would just use the default of 64kb), so it reads
something like:
Actually, we've had bugs related to the handling of the chunk size in early 1.0
releases and if I remember correctly that was fully fixed only in
1.0.5. So it is
possible that your 1.0.3 sstables are cmpressed but with a ridiculously low
chunk size (like 64 bytes instead of 64K), which would amount to no
compression.
In that case then yes it is a bug, but one that has been fixed by
upgrading to 1.1.
--
Sylvain
create column family test
with column_type = 'Standard'
and comparator = 'BytesType'
and default_validation_class = 'UTF8Type'
and key_validation_class = 'BytesType'
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and compression_options = {'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
Would this cause any issues?
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/BulkLoading-SSTables-and-compression-tp7580849p7580933.html
Sent from the cassandra-u...@incubator.apache.org mailing list archive at
Nabble.com.
Re: Nodes marked dead…. leap second?
Hi, I did have the same problem with cassandra 1.1.1 on Ubuntu 11.10 I had to reboot all nodes I'm interested in any information about this. Thanks Jean Paul 2012/7/2 Filippo Diotalevi fili...@ntoklo.com Hi, we had some really weird issues during the weekend, with our cassandra nodes starting marking as dead other (working) nodes in the cluster. That happened all Sunday, and it's still happening. Node are marked dead and up all the time…. Some example logs: INFO [GossipTasks:1] 2012-07-02 06:55:01,804 Gossiper.java (line 818) InetAddress /xx.xx.xx.233 is now dead. INFO [GossipTasks:1] 2012-07-02 06:55:01,805 Gossiper.java (line 818) InetAddress /xx.xx.xx.235 is now dead. INFO [GossipStage:1] 2012-07-02 06:55:21,748 Gossiper.java (line 804) InetAddress /xx.xx.xx.233 is now UP INFO [GossipStage:1] 2012-07-02 06:55:21,893 Gossiper.java (line 804) InetAddress /xx.xx.xx.235 is now UP INFO [GossipTasks:1] 2012-07-02 06:56:03,877 Gossiper.java (line 818) InetAddress /xx.xx.xx.235 is now dead. INFO [GossipTasks:1] 2012-07-02 06:57:58,537 Gossiper.java (line 818) InetAddress /xx.xx.xx.233 is now dead. INFO [GossipStage:1] 2012-07-02 06:59:06,444 Gossiper.java (line 804) InetAddress /xx.xx.xx.233 is now UP I couldn't find any real exception in the logs, but I noticed that the first error occurred at INFO [GossipTasks:1] 2012-07-01 02:00:31,169 Gossiper.java (line 818) InetAddress /xx.xx.xx.234 is now dead. 2012-07-01 02:00:31,169, in the German timezone were the machine is hosted, is June 30th 23:59:60 UTC, the leap second that caused quite a few issues this weekend. Can it be the cause of the cluster failure? Has anybody noticed similar issues? ( also see https://twitter.com/redditstatus/status/219244389044731904 ) I'm running Ubuntu 10.04.3 LTS. Many thanks, -- Filippo Diotalevi -- - Jean Paul Adant - Créative-Ingénierie jean.paul.ad...@gmail.com
Re: Nodes marked dead…. leap second?
Bug: https://lkml.org/lkml/2012/6/30/122 Simple fix to reset the leap second flag: date; date `date +%m%d%H%M%C%y.%S`; date; /Henrik On Mon, Jul 2, 2012 at 1:56 PM, Jean Paul Adant jean.paul.ad...@gmail.comwrote: Hi, I did have the same problem with cassandra 1.1.1 on Ubuntu 11.10 I had to reboot all nodes I'm interested in any information about this. Thanks Jean Paul 2012/7/2 Filippo Diotalevi fili...@ntoklo.com Hi, we had some really weird issues during the weekend, with our cassandra nodes starting marking as dead other (working) nodes in the cluster. That happened all Sunday, and it's still happening. Node are marked dead and up all the time…. Some example logs: INFO [GossipTasks:1] 2012-07-02 06:55:01,804 Gossiper.java (line 818) InetAddress /xx.xx.xx.233 is now dead. INFO [GossipTasks:1] 2012-07-02 06:55:01,805 Gossiper.java (line 818) InetAddress /xx.xx.xx.235 is now dead. INFO [GossipStage:1] 2012-07-02 06:55:21,748 Gossiper.java (line 804) InetAddress /xx.xx.xx.233 is now UP INFO [GossipStage:1] 2012-07-02 06:55:21,893 Gossiper.java (line 804) InetAddress /xx.xx.xx.235 is now UP INFO [GossipTasks:1] 2012-07-02 06:56:03,877 Gossiper.java (line 818) InetAddress /xx.xx.xx.235 is now dead. INFO [GossipTasks:1] 2012-07-02 06:57:58,537 Gossiper.java (line 818) InetAddress /xx.xx.xx.233 is now dead. INFO [GossipStage:1] 2012-07-02 06:59:06,444 Gossiper.java (line 804) InetAddress /xx.xx.xx.233 is now UP I couldn't find any real exception in the logs, but I noticed that the first error occurred at INFO [GossipTasks:1] 2012-07-01 02:00:31,169 Gossiper.java (line 818) InetAddress /xx.xx.xx.234 is now dead. 2012-07-01 02:00:31,169, in the German timezone were the machine is hosted, is June 30th 23:59:60 UTC, the leap second that caused quite a few issues this weekend. Can it be the cause of the cluster failure? Has anybody noticed similar issues? ( also see https://twitter.com/redditstatus/status/219244389044731904 ) I'm running Ubuntu 10.04.3 LTS. Many thanks, -- Filippo Diotalevi -- - Jean Paul Adant - Créative-Ingénierie jean.paul.ad...@gmail.com
Re: BulkLoading SSTables and compression
Just to clarify, the data that we're loading SSTables from (v1.0.3) doesn't have compression enabled on any of the CF's. So in theory the compression should occur on the receiving end (v1.1.1) as we're going from uncompressed data to compressed data. So I'm not sure if the bug you mention is causing the behaviour we're seeing here. The only thing I can think of is that the upgradesstables option follows a slightly different path to the bulk uploader when it comes to generating the sstables that have been flushed to disk? -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/BulkLoading-SSTables-and-compression-tp7580849p7580938.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
cassandra on re-Start
Hi, I have set up cassandra on my linux system. When I restarted the system , it is showing the keyspace does not exist. Not even letting me to create the keyspace with the same name again. Did anybody find this issue? Any solutions on it? Cheers, Puneet
forceUserDefinedCompaction in 1.1.0
Hi guys, We have a 6-node 1.0.9 cluster for production and 3-node 1.1.0 cluster for testing the new version of Cassandra. In both we insert data in a particular CF with always a TTL of 31 days. To clean up the files faster we use the forceUserDefinedCompaction to manually force compaction on the old sstables which we are sure the data has expired. In 1.0 this works perfect, but in the 1.1 the command executes without error, but in the log of the node I see the following: INFO [CompactionExecutor:546] 2012-07-02 15:05:11,837 CompactionManager.java (line 337) Will not compact MapData024CTwT/MapData024CTwT-HOS-hc-21150: it is not an active sstable INFO [CompactionExecutor:546] 2012-07-02 15:05:11,838 CompactionManager.java (line 350) No file to compact for user defined compaction I am pretty sure the sstable because it should contain partly expired data en partly data which is still active. Does this have to do something with the new directory structure from 1.1 ? Or are the parameters changed from the function? Kind regards, Pieter Callewaert
RE: forceUserDefinedCompaction in 1.1.0
Hi, While I was typing my mail I had the idea to try with the new directory layout. It seems you have to change the parameter settings from 1.0 to 1.1 In 1.0: Param 1: keyspace name Param 2: datafile name In 1.1: Param 1: keyspace name Param 2: column family name/datafile name Don't know if this is a bug or a breaking change ? Kind regards, Pieter Callewaert From: Pieter Callewaert [mailto:pieter.callewa...@be-mobile.be] Sent: maandag 2 juli 2012 15:10 To: user@cassandra.apache.org Subject: forceUserDefinedCompaction in 1.1.0 Hi guys, We have a 6-node 1.0.9 cluster for production and 3-node 1.1.0 cluster for testing the new version of Cassandra. In both we insert data in a particular CF with always a TTL of 31 days. To clean up the files faster we use the forceUserDefinedCompaction to manually force compaction on the old sstables which we are sure the data has expired. In 1.0 this works perfect, but in the 1.1 the command executes without error, but in the log of the node I see the following: INFO [CompactionExecutor:546] 2012-07-02 15:05:11,837 CompactionManager.java (line 337) Will not compact MapData024CTwT/MapData024CTwT-HOS-hc-21150: it is not an active sstable INFO [CompactionExecutor:546] 2012-07-02 15:05:11,838 CompactionManager.java (line 350) No file to compact for user defined compaction I am pretty sure the sstable because it should contain partly expired data en partly data which is still active. Does this have to do something with the new directory structure from 1.1 ? Or are the parameters changed from the function? Kind regards, Pieter Callewaert
Any reason to limit one's self to a single high level java client?
I recognize that behind the scenes there's connection pooling and all kinds of nice asynchronous dispatch of requests to cassandra, but is there any sort of reason to avoid using different Java clients in the same application? I'm noticing that some are better suited to certain kinds activity than others. Dave
Re: SnappyCompressor and Cassandra 1.1.1
I've tested this and added a not to issue 4400. Hopefully that¹s the correct thing to do. Andy On 02/07/2012 08:13, Sylvain Lebresne sylv...@datastax.com wrote: Yeah, we tried to catch when Snappy couldn't load (and deactivate it when that happens) but apparently we've missed a few of the exceptions that can be thrown in that case. I've created https://issues.apache.org/jira/browse/CASSANDRA-4400 to fix that. If you could try the patch on that issue and check it works for you that would be awesome since I don't have a Raspberry Pi myself to test. -- Sylvain On Sun, Jul 1, 2012 at 3:26 PM, Andy Cobley acob...@computing.dundee.ac.uk wrote: I'm running Cassandra on Raspberry Pi (for educational reason) and have been successfully running 1.1.0 for some time. However there is no native build of SnappyCompressor for the platform (I'm currently working n rectifying that if I can) so that compression is unavailable. When I try and start 1.1.1 on the platform I'm getting the following error which looks to me like 1.1.1 is trying to load snappy compressor at startup and falls over when to can't find it. Thats not been the case with 1.1.0: INFO 14:22:07,600 Global memtable threshold is enabled at 35MB java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java :57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI mpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317) at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219) at org.xerial.snappy.Snappy.clinit(Snappy.java:44) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor .java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompr essor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.clinit(SnappyCompress or.java:37) at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:76) at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescripto r.java:439) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescripto r.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassan draDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCas sandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106 ) Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1681) at java.lang.Runtime.loadLibrary0(Runtime.java:840) at java.lang.System.loadLibrary(System.java:1047) at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java: 52) ... 17 more ERROR 14:22:09,934 Exception encountered during startup org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) at org.xerial.snappy.Snappy.clinit(Snappy.java:44) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor .java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompr essor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.clinit(SnappyCompress or.java:37) at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:76) at org.apache.cassandra.config.KSMetaData.systemKeyspace(KSMetaData.java:79) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescripto r.java:439) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescripto r.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassan draDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCas sandraDaemon.java:353) at org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:106 ) org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) at org.xerial.snappy.Snappy.clinit(Snappy.java:44) at org.apache.cassandra.io.compress.SnappyCompressor.create(SnappyCompressor .java:45) at org.apache.cassandra.io.compress.SnappyCompressor.isAvailable(SnappyCompr essor.java:55) at org.apache.cassandra.io.compress.SnappyCompressor.clinit(SnappyCompress or.java:37) at org.apache.cassandra.config.CFMetaData.clinit(CFMetaData.java:76) at
frequent node up/downs
Hello, I recently set up a 2 node cassandra cluster on dedicated hardware. In the logs there have been a lot of InetAddress xxx is now dead' or UP messages. Comparing the log messages between the 2 nodes, they seem to coincide with extremely long ParNew collections. I have seem some of up to 50 seconds. The installation is pretty vanilla, I didn't change any settings and the machines don't seem particularly busy - cassandra is the only thing running on the machine with an 8GB heap. The machine has 64GB of RAM and CPU/IO usage looks pretty light. I do see a lot of 'Heap is xxx full. You may need to reduce memtable and/or cache sizes' messages. Would this help with the long ParNew collections? That message seems to be triggered on a full collection.
RE: frequent node up/downs
Hi, Had the same problem this morning, seems related to the leap second bug. Rebooting the nodes fixed it for me, but there seems to be a fix also without rebooting the server. Kind regards, Pieter From: feedly team [mailto:feedly...@gmail.com] Sent: maandag 2 juli 2012 17:09 To: user@cassandra.apache.org Subject: frequent node up/downs Hello, I recently set up a 2 node cassandra cluster on dedicated hardware. In the logs there have been a lot of InetAddress xxx is now dead' or UP messages. Comparing the log messages between the 2 nodes, they seem to coincide with extremely long ParNew collections. I have seem some of up to 50 seconds. The installation is pretty vanilla, I didn't change any settings and the machines don't seem particularly busy - cassandra is the only thing running on the machine with an 8GB heap. The machine has 64GB of RAM and CPU/IO usage looks pretty light. I do see a lot of 'Heap is xxx full. You may need to reduce memtable and/or cache sizes' messages. Would this help with the long ParNew collections? That message seems to be triggered on a full collection.
Re: frequent node up/downs
Yeah! Look that. http://arstechnica.com/business/2012/07/one-day-later-the-leap-second-v-the-internet-scorecard/ I had the same problem. The solution was rebooting. On Mon, 2 Jul 2012 11:08:57 -0400 feedly team feedly...@gmail.com wrote: Hello, I recently set up a 2 node cassandra cluster on dedicated hardware. In the logs there have been a lot of InetAddress xxx is now dead' or UP messages. Comparing the log messages between the 2 nodes, they seem to coincide with extremely long ParNew collections. I have seem some of up to 50 seconds. The installation is pretty vanilla, I didn't change any settings and the machines don't seem particularly busy - cassandra is the only thing running on the machine with an 8GB heap. The machine has 64GB of RAM and CPU/IO usage looks pretty light. I do see a lot of 'Heap is xxx full. You may need to reduce memtable and/or cache sizes' messages. Would this help with the long ParNew collections? That message seems to be triggered on a full collection. -- Marcus Both
Re: ClassCastException during Cassandra server startup
Thierry, Key cache files are stored inside your saved_caches_directory defined in cassandra.yaml, which has default value of /var/lib/cassandra/saved_caches. Yuki On Monday, July 2, 2012 at 4:00 AM, Thierry Templier wrote: Hello Yuki, Could you give me hints about where to find these files. I have a look in the installation folder of Cassandra and in the /var/lib/cassandra folder? Thanks very much for your help. Thierry That was bug in 1.1.1 and fixed in https://issues.apache.org/jira/browse/CASSANDRA-4331. Workaround is deleting the key cache files for your index CFs should fix this. Yuki
bulk load glitch
I'm attempting to perform a bulk load by calling the jmx:bulkLoad method on
several nodes in parallel. In a Casssandra log
file I see a few occurrences of the following:
INFO [GossipTasks:1] 2012-07-02 10:12:33,626 Gossiper.java (line 759)
InetAddress /10.4.0.3 is now dead.
ERROR [GossipTasks:1] 2012-07-02 10:12:53,630 StreamOutSession.java (line 232)
StreamOutSession /10.4.0.3 failed because {} died or was restarted/removed
INFO [GossipStage:1] 2012-07-02 10:18:40,008 Gossiper.java (line 745)
InetAddress /10.4.0.3 is now UP
I can't tell whether the bulk load process recovered from the transient dead
node, or whether I need to start over.
Does anybody know?
Brian
This is Cassanrda 1.0.5, by the way.
faillout from AWS outage - HELP
all, my 4 node cluster seems pretty screwed up after the AWS outage. we found all our machines with their cpu stuck at 100%. so i went to restart each cassandra node one by one. i did node with token id 0 first. i came back but doesn't look like it doing anything. once i thought it was up i went and restarted the next. this one got stuck on the AMI init startup. i cancelled it, rebooted againg and now it't suck with [INFO] 07/02/12-15:42:29 Received 0 of 1 responses from: suggestions on how to fix this? thanks, deno
Re: jscv CPU Consumption
Thanks for the answer. In fact this is maybe simpler. We do a lot of inserts and a couple of reads. - tpstats show us the pending read stage was growing. - *It seems* there was an issue with one of our request. We have 4 indexed columns (UT8Type) First, one of the indexed value was ---. we replaced this value by and empty string and it works better. (running the query in CLI) Yet, we still observed pending Read Stage after 10 minutes of run (on a couple of nodes). Do you have any ideas, advices on how to investigate in more details? Any help would be appreciate. On Sun, Jul 1, 2012 at 8:56 AM, Jonathan Ellis jbel...@gmail.com wrote: Sounds like http://wiki.apache.org/cassandra/FAQ#ubuntu_ec2_hangs to me. On Fri, Jun 29, 2012 at 1:45 AM, Olivier Mallassi omalla...@octo.com wrote: Hi all We have a 12 servers clusters (8 cores by machines..). OS is Ubuntu 10.04.2. On one of the machine (only one) and without any load (no inserts, no reads), we have a huge CPU Load whereas there is no activities (no compaction in progress etc...) A top on the machine show us the process jscv is using all the available CPUs. Is that link to JNA? do you have any ideas? Cheers -- Olivier Mallassi OCTO Technology 50, Avenue des Champs-Elysées 75008 Paris Mobile: (33) 6 28 70 26 61 Tél: (33) 1 58 56 10 00 Fax: (33) 1 58 56 10 01 http://www.octo.com Octo Talks! http://blog.octo.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Olivier Mallassi OCTO Technology 50, Avenue des Champs-Elysées 75008 Paris Mobile: (33) 6 28 70 26 61 Tél: (33) 1 58 56 10 00 Fax: (33) 1 58 56 10 01 http://www.octo.com Octo Talks! http://blog.octo.com
Secondary Index, performance , data type
Hi all We have 4 indexed columns; all configured in UT8Type even if one columns is a date and the other an integer). 1/ the read query we run can have up to 4 criteria *select my_cf where columnA = a and columnB = b and columnC = c and columnD = d* This query, is fast (500ms) up to 3 criterias but when we add the fourth one, the exection time is 9,5s. Do you have any ideas? is there any way to understand how cassandra internally run the query (a kind of explain plan)? 2/ Is there any limitations on the number of criterias we can usually have? 3/ Even if we have different data type (date, string, int), we have all stored them as UTF8Type. Could we expect performance improvements if we use DateType, LongType? Many thx for all your answers. -- Olivier Mallassi OCTO Technology 50, Avenue des Champs-Elysées 75008 Paris Mobile: (33) 6 28 70 26 61 Tél: (33) 1 58 56 10 00 Fax: (33) 1 58 56 10 01 http://www.octo.com Octo Talks! http://blog.octo.com
Re: faillout from AWS outage - HELP
Could be related to the leap second bug.. try this: Bug: https://lkml.org/lkml/2012/6/30/122 Simple fix to reset the leap second flag: date; date `date +%m%d%H%M%C%y.%S`; date; mvg Joost On Mon, Jul 2, 2012 at 5:43 PM, Deno Vichas d...@syncopated.net wrote: all, my 4 node cluster seems pretty screwed up after the AWS outage. we found all our machines with their cpu stuck at 100%. so i went to restart each cassandra node one by one. i did node with token id 0 first. i came back but doesn't look like it doing anything. once i thought it was up i went and restarted the next. this one got stuck on the AMI init startup. i cancelled it, rebooted againg and now it't suck with [INFO] 07/02/12-15:42:29 Received 0 of 1 responses from: suggestions on how to fix this? thanks, deno -- Joost van de Wijgerd Visseringstraat 21B 1051KH Amsterdam +31624111401 joost.van.de.wijgerd@Skype http://www.linkedin.com/in/jwijgerd
Re: faillout from AWS outage - HELP
the node that doesn't want to start just spit out - /EC2 is experiencing some issues and has not allocated all of the resources in under 10 minutes. Aborting the clustering of this reservation. Please try again. Please visit http://datastax.com/ami for this AMI's feature set. / On 7/2/2012 8:43 AM, Deno Vichas wrote: all, my 4 node cluster seems pretty screwed up after the AWS outage. we found all our machines with their cpu stuck at 100%. so i went to restart each cassandra node one by one. i did node with token id 0 first. i came back but doesn't look like it doing anything. once i thought it was up i went and restarted the next. this one got stuck on the AMI init startup. i cancelled it, rebooted againg and now it't suck with [INFO] 07/02/12-15:42:29 Received 0 of 1 responses from: suggestions on how to fix this? thanks, deno
MultigetSliceQuery count 1 returning multiple results
We are using hector.rb to do a MultigetSliceQuery over one row, with count 1. Most of the time it works, but occasionally we are getting back 2 columns. Has anyone seen this before? Cassandra: 1.1.0-SNAPSHOT Hector: 1.0.2 Hector.rb: 0.0.1 Gem Code: https://github.com/jashmenn/hector.rb/blob/master/lib/hector/hector.rb#L126 Example Query: client.get_range(column_family, row_id, row_id, :start = time, :finish = (time - 1.day), :count = 1, :reversed = true) client.get_range(column_family, row_id, row_id, :start = time, :finish = (time + 1.day), :count = 1) Thanks, Owen
Re: Node won't start (faillout from AWS outage) - HELP
Your missing the sliced_buffer_size_in_kb property in your cassandra.yaml file. 2012/7/2 Deno Vichas d...@syncopated.net i'm seeing the following exception in my log now. this is even after re-creating my config file. INFO [main] 2012-07-02 16:58:17,850 AbstractCassandraDaemon.java (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO [main] 2012-07-02 16:58:17,853 AbstractCassandraDaemon.java (line 122) Heap size: 1936719872/1937768448 INFO [main] 2012-07-02 16:58:17,854 AbstractCassandraDaemon.java (line 123) Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share /cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.1.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.1.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar INFO [main] 2012-07-02 16:58:19,309 CLibrary.java (line 111) JNA mlockall successful INFO [main] 2012-07-02 16:58:19,319 DatabaseDescriptor.java (line 127) Loading settings from file:/etc/cassandra/cassandra.yaml ERROR [main] 2012-07-02 16:58:19,517 DatabaseDescriptor.java (line 481) Fatal configuration error error Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create property=sliced_buffer_size_in_kb for JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config in reader, line 4, column 1: cluster_name: 'StockTouchCluster' ^ at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372) at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177) at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:136) at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122) at org.yaml.snakeyaml.Loader.load(Loader.java:52) at org.yaml.snakeyaml.Yaml.load(Yaml.java:166) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:143) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160) Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=sliced_buffer_size_in_kb for JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305) at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184) at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:370) ... 14 more Caused by: org.yaml.snakeyaml.error.YAMLException: Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.getProperty(Constructor.java:342) at
Re: Node won't start (faillout from AWS outage) - HELP
Just the opposite, I think. The property value exists in the yaml file but does not have a corresponding definition in the Config class. Typically caused by a version mismatch in my experience. On Jul 2, 2012, at 1:20 PM, Robin Verlangen wrote: Your missing the sliced_buffer_size_in_kb property in your cassandra.yaml file. 2012/7/2 Deno Vichas d...@syncopated.net i'm seeing the following exception in my log now. this is even after re-creating my config file. INFO [main] 2012-07-02 16:58:17,850 AbstractCassandraDaemon.java (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO [main] 2012-07-02 16:58:17,853 AbstractCassandraDaemon.java (line 122) Heap size: 1936719872/1937768448 INFO [main] 2012-07-02 16:58:17,854 AbstractCassandraDaemon.java (line 123) Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share /cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.1.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.1.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar INFO [main] 2012-07-02 16:58:19,309 CLibrary.java (line 111) JNA mlockall successful INFO [main] 2012-07-02 16:58:19,319 DatabaseDescriptor.java (line 127) Loading settings from file:/etc/cassandra/cassandra.yaml ERROR [main] 2012-07-02 16:58:19,517 DatabaseDescriptor.java (line 481) Fatal configuration error error Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create property=sliced_buffer_size_in_kb for JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config in reader, line 4, column 1: cluster_name: 'StockTouchCluster' ^ at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372) at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177) at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:136) at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122) at org.yaml.snakeyaml.Loader.load(Loader.java:52) at org.yaml.snakeyaml.Yaml.load(Yaml.java:166) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:143) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160) Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=sliced_buffer_size_in_kb for JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305) at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184) at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:370) ... 14 more Caused by:
Re: Node won't start (faillout from AWS outage) - HELP
no, it's there... after editing the first couple lines of the config file there spot where it complains changes. On 7/2/2012 10:20 AM, Robin Verlangen wrote: Your missing the sliced_buffer_size_in_kb property in your cassandra.yaml file. 2012/7/2 Deno Vichas d...@syncopated.net mailto:d...@syncopated.net i'm seeing the following exception in my log now. this is even after re-creating my config file. INFO [main] 2012-07-02 16:58:17,850 AbstractCassandraDaemon.java (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO [main] 2012-07-02 16:58:17,853 AbstractCassandraDaemon.java (line 122) Heap size: 1936719872/1937768448 INFO [main] 2012-07-02 16:58:17,854 AbstractCassandraDaemon.java (line 123) Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share /cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.1.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.1.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar INFO [main] 2012-07-02 16:58:19,309 CLibrary.java (line 111) JNA mlockall successful INFO [main] 2012-07-02 16:58:19,319 DatabaseDescriptor.java (line 127) Loading settings from file:/etc/cassandra/cassandra.yaml ERROR [main] 2012-07-02 16:58:19,517 DatabaseDescriptor.java (line 481) Fatal configuration error error Can't construct a java object for tag:yaml.org http://yaml.org,2002:org.apache.cassandra.config.Config; exception=Cannot create property=sliced_buffer_size_in_kb for JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config in reader, line 4, column 1: cluster_name: 'StockTouchCluster' ^ at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372) at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177) at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:136) at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122) at org.yaml.snakeyaml.Loader.load(Loader.java:52) at org.yaml.snakeyaml.Yaml.load(Yaml.java:166) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:143) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160) Caused by: org.yaml.snakeyaml.error.YAMLException: Cannot create property=sliced_buffer_size_in_kb for JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:305) at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:184) at
Re: cassandra on re-Start
On Mon, Jul 2, 2012 at 5:43 AM, puneet loya puneetl...@gmail.com wrote: When I restarted the system , it is showing the keyspace does not exist. Not even letting me to create the keyspace with the same name again. Paste the error you get. =Rob -- =Robert Coli AIMGTALK - rc...@palominodb.com YAHOO - rcoli.palominob SKYPE - rcoli_palominodb
Re: Node won't start (faillout from AWS outage) - HELP
should be the same version. i did notice some extra files in my etc dir. there's a few dpkg-new files. ubuntu@ip-10-40-207-151:~/datastax_ami$ ll /etc/cassandra/ total 100 drwxr-xr-x 2 root root 4096 2012-07-02 18:13 ./ drwxr-xr-x 99 root root 4096 2012-07-02 16:31 ../ -rw-r--r-- 1 root root 7649 2012-02-21 07:10 cassandra-env.sh -rw-r--r-- 1 root root 7690 2012-06-01 17:54 cassandra-env.sh.dpkg-new -rw-r--r-- 1 root root 18424 2012-07-02 18:11 cassandra.yaml -rw-r--r-- 1 root root 19234 2012-07-02 16:54 cassandra.yaml.bak -rw-r--r-- 1 root root 24911 2012-07-02 17:11 cassandra.yaml.dpkg-new -rw-r--r-- 1 root root 1849 2012-07-02 17:12 log4j-server.properties -rw-r--r-- 1 root root 1850 2012-06-01 17:54 log4j-server.properties.dpkg-new* * On 7/2/2012 10:25 AM, Brian Jeltema wrote: Just the opposite, I think. The property value exists in the yaml file but does not have a corresponding definition in the Config class. Typically caused by a version mismatch in my experience. On Jul 2, 2012, at 1:20 PM, Robin Verlangen wrote: Your missing the sliced_buffer_size_in_kb property in your cassandra.yaml file. 2012/7/2 Deno Vichas d...@syncopated.net mailto:d...@syncopated.net i'm seeing the following exception in my log now. this is even after re-creating my config file. INFO [main] 2012-07-02 16:58:17,850 AbstractCassandraDaemon.java (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO [main] 2012-07-02 16:58:17,853 AbstractCassandraDaemon.java (line 122) Heap size: 1936719872/1937768448 INFO [main] 2012-07-02 16:58:17,854 AbstractCassandraDaemon.java (line 123) Classpath: /usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/avro-1.4.0-fixes.jar:/usr/share/cassandra/lib/avro-1.4.0-sources-fixes.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang-2.4.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/guava-r08.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jline-0.9.94.jar:/usr/share/cassandra/lib/joda-time-1.6.2.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.7.0.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/metrics-core-2.0.3.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.6.1.jar:/usr/share /cassandra/lib/slf4j-log4j12-1.6.1.jar:/usr/share/cassandra/lib/snakeyaml-1.6.jar:/usr/share/cassandra/lib/snappy-java-1.0.4.1.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/apache-cassandra-1.1.1.jar:/usr/share/cassandra/apache-cassandra-thrift-1.1.1.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/stress.jar:/usr/share/java/jna.jar:/etc/cassandra:/usr/share/java/commons-daemon.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar INFO [main] 2012-07-02 16:58:19,309 CLibrary.java (line 111) JNA mlockall successful INFO [main] 2012-07-02 16:58:19,319 DatabaseDescriptor.java (line 127) Loading settings from file:/etc/cassandra/cassandra.yaml ERROR [main] 2012-07-02 16:58:19,517 DatabaseDescriptor.java (line 481) Fatal configuration error error Can't construct a java object for tag:yaml.org http://yaml.org/,2002:org.apache.cassandra.config.Config; exception=Cannot create property=sliced_buffer_size_in_kb for JavaBean=org.apache.cassandra.config.Config@371bed71; Unable to find property 'sliced_buffer_size_in_kb' on class: org.apache.cassandra.config.Config in reader, line 4, column 1: cluster_name: 'StockTouchCluster' ^ at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:372) at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:177) at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:136) at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:122) at org.yaml.snakeyaml.Loader.load(Loader.java:52) at org.yaml.snakeyaml.Yaml.load(Yaml.java:166) at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:143) at org.apache.cassandra.config.DatabaseDescriptor.clinit(DatabaseDescriptor.java:118) at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126) at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
datastax aws ami
all, i've got a datastax 2.1 ami instance that's screwed up. for some reason it won't read the config file. what's the recommended way to replace this node with a new one? it doesn't seem like you can use the ami to bring up single nodes as it want to do whole clusters. thanks, deno
[RELEASE] Apache Cassandra 1.1.2 released
The Cassandra team is pleased to announce the release of Apache Cassandra version 1.1.2. Cassandra is a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. You can read more here: http://cassandra.apache.org/ Downloads of source and binary distributions are listed in our download section: http://cassandra.apache.org/download/ This version is a maintenance/bug fix release[1] on the 1.1 series. As always, please pay attention to the release notes[2] and Let us know[3] if you were to encounter any problem. Enjoy! [1]: http://goo.gl/kt5e9 (CHANGES.txt) [2]: http://goo.gl/nfE2w (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA
Re: datastax aws ami
is the 2.1 image still around? On 7/2/2012 11:24 AM, Deno Vichas wrote: all, i've got a datastax 2.1 ami instance that's screwed up. for some reason it won't read the config file. what's the recommended way to replace this node with a new one? it doesn't seem like you can use the ami to bring up single nodes as it want to do whole clusters. thanks, deno
Thrift version and OOM errors
Hello All,
We are using Cassandra 1.0.7 on AWS on mediums (that is 3.8G RAM, 1 Core),
running Ubuntu 12.04. We have three nodes in the cluster and we hit only
one node from our application. Thrift version is 0.6.1 (we changed from 0.8
because we thought there was a compatibility problem between thrift and
Cassandra ('old client' according to the output.log). We are still not sure
what version of thrift to use with Cassandra 1.0.7 (we are still getting
the same message regarding the 'old client'). I would appreciate any help
on that please.
Below, I am sharing the errors we are getting from the output.log file.
First three errors are not responsible for the crash, only the OOM error
is, but something seems to be really wrong there...
Error #1
ERROR 14:00:12,057 Thrift error occurred during processing of message.
org.apache.thrift.TException: Message length exceeded: 1970238464
at
org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
at
org.apache.thrift.protocol.TBinaryProtocol.readBinary(TBinaryProtocol.java:363)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:102)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:112)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:121)
at org.apache.thrift.protocol.TProtocolUtil.skip(TProtocolUtil.java:60)
at org.apache.cassandra.thrift.Mutation.read(Mutation.java:355)
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18966)
at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Error #2
ERROR 14:03:48,004 Error occurred during processing of message.
java.lang.StringIndexOutOfBoundsException: String index out of range:
-2147418111
at java.lang.String.checkBounds(String.java:397)
at java.lang.String.init(String.java:442)
at
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:339)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:210)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Error #3
ERROR 14:07:24,415 Thrift error occurred during processing of message.
org.apache.thrift.protocol.TProtocolException: Missing version in
readMessageBegin, old client?
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:213)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Error #4
ERROR 16:07:10,168 Thrift error occurred during processing of message.
org.apache.thrift.TException: Message length exceeded: 218104076
at
org.apache.thrift.protocol.TBinaryProtocol.checkReadLength(TBinaryProtocol.java:393)
at
org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:352)
at
org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:347)
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_args.read(Cassandra.java:18958)
at
org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.process(Cassandra.java:3441)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
java.lang.OutOfMemoryError: Java heap space
Dumping heap to /var/lib/cassandra/java_1341224307.hprof ...
INFO 16:07:18,882 GC for Copy: 886 ms for 1 collections, 2242700896 used;
max is 2670985216
Java HotSpot(TM) 64-Bit Server VM warning: record is too large
Heap dump file created [4429997807 bytes in 95.755 secs]
INFO 16:08:54,749 GC for ConcurrentMarkSweep: 1157 ms for 4 collections,
Re: Cassandra and massive TTL expirations cause HEAP issue
After 10 days my cluster crashes due to a java.lang.OutOfMemoryError during compaction of the big column family that contains roughly 95% of the data. Does this column family have very wide rows ? simply some tweaks I need to make in the yaml file. I have tried: The main things that reduce the impact compaction has on memory are: in_memory_compaction_limit_in_mb concurrent_compactors Of the top of my head I cannot think of any shortcuts taken by compaction if/when all data in an SSTable is past TTL. I think there was talk of something like that though. Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/06/2012, at 2:38 AM, Nils Pommerien wrote: Hello, I am evaluating Cassandra in a log retrieval application. My ring conists of3 m2.xlarge instances (17.1 GB memory, 6.5 ECU (2 virtual cores with 3.25 EC2 Compute Units each), 420 GB of local instance storage, 64-bit platform) and I am writing at roughly 220 writes/sec. Per day I am adding roughly 60GB of data. All of this sounds simple and easy and all three nodes are humming along with basically no load. The issue is that I am writing all my data with a TTL of 10 days. After 10 days my cluster crashes due to a java.lang.OutOfMemoryError during compaction of the big column family that contains roughly 95% of the data. So basically after 10 days my data set is 600GB and after 10 days Cassandra would have to tombstone and purge 60GB of data at the same rate of roughly 220 deletes/second. I am not sure if Cassandra should be able to do it, whether I should take a partitioning approach (one CF per day), or if there is simply some tweaks I need to make in the yaml file. I have tried: Decrease flush-largest-memtables-at to .4 reduce_cache_sizes_at and reduce_cache_capacity_to set to 1 Now, the issue remains the same: WARN [ScheduledTasks:1] 2012-06-11 19:39:42,017 GCInspector.java (line 145) Heap is 0.9920103380107628 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically. Eventually it will just die with this message. This affects all nodes in the cluster, not just one. Dump file is incomplete: file size limit ERROR 19:39:39,695 Exception in thread Thread[ReadStage:134,5,main] java.lang.OutOfMemoryError: Java heap space ERROR 19:39:39,724 Exception in thread Thread[MutationStage:57,5,main] java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.utils.FBUtilities.hashToBigInteger(FBUtilities.java:213) at org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java:154) at org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.java:47) at org.apache.cassandra.db.RowPosition.forKey(RowPosition.java:54) Any help is highly appreciated. It would be cool to tweak it in a way that I can have a moving window of 10 days in Cassandra while dropping the old data… Or, if there is any other recommended way to deal with such sliding time windows I am open for ideas. Thank you for your help!
Re: cassandra 1.0.9 error - Read an invalid frame size of 0
Gurpreet , If you can reproduce this please create a ticket on https://issues.apache.org/jira/browse/CASSANDRA Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 19/06/2012, at 6:38 AM, Gurpreet Singh wrote: I found a fix for this one, rather a workaround. I changed the rpc_server_type in cassandra.yaml, from hsha to sync, and the error went away. I guess, there is some issue with the thrift nonblocking server. Thanks Gurpreet On Wed, May 16, 2012 at 7:04 PM, Gurpreet Singh gurpreet.si...@gmail.com wrote: Thanks Aaron. will do! On Mon, May 14, 2012 at 1:14 PM, aaron morton aa...@thelastpickle.com wrote: Are you using framed transport on the client side ? Try the Hector user list for hector specific help https://groups.google.com/forum/?fromgroups#!searchin/hector-users Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 12/05/2012, at 5:44 AM, Gurpreet Singh wrote: This is hampering our testing of cassandra a lot, and our move to cassandra 1.0.9. Has anyone seen this before? Should I be trying a different version of cassandra? /G On Thu, May 10, 2012 at 11:29 PM, Gurpreet Singh gurpreet.si...@gmail.com wrote: Hi, i have created 1 node cluster of cassandra 1.0.9. I am setting this up for testing reads/writes. I am seeing the following error in the server system.log ERROR [Selector-Thread-7] 2012-05-10 22:44:02,607 TNonblockingServer.java (line 467) Read an invalid frame size of 0. Are you using TFramedTransport on the client side? Initially i was using a old hector 0.7.x, but even after switching to hector 1.0-5 and thrift version 0.6.1, i still see this error. I am using 20 threads writing/reading from cassandra. The max write batch size is 10 with payload size constant per key to be 600 bytes. On the client side, i see Hector exceptions happenning coinciding with these messages on the server. Any ideas why these errors are happenning? Thanks Gurpreet
Re: repair never finishing 1.0.7
The nodes in DC1 need to be able to reach the nodes in DC2 on the public (NAT'd) IP. Others may be able to provide some more details . Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/06/2012, at 9:51 PM, Andras Szerdahelyi wrote: Aaron, The broadcast_address allows a node to broadcast an address that is different to the ones it's bound to on the local interfaces https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L270 Yes and thats not where the problem is IMO.. If you broadcast your translated address ( say 1.2.3.4, a public ip ) , nodes outside your VPN'd network will have no problems connecting as long as they can route to this address ( which they should ), but any other nodes on the local net ( e.g. 10.0.1.2 ) won't be able to connect/route to their neighbor who's telling them to open the return socket to 1.2.3.4 Am i getting this right? At least this is what i have experienced not so long ago: DC1 nodes a) 10.0.1.1 translated to 1.2.3.4 on NAT b) 10.0.1.2 translated to 1.2.3.5 on NAT DC2 nodes a) 10.0.2.1 translated to 1.2.4.4 on NAT b) 10.0.2.2 translated to 1.2.4.5 on NAT Let's assume DC2 nodes' broadcast_addresses are their public addresses. if, DC1:a and DC1:b broadcast their public address, 1.2.3.4 and 1.2.3.5, they are advertising an address that is not routable on their network ( loopback ) but DC2:a and DC2:b can connect/route to them just fine. Nodetool ring on any DC1 node says the others in DC1 are down, everything else is up . Nodetool ring on any DC2 node says everything is up. if DC1:a and DC1:b broadcast their private address, they can connect to each other fine, but DC2:a and DC2:b will have no chance to route to them. Nodetool ring on any DC1 node says everything is up. Nodetool ring on any DC2 node says DC1 nodes are down. regards, Andras On 27 Jun 2012, at 11:29, aaron morton wrote: Setting up a Cassandra ring across NAT ( without a VPN ) is impossible in my experience. The broadcast_address allows a node to broadcast an address that is different to the ones it's bound to on the local interfaces https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L270 1) How can I make sure that the JIRA issue above is my real problem? (I see no errors or warns in the logs; no other activity) If the errors are not there it is not your problem. - a full cluster restart allows the first attempted repair to complete (haven't tested yet; this is not practical even if it works) Rolling restart of the nodes involved in the repair is sufficient. Double checking the networking and check the logs on both sides of the transfer for errors or warnings. The code around streaming is better at failing loudly now days. If you dont see anything set DEBUG logging on org.apache.cassandra.streaming.FileStreamTask. That will let you know if things start and progress. Hope that helps. - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 26/06/2012, at 6:16 PM, Alexandru Sicoe wrote: Hi Andras, I am not using a VPN. The system has been running successfully in this configuration for a couple of weeks until I noticed the repair is not working. What happens is that I configure the IP Tables of the machine on each Cassandra node to forward packets that are sent to any of the IPs in the other DC (on ports 7000, 9160 and 7199) to be sent to the gateway IP. The gateway does the NAT sending the packets on the other side to the real destination IP, having replaced the source IP with the initial sender's IP (at least in my understanding of it). What might be the problem given the configuration? How to fix this? Cheers, Alex On Mon, Jun 25, 2012 at 12:47 PM, Andras Szerdahelyi andras.szerdahe...@ignitionone.com wrote: The DCs are communicating over a gateway where I do NAT for ports 7000, 9160 and 7199. Ah, that sounds familiar. You don't mention if you are VPN'd or not. I'll assume you are not. So, your nodes are behind network address translation - is that to say they advertise ( broadcast ) their internal or translated/forwarded IP to each other? Setting up a Cassandra ring across NAT ( without a VPN ) is impossible in my experience. Either the nodes on your local network won't be able to communicate with each other, because they broadcast their translated ( public ) address which is normally ( router configuration ) not routable from within the local network, or the nodes broadcast their internal IP, in which case the outside nodes are helpless in trying to connect to a local net. On DC2 nodes/the node you issue the repair on, check for any sockets being opened to the internal addresses of the nodes in DC1. regards, Andras On 25 Jun 2012, at 11:57, Alexandru Sicoe
Re: CQL / ASSUME for keys
I'm trying to do the following : update keyspace.CF set '2' = '2' + 12 WHERE KEY = 'mykey'; CQL does not support expressions in the SET clause. http://www.datastax.com/docs/1.0/references/cql/UPDATE Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 28/06/2012, at 2:08 AM, Alain RODRIGUEZ wrote: Hi, I'm trying to do the following : update keyspace.CF set '2' = '2' + 12 WHERE KEY = 'mykey'; And got this answer: Bad Request: cannot parse 'mykey' as hex bytes Using this doesn't help: assume keyspace.CF(KEY) VALUES ARE text; (Found here http://www.datastax.com/docs/1.0/references/cql/ASSUME and I'm using C* 1.0.9) Show schema in cli gives : create column family CF with column_type = 'Standard' and comparator = 'UTF8Type' and default_validation_class = 'CounterColumnType' and key_validation_class = 'BytesType' and rows_cached = 0.0 and row_cache_save_period = 0 and row_cache_keys_to_save = 2147483647 and keys_cached = 2000.0 and key_cache_save_period = 14400 and read_repair_chance = 1.0 and gc_grace = 864000 and min_compaction_threshold = 4 and max_compaction_threshold = 32 and replicate_on_write = true and row_cache_provider = 'ConcurrentLinkedHashCacheProvider' and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'; What would be the consequences of changing the key_validation_class from 'BytesType' to 'UTF8Type' (in production)? Shouldn't my assume command allow me to update my data even if I don't give the key as Bytes ? Alain
Re: Enable CQL3 from Astyanax
Thanks for contributing. I'm behind the curve on CQL 3, but here is a post about some of the changes http://www.datastax.com/dev/blog/whats-new-in-cql-3-0 Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 28/06/2012, at 2:30 AM, Thierry Templier wrote: Hello Aaron, I created an issue on the Astyanax github for this problem. I added a fix to support CQL3 in the tool. See the link https://github.com/Netflix/astyanax/issues/75. Thierry Had a quick look, the current master does not appear to support it. Cheers
Re: Node crashing during read repair
Is this still an issue ? It looks like something shut down the messaging service. Was there anything else in the logs ? Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 28/06/2012, at 3:49 AM, Robin Verlangen wrote: Hi there, Today I found one node (running 1.1.1 in a 3 node cluster) being dead for the third time this week, it died with the following message: ERROR [ReadRepairStage:3] 2012-06-27 14:28:30,929 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[ReadRepairStage:3,5,main] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60) at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:566) at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:439) at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:391) at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:372) at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:460) at org.apache.cassandra.service.RowRepairResolver.scheduleRepairs(RowRepairResolver.java:136) at org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:94) at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Is this a common bug in 1.1.1, or did I get a race condition? -- With kind regards, Robin Verlangen Software engineer W http://www.robinverlangen.nl E ro...@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Re: bulk load glitch
On Mon, Jul 2, 2012 at 10:35 AM, Brian Jeltema brian.jelt...@digitalenvoy.net wrote: I can't tell whether the bulk load process recovered from the transient dead node, or whether I need to start over. Does anybody know? You need to start over if the failure detector tripped, but it will retry a few times for regular network errors. -Brandon
Re: frequent node up/downs
Yeah I noticed the leap second problem and ran the suggested fix, but I have been facing these problems before Saturday and still see the occasional failures after running the fix. Thanks. On Mon, Jul 2, 2012 at 11:17 AM, Marcus Both mb...@terra.com.br wrote: Yeah! Look that. http://arstechnica.com/business/2012/07/one-day-later-the-leap-second-v-the-internet-scorecard/ I had the same problem. The solution was rebooting. On Mon, 2 Jul 2012 11:08:57 -0400 feedly team feedly...@gmail.com wrote: Hello, I recently set up a 2 node cassandra cluster on dedicated hardware. In the logs there have been a lot of InetAddress xxx is now dead' or UP messages. Comparing the log messages between the 2 nodes, they seem to coincide with extremely long ParNew collections. I have seem some of up to 50 seconds. The installation is pretty vanilla, I didn't change any settings and the machines don't seem particularly busy - cassandra is the only thing running on the machine with an 8GB heap. The machine has 64GB of RAM and CPU/IO usage looks pretty light. I do see a lot of 'Heap is xxx full. You may need to reduce memtable and/or cache sizes' messages. Would this help with the long ParNew collections? That message seems to be triggered on a full collection. -- Marcus Both
Re: faillout from AWS outage - HELP
yes, you where correct. resetting the flag and bouncing cassandra fixed it. thx On 7/2/2012 8:59 AM, Joost van de Wijgerd wrote: Could be related to the leap second bug.. try this: Bug: https://lkml.org/lkml/2012/6/30/122 Simple fix to reset the leap second flag: date; date `date +%m%d%H%M%C%y.%S`; date; mvg Joost On Mon, Jul 2, 2012 at 5:43 PM, Deno Vichas d...@syncopated.net wrote: all, my 4 node cluster seems pretty screwed up after the AWS outage. we found all our machines with their cpu stuck at 100%. so i went to restart each cassandra node one by one. i did node with token id 0 first. i came back but doesn't look like it doing anything. once i thought it was up i went and restarted the next. this one got stuck on the AMI init startup. i cancelled it, rebooted againg and now it't suck with [INFO] 07/02/12-15:42:29 Received 0 of 1 responses from: suggestions on how to fix this? thanks, deno
Re: hector timeouts
is anybody using kafka? what other options is there? currently i need to do around 50,000 (is that a lot?) a minute. On 7/1/2012 11:39 AM, aaron morton wrote: Using Cassandra as a queue is generally thought of as a bas idea, owing to the high delete workload. Levelled compaction handles it better but it is still no the best approach. Depending on your needs consider running http://incubator.apache.org/kafka/ could you share some details on this? we're using hector and we see random timeout warns in the logs and not sure how to address them. First determine if they are server side or client side timeouts. Then determine what the query was. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 29/06/2012, at 7:02 AM, Deno Vichas wrote: On 6/28/2012 9:37 AM, David Leimbach wrote: That coupled with Hector timeout issues became a real problem for us. could you share some details on this? we're using hector and we see random timeout warns in the logs and not sure how to address them. thanks, deno
Re: hector timeouts
lots of folks use Apache Kafka, check out https://cwiki.apache.org/confluence/display/KAFKA/Powered+By just to name a few you can read about the performance for yourself http://incubator.apache.org/kafka/performance.html @ http://www.medialets.com we use Kafka upstream of Cassandra acting like a queue so our workers can do their business logic prior to storing their results in Cassandra, Hadoop MySQL this decouples our backend analytics from our forwarding facing system keeping our forward facing system (ad serving to mobile devices) as fast as possible and our backend results near realtime (seconds from data coming in) here are some papers and presentations https://cwiki.apache.org/confluence/display/KAFKA/Kafka+papers+and+presentations On Mon, Jul 2, 2012 at 10:09 PM, Deno Vichas d...@syncopated.net wrote: is anybody using kafka? what other options is there? currently i need to do around 50,000 (is that a lot?) a minute. On 7/1/2012 11:39 AM, aaron morton wrote: Using Cassandra as a queue is generally thought of as a bas idea, owing to the high delete workload. Levelled compaction handles it better but it is still no the best approach. Depending on your needs consider running http://incubator.apache.org/kafka/ could you share some details on this? we're using hector and we see random timeout warns in the logs and not sure how to address them. First determine if they are server side or client side timeouts. Then determine what the query was. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 29/06/2012, at 7:02 AM, Deno Vichas wrote: On 6/28/2012 9:37 AM, David Leimbach wrote: That coupled with Hector timeout issues became a real problem for us. could you share some details on this? we're using hector and we see random timeout warns in the logs and not sure how to address them. thanks, deno -- /* Joe Stein http://www.linkedin.com/in/charmalloc Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop */
Re: frequent node up/downs
Couple more details. I confirmed that swap space is not being used (free -m shows 0 swap) and cassandra.log has a message like JNA mlockall successful. top shows the process having 9g in resident memory but 21.6g in virtual...What accounts for the much larger virtual number? some kind of off-heap memory? I'm a little puzzled as to why I would get such long pauses without swapping. I uncommented all the gc logging options in cassandra-env.sh to try to see what is going on when the node freezes. Thanks Kireet On Mon, Jul 2, 2012 at 9:51 PM, feedly team feedly...@gmail.com wrote: Yeah I noticed the leap second problem and ran the suggested fix, but I have been facing these problems before Saturday and still see the occasional failures after running the fix. Thanks. On Mon, Jul 2, 2012 at 11:17 AM, Marcus Both mb...@terra.com.br wrote: Yeah! Look that. http://arstechnica.com/business/2012/07/one-day-later-the-leap-second-v-the-internet-scorecard/ I had the same problem. The solution was rebooting. On Mon, 2 Jul 2012 11:08:57 -0400 feedly team feedly...@gmail.com wrote: Hello, I recently set up a 2 node cassandra cluster on dedicated hardware. In the logs there have been a lot of InetAddress xxx is now dead' or UP messages. Comparing the log messages between the 2 nodes, they seem to coincide with extremely long ParNew collections. I have seem some of up to 50 seconds. The installation is pretty vanilla, I didn't change any settings and the machines don't seem particularly busy - cassandra is the only thing running on the machine with an 8GB heap. The machine has 64GB of RAM and CPU/IO usage looks pretty light. I do see a lot of 'Heap is xxx full. You may need to reduce memtable and/or cache sizes' messages. Would this help with the long ParNew collections? That message seems to be triggered on a full collection. -- Marcus Both
