Re: Creating counter columns in cassandra
Hi all,
Thanks all for the valuable feedback. I have a problem with running queries
with Cqlsh.
My query is SELECT * FROM rule1 WHERE sms=3;
java.lang.NumberFormatException: An hex string representing bytes must have
an even length
at org.apache.cassandra.utils.Hex.hexToBytes(Hex.java:52)
at
org.apache.cassandra.utils.ByteBufferUtil.hexToBytes(ByteBufferUtil.java:501)
at
org.apache.cassandra.db.marshal.CounterColumnType.fromString(CounterColumnType.java:57)
at org.apache.cassandra.cql.Term.getByteBuffer(Term.java:96)
at
org.apache.cassandra.cql.QueryProcessor.multiRangeSlice(QueryProcessor.java:185)
at
org.apache.cassandra.cql.QueryProcessor.processStatement(QueryProcessor.java:484)
at org.apache.cassandra.cql.QueryProcessor.process(QueryProcessor.java:877)
at
org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1235)
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3542)
at
org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3530)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
but when I say SELECT * FROM rule1 WHERE sms=03; no exceptions are shown.
But though I have entries where sms count =3 that entry is not retrieved.
And for queries like SELECT * FROM rule1 WHERE sms=03;
Bad Request: No indexed columns present in by-columns clause with equals
operator
Can anyone recognize the problem here??
Following are the methods I used.
//for indexing columns
void indexColumn(String idxColumnName,String CountercfName){
Cluster cluster = HFactory.getOrCreateCluster(
BasicConf.CASSANDRA_CLUSTER, BasicConf.CLUSTER_PORT);
KeyspaceDefinition keyspaceDefinition =
cluster.describeKeyspace(BasicConf.KEYSPACE);
ListColumnFamilyDefinition cdfs = keyspaceDefinition.getCfDefs();
ColumnFamilyDefinition cfd = null;
for(ColumnFamilyDefinition c:cdfs){
if(c.getName().toString().equals(CountercfName)) {
System.out.println(c.getName());
cfd=c;
break;
}
}
BasicColumnFamilyDefinition columnFamilyDefinition = new
BasicColumnFamilyDefinition(cfd);
BasicColumnDefinition bcdf = new BasicColumnDefinition();
bcdf.setName(StringSerializer.get().toByteBuffer(idxColumnName));
bcdf.setIndexName(idxColumnName+index);
bcdf.setIndexType(ColumnIndexType.KEYS);
bcdf.setValidationClass(ComparatorType.COUNTERTYPE.getClassName());
columnFamilyDefinition.addColumnDefinition(bcdf);
cluster.updateColumnFamily(new ThriftCfDef(columnFamilyDefinition));
}
// for adding a new counter column
void insertCounterColumn(String cfName, String counterColumnName,
String phoneNumberKey) {
MutatorString mutator = HFactory.createMutator(keyspace,
StringSerializer.get());
mutator.insertCounter(phoneNumberKey, cfName, HFactory
.createCounterColumn(counterColumnName, 1L,
StringSerializer.get()));
mutator.execute();
CounterQueryString, String counter = new ThriftCounterColumnQueryString,
String(
keyspace, StringSerializer.get(), StringSerializer.get());
counter.setColumnFamily(cfName).setKey(phoneNumberKey)
.setName(counterColumnName);
indexColumn(columnName, cfName);
}
// incrementing counter values
void incrementCounter(String ruleName, String columnName,
HashMapString, Long entries) {
MutatorString mutator = HFactory.createMutator(keyspace,
StringSerializer.get());
SetString keys = entries.keySet();
for (String s : keys) {
mutator.incrementCounter(s, ruleName, columnName, entries.get(s));
}
mutator.execute();
}
On Sun, Jul 29, 2012 at 3:29 PM, Paolo Bernardi berna...@gmail.com wrote:
On Sun, Jul 29, 2012 at 9:30 AM, Abhijit Chanda
abhijit.chan...@gmail.com wrote:
There should be at least one = (equals) in the WHERE case on key or
secondary index column, this is the Cassandra limitation.
Yep, it's still there (see validateFilterClauses from line 531):
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/thrift/ThriftValidation.java
It's not probably going to change anytime soon (or later, probably ;-)
), secondary indexes are still CFs underneath:
http://grokbase.com/t/cassandra/user/1099pty59r/indexoperator-lt-lte-gt-gte-implementation#20100910dx1fwm8z9ek1cvec10jvmpzfa4
From that message you also get an hint on how to proceed in this situation:
1) Add a fake equality condition to the query;
2) Go for a range scan, which is more efficient than the hack above.
Depending on the speed than you need on the writing side compared to
[RELEASE] Apache Cassandra 1.0.11 released
The Cassandra team is pleased to announce the release of Apache Cassandra version 1.0.11. Cassandra is a highly scalable second-generation distributed database, bringing together Dynamo's fully distributed design and Bigtable's ColumnFamily-based data model. You can read more here: http://cassandra.apache.org/ Downloads of source and binary distributions are listed in our download section: http://cassandra.apache.org/download/ This version is maintenance/bug fix release[1]. As always, please pay attention to the release notes[2] and Let us know[3] if you were to encounter any problem. Have fun! [1]: http://goo.gl/KLR6t (CHANGES.txt) [2]: http://goo.gl/5lYPe (NEWS.txt) [3]: https://issues.apache.org/jira/browse/CASSANDRA
Read latency skyrockets after 95-98th percentile
hey list, i've been trying to understand why we are seeing rather nasty read latency peaks in as much as 2% of our total read ( not sure what the underlying thrift call is, should be get_slice ) requests that we have been **unable to tie to garbage collection or blocking I/O.** This is what i mean by nasty : Mean: 1.186341, Max: 41.912 stDev: 2.2490502737495333, 75th:1.02424999, 95th: 2.11034977, 98th: 5.21026034, 99th: 13.248351, 999th: 41.8945400041 This is measured around the Astyanax / Hector wrappers for running column slice queries and iterating over their results. FWIW they perform very similarly. Now, the mean is awesome - 1.2ms for a read is really really fast, i think thrift overhead makes up for probably 1ms alone. What i can't live with is pretty much everything thats beyond the 98th percentile. Perhaps my expectations are misaligned to begin with and this is great, but the dataset under test that yielded this result ( see below ) is extremely small - only a 1000 items per test made up of a few small dynamic composite columns and the read is being done sequentially, from a single threaded client. I just don't understand whats causing the peaks. Here's the complete run-down on the setup: The data is dynamic composite columns with string or integer values. The DynamicComposites themselves are made up of 3 Components each: String,String,Integer and String,Integer,Integer The query is getting all columns ( in this particular test, that will be a single column ) of one of the composite types ( String, String, Integer or String, Integer, Integer ) for a particular string key. ( How this is achieved is more or less at https://gist.github.com/3207894 ) The production cluster is set up as follows: 30-40 read/sec clients from multiple hosts, NetworkTopologyStrategy, 2 replicas per DC, read consistency: ONE, 2DCs, 2 nodes per DC ) 8GB multi-core physical hosts most settings on default in cassandra.yaml In an attempt to isolate the problem, i've done the following ( this is where the results above are coming from but we see roughly the same numbers in prod ) - created a short unit test around the ( two ) classes that wrap the Astyanax and Hector libraries in our application, in which i'm only testing the read performance - in setUp() i start Cassandra with CassandraStoreKeeperTest.cassandra = new CassandraDaemon(); CassandraStoreKeeperTest.cassandra.init(null); CassandraStoreKeeperTest.cassandra.start(); - same cassandra yaml as in prod, file at https://gist.github.com/3207894 ( mostly defaults ) - i pass the following flags to the JVM running my unit test: -server -Xmx2G and all the GC flags from cassandra-env.sh, plus the jamm java agent - i warm up the embedded cassandra with 1k reads and 1k writes - the test is 1000 single threaded, sequential reads via Hector and Astyanax ( no significant differences between the two ) via thrift to the embedded single-node Cassandra instance - i read back the 1000 items saved in the warm-up The results are Mean: 1.186341, Max: 41.912 stDev: 2.2490502737495333, 75th:1.02424999, 95th: 2.11034977, 98th: 5.21026034, 99th: 13.248351, 999th: 41.8945400041 here's the GC output during the read [GC [ParNew Desired survivor size 1081344 bytes, new threshold 1 (max 1) - age 1: 281480 bytes, 281480 total : 17322K-308K(19136K), 0.0037375 secs] 28742K-11885K(83008K), 0.0038170 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] [GC [ParNew Desired survivor size 1081344 bytes, new threshold 1 (max 1) - age 1: 261360 bytes, 261360 total : 17332K-386K(19136K), 0.0034069 secs] 28909K-12098K(83008K), 0.0034849 secs] [Times: user=0.00 sys=0.01, real=0.00 secs] [GC [ParNew Desired survivor size 1081344 bytes, new threshold 1 (max 1) - age 1: 241336 bytes, 241336 total : 17410K-383K(19136K), 0.0035252 secs] 29122K-12209K(83008K), 0.0036062 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] Doesn't look like GC ( i could be wrong ) - what else could it be ? all suggestions are highly appreciated.. Thanks! Andras ps: this is with Cassandra 1.1.0
RE: Read latency skyrockets after 95-98th percentile
What is a data load? Does it fits in RAM? I bet it's due to GC. Since this is a 1 node only, dynamic snitch is out of scope. Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsiderhttp://twitter.com/#!/adforminsider What is Adform: watch this short videohttp://vimeo.com/adform/display [Adform News] http://www.adform.com Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Andras Szerdahelyi [mailto:andras.szerdahe...@ignitionone.com] Sent: Tuesday, July 31, 2012 15:53 To: user@cassandra.apache.org Subject: Read latency skyrockets after 95-98th percentile hey list, i've been trying to understand why we are seeing rather nasty read latency peaks in as much as 2% of our total read ( not sure what the underlying thrift call is, should be get_slice ) requests that we have been **unable to tie to garbage collection or blocking I/O.** This is what i mean by nasty : Mean: 1.186341, Max: 41.912 stDev: 2.2490502737495333, 75th:1.02424999, 95th: 2.11034977, 98th: 5.21026034, 99th: 13.248351, 999th: 41.8945400041 This is measured around the Astyanax / Hector wrappers for running column slice queries and iterating over their results. FWIW they perform very similarly. Now, the mean is awesome - 1.2ms for a read is really really fast, i think thrift overhead makes up for probably 1ms alone. What i can't live with is pretty much everything thats beyond the 98th percentile. Perhaps my expectations are misaligned to begin with and this is great, but the dataset under test that yielded this result ( see below ) is extremely small - only a 1000 items per test made up of a few small dynamic composite columns and the read is being done sequentially, from a single threaded client. I just don't understand whats causing the peaks. Here's the complete run-down on the setup: The data is dynamic composite columns with string or integer values. The DynamicComposites themselves are made up of 3 Components each: String,String,Integer and String,Integer,Integer The query is getting all columns ( in this particular test, that will be a single column ) of one of the composite types ( String, String, Integer or String, Integer, Integer ) for a particular string key. ( How this is achieved is more or less at https://gist.github.com/3207894 ) The production cluster is set up as follows: 30-40 read/sec clients from multiple hosts, NetworkTopologyStrategy, 2 replicas per DC, read consistency: ONE, 2DCs, 2 nodes per DC ) 8GB multi-core physical hosts most settings on default in cassandra.yaml In an attempt to isolate the problem, i've done the following ( this is where the results above are coming from but we see roughly the same numbers in prod ) - created a short unit test around the ( two ) classes that wrap the Astyanax and Hector libraries in our application, in which i'm only testing the read performance - in setUp() i start Cassandra with CassandraStoreKeeperTest.cassandra = new CassandraDaemon(); CassandraStoreKeeperTest.cassandra.init(null); CassandraStoreKeeperTest.cassandra.start(); - same cassandra yaml as in prod, file at https://gist.github.com/3207894 ( mostly defaults ) - i pass the following flags to the JVM running my unit test: -server -Xmx2G and all the GC flags from cassandra-env.sh, plus the jamm java agent - i warm up the embedded cassandra with 1k reads and 1k writes - the test is 1000 single threaded, sequential reads via Hector and Astyanax ( no significant differences between the two ) via thrift to the embedded single-node Cassandra instance - i read back the 1000 items saved in the warm-up The results are Mean: 1.186341, Max: 41.912 stDev: 2.2490502737495333, 75th:1.02424999, 95th: 2.11034977, 98th: 5.21026034, 99th: 13.248351, 999th: 41.8945400041 here's the GC output during the read [GC [ParNew Desired survivor size 1081344 bytes, new threshold 1 (max 1) - age 1: 281480 bytes, 281480 total : 17322K-308K(19136K), 0.0037375 secs] 28742K-11885K(83008K), 0.0038170 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] [GC [ParNew Desired survivor size 1081344 bytes, new threshold 1 (max 1) - age 1: 261360 bytes, 261360 total : 17332K-386K(19136K), 0.0034069 secs] 28909K-12098K(83008K), 0.0034849 secs] [Times: user=0.00 sys=0.01, real=0.00
Re: Read latency skyrockets after 95-98th percentile
Victor, the data is a single composite column String, String, Integer with a string value. The key is a UTF8 string I can reproduce this exact behavior with only 100 of these items on a 2GB heap. Also, please see -XX:+PrintGCDetails output below, i'm not convinced it's GC. With a 100 items no garbage collection is logged at all. here's the GC output during the read [GC [ParNew Desired survivor size 1081344 bytes, new threshold 1 (max 1) - age 1: 281480 bytes, 281480 total : 17322K-308K(19136K), 0.0037375 secs] 28742K-11885K(83008K), 0.0038170 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] [GC [ParNew Desired survivor size 1081344 bytes, new threshold 1 (max 1) - age 1: 261360 bytes, 261360 total : 17332K-386K(19136K), 0.0034069 secs] 28909K-12098K(83008K), 0.0034849 secs] [Times: user=0.00 sys=0.01, real=0.00 secs] [GC [ParNew Desired survivor size 1081344 bytes, new threshold 1 (max 1) - age 1: 241336 bytes, 241336 total : 17410K-383K(19136K), 0.0035252 secs] 29122K-12209K(83008K), 0.0036062 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] thanks, Andras On 31 Jul 2012, at 16:04, Viktor Jevdokimov wrote: What is a data load? Does it fits in RAM? I bet it’s due to GC. Since this is a 1 node only, dynamic snitch is out of scope. Best regards / Pagarbiai Viktor Jevdokimov Senior Developer Email: viktor.jevdoki...@adform.commailto:viktor.jevdoki...@adform.com Phone: +370 5 212 3063, Fax +370 5 261 0453 J. Jasinskio 16C, LT-01112 Vilnius, Lithuania Follow us on Twitter: @adforminsiderhttp://twitter.com/#!/adforminsider What is Adform: watch this short videohttp://vimeo.com/adform/display signature-logo29.pnghttp://www.adform.com Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies. From: Andras Szerdahelyi [mailto:andras.szerdahe...@ignitionone.com] Sent: Tuesday, July 31, 2012 15:53 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Read latency skyrockets after 95-98th percentile hey list, i've been trying to understand why we are seeing rather nasty read latency peaks in as much as 2% of our total read ( not sure what the underlying thrift call is, should be get_slice ) requests that we have been **unable to tie to garbage collection or blocking I/O.** This is what i mean by nasty : Mean: 1.186341, Max: 41.912 stDev: 2.2490502737495333, 75th:1.02424999, 95th: 2.11034977, 98th: 5.21026034, 99th: 13.248351, 999th: 41.8945400041 This is measured around the Astyanax / Hector wrappers for running column slice queries and iterating over their results. FWIW they perform very similarly. Now, the mean is awesome - 1.2ms for a read is really really fast, i think thrift overhead makes up for probably 1ms alone. What i can't live with is pretty much everything thats beyond the 98th percentile. Perhaps my expectations are misaligned to begin with and this is great, but the dataset under test that yielded this result ( see below ) is extremely small - only a 1000 items per test made up of a few small dynamic composite columns and the read is being done sequentially, from a single threaded client. I just don't understand whats causing the peaks. Here's the complete run-down on the setup: The data is dynamic composite columns with string or integer values. The DynamicComposites themselves are made up of 3 Components each: String,String,Integer and String,Integer,Integer The query is getting all columns ( in this particular test, that will be a single column ) of one of the composite types ( String, String, Integer or String, Integer, Integer ) for a particular string key. ( How this is achieved is more or less at https://gist.github.com/3207894 ) The production cluster is set up as follows: 30-40 read/sec clients from multiple hosts, NetworkTopologyStrategy, 2 replicas per DC, read consistency: ONE, 2DCs, 2 nodes per DC ) 8GB multi-core physical hosts most settings on default in cassandra.yaml In an attempt to isolate the problem, i've done the following ( this is where the results above are coming from but we see roughly the same numbers in prod ) - created a short unit test around the ( two ) classes that wrap the Astyanax and Hector libraries in our application, in which i'm only testing the read performance - in setUp() i start Cassandra with CassandraStoreKeeperTest.cassandra = new CassandraDaemon(); CassandraStoreKeeperTest.cassandra.init(null); CassandraStoreKeeperTest.cassandra.start(); - same cassandra yaml as in prod, file at
Re: unsubscribe
On Mon, Jul 30, 2012 at 8:31 PM, Alvin UW alvi...@gmail.com wrote: thx http://wiki.apache.org/cassandra/FAQ#unsubscribe You are welcome. -- Eric Evans Acunu | http://www.acunu.com | @acunu
Cannot create CF in Cassandra 1.1.2
My cassandra used to work fine, now it does not create a table. No error message, but the table is not there. I execute create table statement many times, the same thing. When I do it, I see in the system.log the the lines below. What can be the problem? Pre-history: before I noticed it is broken, I went through some issues related to the lack of space, so I was adding more storage and restarting both servers (I have my thing partitioned between 2 nodes). Is there any workaround to bring it back to normal, besides cleaning it all up and starting from scratch? = INFO [MigrationStage:1] 2012-07-31 13:51:20,090 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columnfamilies@766316124(1520/1900 serialized/live bytes, 20 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,091 Memtable.java (line 266) Writing Memtable-schema_columnfamilies@766316124(1520/1900 serialized/live bytes, 20 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,155 Memtable.java (line 307) Completed flushing /Data_Store1/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hd-54-Data.db (1575 bytes) for commitlog position ReplayPosition(segmentId=3346642617210212, position=12976) INFO [MigrationStage:1] 2012-07-31 13:51:20,156 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columns@2106973490(2048/2560 serialized/live bytes, 30 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,157 Memtable.java (line 266) Writing Memtable-schema_columns@2106973490(2048/2560 serialized/live bytes, 30 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,211 Memtable.java (line 307) Completed flushing /Data_Store1/cassandra/data/system/schema_columns/system-schema_columns-hd-54-Data.db (2111 bytes) for commitlog position ReplayPosition(segmentId=3346642617210212, position=12976) This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (Moon Capital). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.
How to create a COLUMNFAMILY with Leveled Compaction?
Sorry for my stupid simple question. How to create a COLUMNFAMILY with Leveled Compaction? There is no example in documentation: http://www.datastax.com/docs/1.1/configuration/storage_configuration#compaction-strategy I try it on Cassandra 1.1.0 and 1.1.2, both failed. The COLUMNFAMILY is still 'SizeTieredCompactionStrategy'. :( Here is my test and output: @host01:/usr/share/cassandracqlsh host01 --cql3 Connected to BookCluster at host01:9160. [cqlsh 2.2.0 | Cassandra 1.1.0 | CQL spec 3.0.0 | Thrift protocol 19.30.0] Use HELP for help. cqlsh cqlsh use demo; cqlsh:demo CREATE COLUMNFAMILY book ... (isbn varchar, ... book_id bigint, ... price int, ... obj varchar, ... PRIMARY KEY (isbn, book_id) ... ) ... WITH compaction_strategy_class='LeveledCompactionStrategy'; cqlsh:demo cqlsh:demo describe COLUMNFAMILY book; CREATE COLUMNFAMILY book ( isbn text PRIMARY KEY ) WITH comment='' AND comparator='CompositeType(org.apache.cassandra.db.marshal.LongType,org.apache.cassandra.db.marshal.UTF8Type)' AND read_repair_chance=0.10 AND gc_grace_seconds=864000 AND default_validation=text AND min_compaction_threshold=4 AND max_compaction_threshold=32 AND replicate_on_write=True AND compaction_strategy_class='SizeTieredCompactionStrategy' AND compression_parameters:sstable_compression='org.apache.cassandra.io.compress.SnappyCompressor'; cqlsh:demo Thanks, Charlie (@mujiang) 一个 木匠 === Data Architect Developer http://mujiang.blogspot.com
RE: Cannot create CF in Cassandra 1.1.2
No, I am using cql 3, but it should not matter. The point is, it used to work, and I am sure that if I just clean up the whole thing and start from scratch, it will work again. Something got corrupted. But I will give a shot to the cli, just to rule out any cql 3 issues. -Original Message- From: rohit bhatia [mailto:rohit2...@gmail.com] Sent: Tuesday, July 31, 2012 2:40 PM To: user@cassandra.apache.org Subject: Re: Cannot create CF in Cassandra 1.1.2 I believe u r using cassandra-cli. Please use help create for the proper syntax.. See http://wiki.apache.org/cassandra/DataModel/ for cassandra datamodel. Also, the flushing events are expected. For example run create keyspace name1 , use name1 , create column family name2 This should work with default settings.. On Tue, Jul 31, 2012 at 11:34 PM, Leonid Ilyevsky lilyev...@mooncapital.com wrote: My cassandra used to work fine, now it does not create a table. No error message, but the table is not there. I execute create table statement many times, the same thing. When I do it, I see in the system.log the the lines below. What can be the problem? Pre-history: before I noticed it is broken, I went through some issues related to the lack of space, so I was adding more storage and restarting both servers (I have my thing partitioned between 2 nodes). Is there any workaround to bring it back to normal, besides cleaning it all up and starting from scratch? = INFO [MigrationStage:1] 2012-07-31 13:51:20,090 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columnfamilies@766316124(1520/1900 serialized/live bytes, 20 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,091 Memtable.java (line 266) Writing Memtable-schema_columnfamilies@766316124(1520/1900 serialized/live bytes, 20 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,155 Memtable.java (line 307) Completed flushing /Data_Store1/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hd-54-Data.db (1575 bytes) for commitlog position ReplayPosition(segmentId=3346642617210212, position=12976) INFO [MigrationStage:1] 2012-07-31 13:51:20,156 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columns@2106973490(2048/2560 serialized/live bytes, 30 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,157 Memtable.java (line 266) Writing Memtable-schema_columns@2106973490(2048/2560 serialized/live bytes, 30 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,211 Memtable.java (line 307) Completed flushing /Data_Store1/cassandra/data/system/schema_columns/system-schema_columns-hd-54-Data.db (2111 bytes) for commitlog position ReplayPosition(segmentId=3346642617210212, position=12976) This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (Moon Capital). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email. This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (Moon Capital). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email.
RE: Cannot create CF in Cassandra 1.1.2
Now I got tired of this, so I cleaned up everything and started from the empty keyspace. Everything is ok so far. How can I prevent this from happening in the future? -Original Message- From: Leonid Ilyevsky [mailto:lilyev...@mooncapital.com] Sent: Tuesday, July 31, 2012 3:11 PM To: 'user@cassandra.apache.org' Subject: RE: Cannot create CF in Cassandra 1.1.2 Yes, I tried it now with CLI, the same thing. No error, it says that schema agrees across the cluster, but when I call describe, the new column family is not there. Another proof is, I can repeat the same create statement multiple times, and it does not complain that it exists already. I am positive that my data is corrupted in some way, so the question is, how do I repair it. -Original Message- From: Leonid Ilyevsky [mailto:lilyev...@mooncapital.com] Sent: Tuesday, July 31, 2012 2:55 PM To: 'user@cassandra.apache.org' Subject: RE: Cannot create CF in Cassandra 1.1.2 No, I am using cql 3, but it should not matter. The point is, it used to work, and I am sure that if I just clean up the whole thing and start from scratch, it will work again. Something got corrupted. But I will give a shot to the cli, just to rule out any cql 3 issues. -Original Message- From: rohit bhatia [mailto:rohit2...@gmail.com] Sent: Tuesday, July 31, 2012 2:40 PM To: user@cassandra.apache.org Subject: Re: Cannot create CF in Cassandra 1.1.2 I believe u r using cassandra-cli. Please use help create for the proper syntax.. See http://wiki.apache.org/cassandra/DataModel/ for cassandra datamodel. Also, the flushing events are expected. For example run create keyspace name1 , use name1 , create column family name2 This should work with default settings.. On Tue, Jul 31, 2012 at 11:34 PM, Leonid Ilyevsky lilyev...@mooncapital.com wrote: My cassandra used to work fine, now it does not create a table. No error message, but the table is not there. I execute create table statement many times, the same thing. When I do it, I see in the system.log the the lines below. What can be the problem? Pre-history: before I noticed it is broken, I went through some issues related to the lack of space, so I was adding more storage and restarting both servers (I have my thing partitioned between 2 nodes). Is there any workaround to bring it back to normal, besides cleaning it all up and starting from scratch? = INFO [MigrationStage:1] 2012-07-31 13:51:20,090 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columnfamilies@766316124(1520/1900 serialized/live bytes, 20 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,091 Memtable.java (line 266) Writing Memtable-schema_columnfamilies@766316124(1520/1900 serialized/live bytes, 20 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,155 Memtable.java (line 307) Completed flushing /Data_Store1/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hd-54-Data.db (1575 bytes) for commitlog position ReplayPosition(segmentId=3346642617210212, position=12976) INFO [MigrationStage:1] 2012-07-31 13:51:20,156 ColumnFamilyStore.java (line 643) Enqueuing flush of Memtable-schema_columns@2106973490(2048/2560 serialized/live bytes, 30 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,157 Memtable.java (line 266) Writing Memtable-schema_columns@2106973490(2048/2560 serialized/live bytes, 30 ops) INFO [FlushWriter:5] 2012-07-31 13:51:20,211 Memtable.java (line 307) Completed flushing /Data_Store1/cassandra/data/system/schema_columns/system-schema_columns-hd-54-Data.db (2111 bytes) for commitlog position ReplayPosition(segmentId=3346642617210212, position=12976) This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any disclosure, copying, forwarding or distribution of this email is strictly prohibited and this email and any attachments should be deleted immediately. This email and any attachments do not constitute an offer to sell or a solicitation of an offer to purchase any interest in any investment vehicle sponsored by Moon Capital Management LP (Moon Capital). Moon Capital does not provide legal, accounting or tax advice. Any statement regarding legal, accounting or tax matters was not intended or written to be relied upon by any person as advice. Moon Capital does not waive confidentiality or privilege as a result of this email. This email, along with any attachments, is confidential and may be legally privileged or otherwise protected from disclosure. Any unauthorized dissemination, copying or use of the contents of this email is strictly prohibited and may be in violation of law. If you are not the intended recipient, any
thrift/cassandra stacktrace....
1. I would think thrift would give better errors, I have no idea where to look(thrift or cassandra?). Anyone have any idea how to figure this out? (I was hoping thrift would give astyanax a better error than it did :( )…. 2. Anyone know if I am feeding my composite column info incorrectly below (as that is what caused this exception since UTF8Type works fine)….(ie. The last line in that code is what causes this issue to occur). ColumnFamilyDefinition def = cluster.makeColumnFamilyDefinition() .setName(colFamily) .setKeyspace(keyspace.getKeyspaceName()) .setComparatorType(CompositeType(UTF8Type, BytesType)); java.lang.RuntimeException: com.netflix.astyanax.connectionpool.exceptions.BadRequestException: BadRequestException: [host=127.0.0.1(127.0.0.1):9160, latency=11(11), attempts=1] InvalidRequestException(why:Not enough bytes to read value of component 0) at com.alvazan.orm.layer3.spi.db.cassandra.CassandraSession.sendChanges(CassandraSession.java:182) at com.alvazan.orm.layer2.nosql.cache.NoSqlWriteCacheImpl.flush(NoSqlWriteCacheImpl.java:78) at com.alvazan.orm.layer2.nosql.cache.NoSqlReadCacheImpl.flush(NoSqlReadCacheImpl.java:52) at com.alvazan.orm.layer1.base.BaseEntityManagerImpl.flush(BaseEntityManagerImpl.java:119) at com.alvazan.test.TestIndexesNew.testBasicIndex(TestIndexesNew.java:42) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) at org.junit.runners.ParentRunner.run(ParentRunner.java:300) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) Caused by: com.netflix.astyanax.connectionpool.exceptions.BadRequestException: BadRequestException: [host=127.0.0.1(127.0.0.1):9160, latency=11(11), attempts=1] InvalidRequestException(why:Not enough bytes to read value of component 0) at com.netflix.astyanax.thrift.ThriftConverter.ToConnectionPoolException(ThriftConverter.java:159) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:60) at com.netflix.astyanax.thrift.AbstractOperationImpl.execute(AbstractOperationImpl.java:27) at com.netflix.astyanax.thrift.ThriftSyncConnectionFactoryImpl$1.execute(ThriftSyncConnectionFactoryImpl.java:131) at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFailoverImpl.java:52) at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHostPartitionConnectionPool.java:229) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:455) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.access$400(ThriftKeyspaceImpl.java:62) at com.netflix.astyanax.thrift.ThriftKeyspaceImpl$1.execute(ThriftKeyspaceImpl.java:115) at com.alvazan.orm.layer3.spi.db.cassandra.CassandraSession.sendChangesImpl(CassandraSession.java:197) at com.alvazan.orm.layer3.spi.db.cassandra.CassandraSession.sendChanges(CassandraSession.java:180) ... 30 more Caused by: InvalidRequestException(why:Not enough bytes to read value of component 0) at org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:20169) at
Re: Dynamic CF
I thought I'd give this a try:
create columnfamily
at_event__ac_c
(
ac_c text,
ac_creation bigint,
ac_name text,
ac_value text,
PRIMARY KEY (ac_c, ac_creation)
) with compression_parameters:sstable_compression = '';
Then, insert a few columns:
begin batch using consistency quorum
insert into at_event__ac_c (ac_c, ac_creation, ac_name, ac_value)
values ('83433.361.t4.l1.cisco.com', 1303167920747402,
'ac_event_type', 'SERV.CPE.CONN')
insert into at_event__ac_c (ac_c, ac_creation, ac_name, ac_value)
values ('83433.361.t4.l1.cisco.com', 1303167920747402, 'ac_vid', '')
insert into at_event__ac_c (ac_c, ac_creation, ac_name, ac_value)
values ('83433.361.t4.l1.cisco.com', 1303167920747402, 'ac_state',
'UP')
apply batch;
then followed up with a query:
cqlsh:op2 select * from at_event__ac_c;
ac_c | ac_creation | ac_name | ac_value
---+--+--+--
83433.361.t4.l1.cisco.com | 1303167920747402 | ac_state | UP
So, I can't get the same index behavior from the ac_creation column
as I get from the ac_c column. That is, ac_c is additive, ac_creation
overwrites. When I turn this around and create a real wide row, like this:
create columnfamily
at_event__ac_c
(
ac_c text,
ac_creation bigint,
ac_event_type text,
ac_vid text,
ac_state text,
PRIMARY KEY (ac_c, ac_creation)
) with compression_parameters:sstable_compression = '';
I get the behavior I want, which is a composite row which is indexed
by ac_c and ac_creation.
If I have 100 columns defined as a static CF (like above), when I do an insert
does it just use the space for the columns inserted, or the columns declared?
-g
On Tue, Jul 10, 2012 at 7:23 AM, Sylvain Lebresne sylv...@datastax.com wrote:
On Fri, Jul 6, 2012 at 10:49 PM, Leonid Ilyevsky
lilyev...@mooncapital.com wrote:
At this point I am really confused about what direction Cassandra is going.
CQL 3 has the benefit of composite keys, but no dynamic columns.
I thought, the whole point of Cassandra was to provide dynamic tables.
CQL3 absolutely provide dynamic tables/wide rows, the syntax is just
different. The typical example for wide rows is a time serie, for
instance keeping all the events for a given event_kind in the same C*
row ordered by time. You declare that in CQL3 using:
CREATE TABLE events (
event_kind text,
time timestamp,
event_name text,
event_details text,
PRIMARY KEY (event_kind, time)
)
The important part in such definition is that one CQL row (i.e a given
event_kind, time, event_name, even_details) does not map to an internal
Cassandra row. More precisely, all events sharing the same event_kind will be
in the same internal row. This is a wide row/dynamic table in the sense of
thrift.
I need to have a huge table to store market quotes, and be able to query it
by name and timestamp (t1 = t = t2), therefore I wanted the composite key.
Loading data to such table using prepared statements (CQL 3-based) was very
slow, because it makes a server call for each row.
You should use a BATCH statement which is the equivalent to batch_mutate.
--
Sylvain
Mixed cluster node with version 1.1.2 and 1.0.6 gives errors
Hi I have 3 node development cluster and all running 1.0.6 version without any issue. As the part of the upgrade to 1.1.2, I just upgrade one node to 1.1.2 version. When start the upgraded 1.1.2 node, the other 1.0.6 nodes getting the below exceptions? 2012-08-01 18:31:15,990 INFO [IncomingTcpConnection] Received connection from newer protocol version. Ignorning 2012-08-01 18:31:16,008 INFO [Gossiper] Node /10.1.161.202 has restarted, now UP 2012-08-01 18:31:16,008 INFO [Gossiper] InetAddress /10.1.161.202 is now UP 2012-08-01 18:31:16,010 ERROR [AbstractCassandraDaemon] Fatal exception in thread Thread[GossipStage:1,5,main] java.lang.UnsupportedOperationException: Not a time-based UUID at java.util.UUID.timestamp(UUID.java:308) at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:98) at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:81) at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:807) at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:850) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:909) at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2012-08-01 18:31:16,013 ERROR [AbstractCassandraDaemon] Fatal exception in thread Thread[GossipStage:1,5,main] java.lang.UnsupportedOperationException: Not a time-based UUID at java.util.UUID.timestamp(UUID.java:308) at org.apache.cassandra.service.MigrationManager.rectify(MigrationManager.java:98) at org.apache.cassandra.service.MigrationManager.onAlive(MigrationManager.java:81) at org.apache.cassandra.gms.Gossiper.markAlive(Gossiper.java:807) at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:850) at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:909) at org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(GossipDigestAckVerbHandler.java:68) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2012-08-01 18:31:16,383 INFO [StorageService] Node /10.1.161.202 state jump to normal 2012-08-01 18:32:17,132 ERROR [AbstractCassandraDaemon] Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: Could not reach schema agreement with /10.1.161.202 in 6ms at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:224) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:239) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 2012-08-01 18:32:17,133 ERROR [AbstractCassandraDaemon] Fatal exception in thread Thread[HintedHandoff:1,1,main] java.lang.RuntimeException: Could not reach schema agreement with /10.1.161.202 in 6ms at org.apache.cassandra.db.HintedHandOffManager.waitForSchemaAgreement(HintedHandOffManager.java:224) at org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:239) at org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:81) at org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:353) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Any thoughts?? /Roshan -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Mixed-cluster-node-with-version-1-1-2-and-1-0-6-gives-errors-tp7581534.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Secondary Index Limitation / DOes it exist in Cassandra 1.1.2
Hi All, Correct me if I am wrong, but I know that secondary indexes are stored to local column families on each node. Previously where the default cache key value was 200,000 rows, and you couldn't really tune the local index column family, that posed a limitation on low cardinality of the possible values of the secondary index column. However, in Cassandra 1.1.2, I don't see the option of tuning cache per row count any more and it is solely memory based. I wonder if this eliminates he previous limitations with secondary indexes. Please advice. Cheers, -Arya
Cassandra FAILED_TO_UNCOMPRESS
Hi, We are using DataStax cassandra (2 nodes) have recently upgraded from 0.8.x to 1.0.x. Followed the steps on http://www.datastax.com/docs/1.0/install/upgrading#upgrading-cassandra-0-8-x-to-1-0-x however on running the final repair on Node1 we get a error message as below. Please can you guide. INFO [AntiEntropyStage:1] 2012-07-30 11:57:41,368 AntiEntropyService.java (line 837) [repair #a99c1a50-da0f-11e1--ca50e7b510fb] requests for merkle tree sent for Standard1 (to [/172.16.0.29, /172.16.0.34]) INFO [AntiEntropyStage:1] 2012-07-30 11:57:41,381 AntiEntropyService.java (line 935) [repair #a99c1a50-da0f-11e1--ca50e7b510fb] Endpoints /172.16.0.29 and /172.16.0.34 are consistent for Indexed1 INFO [AntiEntropyStage:1] 2012-07-30 11:57:41,382 AntiEntropyService.java (line 762) [repair #a99c1a50-da0f-11e1--ca50e7b510fb] Indexed1 is fully synced (1 remaining column family to sync for this session) INFO [CompactionExecutor:5] 2012-07-30 12:56:15,998 AutoSavingCache.java (line 269) Saved Keyspace1-Standard1-KeyCache (35450 items) in 542 ms INFO [CompactionExecutor:6] 2012-07-30 13:56:16,020 AutoSavingCache.java (line 269) Saved Keyspace1-Standard1-KeyCache (35450 items) in 557 ms ERROR [ValidationExecutor:1] 2012-07-30 14:18:51,390 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[ValidationExecutor:1,1,main] java.io.IOError: java.io.IOException: FAILED_TO_UNCOMPRESS(5) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:97) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:137) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:97) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:82) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:118) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:101) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:614) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135) at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:849) at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:64) at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:440) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: FAILED_TO_UNCOMPRESS(5) at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78) at org.xerial.snappy.SnappyNative.rawUncompress(Native Method) at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391) at org.apache.cassandra.io.compress.SnappyCompressor.uncompress(SnappyCompressor.java:52) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:89) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:75) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377) at org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:396) at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:358) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120) at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:144) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:232) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:110) ... 19 more INFO
Re: CQL3 and column slices
In the absence of streaming, how does one retrieve a large result set in CQL3? E.g., to use the same example: CREATE TABLE bug_test (a int, b int, c int, d int, e int, f text, PRIMARY KEY (a, b, c, d, e) ); with some data in it: SELECT * FROM bug_test; Results: a | b | c | d | e | f --+-- 1 | 1 | 1 | 1 | 1 | 1 1 | 1 | 1 | 1 | 2 | 2 1 | 1 | 1 | 1 | 3 | 3 1 | 1 | 1 | 1 | 5 | 5 1 | 1 | 1 | 2 | 1 | 1 The query SELECT * FROM bug_test WHERE a = 1 AND b = 1; will retrieve the above 5 rows. But what if my dataset had a million rows that match the query? How can I retrieve them? Thanks! Thorsten On 7/24/2012 11:58 PM, Sylvain Lebresne wrote: The short answer is yes, we are looking into adding streaming of results to solve that problem (https://issues.apache.org/jira/browse/CASSANDRA-4415). -- Sylvain On Tue, Jul 24, 2012 at 6:51 PM, Josep Blanquer blanq...@rightscale.com wrote: Thank Sylvain, The main argument for this is pagination. Let me try to explain the use cases, and compare it to RDBMS for better illustration: 1- Right now, Cassandra doesn't stream the requests, so large resultsets are a royal pain in the neck to deal with. I.e., if I have a range_slice, or even a slice query that cuts across 1 million columns...I have to completely eat it all in the client receiving the response. That is, I'll need to store 1 million results in the client no matter what, and that can be quite prohibitive. 2- In an effort to alleviate that, one can be smarter in the client and play the pagination game...i.e., start slicing at some column and get the next N results, then start the slice at the last column seen and get N moreetc. That results in many more queries from the smart client, but at least it would allow you to handle large result sets. (That's where the need for the CQL query in my original email was about). 3- There's another important factor related to this problem in my opinion: the LIMIT clause in Cassandra (in both CQL or Thrift) is a required field. What I mean by required is that cassandra requires an explicit count to operate underneath. So it is really different from RDBMS' semantics where no LIMIT means you'll get all the results (instead of the high, yet still bound count of 10K or 20K max resultset row cassandra enforces by defaul)...and I cannot tell you how many problems we've had with developers forgetting about these default counts in queries, and realizing that some had results truncated because of that...in my mind, LIMIT should be to only used restrict results...queries with no LIMIT should always return all results (much like RDBMS)...otherwise the query seems the same but it is semantically different. So, all in all I think that the main problem/use case I'm facing is that Cassandra cannot stream resultsets. If it did, I believe that the need for my pagination use case would basically disappear, since it'd be the transport/client that would throttle how many results are stored in the client buffer at any point time. At the same time, I believe that with a streaming protocol you could simply change Cassandra internals to have infinite default limits...since there wouldn't be no reason to stop scanning (unless an explicit LIMIT clause was specified by the client). That would give you not only the SQL-equivalent syntax, but also the equivalent semantics of most current DBs. I hope that makes sense. That being said, are there any plans for streaming results? I believe that without that (and especially with the new CQL restrictions) it make much more difficult to use Cassandra with wide rows and large resultsets (which, in my mind is one of its sweet spots ). I believe that if that doesn't happen it would a) force the clients to be built in a much more complex and inefficient way to handle wide rows or b) will force users to use different, less efficient datamodels for their data. Both seem bad propositions to me, as they wouldn't be taking advantage of Cassandra's power, therefore diminishing its value. Cheers, Josep M. On Tue, Jul 24, 2012 at 3:11 AM, Sylvain Lebresne sylv...@datastax.com wrote: On Tue, Jul 24, 2012 at 12:09 AM, Josep Blanquer blanq...@rightscale.com wrote: is there some way to express that in CQL3? something logically equivalent to SELECT * FROM bug_test WHERE a:b:c:d:e 1:1:1:1:2?? No, there isn't. Not currently at least. But feel free of course to open a ticket/request on https://issues.apache.org/jira/browse/CASSANDRA. I note that I would be curious to know the concrete use case you have for such type of queries. It would also help as an argument to add such facilities more quickly (or at all). Typically, we should support it in CQL3 because it was possible with thrift is definitively an argument, but a much weaker one without concrete examples of why it might be useful in the first place. -- Sylvain
