Re: Murmur Long.MIN_VALUE token allowed?
Hi Aaron, thanks for your response. But that is exactly what scares me: RandomPartitioner.MIN is -1, which is not a valid token :-) And my feeling gets worse when I look at Murmur3Partitioner.normalize(). This one explicitly excludes Long.MIN_VALUE by changing it to Long.MAX_VALUE. I think I'll just avoid it in the future. Better safe than sorry... cheers, Christian On Tue, Dec 10, 2013 at 8:24 AM, Aaron Morton aa...@thelastpickle.comwrote: AFAIK any value that is a valid output from murmor3 is a valid token. The Murmur3Partitioner set’s min and max to long min and max… public static final LongToken MINIMUM = new LongToken(Long.MIN_VALUE); public static final long MAXIMUM = Long.MAX_VALUE; Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 5/12/2013, at 12:38 am, horschi hors...@gmail.com wrote: Hi, I just realized that I can move a node to Long.MIN_VALUE: 127.0.0.1 rack1 Up Normal 1011.58 KB 100.00% -9223372036854775808 Is that really a valid token for Murmur3Partitioner ? I thought that Long.MIN_VALUE (like -1 for Random) is not a regular token. Shouldn't be only used for token-range-scans? kind regards, Christian
Re: Try to configure commitlog_archiving.properties
Vicky Kak vicky.kak at gmail.com writes: Why, can you give me a good example and the good way to configure archive commit logs ? Take a look at the cassandra code ;) Taking a look to the code ? i'm not a develloper but a DBA, where should i look ? Thank you. Regards, Bonnet Jonathan.
Re: Try to configure commitlog_archiving.properties
Hi, There is some docs on the internet for this operations. It is basically as presented in the archive-commitlog file. (commitlog_archiving.properties). The way the operations work: The operation is called automatically with parameters that give you control over what you want to do with it. It looks to me like your copy command for the archiving wants to copy every file. This is unnecessary. Cassandra will give you the arguments. E.g. Documentation for the archive command reads: # Command to execute to archive a commitlog segment # Parameters: %path = Fully qualified path of the segment to archive # %name = Name of the commit log. # Example: archive_command=/bin/ln %path /backup/%name # # Limitation: *_command= expects one command with arguments. STDOUT # and STDIN or multiple commands cannot be executed. You might want # to script multiple commands and add a pointer here. Argument 1 will give you the path to the files you'd want to copy, while argument 2 will give uou the name if it. You can then create a command: archive_command=/bin/bash /home/cassandra/scripts/cassandra-archive.sh %path %name The above would be an example for me. As the commands by default only execute 1 command, I have them point to a custom script that does what I desire. My script then looks something like this: #! /bin/bash # use bzip to compress the file bzip2 --best -k $1 # move to commit log archive mv $1.bz2 $HOME/commitlog_restore/$2.bz2 I compress my commitlog and then move it somewhere else. Cassandra will call this operation first, and then delete the commitlog. You can apply similar behaviours to all of those commands. I would recommend to set up a cleanup scripts, otherwise your saved commit-log files will floud your harddrive. I would also test this locally first to make sure it works. For testing locally, bare in mind that you may wanna set your max file sizes to very low values as the command is only called when the commitlog is deleted. So having commitlogs that are 250 MB or bigger will need a lot of writing to make the event trigger. I hope I got that right and it helps. Cheers. FYI there was a bug with this, so you may wanna be on the right version: https://issues.apache.org/jira/browse/CASSANDRA-5909 On 06/12/13 15:33, Vicky Kak wrote: Why, can you give me a good example and the good way to configure archive commit logs ? Take a look at the cassandra code ;) On Fri, Dec 6, 2013 at 3:34 PM, Bonnet Jonathan jonathan.bon...@externe.bnpparibas.com mailto:jonathan.bon...@externe.bnpparibas.com wrote: Hello, I try to configure commitlog_archiving.properties to take advantage of backup and restore at a point of time, but there is no ressources on internet for that. So i need some help. If I understand I have 4 parameters: archive_command= restore_command= restore_directories= restore_point_in_time= Forget for the moment the restore_point_in_time, should i put one command only for the archive_command; my wish is to copy all the commitlogs on another directory: restore_directories=/produits/cassandra/cassandra_data/archived_logs archive_command=/bin/cp - f /produits/cassandra/cassandra_commit/*.log /produits/cassandra/cassandra_ data/archived_logs restore_command=/bin/cp - f /produits/cassandra/cassandra_data/archived_logs/*.log /produits/cassandr a/cassandra_commit But it doesn't work when i restart the node which interest me, it doesn't copy anything. Why, can you give me a good example and the good way to configure archive commit logs ? there's nothing on the net except Datastax Website. Thanks for your answear. Regards, Bonnet Jonathan.
What is the fastest way to get data into Cassandra 2 from a Java application?
I have tried the DataStax Java driver and it seems the fastest way to insert data is to compose a CQL string with all parameters inline. This loop takes 2500ms or so on my test cluster: PreparedStatement ps = session.prepare(INSERT INTO perf_test.wibble (id, info) VALUES (?, ?)) for (int i = 0; i 1000; i++) session.execute(ps.bind( + i, aa + i)); The same loop with the parameters inline is about 1300ms. It gets worse if there are many parameters. I know I can use batching to insert all the rows at once but thats not the purpose of this test. I also tried using session.execute(cql, params) and it is faster but still doesn't match inline values. Composing CQL strings is certainly convenient and simple but is there a much faster way? Thanks David I have also posted this on Stackoverflow if anyone wants the points: http://stackoverflow.com/questions/20491090/what-is-the-fastest-way-to-get-data-into-cassandra-2-from-a-java-application
Re: What is the fastest way to get data into Cassandra 2 from a Java application?
I should probably give you a number which is about 300 meg / s via thrift api and use 1mb batches On Dec 10, 2013, at 5:14 AM, graham sanderson gra...@vast.com wrote: Perhaps not the way forward, however I can bulk insert data via astyanax at a rate that maxes out our (fast) networks. That said for our next release (of this part of our product - our other current is node.js via binary protocol) we will be looking at insert speed via java driver, and also alternative scala/java implementations of the binary protocol. On Dec 10, 2013, at 4:49 AM, David Tinker david.tin...@gmail.com wrote: I have tried the DataStax Java driver and it seems the fastest way to insert data is to compose a CQL string with all parameters inline. This loop takes 2500ms or so on my test cluster: PreparedStatement ps = session.prepare(INSERT INTO perf_test.wibble (id, info) VALUES (?, ?)) for (int i = 0; i 1000; i++) session.execute(ps.bind( + i, aa + i)); The same loop with the parameters inline is about 1300ms. It gets worse if there are many parameters. I know I can use batching to insert all the rows at once but thats not the purpose of this test. I also tried using session.execute(cql, params) and it is faster but still doesn't match inline values. Composing CQL strings is certainly convenient and simple but is there a much faster way? Thanks David I have also posted this on Stackoverflow if anyone wants the points: http://stackoverflow.com/questions/20491090/what-is-the-fastest-way-to-get-data-into-cassandra-2-from-a-java-application smime.p7s Description: S/MIME cryptographic signature
Re: What is the fastest way to get data into Cassandra 2 from a Java application?
Hmm. I have read that the thrift interface to Cassandra is out of favour and the CQL interface is in. Where does that leave Astyanax? On Tue, Dec 10, 2013 at 1:14 PM, graham sanderson gra...@vast.com wrote: Perhaps not the way forward, however I can bulk insert data via astyanax at a rate that maxes out our (fast) networks. That said for our next release (of this part of our product - our other current is node.js via binary protocol) we will be looking at insert speed via java driver, and also alternative scala/java implementations of the binary protocol. On Dec 10, 2013, at 4:49 AM, David Tinker david.tin...@gmail.com wrote: I have tried the DataStax Java driver and it seems the fastest way to insert data is to compose a CQL string with all parameters inline. This loop takes 2500ms or so on my test cluster: PreparedStatement ps = session.prepare(INSERT INTO perf_test.wibble (id, info) VALUES (?, ?)) for (int i = 0; i 1000; i++) session.execute(ps.bind( + i, aa + i)); The same loop with the parameters inline is about 1300ms. It gets worse if there are many parameters. I know I can use batching to insert all the rows at once but thats not the purpose of this test. I also tried using session.execute(cql, params) and it is faster but still doesn't match inline values. Composing CQL strings is certainly convenient and simple but is there a much faster way? Thanks David I have also posted this on Stackoverflow if anyone wants the points: http://stackoverflow.com/questions/20491090/what-is-the-fastest-way-to-get-data-into-cassandra-2-from-a-java-application -- http://qdb.io/ Persistent Message Queues With Replay and #RabbitMQ Integration
Re: Exactly one wide row per node for a given CF?
I have a question about this statement: When rows get above a few 10¹s of MB things can slow down, when they get above 50 MB they can be a pain, when they get above 100MB it¹s a warning sign. And when they get above 1GB, well you you don¹t want to know what happens then. I tested a data model that I created. Here¹s the schema for the table in question: CREATE TABLE bdn_index_pub ( tree INT, pord INT, hpath VARCHAR, PRIMARY KEY (tree, pord) ); As a test, I inserted 100 million records. tree had the same value for every record, and I had 100 million values for pord. hpath averaged about 50 characters in length. My understanding is that all 100 million strings would have been stored in a single row, since they all had the same value for the first component of the primary key. I didn¹t look at the size of the table, but it had to be several gigs (uncompressed). Contrary to what Aaron says, I do want to know what happens, because I didn¹t experience any issues with this table during my test. Inserting was fast. The last batch of records inserted in approximately the same amount of time as the first batch. Querying the table was fast. What I didn¹t do was test the table under load, nor did I try this in a multi-node cluster. If this is bad, can somebody suggest a better pattern? This table was designed to support a query like this: select hpath from bdn_index_pub where tree = :tree and pord = :start and pord = :end. In my application, most trees will have less than a million records. A handful will have 10¹s of millions, and one of them will have 100 million. If I need to break up my rows, my first instinct would be to divide each tree into blocks of say 10,000 and change tree to a string that contains the tree and the block number. Something like this: 17:0, 0, /¹ 17:0, , ¹/a/b/c¹ 17:1,1, /a/b/d¹ I¹d then need to issue an extra query for ranges that crossed block boundaries. Any suggestions on a better pattern? Thanks Robert From: Aaron Morton aa...@thelastpickle.com Reply-To: user@cassandra.apache.org Date: Tuesday, December 10, 2013 at 12:33 AM To: Cassandra User user@cassandra.apache.org Subject: Re: Exactly one wide row per node for a given CF? But this becomes troublesome if I add or remove nodes. What effectively I want is to partition on the unique id of the record modulus N (id % N; where N is the number of nodes). This is exactly the problem consistent hashing (used by cassandra) is designed to solve. If you hash the key and modulo the number of nodes, adding and removing nodes requires a lot of data to move. I want to be able to randomly distribute a large set of records but keep them clustered in one wide row per node. Sounds like you should revisit your data modelling, this is a pretty well known anti pattern. When rows get above a few 10¹s of MB things can slow down, when they get above 50 MB they can be a pain, when they get above 100MB it¹s a warning sign. And when they get above 1GB, well you you don¹t want to know what happens then. It¹s a bad idea and you should take another look at the data model. If you have to do it, you can try the ByteOrderedPartitioner which uses the row key as a token, given you total control of the row placement. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 4/12/2013, at 8:32 pm, Vivek Mishra mishra.v...@gmail.com wrote: So Basically you want to create a cluster of multiple unique keys, but data which belongs to one unique should be colocated. correct? -Vivek On Tue, Dec 3, 2013 at 10:39 AM, onlinespending onlinespend...@gmail.com wrote: Subject says it all. I want to be able to randomly distribute a large set of records but keep them clustered in one wide row per node. As an example, lets say I¹ve got a collection of about 1 million records each with a unique id. If I just go ahead and set the primary key (and therefore the partition key) as the unique id, I¹ll get very good random distribution across my server cluster. However, each record will be its own row. I¹d like to have each record belong to one large wide row (per server node) so I can have them sorted or clustered on some other column. If I say have 5 nodes in my cluster, I could randomly assign a value of 1 - 5 at the time of creation and have the partition key set to this value. But this becomes troublesome if I add or remove nodes. What effectively I want is to partition on the unique id of the record modulus N (id % N; where N is the number of nodes). I have to imagine there¹s a mechanism in Cassandra to simply randomize the partitioning without even using a key (and then clustering on some column). Thanks for any help.
Re: list all nodes as seeds (excluding self)
My understanding is that a node won't auto-bootstrap if it thinks it's a seed node. So when adding a new node to an existing cluster, I want to make sure it will auto-bootstrap, and I don't want to do 2 edits to the config file (first start without node as seed, then add node as seed). On 12/09/2013 11:54 PM, Daneel Yaitskov wrote: What is the problem to put all nodes on the seed list without any exclusion. Constant list is the simplest solution. You should write it once in cassandra.yaml, tar with everything you need (cassandra, jdk) and copy as many as number of your nodes on the list. Really 1 thing you should generate is rpc address and listen address. I use self extracting bash archives cat EOF listen_address: $HOSTNAME EOF On Mon, Dec 9, 2013 at 7:32 PM, Anne Sullivan acot...@gmail.com wrote: For ease of maintenance and because we'll likely have many deployments where the cluster size is very small (2 - 5 nodes), I'm wondering if I can set my seed_provider list to contain all nodes except the local node's IP. ie) For nodes A-C A- B, C B- A, C C- A, B I think my question is more or less In line with this comment, I'm wondering if satisfying ONLY 2) is safe: https://issues.apache.org/jira/browse/CASSANDRA-5836?focusedCommentId=13727032 Datastax docs suggest that "every node should have the same list of seeds", and also "To prevent partitions in gossip communications, use the same list of seed nodes in all nodes in a cluster". I wouldn't end up with gossip partitions in the example above, so if that's the only reason for the recommendation of keeping the list consistent across all nodes then it should be ok. Reading other comments regarding the seed list / bootstrapping, I *think* this will be ok but would appreciate confirmation/comments. My goal is to have all nodes auto-bootstrap. When adding a new node, I don't want to do 2 edits to the config file (first start without node as seed, then add node as seed). thanks Anne -- Daneel S. Yaitskov
Recurring actions with 4 hour interval
Hello, We've been having a lot of problems with extremely long GC (and still do) which I've asked about several times on this list (I can find links to those discussions if anyone is interested). We noticed a pattern that the GC pauses may be related to something happening every 4 hours. Is there anything specific happening within Cassandra with a 4 hour interval? Any help is much appreciated, Joel Samuelsson
setting PIG_INPUT_INITIAL_ADDRESS environment . variable in Oozie for cassandra ...¿?
Hi, I have an error with pig action in oozie 4.0.0 using cassandraStorage. (cassandra 1.2.10) I can run pig scripts right with cassandra. but whe I try to use cassandraStorage to load data I have this error: *Run pig script using PigRunner.run() for Pig version 0.8+* *Apache Pig version 0.10.0 (r1328203) * *compiled Apr 20 2012, 00:33:25* *Run pig script using PigRunner.run() for Pig version 0.8+* *2013-12-10 12:24:39,084 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0 (r1328203) compiled Apr 20 2012, 00:33:25* *2013-12-10 12:24:39,084 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0 (r1328203) compiled Apr 20 2012, 00:33:25* *2013-12-10 12:24:39,095 [main] INFO org.apache.pig.Main - Logging error messages to: /tmp/hadoop-ec2-user/mapred/local/taskTracker/ec2-user/jobcache/job_201312100858_0007/attempt_201312100858_0007_m_00_0/work/pig-job_201312100858_0007.log* *2013-12-10 12:24:39,095 [main] INFO org.apache.pig.Main - Logging error messages to: /tmp/hadoop-ec2-user/mapred/local/taskTracker/ec2-user/jobcache/job_201312100858_0007/attempt_201312100858_0007_m_00_0/work/pig-job_201312100858_0007.log* *2013-12-10 12:24:39,501 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://10.228.243.18:9000 http://10.228.243.18:9000* *2013-12-10 12:24:39,501 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://10.228.243.18:9000 http://10.228.243.18:9000* *2013-12-10 12:24:39,510 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: 10.228.243.18:9001 http://10.228.243.18:9001* *2013-12-10 12:24:39,510 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: 10.228.243.18:9001 http://10.228.243.18:9001* *2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2245: * *file testCassandra.pig, line 7, column 7 Cannot get schema from loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage* *2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2245: * *file testCassandra.pig, line 7, column 7 Cannot get schema from loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage* *2013-12-10 12:24:40,505 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2245: * *file testCassandra.pig, line 7, column 7 Cannot get schema from loadFunc org.apache.cassandra.hadoop.pig.CassandraStorage* * at org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:155)* * at org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:110)* * at org.apache.pig.newplan.logical.relational.LOStore.getSchema(LOStore.java:68)* * at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.validate(SchemaAliasVisitor.java:60)* * at org.apache.pig.newplan.logical.visitor.SchemaAliasVisitor.visit(SchemaAliasVisitor.java:84)* * at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:77)* * at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)* * at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)* * at org.apache.pig.PigServer$Graph.compile(PigServer.java:1617)* * at org.apache.pig.PigServer$Graph.compile(PigServer.java:1611)* * at org.apache.pig.PigServer$Graph.access$200(PigServer.java:1334)* * at org.apache.pig.PigServer.execute(PigServer.java:1239)* * at org.apache.pig.PigServer.executeBatch(PigServer.java:362)* * at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:132)* * at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:193)* * at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)* * at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)* * at org.apache.pig.Main.run(Main.java:430)* * at org.apache.pig.PigRunner.run(PigRunner.java:49)* * at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)* * at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)* * at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)* * at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)* * at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)* * at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)* * at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)* * at java.lang.reflect.Method.invoke(Method.java:601)* * at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)* * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)*
Re: Recurring actions with 4 hour interval
As far as I know there is nothing hard coded in Cassandra that kicks in every 4 hours. Turn on GC logging, maybe dump the output of jstats to a file and correlate this data with the Cassandra logs. Cassandra logs are pretty good in telling you what is going on. 2013/12/10 Joel Samuelsson samuelsson.j...@gmail.com Hello, We've been having a lot of problems with extremely long GC (and still do) which I've asked about several times on this list (I can find links to those discussions if anyone is interested). We noticed a pattern that the GC pauses may be related to something happening every 4 hours. Is there anything specific happening within Cassandra with a 4 hour interval? Any help is much appreciated, Joel Samuelsson
Re: Recurring actions with 4 hour interval
2.0.3: system tables have a 1 hour memtable_flush_period which I have observed to trigger compaction on the 4 hour mark. Going by memory tho... -ml On Tue, Dec 10, 2013 at 10:31 AM, Andre Sprenger andre.spren...@getanet.dewrote: As far as I know there is nothing hard coded in Cassandra that kicks in every 4 hours. Turn on GC logging, maybe dump the output of jstats to a file and correlate this data with the Cassandra logs. Cassandra logs are pretty good in telling you what is going on. 2013/12/10 Joel Samuelsson samuelsson.j...@gmail.com Hello, We've been having a lot of problems with extremely long GC (and still do) which I've asked about several times on this list (I can find links to those discussions if anyone is interested). We noticed a pattern that the GC pauses may be related to something happening every 4 hours. Is there anything specific happening within Cassandra with a 4 hour interval? Any help is much appreciated, Joel Samuelsson
Re: Try to configure commitlog_archiving.properties
Thanks a lot, It Works, i see commit log bein archived. I'll try tomorrow the restore command. Thanks again. Bonnet Jonathan.
Re: Exactly one wide row per node for a given CF?
comments below On Dec 9, 2013, at 11:33 PM, Aaron Morton aa...@thelastpickle.com wrote: But this becomes troublesome if I add or remove nodes. What effectively I want is to partition on the unique id of the record modulus N (id % N; where N is the number of nodes). This is exactly the problem consistent hashing (used by cassandra) is designed to solve. If you hash the key and modulo the number of nodes, adding and removing nodes requires a lot of data to move. I want to be able to randomly distribute a large set of records but keep them clustered in one wide row per node. Sounds like you should revisit your data modelling, this is a pretty well known anti pattern. When rows get above a few 10’s of MB things can slow down, when they get above 50 MB they can be a pain, when they get above 100MB it’s a warning sign. And when they get above 1GB, well you you don’t want to know what happens then. It’s a bad idea and you should take another look at the data model. If you have to do it, you can try the ByteOrderedPartitioner which uses the row key as a token, given you total control of the row placement. You should re-read my last paragraph as an example that would clearly benefit from such an approach. If one understands how paging works then you’ll see why how you’d benefit from grouping probabilistically similar data within each node, but also wanting to split data across nodes to reduce hot spotting. Regardless, I no longer think it’s necessary to have a single wide rode per node. Several wide rows per node is just as good, since for all practical purposes paging in the first N key/values per M rows on a node is the same as reading in the first N*M key/values from a single row. So I’m going to do what I alluded to before. Treat the LSB of a record’s id as the partition key, and then cluster on something meaningful (such as geohash) and the prefix of the id. create table test ( id_prefix int, id_suffix int, geohash text, value text, primary key (id_suffix, geohash, id_prefix)); So if this was for a collection of users, they would be randomly distributed across nodes to increase parallelism and reduce hotspots, but within each wide row they'd be meaningfully clustered by geographic region, so as to increase the probability that paged in data is going to contain more of the working set. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 4/12/2013, at 8:32 pm, Vivek Mishra mishra.v...@gmail.com wrote: So Basically you want to create a cluster of multiple unique keys, but data which belongs to one unique should be colocated. correct? -Vivek On Tue, Dec 3, 2013 at 10:39 AM, onlinespending onlinespend...@gmail.com wrote: Subject says it all. I want to be able to randomly distribute a large set of records but keep them clustered in one wide row per node. As an example, lets say I’ve got a collection of about 1 million records each with a unique id. If I just go ahead and set the primary key (and therefore the partition key) as the unique id, I’ll get very good random distribution across my server cluster. However, each record will be its own row. I’d like to have each record belong to one large wide row (per server node) so I can have them sorted or clustered on some other column. If I say have 5 nodes in my cluster, I could randomly assign a value of 1 - 5 at the time of creation and have the partition key set to this value. But this becomes troublesome if I add or remove nodes. What effectively I want is to partition on the unique id of the record modulus N (id % N; where N is the number of nodes). I have to imagine there’s a mechanism in Cassandra to simply randomize the partitioning without even using a key (and then clustering on some column). Thanks for any help.
Re: Exactly one wide row per node for a given CF?
Where you’ll run into trouble is with compaction. It looks as if pord is some sequentially increasing value. Try your experiment again with a clustering key which is a bit more random at the time of insertion. On Dec 10, 2013, at 5:41 AM, Robert Wille rwi...@fold3.com wrote: I have a question about this statement: When rows get above a few 10’s of MB things can slow down, when they get above 50 MB they can be a pain, when they get above 100MB it’s a warning sign. And when they get above 1GB, well you you don’t want to know what happens then. I tested a data model that I created. Here’s the schema for the table in question: CREATE TABLE bdn_index_pub ( tree INT, pord INT, hpath VARCHAR, PRIMARY KEY (tree, pord) ); As a test, I inserted 100 million records. tree had the same value for every record, and I had 100 million values for pord. hpath averaged about 50 characters in length. My understanding is that all 100 million strings would have been stored in a single row, since they all had the same value for the first component of the primary key. I didn’t look at the size of the table, but it had to be several gigs (uncompressed). Contrary to what Aaron says, I do want to know what happens, because I didn’t experience any issues with this table during my test. Inserting was fast. The last batch of records inserted in approximately the same amount of time as the first batch. Querying the table was fast. What I didn’t do was test the table under load, nor did I try this in a multi-node cluster. If this is bad, can somebody suggest a better pattern? This table was designed to support a query like this: select hpath from bdn_index_pub where tree = :tree and pord = :start and pord = :end. In my application, most trees will have less than a million records. A handful will have 10’s of millions, and one of them will have 100 million. If I need to break up my rows, my first instinct would be to divide each tree into blocks of say 10,000 and change tree to a string that contains the tree and the block number. Something like this: 17:0, 0, ‘/’ … 17:0, , ’/a/b/c’ 17:1,1, ‘/a/b/d’ … I’d then need to issue an extra query for ranges that crossed block boundaries. Any suggestions on a better pattern? Thanks Robert From: Aaron Morton aa...@thelastpickle.com Reply-To: user@cassandra.apache.org Date: Tuesday, December 10, 2013 at 12:33 AM To: Cassandra User user@cassandra.apache.org Subject: Re: Exactly one wide row per node for a given CF? But this becomes troublesome if I add or remove nodes. What effectively I want is to partition on the unique id of the record modulus N (id % N; where N is the number of nodes). This is exactly the problem consistent hashing (used by cassandra) is designed to solve. If you hash the key and modulo the number of nodes, adding and removing nodes requires a lot of data to move. I want to be able to randomly distribute a large set of records but keep them clustered in one wide row per node. Sounds like you should revisit your data modelling, this is a pretty well known anti pattern. When rows get above a few 10’s of MB things can slow down, when they get above 50 MB they can be a pain, when they get above 100MB it’s a warning sign. And when they get above 1GB, well you you don’t want to know what happens then. It’s a bad idea and you should take another look at the data model. If you have to do it, you can try the ByteOrderedPartitioner which uses the row key as a token, given you total control of the row placement. Cheers - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 4/12/2013, at 8:32 pm, Vivek Mishra mishra.v...@gmail.com wrote: So Basically you want to create a cluster of multiple unique keys, but data which belongs to one unique should be colocated. correct? -Vivek On Tue, Dec 3, 2013 at 10:39 AM, onlinespending onlinespend...@gmail.com wrote: Subject says it all. I want to be able to randomly distribute a large set of records but keep them clustered in one wide row per node. As an example, lets say I’ve got a collection of about 1 million records each with a unique id. If I just go ahead and set the primary key (and therefore the partition key) as the unique id, I’ll get very good random distribution across my server cluster. However, each record will be its own row. I’d like to have each record belong to one large wide row (per server node) so I can have them sorted or clustered on some other column. If I say have 5 nodes in my cluster, I could randomly assign a value of 1 - 5 at the time of creation and have the partition key set to this value. But this becomes troublesome if I add or remove nodes. What effectively I want is to partition on the unique id of
Re: Exactly one wide row per node for a given CF?
You could shard your rows like the following.
You would need over 100 shards, possibly... so testing is in order :)
Michael
-- put this in file and run using 'cqlsh -f file
DROP KEYSPACE robert_test;
CREATE KEYSPACE robert_test WITH replication = {
'class': 'SimpleStrategy',
'replication_factor' : 1
};
USE robert_test;
CREATE TABLE bdn_index_pub (
tree int,
shard int,
pord int,
hpath text,
PRIMARY KEY ((tree, shard), pord)
);
-- shard is calculated as pord % 12
COPY bdn_index_pub (tree, shard, pord, hpath) FROM STDIN;
1, 1, 1, Chicago
5, 3, 15, New York
1, 5, 5, Melbourne
3, 2, 2, San Francisco
1, 3, 3, Palo Alto
\.
SELECT * FROM bdn_index_pub
WHERE shard IN (0,1,2,3,4,5,6,7,8,9,10,11)
AND tree = 1
AND pord 4
AND pord 0
ORDER BY pord desc
;
-- returns:
-- tree | shard | pord | hpath
+---+--+--
--1 | 3 |3 | Palo Alto
--1 | 1 |1 |Chicago
On Tue, Dec 10, 2013 at 8:41 AM, Robert Wille rwi...@fold3.com wrote:
I have a question about this statement:
When rows get above a few 10’s of MB things can slow down, when they get
above 50 MB they can be a pain, when they get above 100MB it’s a warning
sign. And when they get above 1GB, well you you don’t want to know what
happens then.
I tested a data model that I created. Here’s the schema for the table in
question:
CREATE TABLE bdn_index_pub (
tree INT,
pord INT,
hpath VARCHAR,
PRIMARY KEY (tree, pord)
);
As a test, I inserted 100 million records. tree had the same value for
every record, and I had 100 million values for pord. hpath averaged about
50 characters in length. My understanding is that all 100 million strings
would have been stored in a single row, since they all had the same value
for the first component of the primary key. I didn’t look at the size of
the table, but it had to be several gigs (uncompressed). Contrary to what
Aaron says, I do want to know what happens, because I didn’t experience any
issues with this table during my test. Inserting was fast. The last batch
of records inserted in approximately the same amount of time as the first
batch. Querying the table was fast. What I didn’t do was test the table
under load, nor did I try this in a multi-node cluster.
If this is bad, can somebody suggest a better pattern? This table was
designed to support a query like this: select hpath from bdn_index_pub
where tree = :tree and pord = :start and pord = :end. In my application,
most trees will have less than a million records. A handful will have 10’s
of millions, and one of them will have 100 million.
If I need to break up my rows, my first instinct would be to divide each
tree into blocks of say 10,000 and change tree to a string that contains
the tree and the block number. Something like this:
17:0, 0, ‘/’
…
17:0, , ’/a/b/c’
17:1,1, ‘/a/b/d’
…
I’d then need to issue an extra query for ranges that crossed block
boundaries.
Any suggestions on a better pattern?
Thanks
Robert
From: Aaron Morton aa...@thelastpickle.com
Reply-To: user@cassandra.apache.org
Date: Tuesday, December 10, 2013 at 12:33 AM
To: Cassandra User user@cassandra.apache.org
Subject: Re: Exactly one wide row per node for a given CF?
But this becomes troublesome if I add or remove nodes. What effectively I
want is to partition on the unique id of the record modulus N (id % N;
where N is the number of nodes).
This is exactly the problem consistent hashing (used by cassandra) is
designed to solve. If you hash the key and modulo the number of nodes,
adding and removing nodes requires a lot of data to move.
I want to be able to randomly distribute a large set of records but keep
them clustered in one wide row per node.
Sounds like you should revisit your data modelling, this is a pretty well
known anti pattern.
When rows get above a few 10’s of MB things can slow down, when they get
above 50 MB they can be a pain, when they get above 100MB it’s a warning
sign. And when they get above 1GB, well you you don’t want to know what
happens then.
It’s a bad idea and you should take another look at the data model. If you
have to do it, you can try the ByteOrderedPartitioner which uses the row
key as a token, given you total control of the row placement.
Cheers
-
Aaron Morton
New Zealand
@aaronmorton
Co-Founder Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 4/12/2013, at 8:32 pm, Vivek Mishra mishra.v...@gmail.com wrote:
So Basically you want to create a cluster of multiple unique keys, but
data which belongs to one unique should be colocated. correct?
-Vivek
On Tue, Dec 3, 2013 at 10:39 AM, onlinespending
onlinespend...@gmail.comwrote:
Subject says it all. I want to be able to randomly distribute a large set
of records but keep them clustered in one wide row per node.
As an example, lets say I’ve got a collection
Re: Drop keyspace via CQL hanging on master/trunk.
Great. Thanks Aaron.
FWIW, I am/was porting Virgil over CQL.
I should be able to release a new REST API for C* (using CQL) shortly.
-brian
---
Brian O'Neill
Chief Architect
Health Market Science
The Science of Better Results
2700 Horizon Drive • King of Prussia, PA • 19406
M: 215.588.6024 • @boneill42 • healthmarketscience.com
This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If you
received this email in error and are not the intended recipient, or the person
responsible to deliver it to the intended recipient, please contact the sender
at the email above and delete this email and any attachments and destroy any
copies thereof. Any review, retransmission, dissemination, copying or other use
of, or taking any action in reliance upon, this information by persons or
entities other than the intended recipient is strictly prohibited.
On Dec 10, 2013, at 1:51 PM, Aaron Morton aa...@thelastpickle.com wrote:
Looks like a bug, will try to fix today
https://issues.apache.org/jira/browse/CASSANDRA-6472
Cheers
-
Aaron Morton
New Zealand
@aaronmorton
Co-Founder Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 6/12/2013, at 10:25 am, Brian O'Neill b...@alumni.brown.edu wrote:
I removed the data directory just to make sure I had a clean environment.
(eliminating the possibility of corrupt keyspaces/files causing problems)
-brian
---
Brian O'Neill
Chief Architect
Health Market Science
The Science of Better Results
2700 Horizon Drive • King of Prussia, PA • 19406
M: 215.588.6024 • @boneill42 •
healthmarketscience.com
This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If
you received this email in error and are not the intended recipient, or the
person responsible to deliver it to the intended recipient, please contact
the sender at the email above and delete this email and any attachments and
destroy any copies thereof. Any review, retransmission, dissemination,
copying or other use of, or taking any action in reliance upon, this
information by persons or entities other than the intended recipient is
strictly prohibited.
From: Jason Wee peich...@gmail.com
Reply-To: user@cassandra.apache.org
Date: Thursday, December 5, 2013 at 4:03 PM
To: user@cassandra.apache.org
Subject: Re: Drop keyspace via CQL hanging on master/trunk.
Hey Brian, just out of curiosity, why would you remove cassandra data
directory entirely?
/Jason
On Fri, Dec 6, 2013 at 2:38 AM, Brian O'Neill b...@alumni.brown.edu wrote:
When running Cassandra from trunk/master, I see a drop keyspace command
hang at the CQL prompt.
To reproduce:
1) Removed my cassandra data directory entirely
2) Fired up cqlsh, and executed the following CQL commands in succession:
bone@zen:~/git/boneill42/cassandra- bin/cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 4.1.0 | Cassandra 2.1-SNAPSHOT | CQL spec 3.1.1 | Thrift protocol
19.38.0]
Use HELP for help.
cqlsh describe keyspaces;
system system_traces
cqlsh create keyspace test_keyspace with replication =3D {'class':'SimpleS=
trategy', 'replication_factor':'1'};
cqlsh describe keyspaces;
system test_keyspace system_traces
cqlsh drop keyspace test_keyspace;
THIS HANGS INDEFINITELY
thoughts? user error? worth filing an issue?
One other note — this happens using the CQL java driver as well.
-brian
---
Brian O'Neill
Chief Architect
Health Market Science
The Science of Better Results
2700 Horizon Drive • King of Prussia, PA • 19406
M: 215.588.6024 • @boneill42 •
healthmarketscience.com
This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If
you received this email in error and are not the intended recipient, or the
person responsible to deliver it to the intended recipient, please contact
the sender at the email above and delete this email and any attachments and
destroy any copies thereof. Any review, retransmission, dissemination,
copying or other use of, or taking any action in reliance upon, this
information by persons or entities other than the intended recipient is
strictly prohibited.
Re: Try to configure commitlog_archiving.properties
On Tue, Dec 10, 2013 at 1:45 AM, Bonnet Jonathan jonathan.bon...@externe.bnpparibas.com wrote: Taking a look to the code ? i'm not a develloper but a DBA, where should i look ? Thank you. In all seriousness, if you plan to operate Cassandra, get used to the idea of reading Java source code. There are many corners of the codebase whose behavior is only understandable via this method. Especially if you want to be sure of how the feature works in the exact version you are running. The commit log archival stuff, which IIRC was contributed by Netflix specifically in order to support a feature of Priam, is likely a good example. =Rob
Re: list all nodes as seeds (excluding self)
Comments added, not sure about the usefulness seeing as the issue is already "resolved" :) -Anne On 12/10/2013 03:12 PM, Robert Coli wrote: On Tue, Dec 10, 2013 at 5:58 AM, Anne Sullivan anne.b.sulli...@alcatel-lucent.com wrote: My understanding is that a node won't auto-bootstrap if it thinks it's a seed node. So when adding a new node to an existing cluster, I want to make sure it will auto-bootstrap, and I don't want to do 2 edits to the config file (first start without node as seed, then add node as seed). Your input, as an operator who doesn't want to do this illogical and error prone conf file dance, is welcome here : https://issues.apache.org/jira/browse/CASSANDRA-5836 "Seed nodes should be able to bootstrap without manual intervention" =Rob
2 nodes cassandra cluster raid10 or JBOD
Hi all, I need to setup 2 nodes Cassandra cluster. I know that Datastax recommends using JBOD as a disk configuration and have replication for the redundancy. I was planning to use RAID 10 but using JBOD can save 50% disk space and increase the performance . But I am not sure I should use JBOD with 2 nodes cluster since there is a higher chance to lose 50% of our cluster compare to a larger cluster. I may prefer to have stronger nodes if I have limited number of nodes. What do you think about that? Is there anyone who has 2 nodes cluster? Best Regards, Cem
Re: What is the fastest way to get data into Cassandra 2 from a Java application?
I can’t speak for Astyanax; their thrift transport I believe is abstracted out, however the object model is very CF wide row vs table-y. I have no idea what the plans are for further Astyanax dev (maybe someone on this list), but I believe the thrift API is not going away, so considering Astyanax/thrift is an option, thought I’d imagine you wouldn’t gain much going down the CQL over thrift method, so you need to be able to model your data in “internal” form. Two reasons we may want to move to the binary protocol for reads: asynchronous ability (which is now in thrift but it seems unlikely to be utilized in cassandra) for writes: compression, since we are (currently) network bandwidth limited for enormous batch inserts (from hadoop) On Dec 10, 2013, at 6:44 AM, David Tinker david.tin...@gmail.com wrote: Hmm. I have read that the thrift interface to Cassandra is out of favour and the CQL interface is in. Where does that leave Astyanax? On Tue, Dec 10, 2013 at 1:14 PM, graham sanderson gra...@vast.com wrote: Perhaps not the way forward, however I can bulk insert data via astyanax at a rate that maxes out our (fast) networks. That said for our next release (of this part of our product - our other current is node.js via binary protocol) we will be looking at insert speed via java driver, and also alternative scala/java implementations of the binary protocol. On Dec 10, 2013, at 4:49 AM, David Tinker david.tin...@gmail.com wrote: I have tried the DataStax Java driver and it seems the fastest way to insert data is to compose a CQL string with all parameters inline. This loop takes 2500ms or so on my test cluster: PreparedStatement ps = session.prepare(INSERT INTO perf_test.wibble (id, info) VALUES (?, ?)) for (int i = 0; i 1000; i++) session.execute(ps.bind( + i, aa + i)); The same loop with the parameters inline is about 1300ms. It gets worse if there are many parameters. I know I can use batching to insert all the rows at once but thats not the purpose of this test. I also tried using session.execute(cql, params) and it is faster but still doesn't match inline values. Composing CQL strings is certainly convenient and simple but is there a much faster way? Thanks David I have also posted this on Stackoverflow if anyone wants the points: http://stackoverflow.com/questions/20491090/what-is-the-fastest-way-to-get-data-into-cassandra-2-from-a-java-application -- http://qdb.io/ Persistent Message Queues With Replay and #RabbitMQ Integration smime.p7s Description: S/MIME cryptographic signature
About Cassandra-Hadoop(Pig) Integration issue
Hello Cassandra users, For one of our our new Big data BI projects, we are using Apache Cassandra 1.2.10 as our primary data store with the support of Hadoop for analytics. For prototyping purpose we have 1 node each for Apache Cassandra/Hadoop. Pig is our choice to process the data from/to C*. Before coming to the actual problem, for one of the CFs, primary key is a composite one. Eg: ((A,B,C),D). Based on the documentation combination of (A,B,C) is our partition key. We are facing couple of issues with Pig talking to C*. Most important are; *Problem 1:* we are not able to apply filters on the columns which are part of partition key. This seems to be very basic requirement to have for C*-Pig integration. To give you more idea on the problem, here is the jira on the same topic. https://issues.apache.org/jira/browse/CASSANDRA-6151 Unfortunately the status of the issue is resolved with resolution as Won't Fix. Not sure why. Because of this we are dead in the water. No clue how to proceed. Someone from community suggested to use filter using Pig. But that seems to be not a idea case as we load whole data from Cassandra and then applying filters before processing. isn't? As per my understanding, If we avoid using filter by partition key columns on Cassandra query then there will be scan for data across clusters. *Problem 2:* If there is filter just on a secondary index column, we get timeouts. Can anyone help me out to solve the above problems. Thanks, Pradeep
Re: nodetool repair keeping an empty cluster busy
Corollary: what is getting shipped over the wire? The ganglia screenshot shows the network traffic on all the three hosts on which I ran the nodetool repair. [image: Inline image 1] remember UN 10.1.2.11 107.47 KB 256 32.9% 1f800723-10e4-4dcd-841f-73709a81d432 rack1 UN 10.1.2.10 127.67 KB 256 32.4% bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639 rack1 UN 10.1.2.12 107.62 KB 256 34.7% 5258f178-b20e-408f-a7bf-b6da2903e026 rack1 Much appreciated. Sven On Wed, Dec 11, 2013 at 3:56 PM, Sven Stark sven.st...@m-square.com.auwrote: Howdy! Not a matter of life or death, just curious. I've just stood up a three node cluster (v1.2.8) on three c3.2xlarge boxes in AWS. Silly me forgot the correct replication factor for one of the needed keyspaces. So I changed it via cli and ran a nodetool repair. Well .. there is no data at all in the keyspace yet, only the definition and nodetool repair ran about 20minutes using 2 of the 8 CPU fully. Any hints what nodetool repair is doing on an empty cluster that makes the host spin so hard? Cheers, Sven == Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie Cpu(s): 22.7%us, 1.0%sy, 2.9%ni, 73.0%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st Mem: 15339196k total, 7474360k used, 7864836k free, 251904k buffers Swap:0k total,0k used,0k free, 798324k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 10840 cassandr 20 0 8354m 4.1g 19m S 218 28.0 35:25.73 jsvc 16675 kafka 20 0 3987m 192m 12m S2 1.3 0:47.89 java 20328 root 20 0 5613m 569m 16m S2 3.8 1:35.13 jsvc 5969 exhibito 20 0 6423m 116m 12m S1 0.8 0:25.87 java 14436 tomcat7 20 0 3701m 167m 11m S1 1.1 0:25.80 java 6278 exhibito 20 0 6487m 119m 9984 S0 0.8 0:22.63 java 17713 storm 20 0 6033m 159m 11m S0 1.1 0:10.99 java 18769 storm 20 0 5773m 156m 11m S0 1.0 0:10.71 java root@xxx-01:~# nodetool -h `hostname` status Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- AddressLoad Tokens Owns Host ID Rack UN 10.1.2.11 107.47 KB 256 32.9% 1f800723-10e4-4dcd-841f-73709a81d432 rack1 UN 10.1.2.10 127.67 KB 256 32.4% bd6b2059-e9dc-4b01-95ab-d7c4fc0ec639 rack1 UN 10.1.2.12 107.62 KB 256 34.7% 5258f178-b20e-408f-a7bf-b6da2903e026 rack1 root@xxx-01:~# nodetool -h `hostname` compactionstats pending tasks: 1 compaction typekeyspace column family completed total unit progress Active compaction remaining time :n/a root@xxx-01:~# nodetool -h `hostname` netstats Mode: NORMAL Not sending any streams. Not receiving any streams. Read Repair Statistics: Attempted: 0 Mismatch (Blocking): 0 Mismatch (Background): 0 Pool NameActive Pending Completed Commandsn/a 0 57155 Responses n/a 0 14573 image.png
Re: What is the fastest way to get data into Cassandra 2 from a Java application?
The session.execute blocks until the C* returns the response. Use the async
version, but do so with caution. If you don't throttle the requests, you
will start seeing timeouts on the client side pretty quickly. For
throttling I've used a Semaphore, but I think Guava's RateLimiter is better
suited. And if you want to wait until all the writes have finished,
definitely use Guava's futures API. Try something like,
PreparedStatement ps = session.prepare(INSERT INTO perf_test.wibble
(id, info) VALUES (?, ?));
RateLimiter permits = RateLimiter.create(500);// you will need to tune
this to your environment
int count = 1000;
final CountDownLatch latch = new CountDownLatch(count);
for (int i = 0; i count; i++) {
ResultSetFuture future = session.executeAsync(ps.bind( + i, aa +
i));
Futures.addCallback(future, new FutureCallbackResultSet() {
public void onSuccess(ResultSet rows) {
latch.countDown();
}
public void onFailure(Throwable t) {
latch.countDown();
// log the error or other error handling
}
});
}
latch.await(); // need to handle and/or throw InterruptedException
On Tue, Dec 10, 2013 at 8:16 PM, graham sanderson gra...@vast.com wrote:
I can’t speak for Astyanax; their thrift transport I believe is abstracted
out, however the object model is very CF wide row vs table-y.
I have no idea what the plans are for further Astyanax dev (maybe someone
on this list), but I believe the thrift API is not going away, so
considering Astyanax/thrift is an option, thought I’d imagine you wouldn’t
gain much going down the CQL over thrift method, so you need to be able to
model your data in “internal” form.
Two reasons we may want to move to the binary protocol
for reads: asynchronous ability (which is now in thrift but it seems
unlikely to be utilized in cassandra)
for writes: compression, since we are (currently) network bandwidth
limited for enormous batch inserts (from hadoop)
On Dec 10, 2013, at 6:44 AM, David Tinker david.tin...@gmail.com wrote:
Hmm. I have read that the thrift interface to Cassandra is out of
favour and the CQL interface is in. Where does that leave Astyanax?
On Tue, Dec 10, 2013 at 1:14 PM, graham sanderson gra...@vast.com
wrote:
Perhaps not the way forward, however I can bulk insert data via
astyanax at a rate that maxes out our (fast) networks. That said for our
next release (of this part of our product - our other current is node.js
via binary protocol) we will be looking at insert speed via java driver,
and also alternative scala/java implementations of the binary protocol.
On Dec 10, 2013, at 4:49 AM, David Tinker david.tin...@gmail.com
wrote:
I have tried the DataStax Java driver and it seems the fastest way to
insert data is to compose a CQL string with all parameters inline.
This loop takes 2500ms or so on my test cluster:
PreparedStatement ps = session.prepare(INSERT INTO perf_test.wibble
(id, info) VALUES (?, ?))
for (int i = 0; i 1000; i++) session.execute(ps.bind( + i, aa +
i));
The same loop with the parameters inline is about 1300ms. It gets
worse if there are many parameters. I know I can use batching to
insert all the rows at once but thats not the purpose of this test. I
also tried using session.execute(cql, params) and it is faster but
still doesn't match inline values.
Composing CQL strings is certainly convenient and simple but is there
a much faster way?
Thanks
David
I have also posted this on Stackoverflow if anyone wants the points:
http://stackoverflow.com/questions/20491090/what-is-the-fastest-way-to-get-data-into-cassandra-2-from-a-java-application
--
http://qdb.io/ Persistent Message Queues With Replay and #RabbitMQ
Integration
--
- John
