Re: memtable mem usage off by 10?
I’m using the datastax rpms, using the bundled launch scripts. grep -i jamm * cassandra-env.sh:# add the jamm javaagent cassandra-env.sh:JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar” And it’s part of the commandline used to start cassandra: /usr/java/latest/bin/java -ea -javaagent:/usr/share/cassandra//lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms10G -Xmx10G -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss240k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=4 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSIncrementalMode -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.7.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.7.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.7.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.3.jar org.apache.cassandra.service.CassandraDaemon From: Benedict Elliott Smith belliottsm...@datastax.commailto:belliottsm...@datastax.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Wednesday 4 June 2014 17:18 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: memtable mem usage off by 10? In that case I would assume the problem is that for some reason JAMM is failing to load, and so the liveRatio it would ordinarily calculate is defaulting to 10 - are you using the bundled cassandra launch scripts? On 4 June 2014 15:51, Idrén, Johan johan.id...@dice.semailto:johan.id...@dice.se wrote: I wasn’t supplying it, I was assuming it was using the default. It does not exist in my config file. Sorry for the confusion. From: Benedict Elliott Smith belliottsm...@datastax.commailto:belliottsm...@datastax.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Wednesday 4 June 2014 16:36 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: memtable mem usage off by 10? Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry, I was going by the documentation. It claims that the property is around in 2.0. But something else is wrong, as Cassandra will crash if you supply an invalid property, implying it's not sourcing the config file you're using. I'm afraid I don't have the context for why it was removed, but it happened as part of the 2.0 release. On 4 June 2014 13:59, Jack Krupansky j...@basetechnology.commailto:j...@basetechnology.com wrote: Yeah, it is in the doc: http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html And I don’t find a Jira issue mentioning it being removed, so... what’s the full story there?! -- Jack Krupansky From: Idrén, Johanmailto:johan.id...@dice.se Sent: Wednesday, June 4, 2014 8:26 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: RE: memtable mem usage off by 10? Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry, I was going by the documentation. It claims that the
CQLSSTableWriter memory leak
I am using Cassandra's CQLSSTableWriter to import a large amount of data into
Cassandra. When I use CQLSSTableWriter to write to a table with compound
primary key, the memory consumption keeps growing. The GC of JVM cannot collect
any used memory. When writing to tables with no compound primary key, the JVM
GC works fine.
My Cassandra version is 2.0.5. The OS is Ubuntu 14.04 x86-64. JVM parameters
are -Xms1g -Xmx2g. This is sufficient for all other non-compound primary key
cases.
The problem can be reproduced by the following test case:
import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.exceptions.InvalidRequestException;
import java.io.IOException;
import java.util.UUID;
class SS {
public static void main(String[] args) {
String schema = create table test.t (x uuid, y uuid, primary key (x,
y));
String insert = insert into test.t (x, y) values (?, ?);
CQLSSTableWriter writer = CQLSSTableWriter.builder()
.inDirectory(/tmp/test/t)
.forTable(schema).withBufferSizeInMB(32)
.using(insert).build();
UUID id = UUID.randomUUID();
try {
for (int i = 0; i 5000; i++) {
UUID id2 = UUID.randomUUID();
writer.addRow(id, id2);
}
writer.close();
} catch (Exception e) {
System.err.println(hell);
}
}
}
Re: Consolidating records and TTL
As Tyler says, with atomic batches which are enabled by default the cluster will keep trying to replay the insert / deletes. Nodes check their local batch log for failed batches, ones where the coordinator did not acknowledge it had successfully completed, every 60 seconds. So there is a window where it’s possible for not all mutations in the batch to be completed. This could happen when a write timeout occurs when processing a batch of 2 rows; the request CL will not have been achieved on one or more of the rows. The coordinator will leave it up to the batch log to replay the request, and the client driver will (by default config) not retry. You can use a model like this. create table ledger ( account int, tx_id timeuuid, sub_total int, primary key (account, tx_id) ); create table account ( account int, total int, last_tx_id timeuuid, primary key (account) ); To get the total: select * from account where account = X; Then get the ledger entries you need select * from ledger where account = X and tx_id last_tx_id; This query will degrade when the partition size in the ledger table gets bigger, as it will need to read the column index (see column_index_size_in_kb in yaml). It will use that to find the first page that contains the rows we are interested in and then read forwards to the end of the row. It’s not the most efficient type of read but if you are going to delete ledger entries this *should* be able to skip over the tombstones without reading them. When you want to update the total in the account write to the account table and update both the total and the last_tx_id. You can then delete ledger entries if needed. Don’t forget to ensure that only one client thread is doing this at a time. Hope that helps. Aaron - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 5/06/2014, at 10:37 am, Tyler Hobbs ty...@datastax.com wrote: Just use an atomic batch that holds both the insert and deletes: http://www.datastax.com/dev/blog/atomic-batches-in-cassandra-1-2 On Tue, Jun 3, 2014 at 2:13 PM, Charlie Mason charlie@gmail.com wrote: Hi All. I have a system thats going to make possibly several concurrent changes to a running total. I know I could use a counter for this. However I have extra meta data I can store with the changes which would allow me to reply the changes. If I use a counter and it looses some writes I can't recover it as I will only have its current total not the extra meta data to know where to replay from. What I was planning to do was write each change of the value to a CQL table with a Time UUID as a row level primary key as well as a partition key. Then when I need to read the running total back I will do a query for all the changes and add them up to get the total. As there could be tens of thousands of these I want to have a period after which these are consolidated. Most won't be any where near that but a few will which I need to be able to support. So I was also going to have a consolidated total table which holds the UUID of the values consolidated up to. Since I can bound the query for the recent updates by the UUID I should be able to avoid all the tombstones. So if the read encounters any changes that can be consolidated it inserts a new consolidated value and deletes the newly consolidated changes. What I am slightly worried about is what happens if the consolidated value insert fails but the deletes to the change records succeed. I would be left with an inconsistent total indefinitely. I have come up with a couple of ideas: 1, I could make it require all nodes to acknowledge it before deleting the difference records. 2, May be I could have another period after its consolidated but before its deleted? 3, Is there anyway I could use the TTL to allow to it to be deleted after a period of time? Chances are another read would come in and fix the value. Anyone got any other suggestions on how I could implement this? Thanks, Charlie M -- Tyler Hobbs DataStax
VPC AWS
Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information...
Re: VPC AWS
Using 2.0.x, we hit https://issues.apache.org/jira/browse/CASSANDRA-2380 when deploying in a VPC. We worked around this by adding a hostname in /etc/hosts. Apart from that, nothing specific. We use only internal ip's, no public ip's and no broadcast etc. It works very well. /Johan From: Alain RODRIGUEZ arodr...@gmail.commailto:arodr...@gmail.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Thursday 5 June 2014 14:12 To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: VPC AWS Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information...
Re: VPC AWS
Hello Alain, We switched from EC2 to VPC a couple of years ago. The process for us was long, slow, and multi step for our (at the time) 6 node cluster. In our case, we don't need to consider multi-DC. However, in our infrastructure we were rapidly running out of IP addresses, and wished to move to VPC to give us a nearly inexhaustible supply. In addition, AWS VPC gives us an additional layer of security for our Cassandra cluster. To do this, we setup our VPC to have both private and public subnets. Public subnets were accessible to the Internet (when instances were assigned a public IP), while private subnets could not (although instances on the subnet could access the Internet via a NAT instance). We wished for to be Cassandra on the private subnet. However, this introduced a complication. EC2 instances would not be able to communicate directly to our VPC instances on a private subnet. So, to achieve this, while still having an operating Cassandra DB without downtime, we essentially had to stage Cassandra instances on our public subnet, assigning IPs and reconfiguring nodes until we had a mixed EC2/VPC Public subnet cluster, then start moving systems to the private subnet, continuing the process until all instances were on a private subnet. During the process we carefully orchestrated configuration like broadcast and seeds to make sure the cluster continued to function properly and all nodes could communicate with each other. We also had to carefully orchestrate the assigning of AWS security groups to make sure everyone could talk to each other during this process. Also keep in mind that the use of public IPs for communications will add to your AWS costs. During our transition we had to do this for a short time while EC2 instances were communicating with VPC instances, but we were able to switch to 100% internal IPs when we completed (you will still get inter availability zone charges regardless) This process was complex enough that I wrote detailed series of steps, for each node in our cluster. -Mike From: Alain RODRIGUEZ arodr...@gmail.com To: user@cassandra.apache.org Sent: Thursday, June 5, 2014 8:12 AM Subject: VPC AWS Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information...
Re: VPC AWS
Hello Alain, We switched from EC2 to VPC a couple of years ago. The process for us was long, slow and multi step. In our case, we don't need to consider multi-DC. However, in our infrastructure we were rapidly running out of IP addresses, and wished to move to VPC to give us a nearly inexhaustible supply. In addition, AWS VPC gives us an additional layer of security for our Cassandra cluster. To do this, we setup our VPC to have both private and public subnets. Public subnets were accessible to the Internet (when instances were assigned a public IP, while private subnets could not (although instances on the subnet could access the Internet via a NAT instance). We wished for Cassandra on the private subnet. So, to achieve this, while still having an operating Cassandra DB without downtime, we essentially had to stage Cassandra instances on our public subnet, assigning IPs and reconfiguring nodes until we had a mixed EC2/VPC Public cluster, then start moving systems to the private subnet, continuing the process until all instances were on a private subnet. During the process we carefully orchestrated configuration like broadcast and seeds to make sure the cluster continued to function properly. We also had to orchestrate the assigning of AWS security groups to make sure everyone could talk to each other during this process. Also keep in mind that the use of public IPs for communications will add to your AWS costs. During our transition we had to do this for a short time while EC2 instances were communicating with VPC instances, but we were able to switch to 100% internal IPs when we completed (you will still get inter availability zone charges regardless) In order to make this successful, I created a script out From: Alain RODRIGUEZ arodr...@gmail.com To: user@cassandra.apache.org Sent: Thursday, June 5, 2014 8:12 AM Subject: VPC AWS Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information...
Re: VPC AWS
I don't think traffic will flow between classic ec2 and vpc directly. There is some kind of gateway bridge instance that sits between, acting as a NAT. I would think that would cause new challenges for: -transitions -clients Sorry this response isn't heavy on content! I'm curious how this thread goes... Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information... -- Will Oberman Civic Science, Inc. 6101 Penn Avenue, Fifth Floor Pittsburgh, PA 15206 (M) 412-480-7835 (E) ober...@civicscience.com
Re: VPC AWS
I think you can define VPC subnet to be public (to have public + private IPs) or private only. Any insight regarding snitches ? What snitch do you guys use ? 2014-06-05 15:06 GMT+02:00 William Oberman ober...@civicscience.com: I don't think traffic will flow between classic ec2 and vpc directly. There is some kind of gateway bridge instance that sits between, acting as a NAT. I would think that would cause new challenges for: -transitions -clients Sorry this response isn't heavy on content! I'm curious how this thread goes... Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information... -- Will Oberman Civic Science, Inc. 6101 Penn Avenue, Fifth Floor Pittsburgh, PA 15206 (M) 412-480-7835 (E) ober...@civicscience.com
Re: VPC AWS
A general thought, if you're using AWS for this, I'd strongly recommend you consider using OpsWorks and custom chef recipes for your node deployment if its an option for you. The easy of provisioning new nodes without the need for snapshotting is certainly worth the hassle, and there are already several Cassandra chef recipes available with a little searching. David On Thu, Jun 5, 2014 at 9:26 AM, William Oberman ober...@civicscience.com wrote: I was making assumptions (sorry!), namely the goal to switching to vpc was getting backend resources (like cassandra) off the public internet. Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: I think you can define VPC subnet to be public (to have public + private IPs) or private only. Any insight regarding snitches ? What snitch do you guys use ? 2014-06-05 15:06 GMT+02:00 William Oberman ober...@civicscience.com: I don't think traffic will flow between classic ec2 and vpc directly. There is some kind of gateway bridge instance that sits between, acting as a NAT. I would think that would cause new challenges for: -transitions -clients Sorry this response isn't heavy on content! I'm curious how this thread goes... Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information... -- Will Oberman Civic Science, Inc. 6101 Penn Avenue, Fifth Floor Pittsburgh, PA 15206 (M) 412-480-7835 (E) ober...@civicscience.com -- Will Oberman Civic Science, Inc. 6101 Penn Avenue, Fifth Floor Pittsburgh, PA 15206 (M) 412-480-7835 (E) ober...@civicscience.com
Re: VPC AWS
Well if you are talking about the transition I think we are going to do it this way - Existing cluster out of any VPC - DC1 - New VPC cluster, with subnet private + public - DC2 - Add / Rebuild DC2 - Switch clients from DC1 to DC2 - Remove DC1 - Switch client inside the VPC - Remove public VPC subnet -- Issue in this last step how to switch cassandra configuation from public to private IPs without any downtime ? What do you think about this approach ? 2014-06-05 15:29 GMT+02:00 David McNelis dmcne...@gmail.com: A general thought, if you're using AWS for this, I'd strongly recommend you consider using OpsWorks and custom chef recipes for your node deployment if its an option for you. The easy of provisioning new nodes without the need for snapshotting is certainly worth the hassle, and there are already several Cassandra chef recipes available with a little searching. David On Thu, Jun 5, 2014 at 9:26 AM, William Oberman ober...@civicscience.com wrote: I was making assumptions (sorry!), namely the goal to switching to vpc was getting backend resources (like cassandra) off the public internet. Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: I think you can define VPC subnet to be public (to have public + private IPs) or private only. Any insight regarding snitches ? What snitch do you guys use ? 2014-06-05 15:06 GMT+02:00 William Oberman ober...@civicscience.com: I don't think traffic will flow between classic ec2 and vpc directly. There is some kind of gateway bridge instance that sits between, acting as a NAT. I would think that would cause new challenges for: -transitions -clients Sorry this response isn't heavy on content! I'm curious how this thread goes... Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information... -- Will Oberman Civic Science, Inc. 6101 Penn Avenue, Fifth Floor Pittsburgh, PA 15206 (M) 412-480-7835 (E) ober...@civicscience.com -- Will Oberman Civic Science, Inc. 6101 Penn Avenue, Fifth Floor Pittsburgh, PA 15206 (M) 412-480-7835 (E) ober...@civicscience.com
Re: memtable mem usage off by 10?
What does /usr/java/latest/bin/java -version print? On 5 June 2014 08:15, Idrén, Johan johan.id...@dice.se wrote: I’m using the datastax rpms, using the bundled launch scripts. grep -i jamm * cassandra-env.sh:# add the jamm javaagent cassandra-env.sh:JVM_OPTS=$JVM_OPTS -javaagent:$CASSANDRA_HOME/lib/jamm-0.2.5.jar” And it’s part of the commandline used to start cassandra: /usr/java/latest/bin/java -ea -javaagent:/usr/share/cassandra//lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms10G -Xmx10G -Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss240k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=4 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSIncrementalMode -XX:+UseCondCardMark -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dlog4j.configuration=log4j-server.properties -Dlog4j.defaultInitOverride=true -Dcassandra-pidfile=/var/run/cassandra/cassandra.pid -cp /etc/cassandra/conf:/usr/share/java/jna.jar:/usr/share/cassandra/lib/antlr-3.2.jar:/usr/share/cassandra/lib/apache-cassandra-2.0.7.jar:/usr/share/cassandra/lib/apache-cassandra-clientutil-2.0.7.jar:/usr/share/cassandra/lib/apache-cassandra-thrift-2.0.7.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.3.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-15.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.1.2.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.2.5.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.1.jar:/usr/share/cassandra/lib/log4j-1.2.16.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-3.6.6.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/servlet-api-2.5-20081211.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/slf4j-log4j12-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.jar:/usr/share/cassandra/lib/snaptree-0.1.jar:/usr/share/cassandra/lib/stress.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.3.jar org.apache.cassandra.service.CassandraDaemon From: Benedict Elliott Smith belliottsm...@datastax.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Wednesday 4 June 2014 17:18 To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: memtable mem usage off by 10? In that case I would assume the problem is that for some reason JAMM is failing to load, and so the liveRatio it would ordinarily calculate is defaulting to 10 - are you using the bundled cassandra launch scripts? On 4 June 2014 15:51, Idrén, Johan johan.id...@dice.se wrote: I wasn’t supplying it, I was assuming it was using the default. It does not exist in my config file. Sorry for the confusion. From: Benedict Elliott Smith belliottsm...@datastax.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Wednesday 4 June 2014 16:36 To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: memtable mem usage off by 10? Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry, I was going by the documentation. It claims that the property is around in 2.0. But something else is wrong, as Cassandra will crash if you supply an invalid property, implying it's not sourcing the config file you're using. I'm afraid I don't have the context for why it was removed, but it happened as part of the 2.0 release. On 4 June 2014 13:59, Jack Krupansky j...@basetechnology.com wrote: Yeah, it is in the doc: http://www.datastax.com/documentation/cassandra/2.0/cassandra/configuration/configCassandra_yaml_r.html And I don’t find a Jira issue mentioning it being removed, so... what’s the full story there?! -- Jack Krupansky *From:* Idrén, Johan johan.id...@dice.se *Sent:* Wednesday, June 4, 2014 8:26 AM *To:* user@cassandra.apache.org *Subject:* RE: memtable mem usage off by 10? Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry, I was going by the documentation. It claims that the property is around in 2.0. If we skip that, part of my reply still makes sense: Having memtable_total_size_in_mb set to 20480, memtables are flushed at a reported value of ~2GB. With a constant overhead of ~10x, as suggested,
Re: VPC AWS
We personally use the EC2Snitch, however, we don't have the multi-region requirements you do, -Mike From: Alain RODRIGUEZ arodr...@gmail.com To: user@cassandra.apache.org Sent: Thursday, June 5, 2014 9:14 AM Subject: Re: VPC AWS I think you can define VPC subnet to be public (to have public + private IPs) or private only. Any insight regarding snitches ? What snitch do you guys use ? 2014-06-05 15:06 GMT+02:00 William Oberman ober...@civicscience.com: I don't think traffic will flow between classic ec2 and vpc directly. There is some kind of gateway bridge instance that sits between, acting as a NAT. I would think that would cause new challenges for: -transitions -clients Sorry this response isn't heavy on content! I'm curious how this thread goes... Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information... -- Will Oberman Civic Science, Inc. 6101 Penn Avenue, Fifth Floor Pittsburgh, PA 15206 (M) 412-480-7835 (E) ober...@civicscience.com
Migration 1.2.14 to 2.0.8 causes Tried to create duplicate hard link at startup
Hi, I'm trying to migrate a development cluster from 1.2.14 to 2.0.8. When starting up 2.0.8, I'm seeing the following error in the logs: INFO 17:40:25,405 Snapshotting drillster, Account to pre-sstablemetamigration ERROR 17:40:25,407 Exception encountered during startup java.lang.RuntimeException: Tried to create duplicate hard link to /Users/tom/cassandra-data/data/drillster/Account/snapshots/pre-sstablemetamigration/drillster-Account-ic-65-Filter.db at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:75) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.snapshotWithoutCFS(LegacyLeveledManifest.java:129) at org.apache.cassandra.db.compaction.LegacyLeveledManifest.migrateManifests(LegacyLeveledManifest.java:91) at org.apache.cassandra.db.compaction.LeveledManifest.maybeMigrateManifests(LeveledManifest.java:617) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:274) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) Does anyone have an idea how to solve this? Thanks, Tom
Re: CQLSSTableWriter memory leak
How many rows (primary key values) are you writing for each partition of the
primary key? I mean, are there relatively few, or are these very wide
partitions?
Oh, I see! You’re writing 50,000,000 rows to a single partition! My, that IS
ambitious.
-- Jack Krupansky
From: Xu Zhongxing
Sent: Thursday, June 5, 2014 3:34 AM
To: user@cassandra.apache.org
Subject: CQLSSTableWriter memory leak
I am using Cassandra's CQLSSTableWriter to import a large amount of data into
Cassandra. When I use CQLSSTableWriter to write to a table with compound
primary key, the memory consumption keeps growing. The GC of JVM cannot collect
any used memory. When writing to tables with no compound primary key, the JVM
GC works fine.
My Cassandra version is 2.0.5. The OS is Ubuntu 14.04 x86-64. JVM parameters
are -Xms1g -Xmx2g. This is sufficient for all other non-compound primary key
cases.
The problem can be reproduced by the following test case:
import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.exceptions.InvalidRequestException;
import java.io.IOException;
import java.util.UUID;
class SS {
public static void main(String[] args) {
String schema = create table test.t (x uuid, y uuid, primary key (x,
y));
String insert = insert into test.t (x, y) values (?, ?);
CQLSSTableWriter writer = CQLSSTableWriter.builder()
.inDirectory(/tmp/test/t)
.forTable(schema).withBufferSizeInMB(32)
.using(insert).build();
UUID id = UUID.randomUUID();
try {
for (int i = 0; i 5000; i++) {
UUID id2 = UUID.randomUUID();
writer.addRow(id, id2);
}
writer.close();
} catch (Exception e) {
System.err.println(hell);
}
}
}
Re: migration to a new model
Michael, I will try to test it up to tomorrow and I will let you know all the results. Thanks a lot! Best regards, Marcelo. 2014-06-04 22:28 GMT-03:00 Laing, Michael michael.la...@nytimes.com: BTW you might want to put a LIMIT clause on your SELECT for testing. -ml On Wed, Jun 4, 2014 at 6:04 PM, Laing, Michael michael.la...@nytimes.com wrote: Marcelo, Here is a link to the preview of the python fast copy program: https://gist.github.com/michaelplaing/37d89c8f5f09ae779e47 It will copy a table from one cluster to another with some transformation- they can be the same cluster. It has 3 main throttles to experiment with: 1. fetch_size: size of source pages in rows 2. worker_count: number of worker subprocesses 3. concurrency: number of async callback chains per worker subprocess It is easy to overrun Cassandra and the python driver, so I recommend starting with the defaults: fetch_size: 1000; worker_count: 2; concurrency: 10. Additionally there are switches to set 'policies' by source and destination: retry (downgrade consistency), dc_aware, and token_aware. retry is useful if you are getting timeouts. For the others YMMV. To use it you need to define the SELECT and UPDATE cql statements as well as the 'map_fields' method. The worker subprocesses divide up the token range among themselves and proceed quasi-independently. Each worker opens a connection to each cluster and the driver sets up connection pools to the nodes in the cluster. Anyway there are a lot of processes, threads, callbacks going at once so it is fun to watch. On my regional cluster of small nodes in AWS I got about 3000 rows per second transferred after things warmed up a bit - each row about 6kb. ml On Wed, Jun 4, 2014 at 11:49 AM, Laing, Michael michael.la...@nytimes.com wrote: OK Marcelo, I'll work on it today. -ml On Tue, Jun 3, 2014 at 8:24 PM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br wrote: Hi Michael, For sure I would be interested in this program! I am new both to python and for cql. I started creating this copier, but was having problems with timeouts. Alex solved my problem here on the list, but I think I will still have a lot of trouble making the copy to work fine. I open sourced my version here: https://github.com/s1mbi0se/cql_record_processor Just in case it's useful for anything. However, I saw CQL has support for concurrency itself and having something made by someone who knows Python CQL Driver better would be very helpful. My two servers today are at OVH (ovh.com), we have servers at AWS but but several cases we prefer other hosts. Both servers have SDD and 64 Gb RAM, I could use the script as a benchmark for you if you want. Besides, we have some bigger clusters, I could run on the just to test the speed if this is going to help. Regards Marcelo. 2014-06-03 11:40 GMT-03:00 Laing, Michael michael.la...@nytimes.com: Hi Marcelo, I could create a fast copy program by repurposing some python apps that I am using for benchmarking the python driver - do you still need this? With high levels of concurrency and multiple subprocess workers, based on my current actual benchmarks, I think I can get well over 1,000 rows/second on my mac and significantly more in AWS. I'm using variable size rows averaging 5kb. This would be the initial version of a piece of the benchmark suite we will release as part of our nyt⨍aбrik project on 21 June for my Cassandra Day NYC talk re the python driver. ml On Mon, Jun 2, 2014 at 2:15 PM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br wrote: Hi Jens, Thanks for trying to help. Indeed, I know I can't do it using just CQL. But what would you use to migrate data manually? I tried to create a python program using auto paging, but I am getting timeouts. I also tried Hive, but no success. I only have two nodes and less than 200Gb in this cluster, any simple way to extract the data quickly would be good enough for me. Best regards, Marcelo. 2014-06-02 15:08 GMT-03:00 Jens Rantil jens.ran...@tink.se: Hi Marcelo, Looks like you can't do this without migrating your data manually: https://stackoverflow.com/questions/18421668/alter-cassandra-column-family-primary-key-using-cassandra-cli-or-cql Cheers, Jens On Mon, Jun 2, 2014 at 7:48 PM, Marcelo Elias Del Valle marc...@s1mbi0se.com.br wrote: Hi, I have some cql CFs in a 2 node Cassandra 2.0.8 cluster. I realized I created my column family with the wrong partition. Instead of: CREATE TABLE IF NOT EXISTS entity_lookup ( name varchar, value varchar, entity_id uuid, PRIMARY KEY ((name, value), entity_id)) WITH caching=all; I used: CREATE TABLE IF NOT EXISTS entitylookup ( name varchar, value varchar, entity_id uuid, PRIMARY KEY (name, value, entity_id)) WITH caching=all; Now I need to migrate the data from the second CF to the first one. I am using Data Stax
Re: nodetool move seems slow
Hi Rob, THX for you response and link to the issue. The move did complete after a restart! Cheers, ~Jason *** From: Robert Coli rc...@eventbrite.commailto:rc...@eventbrite.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Wednesday, June 4, 2014 at 5:01 PM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Cc: Francois Richard frich...@yahoo-inc.commailto:frich...@yahoo-inc.com Subject: Re: nodetool move seems slow On Wed, Jun 4, 2014 at 2:34 PM, Jason Tyler jaty...@yahoo-inc.commailto:jaty...@yahoo-inc.com wrote: I wrote 'apparent progress' because it reports “MOVING” and the Pending Commands/Responses are changing over time. However, I haven’t seen the individual .db files progress go above 0%. Your move is hung. Restart the affected nodes [1] and then restart the move. =Rob [1] https://issues.apache.org/jira/browse/CASSANDRA-3486
Re: Cassandra 2.0 unbalanced ring with vnodes after adding new node
Actually, I have the same doubt. The same happens to me, but I guess it's because of lack of knowledge in Cassandra vnodes, somehow... I just added 3 nodes to my old 2 nodes cluster, now I have a 5 nodes cluster. As rows should be in a node calculated by HASH / number of nodes, adding a new node should move data from all other nodes to the new ones, right? Considering I have an enough number of different row keys. I noticed that: 1. Even reading data with read consistency = ALL, I get the wrong results while the repair is not complete. Should this happen? 2. I have run nodetool repair in each new node and nodetool cleanup in the 2 old nodes. There is some streaming happening, but it's really slow, considering my bandwith and use of SSDs. What should I do make the data stream from the old nodes to the new ones faster? And everytime I add new nodes to the cluster I will have to stop my processes that reads data from cassandra until the move is complete? Isn't there any other way? Best regards, Marcelo. 2014-06-04 13:52 GMT-03:00 Владимир Рудев vladimir.ru...@gmail.com: Hello to everyone! Please, can someone explain where we made a mistake? We have cluster with 4 nodes which uses vnodes(256 per node, default settings), snitch is default on every node: SimpleSnitch. These four nodes was from beginning of a cluster. In this cluster we have keyspace with this options: Keyspace: K: Replication Strategy: org.apache.cassandra.locator.SimpleStrategy Durable Writes: true Options: [replication_factor:3] All was normal and nodetool status K shows that each node owns 75% of all key range. All 4 nodes are located in same datacenter and have same first two bytes in IP address(others are different). Then we buy new server on different datacenter and add it to the cluster with same settings as in previous four nodes(difference only in listen_address), assuming that the effective own of each node for this keyspace will be 300/5=60% or near. But after 3-5 minutes after start nodetool status K show this: nodetool status K; Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- AddressLoad Tokens Owns (effective) Host ID Rack UN N1 6,06 GB256 50.0% 62f295b3-0da6-4854-a53a-f03d6b424b03 rack1 UN N2 5,89 GB256 50.0% af4e4a23-2610-44dd-9061-09c7a6512a54 rack1 UN N3 6,02 GB256 50.0% 0f0e4e78-6fb2-479f-ad76-477006f76795 rack1 UN N4 5,8 GB 256 50.0% 670344c0-9856-48cf-9ec9-1a98f9a89460 rack1 UN N5 7,51 GB256 100.0% 82473d14-9e36-4ae7-86d2-a3e526efb53f rack1 N5 is newly added node nodetool repair -pr on N5 doesn't change anything nodetool describering K shows that new node N5 participate in EACH range. This is not we want at all. It looks like cassandra add new node to each range because it located in different datacenter, but all settings and output are exactly prevent this. Also interesting point is that while in all config files snitch is defined as SimpleSnitch the output of the command nodetool describecluster is: Cluster Information: Name: Some Cluster Name Snitch: org.apache.cassandra.locator.*DynamicEndpointSnitch* Partitioner: org.apache.cassandra.dht.Murmur3Partitioner Schema versions: 26b8fa37-e666-31ed-aa3b-85be75f2aa1a: [N1, N2, N3, N4, N5] We use Cassandra 2.0.6 Questions we have at this moment: 1. How to rebalance ring so all nodes will own 60% of range? 1a. Removing node from cluster and adding it again is a solution? 2. Where we possibly make a mistake when adding new node? 3. If we add new 6th node to ring it will take 50% from N5 or some portion from each node? Thanks in advance! -- С уважением, Владимир Рудев (With regards, Vladimir Rudev) vladimir.ru...@gmail.com
Re: VPC AWS
The implementation of moving from EC2 to a VPC was a bit of a juggling act. Our motivation was two fold: 1) We were running out of static IP addresses, and it was becoming increasingly difficult in EC2 to design around limiting the number of static IP addresses to the number of public IP addresses EC2 allowed 2) VPC affords us an additional level of security that was desirable. However, we needed to consider the following limitations: 1) By default, you have a limited number of available public IPs for both EC2 and VPC. 2) AWS security groups need to be configured to allow traffic for Cassandra to/from instances in EC2 and the VPC. You are correct at the high level that the migration goes from EC2-Public VPC (VPC with an Internet Gateway)-Private VPC (VPC with a NAT). The first phase was moving instances to the public VPC, setting broadcast and seeds to the public IPs we had available. Basically: 1) Take down a node, taking a snapshot for a backup 2) Restore the node on the public VPC, assigning it to the correct security group, manually setting the seeds to other available nodes 3) Verify the cluster can communicate 4) Repeat Realize the NAT instance on the private subnet will also require a public IP. What got really interesting is that near the end of the process we ran out of available IPs, requiring us to switch the final node that was on EC2 directly to the private VPC (and taking down two nodes at once, which our setup allowed given we had 6 nodes with an RF of 3). What we did, and highly suggest for the switch, is to write down every step that has to happen on every node during the switch. In our case, many of the moved nodes required slightly different configurations for items like the seeds. Its been a couple of years, so my memory on this maybe a little fuzzy :) -Mike From: Aiman Parvaiz ai...@shift.com To: user@cassandra.apache.org; Michael Theroux mthero...@yahoo.com Sent: Thursday, June 5, 2014 12:55 PM Subject: Re: VPC AWS Michael, Thanks for the response, I am about to head in to something very similar if not exactly same. I envision things happening on the same lines as you mentioned. I would be grateful if you could please throw some more light on how you went about switching cassandra nodes from public subnet to private with out any downtime. I have not started on this project yet, still in my research phase. I plan to have a ec2+public VPC cluster and then decomission ec2 nodes to have everything in public subnet, next would be to move it to private subnet. Thanks On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux mthero...@yahoo.com wrote: We personally use the EC2Snitch, however, we don't have the multi-region requirements you do, -Mike From: Alain RODRIGUEZ arodr...@gmail.com To: user@cassandra.apache.org Sent: Thursday, June 5, 2014 9:14 AM Subject: Re: VPC AWS I think you can define VPC subnet to be public (to have public + private IPs) or private only. Any insight regarding snitches ? What snitch do you guys use ? 2014-06-05 15:06 GMT+02:00 William Oberman ober...@civicscience.com: I don't think traffic will flow between classic ec2 and vpc directly. There is some kind of gateway bridge instance that sits between, acting as a NAT. I would think that would cause new challenges for: -transitions -clients Sorry this response isn't heavy on content! I'm curious how this thread goes... Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We use private ones for listen address. Machines inside the VPC will only have private IP AFAIK. Should I keep using a broadcast address ? Is there any other incidence when switching to a VPC ? Sorry if the topic was already discussed, I was unable to find any useful information... -- Will Oberman Civic Science, Inc. 6101 Penn Avenue, Fifth Floor Pittsburgh, PA 15206 (M) 412-480-7835 (E) ober...@civicscience.com
Re: VPC AWS
Thanks for this info Michael. As far as restoring node in public VPC is concerned I was thinking ( and I might be wrong here) if we can have a ring spread across EC2 and public subnet of a VPC, this way I can simply decommission nodes in Ec2 as I gradually introduce new nodes in public subnet of VPC and I will end up with a ring in public subnet and then migrate them from public to private in a similar way may be. If anyone has any experience/ suggestions with this please share, would really appreciate it. Aiman On Thu, Jun 5, 2014 at 10:37 AM, Michael Theroux mthero...@yahoo.com wrote: The implementation of moving from EC2 to a VPC was a bit of a juggling act. Our motivation was two fold: 1) We were running out of static IP addresses, and it was becoming increasingly difficult in EC2 to design around limiting the number of static IP addresses to the number of public IP addresses EC2 allowed 2) VPC affords us an additional level of security that was desirable. However, we needed to consider the following limitations: 1) By default, you have a limited number of available public IPs for both EC2 and VPC. 2) AWS security groups need to be configured to allow traffic for Cassandra to/from instances in EC2 and the VPC. You are correct at the high level that the migration goes from EC2-Public VPC (VPC with an Internet Gateway)-Private VPC (VPC with a NAT). The first phase was moving instances to the public VPC, setting broadcast and seeds to the public IPs we had available. Basically: 1) Take down a node, taking a snapshot for a backup 2) Restore the node on the public VPC, assigning it to the correct security group, manually setting the seeds to other available nodes 3) Verify the cluster can communicate 4) Repeat Realize the NAT instance on the private subnet will also require a public IP. What got really interesting is that near the end of the process we ran out of available IPs, requiring us to switch the final node that was on EC2 directly to the private VPC (and taking down two nodes at once, which our setup allowed given we had 6 nodes with an RF of 3). What we did, and highly suggest for the switch, is to write down every step that has to happen on every node during the switch. In our case, many of the moved nodes required slightly different configurations for items like the seeds. Its been a couple of years, so my memory on this maybe a little fuzzy :) -Mike -- *From:* Aiman Parvaiz ai...@shift.com *To:* user@cassandra.apache.org; Michael Theroux mthero...@yahoo.com *Sent:* Thursday, June 5, 2014 12:55 PM *Subject:* Re: VPC AWS Michael, Thanks for the response, I am about to head in to something very similar if not exactly same. I envision things happening on the same lines as you mentioned. I would be grateful if you could please throw some more light on how you went about switching cassandra nodes from public subnet to private with out any downtime. I have not started on this project yet, still in my research phase. I plan to have a ec2+public VPC cluster and then decomission ec2 nodes to have everything in public subnet, next would be to move it to private subnet. Thanks On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux mthero...@yahoo.com wrote: We personally use the EC2Snitch, however, we don't have the multi-region requirements you do, -Mike -- *From:* Alain RODRIGUEZ arodr...@gmail.com *To:* user@cassandra.apache.org *Sent:* Thursday, June 5, 2014 9:14 AM *Subject:* Re: VPC AWS I think you can define VPC subnet to be public (to have public + private IPs) or private only. Any insight regarding snitches ? What snitch do you guys use ? 2014-06-05 15:06 GMT+02:00 William Oberman ober...@civicscience.com: I don't think traffic will flow between classic ec2 and vpc directly. There is some kind of gateway bridge instance that sits between, acting as a NAT. I would think that would cause new challenges for: -transitions -clients Sorry this response isn't heavy on content! I'm curious how this thread goes... Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite sure that some people are already using VPC. If you can point me to any documentation regarding VPC / Cassandra, it would be very nice of you. We have only one DC for now, but we need to remain multi DC compatible, since we will add DC very soon. Else, I would like to know if I should keep using EC2MultiRegionSnitch or change the snitch to anything else. What about broadcast/listen ip, seeds...? We currently use public ip as for broadcast address and for seeds. We
Re: VPC AWS
You can have a ring spread across EC2 and the public subnet of a VPC. That is how we did our migration. In our case, we simply replaced the existing EC2 node with a new instance in the public VPC, restored from a backup taken right before the switch. -Mike From: Aiman Parvaiz ai...@shift.com To: Michael Theroux mthero...@yahoo.com Cc: user@cassandra.apache.org user@cassandra.apache.org Sent: Thursday, June 5, 2014 2:39 PM Subject: Re: VPC AWS Thanks for this info Michael. As far as restoring node in public VPC is concerned I was thinking ( and I might be wrong here) if we can have a ring spread across EC2 and public subnet of a VPC, this way I can simply decommission nodes in Ec2 as I gradually introduce new nodes in public subnet of VPC and I will end up with a ring in public subnet and then migrate them from public to private in a similar way may be. If anyone has any experience/ suggestions with this please share, would really appreciate it. Aiman On Thu, Jun 5, 2014 at 10:37 AM, Michael Theroux mthero...@yahoo.com wrote: The implementation of moving from EC2 to a VPC was a bit of a juggling act. Our motivation was two fold: 1) We were running out of static IP addresses, and it was becoming increasingly difficult in EC2 to design around limiting the number of static IP addresses to the number of public IP addresses EC2 allowed 2) VPC affords us an additional level of security that was desirable. However, we needed to consider the following limitations: 1) By default, you have a limited number of available public IPs for both EC2 and VPC. 2) AWS security groups need to be configured to allow traffic for Cassandra to/from instances in EC2 and the VPC. You are correct at the high level that the migration goes from EC2-Public VPC (VPC with an Internet Gateway)-Private VPC (VPC with a NAT). The first phase was moving instances to the public VPC, setting broadcast and seeds to the public IPs we had available. Basically: 1) Take down a node, taking a snapshot for a backup 2) Restore the node on the public VPC, assigning it to the correct security group, manually setting the seeds to other available nodes 3) Verify the cluster can communicate 4) Repeat Realize the NAT instance on the private subnet will also require a public IP. What got really interesting is that near the end of the process we ran out of available IPs, requiring us to switch the final node that was on EC2 directly to the private VPC (and taking down two nodes at once, which our setup allowed given we had 6 nodes with an RF of 3). What we did, and highly suggest for the switch, is to write down every step that has to happen on every node during the switch. In our case, many of the moved nodes required slightly different configurations for items like the seeds. Its been a couple of years, so my memory on this maybe a little fuzzy :) -Mike From: Aiman Parvaiz ai...@shift.com To: user@cassandra.apache.org; Michael Theroux mthero...@yahoo.com Sent: Thursday, June 5, 2014 12:55 PM Subject: Re: VPC AWS Michael, Thanks for the response, I am about to head in to something very similar if not exactly same. I envision things happening on the same lines as you mentioned. I would be grateful if you could please throw some more light on how you went about switching cassandra nodes from public subnet to private with out any downtime. I have not started on this project yet, still in my research phase. I plan to have a ec2+public VPC cluster and then decomission ec2 nodes to have everything in public subnet, next would be to move it to private subnet. Thanks On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux mthero...@yahoo.com wrote: We personally use the EC2Snitch, however, we don't have the multi-region requirements you do, -Mike From: Alain RODRIGUEZ arodr...@gmail.com To: user@cassandra.apache.org Sent: Thursday, June 5, 2014 9:14 AM Subject: Re: VPC AWS I think you can define VPC subnet to be public (to have public + private IPs) or private only. Any insight regarding snitches ? What snitch do you guys use ? 2014-06-05 15:06 GMT+02:00 William Oberman ober...@civicscience.com: I don't think traffic will flow between classic ec2 and vpc directly. There is some kind of gateway bridge instance that sits between, acting as a NAT. I would think that would cause new challenges for: -transitions -clients Sorry this response isn't heavy on content! I'm curious how this thread goes... Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some questions regarding this switch and the Cassandra configuration inside a VPC. Actually I found no documentation on this topic, but I am quite
Re: VPC AWS
Cool, thanks again for this. On Thu, Jun 5, 2014 at 11:51 AM, Michael Theroux mthero...@yahoo.com wrote: You can have a ring spread across EC2 and the public subnet of a VPC. That is how we did our migration. In our case, we simply replaced the existing EC2 node with a new instance in the public VPC, restored from a backup taken right before the switch. -Mike -- *From:* Aiman Parvaiz ai...@shift.com *To:* Michael Theroux mthero...@yahoo.com *Cc:* user@cassandra.apache.org user@cassandra.apache.org *Sent:* Thursday, June 5, 2014 2:39 PM *Subject:* Re: VPC AWS Thanks for this info Michael. As far as restoring node in public VPC is concerned I was thinking ( and I might be wrong here) if we can have a ring spread across EC2 and public subnet of a VPC, this way I can simply decommission nodes in Ec2 as I gradually introduce new nodes in public subnet of VPC and I will end up with a ring in public subnet and then migrate them from public to private in a similar way may be. If anyone has any experience/ suggestions with this please share, would really appreciate it. Aiman On Thu, Jun 5, 2014 at 10:37 AM, Michael Theroux mthero...@yahoo.com wrote: The implementation of moving from EC2 to a VPC was a bit of a juggling act. Our motivation was two fold: 1) We were running out of static IP addresses, and it was becoming increasingly difficult in EC2 to design around limiting the number of static IP addresses to the number of public IP addresses EC2 allowed 2) VPC affords us an additional level of security that was desirable. However, we needed to consider the following limitations: 1) By default, you have a limited number of available public IPs for both EC2 and VPC. 2) AWS security groups need to be configured to allow traffic for Cassandra to/from instances in EC2 and the VPC. You are correct at the high level that the migration goes from EC2-Public VPC (VPC with an Internet Gateway)-Private VPC (VPC with a NAT). The first phase was moving instances to the public VPC, setting broadcast and seeds to the public IPs we had available. Basically: 1) Take down a node, taking a snapshot for a backup 2) Restore the node on the public VPC, assigning it to the correct security group, manually setting the seeds to other available nodes 3) Verify the cluster can communicate 4) Repeat Realize the NAT instance on the private subnet will also require a public IP. What got really interesting is that near the end of the process we ran out of available IPs, requiring us to switch the final node that was on EC2 directly to the private VPC (and taking down two nodes at once, which our setup allowed given we had 6 nodes with an RF of 3). What we did, and highly suggest for the switch, is to write down every step that has to happen on every node during the switch. In our case, many of the moved nodes required slightly different configurations for items like the seeds. Its been a couple of years, so my memory on this maybe a little fuzzy :) -Mike -- *From:* Aiman Parvaiz ai...@shift.com *To:* user@cassandra.apache.org; Michael Theroux mthero...@yahoo.com *Sent:* Thursday, June 5, 2014 12:55 PM *Subject:* Re: VPC AWS Michael, Thanks for the response, I am about to head in to something very similar if not exactly same. I envision things happening on the same lines as you mentioned. I would be grateful if you could please throw some more light on how you went about switching cassandra nodes from public subnet to private with out any downtime. I have not started on this project yet, still in my research phase. I plan to have a ec2+public VPC cluster and then decomission ec2 nodes to have everything in public subnet, next would be to move it to private subnet. Thanks On Thu, Jun 5, 2014 at 8:14 AM, Michael Theroux mthero...@yahoo.com wrote: We personally use the EC2Snitch, however, we don't have the multi-region requirements you do, -Mike -- *From:* Alain RODRIGUEZ arodr...@gmail.com *To:* user@cassandra.apache.org *Sent:* Thursday, June 5, 2014 9:14 AM *Subject:* Re: VPC AWS I think you can define VPC subnet to be public (to have public + private IPs) or private only. Any insight regarding snitches ? What snitch do you guys use ? 2014-06-05 15:06 GMT+02:00 William Oberman ober...@civicscience.com: I don't think traffic will flow between classic ec2 and vpc directly. There is some kind of gateway bridge instance that sits between, acting as a NAT. I would think that would cause new challenges for: -transitions -clients Sorry this response isn't heavy on content! I'm curious how this thread goes... Will On Thursday, June 5, 2014, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi guys, We are going to move from a cluster made of simple Amazon EC2 servers to a VPC cluster. We are using Cassandra 1.2.11 and I have some
Re: High latency on 5 node Cassandra Cluster
Yes, I meant RF: 2,2,1. We have the following in our network topo config: # cass08: us-west-1c private.ip=DC-A:1 public.ip=DC-A:1 # cass09: us-west-1b private.ip=DC-A:2 public.ip=DC-A:2 # cass10: Linode-Fremont-CA public.ip=DC-C:1 # cass11: us-west-2c private.ip=DC-B:1 public.ip=DC-B:1 # cass12: us-west-2b private.ip=DC-B:2 public.ip=DC-B:2 We're figuring out the upgrade right now. In general, what is the expected upgrade time per node for ~50GB (on each node)? Thanks arup On Wed, Jun 4, 2014 at 11:08 AM, Robert Coli rc...@eventbrite.com wrote: On Wed, Jun 4, 2014 at 12:12 AM, Arup Chakrabarti a...@pagerduty.com wrote: Size: 5 nodes (2 in AWS US-West-1, 2 in AWS US-West-2, 1 in Linode Fremont) Replication Factor: 5 You're operating with a single-DC strategy across multiple data centers? If so, I'm surprised you get sane latency ever. (Or do you mean RF : 2,2,1?) I agree with others that problems which can cause cluster wide outages exist in Gossip in the version of Cassandra you are running. As a general piece of feedback, I suggest an upgrade, first to 1.1 HEAD, then 1.2.16. =Rob
Re: Consolidating records and TTL
Hi All, Thanks for the replies. These detailed explanations are extremely useful. The are much appreciated! One other question that came to mind with this. Is there any way I can use a TTLs to keep the old account entries around for a fixed length of time rather than deleting them immediately? That way if I had an issue I could replay the process form one of these earlier points. For the regular reads of the account table I can use a limit of 1, with the reverse comparator set I should always get the newest record without hitting an older record first. I can't do the initial account insert with a TTL as I can't guarantee when a new value would come along and so replace this account record. However when I insert the new account record, instead of deleting the old one could I reinsert it with a TTL of say 1 month. How would compaction handle this. Would the original record get compacted away after 1 month + the GC Grace period or would it hang around still? Thanks, Charlie M On Thu, Jun 5, 2014 at 5:32 PM, James Campbell ja...@breachintelligence.com wrote: Thanks for creating and opening the discussion on this use case. I have been evaluating Cassandra for a very similar problem, but with the small twist that I'd like to roll up the ledger entries into the aggregated 'account' information on a regular basis to ensure that even rarely-read (but often written) data will be periodically rolled up and ready for quick reads. There is also a larger twist that for me 'account' is a compound key and the typical read pattern will include reading from multiple rows within the same partition (each of which would require separate merging from the ledger). This response prompts three questions for me about that: 1. Just to clarify the requirement that only one client thread does this at a time, I assume you mean only one thread should handle any given 'account' at a time, right? There wouldn't be problems having multiple clients do this data maintenance on different accounts at the same time, I hope. 2. For my use case, hadoop integration seems the natural fit, because I'd like to batch these updates to run at night during query down-time, but the current hadoop tools don't really support reading from or writing to multiple tables, as is required for this implementation. I've seen a few posts from people who have written homegrown input/output formats for hadoop, but I haven't tried to use them or evaluate their stability. Is there another mechanism I should be thinking about for that sort of batch updating? James Campbell -- *From:* Aaron Morton aa...@thelastpickle.com *Sent:* Thursday, June 5, 2014 5:26 AM *To:* Cassandra User *Cc:* charlie@gmail.com *Subject:* Re: Consolidating records and TTL As Tyler says, with atomic batches which are enabled by default the cluster will keep trying to replay the insert / deletes. Nodes check their local batch log for failed batches, ones where the coordinator did not acknowledge it had successfully completed, every 60 seconds. So there is a window where it’s possible for not all mutations in the batch to be completed. This could happen when a write timeout occurs when processing a batch of 2 rows; the request CL will not have been achieved on one or more of the rows. The coordinator will leave it up to the batch log to replay the request, and the client driver will (by default config) not retry. You can use a model like this. create table ledger ( account int, tx_id timeuuid, sub_total int, primary key (account, tx_id) ); create table account ( account int, total int, last_tx_id timeuuid, primary key (account) ); To get the total: select * from account where account = X; Then get the ledger entries you need select * from ledger where account = X and tx_id last_tx_id; This query will degrade when the partition size in the ledger table gets bigger, as it will need to read the column index (see column_index_size_in_kb in yaml). It will use that to find the first page that contains the rows we are interested in and then read forwards to the end of the row. It’s not the most efficient type of read but if you are going to delete ledger entries this *should* be able to skip over the tombstones without reading them. When you want to update the total in the account write to the account table and update both the total and the last_tx_id. You can then delete ledger entries if needed. Don’t forget to ensure that only one client thread is doing this at a time. Hope that helps. Aaron - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 5/06/2014, at 10:37 am, Tyler Hobbs ty...@datastax.com wrote: Just use an atomic batch that holds both the insert and deletes:
Sizing a new Cassandra cluster
I have been able to find lots of general information about sizing each node in a new Cassandra cluster, but have not come across any specific recommendations about the total size and configuration of the cluster (the number of nodes required per data center, the number of data centers, throughput requirements between data centers, etc.). I am currently in the process of sizing a new Cassandra cluster to support the following: * Probably more write intensive than read, or at least 65% / 35%. * Writes per day: 200,000,000 (~2315 per second). * Data retention = 30 days. * Replication Factor = 3. (I anticipate reads and writes of CL = Quorum or Quorum Local.) * My developers estimate a payload of ~300 bytes per record. * Throughput per second (MiB): (Records per second * Replication Facor * Event Payload) / 1024 / 1024 = 1.99 MiB/Sec. * Storage required (TiB): (Events per day * Event Payload * Replication Factor * Data Retention * 2) / 1024 / 1024 / 1024 / 1024 = 9.82 TiB. * Size doubled to provide room for Compaction. I’m wondering if I’m on the right track with my math, and if the following configuration would perform well, and leave a little overhead: * 2 Data Centers (they could co-exist with the application clusters). * 12 nodes (6 per data center) with: * 1 TiB storage capacity each. * I’ve seen varying information for RAID usage / configuration. Is a RAID 1 mirrored over 2x 1 TiB SSD drives performant? That might be a good configuration for us, and provide some high availability so that we can lose a drive without having to repair a node. Or is it better to buy an additional node for extra capacity, save the data to single SSDs and let it fail? (Or stripe 2x 500 GiB SSD drives…) * Do we need to store the CommitLog on a separate drive if we’re using SSD? How much space do we leave for it? Do we really need separate controllers? * 8 CPU cores. * 32GB RAM. Thoughts? Is this enough? Overkill? —Jeremy
Re: CQLSSTableWriter memory leak
Is writing too many rows to a single partition the cause of memory consumption?
What I want to achieve is this: say I have 5 partition ID. Each corresponds to
50 million IDs. Given a partition ID, I need to get its corresponding 50
million IDs. Is there another way to design the schema to avoid such a compound
primary key?
I could use the clustering IDs as the primary key, and create index on the
partition ID. But is that equivalent to create another table with compound keys?
At 2014-06-06 00:16:13, Jack Krupansky j...@basetechnology.com wrote:
How many rows (primary key values) are you writing for each partition of the
primary key? I mean, are there relatively few, or are these very wide
partitions?
Oh, I see! You’re writing 50,000,000 rows to a single partition! My, that IS
ambitious.
-- Jack Krupansky
From:Xu Zhongxing
Sent: Thursday, June 5, 2014 3:34 AM
To:user@cassandra.apache.org
Subject: CQLSSTableWriter memory leak
I am using Cassandra's CQLSSTableWriter to import a large amount of data into
Cassandra. When I use CQLSSTableWriter to write to a table with compound
primary key, the memory consumption keeps growing. The GC of JVM cannot collect
any used memory. When writing to tables with no compound primary key, the JVM
GC works fine.
My Cassandra version is 2.0.5. The OS is Ubuntu 14.04 x86-64. JVM parameters
are -Xms1g -Xmx2g. This is sufficient for all other non-compound primary key
cases.
The problem can be reproduced by the following test case:
import org.apache.cassandra.io.sstable.CQLSSTableWriter;
import org.apache.cassandra.exceptions.InvalidRequestException;
import java.io.IOException;
import java.util.UUID;
class SS {
public static void main(String[] args) {
String schema = create table test.t (x uuid, y uuid, primary key (x,
y));
String insert = insert into test.t (x, y) values (?, ?);
CQLSSTableWriter writer = CQLSSTableWriter.builder()
.inDirectory(/tmp/test/t)
.forTable(schema).withBufferSizeInMB(32)
.using(insert).build();
UUID id = UUID.randomUUID();
try {
for (int i = 0; i 5000; i++) {
UUID id2 = UUID.randomUUID();
writer.addRow(id, id2);
}
writer.close();
} catch (Exception e) {
System.err.println(hell);
}
}
}
Bad Request: Type error: cannot assign result of function token (type bigint) to id (type int)
I'm building a new schema which I need to read externally by paging through the result set. My understanding from reading the documentation , and this list, is that I can do that but I need to use the token() function. Only it doesn't work. Here's a reduction: create table test_paging ( id int, primary key(id) ); insert into test_paging (id) values (1); insert into test_paging (id) values (2); insert into test_paging (id) values (3); insert into test_paging (id) values (4); insert into test_paging (id) values (5); select * from test_paging where id token(0); … but it gives me: Bad Request: Type error: cannot assign result of function token (type bigint) to id (type int) … What's that about? I can't find any documentation for this and there aren't any concise examples. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* Skype: *burtonator* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com War is peace. Freedom is slavery. Ignorance is strength. Corporations are people.
Re: Bad Request: Type error: cannot assign result of function token (type bigint) to id (type int)
You should read through the token docs, it has examples and specifications: http://cassandra.apache.org/doc/cql3/CQL.html#tokenFun On Thu, Jun 5, 2014 at 10:22 PM, Kevin Burton bur...@spinn3r.com wrote: I'm building a new schema which I need to read externally by paging through the result set. My understanding from reading the documentation , and this list, is that I can do that but I need to use the token() function. Only it doesn't work. Here's a reduction: create table test_paging ( id int, primary key(id) ); insert into test_paging (id) values (1); insert into test_paging (id) values (2); insert into test_paging (id) values (3); insert into test_paging (id) values (4); insert into test_paging (id) values (5); select * from test_paging where id token(0); … but it gives me: Bad Request: Type error: cannot assign result of function token (type bigint) to id (type int) … What's that about? I can't find any documentation for this and there aren't any concise examples. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* Skype: *burtonator* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com War is peace. Freedom is slavery. Ignorance is strength. Corporations are people. -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
Re: Bad Request: Type error: cannot assign result of function token (type bigint) to id (type int)
Sorry, the datastax docs are actually a bit better: http://www.datastax.com/documentation/cql/3.0/cql/cql_using/paging_c.html Jon On Thu, Jun 5, 2014 at 10:46 PM, Jonathan Haddad j...@jonhaddad.com wrote: You should read through the token docs, it has examples and specifications: http://cassandra.apache.org/doc/cql3/CQL.html#tokenFun On Thu, Jun 5, 2014 at 10:22 PM, Kevin Burton bur...@spinn3r.com wrote: I'm building a new schema which I need to read externally by paging through the result set. My understanding from reading the documentation , and this list, is that I can do that but I need to use the token() function. Only it doesn't work. Here's a reduction: create table test_paging ( id int, primary key(id) ); insert into test_paging (id) values (1); insert into test_paging (id) values (2); insert into test_paging (id) values (3); insert into test_paging (id) values (4); insert into test_paging (id) values (5); select * from test_paging where id token(0); … but it gives me: Bad Request: Type error: cannot assign result of function token (type bigint) to id (type int) … What's that about? I can't find any documentation for this and there aren't any concise examples. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* Skype: *burtonator* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com War is peace. Freedom is slavery. Ignorance is strength. Corporations are people. -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade -- Jon Haddad http://www.rustyrazorblade.com skype: rustyrazorblade
