Re: Nodetool repair exceptions in Cassandra 2.0.2
Thank you for the reply Aaron. Unfortunately, I could not seem to find any additional info in the logs. However, upgrading from 2.0.2 to 2.0.3 seems to have done the trick! Best regards, -David Laube On Dec 11, 2013, at 6:51 PM, Aaron Morton aa...@thelastpickle.com wrote: [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f for range (1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException: [repair #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 the 10.x.x.48 node sent a tree response (merkle tree) to this node that did not contain the tree. This node then killed the repair session. Look for log messages on 10.x.x.48 that correlate with the repair session ID above. They may look like logger.error(Failed creating a merkle tree for + desc + , + initiator + (see log for details)”); or logger.info(String.format([repair #%s] Sending completed merkle tree to %s for %s/%s, desc.sessionId, initiator, desc.keyspace, desc.columnFamily)); Hope that helps. - Aaron Morton New Zealand @aaronmorton Co-Founder Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 10/12/2013, at 12:57 pm, Laing, Michael michael.la...@nytimes.com wrote: My experience is that you must upgrade to 2.0.3 ASAP to fix this. Michael On Mon, Dec 9, 2013 at 6:39 PM, David Laube d...@stormpath.com wrote: Hi All, We are running Cassandra 2.0.2 and have recently stumbled upon an issue with nodetool repair. Upon running nodetool repair on each of the 5 nodes in the ring (one at a time) we observe the following exceptions returned to standard out; [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f for range (1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException: [repair #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 [2013-12-08 11:04:02,063] Repair session 284c8b40-5ff8-11e3-97c0-5973cc397f8f for range (-109256956528331396,-89316884701275697] failed with error org.apache.cassandra.exceptions.RepairException: [repair #284c8b40-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.103 [2013-12-08 11:04:02,070] Repair session 399e7160-5ff8-11e3-97c0-5973cc397f8f for range (8901153810410866970,8915879751739915956] failed with error org.apache.cassandra.exceptions.RepairException: [repair #399e7160-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (8901153810410866970,8915879751739915956]] Validation failed in /10.x.x.103 [2013-12-08 11:04:02,072] Repair session 3ea73340-5ff8-11e3-97c0-5973cc397f8f for range (1149084504576970235,1190026362216198862] failed with error org.apache.cassandra.exceptions.RepairException: [repair #3ea73340-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1149084504576970235,1190026362216198862]] Validation failed in /10.x.x.103 [2013-12-08 11:04:02,091] Repair session 6f0da460-5ff8-11e3-97c0-5973cc397f8f for range (-5407189524618266750,-5389231566389960750] failed with error org.apache.cassandra.exceptions.RepairException: [repair #6f0da460-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (-5407189524618266750,-5389231566389960750]] Validation failed in /10.x.x.103 [2013-12-09 23:16:36,962] Repair session 7efc2740-6127-11e3-97c0-5973cc397f8f for range (1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException: [repair #7efc2740-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 [2013-12-09 23:16:36,986] Repair session a8c44260-6127-11e3-97c0-5973cc397f8f for range (-109256956528331396,-89316884701275697] failed with error org.apache.cassandra.exceptions.RepairException: [repair #a8c44260-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.210 The /var/log/cassandra/system.log shows similar info as above with no real explanation as to the root cause behind the exception(s). There also does not appear to be any additional info in /var/log/cassandra/cassandra.log. We have tried restoring a recent snapshot of the keyespace in question to a separate staging ring and the repair runs successfully and without exception there. This is even after we tried insert/delete on the keyspace in the separate staging ring. Has anyone seen this behavior before and what can we do to resolve this? Any assistance would be greatly appreciated. Best regards, -Dave
Nodetool repair exceptions in Cassandra 2.0.2
Hi All, We are running Cassandra 2.0.2 and have recently stumbled upon an issue with nodetool repair. Upon running nodetool repair on each of the 5 nodes in the ring (one at a time) we observe the following exceptions returned to standard out; [2013-12-08 11:04:02,047] Repair session ff16c510-5ff7-11e3-97c0-5973cc397f8f for range (1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException: [repair #ff16c510-5ff7-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 [2013-12-08 11:04:02,063] Repair session 284c8b40-5ff8-11e3-97c0-5973cc397f8f for range (-109256956528331396,-89316884701275697] failed with error org.apache.cassandra.exceptions.RepairException: [repair #284c8b40-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.103 [2013-12-08 11:04:02,070] Repair session 399e7160-5ff8-11e3-97c0-5973cc397f8f for range (8901153810410866970,8915879751739915956] failed with error org.apache.cassandra.exceptions.RepairException: [repair #399e7160-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (8901153810410866970,8915879751739915956]] Validation failed in /10.x.x.103 [2013-12-08 11:04:02,072] Repair session 3ea73340-5ff8-11e3-97c0-5973cc397f8f for range (1149084504576970235,1190026362216198862] failed with error org.apache.cassandra.exceptions.RepairException: [repair #3ea73340-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1149084504576970235,1190026362216198862]] Validation failed in /10.x.x.103 [2013-12-08 11:04:02,091] Repair session 6f0da460-5ff8-11e3-97c0-5973cc397f8f for range (-5407189524618266750,-5389231566389960750] failed with error org.apache.cassandra.exceptions.RepairException: [repair #6f0da460-5ff8-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (-5407189524618266750,-5389231566389960750]] Validation failed in /10.x.x.103 [2013-12-09 23:16:36,962] Repair session 7efc2740-6127-11e3-97c0-5973cc397f8f for range (1246984843639507027,1266616572749926276] failed with error org.apache.cassandra.exceptions.RepairException: [repair #7efc2740-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family1, (1246984843639507027,1266616572749926276]] Validation failed in /10.x.x.48 [2013-12-09 23:16:36,986] Repair session a8c44260-6127-11e3-97c0-5973cc397f8f for range (-109256956528331396,-89316884701275697] failed with error org.apache.cassandra.exceptions.RepairException: [repair #a8c44260-6127-11e3-97c0-5973cc397f8f on keyspace_name/col_family2, (-109256956528331396,-89316884701275697]] Validation failed in /10.x.x.210 The /var/log/cassandra/system.log shows similar info as above with no real explanation as to the root cause behind the exception(s). There also does not appear to be any additional info in /var/log/cassandra/cassandra.log. We have tried restoring a recent snapshot of the keyespace in question to a separate staging ring and the repair runs successfully and without exception there. This is even after we tried insert/delete on the keyspace in the separate staging ring. Has anyone seen this behavior before and what can we do to resolve this? Any assistance would be greatly appreciated. Best regards, -Dave
Re: sstableloader does not support client encryption on Cassandra 2.0?
Thank you Tyler. I took your advice and I have opened https://issues.apache.org/jira/browse/CASSANDRA-6378 Best regards, -David Laube On Nov 19, 2013, at 9:51 AM, Tyler Hobbs ty...@datastax.com wrote: I think this is just an oversight; would you mind opening a ticket here? https://issues.apache.org/jira/browse/CASSANDRA On Mon, Nov 18, 2013 at 12:37 PM, David Laube d...@stormpath.com wrote: Hi All, We have been testing backup/restore from one ring to another and we recently stumbled upon an issue with sstableloader. When client_enc_enable: true, the exception below is generated. When client_enc_enable is set to false, the sstableloader is able to get to the point where it is discovers endpoints, connects to stream data, etc. ==BEGIN EXCEPTION== sstableloader --debug -d x.x.x.248,x.x.x.108,x.x.x.113 /tmp/import/keyspace_name/columnfamily_name Exception in thread main java.lang.RuntimeException: Could not retrieve endpoint ranges: at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:226) at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:149) at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:68) Caused by: org.apache.thrift.transport.TTransportException: Frame size (352518400) larger than max length (16384000)! at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.cassandra.thrift.Cassandra$Client.recv_describe_partitioner(Cassandra.java:1292) at org.apache.cassandra.thrift.Cassandra$Client.describe_partitioner(Cassandra.java:1280) at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:199) ... 2 more ==END EXCEPTION== Has anyone seen this before or can someone confirm that SSL/encryption is not supported under the open source project and only with d-stax enterprise? Thanks, -David Laube -- Tyler Hobbs DataStax
sstableloader does not support client encryption on Cassandra 2.0?
Hi All, We have been testing backup/restore from one ring to another and we recently stumbled upon an issue with sstableloader. When client_enc_enable: true, the exception below is generated. When client_enc_enable is set to false, the sstableloader is able to get to the point where it is discovers endpoints, connects to stream data, etc. ==BEGIN EXCEPTION== sstableloader --debug -d x.x.x.248,x.x.x.108,x.x.x.113 /tmp/import/keyspace_name/columnfamily_name Exception in thread main java.lang.RuntimeException: Could not retrieve endpoint ranges: at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:226) at org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:149) at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:68) Caused by: org.apache.thrift.transport.TTransportException: Frame size (352518400) larger than max length (16384000)! at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137) at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.cassandra.thrift.Cassandra$Client.recv_describe_partitioner(Cassandra.java:1292) at org.apache.cassandra.thrift.Cassandra$Client.describe_partitioner(Cassandra.java:1280) at org.apache.cassandra.tools.BulkLoader$ExternalClient.init(BulkLoader.java:199) ... 2 more ==END EXCEPTION== Has anyone seen this before or can someone confirm that SSL/encryption is not supported under the open source project and only with d-stax enterprise? Thanks, -David Laube
Read inconsistency after backup and restore to different cluster
Hi All, After running through our backup and restore process FROM our test production TO our staging environment, we are seeing inconsistent reads from the cluster we restored to. We have the same number of nodes in both clusters. For example, we will select data from a column family on the newly restored cluster but sometimes the expected data is returned and other times it is not. These selects are carried out one after another with very little delay. It is almost as if the data only exists on some of the nodes, or perhaps the token ranges are dramatically different --again, we are using vnodes so I am not exactly sure how this plays into the equation. We are running Cassadra 2.0.2 with vnodes and deploying via chef. The backup and restore process is currently orchestrated using bash scripts and chef's distributed SSH. I have outlined the process below for review. (I) Backup cluster-A (with existing prod data): 1. Run nodetool flush on each of the nodes in a 5 node ring. 2. Run nodetool snapshot keyspace_name on each of the nodes in a 5 node ring. 3. Archive the snapshot data from the snapshots directory in each node, creating a single archive of the snapshot. 4. Copy the snapshot data archive for each of the nodes to s3. (II) Restore backup FROM cluster-A TO cluster-B: *NOTE: cluster-B is a freshly deployed ring with no data, but a different cluster-name used for staging. 1. Deploy 5 nodes as part of the cluster-B ring. 2. Create keyspace_name keyspace and column families on cluster-B. 3. Stop Cassandra on all 5 nodes in the cluster-B ring. 4. Clear commit logs on cluster-B with: rm -f /var/lib/cassandra/commitlog/* 5. Copy 1 of the 5 snapshot archives from cluster-A to each of the five nodes in the new cluster-B ring. 6. Extract the archives to /var/lib/cassandra/data/keyspace_name ensuring that the column family directories and associated .DB files are in place under /var/lib/cassandra/data/keyspace_name/columfamily1/ ….etc. 7.Start Cassandra on each of the nodes in cluster-B. 8. Run nodetool repair on each of the nodes in cluster-B. Please let me know if you see any major errors or deviation from best practices which could be contributing to our read inconsistencies. I'll be happy to answer any specific question you may have regarding our configuration. Thank you in advance! Best regards, -David Laube
Re: Read inconsistency after backup and restore to different cluster
Thank you for the detailed reply Rob! I have replied to your comments in-line below; On Nov 14, 2013, at 1:15 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Nov 14, 2013 at 12:37 PM, David Laube d...@stormpath.com wrote: It is almost as if the data only exists on some of the nodes, or perhaps the token ranges are dramatically different --again, we are using vnodes so I am not exactly sure how this plays into the equation. The token ranges are dramatically different, due to vnode random token selection from not setting initial_token, and setting num_tokens. You can verify this by listing the tokens per physical node in nodetool gossipinfo or (iirc) nodetool status. 5. Copy 1 of the 5 snapshot archives from cluster-A to each of the five nodes in the new cluster-B ring. I don't understand this at all, do you mean that you are using one source node's data to load each of of the target nodes? Or are you just saying there's a 1:1 relationship between source snapshots and target nodes to load into? Unless you have RF=N, using one source for 5 target nodes won't work. We have configured RF=3 for the keyspace in question. Also, from a client perspective, we read with CL=1 and write with CL=QUORUM. Since we have 5 nodes total in cluster-A, we snapshot keyspace_name on each of the five nodes which results in a snapshot directory on each of the five nodes that we archive and ship off to s3. We then take the snapshot archive generated FROM cluster-A_node1 and copy/extract/restore TO cluster-B_node1, then we take the snapshot archive FROM cluster-A_node2 and copy/extract/restore TO cluster-B_node2 and so on and so forth. To do what I think you're attempting to do, you have basically two options. 1) don't use vnodes and do a 1:1 copy of snapshots 2) use vnodes and a) get a list of tokens per node from the source cluster b) put a comma delimited list of these in initial_token in cassandra.yaml on target nodes c) probably have to un-set num_tokens (this part is unclear to me, you will have to test..) d) set auto_bootstrap:false in cassandra.yaml e) start target nodes, they will not-bootstrap into the same ranges as the source cluster f) load schema / copy data into datadir (being careful of https://issues.apache.org/jira/browse/CASSANDRA-6245) g) restart node or use nodetool refresh (I'd probably restart the node to avoid the bulk rename that refresh does) to pick up sstables h) remove auto_bootstrap:false from cassandra.yaml I *believe* this *should* work, but have never tried it as I do not currently run with vnodes. It should work because it basically makes implicit vnode tokens explicit in the conf file. If it *does* work, I'd greatly appreciate you sharing details of your experience with the list. I'll start with parsing out the token ranges that our vnode config ends up assigning in cluster-A, and doing some creative config work on the target cluster-B we are trying to restore to as you have suggested. Depending on what additional comments/recommendation you or another member of the list may have (if any) based on the clarification I've made above, I will absolutely report back my findings here. General reference on tasks of this nature (does not consider vnodes, but treat vnodes as just a lot of physical nodes and it is mostly relevant) : http://www.palominodb.com/blog/2012/09/25/bulk-loading-options-cassandra =Rob
Automated backup and restore of Cassandra 2.0
Hi All, I was wondering if anyone on the list could make some recommendations as to how you are currently backing up and restoring your ring in an automated manner. We are deploying Cassandra 2.0 with Chef and as part of our release process, we test our application in a full staging environment before going to production. From what I understand about the snapshot/restore process, this requires that we restore X number of individual node snapshots to X number of nodes in a separate staging-ring. I would like to handle this either in-house or with open source software instead of going down the Datastax route. Can anyone suggest some methods of accomplishing this goal in a straight-forward way? Thanks! Best regards, -David Laube
cqlsh error after enabling encryption
Hi All, After enabling encryption on our Cassandra 1.2.8 nodes, we receiving the error Connection error: TSocket read 0 bytes while attempting to use CQLsh to talk to the ring. I've followed the docs over at http://www.datastax.com/documentation/cassandra/1.2/webhelp/cassandra/security/secureCqlshSSL_t.html but can't seem to figure out why this isn't working. Inter-node communication seems to be working properly since nodetool status shows our nodes as up, but the CQLsh client is unable to talk to a single node or any node in the cluster (specifying the IP in .cqlshrc or on the CLI) for some reason. I'm providing the applicable config file entries below for review. Any insight or suggestions would be greatly appreciated! :) My ~/.cqlshrc file: [connection] hostname = 127.0.0.1 port = 9160 factory = cqlshlib.ssl.ssl_transport_factory [ssl] certfile = /etc/cassandra/conf/cassandra_client.crt validate = true ## Optional, true by default. [certfiles] ## Optional section, overrides the default certfile in the [ssl] section. 192.168.1.3 = ~/keys/cassandra01.cert 192.168.1.4 = ~/keys/cassandra02.cert Our cassandra.yaml file config blocks: …snip… server_encryption_options: internode_encryption: all keystore: /etc/cassandra/conf/.keystore keystore_password: yeah-right truststore: /etc/cassandra/conf/.truststore truststore_password: yeah-right # More advanced defaults below: # protocol: TLS # algorithm: SunX509 # store_type: JKS # cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA] # require_client_auth: false # enable or disable client/server encryption. client_encryption_options: enabled: true keystore: /etc/cassandra/conf/.keystore keystore_password: yeah-right # require_client_auth: false # Set trustore and truststore_password if require_client_auth is true # truststore: conf/.truststore # truststore_password: cassandra # More advanced defaults below: protocol: TLS algorithm: SunX509 store_type: JKS cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA] …snip... Thanks, -David Laube
Cassandra JVM heap sizes on EC2
Hi All, We are evaluating our JVM heap size configuration on Cassandra 1.2.8 and would like to get some feedback from the community as to what the proper JVM heap size should be for cassandra nodes deployed on to Amazon EC2. We are running m2.4xlarge EC2 instances (64GB RAM, 8 core, 2 x 840GB disks) --so we will have plenty of RAM. I've already consulted the docs at http://www.datastax.com/documentation/cassandra/1.2/mobile/cassandra/operations/ops_tune_jvm_c.html but would love to hear what is working or not working for you in the wild. Since Datastax cautions against using more than 8GB, I'm wondering if it is even advantageous to use even slightly more. Thanks, -David Laube
