Performance tuning for read throughput w/ token scans?
[Cassandra 2.1.5] I'm trying to explore my options for increasing read throughput with token scans (SELECT * FROM x WHERE token(y) L AND token(y) L). So far I've started by reading an entire virtual token range from a single node. Currently on a single query I can read about 57,286.03 rows/s which translates to 5.5 MiB/s. However under load (even under heavy load) my disk utilization never gets that high (SSDs, less than 10%) - nor does my network utilization (1gbit). So far I've tried - - Moving to the G1 collector (started with the cassandra-env that is was linked from CASSANDRA-7486) - which reduced timeouts which I think were caused longish pauses - Enabling TIMEHORIZON message coalescing I'm still very new to JVM tuning but I used jstack to inspect what was going on in threads with high cpu usage. Its almost always either OutBoundTcpConnection stack/thread or SEPWorker stack/thread - and judging by what the SEPWorker does (I mostly see compares like https://github.com/apache/cassandra/blob/cassandra-2.1.5/src/java/org/apache/cassandra/db/composites/AbstractCType.java#L185), I think I might be CPU bound? (I'm still new to the actual Cassandra source code, so apologies if that doesn't make sense either). Given this information, does anyone have any pointers on what levers I could pull next or other things I can look to measure? Thanks for any help, Nimi
Re: sstableloader Could not retrieve endpoint ranges
Fabien, thanks for the reply. We do have Thrift enabled. From what I can tell, the Could not retrieve endpoint ranges: crops up under various circumstances. From further reading on sstableloader, it occurred to me that it might be a safer bet to use the JMX StorageService bulkLoad command, considering that the data to import was already on one of the Cassandra nodes, just in an arbitrary directory outside the Cassandra data directories. I was able to get this bulkLoad command to fail with a message that the directory structure did not follow the expected keyspace/table/ pattern. So I created a keyspace directory and then a table directory within that and moved all the files under the table directory. Executed bulkLoad, passing in that directory. It succeeded. Then I went and ran a nodetool refresh on the table in question. Only one problem. If I then went to query the table for, well, anything, nothing came back. And this was after successfully querying the table before and truncating the table just prior to the bulkLoad, so that I knew that only the data coming from the bulkLoad could show up there. Oh, and for good measure, I stopped and started all the nodes too. No luck still. What's puzzling about this is that the bulkLoad silently succeeds, even though it doesn't appear to be doing anything. I haven't bothered yet to check the Cassandra logs. On Fri, Jun 19, 2015 at 12:28 AM, Fabien Rousseau fabifab...@gmail.com wrote: Hi, I already got this error on a 2.1 clusters because thrift was disabled. So you should check that thrift is enabled and accessible from the sstableloader process. Hope this help Fabien Le 19 juin 2015 05:44, Mitch Gitman mgit...@gmail.com a écrit : I'm using sstableloader to bulk-load a table from one cluster to another. I can't just copy sstables because the clusters have different topologies. While we're looking to upgrade soon to Cassandra 2.0.x, we're on Cassandra 1.2.19. The source data comes from a nodetool snapshot. Here's the command I ran: sstableloader -d *IP_ADDRESSES_OF_SEED_NOTES* */SNAPSHOT_DIRECTORY/* Here's the result I got: Could not retrieve endpoint ranges: -pr,--principal kerberos principal -k,--keytab keytab location --ssl-keystoressl keystore location --ssl-keystore-password ssl keystore password --ssl-keystore-type ssl keystore type --ssl-truststore ssl truststore location --ssl-truststore-password ssl truststore password --ssl-truststore-type ssl truststore type Not sure what to make of this, what with the hints at security arguments that pop up. The source and destination clusters have no security. Hoping this might ring a bell with someone out there.
Re: sstableloader Could not retrieve endpoint ranges
I checked the system.log for the Cassandra node that I did the jconsole JMX session against and which had the data to load. Lot of log output indicating that it's busy loading the files. Lot of stacktraces indicating a broken pipe. I have no reason to believe there are connectivity issues between the nodes, but verifying that is beyond my expertise. What's indicative is this last bit of log output: INFO [Streaming to /10.205.55.101:5] 2015-06-19 21:20:45,441 StreamReplyVerbHandler.java (line 44) Successfully sent /srv/cas-snapshot-06-17-2015/endpoints/endpoint_messages/endpoints-endpoint_messages-ic-34-Data.db to /10.205.55.101 INFO [Streaming to /10.205.55.101:5] 2015-06-19 21:20:45,457 OutputHandler.java (line 42) Streaming session to /10.205.55.101 failed ERROR [Streaming to /10.205.55.101:5] 2015-06-19 21:20:45,458 CassandraDaemon.java (line 253) Exception in thread Thread[Streaming to / 10.205.55.101:5,5,RMI Runtime] java.lang.RuntimeException: java.io.IOException: Broken pipe at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at org.apache.cassandra.streaming.compress.CompressedFileStreamTask.stream(CompressedFileStreamTask.java:93) at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:91) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 3 more And then right after that I see what appears to be the output from the nodetool refresh: INFO [RMI TCP Connection(2480)-10.2.101.114] 2015-06-19 21:22:56,877 ColumnFamilyStore.java (line 478) Loading new SSTables for endpoints/endpoint_messages... INFO [RMI TCP Connection(2480)-10.2.101.114] 2015-06-19 21:22:56,878 ColumnFamilyStore.java (line 524) No new SSTables were found for endpoints/endpoint_messages Notice that Cassandra hasn't found any new SSTables, even though it was just so busy loading them. What's also noteworthy is that the output from the originating node shows it successfully sent endpoints-endpoint_messages-ic-34-Data.db to another node. But then in the system.log for that destination node, I see no mention of that file. What I do see on the destination node are a few INFO messages about streaming one of the .db files, and every time that's immediately followed by an error message: INFO [Thread-108] 2015-06-19 21:20:45,453 StreamInSession.java (line 142) Streaming of file /srv/cas-snapshot-06-17-2015/endpoints/endpoint_messages/endpoints-endpoint_messages-ic-26-Data.db sections=1 progress=0/105137329 - 0% for org.apache.cassandra.streaming.StreamInSession@46c039ef failed: requesting a retry. ERROR [Thread-109] 2015-06-19 21:20:45,456 CassandraDaemon.java (line 253) Exception in thread Thread[Thread-109,5,main] java.lang.RuntimeException: java.nio.channels.AsynchronousCloseException at com.google.common.base.Throwables.propagate(Throwables.java:160) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) at java.lang.Thread.run(Thread.java:745) Caused by: java.nio.channels.AsynchronousCloseException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:412) at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:203) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ... 1 more I don't know, I'm seeing enough flakiness here as to consider Cassandra bulk-loading a lost cause, even if there is something wrong and fixable about my particular cluster. On to exporting and re-importing data at the proprietary application level. Life is too short. On Fri, Jun 19, 2015 at 2:40 PM, Mitch Gitman mgit...@gmail.com wrote: Fabien, thanks for the reply. We do have Thrift enabled. From what I can tell, the Could not retrieve endpoint ranges: crops up under various circumstances. From further reading on sstableloader, it occurred to me that it might be a safer bet to use the JMX StorageService bulkLoad command, considering that the data to import was already on one of the Cassandra nodes, just in an arbitrary directory outside the Cassandra data directories. I was able to get this bulkLoad command to fail with a message that the directory structure did not follow the expected keyspace/table/ pattern. So I created
RE: Code review - Spark SQL command-line client for Cassandra
Hi Matthew,
It looks fine to me. I have built a similar service that allows a user to
submit a query from a browser and returns the result in JSON format.
Another alternative is to leave a Spark shell or one of the notebooks (Spark
Notebook, Zeppelin, etc.) session open and run queries from there. This model
works only if people give you the queries to execute.
Mohammed
From: Matthew Johnson [mailto:matt.john...@algomi.com]
Sent: Friday, June 19, 2015 2:21 AM
To: user@cassandra.apache.org
Subject: Code review - Spark SQL command-line client for Cassandra
Hi all,
I have been struggling with Cassandra’s lack of adhoc query support (I know
this is an anti-pattern of Cassandra, but sometimes management come over and
ask me to run stuff and it’s impossible to explain that it will take me a while
when it would take about 10 seconds in MySQL) so I have put together the
following code snippet that bundles DataStax’s Cassandra Spark connector and
allows you to submit Spark SQL to it, outputting the results in a text file.
Does anyone spot any obvious flaws in this plan?? (I have a lot more error
handling etc in my code, but removed it here for brevity)
private void run(String sqlQuery) {
SparkContext scc = new SparkContext(conf);
CassandraSQLContext csql = new CassandraSQLContext(scc);
DataFrame sql = csql.sql(sqlQuery);
String folderName = /tmp/output_ + System.currentTimeMillis();
LOG.info(Attempting to save SQL results in folder: + folderName);
sql.rdd().saveAsTextFile(folderName);
LOG.info(SQL results saved);
}
public static void main(String[] args) {
String sparkMasterUrl = args[0];
String sparkHost = args[1];
String sqlQuery = args[2];
SparkConf conf = new SparkConf();
conf.setAppName(Java Spark SQL);
conf.setMaster(sparkMasterUrl);
conf.set(spark.cassandra.connection.host, sparkHost);
JavaSparkSQL app = new JavaSparkSQL(conf);
app.run(sqlQuery, printToConsole);
}
I can then submit this to Spark with ‘spark-submit’:
./spark-submit --class com.algomi.spark.JavaSparkSQL --master
spark://sales3:7077
spark-on-cassandra-0.0.1-SNAPSHOT-jar-with-dependencies.jar
spark://sales3:7077 sales3 select * from mykeyspace.operationlog
It seems to work pretty well, so I’m pretty happy, but wondering why this isn’t
common practice (at least I haven’t been able to find much about it on Google)
– is there something terrible that I’m missing?
Thanks!
Matthew
RE: nodetool repair
Hi, For the record I've succesfully used https://github.com/BrianGallew/cassandra_range_repair to make smooth repairing. Could maybe also be of interest don't know... Cheers, Jens – Skickat från Mailbox On Fri, Jun 19, 2015 at 8:36 PM, null sean_r_dur...@homedepot.com wrote: It seems to me that running repair on any given node may also induce repairs to related replica nodes. For example, if I run repair on node A and node B has some replicas, data might stream from A to B (assuming A has newer/more data). Now, that does NOT mean that node B will be fully repaired. You still need to run repair -pr on all nodes before gc_grace_seconds. You can run repairs on multiple nodes at the same time. However, you might end up with a large amount of streaming, if many repairs are needed. So, you should be aware of a performance impact. I run weekly repairs on one node at a time, if possible. On, larger rings, though, I run repairs on multiple nodes staggered by a few hours. Once your routine maintenance is established, repairs will not run for very long. But, if you have a large ring that hasn’t been repaired, those first repairs may take days (but should get faster as you get further through the ring). Sean Durity From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Friday, June 19, 2015 3:56 AM To: user@cassandra.apache.org Subject: Re: nodetool repair Hi, This is not necessarily true. Repair will induce compactions only if you have entropy in your cluster. If not it will just read your data to compare all the replica of each piece of data (using indeed cpu and disk IO). If there is some data missing it will repair it. Though, due to merkle tree size, you will generally stream more data than just the data needed. To limit this downside and the compactions amount, use range repairs -- http://www.datastax.com/dev/blog/advanced-repair-techniques. About tombstones, they will be evicted only after gc_grace_period and only if all the parts of the row are part of the compaction. C*heers, Alain 2015-06-19 9:08 GMT+02:00 arun sirimalla arunsi...@gmail.commailto:arunsi...@gmail.com: Yes compactions will remove tombstones On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction, does it mean that it would also clean up my tombstone from my LeveledCompactionStrategy tables at the same time? Thanks for your help. On 19 Jun 2015, at 07:56 , arun sirimalla arunsi...@gmail.commailto:arunsi...@gmail.com wrote: Hi Jean, Running nodetool repair on a node will repair only that node in the cluster. It is recommended to run nodetool repair on one node at a time. Few things to keep in mind while running repair 1. Running repair will trigger compactions 2. Increase in CPU utilization. Run node tool repair with -pr option, so that it will repair only the range that node is responsible for. On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? Kind regards On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.commailto:j...@jonhaddad.com wrote: If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help. -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera)
Re: Deploying OpsCenter behind a HTTP(S) proxy
Hi Ben, thank for your reply. That was I was afraid of actually as it means there's no easy solution to implement I guess. I think some guys in my team are in contact with DS people, so I may have a look there. Jonathan On 06/18/2015 07:26 PM, Ben Bromhead wrote: OpsCenter is a little bit tricky to simply just rewrite urls, the lhr requests and rest endpoints it hits are all specified a little differently in the javascript app it loads. We ended up monkey patching a buttload of the js files to get all the requests working properly with our proxy. Everytime a new release of OpsCenter comes out we have to rework it. If you are a DSE customer I would raise it as a support issue :) On 18 June 2015 at 02:29, Spencer Brown lilspe...@gmail.com mailto:lilspe...@gmail.com wrote: First, your firewall should really be your frontend There operational frontend is apache, which is common. You want every url with opscenter in it handled elsewhere. You could also set up proxies for /. cluster-configs, etc... Then there is mod_rewrite, which provides a lot more granularity about when you want what gets handled where.I set up the architectural infrastructure for Orbitz and some major banks, and I'd be happpy to help you out on this. I charge $30/hr., but what you need isn't very complex so we're really just talking $100. On Thu, Jun 18, 2015 at 5:13 AM, Jonathan Ballet jbal...@gfproducts.ch mailto:jbal...@gfproducts.ch wrote: Hi, I'm looking for information on how to correctly deploy an OpsCenter instance behind a HTTP(S) proxy. I have a running instance of OpsCenter 5.1 reachable at http://opscenter:/opscenter/ but I would like to be able to serve this kind of tool under a single hostname on HTTPS along with other tools of this kind, for easier convenience. I'm currently using Apache as my HTTP front-end and I tried this naive configuration: VirtualHost *:80 ServerName tools ... ProxyPreserveHost On # Proxy to OpsCenter # ProxyPass /opscenter/ http://opscenter:/opscenter/ ProxyPassReverse/opscenter/ http://opscenter:/opscenter/ /VirtualHost This doesn't quite work, as OpsCenter seem to also serve specific endpoints from / directly Of course, it doesn't correctly work, as OpsCenter seem to also serve specific data from / directly, such as: /cluster-configs /TestCluster /meta /rc /tcp Is there something I can configure in OpsCenter so that it serves these URLs from somewhere else, or a list of known URLs that I can remap on the proxy, or better yet, a known proxy configuration to put in front of OpsCenter? Regards, Jonathan -- Ben Bromhead Instaclustr | www.instaclustr.com https://www.instaclustr.com/ | @instaclustr http://twitter.com/instaclustr | (650) 284 9692
Re: sstableloader Could not retrieve endpoint ranges
Hi, I already got this error on a 2.1 clusters because thrift was disabled. So you should check that thrift is enabled and accessible from the sstableloader process. Hope this help Fabien Le 19 juin 2015 05:44, Mitch Gitman mgit...@gmail.com a écrit : I'm using sstableloader to bulk-load a table from one cluster to another. I can't just copy sstables because the clusters have different topologies. While we're looking to upgrade soon to Cassandra 2.0.x, we're on Cassandra 1.2.19. The source data comes from a nodetool snapshot. Here's the command I ran: sstableloader -d *IP_ADDRESSES_OF_SEED_NOTES* */SNAPSHOT_DIRECTORY/* Here's the result I got: Could not retrieve endpoint ranges: -pr,--principal kerberos principal -k,--keytab keytab location --ssl-keystoressl keystore location --ssl-keystore-password ssl keystore password --ssl-keystore-type ssl keystore type --ssl-truststore ssl truststore location --ssl-truststore-password ssl truststore password --ssl-truststore-type ssl truststore type Not sure what to make of this, what with the hints at security arguments that pop up. The source and destination clusters have no security. Hoping this might ring a bell with someone out there.
Re: nodetool repair
Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction, does it mean that it would also clean up my tombstone from my LeveledCompactionStrategy tables at the same time? Thanks for your help. On 19 Jun 2015, at 07:56 , arun sirimalla arunsi...@gmail.commailto:arunsi...@gmail.com wrote: Hi Jean, Running nodetool repair on a node will repair only that node in the cluster. It is recommended to run nodetool repair on one node at a time. Few things to keep in mind while running repair 1. Running repair will trigger compactions 2. Increase in CPU utilization. Run node tool repair with -pr option, so that it will repair only the range that node is responsible for. On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? Kind regards On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.commailto:j...@jonhaddad.com wrote: If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help. -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html
Re: nodetool repair
Yes compactions will remove tombstones On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay jean.tremb...@zen-innovations.com wrote: Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction, does it mean that it would also clean up my tombstone from my LeveledCompactionStrategy tables at the same time? Thanks for your help. On 19 Jun 2015, at 07:56 , arun sirimalla arunsi...@gmail.com wrote: Hi Jean, Running nodetool repair on a node will repair only that node in the cluster. It is recommended to run nodetool repair on one node at a time. Few things to keep in mind while running repair 1. Running repair will trigger compactions 2. Increase in CPU utilization. Run node tool repair with -pr option, so that it will repair only the range that node is responsible for. On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay jean.tremb...@zen-innovations.com wrote: Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? Kind regards On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.com wrote: If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.com wrote: Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help. -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html
Code review - Spark SQL command-line client for Cassandra
Hi all,
I have been struggling with Cassandra’s lack of adhoc query support (I know
this is an anti-pattern of Cassandra, but sometimes management come over
and ask me to run stuff and it’s impossible to explain that it will take me
a while when it would take about 10 seconds in MySQL) so I have put
together the following code snippet that bundles DataStax’s Cassandra Spark
connector and allows you to submit Spark SQL to it, outputting the results
in a text file.
Does anyone spot any obvious flaws in this plan?? (I have a lot more error
handling etc in my code, but removed it here for brevity)
*private* *void* run(String sqlQuery) {
SparkContext scc = *new* SparkContext(conf);
CassandraSQLContext csql = *new* CassandraSQLContext(scc);
DataFrame sql = csql.sql(sqlQuery);
String folderName = /tmp/output_ + System.*currentTimeMillis*();
*LOG*.info(Attempting to save SQL results in folder: +
folderName);
sql.rdd().saveAsTextFile(folderName);
*LOG*.info(SQL results saved);
}
*public* *static* *void* main(String[] args) {
String sparkMasterUrl = args[0];
String sparkHost = args[1];
String sqlQuery = args[2];
SparkConf conf = *new* SparkConf();
conf.setAppName(Java Spark SQL);
conf.setMaster(sparkMasterUrl);
conf.set(spark.cassandra.connection.host, sparkHost);
JavaSparkSQL app = *new* JavaSparkSQL(conf);
app.run(sqlQuery, printToConsole);
}
I can then submit this to Spark with ‘spark-submit’:
Ø *./spark-submit --class com.algomi.spark.JavaSparkSQL --master
spark://sales3:7077
spark-on-cassandra-0.0.1-SNAPSHOT-jar-with-dependencies.jar
spark://sales3:7077 sales3 select * from mykeyspace.operationlog *
It seems to work pretty well, so I’m pretty happy, but wondering why this
isn’t common practice (at least I haven’t been able to find much about it
on Google) – is there something terrible that I’m missing?
Thanks!
Matthew
Long nodetool repair use nodetool repair incremental
Hi, I understand that we must repair the DB on a regular basis. Now I also see that making a repair is using lots of resources in the cluster so I need to do this during the weekend because I really would like to have high performance at least during the week days. In the documentation I see the incremental nodetool repair”. That really seems to be the option for me. Few questions: 1) Why is it written in the documentation that it is suggested to do incremental repair daily, and full repair on the weekends? Is repair incremental not good enough or not safe? 2) In order to use incremental repair we need to do migration steps. One of these steps is to run the tool sstablerepairedset. What is the parameter sstables? Is it the name (without extension) of all *.db files contained in the data/cfdir? 3) Is the sstablerepairedset really needed when making a repair incremental on an empty DB? Thanks for your help Jean
RE: Cassandra Metrics
Hi, One valuable (IMHO) entry point is : « Guide to Cassandra Thread Pools » http://blackbird.io/guide-to-cassandra-thread-pools Take a look. Regards, Dominique De : pushdlim...@gmail.com [mailto:pushdlim...@gmail.com] De la part de Saurabh Chandolia Envoyé : vendredi 19 juin 2015 11:42 À : user@cassandra.apache.org Objet : Cassandra Metrics Hi, I have recently started using Cassandra. As of now, I am only using cfstats and cfhistograms to monitor individual CF stats. What all cassandra metrics should I watch for stability and performance? Are there any tools to do the same? Is there any performance overhead if I start monitoring too many metrics? Thanks - Saurabh
Re: Deploying OpsCenter behind a HTTP(S) proxy
Well, it does sound like you just need proxies for cluster-configs, TestCluster, etc... Your site is called opscenter and then you proxy a directory called opscenter. But the 2 have nothing to do with each other. Your site can be called opscenter and then you need a proxy for TestCluster, etc... You haven't proxied opscenter the host - you've proxied opscenter the directory. Because opscenter the host has many directories with names other than opscenter, you need to have proxy rules for all those too. Or, as I'd said, you may be better off using rewrite if there are proxies that proxy is not complicated enough for. On Fri, Jun 19, 2015 at 3:15 AM, Jonathan Ballet jbal...@gfproducts.ch wrote: Hi Spencer, I certainly know how to configure a proxy or how to rewrite URLs if I need to and we are currently not looking for a contractor, but thanks for your message! :) Jonathan On 06/18/2015 11:29 AM, Spencer Brown wrote: First, your firewall should really be your frontend There operational frontend is apache, which is common. You want every url with opscenter in it handled elsewhere. You could also set up proxies for /. cluster-configs, etc... Then there is mod_rewrite, which provides a lot more granularity about when you want what gets handled where.I set up the architectural infrastructure for Orbitz and some major banks, and I'd be happpy to help you out on this. I charge $30/hr., but what you need isn't very complex so we're really just talking $100. On Thu, Jun 18, 2015 at 5:13 AM, Jonathan Ballet jbal...@gfproducts.ch mailto:jbal...@gfproducts.ch wrote: Hi, I'm looking for information on how to correctly deploy an OpsCenter instance behind a HTTP(S) proxy. I have a running instance of OpsCenter 5.1 reachable at http://opscenter:/opscenter/ but I would like to be able to serve this kind of tool under a single hostname on HTTPS along with other tools of this kind, for easier convenience. I'm currently using Apache as my HTTP front-end and I tried this naive configuration: VirtualHost *:80 ServerName tools ... ProxyPreserveHost On # Proxy to OpsCenter # ProxyPass /opscenter/ http://opscenter:/opscenter/ ProxyPassReverse/opscenter/ http://opscenter:/opscenter/ /VirtualHost This doesn't quite work, as OpsCenter seem to also serve specific endpoints from / directly Of course, it doesn't correctly work, as OpsCenter seem to also serve specific data from / directly, such as: /cluster-configs /TestCluster /meta /rc /tcp Is there something I can configure in OpsCenter so that it serves these URLs from somewhere else, or a list of known URLs that I can remap on the proxy, or better yet, a known proxy configuration to put in front of OpsCenter? Regards, Jonathan
Record deletion job
Hi Is their any way to schedule a job in cassandra to delete the recrods which are older than a specific time period. Excluding the option of TTL. Regards Anil Sent from Samsung Mobile
Re: Deploying OpsCenter behind a HTTP(S) proxy
Hi Spencer, I certainly know how to configure a proxy or how to rewrite URLs if I need to and we are currently not looking for a contractor, but thanks for your message! :) Jonathan On 06/18/2015 11:29 AM, Spencer Brown wrote: First, your firewall should really be your frontend There operational frontend is apache, which is common. You want every url with opscenter in it handled elsewhere. You could also set up proxies for /. cluster-configs, etc... Then there is mod_rewrite, which provides a lot more granularity about when you want what gets handled where.I set up the architectural infrastructure for Orbitz and some major banks, and I'd be happpy to help you out on this. I charge $30/hr., but what you need isn't very complex so we're really just talking $100. On Thu, Jun 18, 2015 at 5:13 AM, Jonathan Ballet jbal...@gfproducts.ch mailto:jbal...@gfproducts.ch wrote: Hi, I'm looking for information on how to correctly deploy an OpsCenter instance behind a HTTP(S) proxy. I have a running instance of OpsCenter 5.1 reachable at http://opscenter:/opscenter/ but I would like to be able to serve this kind of tool under a single hostname on HTTPS along with other tools of this kind, for easier convenience. I'm currently using Apache as my HTTP front-end and I tried this naive configuration: VirtualHost *:80 ServerName tools ... ProxyPreserveHost On # Proxy to OpsCenter # ProxyPass /opscenter/ http://opscenter:/opscenter/ ProxyPassReverse/opscenter/ http://opscenter:/opscenter/ /VirtualHost This doesn't quite work, as OpsCenter seem to also serve specific endpoints from / directly Of course, it doesn't correctly work, as OpsCenter seem to also serve specific data from / directly, such as: /cluster-configs /TestCluster /meta /rc /tcp Is there something I can configure in OpsCenter so that it serves these URLs from somewhere else, or a list of known URLs that I can remap on the proxy, or better yet, a known proxy configuration to put in front of OpsCenter? Regards, Jonathan
abnormal log after remove a node
I have a C* 2.1.5 with 24 nodes.A few days ago ,I have remove a node from this cluster using nodetool decommission. But tody I find some log like this: INFO [GossipStage:1] 2015-06-19 17:38:05,616 Gossiper.java:968 - InetAddress /172.19.105.41 is now DOWN INFO [GossipStage:1] 2015-06-19 17:38:05,617 StorageService.java:1885 - Removing tokens [-1014432261309809702, -1055322450438958612, -1120728727235087395, -1191392141261832305, -1203676771883970142, -1215563040745505837, -1215648909329054362, -1269531760567530381, -1278047879489577908, -1313427877031136549, -1342822572958042617, -1350792764922315814, -1383390744017639599, -139000372807970456, -140827955201469664, -1631551789771606023, -1633789813430312609, -1795528665156349205, -1836619444785023397, -1879127294549041822, -1962337787208890426, -2022309807234530256, -2033402140526360327, -2089413865145942100, -210961549458416802, -2148530352195763113, -2184481573787758786, -610790268720205, -2340762266634834427, -2513416003567685694, -2520971378752190013, -2596695976621541808, -2620636796023437199, -2640378596436678113, -2679143017361311011, -2721176590519112233, -2749213392354746126, -279267896827516626, -2872377759991294853, -2904711688111888325, -290489381926812623, -3000574339499272616, -301428600802598523, -3019280155316984595, -3024451041907074275, -3056898917375012425, -3161300347260716852, -3166392383659271772, -3327634380871627036, -3530685865340274372, -3563112657791369745, -366930313427781469, -3729582520450700795, -3901838244986519991, -4065326606010524312, -4174346928341550117, -4184239233207315432, -4204369933734181327, -4206479093137814808, -421410317165821100, -4311166118017934135, -4407123461118340117, -4466364858622123151, -4466939645485100087, -448955147512581975, -4587780638857304626, -4649897584350376674, -4674234125365755024 , -4833801201210885896, -4857586579802212277, -4868896650650107463, -4980063310159547694, -4983471821416248610, -4992846054037653676, -5026994389965137674, -514302500353679181 0, -5198414516309928594, -5245363745777287346, -5346838390293957674, -5374413419545696184, -5427881744040857637, -5453876964430787287, -5491923669475601173, -55219734138599212 6, -5523011502670737422, -5537121117160410549, -5557015938925208697, -5572489682738121748, -5745899409803353484, -5771239101488682535, -5893479791287484099, -59766730414807540 44, -6014643892406938367, -6086002438656595783, -6129360679394503700, -6224240257573911174, -6290393495130499466, -6378712056928268929, -6430306056990093461, -6800188263839065 013, -6912720411187525051, -7160327814305587432, -7175004328733776324, -7272070430660252577, -7307945744786025148, -742448651973108101, -7539255117639002578, -7657460716997978 94, -7846698077070579798, -7870621904906244395, -7900841391761900719, -7918145426423910061, -7936795453892692473, -8070255024778921411, -8086888710627677669, -8124855925323654 631, -8175270408138820500, -8271197636596881168, -8336685710406477123, -8466220397076441627, -8534337908154758270, -8550484400487603561, -862246738021989870, -8727219287242892 185, -8895705475282612927, -8921801772904834063, -9057266752652143883, -9059183540698454288, -9067986437682229598, -9148183367896132028, -962208188860606543, 10859447725819218 30, 1189775396643491793, 1253728955879686947, 1389982523380382228, 1429632314664544045, 143610053770130548, 150118120072602242, 1575692041584712198, 1624575905722628764, 17894 76212785155173, 1995296121962835019, 2041217364870030239, 2120277336231792146, 2124445736743406711, 2154979704292433983, 2340726755918680765, 23481654796845972, 23620268084352 24407, 2366144489007464626, 2381492708106933027, 2398868971489617398, 2427315953339163528, 2433999003913998534, 2633074510238705620, 266659839023809792, 2677817641360639089, 2 719725410894526151, 2751925111749406683, 2815703589803785617, 3041515796379693113, 3044903149214270978, 3094954503756703989, 3243933267690865263, 3246086646486800371, 33270068 97333869434, 3393657685587750192, 3395065499228709345, 3426126123948029459, 3500469615600510698, 3644011364716880512, 3693249207133187620, 3776164494954636918, 38780676797 8035, 3872151295451662867, 3937077827707223414, 4041082935346014761, 4060208918173638435, 4086747843759164940, 4165638694482690057, 4203996339238989224, 4220155275330961826, 4 366784953339236686, 4390116924352514616, 4391225331964772681, 4392419346255765958, 4448400054980766409, 4463335839328115373, 4547306976104362915, 4588174843388248100, 48438580 67983993745, 4912719175808770608, 499628843707992459, 5004392861473086088, 5021047773702107258, 510226752691159107, 5109551630357971118, 5157669927051121583, 51627694176199618 24, 5238710860488961530, 5245958115092331518, 5302459768185143407, 5373077323749320571, 5445982956737768774, 5526076427753104565, 5531878975169972758, 5590672474842108747, 561 8238086143944892, 5645763748154253201, 5648082473497629258, 5799608283794045232, 5968931466409317704, 6080339666926312644, 6222992739052178144, 6329332485451402638,
Cassandra Metrics
Hi, I have recently started using Cassandra. As of now, I am only using cfstats and cfhistograms to monitor individual CF stats. What all cassandra metrics should I watch for stability and performance? Are there any tools to do the same? Is there any performance overhead if I start monitoring too many metrics? Thanks - Saurabh
Re: nodetool repair
Hi, This is not necessarily true. Repair will induce compactions only if you have entropy in your cluster. If not it will just read your data to compare all the replica of each piece of data (using indeed cpu and disk IO). If there is some data missing it will repair it. Though, due to merkle tree size, you will generally stream more data than just the data needed. To limit this downside and the compactions amount, use range repairs -- http://www.datastax.com/dev/blog/advanced-repair-techniques. About tombstones, they will be evicted only after gc_grace_period and only if all the parts of the row are part of the compaction. C*heers, Alain 2015-06-19 9:08 GMT+02:00 arun sirimalla arunsi...@gmail.com: Yes compactions will remove tombstones On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay jean.tremb...@zen-innovations.com wrote: Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction, does it mean that it would also clean up my tombstone from my LeveledCompactionStrategy tables at the same time? Thanks for your help. On 19 Jun 2015, at 07:56 , arun sirimalla arunsi...@gmail.com wrote: Hi Jean, Running nodetool repair on a node will repair only that node in the cluster. It is recommended to run nodetool repair on one node at a time. Few things to keep in mind while running repair 1. Running repair will trigger compactions 2. Increase in CPU utilization. Run node tool repair with -pr option, so that it will repair only the range that node is responsible for. On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay jean.tremb...@zen-innovations.com wrote: Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? Kind regards On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.com wrote: If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.com wrote: Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help. -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html
Re: counters still inconsistent after repair
Thanks Rob, this was helpful. More counters will be added soon, I'll let you know if those have any problems. On Mon, Jun 15, 2015 at 4:32 PM, Robert Coli rc...@eventbrite.com wrote: On Mon, Jun 15, 2015 at 2:52 PM, Dan Kinder dkin...@turnitin.com wrote: Potentially relevant facts: - Recently upgraded to 2.1.6 from 2.0.14 - This table has ~million rows, low contention, and fairly high increment rate Can you repro on a counter that was created after the upgrade? Mainly wondering: - Is this known or expected? I know Cassandra counters have had issues but thought by now it should be able to keep a consistent counter or at least repair it... All counters which haven't been written to after 2.1 new counters are still on disk as old counters and will remain that way until UPDATEd and then compacted together with all old shards. Old counters can exhibit this behavior. - Any way to reset this counter? Per Aleksey (in IRC) you can turn a replica for an old counter into a new counter by UPDATEing it once. In order to do that without modifying the count, you can [1] : UPDATE tablename SET countercolumn = countercolumn +0 where id = 1; The important caveat that this must be done at least once per shard, with one shard per RF. The only way one can be sure that all shards have been UPDATEd is by contacting each replica node and doing the UPDATE + 0 there, because local writes are preferred. To summarize, the optimal process to upgrade your pre-existing counters to 2.1-era new counters : 1) get a list of all counter keys 2) get a list of replicas per counter key 3) connect to each replica for each counter key and issue an UPDATE + 0 for that counter key 4) run a major compaction As an aside, Aleksey suggests that the above process is so heavyweight that it may not be worth it. If you just leave them be, all counters you're actually used will become progressively more accurate over time. =Rob [1] Special thanks to Jeff Jirsa for verifying that this syntax works. -- Dan Kinder Senior Software Engineer Turnitin – www.turnitin.com dkin...@turnitin.com
Re: nodetool repair
Thank you for your reply. On 19 Jun 2015, at 20:36, sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com sean_r_dur...@homedepot.commailto:sean_r_dur...@homedepot.com wrote: It seems to me that running repair on any given node may also induce repairs to related replica nodes. For example, if I run repair on node A and node B has some replicas, data might stream from A to B (assuming A has newer/more data). Now, that does NOT mean that node B will be fully repaired. You still need to run repair -pr on all nodes before gc_grace_seconds. You can run repairs on multiple nodes at the same time. However, you might end up with a large amount of streaming, if many repairs are needed. So, you should be aware of a performance impact. I run weekly repairs on one node at a time, if possible. On, larger rings, though, I run repairs on multiple nodes staggered by a few hours. Once your routine maintenance is established, repairs will not run for very long. But, if you have a large ring that hasn’t been repaired, those first repairs may take days (but should get faster as you get further through the ring). Sean Durity From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Friday, June 19, 2015 3:56 AM To: user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: nodetool repair Hi, This is not necessarily true. Repair will induce compactions only if you have entropy in your cluster. If not it will just read your data to compare all the replica of each piece of data (using indeed cpu and disk IO). If there is some data missing it will repair it. Though, due to merkle tree size, you will generally stream more data than just the data needed. To limit this downside and the compactions amount, use range repairs -- http://www.datastax.com/dev/blog/advanced-repair-techniques. About tombstones, they will be evicted only after gc_grace_period and only if all the parts of the row are part of the compaction. C*heers, Alain 2015-06-19 9:08 GMT+02:00 arun sirimalla arunsi...@gmail.commailto:arunsi...@gmail.com: Yes compactions will remove tombstones On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction, does it mean that it would also clean up my tombstone from my LeveledCompactionStrategy tables at the same time? Thanks for your help. On 19 Jun 2015, at 07:56 , arun sirimalla arunsi...@gmail.commailto:arunsi...@gmail.com wrote: Hi Jean, Running nodetool repair on a node will repair only that node in the cluster. It is recommended to run nodetool repair on one node at a time. Few things to keep in mind while running repair 1. Running repair will trigger compactions 2. Increase in CPU utilization. Run node tool repair with -pr option, so that it will repair only the range that node is responsible for. On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? Kind regards On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.commailto:j...@jonhaddad.com wrote: If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help. -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera)
RE: nodetool repair
It seems to me that running repair on any given node may also induce repairs to related replica nodes. For example, if I run repair on node A and node B has some replicas, data might stream from A to B (assuming A has newer/more data). Now, that does NOT mean that node B will be fully repaired. You still need to run repair -pr on all nodes before gc_grace_seconds. You can run repairs on multiple nodes at the same time. However, you might end up with a large amount of streaming, if many repairs are needed. So, you should be aware of a performance impact. I run weekly repairs on one node at a time, if possible. On, larger rings, though, I run repairs on multiple nodes staggered by a few hours. Once your routine maintenance is established, repairs will not run for very long. But, if you have a large ring that hasn’t been repaired, those first repairs may take days (but should get faster as you get further through the ring). Sean Durity From: Alain RODRIGUEZ [mailto:arodr...@gmail.com] Sent: Friday, June 19, 2015 3:56 AM To: user@cassandra.apache.org Subject: Re: nodetool repair Hi, This is not necessarily true. Repair will induce compactions only if you have entropy in your cluster. If not it will just read your data to compare all the replica of each piece of data (using indeed cpu and disk IO). If there is some data missing it will repair it. Though, due to merkle tree size, you will generally stream more data than just the data needed. To limit this downside and the compactions amount, use range repairs -- http://www.datastax.com/dev/blog/advanced-repair-techniques. About tombstones, they will be evicted only after gc_grace_period and only if all the parts of the row are part of the compaction. C*heers, Alain 2015-06-19 9:08 GMT+02:00 arun sirimalla arunsi...@gmail.commailto:arunsi...@gmail.com: Yes compactions will remove tombstones On Thu, Jun 18, 2015 at 11:46 PM, Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Perfect thank you. So making a weekly nodetool repair -pr” on all nodes one after the other will repair my cluster. That is great. If it does a compaction, does it mean that it would also clean up my tombstone from my LeveledCompactionStrategy tables at the same time? Thanks for your help. On 19 Jun 2015, at 07:56 , arun sirimalla arunsi...@gmail.commailto:arunsi...@gmail.com wrote: Hi Jean, Running nodetool repair on a node will repair only that node in the cluster. It is recommended to run nodetool repair on one node at a time. Few things to keep in mind while running repair 1. Running repair will trigger compactions 2. Increase in CPU utilization. Run node tool repair with -pr option, so that it will repair only the range that node is responsible for. On Thu, Jun 18, 2015 at 10:50 PM, Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Thanks Jonathan. But I need to know the following: If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? Kind regards On 18 Jun 2015, at 19:19 , Jonathan Haddad j...@jonhaddad.commailto:j...@jonhaddad.com wrote: If you're using DSE, you can schedule it automatically using the repair service. If you're open source, check out Spotify cassandra reaper, it'll manage it for you. https://github.com/spotify/cassandra-reaper On Thu, Jun 18, 2015 at 12:36 PM Jean Tremblay jean.tremb...@zen-innovations.commailto:jean.tremb...@zen-innovations.com wrote: Hi, I want to make on a regular base repairs on my cluster as suggested by the documentation. I want to do this in a way that the cluster is still responding to read requests. So I understand that I should not use the -par switch for that as it will do the repair in parallel and consume all available resources. If you issue a “nodetool repair” on one node will it repair all the nodes in the cluster or only the one on which we issue the command? If it repairs only one node, do I have to wait that the nodetool repair ends, and only then issue another “nodetool repair” on the next node? If we had down time periods I would issue a nodetool -par, but we don’t have down time periods. Sorry for the stupid questions. Thanks for your help. -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the
