Re: apache-cassandra 2.2.8 rpm

2018-06-05 Thread Carlos Rolo
ade to the 2.2.12. [image: Pythian] *Carlos Rolo* | Open Source Consultant | [image: LinkedIn] <https://www.linkedin.com/in/carlosjuzarterolo/> *m* +351 918 918 100 r...@pythian.com *www.pythian.com* <https://www.google.com/url?q=https%3A%2F%2Fwww.pythian.com=D=1=AFQjCNHhR4YJfBb19Qxgli

Re: cassandra repair takes ages

2018-04-22 Thread Carlos Rolo
> throughout all the nodes. > > Are there any more downsides to run nodetool repair -pr simultaneously on > each node, besides the cpu and mem overload? > > Also if someone can clarify about the safety of an incremental repair. > > > thanks, > > nuno > ---

Re: cassandra repair takes ages

2018-04-20 Thread Carlos Rolo
Changing the datadrives to SSD would help to speed up the repairs. Also don't run 3 node, RF2. That makes Quorum = All. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 |

Re: Latest version and Features

2018-04-12 Thread Carlos Rolo
com/in/carlosjuzarterolo>* Mobile: +351 918 918 100 www.pythian.com On Thu, Apr 12, 2018 at 5:02 AM, Michael Shuler <mich...@pbandjelly.org> wrote: > On 04/11/2018 06:12 PM, Carlos Rolo wrote: > > > > I blogged about this decision recently > > here: https://blog.pythian.co

Re: Latest version and Features

2018-04-11 Thread Carlos Rolo
If you are on 3.1.0 I would move forward to 3.11.2. I blogged about this decision recently here: https://blog.pythian.com/what-cassandra-version-should-i-use-2018/ Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data

Re: Is this SSTable restore merging scenario possible ?

2018-03-21 Thread Carlos Rolo
As said before, as long as you rename the UUIDs to match you should be good. The Production "win out" depends on the timestamps. In Cassandra last write wins, so as long as, for the same row, the production timestamps are more recent than the secondary cluster, the production data would "win

Re: Is there any limit in the number of partitions that a table can have

2018-03-07 Thread Carlos Rolo
in. > > F Javier Pareja > > On Wed, Mar 7, 2018 at 5:20 PM, Jeff Jirsa <jji...@gmail.com> wrote: > >> >> >> On Wed, Mar 7, 2018 at 7:13 AM, Carlos Rolo <r...@pythian.com> wrote: >> >>> Hi Jeff, >>> >>> Could you ex

Re: Is there any limit in the number of partitions that a table can have

2018-03-07 Thread Carlos Rolo
Hi Jeff, Could you expand: "Tables without clustering keys are often deceptively expensive to compact, as a lot of work (relative to the other cell boundaries) happens on partition boundaries." This is something I didn't know and highly interesting to know more about! -- Carlos Rolo O

Re: [External] Re: Whch version is the best version to run now?

2018-03-06 Thread Carlos Rolo
Hello, Our 5 cents. Either 3.0.16 or 3.11.x We are really happy with the way 3.11.1/2 is behaving. We still have a lot of really well behaving Clusters in 2.1/2.2 latest. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data

Re: Cassandra Summit 2019 / Cassandra Summit 2018

2018-02-27 Thread Carlos Rolo
Hello all, I'm interested planning/organizing a small kinda of NGCC in Lisbon, Portugal in late May early June. Just waiting for the venue to confirm possible dates. Would be a 1day event kinda last year, is this something people would be interested? I can push a google form for accessing the

Re: What happens if multiple processes send create table if not exist statement to cassandra?

2018-01-27 Thread Carlos Rolo
Don't do that. Worst case you might get different schemas in flight and no agreement on your cluster. If you are already doing that, check "nodetool describecluster" after you do that. Like Jeff said, it is likely to cause problems. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax

RE: Meltdown/Spectre Linux patch - Performance impact on Cassandra?

2018-01-13 Thread Carlos Rolo
We are seeing almost no impact on Azure (+1 or 2%). Non-patched OS. On AWS we don't have any solid data yet. On 13 Jan 2018 09:46, "Steinmaurer, Thomas" < thomas.steinmau...@dynatrace.com> wrote: Hello Ben, thanks for the notice. Similar here + others reporting as well:

Re: Stable Cassandra 3.x version for production

2017-11-07 Thread Carlos Rolo
DSE is already pushing 3.11 in 5.1 If you're going into 3.x either 3.11.1 or 3.0.15. I would recommend 3.11.1 to take advantage of a couple of features (ex: Slow query log) that you don't have on 3.0. If you're stable on 2.x and you don't need any new feature, you can leave it there safely

Re: Schema Mismatch Issue in Production

2017-10-12 Thread Carlos Rolo
Which version are you running? I got stuck in a similar situation (With a lot more nodes) and the only way to make it good was to stop the whole cluster, start nodes 1 by 1. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your

Re: Materialized views stability

2017-10-02 Thread Carlos Rolo
I've been dealing with MV extensively, and I second Blake. MVs are not suitable for production. Unless you're ready for the pain (The out of sync is a major pain point), I would not go that way. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Re: Reaper 0.7 is released!

2017-09-27 Thread Carlos Rolo
Thanks a lot for the release! On 27 Sep 2017 10:21 pm, "Jon Haddad" wrote: > We’ve discussed it, and it’s in GitHub, but we haven’t scheduled it yet. > For now we’re trying to make it easier to use, cut down on it’s query > count, and ensure what’s there is solid. The team

Re: Self-healing data integrity?

2017-09-14 Thread Carlos Rolo
Wouldn't be easier for 1) The CRC to be checked by the sender, and don't send if it doesn't match? 2) And once the stream ends, you could compare the 2 CRCs to see if something got weird during transfer? Also you could implement this in 2 pieces instead of reviewing the streaming architecture

Re: Manual repair not showing in the log.

2017-09-07 Thread Carlos Rolo
Can you check if you have any validation compaction running in nodetool compactionstats? On 7 Sep 2017 7:56 pm, "Mark Furlong" wrote: I have started a repair and I received the message ‘Starting repair command #1, repairing 25301 ranges for keyspace x

Re: truncate table in C* 3.11.0

2017-09-07 Thread Carlos Rolo
If you waited less than 60s, no warning/error was issued. Do the following: * Check if all nodes are up (truncate fails if not) * Check if you got a snapshot generated (unless you have auto_snapshot disabled) * Check if you have still the sstables in the directories (you shouldn't) If it didn't

Re: Cassandra Setup Question

2017-08-23 Thread Carlos Rolo
Use networktopologystrategy as replication strategy and make sure you have dc1: 3 and dc2: 3. This way you have 3 replicas in each DC. On 23 Aug 2017 12:53, "Jonathan Baynes" wrote: > Hi Community, > > > > Quick question regarding Replication Factor. > > > > In

Re: Adding a new node with the double of disk space

2017-08-18 Thread Carlos Rolo
...@reddit.com.invalid> > wrote: > >> Are you saying if a node had double the hardware capacity in every way it >> would be a bad idea to up num_tokens? I thought that was the whole idea of >> that setting though? >> >> On Thu, Aug 17, 2017 at 9:52 AM, Carlos Ro

Re: Adding a new node with the double of disk space

2017-08-17 Thread Carlos Rolo
No. If you would double all the hardware on that node vs the others would still be a bad idea. Keep the cluster uniform vnodes wise. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter: @cjrolo |

Re: c* updates not getting reflected.

2017-07-11 Thread Carlos Rolo
What consistency are you using on those queries? On 11 Jul 2017 19:09, "techpyaasa ." wrote: > Hi, > > We have a table with following schema: > > CREATE TABLE ks1.cf1 ( pid bigint, cid bigint, resp_json text, status int, > PRIMARY KEY (pid, cid) ) WITH CLUSTERING ORDER BY

Re: Reaper v0.6.1 released

2017-06-15 Thread Carlos Rolo
Great! Thanks a lot! Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: *linkedin.com/in/carlosjuzarterolo * Mobile:

Re: Stable version apache cassandra 3.X /3.0.X

2017-05-31 Thread Carlos Rolo
On sync in Jon. Only go 3.0.x if you REALLY need something from there (ex: MV) even then, be carefull. 3.x wait for 3.11.x. 3.10 if you REALLY need something from there right now. Latest 2.2.x or 2.1.x if you are just doing baseline Cassandra and need the stability. Regards, Carlos Juzarte

Re: Slowness in C* cluster after implementing multiple network interface configuration.

2017-05-24 Thread Carlos Rolo
It might be a bug. Cassandra, AFAIK, scans those files for changes and updates the topology (So you don't need a restart if you change the files). It might be the case that the absence of the file, is still noticed by Cassandra even if it is not really used. I can do a small test to confirm, if

Re: Cassandra Schema version mismatch

2017-05-06 Thread Carlos Rolo
Sometimes "resetlocalschema" (happened to me, didn't check why) will not work, and you need to stop the offending nodes and bring them back one by one.. That solved the issue. In the cases I've seen this happened the clusters in question where either: a) A couple of nodes down (cloud provider

Re: Cassandra Schema version mismatch

2017-05-05 Thread Carlos Rolo
Are you changing the schema in a dynamic fashion? If you get problems (network, gc pauses, etc) during the schema changes it might lead to that. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter:

Re: cassandra OOM

2017-04-25 Thread Carlos Rolo
To add some contribution to this thread, we have seen both cases. CMS easily outperforming G1 for the same Heapsize and the inverse too. On the same cluster different workloads (datacenter based) we have both collectors because of performance based on the workload. It would be good to colect this

Re: Drop tables takes too long

2017-04-20 Thread Carlos Rolo
You have 4800 Tables in total? That is a lot of tables, plus MVs? or MVs are already considered in the 60*80 account? I would recommend to reduce the table number. Other thing is that you need to check your log file for GC Pauses, and how long those pauses take. You also might need to increase

Re: WriteTimeoutException with LWT after few milliseconds

2017-04-12 Thread Carlos Rolo
You can try to use TRACING to debug the situation, but for a LWT to fail so fast, the most probable cause is what you stated: "It is possible that there are concurrent inserts on the same PK - actually thats the reason why I use LWTs." AKA, someone inserted first. Regards, Carlos Juzarte Rolo

Re: too many compactions pending and compaction is slow on few tables

2017-04-07 Thread Carlos Rolo
Is not a good idea to do LCS on spinning. Change to STCS, and reduce the compactors to 2 (if you have more than 2). Check if that helps. On Apr 7, 2017 20:18, "Matija Gobec" wrote: > It does as the "new" data, even if the values are the same, has new write > time

Re: Node always dieing

2017-04-06 Thread Carlos Rolo
i3 are having those issues more than the other instances it seems. Not the first report I heard about. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:

Re: Node always dieing

2017-04-06 Thread Carlos Rolo
There was some issue with the i3 instances and Cassandra. Did you had this cluster running always on i3? On Apr 6, 2017 13:06, "Cogumelos Maravilha" wrote: > Limit Soft Limit Hard Limit > Units > Max cpu time unlimited

Re: Archive node

2017-03-06 Thread Carlos Rolo
I would not suggest to do that, because the new "Archive" node would be a new DC that you would need to build (Operational wise). You could also snapshot the old one once it finishes and use SSTableloader to push it into your Development DC. This way you have isolation from Production. Plus no

Re: Read exceptions after upgrading to 3.0.10

2017-02-24 Thread Carlos Rolo
By any chances are you using the PHP/C++ driver? -- --

Re: Global TTL vs Insert TTL

2017-02-01 Thread Carlos Rolo
erform >> optimization when compacting. >> >> For example if it can see than the maxTimestamp of an SSTable is older >> than the table Global TTL, the SSTable can be entirely dropped during >> compaction >> >> Using dynamic TTL at runtime, since Cassandra d

Re: Global TTL vs Insert TTL

2017-02-01 Thread Carlos Rolo
To reinforce Alain statement: "I would say that the unsafe part is more about using C* 3.9" this is key. You would be better on 3.0.x unless you need features on the 3.x series. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your

Re: Has anyone deployed a production cluster with less than 6 nodes per DC?

2016-12-26 Thread Carlos Rolo
It depends on a lot of factors. What causes the cluster to get crazy? I/O, Network, CPU? I manage clusters of all sizes (even 3 nodes per DC) but it all depends on usage and configuration. Regards, Carlos Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect /

Re: Join_ring=false Use Cases

2016-12-20 Thread Carlos Rolo
Beware the Java Driver limitations around whitelisting IPs. Works fine in Python. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:

Re: Failure when setting up cassandra in cluster

2016-08-22 Thread Carlos Rolo
If Ryan answer doesn't help, post Cassandra version. There is a bug with cql and some python version that would lead to that error. Also, please post "nodetool status". Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data

Re: Support/Consulting companies

2016-08-22 Thread Carlos Rolo
https://www.pythian.com/ Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: *linkedin.com/in/carlosjuzarterolo * Mobile:

Re: IF EXISTS checks on all nodes?

2016-05-12 Thread Carlos Rolo
Hello, As far as I know, lightweight transactions only apply to a single partition, so in your case it will only execute on the nodes responsible for that partition. And as a consequence, those nodes will all be in the same state when the transaction ends (If it would apply). Please refer to

Re: COPY TO export fails with

2016-05-10 Thread Carlos Rolo
Hello, That is a lot of data to do an "COPY TO. If you want a fast way to export, and you're fine with Java, you can use Cassandra SSTableReader classes to read the sstables directly. Spark also works. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra

Re: Changing snitch from PropertyFile to Gossip

2016-04-25 Thread Carlos Rolo
r is there a specific reason? > > C*heers, > --- > Alain Rodriguez - al...@thelastpickle.com > France > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > 2016-04-24 15:44 GMT+02:00 Carlos Rolo <r...@pythian.com>

Re: Changing snitch from PropertyFile to Gossip

2016-04-24 Thread Carlos Rolo
As long as the topology doesn't change, yes. Repair once you finish. Em 24/04/2016 13:23, "AJ" escreveu: > Is it possible to do this without down time i.e. run in mixed mode while > doing a rolling upgrade? -- --

Re: Most stable version?

2016-04-22 Thread Carlos Rolo
are you suggesting at some point there will be a stable train for 3? > (or that 3.x will be bumped to 4.0 when stable)? > > We're on 2.2.5 and haven't seen any major problems with it. > > -J > > > > Sent via iPhone > > On Apr 22, 2016, at 03:34, Carlos Rolo <r..

Re: Most stable version?

2016-04-22 Thread Carlos Rolo
.X? or > latest 2.2.x is recommended? > > Currently I use 2.2.5, but is the latest 2.1.13 recommended for production > use? > > Regards, > Satoshi > > > On Mon, Apr 18, 2016 at 11:45 PM, Carlos Rolo <r...@pythian.com> wrote: > >> Sorry to resurrect this now, but I d

Re: Most stable version?

2016-04-18 Thread Carlos Rolo
Sorry to resurrect this now, but I don't consider anything after 3.0.x stable. I wrote a blog post about this to be clear: https://www.pythian.com/blog/cassandra-version-production/ Use it and pick a version based on your needs. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax

Re: Do I have to use repair -inc with the option -par forcely?

2016-02-16 Thread Carlos Rolo
+1 on what Alain said, but I do think if you are high enough on a 2.1.x (will look later) version you don't need to follow the documentation. It is outdated. Run a full repair, the you can start incremental repairs since the SSTables will have the metadata on them about the last repair. Wait

Re: Latest stable release

2016-02-08 Thread Carlos Rolo
I honestly go with 2.1.13 unless you need the features on 2.2.x. I would not recommend 3.x for now (unless you need the features). Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter: @cjrolo |

Re: Cassandra is consuming a lot of disk space

2016-01-13 Thread Carlos Rolo
You can check if the snapshot exists in the snapshot folder. Repairs stream sstables over, than can temporary increase disk space. But I think Carlos Alonso might be correct. Running compactions might be the issue. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data

Re: Unable to start one Cassandra node: OutOfMemoryError

2015-12-10 Thread Carlos Rolo
Dealt with that recently, and the only solution that made it work was to increase heap sizes. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: @cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo *

Re: scylladb

2015-11-11 Thread Carlos Rolo
com/in/carlosjuzarterolo>* Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649 www.pythian.com On Wed, Nov 11, 2015 at 4:46 PM, Dani Traphagen <dani.trapha...@datastax.com > wrote: > Killer, @cjrolo. Will you update via this thread? > > On Wed, Nov 11, 2015 at 7:57 AM, Carlos R

Re: scylladb

2015-11-11 Thread Carlos Rolo
Not yet, but not far from doing it. No rain here yet! :) On a more serious tone, should be done before end of the Month. -- --

Re: scylladb

2015-11-05 Thread Carlos Rolo
I will not try until multi-DC is implemented. More than an month has passed since I looked for it, so it could possibly be in place, if so I may take some time to test it. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: @cjrolo | Linkedin:

Re: scylladb

2015-11-05 Thread Carlos Rolo
On Thursday, November 5, 2015, Carlos Rolo <r...@pythian.com> wrote: > >> I will not try until multi-DC is implemented. More than an month has >> passed since I looked for it, so it could possibly be in place, if so I may >> take some time to test it. >> >> Reg

Re: any update about CASSANDRA-10420

2015-10-16 Thread Carlos Rolo
Can you provide more information? The description is generic, driver version and a test case that the devs can use to reproduce the bug would be optimal. I'm not involved in the development of cassandra, but that bug description doesn't seem complete. Maybe it is a driver bug and not a Cassandra

Re: Removed node is not completely removed

2015-10-14 Thread Carlos Rolo
Check system.peers table to see if the IP is still there. If so edit the table and remove the offending IP. You are probably running into this: https://issues.apache.org/jira/browse/CASSANDRA-6053 Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian |

Re: howto do sql query like in a relational database

2015-07-22 Thread Carlos Rolo
Hello Anton, You need to look into Datastax Entreprise (DSE) Offering. It integrates Solr search which allows you to do searches like the one you mention. There are also some opensource projects doing this kind of integration, so its up to you. And as Oded mentioned Cassandra really shines on

Re: Seed gossip version error

2015-07-21 Thread Carlos Rolo
That error should only occur when you have a mismatch between the Seed version and the new node version. Are you sure all your nodes are running in the same version? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin:

Re: cassandra repair error

2015-07-16 Thread Carlos Rolo
Hello, It seems that is a problem with Cassandra trying to delete a directory that is not empty. First time seeing this error. Are your harddrives with enough space and working correctly? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo

Re: Wrong peers

2015-07-06 Thread Carlos Rolo
There is a bug in Jira related to this, it is not a driver issue, is a Cassandra issue. It is solved on 2.0.14 I think. I will post the ticket once I find it. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin:

Re: Error while adding a new node.

2015-07-02 Thread Carlos Rolo
Indeed you should upgrade to 2.1.7. And then report if you are still facing problems. Versions up to 2.1.5 (in the 2.1.x series) are not considered stable. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin:

Re: [MASSMAIL]Re: Error while adding a new node.

2015-07-02 Thread Carlos Rolo
mlor...@uci.cu wrote: The recommended version to use is 2.1.5 because, like you Carlos said, 2.1.6 and 2.1.7 are very new to consider them like stable. On 02/07/15 08:55, Carlos Rolo wrote: Indeed you should upgrade to 2.1.7. And then report if you are still facing problems. Versions up

Re: compaction occurring very frequently

2015-06-18 Thread Carlos Rolo
Hi, 2.1.x does have compaction issues. Upgrading to 2.1.6 should help you. You can also search this mailing list for some recommendations for that specific problem since it is a recurrent issue. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter:

Re: Replication in one datacetner only

2015-06-16 Thread Carlos Rolo
Yes, set the replication factor of the keyspace to only use the datacenter you want. More information here: http://docs.datastax.com/en/cql/3.1/cql/cql_reference/create_keyspace_r.html Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo |

Re: Missing data

2015-06-15 Thread Carlos Rolo
Hi Jean, The problem of that Warning is that you are reading too many tombstones per request. If you do have Tombstones without doing DELETE it because you probably TTL'ed the data when inserting (By mistake? Or did you set default_time_to_live in your table?). You can use nodetool cfstats to

Re: Question about nodetool status ... output

2015-06-12 Thread Carlos Rolo
Your data model also contributes to the balance (or lack of) of the cluster. If you have a really bad data partitioning Cassandra will not do any magic. Regarding that cluster, I would decommission the x.52 node and add it again with the correct configuration. After the bootstrap, run a cleanup.

Re: Lucene index plugin for Apache Cassandra

2015-06-12 Thread Carlos Rolo
Seems like an interesting tool! What operational recommendations would you make to users of this tool (Extra hardware capacity, extra metrics to monitor, etc)? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin:

Re: Hundreds of sstables after every Repair

2015-06-09 Thread Carlos Rolo
Hello, Do you have your clocks synced across your cluster? Are you using NTP and have it properly configured? Sometimes clock out of sync can trigger weird behaviour. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin:

Re: Ghost compaction process

2015-06-08 Thread Carlos Rolo
HI, Is it 2.0.14 or 2.1.4? If you are on 2.1.4 I would recommend an upgrade to 2.1.5 regardless of that issue. From the data you provide it is difficult to access what is the issue. If you are running with RF=2 you can always add another node and kill that one if that is the only node that shows

Re: Reading too many tombstones

2015-06-04 Thread Carlos Rolo
The TTL data will only be removed after the gc_grace_seconds. So your data with 30 days TTL will be still in Cassandra for 10 days more (40 in total). Is your data being there for more than that? Otherwise it is expected behaviour and probably you should do something on your data model to avoid

Re: How to interpret some GC logs

2015-06-03 Thread Carlos Rolo
GC Logs are a weird science. I use a couple of resources to get through them. Regarding your question my 1.8.0_40 always have the first the -. I greped through 2h of logs, and on a test environment. I use the following set of options: -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC

Re: Multiple cassandra instances per physical node

2015-05-21 Thread Carlos Rolo
Hi, I also advice against multiple instances on the same hardware. If you have really big boxes why not virtualize? Other option is experiment with CCM. Although there are some limitations with CCM (ex: JNA is disabled) If you follow up on this I would to hear how it went. Em 21/05/2015 19:33,

Re: minimum bandwidth requirement between two Geo Redundant sites of Cassandra database

2015-04-28 Thread Carlos Rolo
Hi, I would not recommend anything below 1Gbps for the bandwidth. Latency try to have it as low as you can. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo

Re: New node got stuck joining the cluster after a while

2015-04-28 Thread Carlos Rolo
Hi, The 2.1.x series is not recommeded for use, especially the first versions. I would downgrade to 2.0.14 or if must stay on 2.1 upgrade your cluster to 2.1.4 or the imminent release of 2.1.5. This mailing list as a few tips how to deal with the 2.1.x releases, but the best way is indeed a

Re: What is 'Read Reuqests' on OpsCenter exaclty?

2015-04-24 Thread Carlos Rolo
. A future version of OpsC will include tooltips with these descriptions for better clarity. On Apr 23, 2015 6:30 AM, Carlos Rolo r...@pythian.com wrote: Probably it takes in account the read repair, plus a read that have consistency != 1 will produce reads on other machines (which are taken

Re: What is 'Read Reuqests' on OpsCenter exaclty?

2015-04-23 Thread Carlos Rolo
Probably it takes in account the read repair, plus a read that have consistency != 1 will produce reads on other machines (which are taken in account). I don't know the internals of opscenter but I would assume that this is the case. If you want to test it further, disable read_repair, and make

Re: LCS Strategy, compaction pending tasks keep increasing

2015-04-21 Thread Carlos Rolo
Are you on version 2.1.x? Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649 www.pythian.com On

Re: Adding nodes to existing cluster

2015-04-20 Thread Carlos Rolo
Start one node at a time. Wait 2 minutes before starting each node. How much data and nodes you have already? Depending on that, the streaming of data can stress on the resources you have. I would recommend to start one and monitor, if things are ok, add another one. And so on. Regards, Carlos

Re: Adding nodes to existing cluster

2015-04-20 Thread Carlos Rolo
by one if we're not using rack awareness? On Mon, Apr 20, 2015 at 2:49 PM, Carlos Rolo r...@pythian.com wrote: Start one node at a time. Wait 2 minutes before starting each node. How much data and nodes you have already? Depending on that, the streaming of data can stress

Re: Best practice: Multiple clusters vs multiple tables in a single cluster?

2015-04-02 Thread Carlos Rolo
Adding a new keyspace should be perfectly fine. Unless you have completely distinct workloads for the different keyspaces. Even so you can balanced some stuff at keyspace/table level. But I would go with a new keyspace not with a new cluster given the small size you say you have. Regards, Carlos

Re: Replication to second data center with different number of nodes

2015-03-30 Thread Carlos Rolo
Sharing my experience here. 1) Never had any issues with different size DCs. If the hardware is the same, keep the # to 256. 2) In most of the cases I keep the 256 vnodes and no performance problems (when they are triggered, the cause is not the vnodes #) Regards, Carlos Juzarte Rolo Cassandra

Re: Upgrade from 2.0.9 to 2.1.3

2015-03-06 Thread Carlos Rolo
I would not recommend an upgrade to 2.1.x for now. Do you have any specific reason to upgrade? For upgrading from 2.0.9 you can just do a direct upgrade. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin:

Re: Write timeout under load but Read is fine

2015-03-06 Thread Carlos Rolo
What is the consistency level you are using? Are you using Thrift or CQL? Are you using SSDs? Check if compactions are running when you get the timeouts. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin:

Re: Unexplained query slowness

2015-02-25 Thread Carlos Rolo
You can use query tracing to check what is happening. Also you fire jconsole/JavaVisualVM and push out some metrics like the 99th read Beans for that column family. A simpler check is using cfstats and look for weird numbers (high number sstables, if you are deleting check how much tombstones per

Re: Setting up JNA on CentOS 6.6. with cassandra20-2.0.12 and Oracle Java 1.7.0_75

2015-02-25 Thread Carlos Rolo
Hello, I always install JNA into the lib directory of java itself Since I normally have java in /opt/java I put the JNA into /opt/java/lib. ~$ grep JNA /var/log/cassandra/system.log INFO HH:MM:SS JNA mlockall successful Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your

Re: Setting up JNA on CentOS 6.6. with cassandra20-2.0.12 and Oracle Java 1.7.0_75

2015-02-25 Thread Carlos Rolo
Also I always install JNA from the JNA page. I did the installation for this blog post in CentOS 6.5: http://www.pythian.com/blog/from-0-to-cassandra-an-exhaustive-approach-to-installing-cassandra/ Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian |

Re: Possible problem with disk latency

2015-02-25 Thread Carlos Rolo
Your latency doesn't seem that high that can cause that problem. I suspect more of a problem with the Cassandra version (2.1.3) than that with the hard drives. I didn't look deep into the information provided but for your reference, the only time I had serious (leading to OOM and all sort of weird

Re: AMI to use to launch a cluster with OpsCenter on AWS

2015-02-23 Thread Carlos Rolo
Regarding AWS the only thing I normally do (besides the normal installation, etc) is setting up the firewall zones so the ports needed for Cassandra are open. You can follow this guide: https://razvantudorica.com/02/create-a-cassandra-cluster-with-opscenter-on-amazon-ec2/a Regards, Carlos

Re: C* 2.1.2 invokes oom-killer

2015-02-19 Thread Carlos Rolo
So compaction doesn't seem to be your problem (You can check with nodetool compactionstats just to be sure). How much is your write latency on your column families? I had OOM related to this before, and there was a tipping point around 70ms. -- --

Re: C* 2.1.2 invokes oom-killer

2015-02-19 Thread Carlos Rolo
rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo http://linkedin.com/in/carlosjuzarterolo* Tel: 1649 www.pythian.com On Thu, Feb 19, 2015 at 10:49 AM, Michał Łowicki mlowi...@gmail.com wrote: On Thu, Feb 19, 2015 at 10:41 AM, Carlos Rolo r...@pythian.com wrote: So

Re: C* 2.1.2 invokes oom-killer

2015-02-19 Thread Carlos Rolo
trickle_fsync_interval_in_kb: 10240 On Thu, Feb 19, 2015 at 12:12 PM, Michał Łowicki mlowi...@gmail.com wrote: On Thu, Feb 19, 2015 at 11:02 AM, Carlos Rolo r...@pythian.com wrote: Do you have trickle_fsync enabled? Try to enable that and see if it solves your problem, since you are getting out of non-heap memory

Re: can't delete tmp file

2015-02-19 Thread Carlos Rolo
You should upgrade to 2.1.3 for sure. Check the changelog here: https://git1-us-west.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-2.1.3 Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo |

Re: run cassandra on a small instance

2015-02-19 Thread Carlos Rolo
of output. Thanks Tim On Thu, Feb 19, 2015 at 3:40 AM, Carlos Rolo r...@pythian.com wrote: I have Cassandra instances running on VMs with smaller RAM (1GB even) and I don't go OOM when testing them. Although I use them in AWS and other providers, never tried Digital Ocean. Does Cassandra just

Re: Many pending compactions

2015-02-16 Thread Carlos Rolo
Hi 100% in agreement with Roland, 2.1.x series is a pain! I would never recommend the current 2.1.x series for production. Clocks is a pain, and check your connectivity! Also check tpstats to see if your threadpools are being overrun. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian

Re: best supported spark connector for Cassandra

2015-02-13 Thread Carlos Rolo
Not for sure ;) If you need Cassandra support I can forward you to someone to talk to at Pythian. Regards, Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo

Re: Adding new node - OPSCenter problems

2015-02-11 Thread Carlos Rolo
Hello, What is the output of nodetool status? All nodes should appear, otherwise there is some configuration error. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo

Re: Two problems with Cassandra

2015-02-11 Thread Carlos Rolo
Hello Pavel, What is the size of the Cluster (# of nodes)? And you need to iterate over the full 1TB every time you do the update? Or just parts of it? IMO information is short to make any kind of assessment of the problem you are having. I can suggest to try a 2.0.x (or 2.1.1) release to see

  1   2   >