Re: TWCS Log Warning

2024-05-23 Thread Jon Haddad
As an aside, if you're not putting a TTL on your data, it's a good idea to be proactive and use multiple tables. For example, one per month or year. This allows you the flexibility to delete your data by dropping old tables. Storing old data in Cassandra is expensive. Once you get to a certain

Re: Replication factor, LOCAL_QUORUM write consistency and materialized views

2024-05-17 Thread Jon Haddad
I strongly suggest you don't use materialized views at all. There are edge cases that in my opinion make them unsuitable for production, both in terms of cluster stability as well as data integrity. Jon On Fri, May 17, 2024 at 8:58 AM Gábor Auth wrote: > Hi, > > I know, I know, the

Re: Change num_tokens in a live cluster

2024-05-16 Thread Jon Haddad
Unless your cluster is very small, using the method of adding / removing nodes will eventually result in putting a much larger portion of your dataset on a very few number of nodes. I *highly* discourage this. The only correct, safe path is Bowen's suggestion of adding another DC and

Re: storage engine series

2024-04-30 Thread Jon Haddad
Thanks Aaron! Just realized I made a mistake, the 4th week's URL is https://www.youtube.com/watch?v=MAxQ0QygcKk. Jon On Tue, Apr 30, 2024 at 4:58 AM Aaron Ploetz wrote: > Nice! This sounds awesome, Jon. > > On Mon, Apr 29, 2024 at 6:25 PM Jon Haddad wrote: > >> Hey everyon

storage engine series

2024-04-29 Thread Jon Haddad
Hey everyone, I'm doing a 4 week YouTube series on the C* storage engine. My first video was last week where I gave an overview into some of the storage engine internals [1]. The next 3 weeks are looking at the new Trie indexes coming in 5.0 [2], running Cassandra on EBS [3], and finally

Trie Memtables

2024-04-09 Thread Jon Haddad
Hey all, Tomorrow at 10:30am PDT I'm taking a look at Trie Memtables tomorrow on my live stream. I'll do some performance comparisons between it and the legacy SkipListMemtable implementation and see what I can learn. https://www.youtube.com/live/Jp5R_-uXORQ?si=NnIoV3jqjHFoD8nF or if you

Re: Datacenter decommissioning on Cassandra 4.1.4

2024-04-08 Thread Jon Haddad
You shouldn’t decom an entire DC before removing it from replication. — Jon Haddad Rustyrazorblade Consulting rustyrazorblade.com On Mon, Apr 8, 2024 at 6:26 AM Michalis Kotsiouros (EXT) via user < user@cassandra.apache.org> wrote: > Hello community, > > In our deployments, we

Re: Query on Performance Dip

2024-04-05 Thread Jon Haddad
Try changing the chunk length parameter on the compression settings to 4kb, and reduce read ahead to 16kb if you’re using EBS or 4KB if you’re using decent local ssd or nvme. Counters read before write. — Jon Haddad Rustyrazorblade Consulting rustyrazorblade.com On Fri, Apr 5, 2024 at 9:27 AM

Re: Query on Performance Dip

2024-03-30 Thread Jon Haddad
Hi, Unfortunately, the numbers you're posting have no meaning without context. The speculative retries could be the cause of a problem, or you could simply be executing enough queries and you have a fairly high variance in latency which triggers them often. It's unclear how many queries / second

Tomorrow 10AM PDT - Examining LWT perf in 5.0

2024-03-19 Thread Jon Haddad
Hey folks, I'm doing a working session tomorrow at 10am PDT, testing LWTs in C* 5.0. I'll be running benchmarks and doing some performance analysis. Come hang out and bring your questions! Jon YouTube: https://www.youtube.com/watch?v=IoWh647LRQ0 LinkedIn:

Streaming a working session with 5.0 - UCS

2024-03-05 Thread Jon Haddad
Hey everyone, Today starting at 10am PT I'm going to be streaming my session messing with 5.0, looking at UCS. I'm doing this with my easy-cass-lab and easy-cass-stress tools using a build of C* from last night. I'll also show some of the cool things you can do with my tools. I'll be running

Re: Check out new features in K8ssandra and Mission Control

2024-02-27 Thread Jon Haddad
Hey Chris - this looks pretty interesting! It looks like there's a lot of functionality in here. * What aspects of Mission Control are dependent on using K8ssandra? * Can Mission Control work without K8ssandra? * Is mission control open source? * I'm not familiar with Vector - does it require an

stress testing & lab provisioning tools

2024-02-26 Thread Jon Haddad
Hey everyone, Over the last several months I've put a lot of work into 2 projects I started back at The Last Pickle, for stress testing Cassandra and for building labs in AWS. You may know them as tlp-stress and tlp-cluster. Since I haven't worked at TLP in almost half a decade, and am the

Re: Remove folders of deleted tables

2023-12-05 Thread Jon Haddad
I can't think of a reason to keep empty directories around, seems like a reasonable change, but I don't think you're butting up against a thing that most people would run into, as snapshots are enabled by default (auto_snapshot: true) and almost nobody changes it. The use case you described

Re: Memory and caches

2023-11-27 Thread Jon Haddad
I haven't found chunk cache to be particularly useful. It's a fairly small cache that could only help when you're dealing with a small hot dataset. I wouldn't bother increasing memory for it. Key cache can be helpful, but it depends on the workload. I generally recommend optimizing for your

Re: Running Large Clusters in Production

2020-07-10 Thread Jon Haddad
I worked on a handful of large clusters (> 200 nodes) using vnodes, and there were some serious issues with both performance and availability. We had to put in a LOT of work to fix the problems. I agree with Jeff - it's way better to manage multiple clusters than a really large one. On Fri,

Re: Upgrading cassandra cluster from 2.1 to 3.X when using custom TWCS

2020-07-09 Thread Jon Haddad
You could also pull TWCS out of the version of Cassandra you want to deploy, fix the imports and change the package name. Then you've got the same version as OSS, just under the name you're using in 2.1. Once you've moved to 3.11, you can switch to the OSS version. On Thu, Jul 9, 2020 at 9:09

Re: Cassandra upgrade from 3.11.3 -> 3.11.6

2020-06-24 Thread Jon Haddad
Generally speaking, don't run mixed versions longer than you have to, and don't upgrade that way. Why? * We don't support it. * We don't even test it. * If you run into trouble and ask for help, the first thing people will tell you is to get all nodes on the same version. Anyone that's doing so

Re: Generating evenly distributed tokens for vnodes

2020-05-29 Thread Jon Haddad
I'm on mobile now so I might be mistaken, but I don't think nodetool move works with multiple tokens On Fri, May 29, 2020, 1:48 PM Kornel Pal wrote: > Hi Anthony, > > Thank you very much for looking into using the script for initial token > generation and for providing multiple detailed methods

Re: Performance of Data Types used for Primary keys

2020-03-06 Thread Jon Haddad
It's not going to matter at all. On Fri, Mar 6, 2020, 2:15 AM Hanauer, Arnulf, Vodacom South Africa (External) wrote: > Hi Cassandra folks, > > > > Is there any difference in performance of general operations if using a > TEXT based Primary key versus a BIGINT Primary key. > > > > Our use-case

Re: Deleting data from future

2020-03-02 Thread Jon Haddad
You can issue a delete using a future timestamp. http://cassandra.apache.org/doc/latest/cql/dml.html#grammar-token-update-parameter Look for USING TIMESTAMP. Jon On Mon, Mar 2, 2020, 3:28 AM Furkan Cifci wrote: > Greetings, > In our C* cluster, one node lost time sync and it went to

Re: Should we use Materialised Views or ditch them ?

2020-02-28 Thread Jon Haddad
I also recommend avoiding them. I've seen too many clusters fall over as a result of their usage. On Fri, Feb 28, 2020 at 9:52 AM Max C. wrote: > The general view of the community is that you should *NOT* use them in > production, due to multiple serious outstanding issues (see Jira). We used

Re: New seed node in the cluster immediately UN without passing for UJ state

2020-02-13 Thread Jon Haddad
Seeds don't bootstrap, don't list new nodes as seeds. On Thu, Feb 13, 2020 at 5:23 PM Sergio wrote: > Hi guys! > > I don't know how but this is the first time that I see such behavior. I > wanted to add a new node in the cluster and it looks to be working fine but > instead to wait for 2-3

Re: [EXTERNAL] Cassandra 3.11.X upgrades

2020-02-12 Thread Jon Haddad
A while ago, on my first cluster, I decided to do an upgrade by adding nodes running 1.2 to an existing cluster running version 1.1. This was a bad decision, and at that point I decided to always play it safe and always stick to a single version, and never bootstrap in a node running different

Re: [RELEASE] Apache Cassandra 4.0-alpha3 released

2020-02-07 Thread Jon Haddad
Thanks for handling this, Mick! On Fri, Feb 7, 2020 at 12:02 PM Mick Semb Wever wrote: > > > The Cassandra team is pleased to announce the release of Apache Cassandra > version 4.0-alpha3. > > Apache Cassandra is a fully distributed database. It is the right choice > when you need scalability

Re: Overload because of hint pressure + MVs

2020-02-07 Thread Jon Haddad
There's a few things you can do here that might help. First off, if you're using the default heap settings, that's a serious problem. If you've got the head room, my recommendation is to use 16GB heap with 12 GB new gen and pin your memtable heap space to 2GB. Set your max tenuring threshold to

Re: Question on large partition key

2019-12-31 Thread Jon Haddad
I suggest checking out Aaron Morton's post on the 3.0 storage engine. https://thelastpickle.com/blog/2016/03/04/introductiont-to-the-apache-cassandra-3-storage-engine.html On Tue, Dec 31, 2019 at 11:20 AM Subroto Barua wrote: > I have a table --- > > create Table mytable ( > > Id text, > >

Re: Streaming Failed during bootstrap of a Replacement node

2019-12-20 Thread Jon Haddad
t; Yes we will upgrade it soon. But before we can upgrade shouldn’t we get > this lost node in the cluster to be replaced ? > > > > > > > > *From:* Jon Haddad > *Sent:* Friday, December 20, 2019 2:13 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Streaming

Re: Streaming Failed during bootstrap of a Replacement node

2019-12-20 Thread Jon Haddad
You should upgrade to Cassandra 3.11.5 before doing anything else. You're running a pretty old and buggy version. There's been hundreds (maybe thousands) of bugs fixed between 3.3 and 3.11.5. On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj wrote: > Hi, > > > > We are seeing the following error

Re: execute is faster than execute_async?

2019-12-11 Thread Jon Haddad
I'm not sure how you're measuring this - could you share your benchmarking code? I ask because execute calls execute_async under the hood: https://github.com/datastax/python-driver/blob/master/cassandra/cluster.py#L2316 I tested the python driver a ways back and found some weird behavior due to

Re: Connection Pooling in v4.x Java Driver

2019-12-10 Thread Jon Haddad
I'm not sure how closely the driver maintainers are following this list. You might want to ask on the Java Driver mailing list: https://groups.google.com/a/lists.datastax.com/forum/#!forum/java-driver-user On Tue, Dec 10, 2019 at 5:10 PM Caravaggio, Kevin < kevin.caravag...@lowes.com> wrote:

Re: AWS ephemeral instances + backup

2019-12-05 Thread Jon Haddad
You can easily do this with bcache or LVM http://rustyrazorblade.com/post/2018/2018-04-24-intro-to-lvm/. Medusa might be a good route to go down if you want to do backups instead: https://thelastpickle.com/blog/2019/11/05/cassandra-medusa-backup-tool-is-open-source.html On Thu, Dec 5, 2019 at

Re: Cassandra 4 alpha/alpha2

2019-11-01 Thread Jon Haddad
A new thing like this would be much better served by the community through several iterations. For instance, over the last year I've developed a tool for spinning up lab clusters, it's here: https://thelastpickle.com/tlp-cluster/ I had to make a *lot* of tradeoffs here. Everything Jeff

Re: Cassandra 4 alpha/alpha2

2019-10-31 Thread Jon Haddad
What artifact did you use and what OS are you on? On Thu, Oct 31, 2019 at 12:40 PM Abdul Patel wrote: > Hey Everyone > > Did anyone was successfull to install either alpha or alpha2 version for > cassandra 4.0? > Found 2 issues : > 1> cassandra-env.sh: > JAVA_VERSION varianle is not defined. >

Re: What is the status of counters? Should I use them?

2019-10-30 Thread Jon Haddad
‐ Original Message ‐‐‐ > On Wednesday, 30 October 2019 16:26, Jon Haddad wrote: > > Counters are good for things like page views, bad for money. Yes they can > under or overcount in certain situations. If your cluster is stable, > you'll see very little of it in practice. > > I've

Re: Where to get old RPMs?

2019-10-30 Thread Jon Haddad
Archives are here: http://archive.apache.org/dist/cassandra/ For example, the RPM for 3.11.x you can find here: http://archive.apache.org/dist/cassandra/redhat/311x/ The old releases are removed by Apache automatically as part of their policy, it's not specific to Cassandra. On Wed, Oct 30,

Re: What is the status of counters? Should I use them?

2019-10-30 Thread Jon Haddad
Counters are good for things like page views, bad for money. Yes they can under or overcount in certain situations. If your cluster is stable, you'll see very little of it in practice. I've done quite a bit of tuning of counters. Here's the main takeaways: * They do a read before a write, so

Re: TWCS and gc_grace_seconds

2019-10-26 Thread Jon Haddad
My coworker Radovan wrote up a post on the relationship between gc grace and hinted handoff: https://thelastpickle.com/blog/2018/03/21/hinted-handoff-gc-grace-demystified.html Jon On Sat, Oct 26, 2019 at 6:45 AM Hossein Ghiyasi Mehr wrote: > It needs to change gc_grace_seconds carefully

Re: Repair Issues

2019-10-24 Thread Jon Haddad
There's some major warning signs for me with your environment. 4GB heap is too low, and Cassandra 3.7 isn't something I would put into production. Your surface area for problems is massive right now. Things I'd do: 1. Never use incremental repair. Seems like you've already stopped doing them,

Re: merge two cluster

2019-10-23 Thread Jon Haddad
Probably not beneficial, I wouldn't do it. Not a fan of multi-tenancy with Cassandra unless the use cases are so small that your noisy neighbor problem is not very noisy at all. For those cases I don't know what you get from Cassandra other than a cool resume. On Wed, Oct 23, 2019 at 12:41 PM

Re: Cassandra Rack - Datacenter Load Balancing relations

2019-10-23 Thread Jon Haddad
TWO us-east-1b >- 6 write TWO us-east-1b > > > Here we have 2 DC read and write > One Rack per DC > One Availability Zone per DC > > Thanks, > > Sergio > > > On Wed, Oct 23, 2019, 1:11 PM Jon Haddad wrote: > >> Personally, I wouldn't eve

Re: Cassandra Rack - Datacenter Load Balancing relations

2019-10-23 Thread Jon Haddad
Personally, I wouldn't ever do this. I recommend separate DCs if you want to keep workloads separate. On Wed, Oct 23, 2019 at 4:06 PM Sergio wrote: > I forgot to comment for > >OPTION C) >1. Node DC RACK AZ 1 read ONE us-east-1a 2 read ONE us-east-1b >2. 3 read ONE

Re: Cassandra 2.1.18 - Question on stream/bootstrap throughput

2019-10-22 Thread Jon Haddad
CPU waiting on memory will look like CPU overhead. There's a good post on the topic by Brendan Gregg: http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html Regarding GC, I agree with Reid. You're probably not going to saturate your network card no matter what your settings,

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

2019-10-21 Thread Jon Haddad
is a good >> thing on AWS (or anything virtualized), you’d have to run your own tests >> and find out. >> >> >> >> R >> >> *From: *Jon Haddad >> *Reply-To: *"user@cassandra.apache.org" >> *Date: *Monday, October 21, 2019 at 12:0

Re: [EXTERNAL] Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

2019-10-21 Thread Jon Haddad
ng harnesses. It isn’t worth our time. As a previous > writer mentioned, there is usually better return on our time tuning the > schema (aka helping developers understand Cassandra’s strengths). > > > > We use 16 – 32 GB heaps, nothing smaller than that. > > > > Sean

Re: GC Tuning https://thelastpickle.com/blog/2018/04/11/gc-tuning.html

2019-10-21 Thread Jon Haddad
I still use ParNew + CMS over G1GC with Java 8. I haven't done a comparison with JDK 11 yet, so I'm not sure if it's any better. I've heard it is, but I like to verify first. The pause times with ParNew + CMS are generally lower than G1 when tuned right, but as Chris said it can be tricky. If

Re: Elevated response times from all nodes in a data center at the same time.

2019-10-16 Thread Jon Haddad
It's possible the queries you're normally running are served out of page cache, and during the latency spike you're hitting your disks. If you're using read ahead you might be hitting a throughput limit on the disks. I've got some numbers and graphs I can share later when I'm not on my phone.

Re: cluster rolling restart

2019-10-16 Thread Jon Haddad
I agree with Jeff here. Ideally you should be so comfortable with rolling restarts that they become second nature. Cassandra is designed to handle them and you should not be afraid to do them regularly. On Wed, Oct 16, 2019, 8:06 AM Jeff Jirsa wrote: > > Personally I encourage you to rolling

Re: Update/where statement Adds Row

2019-09-12 Thread Jon Haddad
Probably not a great idea unless you're using it sparingly. Using LWTs without knowing all the caveats is likely to lead to terrible cluster performance. On Wed, Sep 11, 2019, 10:59 PM A wrote: > Is it ok if I do this? > > ... where email = em AND company_id = id IF EXISTS > > > > > > Sent

Re: Is it possible to build multi cloud cluster for Cassandra

2019-09-05 Thread Jon Haddad
Technically, not a problem. Use GossipingPropertyFileSnitch to keep things simple and you can go across whatever cloud providers you want without issue. The biggest issue you're going to have isn't going to be Cassandra, it's having the expertise in the different cloud providers to understand

Re: New column

2019-08-22 Thread Jon Haddad
Just to close the loop on this, I did a release of tlp-stress last night, which now has this workload (AllowFiltering). You can grab a deb, rpm, tarball or docker image. Docs are here: http://thelastpickle.com/tlp-stress/ Jon On Mon, Aug 19, 2019 at 2:21 PM Jon Haddad wrote: > It'll be ab

Re: Disk space utilization by from some Cassandra

2019-08-21 Thread Jon Haddad
This advice hasn't been valid for a long time now for most use cases. The only time you need to reserve 50% disk space is if you're going to be running major compactions against a table in your cluster that occupies 50% of its total disk space. Nowadays, that's far less common than it was when

Re: New column

2019-08-19 Thread Jon Haddad
than 100 records and > pass partition key in where clause we wouldnt see issues using new column > and allow filtering? Can you please point me to any doc how allow > filtering works. I was in assumption of it goes through all the partitions > > > On Sun, Aug 18, 2019, 4:33 PM Jon Ha

Re: New column

2019-08-18 Thread Jon Haddad
If you're giving the partition key you won't scan the whole table. The overhead will depend on the size or the partition. Would be an interesting workload for our tlp-stress tool, I'll code something up for the next release. On Sun, Aug 18, 2019, 12:58 PM Rahul Reddy wrote: > Hello, > > We

Re: Datafile Corruption

2019-08-08 Thread Jon Haddad
Any chance you're using NVMe with an older Linux kernel? I've seen a *lot* filesystem errors from using older CentOS versions. You'll want to be using a version > 4.15. On Thu, Aug 8, 2019 at 9:31 AM Philip Ó Condúin wrote: > *@Jeff *- If it was hardware that would explain it all, but do you

Re: Cassandra read requests not getting timeout

2019-08-05 Thread Jon Haddad
I think this might be because the timeout only applied to each request, and the driver is paginating in the background. Each page is a new request. On Mon, Aug 5, 2019, 12:08 AM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Mon, Aug 5, 2019 at 8:50 AM nokia ceph > wrote: > >> Hi

Re: Cheat Sheet for Unix based OS, Performance troubleshooting

2019-07-28 Thread Jon Haddad
http://www.brendangregg.com/linuxperf.html On Sat, Jul 27, 2019 at 2:45 AM Paul Chandler wrote: > I have always found Amy's Cassandra 2.1 tuning guide great for the Linux > performance tuning: > https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html > > Sent from my iPhone > > On 26

Re: Performance impact with ALLOW FILTERING clause.

2019-07-25 Thread Jon Haddad
If you're thinking about rewriting your data to be more performant when doing analytics, you might as well go the distance and put it in an analytics friendly format like Parquet. My 2 cents. On Thu, Jul 25, 2019 at 11:01 AM ZAIDI, ASAD A wrote: > Thank you all for your insights. > > > > When

Re: Materialized View's additional PrimaryKey column

2019-07-25 Thread Jon Haddad
gt; create) on primaryKey columns because we suppose that cassandra developers > can manage this unpreferred operation better then us. I'm really confused > now. > > > > On Wednesday, July 24, 2019, 11:30:15 PM GMT+3, Jon Haddad < > j...@jonhaddad.com> wrote: > &

Re: Materialized View's additional PrimaryKey column

2019-07-24 Thread Jon Haddad
I really, really advise against using MVs. I've had to help a number of teams move off them. Not sure what list of bugs you read, but if the list didn't include "will destabilize your cluster to the point of constant downtime" then the list was incomplete. Jon On Wed, Jul 24, 2019 at 6:32 AM

Re: Compaction throughput

2019-07-19 Thread Jon Haddad
It's a limit on the total compaction throughput. On Fri, Jul 19, 2019 at 10:39 AM Vlad wrote: > Hi, > > is 'nodetool setcompactionthroughput' sets limit for all compactions on > the node, or is it per compaction thread? > > Thanks. >

Re: Running Node Repair After Changing RF or Replication Strategy for a Keyspace

2019-06-28 Thread Jon Haddad
Yep - not to mention the increased complexity and overhead of going from ONE to QUORUM, or the increased cost of QUORUM in RF=5 vs RF=3. If you're in a cloud provider, I've found you're almost always better off adding a new DC with a higher RF, assuming you're on NTS like Jeff mentioned. On Fri,

Re: Recover lost node from backup or evict/re-add?

2019-06-12 Thread Jon Haddad
100% agree with Sean. I would only use Cassandra backups in a case where you need to restore from full cluster loss. Example: An entire DC burns down, tornado, flooding. Your routine node replacement after a failure should be replace_address_first_boot. To ensure this goes smoothly, run

Re: Collecting Latency Metrics

2019-05-30 Thread Jon Haddad
Yep. I would *never* use mean when it comes to performance to make any sort of decisions. I prefer to graph all the p99 latencies as well as the max. Some good reading on the topic: https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/ On Thu, May 30, 2019 at 7:35 AM Chris

Re: Re: How to set up a cluster with allocate_tokens_for_keyspace?

2019-05-05 Thread Jon Haddad
; > Sent using Zoho Mail > > > > Forwarded message > From: Jon Haddad > To: > Date: Sat, 04 May 2019 22:10:39 +0430 > Subject: Re: How to set up a cluster with allocate_tokens_for_keyspace? > Forwarded message > > That line is only r

Re: Priority in IN () cqlsh comand

2019-05-05 Thread Jon Haddad
Do separate queries for each partition you want. There's no benefit in using the IN() clause here, and performance is significantly worse with multi-partition IN(), especially if the partitions are small. On Sun, May 5, 2019 at 4:52 AM Soheil Pourbafrani wrote: > > Hi, > > I want to run cqlsh

Re: How to set up a cluster with allocate_tokens_for_keyspace?

2019-05-04 Thread Jon Haddad
That line is only relevant for when you're starting your cluster and you need to define your initial tokens in a non-random way. Random token distribution doesn't work very well when you only use 4 tokens. Once you get the cluster set up you don't need to specify tokens anymore, you can just use

Re: Increasing the size limits implications

2019-04-30 Thread Jon Haddad
Just curious - why are you using such large batches? Most of the time when someone asks this question, it's because they're using batches as they would in an RDBMS, because larger transactions improve performance. That doesn't apply with Cassandra. Batches are OK at keeping multiple tables in

Re: [EXTERNAL] multiple Cassandra instances per server, possible?

2019-04-18 Thread Jon Haddad
Agreed with Jeff here. The whole "community recommends no more than 1TB" has been around, and inaccurate, for a long time. The biggest issue with dense nodes is how long it takes to replace them. 4.0 should help with that under certain circumstances. On Thu, Apr 18, 2019 at 6:57 AM Jeff Jirsa

Re: 2.1.9 --> 2.2.13 upgrade node startup after upgrade very slow

2019-04-17 Thread Jon Haddad
Let me be more specific - run the async java profiler and generate a flame graph to determine where CPU time is spent. On Wed, Apr 17, 2019 at 11:36 AM Jon Haddad wrote: > > Run the async java profiler on the node to determine what it's doing: > https://github.com/jvm-profiling-to

Re: 2.1.9 --> 2.2.13 upgrade node startup after upgrade very slow

2019-04-17 Thread Jon Haddad
Run the async java profiler on the node to determine what it's doing: https://github.com/jvm-profiling-tools/async-profiler On Wed, Apr 17, 2019 at 11:31 AM Carl Mueller wrote: > > No, we just did the package upgrade 2.1.9 --> 2.2.13 > > It definitely feels like some indexes are being

Re: Questions about C* performance related to tombstone

2019-04-09 Thread Jon Haddad
Normal deletes are fine. Sadly there's a lot of hand wringing about tombstones in the generic sense which leads people to try to work around *every* case where they're used. This is unnecessary. A tombstone over a single row isn't a problem, especially if you're only fetching that one row back.

Re: How to monitor datastax driver compression performance?

2019-04-09 Thread Jon Haddad
; Thanks for your help Jon. > > > El lun., 8 abr. 2019 a las 19:13, Jon Haddad () escribió: >> >> If it were me, I'd look at raw request rates (in terms of requests / >> second as well as request latency), network throughput and then some >> flame graphs of both the s

Re: How to monitor datastax driver compression performance?

2019-04-08 Thread Jon Haddad
If it were me, I'd look at raw request rates (in terms of requests / second as well as request latency), network throughput and then some flame graphs of both the server and your application: https://github.com/jvm-profiling-tools/async-profiler. I've created an issue in tlp-stress to add

Re: Assassinate fails

2019-04-04 Thread Jon Haddad
blem is often seen when logging in with the default cassandra user. > > Within cqlsh, there is code that forces the default cassandra user to > > connect by querying system_auth at QUORUM consistency. This can be > > problematic in larger clusters, and is another reason why you shou

Re: Assassinate fails

2019-04-04 Thread Jon Haddad
dra user. > Within cqlsh, there is code that forces the default cassandra user to connect > by querying system_auth at QUORUM consistency. This can be problematic in > larger clusters, and is another reason why you should never use the default > cassandra user. > > > >

Re: Assassinate fails

2019-04-04 Thread Jon Haddad
Ken, Alain is right about the system tables. What you're describing only works on non-local tables. Changing the CL doesn't help with keyspaces that use LocalStrategy. Here's the definition of the system keyspace: CREATE KEYSPACE system WITH replication = {'class': 'LocalStrategy'} AND

Re: Cassandra Possible read/write race condition in LOCAL_ONE?

2019-03-28 Thread Jon Haddad
I'm reading the OP as doing this from a single server, if that's the case QUORUM / LOCAL_QUORUM will work. On Thu, Mar 28, 2019 at 3:29 PM Jeff Jirsa wrote: > > Yes it can race; if you don't want to race, you'd want to use SERIAL or > LOCAL_SERIAL. > > On Thu, Mar 28, 2019 at 3:04 PM Richard

Re: Garbage Collector

2019-03-19 Thread Jon Haddad
G1 is optimized for high throughput with higher pause times. It's great if you have mixed / unpredictable workloads, and as Elliott mentioned is mostly set & forget. ZGC requires Java 11, which is only supported on trunk. I plan on messing with it soon, but I haven't had time yet. We'll share

Re: Fw: read request is slow

2019-03-18 Thread Jon Haddad
;>> Тема: Re: read request is slow >>> От: Dieudonné Madishon NGAYA >>> Кому: user@cassandra.apache.org >>> Копия: >>> >>> >>> >>> For your information,since cassandra 3.0, it includes ttop and other >>> options inside sjk >

Re: read request is slow

2019-03-16 Thread Jon Haddad
> >> >> >> And >> >> >> >> heap with 50% of that as a starting point? Hw do I do this? >> >> >> >> Thanks >> >> >> >> >> >> *From:* Dieudonné Madishon NGAYA [mailto:dmng...@gmail.com] >> *Sent:* Saturday, M

Re: read request is slow

2019-03-15 Thread Jon Haddad
1. What was the read request? Are you fetching a single row, a million, something else? 2. What are your GC settings? 3. What's the hardware in use? What resources have been allocated to each instance? 4. Did you see this issue after a single request or is the cluster under heavy load? If

JVM Tuning post

2018-04-11 Thread Jon Haddad
Hey folks, We (The Last Pickle) have helped a lot of teams with JVM tuning over the years, finally managed to write some stuff down. We’re hoping the community finds it helpful. http://thelastpickle.com/blog/2018/04/11/gc-tuning.html

Re: Text or....

2018-04-04 Thread Jon Haddad
Depending on the compression rate, I think it would generate less garbage on the Cassandra side if you compressed it client side. Something to test out. > On Apr 4, 2018, at 7:19 AM, Jeff Jirsa wrote: > > Compressing server side and validating checksums is hugely important

Backup & Restore w/ AWS Blog Post

2018-04-03 Thread Jon Haddad
Hey folks. We (The Last Pickle) have helped a number of clients set up backup & restore on AWS over the last couple of years. Alain has been working on a thorough blog post over the last several months to try to document pros, cons and techniques. Hopefully it proves to be helpful to the

Re: nodetool repair and compact

2018-04-01 Thread Jon Haddad
You’ll find the answers to your questions (and quite a bit more) in this blog post from my coworker: http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html Repair doesn’t clean up tombstones,

Re: Fast Writes to Cassandra Failing Through Python Script

2018-03-15 Thread Jon Haddad
TWCS does SizeTieredCompaction within the window, so it’s not likely to make a difference. I’m +1’ing what Jeff said, 128ms memtable_flush_period_in_ms is almost certainly your problem, unless you’ve changed other settings and haven’t told us about them. > On Mar 15, 2018, at 9:54 AM, Affan

Re: What versions should the documentation support now?

2018-03-12 Thread Jon Haddad
Docs for 3.0 go in the 3.0 branch. I’ve never heard of anyone shipping docs for multiple versions, I don’t know why we’d do that. You can get the docs for any version you need by downloading C*, the docs are included. I’m a firm -1 on changing that process. Jon > On Mar 12, 2018, at 9:19

Re: Adding disk to operating C*

2018-03-09 Thread Jon Haddad
I agree with Jeff - I usually advise teams to cap their density around 3TB, especially with TWCS. Read heavy workloads tend to use smaller datasets and ring size ends up being a function of performance tuning. Since 2.2 bootstrap can now be resumed, which helps quite a bit with the

Re: Filling in the blank To Do sections on the Apache Cassandra web site

2018-02-27 Thread Jon Haddad
<kenbrot...@yahoo.com.INVALID> > wrote: > > I was just getting ready to install sphinx. Cool. > > From: Jon Haddad [mailto:jonathan.had...@gmail.com] On Behalf Of Jon Haddad > Sent: Tuesday, February 27, 2018 9:51 AM > To: user@cassandra.apache.org > Subject:

Re: Filling in the blank To Do sections on the Apache Cassandra web site

2018-02-27 Thread Jon Haddad
The docs have been in tree for years :) https://github.com/apache/cassandra/tree/trunk/doc There’s even a docker image to build them so you don’t need to mess with sphinx. Check the README for instructions. Jon > On Feb 27, 2018, at 9:49

Re: How to Parse raw CQL text?

2018-02-26 Thread Jon Haddad
Yes ideally. I’ve been spending a bit of time in the parser the last week. There’s a lot of internals which are still using old terminology and are pretty damn confusing. I’m doing a little investigation into exposing some of the information while also modernizing it. > On Feb 26, 2018,

Re: Gathering / Curating / Organizing Cassandra Best Practices & Patterns

2018-02-24 Thread Jon Haddad
DataStax academy is great but no, no work needs to be or should be aligned with it. Datastax is an independent company trying to make a profit, they could yank their docs at any time. There’s a reason why we started doing the docs in-tree, there was too much of a reliance on DS documentation.

Re: Is it possible / makes it sense to limit concurrent streaming during bootstrapping new nodes?

2018-02-24 Thread Jon Haddad
We don’t have this documented *anywhere* right now, I’ve created a JIRA to update the site with the relevant info on this topic: https://issues.apache.org/jira/browse/CASSANDRA-14258 <https://issues.apache.org/jira/browse/CASSANDRA-14258> > On Feb 24, 2018, at 7:44 AM, Jon

Re: Is it possible / makes it sense to limit concurrent streaming during bootstrapping new nodes?

2018-02-24 Thread Jon Haddad
You can’t migrate down that way. The last several nodes you have up will get completely overwhelmed, and you’ll be completely screwed. Please do not give advice like this unless you’ve actually gone through the process or at least have an understanding of how the data will be shifted. Adding

Re: Initializing a multiple node cluster (multiple datacenters)

2018-02-23 Thread Jon Haddad
In my opinion and experience, this isn’t a real problem, since you define a list of seeds as the first few nodes you add to a cluster. When would you add a node to an existing cluster and mark itself as a seed? It’s neither practical or something you’d do by accident. > On Feb 23, 2018,

Re: Initializing a multiple node cluster (multiple datacenters)

2018-02-22 Thread Jon Haddad
e very > happy to get more people involved in the docs. > > > > On Thu, Feb 22, 2018 at 12:56 PM Kenneth Brotman > <kenbrot...@yahoo.com.invalid <mailto:kenbrot...@yahoo.com.invalid>> wrote: > > That information would have saved me time too. Thanks for making a JIRA for

Re: Initializing a multiple node cluster (multiple datacenters)

2018-02-22 Thread Jon Haddad
Great question. Unfortunately, our OSS docs lack a step by step process on how to add a DC, I’ve created a JIRA to do that: https://issues.apache.org/jira/browse/CASSANDRA-14254 The datastax docs are pretty good for this though:

Re: Cassandra Needs to Grow Up by Version Five!

2018-02-21 Thread Jon Haddad
Ken, Maybe it’s not clear how open source projects work, so let me try to explain. There’s a bunch of us who either get paid by someone or volunteer on our free time. The folks that get paid, (yay!) usually take direction on what the priorities are, and work on projects that directly affect

Re: Memtable flush -> SSTable: customizable or same for all compaction strategies?

2018-02-20 Thread Jon Haddad
The file format is independent from compaction. A compaction strategy only selects sstables to be compacted, that’s it’s only job. It could have side effects, like generating other files, but any decent compaction strategy will account for the fact that those other files don’t exist. I

  1   2   3   >