Re: Expensive metrics?

2018-02-28 Thread Chris Lohfink
> our histograms when built with a Meter use a ExponentiallyDecayingReservoir > but our histograms built directly use DecayingEstimatedHistogramReservoir > algorithm Meters dont use a decaying reservoir, they use EMWA

Re: Expensive metrics?

2018-02-28 Thread Michael Burman
Hi, I wrote CASSANDRA-14281 for the initial idea and where I ended up with my current prototype. This maintains the current layout of the JMX metrics so it shouldn't be visible to users. Should, because I couldn't really find any definition of our metrics. For example, our histograms when

Re: Expensive metrics?

2018-02-27 Thread Nate McCall
Hi Micke, There is some good research in here - have you had a chance to create some issues in Jira from this? On Fri, Feb 23, 2018 at 6:28 AM, Michael Burman wrote: > Hi, > > I was referring to this article by Shipilev (there are few small issues > forgotten in that url you

Re: Expensive metrics?

2018-02-22 Thread Michael Burman
Hi, I was referring to this article by Shipilev (there are few small issues forgotten in that url you pasted): https://shipilev.net/blog/2014/nanotrusting-nanotime/ And his lovely recommendation on it: "System.nanoTime is as bad as String.intern now: you can use it, but use it wisely. ".

Re: Expensive metrics?

2018-02-22 Thread Michael Burman
Hi, I've looked at the high level the metrics' expense. It's around ~4% of the total CPU time in my machine. But the problem with that higher level measurement is that it does not show waits. When I push writes to the Cassandra (through CQL) I'm mostly getting stalls according to the kernel

Re: Expensive metrics?

2018-02-22 Thread Jonathan Haddad
Hey Micke, very cool you're looking to improve C*'s performance, we would absolutely benefit from it. Have you done any other benchmarks beside the micro one to determine the total effect of these metrics on the system overall? Microbenchmarks are a great way to tune small sections of code but

Re: Expensive metrics?

2018-02-22 Thread Jeremiah D Jordan
re: nanoTime vs currentTimeMillis there is a good blog post here about the timing of both and how your choice of Linux clock source can drastically effect the speed of the calls, and also showing that in general on linux there is no perf improvement for one over the other.

Re: Expensive metrics?

2018-02-22 Thread Blake Eggleston
Hi Micke, This is really cool, thanks for taking the time to investigate this. I believe the metrics around memtable insert time come in handy in identifying high partition contention in the memtable. I know I've been involved in a situation over the past year where we got actionable info from

Expensive metrics?

2018-02-22 Thread Michael Burman
Hi, I wanted to get some input from the mailing list before making a JIRA and potential fixes. I'll touch the performance more on latter part, but there's one important question regarding the write latency metric recording place. Currently we measure the writeLatency (and metric write