[
https://issues.apache.org/jira/browse/CASSANDRA-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17929088#comment-17929088
]
Maxim Muzafarov commented on CASSANDRA-20333:
---------------------------------------------
thanks for sharing these thoughts, Benedict. The idea of the CASSANDRA-19365 is
to provide a consistent snapshot of the buckets during reads, as before a user
could see a partially rescaled state while the rescaling was in progress. The
rebase part, however, remains unchanged. We don't block reads or writes while
keeping the state consistent, by losing a few more recent updates - the
solution is suboptimal.
With the {_}ThreadLocal approach{_}, and from a read and rescale point of view,
we could run into the same problems we have with the current striped approach.
So we either have to block on reads or lose some recent updates. Could you or
Dmitry elaborate on it a bit more? There is also a memory footprint concern
that I have in mind, but I think having a poc with the real numbers should help
us decide on that.
I like the idea with the _backward decay_ approach you shared, I could try to
implement that and see how it helps with all the guarantees we put it on it.
> Reduce DecayingEstimatedHistogramReservoir update cost
> ------------------------------------------------------
>
> Key: CASSANDRA-20333
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20333
> Project: Apache Cassandra
> Issue Type: Improvement
> Components: Observability/Metrics
> Reporter: Dmitry Konstantinov
> Assignee: Dmitry Konstantinov
> Priority: Normal
>
> Based on the discussions in CASSANDRA-20250
> [~benedict]:
> {quote}We can probably improve our reservoir performance if we want to,
> perhaps in a follow-up patch? For instance, we could have a small
> thread-local buffer of (time, latency) pairs that we periodically flush
> together, so that we amortise the memory latency costs. Or we could explore
> maintaining a per-thread HdrHistogram, that we periodically flush. This would
> be a good time to explore fully migrating to HdrHistogram, as it has built-in
> merge semantics iirc. I am not sure what the decayed version would look like
> there, but I am certain we could maintain a separate decayed HdrHistogram.
> Having a thread-local buffer of updates we intend to flush to the histograms
> would amortise the latency penalties without fundamentally redesigning
> anything (as well as reducing contention).
> Other possibilities might include e.g. changing the bucket distribution so we
> don't need a LUT for computing lg2, although the above would gracefully
> handle any contribution this has as well.
> {quote}
>
> Other ideas about squeezing extra bits from the current design:
> * bucket id can be calculated once (currently we do it 2 times for decaying
> and current buckets), like:
> {code:java}
> int stripe = (int) (Thread.currentThread().getId() & (nStripes - 1));
> int bucket = stripedIndex(index, stripe);
> rescaledDecayingBuckets.update(bucket, now);
> updateBucket(buckets, bucket, 1); {code}
> * for histograms on highly loaded paths we can use another number of stripes
> (by default it is 2, we can set for example 4 for them)
> * I noticed some variation in performance for a micro-benchmark (existing
> one: DecayingEstimatedHistogramBench) depending on what exact value for
> distributionPrime is used (but I need to double check it)
> * forwardDecayWeight function depends on SampledClock value, so we can try
> to recalculate the weight only when time is changed
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
