[
https://issues.apache.org/jira/browse/CASSANDRA-19666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872003#comment-17872003
]
Maxim Muzafarov commented on CASSANDRA-19666:
---------------------------------------------
Hello [~maedhroz] ,
I've changed how metrics are stored from a ConcurrentHashMap to several nested
maps grouped by metric type and scope. According to the benchmark, this has
performed well browsing the collection without any significant overhead.
h3. The patch is ready for the review:
[https://github.com/apache/cassandra/pull/3452/files]
Tests are here:
[https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch-5/45/]
Can you please review the changes?
–
The size of the collection containing the keyspaces metrics is largest during
the node run, so this is the most interesting case for us in terms of improving
performance. The following queries are the most interesting for us and work
well:
{code:java}
SELECT * FROM system_metrics.keyspace_group ;
SELECT * FROM system_metrics.keyspace_group WHERE name = ? ;
SELECT * FROM system_metrics.keyspace_group WHERE scope = ? ;
SELECT * FROM system_metrics.keyspace_group WHERE scope = ? AND type = ? ;
{code}
The patch includes the following high-level changes:
- pushed down the RowFilter to filter out data _before_ we build the
collection;
- List<Map.Entry<String, Object>> filter to filter out collection in advance;
- TreeMetricRegistry stores metrics grouped by group, type, and scope params;
h3. The benchmarks are the following:
Memory usage when creating 100 keyspaces in sequence.
!Memory_MetricsRegistry_SUMMARY.png|width=50%,height=50%!
Search by metrics name:
{{SELECT * FROM system_metrics.keyspace_group WHERE name = ?}}
!select_keyspace_vt_by_primary_key_19666_vs_trunk.png|width=50%,height=50%!
Search by metric scope:
{{SELECT * FROM system_metrics.keyspace_group WHERE scope = ?}}
!select_keyspace_vt_by_scope_19666_vs_trunk.png|width=50%,height=50%!
> Split the metrics collection into smaller groups
> ------------------------------------------------
>
> Key: CASSANDRA-19666
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19666
> Project: Cassandra
> Issue Type: Improvement
> Components: Observability/Metrics
> Reporter: Maxim Muzafarov
> Assignee: Maxim Muzafarov
> Priority: Urgent
> Fix For: 5.x
>
> Attachments: Memory_MetricsRegistry_SUMMARY.png,
> select_keyspace_vt_by_primary_key_19666_vs_trunk.png,
> select_keyspace_vt_by_scope_19666_vs_trunk.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Historically, all metrics are stored in a single collection indexed by a
> unique metric name - {{{}ConcurrentMap<String, Metric>{}}}. This storage
> technique doesn't offer any advantages over other options we might have
> below, and causes some performance and usability issues when we need to
> traverse this huge collection while filtering the metrics.
> The following possible solutions could be used:
> - Use
> https://www.javadoc.io/doc/io.dropwizard.metrics/metrics-core/4.0.2/com/codahale/metrics/SharedMetricRegistries.html
> , which provides a way to store metrics in their dedicated groups;
> - Implement a hierarchical storage structure, based on the knowledge of how
> the metric key is formed. For example,
> https://the-asf.slack.com/archives/CK23JSY2K/p1716568216701789?thread_ts=1716324924.466269&cid=CK23JSY2K
> Each of the above options assumes, that the metric groups are statically
> registered and known at the time a node starts. This may also be an option to
> consider for dynamically registering metric groups and in turn registering
> corresponding virtual tables and JmxMbean. In the latter case, we'll need to
> implement the notification mechanism for the drivers, as they are not
> currently notified about newly created virtual tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
