[
https://issues.apache.org/jira/browse/SOLR-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18142626#comment-18142626
]
Chris M. Hostetter commented on SOLR-18640:
-------------------------------------------
I've attached a patch that goes the route of each node emitting counts for
every DocCollection that is already resolved in it's local ClusterState –
except for the Overseer, which emits counts for every DocCollection to ensure
we can get counts for collections even if every node hosting that collection is
DOWN.
Labels differentiate collection, shard, replica type, and replica status.
here's an example of the current output from the overseer when there is a 1x1
gettingstarted collection on a node that is completely down, and a second
collection with 1 tlog replica, and 2 pull replicas...
{noformat}
# HELP solr_nocommit_replicas_total Number of replicas for each collection
(known to this node) by type and status
# TYPE solr_nocommit_replicas_total counter
solr_nocommit_replicas_total{category="CONTAINER",collection="gettingstarted",otel_scope_name="org.apache.solr",shard="shard1",state="DOWN",type="NRT"}
1.0
solr_nocommit_replicas_total{category="CONTAINER",collection="tlog_and_pull_col",otel_scope_name="org.apache.solr",shard="shard1",state="ACTIVE",type="PULL"}
2.0
solr_nocommit_replicas_total{category="CONTAINER",collection="tlog_and_pull_col",otel_scope_name="org.apache.solr",shard="shard1",state="ACTIVE",type="TLOG"}
1.0
{noformat}
And here's the output from a differnet (live) not in the cluster that only
hosts one of those PULL replicas (but is aware of the entire collection)...
{noformat}
# HELP solr_nocommit_replicas_total Number of replicas for each collection
(known to this node) by type and status
# TYPE solr_nocommit_replicas_total counter
solr_nocommit_replicas_total{category="CONTAINER",collection="tlog_and_pull_col",otel_scope_name="org.apache.solr",shard="shard1",state="ACTIVE",type="PULL"}
2.0
solr_nocommit_replicas_total{category="CONTAINER",collection="tlog_and_pull_col",otel_scope_name="org.apache.solr",shard="shard1",state="ACTIVE",type="TLOG"}
1.0
{noformat}
One thing I'm on the fence about is adding another permutation to the Labels
that indicates the number of replicas (of each collection/type/shard) that are
"alive" – from the point of view of the reporting node – ie:
{{Replica.isActive(Set<String> liveNodes)}} ... since that may also be useful
info in the event of network partitioning ... but i worry that might get
confusing compared to the "status=ACTIVE" (as published in the clusterstate)
> Solr 10.x replica status metrics
> --------------------------------
>
> Key: SOLR-18640
> URL: https://issues.apache.org/jira/browse/SOLR-18640
> Project: Solr
> Issue Type: Sub-task
> Reporter: Chris M. Hostetter
> Priority: Major
> Attachments: SOLR-18640.patch
>
>
> As noted in SOLR-18147 , Solr 10's otel metrics are missing many replica
> status related metrics from Solr 9 dur to the removal of the prometheus
> exporter...
> {quote}One of the panels I really like from the old one were things around
> state on replicas from Solr. Problem is I think that info only came from the
> prometheus exporter because it parsed a cluster state API and not metrics
> endpoint. I'd like reintroduce those metrics back properly so they are
> actually exposed letting us create things like state timelines of Solr from
> down to recovering or active. But I guess until then that will be missing.
> {quote}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
