[ 
https://issues.apache.org/jira/browse/SOLR-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144096#comment-18144096
 ] 

Jan Høydahl commented on SOLR-18640:
------------------------------------

I suppose you didn't expect much review on a patch in JIRA, since it is so 
cumbersome, but I had Claude take it for a spin, and here is the LLM feedback, 
and I see it half-way agrees with my wish to publish these metrics from the 
overseer only. The rest of this comment is 🤖 answer:

h3. Instrument type

{{observableLongCounter}} exports this as a Prometheus counter 
({{solr_nocommit_replicas_total}}), but replica counts move both ways - 
deleting a replica, or an ACTIVE\->RECOVERING transition, decreases a series. 
PromQL reads every decrease as a counter reset, so {{rate()}}/{{increase()}} 
give spurious spikes, and for the OTLP push work in SOLR-18536 delta 
temporality would emit negative deltas on a monotonic sum. 
{{SolrMetricsContext.observableLongGauge}} is the right fit and is what we 
already use for the same shape of data - see {{solr_cores_loaded}} in 
{{SolrCores}}, which records a current count across several attribute 
permutations. That also settles the naming nocommit: as a gauge it can just be 
{{solr.replicas}}, with no {{_total}} suffix.

h3. "is this correct?" on the Overseer check - no, and it fails open

{{Overseer.closed}} defaults to {{false}}, and the only thing that sets it true 
on a non-leader is {{joinedElectionFired()}} -> {{overseer.close()}}. But 
{{ZkController.init()}} gates that on 
{{cc.nodeRoles.isOverseerAllowedOrPreferred()}}, so a node that never joins the 
election never gets closed and {{!isClosed()}} stays true forever. I confirmed 
this with a scratch test: a node started with {{overseer:disallowed}} reports 
itself as the active Overseer, and would therefore force-resolve every lazy 
CollectionRef on every scrape. Since {{Role.OVERSEER.modeWhenRoleIsAbsent()}} 
is {{disallowed}}, just starting with {{\-Dsolr.node.roles=data:on}} is enough 
to hit it, as is any {{coordinator:on}} node. The same predicate is used by the 
test's own Overseer-discovery helper, so it can silently pick the wrong node.

h3. NPE on the Overseer path

{{skipCollection}} ANDs the {{null == cr}} check with {{notActiveOverseer}}, so 
on the Overseer a null ref falls through to {{cr.get(true)}}. More likely in 
practice: {{dc}} is unchecked, and {{LazyCollectionRef.get(true)}} returns null 
for a collection deleted between {{getCollectionNames()}} and the resolve - 
{{fetchCollectionState}} returns null on NoNodeException, and 
{{getCollectionNames()}} is explicitly documented as "a close approximation". 
The SDK swallows the NPE, so the symptom is the whole instrument silently 
missing from that scrape.

h3. Scrape cost

{{LAZY_CACHE_TIME}} is {{STATE_UPDATE_DELAY}} = 2s, so at any normal scrape 
interval the Overseer re-fetches every unwatched collection from ZooKeeper on 
every scrape, serialized inside the callback. {{collectionStream()}} carries 
the warning for exactly this: "Use this sparingly; some users have thousands of 
collections!"

h3. Per-node vs Overseer

I like the staleness-detection idea in principle, but I don't think this shape 
delivers it. Because non-Overseer nodes only report collections in their local 
snapshot while the Overseer reports all, {{min by (collection, shard, type, 
state)}} across nodes measures which nodes happen to watch a collection, not 
whether they disagree - {{testLazy}} asserts precisely that asymmetry, and a 
min/max alert would fire permanently on any cluster where collections aren't 
hosted everywhere. Meanwhile the full {{collection x shard x type x state}} 
breakdown gets multiplied by node count, and since both roles emit the same 
metric name with no distinguishing label, the natural {{sum by (collection)}} 
inflates counts per reporting node (visible in your own paste: Overseer says 
PULL=2 and the other live node also says PULL=2).

I'd rather split the two semantics:
 * per-node, counting only replicas hosted on that node - cheap, no lazy 
resolution, and {{sum()}} across nodes is correct and gives the cluster-wide 
picture
 * Overseer-only under a separate name for the authoritative view, covering the 
case where every node hosting a collection is DOWN

Stale-view detection then deserves its own bounded metric (a local 
{{DocCollection.getZNodeVersion()}} or cluster-state-age gauge, one series per 
node) rather than 50x the cardinality as a side effect.

On the {{Replica.isActive(liveNodes)}} nocommit: I'd leave it out for now. It's 
a different question from the published state, and folding it into this 
metric's labels invites exactly the confusion you're worried about - better as 
its own metric if we want it.

> Solr 10.x replica status metrics
> --------------------------------
>
>                 Key: SOLR-18640
>                 URL: https://issues.apache.org/jira/browse/SOLR-18640
>             Project: Solr
>          Issue Type: Sub-task
>            Reporter: Chris M. Hostetter
>            Priority: Major
>         Attachments: SOLR-18640.patch
>
>
> As noted in SOLR-18147 , Solr 10's otel metrics are missing many replica 
> status related metrics from Solr 9 dur to the removal of the prometheus 
> exporter...
> {quote}One of the panels I really like from the old one were things around 
> state on replicas from Solr. Problem is I think that info only came from the 
> prometheus exporter because it parsed a cluster state API and not metrics 
> endpoint. I'd like reintroduce those metrics back properly so they are 
> actually exposed letting us create things like state timelines of Solr from 
> down to recovering or active. But I guess until then that will be missing.
> {quote}



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to