[
https://issues.apache.org/jira/browse/SOLR-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18144096#comment-18144096
]
Jan Høydahl commented on SOLR-18640:
------------------------------------
I suppose you didn't expect much review on a patch in JIRA, since it is so
cumbersome, but I had Claude take it for a spin, and here is the LLM feedback,
and I see it half-way agrees with my wish to publish these metrics from the
overseer only. The rest of this comment is 🤖 answer:
h3. Instrument type
{{observableLongCounter}} exports this as a Prometheus counter
({{solr_nocommit_replicas_total}}), but replica counts move both ways -
deleting a replica, or an ACTIVE\->RECOVERING transition, decreases a series.
PromQL reads every decrease as a counter reset, so {{rate()}}/{{increase()}}
give spurious spikes, and for the OTLP push work in SOLR-18536 delta
temporality would emit negative deltas on a monotonic sum.
{{SolrMetricsContext.observableLongGauge}} is the right fit and is what we
already use for the same shape of data - see {{solr_cores_loaded}} in
{{SolrCores}}, which records a current count across several attribute
permutations. That also settles the naming nocommit: as a gauge it can just be
{{solr.replicas}}, with no {{_total}} suffix.
h3. "is this correct?" on the Overseer check - no, and it fails open
{{Overseer.closed}} defaults to {{false}}, and the only thing that sets it true
on a non-leader is {{joinedElectionFired()}} -> {{overseer.close()}}. But
{{ZkController.init()}} gates that on
{{cc.nodeRoles.isOverseerAllowedOrPreferred()}}, so a node that never joins the
election never gets closed and {{!isClosed()}} stays true forever. I confirmed
this with a scratch test: a node started with {{overseer:disallowed}} reports
itself as the active Overseer, and would therefore force-resolve every lazy
CollectionRef on every scrape. Since {{Role.OVERSEER.modeWhenRoleIsAbsent()}}
is {{disallowed}}, just starting with {{\-Dsolr.node.roles=data:on}} is enough
to hit it, as is any {{coordinator:on}} node. The same predicate is used by the
test's own Overseer-discovery helper, so it can silently pick the wrong node.
h3. NPE on the Overseer path
{{skipCollection}} ANDs the {{null == cr}} check with {{notActiveOverseer}}, so
on the Overseer a null ref falls through to {{cr.get(true)}}. More likely in
practice: {{dc}} is unchecked, and {{LazyCollectionRef.get(true)}} returns null
for a collection deleted between {{getCollectionNames()}} and the resolve -
{{fetchCollectionState}} returns null on NoNodeException, and
{{getCollectionNames()}} is explicitly documented as "a close approximation".
The SDK swallows the NPE, so the symptom is the whole instrument silently
missing from that scrape.
h3. Scrape cost
{{LAZY_CACHE_TIME}} is {{STATE_UPDATE_DELAY}} = 2s, so at any normal scrape
interval the Overseer re-fetches every unwatched collection from ZooKeeper on
every scrape, serialized inside the callback. {{collectionStream()}} carries
the warning for exactly this: "Use this sparingly; some users have thousands of
collections!"
h3. Per-node vs Overseer
I like the staleness-detection idea in principle, but I don't think this shape
delivers it. Because non-Overseer nodes only report collections in their local
snapshot while the Overseer reports all, {{min by (collection, shard, type,
state)}} across nodes measures which nodes happen to watch a collection, not
whether they disagree - {{testLazy}} asserts precisely that asymmetry, and a
min/max alert would fire permanently on any cluster where collections aren't
hosted everywhere. Meanwhile the full {{collection x shard x type x state}}
breakdown gets multiplied by node count, and since both roles emit the same
metric name with no distinguishing label, the natural {{sum by (collection)}}
inflates counts per reporting node (visible in your own paste: Overseer says
PULL=2 and the other live node also says PULL=2).
I'd rather split the two semantics:
* per-node, counting only replicas hosted on that node - cheap, no lazy
resolution, and {{sum()}} across nodes is correct and gives the cluster-wide
picture
* Overseer-only under a separate name for the authoritative view, covering the
case where every node hosting a collection is DOWN
Stale-view detection then deserves its own bounded metric (a local
{{DocCollection.getZNodeVersion()}} or cluster-state-age gauge, one series per
node) rather than 50x the cardinality as a side effect.
On the {{Replica.isActive(liveNodes)}} nocommit: I'd leave it out for now. It's
a different question from the published state, and folding it into this
metric's labels invites exactly the confusion you're worried about - better as
its own metric if we want it.
> Solr 10.x replica status metrics
> --------------------------------
>
> Key: SOLR-18640
> URL: https://issues.apache.org/jira/browse/SOLR-18640
> Project: Solr
> Issue Type: Sub-task
> Reporter: Chris M. Hostetter
> Priority: Major
> Attachments: SOLR-18640.patch
>
>
> As noted in SOLR-18147 , Solr 10's otel metrics are missing many replica
> status related metrics from Solr 9 dur to the removal of the prometheus
> exporter...
> {quote}One of the panels I really like from the old one were things around
> state on replicas from Solr. Problem is I think that info only came from the
> prometheus exporter because it parsed a cluster state API and not metrics
> endpoint. I'd like reintroduce those metrics back properly so they are
> actually exposed letting us create things like state timelines of Solr from
> down to recovering or active. But I guess until then that will be missing.
> {quote}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]