[
https://issues.apache.org/jira/browse/SOLR-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141172#comment-18141172
]
Jan Høydahl commented on SOLR-18634:
------------------------------------
Prop names should be named like Java packages, major categories first then leaf
at the end, so solr.cloud.delete…. does not stand on its own feet, next element
after cloud should be sub system, not a verb
> Detect replicas registered in ZK that have no core on disk at startup
> ---------------------------------------------------------------------
>
> Key: SOLR-18634
> URL: https://issues.apache.org/jira/browse/SOLR-18634
> Project: Solr
> Issue Type: Improvement
> Components: SolrCloud
> Reporter: Jan Høydahl
> Priority: Major
>
> h3. Problem
> When a SolrCloud node starts with an empty data directory (typically a lost
> or re-provisioned volume in Kubernetes), ZooKeeper still lists the replicas
> that lived on that node, but core discovery finds no {{core.properties}} for
> them.
> Today {{ZkController}} publishes all of the node's replicas as DOWN at
> startup, and then registers only the cores it finds on disk. Nothing compares
> the two lists. A replica that exists in ZK but not on disk stays DOWN
> forever, and nothing is logged about why. The cluster keeps serving from the
> other replicas, so the condition is easy to miss until someone looks at the
> cloud graph.
> The manual fix is a CoreAdmin CREATE with the existing {{coreNodeName}},
> which the ref guide already documents as the way to "assign a new core to
> replace an existing replica". The new core then does a full replication from
> the leader. Many users instead resort to DELETEREPLICA + ADDREPLICA because
> this path is not well known. See the operator-side discussion in
> https://github.com/apache/solr-operator/issues/787.
> h3. Proposal
> # *Always:* after core discovery, diff the ZK replicas for this node against
> the discovered cores and log a WARN per orphaned replica, naming collection,
> shard, {{coreNodeName}} and whether an active leader exists elsewhere in the
> shard.
> # *Opt-in:* a system property / env var, e.g.
> {{\-Dsolr.cloud.recreateMissingCores=true}} /
> {{SOLR_CLOUD_RECREATE_MISSING_CORES=true}}, default {{false}}. When enabled,
> each orphaned replica is recreated locally through the existing
> {{coreNodeName}} path and left to recover from the leader.
> The repair must be guarded: only recreate a core when the shard has an ACTIVE
> leader on another live node. If the lost replica was the shard's only copy,
> recreating it would make an empty core the leader and hide the data loss. In
> that case only the WARN is emitted, and the replica stays DOWN so that an
> operator can restore from backup. We could guard on free disk space too,
> but it is likely better to recreate the core and let it fail replication as
> it normally would on low disk.
> Out of scope: partial disk loss where {{core.properties}} survives but the
> index does not (see SOLR-18008), and any automatic DELETEREPLICA / ADDREPLICA.
> ----
> 🤖 Issue text authored with AI assistance, approved by me
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
