w3ll1ngt commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3627482754
##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,393 @@ queries with JOINs at massive scale and expect significant
performance benefits.
* Adjust link:data-rebalancing[data rebalancing settings] to ensure that
rebalancing completes faster when your cluster topology changes.
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating
great performance across different scenarios with varying loads. Therefore, in
general terms, a healthy cluster is one whose behavior aligns with your
expectations. However, there are some universal aspects that apply to all
deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned
topology changes or temporary load spikes.
+
+The key properties are:
+
+* The cluster is in the intended link:monitoring-metrics/cluster-states[state]
and serves only the operations allowed by that state.
+* Baseline topology, when it is used or managed manually, matches the expected
data-bearing server nodes.
+* Data remains consistent:
link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports
no partition conflicts when the cluster is idle.
+* Expected nodes are present, and no node segmentation or repeated membership
churn is reported.
+* link:configuring-caches/partition-loss-policy[Lost partitions] are absent.
+* Long-running transactions, Partition Map Exchange (PME), rebalancing,
checkpointing, and executor queues converge instead of accumulating.
+
+There is no single command or metric that proves cluster health for every
deployment.
+
+Use several signals together.
+A simple client connection or SQL liveness check can prove only that a
particular client or query path is reachable; it does not check user
partitions, backup consistency, baseline membership, or all server nodes.
+Similarly, `control.(sh|bat) --cache idle_verify` is an important consistency
check, but it is still not a complete health check.
+
+=== Check the Intended State and Node Membership
+
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally
enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for
example planned maintenance; an inactive cluster does not serve the data
workload.
+The criterion is whether the actual state matches the state that was
intentionally set for the deployment.
+
+Check baseline topology when the cluster uses persistence, when baseline
autoadjustment is disabled, or when you intentionally manage the set of
data-bearing nodes.
+In pure in-memory clusters with the default immediate autoadjustment, baseline
topology normally follows the current server topology automatically.
+If autoadjustment is disabled, the baseline changes only after an operator
changes it.
+If autoadjustment is configured with a non-zero timeout, the baseline is
updated only after the topology remains unchanged for that timeout.
+In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes`
and `Other nodes` with the expected set of server nodes.
+
+Run:
+
+[source,shell]
+----
+control.(sh|bat) --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+ ConsistentId=node-1, State=ONLINE, Order=1
+ ConsistentId=node-2, State=ONLINE, Order=2
+ ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+Example: one baseline node is offline:
+
+[source,text]
+----
+Baseline nodes:
+ ConsistentId=node-1, State=ONLINE, Order=1
+ ConsistentId=node-2, State=OFFLINE, Order=2
+ ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+----
+
+If a baseline node is `OFFLINE`, an expected data-bearing server is missing.
+If other primary or backup copies are available, its absence does not cause
partition loss.
+To check for partition loss, query the partition states as described in
<<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>.
+
+If an online server node has joined the cluster but is not in the baseline,
the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+ ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may
have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the
link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline
auto-adjustment policy] and the current maintenance or scale-out procedure,
then use the documented baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]: partitions
are redistributed according to the new affinity assignment.
+Plan for the additional network, CPU, and storage load, especially in clusters
with persistence.
+
+Use topology changes to distinguish planned activity from instability.
+A server `JOIN`, `LEFT`, or `FAIL` event changes cluster membership and
triggers PME.
+Cache or SQL schema changes can also trigger PME without server node loss.
+Therefore, a topology version or PME metric change is useful only when
interpreted together with maintenance actions and node logs.
+
+Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to
confirm that membership is stable when no maintenance is in progress.
+In logs, look for repeated node join, left, fail, segmentation, and
exchange-worker messages.
+Investigate unexpected repeated membership churn, node segmentation, network
failures, or a PME that does not finish.
+For PME metrics and transaction checks, see
<<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+control.(sh|bat) --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found:
[counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between
primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies
are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be
incomplete.
+A successful `idle_verify` result is an important confirmation of consistency,
but it still does not prove overall cluster health check success.
+
+[#confirm-that-rebalancing-converges]
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is
expected, but it should converge. Rebalance progress should move toward
completion.
+
+Use the
link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system
view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner;
this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate
immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or
`LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache
event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+Example: one partition is lost:
+
+[source,text]
+----
+CACHE_GROUP_ID | PARTITION_ID | NODE_ID | STATE |
IS_PRIMARY
+1544803905 | 5 | 0f4d6f30-3e04-4f68-b6a2-6b89f1795c0d | LOST |
true
+----
+
+If the query returns `LOST`, follow the
link:configuring-caches/partition-loss-policy[Partition Loss Policy] recovery
procedure.
+If a failed node returns, its persistent data may become available again, but
the affected partitions remain in the `LOST` state.
+If the required data is available again, reset the lost partitions:
+
+[source,shell]
+----
+control.(sh|bat) --cache reset_lost_partitions cacheName1,cacheName2,...
+----
+
+If data is still missing, resetting the partitions does not restore it; the
command only clears the `LOST` state.
Review Comment:
added on with some explanarory examples
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
