zstan commented on code in PR #13130: URL: https://github.com/apache/ignite/pull/13130#discussion_r3643287974
########## docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc: ########## @@ -47,3 +47,460 @@ queries with JOINs at massive scale and expect significant performance benefits. * Adjust link:data-rebalancing[data rebalancing settings] to ensure that rebalancing completes faster when your cluster topology changes. +== How to assess cluster health + +Cluster health is a complex set of metrics and states that need to be analyzed together. Apache Ignite is capable of demonstrating great performance across different scenarios with varying loads. Therefore, in general terms, a healthy cluster is one whose behavior aligns with your expectations. However, there are some universal aspects that apply to all deployments and warrant attention. + +It is important to understand that a healthy cluster may undergo planned topology changes or temporary load spikes. + +The key properties are: + +* The cluster is in the intended link:monitoring-metrics/cluster-states[state] and serves only the operations allowed by that state. +* link:clustering/baseline-topology[Baseline topology], when it is used or managed manually, matches the expected data-bearing server nodes. +* Data remains consistent: link:tools/control-script#verifying-partition-checksums[`idle_verify`] reports no partition conflicts when the cluster is idle. +* Expected nodes are present, with no unexpected or repeated node `JOIN`, `LEFT`, or `FAIL` events and no reported node segmentation. +* link:configuring-caches/partition-loss-policy[Lost partitions] are absent. +* The number of link:tools/control-script#transaction-management[long-running transactions] and their durations do not keep increasing; active link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchanges (PMEs)] complete; link:monitoring-metrics/new-metrics-system#monitoring-rebalancing[rebalancing] finishes; link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint-related metrics] and link:monitoring-metrics/new-metrics#thread-pools[executor queue sizes] return to their usual ranges after the link:monitoring-metrics/new-metrics-system#monitoring-topology[topology] has stabilized. + +There is no single command or metric that proves cluster health for every deployment. + +=== Check the Intended State and Node Membership + +Start with the link:monitoring-metrics/cluster-states[cluster state]. + +In the examples below, `control.(sh|bat)` is documentation shorthand. +Run `control.sh` on Unix-like systems or `control.bat` on Windows. + +Run the link:tools/control-script#getting-cluster-state[cluster-state command]: + +[source,shell] +---- +control.(sh|bat) --state +---- + +Relevant output: + +[source,text] +---- +Command [STATE] started +Arguments: --state +-------------------------------------------------------------------------------- +Cluster state: ACTIVE +Command [STATE] finished with code: 0 +---- + +`ACTIVE` is expected for normal read-write operation. +`ACTIVE_READ_ONLY` is normal when the cluster has been intentionally placed in read-only mode. +`INACTIVE` is acceptable only when it matches the current operation, for example, planned maintenance; an inactive cluster does not serve the data workload. +The criterion is whether the actual state matches the state that was intentionally set for the deployment. + +Check baseline topology when the cluster uses persistence, when baseline autoadjustment is disabled, or when you intentionally manage the set of data-bearing nodes. +In pure in-memory clusters with the default immediate autoadjustment, baseline topology normally follows the current server topology automatically. +If autoadjustment is disabled, the baseline changes only after an operator changes it. +If autoadjustment is configured with a non-zero timeout, the baseline is updated only after the topology remains unchanged for that timeout. +In both cases, run `control.(sh|bat) --baseline` and compare `Baseline nodes` and `Other nodes` with the expected set of server nodes. + +Run the link:tools/control-script#getting-nodes-registered-in-baseline-topology[baseline-topology command]: + +[source,shell] +---- +control.(sh|bat) --baseline +---- + +Relevant output for a cluster where all baseline nodes are online: + +[source,text] +---- +Cluster state: ACTIVE +Current topology version: 3 +Baseline auto adjustment disabled: softTimeout=300000 + +Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1) + +Baseline nodes: + ConsistentId=node-1, State=ONLINE, Order=1 + ConsistentId=node-2, State=ONLINE, Order=2 + ConsistentId=node-3, State=ONLINE, Order=3 +-------------------------------------------------------------------------------- +Number of baseline nodes: 3 + +Other nodes not found. +---- + +Example: one baseline node is offline: + +[source,text] +---- +Baseline nodes: + ConsistentId=node-1, State=ONLINE, Order=1 + ConsistentId=node-2, State=OFFLINE, Order=2 + ConsistentId=node-3, State=ONLINE, Order=3 +-------------------------------------------------------------------------------- +Number of baseline nodes: 3 +---- + +If a baseline node is `OFFLINE`, an expected data-bearing server is missing. +If other primary or backup copies are available, its absence does not cause partition loss. +To check for partition loss, query the partition states as described in <<confirm-that-rebalancing-converges,Confirm That Rebalancing Converges>>. + +If an online server node has joined the cluster but is not in the baseline, the command shows it under `Other nodes`: + +[source,text] +---- +Other nodes: + ConsistentId=node-4, Order=4 +Number of other nodes: 1 +---- + +The baseline contains server nodes that are intended to store data. +Client nodes are not part of the baseline. +An online server node in `Other nodes` is not always an error: the node may have been prepared intentionally but not yet introduced into the data topology. +If the node is expected to store data, first check the link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline auto-adjustment policy] and the current maintenance or scale-out procedure, then use the documented baseline change procedure. +Changing the baseline can start link:data-rebalancing[rebalancing]: partitions are redistributed according to the new affinity assignment. +Plan for the additional network, CPU, and storage load, especially in clusters with persistence. + +Use topology changes to distinguish planned activity from instability. +When a server node joins, leaves, or fails, cluster membership changes and triggers a link:data-modeling/data-partitioning#partition-map-exchange[Partition Map Exchange (PME)]. +A dynamic cache start or stop can also trigger PME even when no server node joins, leaves, or fails. +Therefore, a topology version or link:monitoring-metrics/new-metrics#partition-map-exchange[PME metric] change is useful only when interpreted together with maintenance actions and node logs. +Use these metrics to establish historical distributions; there is no universal duration threshold. + +[source,sql] +---- +SELECT NAME, VALUE +FROM SYS.METRICS +WHERE NAME IN ( + 'pme.Duration', + 'pme.CacheOperationsBlockedDuration' +) +ORDER BY NAME; +---- + +Run `control.(sh|bat) --baseline` repeatedly or monitor topology metrics to confirm that membership is stable when no maintenance is in progress. +In logs, look for repeated `JOIN`, `LEFT`, `FAIL`, or segmentation events and exchange-worker messages. +Investigate unexpected repeated `JOIN`, `LEFT`, or `FAIL` events, node segmentation, network failures, or a PME that does not finish. +For PME metrics and transaction checks, see <<check-transactions-and-sql-queries,Check Transactions and SQL Queries>>. + +=== Verify Partition Consistency + +When the cluster is expected to be idle, run: + +[source,shell] +---- +control.(sh|bat) --cache idle_verify +---- + +Successful result: + +[source,text] +---- +The check procedure has finished, no conflicts have been found. +---- + +The beginning of a conflict result uses this format: + +[source,text] +---- +The check procedure has failed, conflict partitions has been found: [counterConflicts=1, hashConflicts=0] +Update counter conflicts: +Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5] +---- + +The command compares partition update counters and partition hashes between primary and backup copies. +Run it only when data updates are stopped. Review Comment: ```suggestion Run it only on idle cluster i.e. without active data updates. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
