Copilot commented on code in PR #13130:
URL: https://github.com/apache/ignite/pull/13130#discussion_r3586795227
##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant
performance benefits.
* Adjust link:data-rebalancing[data rebalancing settings] to ensure that
rebalancing completes faster when your cluster topology changes.
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating
great performance across different scenarios with varying loads. Therefore, in
general terms, a healthy cluster is one whose behavior aligns with your
expectations. However, there are some universal aspects that apply to all
deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned
topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact
and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of
accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is
reachable, but it does not check user partitions, backup consistency, baseline
membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check,
but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally
enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for
example planned maintenance; an inactive cluster does not serve the data
workload.
+The criterion is whether the actual state matches the state that was
intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+ ConsistentId=node-1, State=ONLINE, Order=1
+ ConsistentId=node-2, State=ONLINE, Order=2
+ ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline,
the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+ ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may
have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the
link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline
auto-adjustment policy] and the deployment plan, then use the documented
baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is
often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node
instability, network issues, or misconfigured discovery. However, a temporary
topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the
major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other
affinity-changing events can change the minor affinity topology version without
a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network
failures, or a partition map exchange that does not finish.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+./control.sh --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found:
[counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between
primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies
are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be
incomplete.
+A successful `idle_verify` result is an important confirmation of consistency,
but it still does not prove overall cluster health check success.
+
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is
expected, but it should converge. Rebalance progress should move toward
completion.
+
+Use the
link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system
view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner;
this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate
immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or
`LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache
event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks
from a shared queue. These queues may grow for a short time under load, but
they should not grow continuously. Sustained queue growth means that a node is
not keeping up with the workload or that message processing is impaired. The
same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between
independent stripes: tasks in the same stripe run sequentially, while different
stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when
overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+ OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As it was said, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated
`DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean
object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name
`communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`;
the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group
`threadPools`, bean name `StripedExecutor`; the attributes include
`TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see
link:monitoring-metrics/new-metrics-system#jmx[JMX] and
link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole
MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+=== Check transactions and SQL queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase
after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+./control.sh --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a
universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+ 'tx.OwnerTransactionsNumber',
+ 'tx.TransactionsHoldingLockNumber',
+ 'tx.LockedKeysNumber',
+ 'pme.Duration',
+ 'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and
failure to return to the usual range after the workload drops.
+For view definitions and metrics, see
link:monitoring-metrics/system-views#transactions[TRANSACTIONS],
link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES],
link:monitoring-metrics/new-metrics#transactions[transaction metrics], and
link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map
Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after
topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations
that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is
described in
link:key-value-api/transactions#long-running-transactions-termination[Long
Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a
PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where
appropriate; see
link:key-value-api/transactions#handling-failed-transactions[Handling Failed
Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a
justified safety margin, and test application behavior when rollback happens.
+
+=== Check checkpoint pressure when persistence is enabled
Review Comment:
The section heading capitalization is inconsistent with other headings in
this document. Consider using title case to match the surrounding style.
##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant
performance benefits.
* Adjust link:data-rebalancing[data rebalancing settings] to ensure that
rebalancing completes faster when your cluster topology changes.
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating
great performance across different scenarios with varying loads. Therefore, in
general terms, a healthy cluster is one whose behavior aligns with your
expectations. However, there are some universal aspects that apply to all
deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned
topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact
and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of
accumulating.
Review Comment:
The document uses `*` for unordered lists earlier in this file, but this new
list uses `-`, which is inconsistent and can render differently depending on
AsciiDoc settings. Consider switching these items to `*` to match the
surrounding style.
##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant
performance benefits.
* Adjust link:data-rebalancing[data rebalancing settings] to ensure that
rebalancing completes faster when your cluster topology changes.
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating
great performance across different scenarios with varying loads. Therefore, in
general terms, a healthy cluster is one whose behavior aligns with your
expectations. However, there are some universal aspects that apply to all
deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned
topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact
and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of
accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is
reachable, but it does not check user partitions, backup consistency, baseline
membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check,
but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally
enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for
example planned maintenance; an inactive cluster does not serve the data
workload.
+The criterion is whether the actual state matches the state that was
intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+ ConsistentId=node-1, State=ONLINE, Order=1
+ ConsistentId=node-2, State=ONLINE, Order=2
+ ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline,
the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+ ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may
have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the
link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline
auto-adjustment policy] and the deployment plan, then use the documented
baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is
often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node
instability, network issues, or misconfigured discovery. However, a temporary
topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the
major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other
affinity-changing events can change the minor affinity topology version without
a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network
failures, or a partition map exchange that does not finish.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+./control.sh --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found:
[counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between
primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies
are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be
incomplete.
+A successful `idle_verify` result is an important confirmation of consistency,
but it still does not prove overall cluster health check success.
+
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is
expected, but it should converge. Rebalance progress should move toward
completion.
+
+Use the
link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system
view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner;
this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate
immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or
`LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache
event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks
from a shared queue. These queues may grow for a short time under load, but
they should not grow continuously. Sustained queue growth means that a node is
not keeping up with the workload or that message processing is impaired. The
same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between
independent stripes: tasks in the same stripe run sequentially, while different
stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when
overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+ OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As it was said, short non-zero queues are acceptable under load.
Review Comment:
"As it was said" is unidiomatic English in this context and reads like a
mistranslation. Consider rephrasing to a more standard form (for example, "As
mentioned above").
##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant
performance benefits.
* Adjust link:data-rebalancing[data rebalancing settings] to ensure that
rebalancing completes faster when your cluster topology changes.
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating
great performance across different scenarios with varying loads. Therefore, in
general terms, a healthy cluster is one whose behavior aligns with your
expectations. However, there are some universal aspects that apply to all
deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned
topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact
and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of
accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is
reachable, but it does not check user partitions, backup consistency, baseline
membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check,
but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally
enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for
example planned maintenance; an inactive cluster does not serve the data
workload.
+The criterion is whether the actual state matches the state that was
intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+ ConsistentId=node-1, State=ONLINE, Order=1
+ ConsistentId=node-2, State=ONLINE, Order=2
+ ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline,
the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+ ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may
have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the
link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline
auto-adjustment policy] and the deployment plan, then use the documented
baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is
often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node
instability, network issues, or misconfigured discovery. However, a temporary
topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the
major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other
affinity-changing events can change the minor affinity topology version without
a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network
failures, or a partition map exchange that does not finish.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+./control.sh --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found:
[counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between
primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies
are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be
incomplete.
+A successful `idle_verify` result is an important confirmation of consistency,
but it still does not prove overall cluster health check success.
+
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is
expected, but it should converge. Rebalance progress should move toward
completion.
+
+Use the
link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system
view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner;
this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate
immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or
`LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache
event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks
from a shared queue. These queues may grow for a short time under load, but
they should not grow continuously. Sustained queue growth means that a node is
not keeping up with the workload or that message processing is impaired. The
same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between
independent stripes: tasks in the same stripe run sequentially, while different
stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when
overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+ OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As it was said, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated
`DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean
object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name
`communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`;
the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group
`threadPools`, bean name `StripedExecutor`; the attributes include
`TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see
link:monitoring-metrics/new-metrics-system#jmx[JMX] and
link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole
MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+=== Check transactions and SQL queries
Review Comment:
The section heading capitalization is inconsistent with the other `=== Check
...` headings in this document (most use title case). Consider capitalizing
this heading for consistency.
##########
docs/_docs/perf-and-troubleshooting/general-perf-tips.adoc:
##########
@@ -47,3 +47,345 @@ queries with JOINs at massive scale and expect significant
performance benefits.
* Adjust link:data-rebalancing[data rebalancing settings] to ensure that
rebalancing completes faster when your cluster topology changes.
+== How to assess cluster health
+
+Cluster health is a complex thing. Apache Ignite is capable of demonstrating
great performance across different scenarios with varying loads. Therefore, in
general terms, a healthy cluster is one whose behavior aligns with your
expectations. However, there are some universal aspects that apply to all
deployments and warrant attention.
+
+It is important to understand that a healthy cluster may undergo planned
topology changes or temporary load spikes.
+
+The key properties are:
+- The cluster serves operations allowed by its intended state.
+- Data remains consistent (`idle_verify` passes, meaning partitions are intact
and there are no conflicts).
+- Expected nodes are present.
+- No pending transactions and no split brain.
+- No lost partitions.
+- Transient activity such as rebalancing or queue growth converges instead of
accumulating.
+
+But the thing is, there is no universal single health metric.
+
+Use several signals together.
+For example, `SELECT 1` can confirm that a client or SQL query path is
reachable, but it does not check user partitions, backup consistency, baseline
membership, or all server nodes.
+Similarly, `control.sh --cache idle_verify` is an important consistency check,
but it is not a complete health check.
+
+=== Check the Intended State and Node Membership
+Start with the link:monitoring-metrics/cluster-states[cluster state].
+
+Run:
+
+[source,shell]
+----
+./control.sh --state
+----
+
+Relevant output:
+
+[source,text]
+----
+Command [STATE] started
+Arguments: --state
+--------------------------------------------------------------------------------
+Cluster state: ACTIVE
+Command [STATE] finished with code: 0
+----
+
+`ACTIVE` is expected for normal read-write operation.
+`ACTIVE_READ_ONLY` is normal when read-only operation was intentionally
enabled.
+`INACTIVE` is acceptable only when it matches the current operation, for
example planned maintenance; an inactive cluster does not serve the data
workload.
+The criterion is whether the actual state matches the state that was
intentionally set for the deployment.
+
+If baseline topology is enabled, check the baseline as well:
+
+Run:
+
+[source,shell]
+----
+./control.sh --baseline
+----
+
+Relevant output for a cluster where all baseline nodes are online:
+
+[source,text]
+----
+Cluster state: ACTIVE
+Current topology version: 3
+Baseline auto adjustment disabled: softTimeout=300000
+
+Current topology version: 3 (Coordinator: ConsistentId=node-1, Order=1)
+
+Baseline nodes:
+ ConsistentId=node-1, State=ONLINE, Order=1
+ ConsistentId=node-2, State=ONLINE, Order=2
+ ConsistentId=node-3, State=ONLINE, Order=3
+--------------------------------------------------------------------------------
+Number of baseline nodes: 3
+
+Other nodes not found.
+----
+
+If an online server node has joined the cluster but is not in the baseline,
the command shows it under `Other nodes`:
+
+[source,text]
+----
+Other nodes:
+ ConsistentId=node-4, Order=4
+Number of other nodes: 1
+----
+
+The baseline contains server nodes that are intended to store data.
+Client nodes are not part of the baseline.
+An online server node in `Other nodes` is not always an error: the node may
have been prepared intentionally but not yet introduced into the data topology.
+If the node is expected to store data, first check the
link:clustering/baseline-topology#baseline-topology-autoadjustment[baseline
auto-adjustment policy] and the deployment plan, then use the documented
baseline change procedure.
+Changing the baseline can start link:data-rebalancing[rebalancing]. This is
often not an easy procedure for a cluster with data in persistence storage.
+
+Frequent unexpected topology changes are not normal and usually indicate node
instability, network issues, or misconfigured discovery. However, a temporary
topology change does not make the cluster unhealthy by itself.
+Server `JOIN`, `LEFT`, and `FAIL` events change cluster membership and the
major topology version.
+Cache start or stop, distributed cache creation, SQL table creation, and other
affinity-changing events can change the minor affinity topology version without
a node loss.
+Planned events are normal.
+Investigate unexpected repeated membership churn, node segmentation, network
failures, or a partition map exchange that does not finish.
+
+=== Verify Partition Consistency
+
+When the cluster is expected to be idle, run:
+
+[source,shell]
+----
+./control.sh --cache idle_verify
+----
+
+Successful result:
+
+[source,text]
+----
+The check procedure has finished, no conflicts have been found.
+----
+
+The beginning of a conflict result uses this format:
+
+[source,text]
+----
+The check procedure has failed, conflict partitions has been found:
[counterConflicts=1, hashConflicts=0]
+Update counter conflicts:
+Conflict partition: PartitionKey [grpId=1544803905, grpName=default, partId=5]
+----
+
+The command compares partition update counters and partition hashes between
primary and backup copies.
+Run it only when data updates are stopped.
+If updates are active, the command can report false conflicts because copies
are changing while hashes are being calculated.
+Partitions in `MOVING` or `LOST` state may be skipped, so the result can be
incomplete.
+A successful `idle_verify` result is an important confirmation of consistency,
but it still does not prove overall cluster health check success.
+
+=== Confirm That Rebalancing Converges
+
+After a topology event, such as a new node join, transient rebalancing is
expected, but it should converge. Rebalance progress should move toward
completion.
+
+Use the
link:monitoring-metrics/system-views#partition_states[PARTITION_STATES] system
view to check partition states:
+
+* `OWNING`: the node is the current primary or backup owner.
+* `MOVING`: a partition copy is being loaded on the node during rebalance.
+* `RENTING`: an old copy is being removed after ownership changes.
+* `EVICTED`: the partition is absent on a node that is no longer an owner;
this is not an error by itself.
+* `LOST`: the partition is unavailable and must not be used; investigate
immediately.
+
+[source,sql]
+----
+SELECT CACHE_GROUP_ID, PARTITION_ID, NODE_ID, STATE, IS_PRIMARY
+FROM SYS.PARTITION_STATES
+WHERE STATE IN ('MOVING', 'RENTING', 'LOST')
+ORDER BY STATE, CACHE_GROUP_ID, PARTITION_ID, NODE_ID;
+----
+
+In steady state, this query usually should not return `MOVING`, `RENTING`, or
`LOST` rows.
+`MOVING` and `RENTING` are expected right after an intended topology or cache
event, but their count should decrease.
+`LOST` is not a normal transient state.
+
+=== Check Execution Queues
+
+Ignite has several internal executors. A regular thread pool executes tasks
from a shared queue. These queues may grow for a short time under load, but
they should not grow continuously. Sustained queue growth means that a node is
not keeping up with the workload or that message processing is impaired. The
same logic applies to the striped executor.
+
+The striped executor divides internal cache and transaction tasks between
independent stripes: tasks in the same stripe run sequentially, while different
stripes can run in parallel.
+If a stripe is blocked, tasks related to that stripe can accumulate even when
overall CPU usage does not look high.
+
+Check queue metrics on every server node:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN ('io.communication.OutboundMessagesQueueSize'
+,'io.discovery.MessageWorkerQueueSize'
+,'threadPools.StripedExecutor.TotalQueueSize'
+,'threadPools.StripedExecutor.DetectStarvation'
+)
+ OR NAME LIKE 'threadPools.%.QueueSize'
+ORDER BY NAME;
+----
+
+Inspect queued striped tasks when the striped queue does not drain:
+
+[source,sql]
+----
+SELECT STRIPE_INDEX, THREAD_NAME, TASK_NAME, DESCRIPTION
+FROM SYS.STRIPED_THREADPOOL_QUEUE
+ORDER BY STRIPE_INDEX, THREAD_NAME;
+----
+
+As it was said, short non-zero queues are acceptable under load.
+
+On an idle node, queues usually return to zero.
+Investigate continuous growth, lack of drain after load stops, repeated
`DetectStarvation=true`, or repeated starvation warnings in logs.
+There is no universal absolute threshold.
+Queue metrics are node-local, so collect them from all server nodes.
+
+JMX uses the metric registry name to build `group` and `name` in the MBean
object name.
+The following mappings are useful for queue checks:
+
+* Registry `io.communication` is exposed as JMX group `io`, bean name
`communication`; the attribute is `OutboundMessagesQueueSize`.
+* Registry `io.discovery` is exposed as JMX group `io`, bean name `discovery`;
the attribute is `MessageWorkerQueueSize`.
+* Registry `threadPools.StripedExecutor` is exposed as JMX group
`threadPools`, bean name `StripedExecutor`; the attributes include
`TotalQueueSize`, `StripesQueueSizes`, and `DetectStarvation`.
+* Regular pools such as `threadPools.GridSystemExecutor` expose `QueueSize`.
+
+For JMX object names and SQL metric access, see
link:monitoring-metrics/new-metrics-system#jmx[JMX] and
link:monitoring-metrics/new-metrics-system#sql-view[SQL View].
+
+.JConsole view of node-local striped executor queue metrics
+image::perf-and-troubleshooting/images/healthy-cluster-queues-jconsole.png[JConsole
MBeans view showing threadPools/StripedExecutor and queue-related attributes]
+
+=== Check transactions and SQL queries
+
+A transaction or query is not unhealthy merely because it runs for some time.
+Investigate when the number or age of active operations continues to increase
after the load drops, or when the same operations repeatedly block other work.
+
+Use the transaction command to list long transactions:
+
+[source,shell]
+----
+./control.sh --tx --min-duration 60 --servers --order DURATION
+----
+
+The value `60` is only an example diagnostic filter in seconds, not a
universal production threshold.
+
+Use the system views for current transactions and SQL queries:
+
+[source,sql]
+----
+SELECT XID, STATE, START_TIME, DURATION, KEYS_COUNT, LABEL
+FROM SYS.TRANSACTIONS
+ORDER BY DURATION DESC;
+----
+
+[source,sql]
+----
+SELECT QUERY_ID, START_TIME, DURATION, INITIATOR_ID, SQL
+FROM SYS.SQL_QUERIES
+ORDER BY DURATION DESC;
+----
+
+Track related metrics:
+
+[source,sql]
+----
+SELECT NAME, VALUE
+FROM SYS.METRICS
+WHERE NAME IN (
+ 'tx.OwnerTransactionsNumber',
+ 'tx.TransactionsHoldingLockNumber',
+ 'tx.LockedKeysNumber',
+ 'pme.Duration',
+ 'pme.CacheOperationsBlockedDuration'
+)
+ORDER BY NAME;
+----
+
+Non-zero transaction counters are normal while work is running.
+The problem is sustained growth, increasing age of the oldest operations, and
failure to return to the usual range after the workload drops.
+For view definitions and metrics, see
link:monitoring-metrics/system-views#transactions[TRANSACTIONS],
link:monitoring-metrics/system-views#sql_queries[SQL_QUERIES],
link:monitoring-metrics/new-metrics#transactions[transaction metrics], and
link:monitoring-metrics/new-metrics#partition-map-exchange[Partition Map
Exchange metrics].
+
+Partition Map Exchange (PME) synchronizes partition distribution after
topology and cache changes.
+At one stage, PME waits for incomplete transactions to finish.
+A long transaction can delay a node join, cache start, and other operations
that depend on exchange.
+
+`TransactionConfiguration.setTxTimeoutOnPartitionMapExchange(...)` is
described in
link:key-value-api/transactions#long-running-transactions-termination[Long
Running Transactions Termination].
+The default is `0`, which means transactions are not rolled back because of a
PME timeout.
+The timeout is applied only when PME starts.
+Incomplete transactions that exceed the configured value can be rolled back.
+Applications must handle `TransactionRollbackException` and retry where
appropriate; see
link:key-value-api/transactions#handling-failed-transactions[Handling Failed
Transactions].
+Do not use a universal timeout value.
+Choose a value above the normal duration of legitimate transactions with a
justified safety margin, and test application behavior when rollback happens.
+
+=== Check checkpoint pressure when persistence is enabled
+
+This check applies only to data regions with Native Persistence enabled.
+A pure in-memory cluster does not perform persistence checkpoints for its
in-memory regions.
+
+A link:persistence/native-persistence#checkpointing[checkpoint] writes dirty
pages from RAM to partition files.
+Checkpointing itself is a normal background operation.
+The problem starts when the application write rate exceeds the effective
storage write speed.
+Under checkpoint-buffer or dirty-page pressure, Ignite can throttle update
threads.
+If the checkpoint buffer is exhausted, update processing can stop until the
checkpoint completes.
+
+Monitor these metrics for persistent data regions and data storage:
+
+* `io.dataregion.<region>.DirtyPages`
+* `io.dataregion.<region>.CheckpointBufferSize`
+* `io.dataregion.<region>.UsedCheckpointBufferSize`
+* `io.dataregion.<region>.TotalThrottlingTime`
+* `io.datastorage.LastCheckpointStart`
+* `io.datastorage.LastCheckpointDuration`
+* `io.datastorage.LastCheckpointPagesWriteDuration`
+* `io.datastorage.LastCheckpointTotalPagesNumber`
+* `io.datastorage.LastCheckpointFsyncDuration`
+
+Planned checkpoints run according to
`DataStorageConfiguration.checkpointFrequency`.
+Dirty-page pressure, checkpoint-buffer pressure, and some administrative
operations can trigger an earlier checkpoint.
+Successive `LastCheckpointStart` values let you estimate the real interval.
+A single early checkpoint does not prove a problem.
+A regularly shortening interval together with high `DirtyPages`, high
`UsedCheckpointBufferSize`, or growing `TotalThrottlingTime` indicates storage
or write pressure.
+
+Approximate checkpoint page-write throughput in MiB/s:
+
+[source,text]
+----
+LastCheckpointTotalPagesNumber
+* configured DataStorageConfiguration.pageSize in bytes
+* 1000
+/ LastCheckpointPagesWriteDuration in milliseconds
+/ 1048576
+----
+
+Do not calculate this when `LastCheckpointPagesWriteDuration` is zero.
+Use the actually configured `DataStorageConfiguration.pageSize`; do not assume
it is always 4 KiB.
+This is an approximate estimate, not a full disk benchmark.
+Compare several checkpoints and check whether there is enough headroom for the
normal write workload.
+
+If headroom is insufficient, use checks and changes that can be verified:
+
+* check storage latency and saturation together with
link:monitoring-metrics/new-metrics-system#monitoring-checkpointing-operations[checkpoint
metrics];
+* use faster production storage devices, as discussed in
link:persistence/persistence-tuning#purchase-production-level-ssds[Purchase
Production-Level SSDs];
+* place data files and WAL on separate physical devices, not just separate
directories on the same disk; see
link:persistence/persistence-tuning#keep-wals-separately[Keep WALs Separately];
+* review
link:persistence/persistence-tuning#adjusting-checkpointing-buffer-size[checkpoint
buffer] and link:persistence/persistence-tuning#pages-writes-throttling[page
write throttling] configuration;
+* distribute write load or add server nodes when the deployment plan allows it;
+* check link:persistence/native-persistence#wal-archive[WAL archive] I/O
contention.
+
+JMX checkpoint and data-region metrics are node-local.
+Open the same charts or attributes on each server node that owns persistent
data.
+
+.JConsole view of node-local persistence checkpoint metrics
+image::perf-and-troubleshooting/images/healthy-cluster-checkpoint-jconsole.png[JConsole
MBeans view showing io/datastorage checkpoint attributes and a persistent
dataregion bean]
+
+=== Check logs and optional features
Review Comment:
The section heading capitalization is inconsistent with other headings in
this document. Consider using title case to match the surrounding style.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
