[
https://issues.apache.org/jira/browse/IGNITE-29054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127505#comment-18127505
]
Ignite TC Bot commented on IGNITE-29054:
----------------------------------------
Discovery: Added SocketReadTimeoutsCount and SocketWriteTimeoutsCount metrics
for TCP discovery socket timeouts.
The Release Note field has been filled with the wording above. Please review it
and edit it if needed.
> Add socket timeouts count metric to TcpDiscoverySpi
> ---------------------------------------------------
>
> Key: IGNITE-29054
> URL: https://issues.apache.org/jira/browse/IGNITE-29054
> Project: Ignite
> Issue Type: Task
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Major
> Labels: metrics-to-log
> Fix For: 2.19
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently, socket read and write timeouts in TcpDiscoverySpi are only
> reported via the log ("Socket write has timed out", "Timed out waiting for
> message to be read", "Timed out waiting for message delivery receipt",
> "Socket operation timed out on handshake"). There is no way to track how
> often such timeouts occur without parsing (grepping) the log.
> We need to expose these as metrics so the count of socket read and write
> timeouts can be monitored through the metrics subsystem (e.g. JMX, system
> views, exporters) instead of relying on log parsing.
> Two counters in the existing `io.discovery` registry:
> - SocketWriteTimeoutsCount — incremented in SocketTimeoutObject.onTimeout(),
> on the sender side whose socket write timed out. Indicates the remote node is
> not accepting data (TCP send buffer full): network congestion in the send
> direction, or the remote node is not reading the socket (GC pause, busy, or
> dead).
> - SocketReadTimeoutsCount — incremented at two SocketTimeoutException catch
> sites in TcpDiscoverySpi.readMessage / readReceipt, on the reader side.
> Indicates the remote node is not sending data (not responding to pings, not
> sending receipts): primary hypothesis is a long GC pause on the remote node
> (as the existing LT.warn messages state: "most probably, the reason is long
> GC pauses on remote node"), alternatively network congestion in the receive
> direction or the remote node is dead/busy.
> The pair of metrics gives a primary triage of the cause of node connectivity
> problems before reading logs: read timeouts point to a potential GC pause on
> the remote node, write timeouts point to a network problem in the send
> direction, both growing points to a full network partition, a long GC pause,
> or a dead node. The metrics give the direction of the problem but do not
> determine the cause uniquely — confirming requires GC logs, process health,
> and network monitoring.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
