[
https://issues.apache.org/jira/browse/IGNITE-29054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-29054:
-----------------------------------
Description:
Currently, socket read and write timeouts in TcpDiscoverySpi are only reported
via the log ("Socket write has timed out", "Timed out waiting for message to be
read", "Timed out waiting for message delivery receipt", "Socket operation
timed out on handshake"). There is no way to track how often such timeouts
occur without parsing (grepping) the log.
We need to expose these as metrics so the count of socket read and write
timeouts can be monitored through the metrics subsystem (e.g. JMX, system
views, exporters) instead of relying on log parsing.
Two counters in the existing `io.discovery` registry:
- SocketWriteTimeoutsCount — incremented in SocketTimeoutObject.onTimeout(), on
the sender side whose socket write timed out. Indicates the remote node is not
accepting data (TCP send buffer full): network congestion in the send
direction, or the remote node is not reading the socket (GC pause, busy, or
dead).
- SocketReadTimeoutsCount — incremented at the two SocketTimeoutException catch
sites in TcpDiscoverySpi.readMessage / readReceipt, on the reader side.
Indicates the remote node is not sending data (not responding to pings, not
sending
receipts): primary hypothesis is a long GC pause on the remote node (as the
existing LT.warn messages state: "most probably, the reason is long GC pauses
on remote node"), alternatively network congestion in the receive direction or
the remote node is dead/busy.
The pair of metrics gives a primary triage of the cause of node connectivity
problems before reading logs: read timeouts point to a potential GC pause on
the remote node, write timeouts point to a network problem in the send
direction, both growing points to a full network partition, a long GC pause, or
a dead node. The metrics give the direction of the problem but do not determine
the cause uniquely — confirming requires GC logs, process health, and network
monitoring.
was:
Currently, socket read and write timeouts in TcpDiscoverySpi are only reported
via the log ("Socket write has timed out", "Timed out waiting for message to be
read", "Timed out waiting for message delivery receipt", "Socket operation
timed out on handshake"). There is no way to track how often such timeouts
occur without parsing (grepping) the log.
We need to expose these as metrics so the count of socket read and write
timeouts can be monitored through the metrics subsystem (e.g. JMX, system
views, exporters) instead of relying on log parsing.
Two counters in the existing `io.discovery` registry:
- SocketWriteTimeoutsCount — incremented in SocketTimeoutObject.onTimeout(), on
the sender side whose socket write timed out. Indicates the remote node is not
accepting data (TCP send buffer full): network congestion in the send
direction, or the remote node is not reading the socket (GC pause, busy, or
dead).
- SocketReadTimeoutsCount — incremented at the three SocketTimeoutException
catch sites in TcpDiscoverySpi.readMessage / readReceipt and
ServerImpl.SocketReader, on the reader side. Indicates the remote node is not
sending data (not responding to pings, not sending
receipts): primary hypothesis is a long GC pause on the remote node (as the
existing LT.warn messages state: "most probably, the reason is long GC pauses
on remote node"), alternatively network congestion in the receive direction or
the remote node is dead/busy.
The pair of metrics gives a primary triage of the cause of node connectivity
problems before reading logs: read timeouts point to a potential GC pause on
the remote node, write timeouts point to a network problem in the send
direction, both growing points to a full network partition, a long GC pause, or
a dead node. The metrics give the direction of the problem but do not determine
the cause uniquely — confirming requires GC logs, process health, and network
monitoring.
> Add socket timeouts count metric to TcpDiscoverySpi
> ---------------------------------------------------
>
> Key: IGNITE-29054
> URL: https://issues.apache.org/jira/browse/IGNITE-29054
> Project: Ignite
> Issue Type: Task
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Major
> Labels: metrics-to-log
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, socket read and write timeouts in TcpDiscoverySpi are only
> reported via the log ("Socket write has timed out", "Timed out waiting for
> message to be read", "Timed out waiting for message delivery receipt",
> "Socket operation timed out on handshake"). There is no way to track how
> often such timeouts occur without parsing (grepping) the log.
> We need to expose these as metrics so the count of socket read and write
> timeouts can be monitored through the metrics subsystem (e.g. JMX, system
> views, exporters) instead of relying on log parsing.
> Two counters in the existing `io.discovery` registry:
> - SocketWriteTimeoutsCount — incremented in SocketTimeoutObject.onTimeout(),
> on the sender side whose socket write timed out. Indicates the remote node is
> not accepting data (TCP send buffer full): network congestion in the send
> direction, or the remote node is not reading the socket (GC pause, busy, or
> dead).
> - SocketReadTimeoutsCount — incremented at the two SocketTimeoutException
> catch sites in TcpDiscoverySpi.readMessage / readReceipt, on the reader side.
> Indicates the remote node is not sending data (not responding to pings, not
> sending
> receipts): primary hypothesis is a long GC pause on the remote node (as the
> existing LT.warn messages state: "most probably, the reason is long GC pauses
> on remote node"), alternatively network congestion in the receive direction
> or the remote node is dead/busy.
> The pair of metrics gives a primary triage of the cause of node connectivity
> problems before reading logs: read timeouts point to a potential GC pause on
> the remote node, write timeouts point to a network problem in the send
> direction, both growing points to a full network partition, a long GC pause,
> or a dead node. The metrics give the direction of the problem but do not
> determine the cause uniquely — confirming requires GC logs, process health,
> and network monitoring.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
