[
https://issues.apache.org/jira/browse/IGNITE-29054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18127090#comment-18127090
]
Ignite TC Bot commented on IGNITE-29054:
----------------------------------------
{panel:title=Branch: [pull/13590/head] Base: [master] : No blockers
found!|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}{panel}
{panel:title=Branch: [pull/13590/head] Base: [master] : New Tests
(3)|borderStyle=dashed|borderColor=#ccc|titleBGColor=#D6F7C1}
{color:#00008b}SPI (Discovery) 2{color} [[tests
2|https://ci2.ignite.apache.org/viewLog.html?buildId=9411539]]
* {color:#013220}IgniteSpiDiscoverySelfTestSuite2:
TcpDiscoverySpiMetricsTest.testSocketReadTimeoutsMetric - PASSED{color}
* {color:#013220}IgniteSpiDiscoverySelfTestSuite2:
TcpDiscoverySpiMetricsTest.testSocketWriteTimeoutsMetric - PASSED{color}
{color:#00008b}Control Utility 1{color} [[tests
1|https://ci2.ignite.apache.org/viewLog.html?buildId=9411450]]
* {color:#013220}IgniteControlUtilityTestSuite:
CommandHandlerParsingTest.testCacheListHelpDescribesRegexPattern - PASSED{color}
{panel}
[TeamCity *--> Run :: All*
Results|https://ci2.ignite.apache.org/viewLog.html?buildId=9411555&buildTypeId=IgniteTests24Java8_RunAll]
{color:#ffffff}tcbot-analysis-comment chainBuildId=9411555
rerunBuildIds=none{color}
> Add socket timeouts count metric to TcpDiscoverySpi
> ---------------------------------------------------
>
> Key: IGNITE-29054
> URL: https://issues.apache.org/jira/browse/IGNITE-29054
> Project: Ignite
> Issue Type: Task
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Major
> Labels: metrics-to-log
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, socket read and write timeouts in TcpDiscoverySpi are only
> reported via the log ("Socket write has timed out", "Timed out waiting for
> message to be read", "Timed out waiting for message delivery receipt",
> "Socket operation timed out on handshake"). There is no way to track how
> often such timeouts occur without parsing (grepping) the log.
> We need to expose these as metrics so the count of socket read and write
> timeouts can be monitored through the metrics subsystem (e.g. JMX, system
> views, exporters) instead of relying on log parsing.
> Two counters in the existing `io.discovery` registry:
> - SocketWriteTimeoutsCount — incremented in SocketTimeoutObject.onTimeout(),
> on the sender side whose socket write timed out. Indicates the remote node is
> not accepting data (TCP send buffer full): network congestion in the send
> direction, or the remote node is not reading the socket (GC pause, busy, or
> dead).
> - SocketReadTimeoutsCount — incremented at two SocketTimeoutException catch
> sites in TcpDiscoverySpi.readMessage / readReceipt, on the reader side.
> Indicates the remote node is not sending data (not responding to pings, not
> sending receipts): primary hypothesis is a long GC pause on the remote node
> (as the existing LT.warn messages state: "most probably, the reason is long
> GC pauses on remote node"), alternatively network congestion in the receive
> direction or the remote node is dead/busy.
> The pair of metrics gives a primary triage of the cause of node connectivity
> problems before reading logs: read timeouts point to a potential GC pause on
> the remote node, write timeouts point to a network problem in the send
> direction, both growing points to a full network partition, a long GC pause,
> or a dead node. The metrics give the direction of the problem but do not
> determine the cause uniquely — confirming requires GC logs, process health,
> and network monitoring.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
