Hello devs!
We are working on replacing a core component of our Kubernetes application
with Apache Ignite 2.17.0. In our architecture we have:
1. A set of 6 "backend" pods - each of them being an Ignite cluster node
2. A set of "frontend" pods - each of them joining the cluster as an Ignite
thick client. Initially we have 9 pods, but they can scale horizontally
depending on the demand (these pods handle user requests).
In general things are stable, but we have had 3 major outages in our
staging environments so far. The behavior in all of the cases was the same:
the frontend pods got stuck in different parts of the code when trying to
communicate with ignite cluster nodes (e.g.: calling invoke on a
IgniteCache).
On the last occasion (July 6th 08:13), we identified that the issue
occurred during the process of a "rollout restart". This process, which is
very common in Kubernetes, consists of simply deleting and re-creating the
pods, one after another. This process is orchestrated by our custom
operator, which ensures that Ignite nodes leave and rejoin the cluster
gracefully, always waiting for each node to gracefully shutdown before
proceeding to the next (we tested this extensively).
For this last occasion, we have analyzed Ignite logs and thread dumps as
much as we could. From what we can see, it seems that there is some issue
during Ignite's internal PME (partition-map-exchange) process. The fact
that the PME process was unable to complete appears to have caused the
hangs we observed at the application level, such as calls to
`cache.invoke`. Below, more details about our investigation:
1. The aforementioned "rollout restart" process started, causing many
Ignite nodes to leave and rejoin the cluster. Three cluster nodes left and
rejoined the cluster gracefully, without any issues whatsoever.
2. The fourth member (let's call it B2, "backend-2") was the fourth to be
restarted, at 08:18:29. This node (B2) was the "coordinator" pod until that
point (as called by Ignite).
3. When B2 left the grid, it ceased to be the coordinator. Another node, B1
("backend-1"), was elected as the new coordinator (as seen in its logs
below).
Coordinator changed [prev=TcpDiscoveryNode
[id=f3be5273-d105-414f-b9a1-7ee66ea4e7bd, consistentId=backend-2,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 100.96.100.10, 127.0.0.1],
sockAddrs=HashSet [/[0:0:0:0:0:0:0:1%lo]:47500, /127.0.0.1:47500, /
100.96.100.10:47500], discPort=47500, order=33, intOrder=24, loc=false,
ver=2.17.0#20250209-sha1:d53d4540, isClient=false], cur=TcpDiscoveryNode
[id=b40946f8-4110-4dca-bc82-701780c8a59f, consistentId=backend-1,
addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 100.96.150.24, 127.0.0.1],
sockAddrs=HashSet [/[0:0:0:0:0:0:0:1%lo]:47500, /100.96.150.24:47500, /
127.0.0.1:47500], discPort=47500, order=35, intOrder=25, loc=true,
ver=2.17.0#20250209-sha1:d53d4540, isClient=false]]
4. Once B1 was elected as the new coordinator, the PME process stopped
working properly. We were able to confirm this by looking closer at the PME
message exchange logs between B1 (the coordinator) and a new thick client
that tried to join the cluster (let's call it C1). When C1 joins the
cluster, it starts an exchange and send a single message:
2026-07-06 08:23:53,502 INFO exchange-worker-#46
"Started exchange init [topVer=882/0, crd=false, evt=NODE_JOINED,
evtNode=bd236ba3-98a3-4bc1-8936-48e6eb9f4da2]"
2026-07-06 08:23:53,670 INFO exchange-worker-#46
"Finished exchange init [topVer=882/0, crd=false]"
But it seems that the coordinator (B1) never answers back. Therefore, C1
keeps waiting indefinitely:
2026-07-06 08:24:03,708 INFO grid-nio-worker-tcp-comm-3
"Exchange future waiting for coordinator response
[crd=b40946f8-4110-4dca-bc82-701780c8a59f,
topVer=AffinityTopologyVersion [topVer=882, minorTopVer=0]]"
2026-07-06 08:24:23,699 INFO grid-nio-worker-tcp-comm-3
"Exchange future waiting for coordinator response
[crd=b40946f8-4110-4dca-bc82-701780c8a59f,
topVer=AffinityTopologyVersion [topVer=882, minorTopVer=0]]"
2026-07-06 08:24:03,671 WARN exchange-worker-#46
"Failed to wait for partition map exchange
[topVer=AffinityTopologyVersion [topVer=882, minorTopVer=0],
node=bd236ba3-98a3-4bc1-8936-48e6eb9f4da2]"
2026-07-06 08:24:03,671 WARN exchange-worker-#46
"Ready affinity version: AffinityTopologyVersion [topVer=-1,
minorTopVer=0]"
2026-07-06 08:24:03,681 WARN exchange-worker-#46
"Last exchange future: GridDhtPartitionsExchangeFuture
[topVer=882/0, evt=NODE_JOINED, evtNode=bd236ba3]"
2026-07-06 08:24:33,493 WARN imdg-service-startup-thread-0
"Still waiting for initial partition map exchange
[fut=GridDhtPartitionsExchangeFuture [evtNode=bd236ba3...]]"
5. If we look at the coordinator node (B1), we can actually confirm it
**received a message from C1** during this exchange:
2026-07-06 08:23:53.666 info {
"timestamp": "2026-07-06 08:23:53,666",
"logLevel": "INFO",
"threadName": "grid-nio-worker-tcp-comm-1-#24%TcpCommunicationSpi%",
"loggerName":
"org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi",
"message": "Accepted incoming communication connection [locAddr=/
100.96.150.24:47100, rmtAddr=/100.96.55.7:47818]"
}
And it actually claims it replied the message (in less than 10ms!):
2026-07-06 08:23:53,679 INFO sys-#41117%backend-1%
"Fast replied to single message
[exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=882, minorTopVer=0],
discoEvt=null,
nodeId=bd236ba3,
evt=NODE_JOINED],
nodeId=bd236ba3]"
6. However, even though B1 is receiving the message from C1 and apparently
answering it, it seems C1 never receives the message as the PME process
never finishes. The system remained in the state for almost an hour, until
we manually intervened and restarted the Ignite cluster. During this time,
many different thick clients tried to join the cluster and they all faced
the same issue: they were all unable to complete the initial PME because
the coordinator never responded (even though it always claimed it did).
=======
** ADDITIONAL CLUES **
1. As mentioned, frontend pods (thick clients) are unable to finish their
initial PME during degradation. However, it looks like the same does not
happen with the backend pods (Ignite nodes), as we confirmed that they did
receive full messages back from the coordinator during this window:
08:19:53.872 "Received full message, will finish exchange
[node=b40946f8, resVer=topVer=872/1]" ← backend-0
sys-#44341
08:19:53.874 "Received full message, will finish exchange
[node=b40946f8, resVer=topVer=872/1]" ← backend-3 sys-#56
08:19:53.876 "Received full message, will finish exchange
[node=b40946f8, resVer=topVer=872/1]" ← backend-2 sys-#58
08:19:53.881 "Received full message, will finish exchange
[node=b40946f8, resVer=topVer=872/1]" ← backend-4 sys-#60
2. An interesting finding is that during the whole degradation, we have
noticed spikes in "Accepted incoming communication connection" log
messages. For example, for pod C1, we saw the following:
08:23:49.324 Accepted incoming communication connection [rmtAddr=/
100.96.150.24:54648]
08:23:49.395 Accepted incoming communication connection [rmtAddr=/
100.96.150.24:54650]
08:23:49.460 Accepted incoming communication connection [rmtAddr=/
100.96.150.24:54662]
... (continuous at ~60ms intervals through 08:23:53 and beyond)
We think this is suspicious; it hints to us that the coordinator (B1) is
opening too many connections to C1. However, we were unable to find any
ERROR log from Ignite indicating connection/network issues.
3. The investigation above focused on a **frontend (thick client) joining
the cluster after the system degraded**, as it is the easiest scenario to
analyze. However, it is also worth mentioning that frontends that were
already part of the cluster before the degradation started were also
affected.
Before the degradation, these "old" frontend pods were normally handling
user requests. When the degradation happened, they started hanging in
unrelated Ignite calls, like cache.invoke:
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
at java.util.concurrent.locks.LockSupport.park([email protected]
/LockSupport.java:341)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:181)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:144)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.invoke(GridDhtAtomicCache.java:802)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearAtomicCache.invoke(GridNearAtomicCache.java:493)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.invoke(IgniteCacheProxyImpl.java:1727)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.invoke(IgniteCacheProxyImpl.java:1773)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.invoke(GatewayProtectedCacheProxy.java:1248)
Our current understanding is that this happened because the PME process was
broken, causing the topology version of these thick clients to be outdated
and never refreshed. This is sustained by the following logs that we see in
these frontend pods:
08:19:53.404 "Started exchange init [topVer=872/1, crd=false,
DISCOVERY_CUSTOM_EVT]"
08:19:53.407 "Finished exchange init [topVer=872/1, crd=false]"
← sent single message, now waiting for full message
08:19:54.413 "Delay process full message without exchange id
(there is exchange in progress) [nodeId=b40946f8]"
← first full message deferred into delayedLatestMsg
08:19:55.981 "Delay process full message without exchange id
(there is exchange in progress) [nodeId=b40946f8]"
← second full message OVERWRITES first in delayedLatestMsg
08:20:03.408 WARN "Failed to wait for partition map exchange
[topVer=872/1]"
08:20:03.409 WARN "Ready affinity version: topVer=872/0"
...which are reporting the same PME problem.
=======
We have not been able to forcefully simulate this degradation. Any attempt
to block the JVM process of the coordinator led to a new coordinator
election and the cluster self-recovered, as expected.
We would appreciate it if anyone could help us understand this issue
further or provide additional insights on how we can continue
troubleshooting it. We will continue investigating this on our side, but at
the moment we are still trying to figure out the best next steps.
At the time of the event, we were using INFO log level for Ignite. If
helpful, we can share more logs and thread dumps taken during the event or
run additional tests.
Thanks!
----
Additional Environment details:
- Apache Ignite version: 2.17.0
- Java: OpenJDK 17
- GC: ZGC
- Persistence: disabled
- TLS for intra-node communication: enabled
