Anton Vinogradov created IGNITE-28955:
-----------------------------------------
Summary: Continuous query start hangs under load with ZooKeeper
discovery
Key: IGNITE-28955
URL: https://issues.apache.org/jira/browse/IGNITE-28955
Project: Ignite
Issue Type: Bug
Reporter: Anton Vinogradov
Fix For: 2.19
Starting a continuous query can hang for minutes on a cluster that uses
ZooKeeper discovery while the same caches are under a heavy update load.
h3. Reproduction
Add a subclass of {{CacheContinuousQueryConcurrentPartitionUpdateTest}} that
runs under ZooKeeper discovery (the patch attached to IGNITE-28673 does exactly
that) and run {{testConcurrentUpdatesAndQueryStartAtomicCacheGroup}}.
On current master it does not finish: 318 s against the 300 s test timeout. TCP
discovery is not affected - the same test class passes in 45 s.
h3. Root cause
The message that completes the routine start and the continuous query
notifications share a topic and a pool:
{code:java}
// GridContinuousProcessor, sending the start result
ctx.io().sendToGridTopic(node, TOPIC_CONTINUOUS, new
ContinuousRoutineStartResultMessage(...), SYSTEM_POOL);
// GridContinuousProcessor, sending CQ notifications
ctx.io().sendToGridTopic(node, TOPIC_CONTINUOUS, msg, SYSTEM_POOL, ackC);
{code}
Under a heavy update load the notifications fill the queue, and the single
{{ContinuousRoutineStartResultMessage}} waits behind them - head-of-line
blocking. {{cache.query(qry)}} waits for that message, so the caller sees a
hang. The analysis in IGNITE-28673 shows the same picture from a heap dump:
millions of retained {{GridContinuousMessage}} and
{{CacheContinuousQueryEntry}} objects and one waiting
{{ContinuousRoutineStartResultMessage}}.
h3. Why only ZooKeeper today
{{GridContinuousProcessor}} picks the path by {{immutableDiscoCustomMsg =
!ctx.discovery().mutableCustomMessages()}}, which reads the
{{@DiscoverySpiMutableCustomMessageSupport}} annotation of the configured SPI.
{{TcpDiscoverySpi}} has it, so the routine start is acknowledged over discovery
and never touches the communication queue. {{ZookeeperDiscoverySpi}} does not,
so it already uses the communication path - and hits this.
h3. Why it matters beyond ZooKeeper
IGNITE-28673 removes the mutable path and makes the communication one the only
path. Merging it without fixing this would extend the hang to TCP discovery,
that is, to everyone. So this blocks IGNITE-28673.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
