This is an automated email from the ASF dual-hosted git repository.
sergey-chugunov-1985 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git
The following commit(s) were added to refs/heads/master by this push:
new 1493bbb3139 IGNITE-29177 Analyze and test the reliability of MDC
historical rebalance mechanism (#13714)
1493bbb3139 is described below
commit 1493bbb3139d4955e5ebc00e8d9ff71d7d55df38
Author: Egor Baranov <[email protected]>
AuthorDate: Tue Oct 27 15:44:49 2026 +0300
IGNITE-29177 Analyze and test the reliability of MDC historical rebalance
mechanism (#13714)
---
docs/_docs/clustering/multi-data-center.adoc | 1 +
docs/_docs/data-rebalancing.adoc | 119 +++++
.../rebalancing/MultiDcRebalancingTest.java | 587 ++++++++++++++++++++-
.../MultiDcSplitBrainRebalanceTest.java | 446 ++++++++++++++++
.../ignite/testsuites/IgniteCacheTestSuite19.java | 2 +
.../tests/ignitetest/services/mdc/mdc_cluster.py | 18 +-
.../tests/mdc/mdc_historical_rebalance_test.py | 142 +++++
7 files changed, 1308 insertions(+), 7 deletions(-)
diff --git a/docs/_docs/clustering/multi-data-center.adoc
b/docs/_docs/clustering/multi-data-center.adoc
index a5a37e25b1f..1eb10bd8f51 100644
--- a/docs/_docs/clustering/multi-data-center.adoc
+++ b/docs/_docs/clustering/multi-data-center.adoc
@@ -246,6 +246,7 @@ During rebalancing, Ignite prefers nodes from the same data
center when such sup
If the required partition data cannot be obtained from the same data center,
Ignite can rebalance from a
remote data center.
This behavior applies to both full and historical rebalance.
+See link:data-rebalancing#rebalance-strategy[Rebalance Strategy] for the
conditions that determine which strategy is used.
== Monitoring
diff --git a/docs/_docs/data-rebalancing.adoc b/docs/_docs/data-rebalancing.adoc
index 2a603f8f224..49a43ca76e7 100644
--- a/docs/_docs/data-rebalancing.adoc
+++ b/docs/_docs/data-rebalancing.adoc
@@ -28,6 +28,83 @@ In pure in-memory clusters, the default behavior is to start
rebalancing immedia
In clusters with persistence, the baseline topology has to be changed manually
(default behavior), or can be changed automatically when
link:clustering/baseline-topology#baseline-topology-autoadjustment[automatic
baseline adjustment] is enabled.
====
+== Rebalancing Strategy [[rebalance-strategy]]
+
+When a node joins or rejoins the cluster, each partition is restored using one
of two strategies:
+
+- *Full rebalance* — the node downloads the entire partition content from
another server node that holds a copy of the partition.
+- *Historical rebalance* — the node restores its own data from the local disk
and downloads only the delta (WAL entries written while it was absent) from the
source node that holds a copy of the partition.
+
+The strategy is selected per partition based on the following rules:
+
+. A full rebalance is used if the node has no local data for the partition
(update counter is zero).
++
+This applies both to a *new node joining for the first time* (it has no data
on disk at all) and to a node that was *evicted from the baseline* and rejoined
(its local partition data was invalidated). In both cases the full partition
content must be downloaded from another node.
+
+. A full rebalance is used if the delta exceeds the total number of entries in
the partition — transferring the full content is more efficient in that case.
++
+Example: a partition contains 100 entries. While the node was absent, 500
writes hit this partition. Downloading 500 delta entries is more expensive than
just downloading all 100 entries → full rebalance.
+
+. A full rebalance is used if the partition size is below the
`historical.rebalance.threshold` (DMS property, default 500 entries), unless
the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.
++
+Example: a partition contains 10 entries. While the node was absent, 3 writes
hit this partition. The delta (3) is smaller than the partition (10), but 10 <
500 — the partition is too small to justify WAL replay overhead → full
rebalance. With `IGNITE_PREFER_WAL_REBALANCE=true` this threshold check is
bypassed, and a historical rebalance is used regardless of the partition size
(provided the other rules above are not triggered).
+
+. A historical rebalance is used when all of the above conditions are not met:
the node has local data, the delta is smaller than the partition, and the
partition exceeds the threshold.
++
+Example: a partition contains 10,000 entries. While the node was absent, 30
writes hit this partition. Downloading 30 delta entries is far cheaper than all
10,000 → historical rebalance.
+
+[NOTE]
+====
+If both nodes report the same update counter for a partition, no rebalance is
performed for that partition.
+
+A node reports counters only for partitions it has written to. If the dataset
is small relative to the partition count, most partitions will have a zero
counter and will not be included in the rebalance.
+====
+
+[NOTE]
+====
+*Historical rebalance requires the delta to be present in the WAL of the
source node.* If the WAL segments that contain the delta were already archived
or deleted by the time the node rejoins — for example, the node was absent for
a long time, or WAL history and archiving are configured with a small retention
— the source node cannot supply the delta. In that case Ignite does not lose
data: the supplier detects the failure to iterate over the WAL, marks the last
checkpoint as not applica [...]
+====
+
+[CAUTION]
+====
+The `rebalanceDelay` (deprecated) setting also prevents a historical
rebalance: when a delay is configured, the rebalance is scheduled outside of
the partition map exchange context, and the historical path (which is tied to
that context) is never taken — every partition is restored with a full
rebalance regardless of the delta size and regardless of
`IGNITE_PREFER_WAL_REBALANCE`.
+====
+
+In a link:clustering/multi-data-center[multi-data-center] setup, the node
requesting the data prefers a source from the same data center that holds a
copy of the partition. If no such node is available, it falls back to a node
from a remote data center. This behavior applies to both full and historical
rebalance.
+
+== Historical Rebalance: Internal Mechanism
+
+The following section describes the internal flow of a historical rebalance.
This information is useful for understanding why the rebalance is chosen as
historical or full, and what happens when the historical path fails.
+
+=== Source Selection
+
+When the node requesting data (the *demander*) needs a partition, it selects a
source node from the available nodes that hold a copy of that partition. In a
link:clustering/multi-data-center[multi-data-center] setup, a node from the
same data center is preferred. The selection happens during the partition map
exchange phase, and the source nodes can be changed during the rebalance if one
of them becomes inapplicable.
+
+=== The Demand-Supply Protocol
+
+1. The demander sends a *demand* message to the chosen source node (supplier),
specifying which partitions it needs and whether it expects a full or
historical transfer.
+2. The supplier iterates over the partition data (or WAL entries, for
historical) and sends *supply* messages back to the demander, batch by batch.
+3. The demander sends the next demand when it is ready for more data. This
repeats until all partitions are either fully transferred or marked as *missed*.
+
+=== Fallback from Historical to Full
+
+If the supplier fails to read the WAL (for example, the WAL segments were
archived or deleted), the historical rebalance for that partition is
interrupted. The following happens:
+
+. The supplier's historical iterator throws an exception.
+. The supplier marks the last checkpoint as *not applicable for WAL rebalance*
(so that subsequent checkpoints will not try to serve WAL for this partition).
+. The supplier marks all remaining partitions as *missed* in the supply
message.
+. The demander, upon receiving the *missed* marking for a partition that was
originally requested as historical, excludes the source node from historical
rebalancing and re-requests the partition as *full* from another available
source node.
+
+This mechanism ensures that data is never lost: the node will always receive
either the historical delta or the full partition content.
+
+=== Protection Against Data Loss on Full Rebalance
+
+When a partition is requested as *full* (not historical), the demander
additionally checks whether the source node's copy of the partition is up to
date. A source node that is known to be behind (its update counter for the
partition is lower than the demander's local counter) is *excluded from full
rebalancing* for that partition — accepting a full partition from such a node
would overwrite newer local data with older data. The demander then selects a
different source or waits for the pa [...]
+
+=== Message Ordering (Striped Executor)
+
+Historical rebalance messages are processed on a *striped* executor: all
supply messages from the same source node are handled sequentially in the order
they were received. This preserves the ordering of WAL entries, which is
required for correct replay. Full rebalance messages are processed on a regular
(non-ordered) executor because individual partition transfers are independent.
+
== Configuring Rebalancing Mode
Ignite supports both synchronous and asynchronous rebalancing.
@@ -147,6 +224,48 @@ The following table lists the properties of
`IgniteConfiguration` related to reb
|===
+== Rebalance Risks
+
+A rebalance transfers a significant volume of data between nodes. Depending on
the cluster size and the data volume, it can have a measurable impact on
availability and performance. The following risks should be considered before
scaling a cluster, particularly across data centers.
+
+=== Bandwidth saturation
+
+During a rebalance, one or more source nodes simultaneously transfer partition
data to the target node. In a
link:clustering/multi-data-center[multi-data-center] setup this traffic
traverses the inter-DC (WAN) link, which is typically the narrowest part of the
topology. A rebalance can saturate this link, degrading or starving all other
traffic between the data centers (client requests, replication, monitoring).
+
+The impact is proportional to the number of source nodes and the size of the
transferred partitions, and inversely proportional to the available link
bandwidth. It is most severe for small clusters (few nodes, each holding a
large share of the data) over limited WAN links.
+
+=== Full cluster stall (synchronous mode)
+
+In the `SYNC` rebalancing mode, all public cache operations are blocked until
the rebalance completes. For a large cache over a slow or high-latency link,
this can result in a prolonged unavailability of the entire cache.
+
+=== Target node disk I/O
+
+The target node must persist all received partitions to disk. This produces a
write I/O spike that can saturate the node's storage. If the node serves live
traffic concurrently, request latency will increase.
+
+=== Source node contention
+
+Each source node transfers its share of the partitions while continuing to
serve requests. This reduces the effective request throughput of the existing
nodes for the duration of the rebalance. The effect is more pronounced in
smaller clusters, where each node holds a larger share of the data.
+
+=== Rebalance timeout
+
+Each rebalance message is subject to `rebalanceTimeout` (default 10 seconds).
Over a slow or high-latency link, transferring a single batch may exceed this
value, causing the rebalance to fail and the target node to remain only
partially populated.
+
+== Recommendations for Cluster Administrators
+
+The following recommendations apply when scaling a cluster, especially when
adding nodes to a remote or a new data center.
+
+. *Use asynchronous rebalancing.* Configure the cache with the `ASYNC`
rebalancing mode so that the cache remains available for operations while the
data is transferred in the background. The `SYNC` mode should be avoided for
large caches or cross-DC transfers.
+. *Throttle the transfer.* Set `rebalanceThrottle` (a delay between batches)
and, if necessary, reduce `rebalanceBatchSize`, to limit the bandwidth consumed
by the rebalance and avoid saturating the inter-DC link.
+. *Schedule during low-load periods.* Start the scaling operation when the
cluster is under minimal load, so that the rebalance traffic competes with as
little production traffic as possible.
+. *Increase the timeout for slow links.* On high-latency or low-bandwidth
links, increase `rebalanceTimeout` (and reduce `rebalanceBatchSize`) so that
individual batches do not time out.
+. *Manage the baseline explicitly.* When the baseline topology is not
auto-adjusted, confirm the new node is fully up and then commit the current
cluster topology as the new baseline via
`ignite.cluster().setBaselineTopology(ignite.cluster().topologyVersion())`, so
that the rebalance starts at the intended moment.
+. *Monitor the rebalance.* Track the
link:monitoring-metrics/new-metrics#caches[rebalancing metrics], the inter-DC
link utilization, and the I/O load of the target node during the operation, and
be prepared to throttle or abort if the impact exceeds expectations.
+
+[NOTE]
+====
+For a node joining for the first time (no local data), the rebalance is always
a full transfer of its partitions, regardless of the
`IGNITE_PREFER_WAL_REBALANCE` setting. The bandwidth, disk, and stall
considerations above therefore apply in full.
+====
+
== Monitoring Rebalancing Process
You can monitor the link:monitoring-metrics/new-metrics#caches[rebalancing
process for specific caches using JMX].
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java
index 9f77dc49626..9834e386c5d 100644
---
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java
@@ -17,6 +17,7 @@
package org.apache.ignite.internal.processors.cache.distributed.rebalancing;
+import java.util.Set;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteException;
@@ -28,20 +29,27 @@ import
org.apache.ignite.configuration.DataRegionConfiguration;
import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.internal.IgniteEx;
import org.apache.ignite.internal.managers.communication.GridIoMessage;
+import org.apache.ignite.internal.processors.cache.GridCacheProxyImpl;
import
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemandMessage;
+import
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtDemandedPartitionsMap;
+import
org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager;
import org.apache.ignite.internal.util.typedef.F;
import org.apache.ignite.lang.IgniteInClosure;
import org.apache.ignite.plugin.extensions.communication.Message;
import org.apache.ignite.spi.IgniteSpiException;
import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.testframework.GridTestUtils;
import org.apache.ignite.testframework.junits.WithSystemProperty;
import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
import org.junit.Test;
import static
org.apache.ignite.IgniteSystemProperties.IGNITE_PREFER_WAL_REBALANCE;
-/** */
-@WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+/**
+ * Tests the MDC rebalance decision logic: when the engine chooses full,
historical, or
+ * skips rebalance after a node rejoins the cluster. Live network split-brain
is covered
+ * by {@link MultiDcSplitBrainRebalanceTest}.
+ */
public class MultiDcRebalancingTest extends GridCommonAbstractTest {
/** */
private static final String DC1 = "DC1";
@@ -91,16 +99,570 @@ public class MultiDcRebalancingTest extends
GridCommonAbstractTest {
/** */
@Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
public void testHistoricalRebalanceSameDC() throws Exception {
checkHistoricalRebalance(false);
}
/** */
@Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
public void testHistoricalRebalanceCrossDC() throws Exception {
checkHistoricalRebalance(true);
}
+ /**
+ * Verifies that no rebalance demand is sent when a node restarts without
any
+ * writes having occurred during its absence.
+ *
+ * <p>Scenario: 2 nodes, 1000 entries, checkpoint. Node 1 stops, nothing is
+ * written, node 1 restarts. Both nodes have identical partition counters,
+ * the CRD sees only one value per partition, no node is "behind", no
demand.
+ */
+ @Test
+ public void testNoRebalanceWhenNoDelta() throws Exception {
+ IgniteCache<Object, Object> cache = startTwoNodeCluster(1);
+
+ for (int i = 0; i < 1000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+ IgniteEx ignite1 = startGrid(1, DC2);
+ awaitPartitionMapExchange(true, true, null);
+ RebalanceAwareCommSPI spi = commSPI(ignite1);
+ assertFalse(spi.historical);
+ assertFalse(spi.full);
+ }
+
+ /**
+ * Verifies that partitions below {@code historical.rebalance.threshold}
(default 500)
+ * are NOT eligible for historical rebalance. The node uses full rebalance
instead.
+ *
+ * <p>Scenario: 2 nodes, only 100 entries (each partition has ~1 entry,
well
+ * below threshold 500). Checkpoint. Node 1 stops, 5 more entries written
+ * (small delta), node 1 restarts. Despite the tiny delta, partition size
< 500
+ * means it is not a historical candidate, so full rebalance is used.
+ */
+ @Test
+ public void testBelowThresholdForcesFull() throws Exception {
+ IgniteCache<Object, Object> cache = startTwoNodeCluster(1);
+
+ for (int i = 0; i < 100; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 100; i < 105; i++)
+ cache.put(i, i);
+
+ IgniteEx ignite1 = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ RebalanceAwareCommSPI spi = commSPI(ignite1);
+ assertTrue(spi.rebalanceMsgCnt > 0);
+ assertFalse(spi.historical);
+ assertTrue(spi.full);
+ }
+
+ /**
+ * Verifies that with a very small dataset, only non-empty partitions
appear in
+ * the rebalance demand (the skipZeros mechanism).
+ *
+ * <p>Scenario: 2 nodes, only 10 entries. Since there are 512 partitions,
at most
+ * ~10 have counter > 0. Node 1 stops, 5 more entries written, node 1
restarts.
+ * The SingleMessage omits all empty partitions, the CRD only detects
differences
+ * for the ~10 non-empty partitions, total demanded is well below 512.
+ */
+ @Test
+ public void testSkipZerosReducesRebalancedPartitions() throws Exception {
+ IgniteCache<Object, Object> cache = startTwoNodeCluster(1);
+
+ for (int i = 0; i < 10; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 10; i < 15; i++)
+ cache.put(i, i);
+
+ IgniteEx ignite1 = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ RebalanceAwareCommSPI spi = commSPI(ignite1);
+ assertTrue(spi.rebalanceMsgCnt > 0);
+ assertTrue("Expected < 512 but got " + spi.totalDemandedParts,
+ spi.totalDemandedParts < 512);
+ }
+
+ /**
+ * Verifies historical rebalance after a DC node comes back following an
outage.
+ *
+ * <p>Scenario: 3 nodes — node0 (DC1), node1 (DC2), node2 (DC1). All in
baseline,
+ * 10000 entries written. node1 (DC2) is stopped and writes nothing while
down.
+ * The two DC1 nodes write 10000 more entries each. Then node1 restarts.
+ * Node1 must get cross-DC historical rebalance with the delta from both
DC1 nodes.
+ */
+ @Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+ public void testDcNodeDownHistoricalRebalance() throws Exception {
+ IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+ for (int i = 0; i < 10_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 10_000; i < 20_000; i++)
+ cache.put(i, i);
+
+ IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 20_000; i < 30_000; i++)
+ cache2.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1 = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ RebalanceAwareCommSPI spi = commSPI(ignite1);
+ assertTrue(spi.rebalanceMsgCnt > 0);
+ assertTrue(spi.historical);
+ assertTrue(spi.crossDcRebalance);
+ }
+
+ /**
+ * Verifies that after a DC node comes back from an outage, no data is
lost. Data
+ * written by both surviving DC1 nodes while the node was down is fully
accessible
+ * from the returning DC2 node after recovery.
+ *
+ * <p>Scenario: 3 nodes, node1 (DC2) down. DC1 nodes write 20000 new
entries while
+ * it is down. After recovery, all 30000 entries are readable from the DC2
node.
+ */
+ @Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+ public void testDcNodeDownNoDataLoss() throws Exception {
+ IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+ for (int i = 0; i < 10_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 10_000; i < 20_000; i++)
+ cache.put(i, i);
+
+ IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 20_000; i < 30_000; i++)
+ cache2.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1 = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 0; i < 30_000; i += 3000) {
+ Object val = cache1.get(i);
+ assertEquals(i, val);
+ }
+ }
+
+ /**
+ * Verifies that after a DC node comes back from an outage, if partition
size is
+ * below {@code historical.rebalance.threshold} (default 500), full
rebalance is
+ * used (no PREFER_WAL_REBALANCE set).
+ *
+ * <p>Scenario: 3 nodes, 100 entries (below threshold). node1 (DC2) down,
DC1
+ * writes 200 more entries. After recovery, full rebalance is expected.
+ */
+ @Test
+ public void testDcNodeDownBelowThresholdFullRebalance() throws Exception {
+ IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+ for (int i = 0; i < 100; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 100; i < 200; i++)
+ cache.put(i, i);
+
+ IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 200; i < 300; i++)
+ cache2.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1 = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ RebalanceAwareCommSPI spi = commSPI(ignite1);
+ assertTrue(spi.rebalanceMsgCnt > 0);
+ assertFalse(spi.historical);
+ assertTrue(spi.full);
+ assertTrue(spi.crossDcRebalance);
+ }
+
+ /**
+ * Verifies that after a DC node comes back from an outage, if the delta
is larger
+ * than the partition size (and no PREFER_WAL_REBALANCE), full rebalance
is used —
+ * transferring the full partition is more efficient than replaying the
delta.
+ *
+ * <p>Scenario: 3 nodes, 1000 base entries. node1 (DC2) down. The DC1
nodes each
+ * write 20000 entries (delta >> partition size). After recovery, full
rebalance
+ * is expected (not historical).
+ */
+ @Test
+ public void testDcNodeDownLargeDeltaFullRebalance() throws Exception {
+ IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+ for (int i = 0; i < 1000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 1000; i < 21_000; i++)
+ cache.put(i, i);
+
+ IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 21_000; i < 41_000; i++)
+ cache2.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1 = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ RebalanceAwareCommSPI spi = commSPI(ignite1);
+ assertTrue(spi.rebalanceMsgCnt > 0);
+ assertFalse(spi.historical);
+ assertTrue(spi.full);
+ assertTrue(spi.crossDcRebalance);
+ }
+
+ /**
+ * Verifies that if one of two available source nodes also goes down while
the DC
+ * node is away, the surviving source node can still serve the historical
rebalance.
+ *
+ * <p>Scenario: 3 nodes (node0 DC1, node1 DC2, node2 DC1). node1 (DC2) is
stopped.
+ * Then node0 (one DC1 source) is ALSO stopped. Only node2 remains. After
recovery,
+ * node1 must get historical rebalance from node2 alone.
+ */
+ @Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+ public void testDcNodeDownOneSourceNodeSurvives() throws Exception {
+ IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+ for (int i = 0; i < 10_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+ stopGrid(0);
+ IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 10_000; i < 20_000; i++)
+ cache2.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1 = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ RebalanceAwareCommSPI spi = commSPI(ignite1);
+ assertTrue(spi.rebalanceMsgCnt > 0);
+ assertTrue(spi.historical);
+ assertTrue(spi.crossDcRebalance);
+ IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 0; i < 20_000; i += 5000) {
+ Object val = cache1.get(i);
+ assertEquals(i, val);
+ }
+ }
+
+ /**
+ * Verifies that concurrent writes on the source node during an in-progress
+ * historical rebalance do not corrupt data. After the rebalance completes,
+ * all entries (old + delta + concurrent) are accessible.
+ *
+ * <p>Scenario: 3 nodes. node1 (DC2) is stopped. DC1 writes 10000 entries
(delta).
+ * node1 restarts (triggers rebalance). While the rebalance is in flight,
+ * DC1 writes another 5000 entries. After rebalance completes, verify all
+ * data is accessible.
+ */
+ @Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+ public void testConcurrentWritesDuringRecovery() throws Exception {
+ IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+ for (int i = 0; i < 10_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 10_000; i < 20_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1 = startGrid(1, DC2);
+ IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 20_000; i < 25_000; i++)
+ cache2.put(i, i);
+
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 0; i < 25_000; i += 5000) {
+ Object val = cache1.get(i);
+ assertEquals(i, val);
+ }
+ }
+
+ /**
+ * Verifies that after a DC node outage recovery, if a new node also joins
+ * (triggering affinity reassignment), the returning node gets a mixed
demand:
+ * historical for its own partitions AND full for reassigned partitions.
+ *
+ * <p>Scenario: 2 nodes initially. node1 (DC2) is stopped, DC1 writes
delta.
+ * Recovery: node1 restarts AND a new node (node2, DC1) joins
simultaneously.
+ * The topology change triggers both historical rebalance (for DC2's old
+ * partitions) and full rebalance (for partitions reassigned to node2).
+ */
+ @Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+ public void testAffinityReassignmentMixedRebalance() throws Exception {
+ IgniteCache<Object, Object> cache = startTwoNodeCluster(1);
+
+ for (int i = 0; i < 10_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 10_000; i < 20_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1 = startGrid(1, DC2);
+ IgniteEx ignite2 = startGrid(2, DC1);
+ resetBaselineTopology();
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ waitRebalanceFinished(ignite2, DEFAULT_CACHE_NAME);
+
+ RebalanceAwareCommSPI spi1 = commSPI(ignite1);
+ assertTrue(spi1.rebalanceMsgCnt > 0);
+ assertTrue(spi1.historical);
+
+ RebalanceAwareCommSPI spi2 = commSPI(ignite2);
+ assertTrue("New node should receive full rebalance", spi2.full);
+
+ IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 0; i < 20_000; i += 5000) {
+ Object val = cache1.get(i);
+ assertEquals(i, val);
+ }
+ }
+
+ /**
+ * Verifies that with {@code rebalanceDelay > 0} the rebalance always uses
the full
+ * strategy.
+ *
+ * <p>Documented limitation (see data-rebalancing.adoc, "Rebalance
Strategy"): when a
+ * rebalance delay is configured, the rebalance is scheduled outside of
the partition
+ * map exchange context, and the historical path (which is tied to that
context) is
+ * never taken — every partition is restored with a full rebalance
regardless of the
+ * delta size and regardless of {@code IGNITE_PREFER_WAL_REBALANCE}.
+ *
+ * <p>Related ticket: IGNITE-14524.
+ */
+ @Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+ public void testRebalanceDelayForcesFull() throws Exception {
+ pds = true;
+ IgniteEx ignite0 = startGrid(0, DC1);
+ startGrid(1, DC2);
+ startGrid(2, DC1);
+ ignite0.cluster().state(ClusterState.ACTIVE);
+ CacheConfiguration<Object, Object> cacheCfg = new
CacheConfiguration<>(DEFAULT_CACHE_NAME);
+ cacheCfg.setBackups(2);
+ cacheCfg.setRebalanceDelay(3000);
+ IgniteCache<Object, Object> cache = ignite0.createCache(cacheCfg);
+
+ for (int i = 0; i < 10_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 10_000; i < 20_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1 = startGrid(1, DC2);
+ RebalanceAwareCommSPI spi = commSPI(ignite1);
+
+ boolean demandSent = GridTestUtils.waitForCondition(() ->
spi.rebalanceMsgCnt > 0, 30_000);
+
+ assertTrue("No rebalance demand received within 30s", demandSent);
+
+ awaitPartitionMapExchange(true, true, null);
+ assertTrue(spi.full);
+ assertFalse(spi.historical);
+ }
+
+ /**
+ * Verifies that after MULTIPLE restarts (stop→write→start, repeated), the
recovering
+ * node accumulates all data across all restart cycles.
+ *
+ * <p>Per IGNITE-17738: after a crash/restart, historical rebalance may
not deliver
+ * updates between LWM and HWM. This test exercises two consecutive
restart cycles
+ * and asserts full data integrity after each.
+ *
+ * <p>Scenario: 3 nodes. 10_000 base entries.
+ * Cycle 1: node1 down → node0 writes 10_000 → node1 restarts → verify
20_000.
+ * Cycle 2: node1 down → node0 writes 10_000 → node1 restarts → verify
30_000.
+ */
+ @Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+ public void testMultipleRestartsNoDataLoss() throws Exception {
+ IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+ for (int i = 0; i < 10_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 10_000; i < 20_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1 = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+
+ IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 0; i < 20_000; i += 5000) {
+ Object val = cache1.get(i);
+ assertEquals(i, val);
+ }
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 20_000; i < 30_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ IgniteEx ignite1Restarted = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1Restarted, DEFAULT_CACHE_NAME);
+
+ IgniteCache<Object, Object> cache1After =
ignite1Restarted.cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 0; i < 30_000; i += 5000) {
+ Object val = cache1After.get(i);
+ assertEquals(i, val);
+ }
+ }
+
+ /**
+ * Verifies that when historical (WAL) rebalance is impossible, the
supplier falls
+ * back to a FULL rebalance and no data is lost.
+ *
+ * <p>Truncated WAL is simulated deterministically by marking the last
checkpoint as
+ * inapplicable for WAL rebalance on the source nodes
+ * ({@code lastCheckpointInapplicableForWalRebalance}) — exactly the flag
the supplier
+ * sets itself when a WAL segment turns out to be unreadable
+ * ({@code GridDhtPartitionSupplier} fallbackToFullRebalance, line 451).
This is the
+ * production situation when WAL segments covering the delta were already
deleted.
+ *
+ * <p>Assertion: despite {@code PREFER_WAL_REBALANCE=true} the recovering
node gets a
+ * FULL rebalance (historical is impossible), and all entries are present
afterwards.
+ */
+ @Test
+ @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+ public void testWalTruncatedForcesFull() throws Exception {
+ IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+ for (int i = 0; i < 10_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+ stopGrid(1);
+
+ for (int i = 10_000; i < 20_000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+
+ int grpId = ((GridCacheProxyImpl<Object,
Object>)grid(0).cachex(DEFAULT_CACHE_NAME))
+ .context().group().groupId();
+
+ markWalRebalanceInapplicable(grid(0), grpId);
+ markWalRebalanceInapplicable(grid(2), grpId);
+
+ IgniteEx ignite1 = startGrid(1, DC2);
+ waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+ RebalanceAwareCommSPI spi = commSPI(ignite1);
+
+ assertTrue(spi.full);
+ assertFalse(spi.historical);
+
+ IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+ for (int i = 0; i < 20_000; i += 5000) {
+ Object val = cache1.get(i);
+ assertEquals(i, val);
+ }
+ }
+
+ /**
+ * Starts a 2-node MDC cluster (node0 in DC1, node1 in DC2) in ACTIVE state
+ * and creates a cache with the given number of backups.
+ *
+ * @param backups Number of backups for the cache.
+ * @return The created cache.
+ */
+ private IgniteCache<Object, Object> startTwoNodeCluster(int backups)
throws Exception {
+ pds = true;
+ startGrid(0, DC1);
+ startGrid(1, DC2);
+ grid(0).cluster().state(ClusterState.ACTIVE);
+
+ CacheConfiguration<Object, Object> cacheCfg = new
CacheConfiguration<>(DEFAULT_CACHE_NAME);
+ cacheCfg.setBackups(backups);
+ return grid(0).createCache(cacheCfg);
+ }
+
+ /**
+ * Starts a 3-node MDC cluster (node0 in DC1, node1 in DC2, node2 in DC1)
in
+ * ACTIVE state and creates a cache with the given number of backups.
+ *
+ * @param backups Number of backups for the cache.
+ * @return The created cache.
+ */
+ private IgniteCache<Object, Object> startThreeNodeCluster(int backups)
throws Exception {
+ pds = true;
+ startGrid(0, DC1);
+ startGrid(1, DC2);
+ startGrid(2, DC1);
+ grid(0).cluster().state(ClusterState.ACTIVE);
+
+ CacheConfiguration<Object, Object> cacheCfg = new
CacheConfiguration<>(DEFAULT_CACHE_NAME);
+ cacheCfg.setBackups(backups);
+ return grid(0).createCache(cacheCfg);
+ }
+
+ /** */
+ private void markWalRebalanceInapplicable(IgniteEx ignite, int grpId) {
+ IgniteCacheDatabaseSharedManager db =
ignite.context().cache().context().database();
+ db.lastCheckpointInapplicableForWalRebalance(grpId);
+ }
+
/** */
private void checkFullRebalance(boolean crossDC) throws Exception {
startGrid(0, DC1);
@@ -178,6 +740,12 @@ public class MultiDcRebalancingTest extends
GridCommonAbstractTest {
/** */
private boolean crossDcRebalance;
+ /** */
+ private boolean full;
+
+ /** */
+ private int totalDemandedParts;
+
/** {@inheritDoc} */
@Override public void sendMessage(
ClusterNode node,
@@ -191,6 +759,20 @@ public class MultiDcRebalancingTest extends
GridCommonAbstractTest {
historical |= demandMsg.partitions().hasHistorical();
+ IgniteDhtDemandedPartitionsMap parts = demandMsg.partitions();
+
+ full |= parts.hasFull();
+
+ if (parts.hasHistorical()) {
+ Set<Integer> histSet = parts.historicalSet();
+ totalDemandedParts += histSet.size();
+ }
+
+ if (parts.hasFull()) {
+ Set<Integer> fullSet = parts.fullSet();
+ totalDemandedParts += fullSet.size();
+ }
+
if
(!ignite.cluster().localNode().dataCenterId().equals(node.dataCenterId()))
crossDcRebalance = true;
}
@@ -199,3 +781,4 @@ public class MultiDcRebalancingTest extends
GridCommonAbstractTest {
}
}
}
+
diff --git
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcSplitBrainRebalanceTest.java
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcSplitBrainRebalanceTest.java
new file mode 100644
index 00000000000..f56fb9d5488
--- /dev/null
+++
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcSplitBrainRebalanceTest.java
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.distributed.rebalancing;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCluster;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cluster.ClusterGroup;
+import org.apache.ignite.cluster.ClusterNode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.TopologyValidator;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.TestRecordingCommunicationSpi;
+import org.apache.ignite.internal.managers.communication.GridIoMessage;
+import
org.apache.ignite.internal.processors.cache.distributed.dht.IgniteCacheTopologySplitAbstractTest;
+import
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemandMessage;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.lang.IgniteInClosure;
+import org.apache.ignite.plugin.extensions.communication.Message;
+import org.apache.ignite.spi.IgniteSpiException;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.GridTestUtils;
+import org.junit.Test;
+
+import static org.apache.ignite.IgniteSystemProperties.IGNITE_DATA_CENTER_ID;
+
+/**
+ * Tests MDC disaster-recovery behavior under a LIVE network partition: the
two data
+ * centers stay up but lose sight of each other, and a topology validator
makes the
+ * isolated minority data center read-only while the majority keeps serving
writes.
+ *
+ * <p>Unlike {@link MultiDcRebalancingTest} (where a DC node is stopped and
writes nothing
+ * while away), here both halves stay alive during the partition. Segmentation
is achieved
+ * by faking discovery socket timeouts across the DC boundary (each half
detects the other
+ * as failed) plus blocking cross-DC communication messages, reusing
+ * {@link IgniteCacheTopologySplitAbstractTest}.
+ */
+public class MultiDcSplitBrainRebalanceTest extends
IgniteCacheTopologySplitAbstractTest {
+ /** */
+ private static final String DC1 = "DC1";
+
+ /** */
+ private static final String DC2 = "DC2";
+
+ /** */
+ private static final int SEG_DC1 = 0;
+
+ /** */
+ private static final int SEG_DC2 = 1;
+
+ /** Per-DC node layout: {0,1,2} -> DC1 (majority), {3} -> DC2 (minority).
*/
+ private static final int[] NODE_DC = {0, 0, 0, 1};
+
+ /** Base discovery port; each node takes base + idx. */
+ private static final int DISCO_PORT_BASE = TcpDiscoverySpi.DFLT_PORT;
+
+ /** discovery port -> segment (0 = DC1, 1 = DC2). Filled on node start. */
+ private final Map<Integer, Integer> portToSeg = new ConcurrentHashMap<>();
+
+ /** */
+ private boolean pds;
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String gridName)
throws Exception {
+ IgniteConfiguration cfg = super.getConfiguration(gridName);
+ int idx = Integer.parseInt(gridName.replaceAll("\\D+", ""));
+ cfg.setConsistentId(gridName);
+
+ TcpDiscoverySpi discoSpi = (TcpDiscoverySpi)cfg.getDiscoverySpi();
+ discoSpi.setLocalPort(DISCO_PORT_BASE + idx);
+ portToSeg.put(DISCO_PORT_BASE + idx, NODE_DC[idx] == 0 ? SEG_DC1 :
SEG_DC2);
+
+ List<String> discoAddrs = new ArrayList<>();
+ for (int i = 0; i < NODE_DC.length; i++) {
+ String addr = "127.0.0.1:" + (DISCO_PORT_BASE + i);
+ discoAddrs.add(addr);
+ }
+ TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
+ ipFinder.setShared(true);
+ ipFinder.setAddresses(discoAddrs);
+ discoSpi.setIpFinder(ipFinder);
+
+ cfg.setUserAttributes(F.asMap(IGNITE_DATA_CENTER_ID, NODE_DC[idx] == 0
? DC1 : DC2));
+
+ DataRegionConfiguration dfltRgn = new DataRegionConfiguration();
+ dfltRgn.setPersistenceEnabled(pds);
+ DataStorageConfiguration dsCfg = new DataStorageConfiguration();
+ dsCfg.setDefaultDataRegionConfiguration(dfltRgn);
+ cfg.setDataStorageConfiguration(dsCfg);
+ cfg.setCommunicationSpi(new DemandRecordingCommSpi());
+
+ return cfg;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int segment(ClusterNode node) {
+ return DC2.equals(node.dataCenterId()) ? SEG_DC2 : SEG_DC1;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected boolean isBlocked(int locPort, int rmtPort) {
+ Integer locSeg = portToSeg.get(locPort);
+ Integer rmtSeg = portToSeg.get(rmtPort);
+ if (locSeg == null || rmtSeg == null)
+ return false;
+ return !locSeg.equals(rmtSeg);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ super.beforeTest();
+ portToSeg.clear();
+ cleanPersistenceDir();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+ stopAllGrids();
+ cleanPersistenceDir();
+ }
+
+ /**
+ * Disaster recovery: a network partition between the data centers makes
the minority
+ * DC read-only while the majority DC keeps serving writes. The cache
carries a
+ * topology validator that keeps it operative only while a node of the
main DC (DC1)
+ * is visible in the topology; the isolated minority (DC2) no longer sees
DC1, so the
+ * cache turns inoperative there and its writes are rejected, whereas the
majority
+ * (DC1) stays writable and preserves its data.
+ *
+ * <p>Scenario: 4 nodes, DC1={node0,node1,node2} (majority, main DC),
DC2={node3}
+ * (minority). After 1000 base entries the network is split; DC1 keeps
writing, DC2
+ * writes are rejected.
+ */
+ @Test
+ public void testNetworkPartitionMinorityBecomesReadOnly() throws Exception
{
+ IgniteCache<Object, Object> cache0 = startSplitBrainCluster(new
MainDcOnlyTopologyValidator());
+
+ splitAndWait();
+
+ cache0.put(1000, 1000);
+ assertEquals(1000, cache0.get(1000));
+
+ assertFalse("Minority DC2 must reject writes while partitioned",
+ tryPut(grid(3), 2000, 2000));
+
+ unsplit();
+
+ logRebalanceAfterHeal("MainDcOnly");
+
+ for (int i = 0; i <= 1000; i += 500) {
+ Object v = cache0.get(i);
+ assertEquals(i, v);
+ }
+ }
+
+ /**
+ * Starts a 4-node MDC cluster (node0,1,2 in DC1, node3 in DC2) in ACTIVE
state with
+ * baseline auto-adjustment disabled, creates a cache with backups=1 and
the given
+ * topology validator (or none), writes 1000 base entries, and forces a
checkpoint.
+ *
+ * @param validator Topology validator to apply, or {@code null} for no
validator.
+ * @return The cache on node0.
+ */
+ private IgniteCache<Object, Object>
startSplitBrainCluster(TopologyValidator validator) throws Exception {
+ pds = true;
+
+ startGrid(0);
+ startGrid(3);
+ grid(0).cluster().state(ClusterState.ACTIVE);
+ grid(0).cluster().baselineAutoAdjustEnabled(false);
+
+ CacheConfiguration<Object, Object> cacheCfg = new
CacheConfiguration<>(DEFAULT_CACHE_NAME);
+ cacheCfg.setBackups(1);
+ if (validator != null)
+ cacheCfg.setTopologyValidator(validator);
+ IgniteCache<Object, Object> cache = grid(0).createCache(cacheCfg);
+
+ startGrid(1);
+ startGrid(2);
+ awaitServers(4, grid(0));
+
+ for (int i = 0; i < 1000; i++)
+ cache.put(i, i);
+
+ forceCheckpoint();
+
+ return cache;
+ }
+
+ /** */
+ private boolean tryPut(IgniteEx ignite, int key, int val) {
+ try {
+ IgniteCache<Object, Object> cache =
ignite.cache(DEFAULT_CACHE_NAME);
+ cache.put(key, val);
+
+ return true;
+ }
+ catch (Exception e) {
+ log.info(">>> Put rejected [node=" + ignite.name() + ", err=" +
e.getMessage() + "]");
+
+ return false;
+ }
+ }
+
+ /**
+ * After unsplit, logs the actual state and the rebalance demand counts
per node.
+ *
+ * <p>Documented limitation: {@code splitAndWait}/{@code unsplit} cannot
merge two
+ * half-rings back — once a half detects the loss of the other and
processes
+ * {@code EVT_NODE_FAILED}, it stops reconnecting to those nodes. No new
topology
+ * version is created across the halves, so no cross-DC rebalance demand
is sent
+ * ({@code rebalanceMsgCnt == 0}). True split-brain merge with
state-synchronization
+ * and rebalance is covered at the ducktape level (netem + control
utility).
+ */
+ private void logRebalanceAfterHeal(String testLabel) {
+ for (Ignite ig : G.allGrids()) {
+ IgniteConfiguration cfg = ig.configuration();
+ DemandRecordingCommSpi spi =
(DemandRecordingCommSpi)cfg.getCommunicationSpi();
+
+ IgniteCluster cluster = ig.cluster();
+ ClusterGroup srvGrp = cluster.forServers();
+ Collection<? extends ClusterNode> srvNodes = srvGrp.nodes();
+ int srvCnt = srvNodes.size();
+
+ log.info(">>> [" + testLabel + "] node=" + ig.name()
+ + ", servers_in_topo=" + srvCnt
+ + ", rebalanceMsgCnt=" + spi.rebalanceMsgCnt);
+ }
+ }
+
+ /**
+ * Waits until the given node's cluster topology includes the expected
number of
+ * server nodes. Returns as soon as the condition holds (30s safety cap).
+ */
+ private void awaitServers(int expected, IgniteEx ignite) throws Exception {
+ IgniteCluster cluster = ignite.cluster();
+ boolean done = GridTestUtils.waitForCondition(() -> {
+ ClusterGroup srvGrp = cluster.forServers();
+ Collection<? extends ClusterNode> srvNodes = srvGrp.nodes();
+ return srvNodes.size() == expected;
+ }, 30_000);
+ assertTrue("Topology did not reach " + expected + " servers on " +
ignite.name(), done);
+ }
+
+ /**
+ * Verifies that with a validator requiring both DCs, the partition makes
BOTH halves
+ * read-only: neither can see the other DC, so neither satisfies the
validator.
+ *
+ * <p>Scenario: 4 nodes, DC1={0,1,2}, DC2={3}. Validator returns false
unless at
+ * least one node from each DC is in the topology. After split, both sides
reject
+ * writes.
+ */
+ @Test
+ public void testNetworkPartitionBothHalvesReadOnly() throws Exception {
+ startSplitBrainCluster(new BothDcsTopologyValidator());
+
+ splitAndWait();
+
+ assertFalse("Majority DC1 must reject writes without DC2",
+ tryPut(grid(0), 1000, 1000));
+
+ assertFalse("Minority DC2 must reject writes without DC1",
+ tryPut(grid(3), 2000, 2000));
+
+ unsplit();
+
+ logRebalanceAfterHeal("BothDcsRequired");
+ }
+
+ /**
+ * Verifies that with a quorum validator (min server nodes), the majority
keeps
+ * writing while the minority loses quorum and becomes read-only.
+ *
+ * <p>Scenario: 4 nodes, DC1={0,1,2} (3 nodes >= threshold=3), DC2={3} (1
node < 3).
+ * After split, DC1 passes the quorum check, DC2 does not.
+ */
+ @Test
+ public void testNetworkPartitionMajorityKeepsQuorum() throws Exception {
+ IgniteCache<Object, Object> cache0 = startSplitBrainCluster(new
MinServerCountTopologyValidator(3));
+
+ splitAndWait();
+
+ cache0.put(1000, 1000);
+ assertEquals(1000, cache0.get(1000));
+
+ assertFalse("Minority DC2 must reject writes without quorum",
+ tryPut(grid(3), 2000, 2000));
+
+ unsplit();
+
+ logRebalanceAfterHeal("MinServerCount3");
+ }
+
+ /**
+ * Verifies that without a topology validator, BOTH halves stay writable
during the
+ * partition (uncontrolled split-brain). Each half elects its own
coordinator and
+ * accepts writes independently.
+ *
+ * <p>Scenario: 4 nodes, DC1={0,1,2}, DC2={3}. No validator set. After
split, both
+ * DC1 and DC2 write successfully.
+ */
+ @Test
+ public void testNetworkPartitionNoValidatorBothWritable() throws Exception
{
+ IgniteCache<Object, Object> cache0 = startSplitBrainCluster(null);
+
+ splitAndWait();
+
+ cache0.put(1000, 1000);
+ assertEquals(1000, cache0.get(1000));
+
+ IgniteCache<Object, Object> cache3 = grid(3).cache(DEFAULT_CACHE_NAME);
+
+ cache3.put(2000, 2000);
+ assertEquals(2000, cache3.get(2000));
+
+ unsplit();
+
+ logRebalanceAfterHeal("NoValidator");
+ }
+
+ /**
+ * Keeps the cache operative only while at least one server node of the
main data
+ * center (DC1) is present in the topology. When a partition isolates the
minority
+ * DC2 from DC1, the cache becomes inoperative on the minority and rejects
writes.
+ */
+ private static class MainDcOnlyTopologyValidator implements
TopologyValidator {
+ /** */
+ private static final long serialVersionUID = 0L;
+
+ /** {@inheritDoc} */
+ @Override public boolean validate(Collection<ClusterNode> nodes) {
+ for (ClusterNode node : nodes) {
+ if (!node.isClient() && DC1.equals(node.dataCenterId()))
+ return true;
+ }
+
+ return false;
+ }
+ }
+
+ /**
+ * Keeps the cache operative only while server nodes from BOTH DCs are
present.
+ * When any DC is isolated, the validator fails on both sides.
+ */
+ private static class BothDcsTopologyValidator implements TopologyValidator
{
+ /** */
+ private static final long serialVersionUID = 0L;
+
+ /** {@inheritDoc} */
+ @Override public boolean validate(Collection<ClusterNode> nodes) {
+ boolean hasDc1 = false;
+ boolean hasDc2 = false;
+
+ for (ClusterNode node : nodes) {
+ if (node.isClient())
+ continue;
+ String dc = node.dataCenterId();
+ if (DC1.equals(dc))
+ hasDc1 = true;
+ if (DC2.equals(dc))
+ hasDc2 = true;
+ }
+
+ return hasDc1 && hasDc2;
+ }
+ }
+
+ /**
+ * Keeps the cache operative only while at least {@code minCnt} server
nodes are
+ * present in the topology.
+ */
+ private static class MinServerCountTopologyValidator implements
TopologyValidator {
+ /** */
+ private static final long serialVersionUID = 0L;
+
+ /** */
+ private final int minCnt;
+
+ /** */
+ MinServerCountTopologyValidator(int minCnt) {
+ this.minCnt = minCnt;
+ }
+
+ /** {@inheritDoc} */
+ @Override public boolean validate(Collection<ClusterNode> nodes) {
+ int serverCnt = 0;
+
+ for (ClusterNode node : nodes) {
+ if (!node.isClient())
+ serverCnt++;
+ }
+
+ return serverCnt >= minCnt;
+ }
+ }
+
+ /**
+ * Communication SPI that records rebalance demand messages while
inheriting the
+ * message-blocking behavior used for segmentation.
+ */
+ private static class DemandRecordingCommSpi extends
TestRecordingCommunicationSpi {
+ /** */
+ private volatile int rebalanceMsgCnt;
+
+ /** {@inheritDoc} */
+ @Override public void sendMessage(
+ ClusterNode node,
+ Message msg,
+ IgniteInClosure<IgniteException> ackClosure
+ ) throws IgniteSpiException {
+ if (msg instanceof GridIoMessage
+ && ((GridIoMessage)msg).message() instanceof
GridDhtPartitionDemandMessage)
+ rebalanceMsgCnt++;
+ super.sendMessage(node, msg, ackClosure);
+ }
+ }
+}
+
diff --git
a/modules/core/src/test/java/org/apache/ignite/testsuites/IgniteCacheTestSuite19.java
b/modules/core/src/test/java/org/apache/ignite/testsuites/IgniteCacheTestSuite19.java
index 6d41d046f43..32d82bf131e 100644
---
a/modules/core/src/test/java/org/apache/ignite/testsuites/IgniteCacheTestSuite19.java
+++
b/modules/core/src/test/java/org/apache/ignite/testsuites/IgniteCacheTestSuite19.java
@@ -34,6 +34,7 @@ import
org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridC
import
org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest;
import
org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingUnmarshallingFailedSelfTest;
import
org.apache.ignite.internal.processors.cache.distributed.rebalancing.MultiDcRebalancingTest;
+import
org.apache.ignite.internal.processors.cache.distributed.rebalancing.MultiDcSplitBrainRebalanceTest;
import
org.apache.ignite.internal.processors.cache.persistence.CleanupRestoredCachesSlowTest;
import org.apache.ignite.testframework.GridTestUtils;
import org.apache.ignite.testframework.junits.DynamicSuite;
@@ -72,6 +73,7 @@ public class IgniteCacheTestSuite19 {
GridTestUtils.addTestIfNeeded(suite,
GridCacheRabalancingDelayedPartitionMapExchangeSelfTest.class, ignoredTests);
GridTestUtils.addTestIfNeeded(suite,
GridCacheRebalancingCancelTest.class, ignoredTests);
GridTestUtils.addTestIfNeeded(suite, MultiDcRebalancingTest.class,
ignoredTests);
+ GridTestUtils.addTestIfNeeded(suite,
MultiDcSplitBrainRebalanceTest.class, ignoredTests);
GridTestUtils.addTestIfNeeded(suite,
CacheStoreTxPutAllMultiNodeTest.class, ignoredTests);
GridTestUtils.addTestIfNeeded(suite,
CleanupRestoredCachesSlowTest.class, ignoredTests);
GridTestUtils.addTestIfNeeded(suite,
ClientCreateCacheGroupOnJoinNodeMapsTest.class, ignoredTests);
diff --git a/modules/ducktests/tests/ignitetest/services/mdc/mdc_cluster.py
b/modules/ducktests/tests/ignitetest/services/mdc/mdc_cluster.py
index a9ccbe4496d..b5b57e780a4 100644
--- a/modules/ducktests/tests/ignitetest/services/mdc/mdc_cluster.py
+++ b/modules/ducktests/tests/ignitetest/services/mdc/mdc_cluster.py
@@ -34,6 +34,7 @@ from ignitetest.services.network_group.configuration import
NetworkGroupStore, C
from ignitetest.services.network_group.manager import NetworkGroupManager
from ignitetest.services.utils.control_utility import ControlUtility
from ignitetest.services.utils.ignite_configuration import
IgniteConfiguration, TcpCommunicationSpi
+from ignitetest.services.utils.ignite_configuration.data_storage import
DataStorageConfiguration
from ignitetest.services.utils.ignite_configuration.discovery import
TcpDiscoverySpi, from_ignite_cluster, \
from_ignite_services
from ignitetest.services.utils.ssl.client_connector_configuration import
ClientConnectorConfiguration
@@ -101,9 +102,12 @@ class MdcCluster:
loaders_per_dc: Union[int, Dict[str, int]] = 0,
client_connector: bool = False,
network_timeout: int = 5_000,
- tcp_connect_timeout: int = 5_000):
+ tcp_connect_timeout: int = 5_000,
+ data_storage: DataStorageConfiguration = None,
+ jvm_opts: List[str] = None):
self.test_context = test.test_context
self.logger = test.logger
+ self._jvm_opts = jvm_opts or []
# A single discovery SPI (hence a single ip finder) shared by both
DCs' server
# services is what makes the two DCs form ONE cluster:
prepare_on_start()
@@ -115,7 +119,8 @@ class MdcCluster:
"version": IgniteVersion(ignite_version),
"discovery_spi": TcpDiscoverySpi(),
"network_timeout": network_timeout,
- "communication_spi":
TcpCommunicationSpi(connect_timeout=tcp_connect_timeout)
+ "communication_spi":
TcpCommunicationSpi(connect_timeout=tcp_connect_timeout),
+ "data_storage": data_storage
}
if client_connector:
@@ -126,7 +131,8 @@ class MdcCluster:
self.srv_per_dc = _per_dc(srv_per_dc)
self.servers: Dict[str, IgniteService] = {
- dc: IgniteService(self.test_context, self.ignite_config,
num_nodes=num, jvm_opts=dc_jvm_opts(dc),
+ dc: IgniteService(self.test_context, self.ignite_config,
num_nodes=num,
+ jvm_opts=dc_jvm_opts(dc) + self._jvm_opts,
startup_timeout_sec=IGNITE_STARTUP_TIMEOUT_SEC)
for dc, num in self.srv_per_dc.items() if num > 0}
@@ -162,9 +168,11 @@ class MdcCluster:
service.config =
service.config._replace(discovery_spi=discovery_spi)
def _app_service(self, dc: str) -> IgniteApplicationService:
- client_cfg = self.ignite_config._replace(client_mode=True,
discovery_spi=from_ignite_cluster(self.servers[dc]))
+ client_cfg = self.ignite_config._replace(
+ client_mode=True,
discovery_spi=from_ignite_cluster(self.servers[dc]),
+ data_storage=None)
- return IgniteApplicationService(self.test_context, client_cfg,
jvm_opts=dc_jvm_opts(dc))
+ return IgniteApplicationService(self.test_context, client_cfg,
jvm_opts=dc_jvm_opts(dc) + self._jvm_opts)
def register(self, dc: str, service):
"""
diff --git
a/modules/ducktests/tests/ignitetest/tests/mdc/mdc_historical_rebalance_test.py
b/modules/ducktests/tests/ignitetest/tests/mdc/mdc_historical_rebalance_test.py
new file mode 100644
index 00000000000..da3ac5c0f1a
--- /dev/null
+++
b/modules/ducktests/tests/ignitetest/tests/mdc/mdc_historical_rebalance_test.py
@@ -0,0 +1,142 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+MDC historical rebalance test: a returning data center must restore its
partitions
+from local disk (only the delta written during its absence), not receive a full
+rebalance from the surviving DC.
+"""
+import re
+from time import sleep
+
+from ignitetest.services.mdc.mdc_cluster import MdcCluster, cross_dc_network,
DC_1, DC_2
+from ignitetest.services.utils.ignite_aware import IgniteAwareService
+from ignitetest.services.utils.ignite_configuration.data_storage import
DataStorageConfiguration, \
+ DataRegionConfiguration
+from ignitetest.utils import cluster, ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH
+
+CACHE_NAME = "mdc-historical-rebalance"
+
+BACKUPS = 1
+
+BASE_ENTRIES = 1024
+DELTA_ENTRIES = 256
+
+JVM_OPTS = ["-Xmx384M", "-Xms384M", "-DIGNITE_PREFER_WAL_REBALANCE=true"]
+
+
+def _persistence_config() -> DataStorageConfiguration:
+ """
+ Data storage configuration with persistence enabled on a small default
data region.
+ Short checkpoint frequency ensures data reaches disk before DC failure.
+ """
+ return DataStorageConfiguration(
+ checkpoint_frequency=10,
+ default=DataRegionConfiguration(persistence_enabled=True,
+ initial_size=16 * 1024 * 1024,
+ max_size=64 * 1024 * 1024)
+ )
+
+
+def _rebalance_metric(svc, metric: str) -> int:
+ """
+ Extracts a numeric field from the last 'Completed (final) rebalancing' log
line
+ of any node in the service.
+ """
+ for node in svc.nodes:
+ cmd = f"grep 'Completed (final) rebalancing' {node.log_file} | tail -1"
+ output = IgniteAwareService.exec_command(node, cmd)
+
+ if output:
+ m = re.search(rf"{metric}=(\d+)", output)
+
+ if m:
+ return int(m.group(1))
+
+ raise AssertionError(f"Rebalance metric '{metric}' not found in {svc}
logs")
+
+
+class MdcHistoricalRebalanceTest(IgniteTest):
+ """
+ Verifies that a data center returning after a full stop (DC failure)
restores
+ its partitions from local disk and only receives the delta written during
its
+ absence, rather than a full rebalance from the surviving DC.
+ """
+ @cluster(num_nodes=4)
+ @ignite_versions(str(DEV_BRANCH))
+ def test_returning_dc_recovers_from_disk(self, ignite_version):
+ """
+ DC1 + DC2 with persistence enabled. Data is written, then DC2 goes down
+ (full stop, not a network partition). While DC2 is away, DC1 accepts
new
+ writes (the delta). DC2 returns with clean=False (disk data preserved)
and
+ must recover: all base data from disk, only the delta via rebalance.
+ """
+ mdc = MdcCluster(self, ignite_version, srv_per_dc=1, runners_per_dc=1,
+ data_storage=_persistence_config(),
+ jvm_opts=JVM_OPTS)
+
+ with cross_dc_network(self.logger, mdc):
+ mdc.start_servers()
+
+ # With persistence the cluster starts INACTIVE; activate it
explicitly.
+ mdc.control(DC_1).activate()
+
+ # Phase 1: populate base data across both DCs.
+ mdc.generate_data(DC_1, CACHE_NAME, 0, BASE_ENTRIES // 2,
backups=BACKUPS)
+ mdc.generate_data(DC_2, CACHE_NAME, BASE_ENTRIES // 2,
BASE_ENTRIES, backups=BACKUPS)
+
+ mdc.check_data(DC_1, CACHE_NAME, 0, BASE_ENTRIES)
+ mdc.check_data(DC_2, CACHE_NAME, 0, BASE_ENTRIES)
+
+ mdc.verify_cache_distribution(CACHE_NAME, copies_per_dc=1)
+
+ # Allow checkpoint to flush data to disk before the DC failure.
+ sleep(15)
+
+ # Phase 2: DC2 goes down (full stop, simulating DC failure).
+ mdc.servers[DC_2].stop()
+
+ # DC1 must remain healthy and accept writes.
+ mdc.check_put_admissibility(DC_1, CACHE_NAME, True)
+
+ # Phase 3: write the delta on the surviving DC1.
+ mdc.generate_data(DC_1, CACHE_NAME, BASE_ENTRIES, BASE_ENTRIES +
DELTA_ENTRIES,
+ backups=BACKUPS)
+
+ mdc.check_data(DC_1, CACHE_NAME, 0, BASE_ENTRIES + DELTA_ENTRIES)
+
+ # Phase 4: DC2 returns (disk data preserved, clean=False).
+ mdc.servers[DC_2].start(clean=False)
+ mdc.servers[DC_2].await_rebalance()
+
+ # Phase 5: verify historical rebalance actually restored data from
disk.
+ hist_entries = _rebalance_metric(mdc.servers[DC_2], "histEntries")
+
+ self.logger.info(f"Historical rebalance:
histEntries={hist_entries}")
+
+ assert hist_entries > 0, \
+ f"Expected data restored from local disk
[histEntries={hist_entries}]"
+
+ # Phase 6: verify full data integrity on both DCs.
+ mdc.check_data(DC_1, CACHE_NAME, 0, BASE_ENTRIES + DELTA_ENTRIES)
+ mdc.check_data(DC_2, CACHE_NAME, 0, BASE_ENTRIES + DELTA_ENTRIES)
+
+ mdc.verify_cache_distribution(CACHE_NAME, copies_per_dc=1)
+
+ mdc.verify_servers_log_clean()
+
+ mdc.stop_servers()