This is an automated email from the ASF dual-hosted git repository.

sergey-chugunov-1985 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ignite.git


The following commit(s) were added to refs/heads/master by this push:
     new 1493bbb3139 IGNITE-29177 Analyze and test the reliability of MDC 
historical rebalance mechanism (#13714)
1493bbb3139 is described below

commit 1493bbb3139d4955e5ebc00e8d9ff71d7d55df38
Author: Egor Baranov <[email protected]>
AuthorDate: Tue Oct 27 15:44:49 2026 +0300

    IGNITE-29177 Analyze and test the reliability of MDC historical rebalance 
mechanism (#13714)
---
 docs/_docs/clustering/multi-data-center.adoc       |   1 +
 docs/_docs/data-rebalancing.adoc                   | 119 +++++
 .../rebalancing/MultiDcRebalancingTest.java        | 587 ++++++++++++++++++++-
 .../MultiDcSplitBrainRebalanceTest.java            | 446 ++++++++++++++++
 .../ignite/testsuites/IgniteCacheTestSuite19.java  |   2 +
 .../tests/ignitetest/services/mdc/mdc_cluster.py   |  18 +-
 .../tests/mdc/mdc_historical_rebalance_test.py     | 142 +++++
 7 files changed, 1308 insertions(+), 7 deletions(-)

diff --git a/docs/_docs/clustering/multi-data-center.adoc 
b/docs/_docs/clustering/multi-data-center.adoc
index a5a37e25b1f..1eb10bd8f51 100644
--- a/docs/_docs/clustering/multi-data-center.adoc
+++ b/docs/_docs/clustering/multi-data-center.adoc
@@ -246,6 +246,7 @@ During rebalancing, Ignite prefers nodes from the same data 
center when such sup
 If the required partition data cannot be obtained from the same data center, 
Ignite can rebalance from a
 remote data center.
 This behavior applies to both full and historical rebalance.
+See link:data-rebalancing#rebalance-strategy[Rebalance Strategy] for the 
conditions that determine which strategy is used.
 
 == Monitoring
 
diff --git a/docs/_docs/data-rebalancing.adoc b/docs/_docs/data-rebalancing.adoc
index 2a603f8f224..49a43ca76e7 100644
--- a/docs/_docs/data-rebalancing.adoc
+++ b/docs/_docs/data-rebalancing.adoc
@@ -28,6 +28,83 @@ In pure in-memory clusters, the default behavior is to start 
rebalancing immedia
 In clusters with persistence, the baseline topology has to be changed manually 
(default behavior), or can be changed automatically when 
link:clustering/baseline-topology#baseline-topology-autoadjustment[automatic 
baseline adjustment] is enabled.
 ====
 
+== Rebalancing Strategy [[rebalance-strategy]]
+
+When a node joins or rejoins the cluster, each partition is restored using one 
of two strategies:
+
+- *Full rebalance* — the node downloads the entire partition content from 
another server node that holds a copy of the partition.
+- *Historical rebalance* — the node restores its own data from the local disk 
and downloads only the delta (WAL entries written while it was absent) from the 
source node that holds a copy of the partition.
+
+The strategy is selected per partition based on the following rules:
+
+. A full rebalance is used if the node has no local data for the partition 
(update counter is zero).
++
+This applies both to a *new node joining for the first time* (it has no data 
on disk at all) and to a node that was *evicted from the baseline* and rejoined 
(its local partition data was invalidated). In both cases the full partition 
content must be downloaded from another node.
+
+. A full rebalance is used if the delta exceeds the total number of entries in 
the partition — transferring the full content is more efficient in that case.
++
+Example: a partition contains 100 entries. While the node was absent, 500 
writes hit this partition. Downloading 500 delta entries is more expensive than 
just downloading all 100 entries → full rebalance.
+
+. A full rebalance is used if the partition size is below the 
`historical.rebalance.threshold` (DMS property, default 500 entries), unless 
the `IGNITE_PREFER_WAL_REBALANCE` system property is set to `true`.
++
+Example: a partition contains 10 entries. While the node was absent, 3 writes 
hit this partition. The delta (3) is smaller than the partition (10), but 10 < 
500 — the partition is too small to justify WAL replay overhead → full 
rebalance. With `IGNITE_PREFER_WAL_REBALANCE=true` this threshold check is 
bypassed, and a historical rebalance is used regardless of the partition size 
(provided the other rules above are not triggered).
+
+. A historical rebalance is used when all of the above conditions are not met: 
the node has local data, the delta is smaller than the partition, and the 
partition exceeds the threshold.
++
+Example: a partition contains 10,000 entries. While the node was absent, 30 
writes hit this partition. Downloading 30 delta entries is far cheaper than all 
10,000 → historical rebalance.
+
+[NOTE]
+====
+If both nodes report the same update counter for a partition, no rebalance is 
performed for that partition.
+
+A node reports counters only for partitions it has written to. If the dataset 
is small relative to the partition count, most partitions will have a zero 
counter and will not be included in the rebalance.
+====
+
+[NOTE]
+====
+*Historical rebalance requires the delta to be present in the WAL of the 
source node.* If the WAL segments that contain the delta were already archived 
or deleted by the time the node rejoins — for example, the node was absent for 
a long time, or WAL history and archiving are configured with a small retention 
— the source node cannot supply the delta. In that case Ignite does not lose 
data: the supplier detects the failure to iterate over the WAL, marks the last 
checkpoint as not applica [...]
+====
+
+[CAUTION]
+====
+The `rebalanceDelay` (deprecated) setting also prevents a historical 
rebalance: when a delay is configured, the rebalance is scheduled outside of 
the partition map exchange context, and the historical path (which is tied to 
that context) is never taken — every partition is restored with a full 
rebalance regardless of the delta size and regardless of 
`IGNITE_PREFER_WAL_REBALANCE`.
+====
+
+In a link:clustering/multi-data-center[multi-data-center] setup, the node 
requesting the data prefers a source from the same data center that holds a 
copy of the partition. If no such node is available, it falls back to a node 
from a remote data center. This behavior applies to both full and historical 
rebalance.
+
+== Historical Rebalance: Internal Mechanism
+
+The following section describes the internal flow of a historical rebalance. 
This information is useful for understanding why the rebalance is chosen as 
historical or full, and what happens when the historical path fails.
+
+=== Source Selection
+
+When the node requesting data (the *demander*) needs a partition, it selects a 
source node from the available nodes that hold a copy of that partition. In a 
link:clustering/multi-data-center[multi-data-center] setup, a node from the 
same data center is preferred. The selection happens during the partition map 
exchange phase, and the source nodes can be changed during the rebalance if one 
of them becomes inapplicable.
+
+=== The Demand-Supply Protocol
+
+1. The demander sends a *demand* message to the chosen source node (supplier), 
specifying which partitions it needs and whether it expects a full or 
historical transfer.
+2. The supplier iterates over the partition data (or WAL entries, for 
historical) and sends *supply* messages back to the demander, batch by batch.
+3. The demander sends the next demand when it is ready for more data. This 
repeats until all partitions are either fully transferred or marked as *missed*.
+
+=== Fallback from Historical to Full
+
+If the supplier fails to read the WAL (for example, the WAL segments were 
archived or deleted), the historical rebalance for that partition is 
interrupted. The following happens:
+
+. The supplier's historical iterator throws an exception.
+. The supplier marks the last checkpoint as *not applicable for WAL rebalance* 
(so that subsequent checkpoints will not try to serve WAL for this partition).
+. The supplier marks all remaining partitions as *missed* in the supply 
message.
+. The demander, upon receiving the *missed* marking for a partition that was 
originally requested as historical, excludes the source node from historical 
rebalancing and re-requests the partition as *full* from another available 
source node.
+
+This mechanism ensures that data is never lost: the node will always receive 
either the historical delta or the full partition content.
+
+=== Protection Against Data Loss on Full Rebalance
+
+When a partition is requested as *full* (not historical), the demander 
additionally checks whether the source node's copy of the partition is up to 
date. A source node that is known to be behind (its update counter for the 
partition is lower than the demander's local counter) is *excluded from full 
rebalancing* for that partition — accepting a full partition from such a node 
would overwrite newer local data with older data. The demander then selects a 
different source or waits for the pa [...]
+
+=== Message Ordering (Striped Executor)
+
+Historical rebalance messages are processed on a *striped* executor: all 
supply messages from the same source node are handled sequentially in the order 
they were received. This preserves the ordering of WAL entries, which is 
required for correct replay. Full rebalance messages are processed on a regular 
(non-ordered) executor because individual partition transfers are independent.
+
 == Configuring Rebalancing Mode
 
 Ignite supports both synchronous and asynchronous rebalancing.
@@ -147,6 +224,48 @@ The following table lists the properties of 
`IgniteConfiguration` related to reb
 |===
 
 
+== Rebalance Risks
+
+A rebalance transfers a significant volume of data between nodes. Depending on 
the cluster size and the data volume, it can have a measurable impact on 
availability and performance. The following risks should be considered before 
scaling a cluster, particularly across data centers.
+
+=== Bandwidth saturation
+
+During a rebalance, one or more source nodes simultaneously transfer partition 
data to the target node. In a 
link:clustering/multi-data-center[multi-data-center] setup this traffic 
traverses the inter-DC (WAN) link, which is typically the narrowest part of the 
topology. A rebalance can saturate this link, degrading or starving all other 
traffic between the data centers (client requests, replication, monitoring).
+
+The impact is proportional to the number of source nodes and the size of the 
transferred partitions, and inversely proportional to the available link 
bandwidth. It is most severe for small clusters (few nodes, each holding a 
large share of the data) over limited WAN links.
+
+=== Full cluster stall (synchronous mode)
+
+In the `SYNC` rebalancing mode, all public cache operations are blocked until 
the rebalance completes. For a large cache over a slow or high-latency link, 
this can result in a prolonged unavailability of the entire cache.
+
+=== Target node disk I/O
+
+The target node must persist all received partitions to disk. This produces a 
write I/O spike that can saturate the node's storage. If the node serves live 
traffic concurrently, request latency will increase.
+
+=== Source node contention
+
+Each source node transfers its share of the partitions while continuing to 
serve requests. This reduces the effective request throughput of the existing 
nodes for the duration of the rebalance. The effect is more pronounced in 
smaller clusters, where each node holds a larger share of the data.
+
+=== Rebalance timeout
+
+Each rebalance message is subject to `rebalanceTimeout` (default 10 seconds). 
Over a slow or high-latency link, transferring a single batch may exceed this 
value, causing the rebalance to fail and the target node to remain only 
partially populated.
+
+== Recommendations for Cluster Administrators
+
+The following recommendations apply when scaling a cluster, especially when 
adding nodes to a remote or a new data center.
+
+. *Use asynchronous rebalancing.* Configure the cache with the `ASYNC` 
rebalancing mode so that the cache remains available for operations while the 
data is transferred in the background. The `SYNC` mode should be avoided for 
large caches or cross-DC transfers.
+. *Throttle the transfer.* Set `rebalanceThrottle` (a delay between batches) 
and, if necessary, reduce `rebalanceBatchSize`, to limit the bandwidth consumed 
by the rebalance and avoid saturating the inter-DC link.
+. *Schedule during low-load periods.* Start the scaling operation when the 
cluster is under minimal load, so that the rebalance traffic competes with as 
little production traffic as possible.
+. *Increase the timeout for slow links.* On high-latency or low-bandwidth 
links, increase `rebalanceTimeout` (and reduce `rebalanceBatchSize`) so that 
individual batches do not time out.
+. *Manage the baseline explicitly.* When the baseline topology is not 
auto-adjusted, confirm the new node is fully up and then commit the current 
cluster topology as the new baseline via 
`ignite.cluster().setBaselineTopology(ignite.cluster().topologyVersion())`, so 
that the rebalance starts at the intended moment.
+. *Monitor the rebalance.* Track the 
link:monitoring-metrics/new-metrics#caches[rebalancing metrics], the inter-DC 
link utilization, and the I/O load of the target node during the operation, and 
be prepared to throttle or abort if the impact exceeds expectations.
+
+[NOTE]
+====
+For a node joining for the first time (no local data), the rebalance is always 
a full transfer of its partitions, regardless of the 
`IGNITE_PREFER_WAL_REBALANCE` setting. The bandwidth, disk, and stall 
considerations above therefore apply in full.
+====
+
 == Monitoring Rebalancing Process
 
 You can monitor the link:monitoring-metrics/new-metrics#caches[rebalancing 
process for specific caches using JMX].
diff --git 
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java
 
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java
index 9f77dc49626..9834e386c5d 100644
--- 
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java
+++ 
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcRebalancingTest.java
@@ -17,6 +17,7 @@
 
 package org.apache.ignite.internal.processors.cache.distributed.rebalancing;
 
+import java.util.Set;
 import org.apache.ignite.Ignite;
 import org.apache.ignite.IgniteCache;
 import org.apache.ignite.IgniteException;
@@ -28,20 +29,27 @@ import 
org.apache.ignite.configuration.DataRegionConfiguration;
 import org.apache.ignite.configuration.DataStorageConfiguration;
 import org.apache.ignite.internal.IgniteEx;
 import org.apache.ignite.internal.managers.communication.GridIoMessage;
+import org.apache.ignite.internal.processors.cache.GridCacheProxyImpl;
 import 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemandMessage;
+import 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtDemandedPartitionsMap;
+import 
org.apache.ignite.internal.processors.cache.persistence.IgniteCacheDatabaseSharedManager;
 import org.apache.ignite.internal.util.typedef.F;
 import org.apache.ignite.lang.IgniteInClosure;
 import org.apache.ignite.plugin.extensions.communication.Message;
 import org.apache.ignite.spi.IgniteSpiException;
 import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi;
+import org.apache.ignite.testframework.GridTestUtils;
 import org.apache.ignite.testframework.junits.WithSystemProperty;
 import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
 import org.junit.Test;
 
 import static 
org.apache.ignite.IgniteSystemProperties.IGNITE_PREFER_WAL_REBALANCE;
 
-/** */
-@WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+/**
+ * Tests the MDC rebalance decision logic: when the engine chooses full, 
historical, or
+ * skips rebalance after a node rejoins the cluster. Live network split-brain 
is covered
+ * by {@link MultiDcSplitBrainRebalanceTest}.
+ */
 public class MultiDcRebalancingTest extends GridCommonAbstractTest {
     /** */
     private static final String DC1 = "DC1";
@@ -91,16 +99,570 @@ public class MultiDcRebalancingTest extends 
GridCommonAbstractTest {
 
     /** */
     @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
     public void testHistoricalRebalanceSameDC() throws Exception {
         checkHistoricalRebalance(false);
     }
 
     /** */
     @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
     public void testHistoricalRebalanceCrossDC() throws Exception {
         checkHistoricalRebalance(true);
     }
 
+    /**
+     * Verifies that no rebalance demand is sent when a node restarts without 
any
+     * writes having occurred during its absence.
+     *
+     * <p>Scenario: 2 nodes, 1000 entries, checkpoint. Node 1 stops, nothing is
+     * written, node 1 restarts. Both nodes have identical partition counters,
+     * the CRD sees only one value per partition, no node is "behind", no 
demand.
+     */
+    @Test
+    public void testNoRebalanceWhenNoDelta() throws Exception {
+        IgniteCache<Object, Object> cache = startTwoNodeCluster(1);
+
+        for (int i = 0; i < 1000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+        IgniteEx ignite1 = startGrid(1, DC2);
+        awaitPartitionMapExchange(true, true, null);
+        RebalanceAwareCommSPI spi = commSPI(ignite1);
+        assertFalse(spi.historical);
+        assertFalse(spi.full);
+    }
+
+    /**
+     * Verifies that partitions below {@code historical.rebalance.threshold} 
(default 500)
+     * are NOT eligible for historical rebalance. The node uses full rebalance 
instead.
+     *
+     * <p>Scenario: 2 nodes, only 100 entries (each partition has ~1 entry, 
well
+     * below threshold 500). Checkpoint. Node 1 stops, 5 more entries written
+     * (small delta), node 1 restarts. Despite the tiny delta, partition size 
< 500
+     * means it is not a historical candidate, so full rebalance is used.
+     */
+    @Test
+    public void testBelowThresholdForcesFull() throws Exception {
+        IgniteCache<Object, Object> cache = startTwoNodeCluster(1);
+
+        for (int i = 0; i < 100; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 100; i < 105; i++)
+            cache.put(i, i);
+
+        IgniteEx ignite1 = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        RebalanceAwareCommSPI spi = commSPI(ignite1);
+        assertTrue(spi.rebalanceMsgCnt > 0);
+        assertFalse(spi.historical);
+        assertTrue(spi.full);
+    }
+
+    /**
+     * Verifies that with a very small dataset, only non-empty partitions 
appear in
+     * the rebalance demand (the skipZeros mechanism).
+     *
+     * <p>Scenario: 2 nodes, only 10 entries. Since there are 512 partitions, 
at most
+     * ~10 have counter > 0. Node 1 stops, 5 more entries written, node 1 
restarts.
+     * The SingleMessage omits all empty partitions, the CRD only detects 
differences
+     * for the ~10 non-empty partitions, total demanded is well below 512.
+     */
+    @Test
+    public void testSkipZerosReducesRebalancedPartitions() throws Exception {
+        IgniteCache<Object, Object> cache = startTwoNodeCluster(1);
+
+        for (int i = 0; i < 10; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 10; i < 15; i++)
+            cache.put(i, i);
+
+        IgniteEx ignite1 = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        RebalanceAwareCommSPI spi = commSPI(ignite1);
+        assertTrue(spi.rebalanceMsgCnt > 0);
+        assertTrue("Expected < 512 but got " + spi.totalDemandedParts,
+            spi.totalDemandedParts < 512);
+    }
+
+    /**
+     * Verifies historical rebalance after a DC node comes back following an 
outage.
+     *
+     * <p>Scenario: 3 nodes — node0 (DC1), node1 (DC2), node2 (DC1). All in 
baseline,
+     * 10000 entries written. node1 (DC2) is stopped and writes nothing while 
down.
+     * The two DC1 nodes write 10000 more entries each. Then node1 restarts.
+     * Node1 must get cross-DC historical rebalance with the delta from both 
DC1 nodes.
+     */
+    @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+    public void testDcNodeDownHistoricalRebalance() throws Exception {
+        IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+        for (int i = 0; i < 10_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 10_000; i < 20_000; i++)
+            cache.put(i, i);
+
+        IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 20_000; i < 30_000; i++)
+            cache2.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1 = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        RebalanceAwareCommSPI spi = commSPI(ignite1);
+        assertTrue(spi.rebalanceMsgCnt > 0);
+        assertTrue(spi.historical);
+        assertTrue(spi.crossDcRebalance);
+    }
+
+    /**
+     * Verifies that after a DC node comes back from an outage, no data is 
lost. Data
+     * written by both surviving DC1 nodes while the node was down is fully 
accessible
+     * from the returning DC2 node after recovery.
+     *
+     * <p>Scenario: 3 nodes, node1 (DC2) down. DC1 nodes write 20000 new 
entries while
+     * it is down. After recovery, all 30000 entries are readable from the DC2 
node.
+     */
+    @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+    public void testDcNodeDownNoDataLoss() throws Exception {
+        IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+        for (int i = 0; i < 10_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 10_000; i < 20_000; i++)
+            cache.put(i, i);
+
+        IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 20_000; i < 30_000; i++)
+            cache2.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1 = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 0; i < 30_000; i += 3000) {
+            Object val = cache1.get(i);
+            assertEquals(i, val);
+        }
+    }
+
+    /**
+     * Verifies that after a DC node comes back from an outage, if partition 
size is
+     * below {@code historical.rebalance.threshold} (default 500), full 
rebalance is
+     * used (no PREFER_WAL_REBALANCE set).
+     *
+     * <p>Scenario: 3 nodes, 100 entries (below threshold). node1 (DC2) down, 
DC1
+     * writes 200 more entries. After recovery, full rebalance is expected.
+     */
+    @Test
+    public void testDcNodeDownBelowThresholdFullRebalance() throws Exception {
+        IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+        for (int i = 0; i < 100; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 100; i < 200; i++)
+            cache.put(i, i);
+
+        IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 200; i < 300; i++)
+            cache2.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1 = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        RebalanceAwareCommSPI spi = commSPI(ignite1);
+        assertTrue(spi.rebalanceMsgCnt > 0);
+        assertFalse(spi.historical);
+        assertTrue(spi.full);
+        assertTrue(spi.crossDcRebalance);
+    }
+
+    /**
+     * Verifies that after a DC node comes back from an outage, if the delta 
is larger
+     * than the partition size (and no PREFER_WAL_REBALANCE), full rebalance 
is used —
+     * transferring the full partition is more efficient than replaying the 
delta.
+     *
+     * <p>Scenario: 3 nodes, 1000 base entries. node1 (DC2) down. The DC1 
nodes each
+     * write 20000 entries (delta >> partition size). After recovery, full 
rebalance
+     * is expected (not historical).
+     */
+    @Test
+    public void testDcNodeDownLargeDeltaFullRebalance() throws Exception {
+        IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+        for (int i = 0; i < 1000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 1000; i < 21_000; i++)
+            cache.put(i, i);
+
+        IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 21_000; i < 41_000; i++)
+            cache2.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1 = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        RebalanceAwareCommSPI spi = commSPI(ignite1);
+        assertTrue(spi.rebalanceMsgCnt > 0);
+        assertFalse(spi.historical);
+        assertTrue(spi.full);
+        assertTrue(spi.crossDcRebalance);
+    }
+
+    /**
+     * Verifies that if one of two available source nodes also goes down while 
the DC
+     * node is away, the surviving source node can still serve the historical 
rebalance.
+     *
+     * <p>Scenario: 3 nodes (node0 DC1, node1 DC2, node2 DC1). node1 (DC2) is 
stopped.
+     * Then node0 (one DC1 source) is ALSO stopped. Only node2 remains. After 
recovery,
+     * node1 must get historical rebalance from node2 alone.
+     */
+    @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+    public void testDcNodeDownOneSourceNodeSurvives() throws Exception {
+        IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+        for (int i = 0; i < 10_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+        stopGrid(0);
+        IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 10_000; i < 20_000; i++)
+            cache2.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1 = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        RebalanceAwareCommSPI spi = commSPI(ignite1);
+        assertTrue(spi.rebalanceMsgCnt > 0);
+        assertTrue(spi.historical);
+        assertTrue(spi.crossDcRebalance);
+        IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 0; i < 20_000; i += 5000) {
+            Object val = cache1.get(i);
+            assertEquals(i, val);
+        }
+    }
+
+    /**
+     * Verifies that concurrent writes on the source node during an in-progress
+     * historical rebalance do not corrupt data. After the rebalance completes,
+     * all entries (old + delta + concurrent) are accessible.
+     *
+     * <p>Scenario: 3 nodes. node1 (DC2) is stopped. DC1 writes 10000 entries 
(delta).
+     * node1 restarts (triggers rebalance). While the rebalance is in flight,
+     * DC1 writes another 5000 entries. After rebalance completes, verify all
+     * data is accessible.
+     */
+    @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+    public void testConcurrentWritesDuringRecovery() throws Exception {
+        IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+        for (int i = 0; i < 10_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 10_000; i < 20_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1 = startGrid(1, DC2);
+        IgniteCache<Object, Object> cache2 = grid(2).cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 20_000; i < 25_000; i++)
+            cache2.put(i, i);
+
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 0; i < 25_000; i += 5000) {
+            Object val = cache1.get(i);
+            assertEquals(i, val);
+        }
+    }
+
+    /**
+     * Verifies that after a DC node outage recovery, if a new node also joins
+     * (triggering affinity reassignment), the returning node gets a mixed 
demand:
+     * historical for its own partitions AND full for reassigned partitions.
+     *
+     * <p>Scenario: 2 nodes initially. node1 (DC2) is stopped, DC1 writes 
delta.
+     * Recovery: node1 restarts AND a new node (node2, DC1) joins 
simultaneously.
+     * The topology change triggers both historical rebalance (for DC2's old
+     * partitions) and full rebalance (for partitions reassigned to node2).
+     */
+    @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+    public void testAffinityReassignmentMixedRebalance() throws Exception {
+        IgniteCache<Object, Object> cache = startTwoNodeCluster(1);
+
+        for (int i = 0; i < 10_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 10_000; i < 20_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1 = startGrid(1, DC2);
+        IgniteEx ignite2 = startGrid(2, DC1);
+        resetBaselineTopology();
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        waitRebalanceFinished(ignite2, DEFAULT_CACHE_NAME);
+
+        RebalanceAwareCommSPI spi1 = commSPI(ignite1);
+        assertTrue(spi1.rebalanceMsgCnt > 0);
+        assertTrue(spi1.historical);
+
+        RebalanceAwareCommSPI spi2 = commSPI(ignite2);
+        assertTrue("New node should receive full rebalance", spi2.full);
+
+        IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 0; i < 20_000; i += 5000) {
+            Object val = cache1.get(i);
+            assertEquals(i, val);
+        }
+    }
+
+    /**
+     * Verifies that with {@code rebalanceDelay > 0} the rebalance always uses 
the full
+     * strategy.
+     *
+     * <p>Documented limitation (see data-rebalancing.adoc, "Rebalance 
Strategy"): when a
+     * rebalance delay is configured, the rebalance is scheduled outside of 
the partition
+     * map exchange context, and the historical path (which is tied to that 
context) is
+     * never taken — every partition is restored with a full rebalance 
regardless of the
+     * delta size and regardless of {@code IGNITE_PREFER_WAL_REBALANCE}.
+     *
+     * <p>Related ticket: IGNITE-14524.
+     */
+    @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+    public void testRebalanceDelayForcesFull() throws Exception {
+        pds = true;
+        IgniteEx ignite0 = startGrid(0, DC1);
+        startGrid(1, DC2);
+        startGrid(2, DC1);
+        ignite0.cluster().state(ClusterState.ACTIVE);
+        CacheConfiguration<Object, Object> cacheCfg = new 
CacheConfiguration<>(DEFAULT_CACHE_NAME);
+        cacheCfg.setBackups(2);
+        cacheCfg.setRebalanceDelay(3000);
+        IgniteCache<Object, Object> cache = ignite0.createCache(cacheCfg);
+
+        for (int i = 0; i < 10_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 10_000; i < 20_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1 = startGrid(1, DC2);
+        RebalanceAwareCommSPI spi = commSPI(ignite1);
+
+        boolean demandSent = GridTestUtils.waitForCondition(() -> 
spi.rebalanceMsgCnt > 0, 30_000);
+
+        assertTrue("No rebalance demand received within 30s", demandSent);
+
+        awaitPartitionMapExchange(true, true, null);
+        assertTrue(spi.full);
+        assertFalse(spi.historical);
+    }
+
+    /**
+     * Verifies that after MULTIPLE restarts (stop→write→start, repeated), the 
recovering
+     * node accumulates all data across all restart cycles.
+     *
+     * <p>Per IGNITE-17738: after a crash/restart, historical rebalance may 
not deliver
+     * updates between LWM and HWM. This test exercises two consecutive 
restart cycles
+     * and asserts full data integrity after each.
+     *
+     * <p>Scenario: 3 nodes. 10_000 base entries.
+     * Cycle 1: node1 down → node0 writes 10_000 → node1 restarts → verify 
20_000.
+     * Cycle 2: node1 down → node0 writes 10_000 → node1 restarts → verify 
30_000.
+     */
+    @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+    public void testMultipleRestartsNoDataLoss() throws Exception {
+        IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+        for (int i = 0; i < 10_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 10_000; i < 20_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1 = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+
+        IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 0; i < 20_000; i += 5000) {
+            Object val = cache1.get(i);
+            assertEquals(i, val);
+        }
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 20_000; i < 30_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        IgniteEx ignite1Restarted = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1Restarted, DEFAULT_CACHE_NAME);
+
+        IgniteCache<Object, Object> cache1After = 
ignite1Restarted.cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 0; i < 30_000; i += 5000) {
+            Object val = cache1After.get(i);
+            assertEquals(i, val);
+        }
+    }
+
+    /**
+     * Verifies that when historical (WAL) rebalance is impossible, the 
supplier falls
+     * back to a FULL rebalance and no data is lost.
+     *
+     * <p>Truncated WAL is simulated deterministically by marking the last 
checkpoint as
+     * inapplicable for WAL rebalance on the source nodes
+     * ({@code lastCheckpointInapplicableForWalRebalance}) — exactly the flag 
the supplier
+     * sets itself when a WAL segment turns out to be unreadable
+     * ({@code GridDhtPartitionSupplier} fallbackToFullRebalance, line 451). 
This is the
+     * production situation when WAL segments covering the delta were already 
deleted.
+     *
+     * <p>Assertion: despite {@code PREFER_WAL_REBALANCE=true} the recovering 
node gets a
+     * FULL rebalance (historical is impossible), and all entries are present 
afterwards.
+     */
+    @Test
+    @WithSystemProperty(key = IGNITE_PREFER_WAL_REBALANCE, value = "true")
+    public void testWalTruncatedForcesFull() throws Exception {
+        IgniteCache<Object, Object> cache = startThreeNodeCluster(2);
+
+        for (int i = 0; i < 10_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+        stopGrid(1);
+
+        for (int i = 10_000; i < 20_000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+
+        int grpId = ((GridCacheProxyImpl<Object, 
Object>)grid(0).cachex(DEFAULT_CACHE_NAME))
+            .context().group().groupId();
+
+        markWalRebalanceInapplicable(grid(0), grpId);
+        markWalRebalanceInapplicable(grid(2), grpId);
+
+        IgniteEx ignite1 = startGrid(1, DC2);
+        waitRebalanceFinished(ignite1, DEFAULT_CACHE_NAME);
+        RebalanceAwareCommSPI spi = commSPI(ignite1);
+
+        assertTrue(spi.full);
+        assertFalse(spi.historical);
+
+        IgniteCache<Object, Object> cache1 = ignite1.cache(DEFAULT_CACHE_NAME);
+
+        for (int i = 0; i < 20_000; i += 5000) {
+            Object val = cache1.get(i);
+            assertEquals(i, val);
+        }
+    }
+
+    /**
+     * Starts a 2-node MDC cluster (node0 in DC1, node1 in DC2) in ACTIVE state
+     * and creates a cache with the given number of backups.
+     *
+     * @param backups Number of backups for the cache.
+     * @return The created cache.
+     */
+    private IgniteCache<Object, Object> startTwoNodeCluster(int backups) 
throws Exception {
+        pds = true;
+        startGrid(0, DC1);
+        startGrid(1, DC2);
+        grid(0).cluster().state(ClusterState.ACTIVE);
+
+        CacheConfiguration<Object, Object> cacheCfg = new 
CacheConfiguration<>(DEFAULT_CACHE_NAME);
+        cacheCfg.setBackups(backups);
+        return grid(0).createCache(cacheCfg);
+    }
+
+    /**
+     * Starts a 3-node MDC cluster (node0 in DC1, node1 in DC2, node2 in DC1) 
in
+     * ACTIVE state and creates a cache with the given number of backups.
+     *
+     * @param backups Number of backups for the cache.
+     * @return The created cache.
+     */
+    private IgniteCache<Object, Object> startThreeNodeCluster(int backups) 
throws Exception {
+        pds = true;
+        startGrid(0, DC1);
+        startGrid(1, DC2);
+        startGrid(2, DC1);
+        grid(0).cluster().state(ClusterState.ACTIVE);
+
+        CacheConfiguration<Object, Object> cacheCfg = new 
CacheConfiguration<>(DEFAULT_CACHE_NAME);
+        cacheCfg.setBackups(backups);
+        return grid(0).createCache(cacheCfg);
+    }
+
+    /** */
+    private void markWalRebalanceInapplicable(IgniteEx ignite, int grpId) {
+        IgniteCacheDatabaseSharedManager db = 
ignite.context().cache().context().database();
+        db.lastCheckpointInapplicableForWalRebalance(grpId);
+    }
+
     /** */
     private void checkFullRebalance(boolean crossDC) throws Exception {
         startGrid(0, DC1);
@@ -178,6 +740,12 @@ public class MultiDcRebalancingTest extends 
GridCommonAbstractTest {
         /** */
         private boolean crossDcRebalance;
 
+        /** */
+        private boolean full;
+
+        /** */
+        private int totalDemandedParts;
+
         /** {@inheritDoc} */
         @Override public void sendMessage(
             ClusterNode node,
@@ -191,6 +759,20 @@ public class MultiDcRebalancingTest extends 
GridCommonAbstractTest {
 
                 historical |= demandMsg.partitions().hasHistorical();
 
+                IgniteDhtDemandedPartitionsMap parts = demandMsg.partitions();
+
+                full |= parts.hasFull();
+
+                if (parts.hasHistorical()) {
+                    Set<Integer> histSet = parts.historicalSet();
+                    totalDemandedParts += histSet.size();
+                }
+
+                if (parts.hasFull()) {
+                    Set<Integer> fullSet = parts.fullSet();
+                    totalDemandedParts += fullSet.size();
+                }
+
                 if 
(!ignite.cluster().localNode().dataCenterId().equals(node.dataCenterId()))
                     crossDcRebalance = true;
             }
@@ -199,3 +781,4 @@ public class MultiDcRebalancingTest extends 
GridCommonAbstractTest {
         }
     }
 }
+
diff --git 
a/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcSplitBrainRebalanceTest.java
 
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcSplitBrainRebalanceTest.java
new file mode 100644
index 00000000000..f56fb9d5488
--- /dev/null
+++ 
b/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/distributed/rebalancing/MultiDcSplitBrainRebalanceTest.java
@@ -0,0 +1,446 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.distributed.rebalancing;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCluster;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cluster.ClusterGroup;
+import org.apache.ignite.cluster.ClusterNode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.TopologyValidator;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.TestRecordingCommunicationSpi;
+import org.apache.ignite.internal.managers.communication.GridIoMessage;
+import 
org.apache.ignite.internal.processors.cache.distributed.dht.IgniteCacheTopologySplitAbstractTest;
+import 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionDemandMessage;
+import org.apache.ignite.internal.util.typedef.F;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.lang.IgniteInClosure;
+import org.apache.ignite.plugin.extensions.communication.Message;
+import org.apache.ignite.spi.IgniteSpiException;
+import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
+import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
+import org.apache.ignite.testframework.GridTestUtils;
+import org.junit.Test;
+
+import static org.apache.ignite.IgniteSystemProperties.IGNITE_DATA_CENTER_ID;
+
+/**
+ * Tests MDC disaster-recovery behavior under a LIVE network partition: the 
two data
+ * centers stay up but lose sight of each other, and a topology validator 
makes the
+ * isolated minority data center read-only while the majority keeps serving 
writes.
+ *
+ * <p>Unlike {@link MultiDcRebalancingTest} (where a DC node is stopped and 
writes nothing
+ * while away), here both halves stay alive during the partition. Segmentation 
is achieved
+ * by faking discovery socket timeouts across the DC boundary (each half 
detects the other
+ * as failed) plus blocking cross-DC communication messages, reusing
+ * {@link IgniteCacheTopologySplitAbstractTest}.
+ */
+public class MultiDcSplitBrainRebalanceTest extends 
IgniteCacheTopologySplitAbstractTest {
+    /** */
+    private static final String DC1 = "DC1";
+
+    /** */
+    private static final String DC2 = "DC2";
+
+    /** */
+    private static final int SEG_DC1 = 0;
+
+    /** */
+    private static final int SEG_DC2 = 1;
+
+    /** Per-DC node layout: {0,1,2} -> DC1 (majority), {3} -> DC2 (minority). 
*/
+    private static final int[] NODE_DC = {0, 0, 0, 1};
+
+    /** Base discovery port; each node takes base + idx. */
+    private static final int DISCO_PORT_BASE = TcpDiscoverySpi.DFLT_PORT;
+
+    /** discovery port -> segment (0 = DC1, 1 = DC2). Filled on node start. */
+    private final Map<Integer, Integer> portToSeg = new ConcurrentHashMap<>();
+
+    /** */
+    private boolean pds;
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String gridName) 
throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(gridName);
+        int idx = Integer.parseInt(gridName.replaceAll("\\D+", ""));
+        cfg.setConsistentId(gridName);
+
+        TcpDiscoverySpi discoSpi = (TcpDiscoverySpi)cfg.getDiscoverySpi();
+        discoSpi.setLocalPort(DISCO_PORT_BASE + idx);
+        portToSeg.put(DISCO_PORT_BASE + idx, NODE_DC[idx] == 0 ? SEG_DC1 : 
SEG_DC2);
+
+        List<String> discoAddrs = new ArrayList<>();
+        for (int i = 0; i < NODE_DC.length; i++) {
+            String addr = "127.0.0.1:" + (DISCO_PORT_BASE + i);
+            discoAddrs.add(addr);
+        }
+        TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
+        ipFinder.setShared(true);
+        ipFinder.setAddresses(discoAddrs);
+        discoSpi.setIpFinder(ipFinder);
+
+        cfg.setUserAttributes(F.asMap(IGNITE_DATA_CENTER_ID, NODE_DC[idx] == 0 
? DC1 : DC2));
+
+        DataRegionConfiguration dfltRgn = new DataRegionConfiguration();
+        dfltRgn.setPersistenceEnabled(pds);
+        DataStorageConfiguration dsCfg = new DataStorageConfiguration();
+        dsCfg.setDefaultDataRegionConfiguration(dfltRgn);
+        cfg.setDataStorageConfiguration(dsCfg);
+        cfg.setCommunicationSpi(new DemandRecordingCommSpi());
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected int segment(ClusterNode node) {
+        return DC2.equals(node.dataCenterId()) ? SEG_DC2 : SEG_DC1;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected boolean isBlocked(int locPort, int rmtPort) {
+        Integer locSeg = portToSeg.get(locPort);
+        Integer rmtSeg = portToSeg.get(rmtPort);
+        if (locSeg == null || rmtSeg == null)
+            return false;
+        return !locSeg.equals(rmtSeg);
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        super.beforeTest();
+        portToSeg.clear();
+        cleanPersistenceDir();
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void afterTest() throws Exception {
+        super.afterTest();
+        stopAllGrids();
+        cleanPersistenceDir();
+    }
+
+    /**
+     * Disaster recovery: a network partition between the data centers makes 
the minority
+     * DC read-only while the majority DC keeps serving writes. The cache 
carries a
+     * topology validator that keeps it operative only while a node of the 
main DC (DC1)
+     * is visible in the topology; the isolated minority (DC2) no longer sees 
DC1, so the
+     * cache turns inoperative there and its writes are rejected, whereas the 
majority
+     * (DC1) stays writable and preserves its data.
+     *
+     * <p>Scenario: 4 nodes, DC1={node0,node1,node2} (majority, main DC), 
DC2={node3}
+     * (minority). After 1000 base entries the network is split; DC1 keeps 
writing, DC2
+     * writes are rejected.
+     */
+    @Test
+    public void testNetworkPartitionMinorityBecomesReadOnly() throws Exception 
{
+        IgniteCache<Object, Object> cache0 = startSplitBrainCluster(new 
MainDcOnlyTopologyValidator());
+
+        splitAndWait();
+
+        cache0.put(1000, 1000);
+        assertEquals(1000, cache0.get(1000));
+
+        assertFalse("Minority DC2 must reject writes while partitioned",
+            tryPut(grid(3), 2000, 2000));
+
+        unsplit();
+
+        logRebalanceAfterHeal("MainDcOnly");
+
+        for (int i = 0; i <= 1000; i += 500) {
+            Object v = cache0.get(i);
+            assertEquals(i, v);
+        }
+    }
+
+    /**
+     * Starts a 4-node MDC cluster (node0,1,2 in DC1, node3 in DC2) in ACTIVE 
state with
+     * baseline auto-adjustment disabled, creates a cache with backups=1 and 
the given
+     * topology validator (or none), writes 1000 base entries, and forces a 
checkpoint.
+     *
+     * @param validator Topology validator to apply, or {@code null} for no 
validator.
+     * @return The cache on node0.
+     */
+    private IgniteCache<Object, Object> 
startSplitBrainCluster(TopologyValidator validator) throws Exception {
+        pds = true;
+
+        startGrid(0);
+        startGrid(3);
+        grid(0).cluster().state(ClusterState.ACTIVE);
+        grid(0).cluster().baselineAutoAdjustEnabled(false);
+
+        CacheConfiguration<Object, Object> cacheCfg = new 
CacheConfiguration<>(DEFAULT_CACHE_NAME);
+        cacheCfg.setBackups(1);
+        if (validator != null)
+            cacheCfg.setTopologyValidator(validator);
+        IgniteCache<Object, Object> cache = grid(0).createCache(cacheCfg);
+
+        startGrid(1);
+        startGrid(2);
+        awaitServers(4, grid(0));
+
+        for (int i = 0; i < 1000; i++)
+            cache.put(i, i);
+
+        forceCheckpoint();
+
+        return cache;
+    }
+
+    /** */
+    private boolean tryPut(IgniteEx ignite, int key, int val) {
+        try {
+            IgniteCache<Object, Object> cache = 
ignite.cache(DEFAULT_CACHE_NAME);
+            cache.put(key, val);
+
+            return true;
+        }
+        catch (Exception e) {
+            log.info(">>> Put rejected [node=" + ignite.name() + ", err=" + 
e.getMessage() + "]");
+
+            return false;
+        }
+    }
+
+    /**
+     * After unsplit, logs the actual state and the rebalance demand counts 
per node.
+     *
+     * <p>Documented limitation: {@code splitAndWait}/{@code unsplit} cannot 
merge two
+     * half-rings back — once a half detects the loss of the other and 
processes
+     * {@code EVT_NODE_FAILED}, it stops reconnecting to those nodes. No new 
topology
+     * version is created across the halves, so no cross-DC rebalance demand 
is sent
+     * ({@code rebalanceMsgCnt == 0}). True split-brain merge with 
state-synchronization
+     * and rebalance is covered at the ducktape level (netem + control 
utility).
+     */
+    private void logRebalanceAfterHeal(String testLabel) {
+        for (Ignite ig : G.allGrids()) {
+            IgniteConfiguration cfg = ig.configuration();
+            DemandRecordingCommSpi spi = 
(DemandRecordingCommSpi)cfg.getCommunicationSpi();
+
+            IgniteCluster cluster = ig.cluster();
+            ClusterGroup srvGrp = cluster.forServers();
+            Collection<? extends ClusterNode> srvNodes = srvGrp.nodes();
+            int srvCnt = srvNodes.size();
+
+            log.info(">>> [" + testLabel + "] node=" + ig.name()
+                + ", servers_in_topo=" + srvCnt
+                + ", rebalanceMsgCnt=" + spi.rebalanceMsgCnt);
+        }
+    }
+
+    /**
+     * Waits until the given node's cluster topology includes the expected 
number of
+     * server nodes. Returns as soon as the condition holds (30s safety cap).
+     */
+    private void awaitServers(int expected, IgniteEx ignite) throws Exception {
+        IgniteCluster cluster = ignite.cluster();
+        boolean done = GridTestUtils.waitForCondition(() -> {
+            ClusterGroup srvGrp = cluster.forServers();
+            Collection<? extends ClusterNode> srvNodes = srvGrp.nodes();
+            return srvNodes.size() == expected;
+        }, 30_000);
+        assertTrue("Topology did not reach " + expected + " servers on " + 
ignite.name(), done);
+    }
+
+    /**
+     * Verifies that with a validator requiring both DCs, the partition makes 
BOTH halves
+     * read-only: neither can see the other DC, so neither satisfies the 
validator.
+     *
+     * <p>Scenario: 4 nodes, DC1={0,1,2}, DC2={3}. Validator returns false 
unless at
+     * least one node from each DC is in the topology. After split, both sides 
reject
+     * writes.
+     */
+    @Test
+    public void testNetworkPartitionBothHalvesReadOnly() throws Exception {
+        startSplitBrainCluster(new BothDcsTopologyValidator());
+
+        splitAndWait();
+
+        assertFalse("Majority DC1 must reject writes without DC2",
+            tryPut(grid(0), 1000, 1000));
+
+        assertFalse("Minority DC2 must reject writes without DC1",
+            tryPut(grid(3), 2000, 2000));
+
+        unsplit();
+
+        logRebalanceAfterHeal("BothDcsRequired");
+    }
+
+    /**
+     * Verifies that with a quorum validator (min server nodes), the majority 
keeps
+     * writing while the minority loses quorum and becomes read-only.
+     *
+     * <p>Scenario: 4 nodes, DC1={0,1,2} (3 nodes >= threshold=3), DC2={3} (1 
node < 3).
+     * After split, DC1 passes the quorum check, DC2 does not.
+     */
+    @Test
+    public void testNetworkPartitionMajorityKeepsQuorum() throws Exception {
+        IgniteCache<Object, Object> cache0 = startSplitBrainCluster(new 
MinServerCountTopologyValidator(3));
+
+        splitAndWait();
+
+        cache0.put(1000, 1000);
+        assertEquals(1000, cache0.get(1000));
+
+        assertFalse("Minority DC2 must reject writes without quorum",
+            tryPut(grid(3), 2000, 2000));
+
+        unsplit();
+
+        logRebalanceAfterHeal("MinServerCount3");
+    }
+
+    /**
+     * Verifies that without a topology validator, BOTH halves stay writable 
during the
+     * partition (uncontrolled split-brain). Each half elects its own 
coordinator and
+     * accepts writes independently.
+     *
+     * <p>Scenario: 4 nodes, DC1={0,1,2}, DC2={3}. No validator set. After 
split, both
+     * DC1 and DC2 write successfully.
+     */
+    @Test
+    public void testNetworkPartitionNoValidatorBothWritable() throws Exception 
{
+        IgniteCache<Object, Object> cache0 = startSplitBrainCluster(null);
+
+        splitAndWait();
+
+        cache0.put(1000, 1000);
+        assertEquals(1000, cache0.get(1000));
+
+        IgniteCache<Object, Object> cache3 = grid(3).cache(DEFAULT_CACHE_NAME);
+
+        cache3.put(2000, 2000);
+        assertEquals(2000, cache3.get(2000));
+
+        unsplit();
+
+        logRebalanceAfterHeal("NoValidator");
+    }
+
+    /**
+     * Keeps the cache operative only while at least one server node of the 
main data
+     * center (DC1) is present in the topology. When a partition isolates the 
minority
+     * DC2 from DC1, the cache becomes inoperative on the minority and rejects 
writes.
+     */
+    private static class MainDcOnlyTopologyValidator implements 
TopologyValidator {
+        /** */
+        private static final long serialVersionUID = 0L;
+
+        /** {@inheritDoc} */
+        @Override public boolean validate(Collection<ClusterNode> nodes) {
+            for (ClusterNode node : nodes) {
+                if (!node.isClient() && DC1.equals(node.dataCenterId()))
+                    return true;
+            }
+
+            return false;
+        }
+    }
+
+    /**
+     * Keeps the cache operative only while server nodes from BOTH DCs are 
present.
+     * When any DC is isolated, the validator fails on both sides.
+     */
+    private static class BothDcsTopologyValidator implements TopologyValidator 
{
+        /** */
+        private static final long serialVersionUID = 0L;
+
+        /** {@inheritDoc} */
+        @Override public boolean validate(Collection<ClusterNode> nodes) {
+            boolean hasDc1 = false;
+            boolean hasDc2 = false;
+
+            for (ClusterNode node : nodes) {
+                if (node.isClient())
+                    continue;
+                String dc = node.dataCenterId();
+                if (DC1.equals(dc))
+                    hasDc1 = true;
+                if (DC2.equals(dc))
+                    hasDc2 = true;
+            }
+
+            return hasDc1 && hasDc2;
+        }
+    }
+
+    /**
+     * Keeps the cache operative only while at least {@code minCnt} server 
nodes are
+     * present in the topology.
+     */
+    private static class MinServerCountTopologyValidator implements 
TopologyValidator {
+        /** */
+        private static final long serialVersionUID = 0L;
+
+        /** */
+        private final int minCnt;
+
+        /** */
+        MinServerCountTopologyValidator(int minCnt) {
+            this.minCnt = minCnt;
+        }
+
+        /** {@inheritDoc} */
+        @Override public boolean validate(Collection<ClusterNode> nodes) {
+            int serverCnt = 0;
+
+            for (ClusterNode node : nodes) {
+                if (!node.isClient())
+                    serverCnt++;
+            }
+
+            return serverCnt >= minCnt;
+        }
+    }
+
+    /**
+     * Communication SPI that records rebalance demand messages while 
inheriting the
+     * message-blocking behavior used for segmentation.
+     */
+    private static class DemandRecordingCommSpi extends 
TestRecordingCommunicationSpi {
+        /** */
+        private volatile int rebalanceMsgCnt;
+
+        /** {@inheritDoc} */
+        @Override public void sendMessage(
+            ClusterNode node,
+            Message msg,
+            IgniteInClosure<IgniteException> ackClosure
+        ) throws IgniteSpiException {
+            if (msg instanceof GridIoMessage
+                && ((GridIoMessage)msg).message() instanceof 
GridDhtPartitionDemandMessage)
+                rebalanceMsgCnt++;
+            super.sendMessage(node, msg, ackClosure);
+        }
+    }
+}
+
diff --git 
a/modules/core/src/test/java/org/apache/ignite/testsuites/IgniteCacheTestSuite19.java
 
b/modules/core/src/test/java/org/apache/ignite/testsuites/IgniteCacheTestSuite19.java
index 6d41d046f43..32d82bf131e 100644
--- 
a/modules/core/src/test/java/org/apache/ignite/testsuites/IgniteCacheTestSuite19.java
+++ 
b/modules/core/src/test/java/org/apache/ignite/testsuites/IgniteCacheTestSuite19.java
@@ -34,6 +34,7 @@ import 
org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridC
 import 
org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest;
 import 
org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingUnmarshallingFailedSelfTest;
 import 
org.apache.ignite.internal.processors.cache.distributed.rebalancing.MultiDcRebalancingTest;
+import 
org.apache.ignite.internal.processors.cache.distributed.rebalancing.MultiDcSplitBrainRebalanceTest;
 import 
org.apache.ignite.internal.processors.cache.persistence.CleanupRestoredCachesSlowTest;
 import org.apache.ignite.testframework.GridTestUtils;
 import org.apache.ignite.testframework.junits.DynamicSuite;
@@ -72,6 +73,7 @@ public class IgniteCacheTestSuite19 {
         GridTestUtils.addTestIfNeeded(suite, 
GridCacheRabalancingDelayedPartitionMapExchangeSelfTest.class, ignoredTests);
         GridTestUtils.addTestIfNeeded(suite, 
GridCacheRebalancingCancelTest.class, ignoredTests);
         GridTestUtils.addTestIfNeeded(suite, MultiDcRebalancingTest.class, 
ignoredTests);
+        GridTestUtils.addTestIfNeeded(suite, 
MultiDcSplitBrainRebalanceTest.class, ignoredTests);
         GridTestUtils.addTestIfNeeded(suite, 
CacheStoreTxPutAllMultiNodeTest.class, ignoredTests);
         GridTestUtils.addTestIfNeeded(suite, 
CleanupRestoredCachesSlowTest.class, ignoredTests);
         GridTestUtils.addTestIfNeeded(suite, 
ClientCreateCacheGroupOnJoinNodeMapsTest.class, ignoredTests);
diff --git a/modules/ducktests/tests/ignitetest/services/mdc/mdc_cluster.py 
b/modules/ducktests/tests/ignitetest/services/mdc/mdc_cluster.py
index a9ccbe4496d..b5b57e780a4 100644
--- a/modules/ducktests/tests/ignitetest/services/mdc/mdc_cluster.py
+++ b/modules/ducktests/tests/ignitetest/services/mdc/mdc_cluster.py
@@ -34,6 +34,7 @@ from ignitetest.services.network_group.configuration import 
NetworkGroupStore, C
 from ignitetest.services.network_group.manager import NetworkGroupManager
 from ignitetest.services.utils.control_utility import ControlUtility
 from ignitetest.services.utils.ignite_configuration import 
IgniteConfiguration, TcpCommunicationSpi
+from ignitetest.services.utils.ignite_configuration.data_storage import 
DataStorageConfiguration
 from ignitetest.services.utils.ignite_configuration.discovery import 
TcpDiscoverySpi, from_ignite_cluster, \
     from_ignite_services
 from ignitetest.services.utils.ssl.client_connector_configuration import 
ClientConnectorConfiguration
@@ -101,9 +102,12 @@ class MdcCluster:
                  loaders_per_dc: Union[int, Dict[str, int]] = 0,
                  client_connector: bool = False,
                  network_timeout: int = 5_000,
-                 tcp_connect_timeout: int = 5_000):
+                 tcp_connect_timeout: int = 5_000,
+                 data_storage: DataStorageConfiguration = None,
+                 jvm_opts: List[str] = None):
         self.test_context = test.test_context
         self.logger = test.logger
+        self._jvm_opts = jvm_opts or []
 
         # A single discovery SPI (hence a single ip finder) shared by both 
DCs' server
         # services is what makes the two DCs form ONE cluster: 
prepare_on_start()
@@ -115,7 +119,8 @@ class MdcCluster:
             "version": IgniteVersion(ignite_version),
             "discovery_spi": TcpDiscoverySpi(),
             "network_timeout": network_timeout,
-            "communication_spi": 
TcpCommunicationSpi(connect_timeout=tcp_connect_timeout)
+            "communication_spi": 
TcpCommunicationSpi(connect_timeout=tcp_connect_timeout),
+            "data_storage": data_storage
         }
 
         if client_connector:
@@ -126,7 +131,8 @@ class MdcCluster:
         self.srv_per_dc = _per_dc(srv_per_dc)
 
         self.servers: Dict[str, IgniteService] = {
-            dc: IgniteService(self.test_context, self.ignite_config, 
num_nodes=num, jvm_opts=dc_jvm_opts(dc),
+            dc: IgniteService(self.test_context, self.ignite_config, 
num_nodes=num,
+                              jvm_opts=dc_jvm_opts(dc) + self._jvm_opts,
                               startup_timeout_sec=IGNITE_STARTUP_TIMEOUT_SEC)
             for dc, num in self.srv_per_dc.items() if num > 0}
 
@@ -162,9 +168,11 @@ class MdcCluster:
             service.config = 
service.config._replace(discovery_spi=discovery_spi)
 
     def _app_service(self, dc: str) -> IgniteApplicationService:
-        client_cfg = self.ignite_config._replace(client_mode=True, 
discovery_spi=from_ignite_cluster(self.servers[dc]))
+        client_cfg = self.ignite_config._replace(
+            client_mode=True, 
discovery_spi=from_ignite_cluster(self.servers[dc]),
+            data_storage=None)
 
-        return IgniteApplicationService(self.test_context, client_cfg, 
jvm_opts=dc_jvm_opts(dc))
+        return IgniteApplicationService(self.test_context, client_cfg, 
jvm_opts=dc_jvm_opts(dc) + self._jvm_opts)
 
     def register(self, dc: str, service):
         """
diff --git 
a/modules/ducktests/tests/ignitetest/tests/mdc/mdc_historical_rebalance_test.py 
b/modules/ducktests/tests/ignitetest/tests/mdc/mdc_historical_rebalance_test.py
new file mode 100644
index 00000000000..da3ac5c0f1a
--- /dev/null
+++ 
b/modules/ducktests/tests/ignitetest/tests/mdc/mdc_historical_rebalance_test.py
@@ -0,0 +1,142 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+MDC historical rebalance test: a returning data center must restore its 
partitions
+from local disk (only the delta written during its absence), not receive a full
+rebalance from the surviving DC.
+"""
+import re
+from time import sleep
+
+from ignitetest.services.mdc.mdc_cluster import MdcCluster, cross_dc_network, 
DC_1, DC_2
+from ignitetest.services.utils.ignite_aware import IgniteAwareService
+from ignitetest.services.utils.ignite_configuration.data_storage import 
DataStorageConfiguration, \
+    DataRegionConfiguration
+from ignitetest.utils import cluster, ignite_versions
+from ignitetest.utils.ignite_test import IgniteTest
+from ignitetest.utils.version import DEV_BRANCH
+
+CACHE_NAME = "mdc-historical-rebalance"
+
+BACKUPS = 1
+
+BASE_ENTRIES = 1024
+DELTA_ENTRIES = 256
+
+JVM_OPTS = ["-Xmx384M", "-Xms384M", "-DIGNITE_PREFER_WAL_REBALANCE=true"]
+
+
+def _persistence_config() -> DataStorageConfiguration:
+    """
+    Data storage configuration with persistence enabled on a small default 
data region.
+    Short checkpoint frequency ensures data reaches disk before DC failure.
+    """
+    return DataStorageConfiguration(
+        checkpoint_frequency=10,
+        default=DataRegionConfiguration(persistence_enabled=True,
+                                        initial_size=16 * 1024 * 1024,
+                                        max_size=64 * 1024 * 1024)
+    )
+
+
+def _rebalance_metric(svc, metric: str) -> int:
+    """
+    Extracts a numeric field from the last 'Completed (final) rebalancing' log 
line
+    of any node in the service.
+    """
+    for node in svc.nodes:
+        cmd = f"grep 'Completed (final) rebalancing' {node.log_file} | tail -1"
+        output = IgniteAwareService.exec_command(node, cmd)
+
+        if output:
+            m = re.search(rf"{metric}=(\d+)", output)
+
+            if m:
+                return int(m.group(1))
+
+    raise AssertionError(f"Rebalance metric '{metric}' not found in {svc} 
logs")
+
+
+class MdcHistoricalRebalanceTest(IgniteTest):
+    """
+    Verifies that a data center returning after a full stop (DC failure) 
restores
+    its partitions from local disk and only receives the delta written during 
its
+    absence, rather than a full rebalance from the surviving DC.
+    """
+    @cluster(num_nodes=4)
+    @ignite_versions(str(DEV_BRANCH))
+    def test_returning_dc_recovers_from_disk(self, ignite_version):
+        """
+        DC1 + DC2 with persistence enabled. Data is written, then DC2 goes down
+        (full stop, not a network partition). While DC2 is away, DC1 accepts 
new
+        writes (the delta). DC2 returns with clean=False (disk data preserved) 
and
+        must recover: all base data from disk, only the delta via rebalance.
+        """
+        mdc = MdcCluster(self, ignite_version, srv_per_dc=1, runners_per_dc=1,
+                         data_storage=_persistence_config(),
+                         jvm_opts=JVM_OPTS)
+
+        with cross_dc_network(self.logger, mdc):
+            mdc.start_servers()
+
+            # With persistence the cluster starts INACTIVE; activate it 
explicitly.
+            mdc.control(DC_1).activate()
+
+            # Phase 1: populate base data across both DCs.
+            mdc.generate_data(DC_1, CACHE_NAME, 0, BASE_ENTRIES // 2, 
backups=BACKUPS)
+            mdc.generate_data(DC_2, CACHE_NAME, BASE_ENTRIES // 2, 
BASE_ENTRIES, backups=BACKUPS)
+
+            mdc.check_data(DC_1, CACHE_NAME, 0, BASE_ENTRIES)
+            mdc.check_data(DC_2, CACHE_NAME, 0, BASE_ENTRIES)
+
+            mdc.verify_cache_distribution(CACHE_NAME, copies_per_dc=1)
+
+            # Allow checkpoint to flush data to disk before the DC failure.
+            sleep(15)
+
+            # Phase 2: DC2 goes down (full stop, simulating DC failure).
+            mdc.servers[DC_2].stop()
+
+            # DC1 must remain healthy and accept writes.
+            mdc.check_put_admissibility(DC_1, CACHE_NAME, True)
+
+            # Phase 3: write the delta on the surviving DC1.
+            mdc.generate_data(DC_1, CACHE_NAME, BASE_ENTRIES, BASE_ENTRIES + 
DELTA_ENTRIES,
+                              backups=BACKUPS)
+
+            mdc.check_data(DC_1, CACHE_NAME, 0, BASE_ENTRIES + DELTA_ENTRIES)
+
+            # Phase 4: DC2 returns (disk data preserved, clean=False).
+            mdc.servers[DC_2].start(clean=False)
+            mdc.servers[DC_2].await_rebalance()
+
+            # Phase 5: verify historical rebalance actually restored data from 
disk.
+            hist_entries = _rebalance_metric(mdc.servers[DC_2], "histEntries")
+
+            self.logger.info(f"Historical rebalance: 
histEntries={hist_entries}")
+
+            assert hist_entries > 0, \
+                f"Expected data restored from local disk 
[histEntries={hist_entries}]"
+
+            # Phase 6: verify full data integrity on both DCs.
+            mdc.check_data(DC_1, CACHE_NAME, 0, BASE_ENTRIES + DELTA_ENTRIES)
+            mdc.check_data(DC_2, CACHE_NAME, 0, BASE_ENTRIES + DELTA_ENTRIES)
+
+            mdc.verify_cache_distribution(CACHE_NAME, copies_per_dc=1)
+
+            mdc.verify_servers_log_clean()
+
+            mdc.stop_servers()

Reply via email to