Caideyipi commented on code in PR #18901:
URL: https://github.com/apache/iotdb/pull/18901#discussion_r4309704243


##########
integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBLoadTsFileClusterIT.java:
##########
@@ -260,6 +261,74 @@ public void testLoadAfterOneDataNodeDown() throws 
Exception {
     }
   }
 
+  /**
+   * A LOAD that runs while one DataNode is down is replicated by consensus to 
the surviving
+   * replicas. When the stopped DataNode restarts, its replicas replay the 
LOAD commands they
+   * missed, so the restarted DataNode ends up serving the very same file.
+   */
+  @Test
+  public void testRestartedDataNodeIsSyncedWithTheLoadedFile() throws 
Exception {
+    final long firstPointCount = generateTsFile(tmpDir, "sync-first.tsfile", 
5_000, 1L);
+
+    try (final Connection connection = EnvFactory.getEnv().getConnection();
+        final Statement statement = connection.createStatement()) {
+      createDatabaseAndSchema(statement);
+      statement.execute("load \"" + tmpDir.getAbsolutePath() + "\"");
+      awaitPointCount(statement, firstPointCount);
+    }
+
+    // LOAD the second file while DataNode 0 is down. With three replicas it 
holds a copy of every
+    // region, so it is the DataNode that has to catch up with the LOAD when 
it comes back.
+    final DataNodeWrapper restartedDataNode = 
EnvFactory.getEnv().getDataNodeWrapper(0);
+    restartedDataNode.stop();
+    Assert.assertFalse(restartedDataNode.isAlive());
+
+    final long secondPointCount = generateTsFile(tmpDir, "sync-second.tsfile", 
3_000, 100_000L);
+    final long expectedPointCount = firstPointCount + secondPointCount;
+    try (final Connection connection =
+            DriverManager.getConnection(
+                Config.IOTDB_URL_PREFIX
+                    + 
EnvFactory.getEnv().getDataNodeWrapper(1).getIpAndPortString(),
+                "root",
+                "root");
+        final Statement statement = connection.createStatement()) {
+      statement.execute("load \"" + tmpDir.getAbsolutePath() + "\"");
+      awaitPointCount(statement, expectedPointCount);
+    }
+
+    restartedDataNode.start();
+    Assert.assertTrue(restartedDataNode.isAlive());
+
+    // The connection is pinned to the restarted DataNode, so this waits for 
the replicas it
+    // replayed the LOAD into and fails if the restart left them behind.
+    awaitPointCount(restartedDataNode, expectedPointCount);

Review Comment:
   **[P2] Verify recovery using the restarted node's local replicas**
   
   `getConnection(dataNode)` fixes the JDBC endpoint, but the restarted node 
still plans a distributed query. Under the default strong read consistency, 
`AbstractFragmentParallelPlanner.selectTargetDataNode()` chooses the first 
available replica in the region route, which can be DN1 or DN2. Those nodes 
already returned `expectedPointCount` before the restart. Consequently, this 
assertion can pass as soon as DN0 accepts connections even if its replicas 
never replay the missed LOAD commands. Please force the storage reads onto the 
restarted node, or verify its local replica data/applied state, so a broken 
catch-up path makes this test fail.



##########
integration-test/src/test/java/org/apache/iotdb/db/it/IoTDBLoadTsFileRegionMigrationIT.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.it;
+
+import org.apache.iotdb.commons.schema.column.ColumnHeaderConstant;
+import org.apache.iotdb.consensus.ConsensusFactory;
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.it.utils.TsFileGenerator;
+import org.apache.iotdb.itbase.category.ClusterIT;
+
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.awaitility.Awaitility;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.sql.Connection;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * LOAD TsFile has to keep working when the topology changes while a file is 
being imported: the
+ * route of a region can be re-resolved after a piece was rejected, and a LOAD 
that notices a
+ * migration falls back to inserting the file through the ordinary write path. 
Either way every
+ * point of the file has to end up in the cluster and the migration has to 
finish.
+ *
+ * <p>The data replication factor is 2 with three DataNodes, so a region 
always has exactly one
+ * DataNode that can be its migration destination.
+ */
+@RunWith(IoTDBTestRunner.class)
+@Category({ClusterIT.class})
+public class IoTDBLoadTsFileRegionMigrationIT {
+
+  private static final long PARTITION_INTERVAL = 100_000_000L;
+  private static final String DATABASE = "root.load_migrate";
+  private static final List<String> DEVICES =
+      Arrays.asList(
+          DATABASE + ".d1", DATABASE + ".d2", DATABASE + ".d3", DATABASE + 
".d4", DATABASE + ".d5");
+  private static final List<String> MEASUREMENTS = Arrays.asList("s1", "s2", 
"s3", "s4");
+  private static final int POINT_COUNT_PER_DEVICE = 20_000;
+  private static final int DATA_REPLICATION_FACTOR = 2;
+  private static final int SCHEMA_REPLICATION_FACTOR = 3;
+  private static final int CONFIG_NODE_NUM = 1;
+  private static final int DATA_NODE_NUM = 3;
+
+  private File tmpDir;
+
+  @BeforeClass
+  public static void setUpClass() {
+    
EnvFactory.getEnv().getConfig().getCommonConfig().setTimePartitionInterval(PARTITION_INTERVAL);
+    
EnvFactory.getEnv().getConfig().getCommonConfig().setEnforceStrongPassword(false);
+    
EnvFactory.getEnv().getConfig().getCommonConfig().setPipeMemoryManagementEnabled(false);
+    
EnvFactory.getEnv().getConfig().getCommonConfig().setDatanodeMemoryProportion("1:10:1:1:1:0");
+    
EnvFactory.getEnv().getConfig().getCommonConfig().setTargetChunkPointNum(1_000);
+    
EnvFactory.getEnv().getConfig().getCommonConfig().setMaxNumberOfPointsInPage(500);
+    EnvFactory.getEnv()
+        .getConfig()
+        .getCommonConfig()
+        .setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
+        
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
+        .setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS)
+        .setSchemaReplicationFactor(SCHEMA_REPLICATION_FACTOR)
+        .setDataReplicationFactor(DATA_REPLICATION_FACTOR);
+    EnvFactory.getEnv().initClusterEnvironment(CONFIG_NODE_NUM, DATA_NODE_NUM);
+  }
+
+  @AfterClass
+  public static void tearDownClass() {
+    EnvFactory.getEnv().cleanClusterEnvironment();
+  }
+
+  @Before
+  public void setUp() throws Exception {
+    tmpDir = new File(Files.createTempDirectory("load-migrate-it").toUri());
+  }
+
+  @After
+  public void tearDown() throws Exception {
+    try (final Connection connection = EnvFactory.getEnv().getConnection();
+        final Statement statement = connection.createStatement()) {
+      statement.execute("delete database " + DATABASE);
+    } catch (final Exception ignored) {
+    }
+
+    final File[] files = tmpDir.listFiles();
+    if (files != null) {
+      for (final File file : files) {
+        Assert.assertTrue(deleteRecursively(file));
+      }
+    }
+    Assert.assertTrue(tmpDir.delete());
+  }
+
+  /**
+   * Starts a LOAD of a file large enough to still be dispatching pieces when 
a region of its
+   * database is migrated away from a DataNode it was routed to, then asserts 
that both the LOAD and
+   * the migration finish with every point of the file imported.
+   */
+  @Test
+  public void testLoadWhileARegionMigrates() throws Exception {
+    final long writtenPointCount = generateTsFile(new File(tmpDir, 
"load-migrating.tsfile"));
+
+    try (final Connection connection = EnvFactory.getEnv().getConnection();
+        final Statement statement = connection.createStatement()) {
+      createDatabaseAndSchema(statement);
+    }
+
+    final ExecutorService loader = Executors.newSingleThreadExecutor();
+    try {
+      final Future<Exception> load =
+          loader.submit(
+              () -> {
+                try (final Connection connection = 
EnvFactory.getEnv().getConnection();
+                    final Statement statement = connection.createStatement()) {
+                  statement.execute("load \"" + tmpDir.getAbsolutePath() + 
"\"");
+                  return null;
+                } catch (final Exception e) {
+                  return e;
+                }
+              });
+
+      // Wait until the LOAD allocated the data region of its points, then 
migrate that region while
+      // the LOAD is still splitting and dispatching the pieces of the file.
+      final int[] migration = awaitMigratableRegion();
+      Assert.assertFalse("the LOAD finished before the migration could overlap 
it", load.isDone());
+      try (final Connection connection = EnvFactory.getEnv().getConnection();
+          final Statement statement = connection.createStatement()) {
+        statement.execute(
+            "migrate region " + migration[0] + " from " + migration[1] + " to 
" + migration[2]);

Review Comment:
   **[P2] Synchronize the LOAD with the migration notification**
   
   `load.isDone()` observes the future before opening another connection and 
submitting `MIGRATE REGION`. The LOAD can finish during that gap. Also, 
`ProcedureManager.submitRegionOperationProcedures()` returns after enqueueing 
the migration, and the DataNodes receive the migration notification later. In 
either case, the remaining checks can pass with the LOAD and migration running 
sequentially, without exercising route refresh or migration fallback. A fast 
LOAD that finishes before the 200 ms region poll instead fails the assertion 
despite correct behavior. Please add a deterministic barrier/test hook that 
pauses the LOAD after dispatching a piece, waits for the migration to become 
active, and then resumes it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to