JackieTien97 commented on code in PR #18814:
URL: https://github.com/apache/iotdb/pull/18814#discussion_r4229825119


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/fragment/QueryContext.java:
##########
@@ -145,7 +145,7 @@ public PatternTreeMap<ModEntry, ModsSerializer> 
loadAllModificationsFromDisk(
   }
 
   protected boolean shouldSkipModification(ModEntry modification) {
-    if (tables != null && modification instanceof TableDeletionEntry) {
+    if (isTableModel() && modification instanceof TableDeletionEntry) {

Review Comment:
   [P2] Preserve the nullable table-filter guard for compaction contexts
   
   A table-model context does not imply that `tables` has been populated. 
`ReadPointCompactionPerformer.perform()` creates its context with 
`createFragmentInstanceContextForCompaction()` and sets `ignoreAllNullRows` to 
`false` for table data, but never calls `collectTable()`. If an input file 
contains a `TableDeletionEntry`, this new condition is true and 
`tables.contains(tableName)` throws a `NullPointerException`. Consequently, 
table-model compaction with the `read_point` performer fails after deletions. 
Please retain the `tables != null` guard: an absent table-name filter must 
allow the modifications through.
   
   <details>
   <summary>Reproduction: real modification file and the production compaction 
context factory</summary>
   
   Add this test under 
`iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/execution/fragment/Review18814CompactionModsTest.java`:
   
   ```java
   /*
    * Licensed to the Apache Software Foundation (ASF) under one
    * or more contributor license agreements.  See the NOTICE file
    * distributed with this work for additional information
    * regarding copyright ownership.  The ASF licenses this file
    * to you under the Apache License, Version 2.0 (the
    * "License"); you may not use this file except in compliance
    * with the License.  You may obtain a copy of the License at
    *
    *     http://www.apache.org/licenses/LICENSE-2.0
    *
    * Unless required by applicable law or agreed to in writing,
    * software distributed under the License is distributed on an
    * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    * KIND, either express or implied.  See the License for the
    * specific language governing permissions and limitations
    * under the License.
    */
   
   package org.apache.iotdb.db.queryengine.execution.fragment;
   
   import 
org.apache.iotdb.db.storageengine.dataregion.modification.DeletionPredicate;
   import org.apache.iotdb.db.storageengine.dataregion.modification.ModEntry;
   import 
org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile;
   import 
org.apache.iotdb.db.storageengine.dataregion.modification.TableDeletionEntry;
   import 
org.apache.iotdb.db.storageengine.dataregion.modification.TagPredicate;
   import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
   
   import org.apache.tsfile.file.metadata.StringArrayDeviceID;
   import org.apache.tsfile.read.common.TimeRange;
   import org.junit.Rule;
   import org.junit.Test;
   import org.junit.rules.TemporaryFolder;
   
   import java.io.File;
   import java.util.List;
   
   import static org.junit.Assert.assertEquals;
   
   public class Review18814CompactionModsTest {
     @Rule public TemporaryFolder temporaryFolder = new TemporaryFolder();
   
     @Test
     public void 
testCompactionContextLoadsTableDeletionsWithoutCollectedTables() throws 
Exception {
       File partition = temporaryFolder.newFolder("db", "0", "0");
       TsFileResource resource = new TsFileResource(new File(partition, 
"1-1-0-0.tsfile"));
       StringArrayDeviceID device = new StringArrayDeviceID("table1", "tag1");
       TableDeletionEntry deletion =
           new TableDeletionEntry(
               new DeletionPredicate("table1", new 
TagPredicate.FullExactMatch(device)),
               new TimeRange(1, 10));
       try (ModificationFile mods = resource.getModFileForWrite()) {
         mods.write(deletion);
       }
       FragmentInstanceContext context =
           
FragmentInstanceContext.createFragmentInstanceContextForCompaction(-1);
       context.setIgnoreAllNullRows(false);
       List<ModEntry> loaded = context.getPathModifications(resource, device, 
"s1");
       assertEquals(1, loaded.size());
       assertEquals(deletion, loaded.get(0));
     }
   }
   ```
   
   Run with JDK 17:
   
   ```sh
   mvn clean test -pl iotdb-core/datanode -am \
     -Dtest=Review18814CompactionModsTest \
     -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false
   ```
   
   Actual comparison: the test passes at merge base 
`d20082b1615e6957e77593c9d2623d6fdc843340` and fails at PR head 
`df3eb05cee0ed12997ba881e4ce70781f62b9227` with:
   
   ```text
   java.lang.NullPointerException: Cannot invoke 
"java.util.Set.contains(Object)" because "this.tables" is null
       at 
org.apache.iotdb.db.queryengine.execution.fragment.QueryContext.shouldSkipModification(QueryContext.java:150)
       at 
org.apache.iotdb.db.queryengine.execution.fragment.QueryContext.loadAllModificationsFromDisk(QueryContext.java:138)
   ```
   
   </details>



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/tools/ImportWAL.java:
##########
@@ -949,17 +972,28 @@ static class WALReplayer implements WALReplayWorker {
           new ReplayDecisionController(System.console()));
     }
 
+    /**
+     * @param treeSession the session which replays tree model entries, it 
carries no database
+     * @param tableSession the session which replays table model entries, null 
when no table
+     *     database is declared
+     * @param tableDatabaseName the target database resolved before replay; 
null means no table
+     *     database is declared, so the entries are replayed with the tree 
model only
+     * @param replayDecisionPrompt the prompt which decides how to handle 
entries that cannot be
+     *     replayed automatically
+     */
     WALReplayer(
         final Session treeSession,
         final Session tableSession,
         final String tableDatabaseName,
         final ReplayDecisionPrompt replayDecisionPrompt) {
       this.treeSession = treeSession;
       this.tableSession = tableSession;
+      this.isTableModel =
+          tableDatabaseName != null && 
PathUtils.isTableModelDatabase(tableDatabaseName);

Review Comment:
   [P2] Resolve the source model per snapshot instead of per target database
   
   A WAL directory is not necessarily confined to one data model: with 
SimpleConsensus, `WALManager` uses `ElasticStrategy` or `RoundRobinStrategy`, 
which can share a WAL node across tree and table DataRegions. For a shared WAL 
containing snapshots from `root.sg` and `db`, specifying `-db db` is needed for 
the table data, but this fixed flag now treats the tree snapshot as table data 
too. It reaches `createTableTabletSchema()`, issues `DESCRIBE` for the tree 
device through the table Session, and fails replay. Choosing a tree database 
instead cannot correctly replay the table snapshot. The snapshot's source model 
needs to be resolved per memtable/entry (for example, from source/checkpoint 
metadata), independently of the target table database.
   
   <details>
   <summary>Reproduction: serialize tree and table snapshots into one WAL file, 
then replay it</summary>
   
   Add the following method to the existing `ImportWALTest`; it reuses that 
class's `createWALFile`, `writeWAL`, and `mockTableSchema` helpers and existing 
imports:
   
   ```java
     @Test
     public void testReview18814MixedModelSnapshotsInSharedWAL() throws 
Exception {
       final PrimitiveMemTable treeMemTable = new PrimitiveMemTable("root.sg", 
"0");
       final StringArrayDeviceID treeDevice = new 
StringArrayDeviceID("root.sg.d1");
       treeMemTable.writeAlignedRow(
           treeDevice,
           Collections.singletonList(new MeasurementSchema("s1", 
TSDataType.INT32)),
           1,
           new Object[] {10});
       final PrimitiveMemTable tableMemTable = new PrimitiveMemTable("db", "1");
       tableMemTable.writeAlignedRow(
           new StringArrayDeviceID("table1", "tag1"),
           Collections.singletonList(new MeasurementSchema("s1", 
TSDataType.INT32)),
           2,
           new Object[] {20});
       final File walFile = createWALFile(0);
       writeWAL(walFile, new WALInfoEntry(1, treeMemTable), new WALInfoEntry(2, 
tableMemTable));
       final Session treeSession = mock(Session.class);
       final Session tableSession = mock(Session.class);
       mockTableSchema(tableSession, "table1", "time", "tag1");
       when(tableSession.executeQueryStatement(
               "DESCRIBE " + 
ImportWAL.WALReplayer.quoteIdentifier(treeDevice.getTableName())))
           .thenThrow(new StatementExecutionException("Tree device is not a 
table in db"));
   
       final ImportWAL.ReplayStatistics statistics =
           ImportWAL.replayWALFiles(
               Collections.singletonList(walFile.toPath()),
               new ImportWAL.WALReplayer(treeSession, tableSession, "db"));
   
       assertEquals(2, statistics.getReplayedOperationCount());
       verify(treeSession).insertAlignedTablet(any(Tablet.class));
       verify(tableSession).insertRelationalTablet(any(Tablet.class));
     }
   ```
   
   The WAL bytes are written and read through the real WAL implementation. 
Session RPC is mocked: the target database has `table1`, and the mock rejects a 
table-schema lookup for the tree device. The expected routing is one 
`insertAlignedTablet` on the tree Session and one `insertRelationalTablet` on 
the table Session.
   
   Run with JDK 17:
   
   ```sh
   mvn clean test -pl iotdb-core/datanode -am \
     '-Dtest=ImportWALTest#testReview18814MixedModelSnapshotsInSharedWAL' \
     -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false
   ```
   
   Actual comparison: the test passes at merge base 
`d20082b1615e6957e77593c9d2623d6fdc843340` and fails at PR head 
`df3eb05cee0ed12997ba881e4ce70781f62b9227`. On the PR head the unexpected 
`DESCRIBE` reaches the mock, throwing `StatementExecutionException: Tree device 
is not a table in db`, which is wrapped in `WALReplayException` before both 
snapshots can be replayed.
   
   </details>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to