Caideyipi commented on code in PR #18874:
URL: https://github.com/apache/iotdb/pull/18874#discussion_r4269822165


##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/schema/TreeDeviceViewFieldDetector.java:
##########
@@ -64,98 +65,191 @@ public class TreeDeviceViewFieldDetector {
   private final ConfigManager configManager;
   private final PartialPath path;
   private final TsTable table;
-  private final Map<String, Set<FieldColumnSchema>> fields;
 
   private TDeviceViewResp result = new TDeviceViewResp(StatusUtils.OK, new 
ConcurrentHashMap<>());
   private final Map<String, String> lowerCase2OriginalMap = new HashMap<>();
 
-  public TreeDeviceViewFieldDetector(
-      final ConfigManager configManager,
-      final TsTable table,
-      final Map<String, Set<FieldColumnSchema>> fields) {
+  public TreeDeviceViewFieldDetector(final ConfigManager configManager, final 
TsTable table) {
     this.configManager = configManager;
     this.path = TreeViewSchema.getPrefixPattern(table);
     this.table = table;
-    this.fields = fields;
   }
 
-  public TSStatus detectMissingFieldTypes() {
-    if (table.getFieldNum() == 0 && Objects.isNull(fields)) {
+  /**
+   * Fills the fields of the view which is being created: a view may declare 
no field at all, in
+   * which case every field under its tree path pattern becomes a column, or 
declare fields whose
+   * type is left to the detection.
+   */
+  public TSStatus detectFieldsForCreation() {
+    if (table.getFieldNum() == 0) {
+      // for the first time creation of the table, user do not assign any 
field;
+      // just detect all the fields of prefix path
       new TreeDeviceViewFieldDetectionTaskExecutor(
               configManager,
               getLatestSchemaRegionMap(),
               table.getTagNum(),
-              TreeViewSchema.isRestrict(table))
+              TreeViewSchema.isRestrict(table),
+              null,
+              false)
           .execute();
       if (result.getStatus().getCode() != 
TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
         return result.getStatus();
       }
       result
           .getDeviewViewFieldTypeMap()
           .forEach(
-              (field, type) -> {
-                final FieldColumnSchema columnSchema =
-                    new FieldColumnSchema(field, 
TSDataType.getTsDataType(type));
-                if (!field.equals(lowerCase2OriginalMap.get(field))) {
-                  TreeViewSchema.setOriginalName(columnSchema, 
lowerCase2OriginalMap.get(field));
-                }
-                table.addColumnSchema(columnSchema);
-              });
-    } else {
-      final Map<String, Set<FieldColumnSchema>> unknownFields;
-      if (Objects.isNull(fields)) {
-        unknownFields = new HashMap<>();
-        for (final TsTableColumnSchema schema : table.getColumnList()) {
-          if (!(schema instanceof FieldColumnSchema)
-              || schema.getDataType() != TSDataType.UNKNOWN) {
-            continue;
-          }
-          final String key = TreeViewSchema.getSourceName(schema);
-          if (!unknownFields.containsKey(key)) {
-            unknownFields.put(key, new HashSet<>());
-          }
-          unknownFields.get(key).add((FieldColumnSchema) schema);
-        }
-      } else {
-        unknownFields = fields;
-      }
+              (field, type) ->
+                  
table.addColumnSchema(toFieldColumnSchemaWithOriginName(field, type)));
+      return StatusUtils.OK;
+    }
+
+    return detectFieldTypes(collectUnknownFieldColumns());
+  }
+
+  /**
+   * Resolves the type of the field columns which are being added to an 
existing view. The columns
+   * to resolve are given by the caller, as only it knows which ones it is 
adding.
+   */
+  public TSStatus detectFieldTypesForColumnAddition(final Map<String, 
FieldColumnSchema> fields) {
+    return detectFieldTypes(fields);
+  }
+
+  /**
+   * The fields which exist under the view's tree path pattern but are not 
columns of the view yet,
+   * as the columns to add. Used after a view is created, to pick up the 
series which were created
+   * before the view was mounted on the DataNodes, and which the DataNodes 
therefore never reported.
+   */
+  public Pair<TSStatus, List<TsTableColumnSchema>> 
detectNewFieldColumnsAfterCreation() {
+    new TreeDeviceViewFieldDetectionTaskExecutor(
+            configManager,
+            getLatestSchemaRegionMap(),
+            table.getTagNum(),
+            TreeViewSchema.isRestrict(table),
+            null,
+            true)
+        .execute();
+    if (result.getStatus().getCode() != 
TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+      return new Pair<>(result.getStatus(), Collections.emptyList());
+    }
+    final List<TsTableColumnSchema> columns = new ArrayList<>();
+    result
+        .getDeviewViewFieldTypeMap()
+        .forEach(
+            (field, type) -> {
+              if (Objects.isNull(table.getColumnSchema(field))) {
+                columns.add(toFieldColumnSchemaWithOriginName(field, type));

Review Comment:
   **[P1] Compare source measurements when deciding which columns are missing**
   
   A field already mapped through `FROM` is treated as missing because this 
checks only its table column name. With only `root.db.d.s1` present, creating 
`v(device TAG, renamed INT32 FIELD FROM s1) WITH (auto_sync_schema=true) AS 
root.db.**` makes the catch-up scan emit another column named `s1`. 
`checkColumnSchemaForTree2TableView()` then removes that candidate because its 
source is already referenced by `renamed`, and the empty addition fails with 
`COLUMN_ALREADY_EXISTS`. The added `AddViewColumnProcedure` is a child of the 
create procedure, so this failure triggers rollback of an otherwise valid view 
creation; turning auto-sync on has the same problem. Exclude fields whose 
source name is already represented before submitting the child, and make an 
already-satisfied reconciliation a success.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -2890,6 +2902,24 @@ public TSStatus updateTemplate(final TUpdateTemplateReq 
req) {
           SchemaEngine.getInstance()
               .updateSubtreeMeasurementCountForTemplate(newTemplate.getId(), 
delta);
         }
+        if (TreeToTableViewCache.getInstance().hasView()) {
+          // addViewColumnsIfNeeded, which collects the views from the index 
and asks CN to add the
+          // columns.
+          try {
+            final Map<PartialPath, List<Pair<String, TSDataType>>> 
device2Measurements =
+                collectDevicesUsingExtendedTemplate(oldTemplate, newTemplate);
+            TriggerAddViewColumnVisitor.addViewColumnsIfNeeded(
+                device2Measurements, ALTER_DEVICE_TEMPLATE_ID_PREFIX + 
newTemplate.getId());

Review Comment:
   **[P1] Run template-extension reconciliation after releasing the ConfigNode 
schema lock**
   
   For an activated template covered by an auto-sync view, this synchronous 
callback creates a lock cycle. `ClusterSchemaManager.extendSchemaTemplate()` is 
synchronized and waits for the DataNode `UPDATE_TEMPLATE` responses while 
holding its monitor. This DataNode handler calls `updateTree2TableViews` and 
waits for `AddViewColumnProcedure`, whose `columnCheck()` calls the 
synchronized `tableColumnCheckForColumnExtension()` on the same manager. That 
procedure cannot progress until template extension returns, and template 
extension cannot return until this callback finishes. A healthy extension 
therefore waits for an RPC or the 30-second procedure timeout to break the 
cycle, blocking other operations that need the manager lock. Schedule 
reconciliation after the update response, or perform it on ConfigNode outside 
that monitor.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/table/TreeToTableViewCache.java:
##########
@@ -0,0 +1,465 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.schemaengine.table;
+
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.path.PathPatternUtil;
+import org.apache.iotdb.commons.schema.node.utils.IMNodeFactory;
+import org.apache.iotdb.commons.schema.table.TreeViewSchema;
+import org.apache.iotdb.commons.schema.table.TsTable;
+import org.apache.iotdb.db.i18n.DataNodeSchemaMessages;
+import 
org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.mem.mnode.IMemMNode;
+import 
org.apache.iotdb.db.schemaengine.schemaregion.mtree.loader.MNodeFactoryLoader;
+import 
org.apache.iotdb.db.schemaengine.schemaregion.view.TreeToTableViewReference;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import static 
org.apache.iotdb.commons.conf.IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD;
+import static org.apache.iotdb.commons.conf.IoTDBConstant.PATH_ROOT;
+
+/**
+ * The index of the tree to table views on DataNode, keyed by the fixed prefix 
of each view's tree
+ * path pattern.
+ *
+ * <p>Only DataNodeTableCache writes this index, and it does so while holding 
its own write lock, so
+ * every write here also holds the write lock of this cache and the lock order 
is always
+ * DataNodeTableCache's write lock and then this cache's write lock. A reader 
only takes this
+ * cache's read lock, so the two locks can never form a cycle.
+ */
+public class TreeToTableViewCache implements ITreeToTableViewCache {
+
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(TreeToTableViewCache.class);
+
+  private final IMNodeFactory<IMemMNode> nodeFactory =
+      MNodeFactoryLoader.getInstance().getMemMNodeIMNodeFactory();
+  // This index describes path prefixes, not the database/device roles in a 
SchemaRegion. clear()
+  // replaces it with a fresh one.
+  private IMemMNode rootNode = nodeFactory.createInternalMNode(null, 
PATH_ROOT);
+
+  private final Map<IMemMNode, Set<TreeToTableViewReference>> viewReferences =
+      new IdentityHashMap<>();
+  private final Map<Pair<String, String>, PartialPath> viewPaths = new 
HashMap<>();
+  private final ReentrantReadWriteLock readWriteLock = new 
ReentrantReadWriteLock();
+
+  /**
+   * Whether any view is registered. A cluster without any view answers "no 
view covers this path"
+   * without taking the lock or walking the trie. It is written only under the 
write lock, and read
+   * without any lock.
+   */
+  private volatile boolean hasView = false;
+
+  private TreeToTableViewCache() {}
+
+  @Override
+  public void clear() {
+    readWriteLock.writeLock().lock();
+    try {
+      // A node is only reachable from its parent, so a fresh root drops the 
whole trie at once.
+      rootNode = nodeFactory.createInternalMNode(null, PATH_ROOT);
+      viewReferences.clear();
+      viewPaths.clear();
+      hasView = false;
+    } finally {
+      readWriteLock.writeLock().unlock();
+    }
+  }
+
+  @Override
+  public void init(final Map<String, List<TsTable>> initialTables) {
+    readWriteLock.writeLock().lock();
+    try {
+      // The index is rebuilt from scratch, so the entry of a view which is 
gone does not survive.
+      clear();
+      initialTables.forEach(
+          (database, tables) -> tables.forEach(table -> registerView(database, 
table)));
+    } finally {
+      readWriteLock.writeLock().unlock();
+    }
+  }
+
+  /**
+   * Registers the view at the path of its tree path pattern. The caller has 
already picked the
+   * tables which belong in the index, see 
DataNodeTableCache#collectInitialViews.
+   */
+  private void registerView(final String database, final TsTable table) {
+    final Optional<String> treePath = 
table.getPropValue(TreeViewSchema.TREE_PATH_PATTERN);
+    if (treePath.isEmpty()) {
+      return;
+    }
+    try {
+      addView(
+          database,
+          table.getTableName(),
+          TreeViewSchema.isRestrict(table),
+          table.getTagNum(),
+          new PartialPath(treePath.get()));
+    } catch (final IllegalPathException e) {
+      LOGGER.warn(
+          DataNodeSchemaMessages
+              
.LOG_FAILED_TO_REGISTER_THE_TREE_TO_TABLE_VIEW_ARG_ARG_DURING_INIT_ILLEGAL_PATH_E9D65F63,
+          database,
+          table.getTableName(),
+          e);
+    }
+  }
+
+  /**
+   * Register a view at a fixed prefix path.
+   *
+   * @parm Partial path, eg: root.db.dev01.**;
+   * @return true if a new reference was registered; false if it was already 
present
+   */
+  // todo, log;
+  @Override
+  public boolean addView(
+      final String database,
+      final String tableName,
+      final boolean isRestrict,
+      final int numOfTags,
+      final PartialPath path)
+      throws IllegalPathException {
+    int prefixLength = checkPartialPath(path);
+    final String[] nodes = path.getNodes();
+    readWriteLock.writeLock().lock();
+    try {
+      final PartialPath registeredPath = viewPaths.get(new Pair<>(database, 
tableName));
+      if (Objects.nonNull(registeredPath) && !Objects.equals(registeredPath, 
path)) {
+        // A view is registered at exactly one path, so the entry left at the 
previous path has to
+        // go, or removing this view by name later would miss it.
+        removeView(database, tableName);
+      }
+      viewPaths.put(new Pair<>(database, tableName), path);
+
+      IMemMNode current = rootNode;
+      for (int i = 1; i < prefixLength; i++) {
+        IMemMNode child = current.getChild(nodes[i]);
+        if (child == null) {
+          child = current.addChild(nodes[i], 
nodeFactory.createInternalMNode(current, nodes[i]));
+        }
+        current = child;
+      }
+
+      final boolean added =
+          viewReferences
+              .computeIfAbsent(current, node -> new HashSet<>())
+              .add(new TreeToTableViewReference(database, tableName, 
isRestrict, numOfTags));

Review Comment:
   **[P2] Replace matching metadata when a view is re-registered at the same 
path**
   
   `TreeToTableViewReference.equals()` compares only database and table names, 
so `HashSet.add()` keeps the old reference when `CREATE OR REPLACE VIEW` 
registers the same view at the same tree path. The old entry is removed above 
only if the path changed; changes to TAG count or restrict therefore never 
reach this index. For example, replacing a restricted one-TAG view at 
`root.db.**` with a restricted two-TAG view still leaves `root.db.site.device` 
uncovered, so subsequent measurements are never synchronized. Changing restrict 
can also leave devices incorrectly covered. Both cases fail in regression tests 
against this commit. Replace the existing reference for that identity even when 
the path is unchanged.



##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/consensus/statemachine/schemaregion/SchemaExecutionVisitor.java:
##########
@@ -105,6 +112,8 @@ public class SchemaExecutionVisitor implements 
PlanVisitor<TSStatus, ISchemaRegi
   @Override
   public TSStatus visitCreateTimeSeries(
       final CreateTimeSeriesNode node, final ISchemaRegion schemaRegion) {
+    triggerAddViewColumn(node, null);
+
     try {
       schemaRegion.createTimeSeries(node, -1);

Review Comment:
   **[P2] Add view columns only for schema changes that actually succeeded**
   
   The trigger commits the view column through ConfigNode before 
`schemaRegion.createTimeSeries()` runs. The tree creation can still fail here, 
for example with `SeriesOverflowException` from `SchemaRegionMemoryImpl` or a 
late alias/path conflict, but the view addition has already committed and is 
never rolled back. An unsuccessful attempt to create `s2 INT32` can therefore 
leave an `INT32` column even though `s2` does not exist; a later successful 
creation of `s2 FLOAT` is ignored because column validation removes existing 
names, leaving the view with the wrong type. The same ordering exists in the 
aligned, multi-series and template activation paths. Trigger reconciliation 
after successful creation and use only the measurements that succeeded in a 
partially successful batch.



##########
iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ProcedureMessages.java:
##########
@@ -1511,8 +1511,17 @@ private ProcedureMessages() {}
   public static final String MESSAGE_ROLLING_BACK_AFTER_FAILURE_ARG_474DF456 = 
"正在回滚,失败原因:%s。";
   public static final String 
MESSAGE_ROLLING_BACK_AFTER_AN_EARLIER_FAILURE_850D0AF5 =
       "正在回滚此前发生的失败。";
+<<<<<<< HEAD
   public static final String 
EXCEPTION_DATANODE_SETSYSTEMSTATUS_RPC_FAILED_ARG_ARG_ARG_7E8E5F9F =

Review Comment:
   **[P1] Resolve the merge conflict and duplicate constant before building the 
zh locale**
   
   This Java source still contains `<<<<<<< HEAD`, `=======`, and `>>>>>>>` 
markers. The `with-zh-locale` profile includes this file, so the Chinese build 
cannot compile: `javac -encoding UTF-8` reports seven syntax errors in this 
block. Removing only the markers still fails because the commit-release logging 
constant introduced at line 1521 is already declared at line 1141. Please 
resolve the block while retaining the required constants once each, then verify 
the Chinese-locale build.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to