Caideyipi commented on code in PR #18874:
URL: https://github.com/apache/iotdb/pull/18874#discussion_r4269822165
##########
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/schema/TreeDeviceViewFieldDetector.java:
##########
@@ -64,98 +65,191 @@ public class TreeDeviceViewFieldDetector {
private final ConfigManager configManager;
private final PartialPath path;
private final TsTable table;
- private final Map<String, Set<FieldColumnSchema>> fields;
private TDeviceViewResp result = new TDeviceViewResp(StatusUtils.OK, new
ConcurrentHashMap<>());
private final Map<String, String> lowerCase2OriginalMap = new HashMap<>();
- public TreeDeviceViewFieldDetector(
- final ConfigManager configManager,
- final TsTable table,
- final Map<String, Set<FieldColumnSchema>> fields) {
+ public TreeDeviceViewFieldDetector(final ConfigManager configManager, final
TsTable table) {
this.configManager = configManager;
this.path = TreeViewSchema.getPrefixPattern(table);
this.table = table;
- this.fields = fields;
}
- public TSStatus detectMissingFieldTypes() {
- if (table.getFieldNum() == 0 && Objects.isNull(fields)) {
+ /**
+ * Fills the fields of the view which is being created: a view may declare
no field at all, in
+ * which case every field under its tree path pattern becomes a column, or
declare fields whose
+ * type is left to the detection.
+ */
+ public TSStatus detectFieldsForCreation() {
+ if (table.getFieldNum() == 0) {
+ // for the first time creation of the table, user do not assign any
field;
+ // just detect all the fields of prefix path
new TreeDeviceViewFieldDetectionTaskExecutor(
configManager,
getLatestSchemaRegionMap(),
table.getTagNum(),
- TreeViewSchema.isRestrict(table))
+ TreeViewSchema.isRestrict(table),
+ null,
+ false)
.execute();
if (result.getStatus().getCode() !=
TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
return result.getStatus();
}
result
.getDeviewViewFieldTypeMap()
.forEach(
- (field, type) -> {
- final FieldColumnSchema columnSchema =
- new FieldColumnSchema(field,
TSDataType.getTsDataType(type));
- if (!field.equals(lowerCase2OriginalMap.get(field))) {
- TreeViewSchema.setOriginalName(columnSchema,
lowerCase2OriginalMap.get(field));
- }
- table.addColumnSchema(columnSchema);
- });
- } else {
- final Map<String, Set<FieldColumnSchema>> unknownFields;
- if (Objects.isNull(fields)) {
- unknownFields = new HashMap<>();
- for (final TsTableColumnSchema schema : table.getColumnList()) {
- if (!(schema instanceof FieldColumnSchema)
- || schema.getDataType() != TSDataType.UNKNOWN) {
- continue;
- }
- final String key = TreeViewSchema.getSourceName(schema);
- if (!unknownFields.containsKey(key)) {
- unknownFields.put(key, new HashSet<>());
- }
- unknownFields.get(key).add((FieldColumnSchema) schema);
- }
- } else {
- unknownFields = fields;
- }
+ (field, type) ->
+
table.addColumnSchema(toFieldColumnSchemaWithOriginName(field, type)));
+ return StatusUtils.OK;
+ }
+
+ return detectFieldTypes(collectUnknownFieldColumns());
+ }
+
+ /**
+ * Resolves the type of the field columns which are being added to an
existing view. The columns
+ * to resolve are given by the caller, as only it knows which ones it is
adding.
+ */
+ public TSStatus detectFieldTypesForColumnAddition(final Map<String,
FieldColumnSchema> fields) {
+ return detectFieldTypes(fields);
+ }
+
+ /**
+ * The fields which exist under the view's tree path pattern but are not
columns of the view yet,
+ * as the columns to add. Used after a view is created, to pick up the
series which were created
+ * before the view was mounted on the DataNodes, and which the DataNodes
therefore never reported.
+ */
+ public Pair<TSStatus, List<TsTableColumnSchema>>
detectNewFieldColumnsAfterCreation() {
+ new TreeDeviceViewFieldDetectionTaskExecutor(
+ configManager,
+ getLatestSchemaRegionMap(),
+ table.getTagNum(),
+ TreeViewSchema.isRestrict(table),
+ null,
+ true)
+ .execute();
+ if (result.getStatus().getCode() !=
TSStatusCode.SUCCESS_STATUS.getStatusCode()) {
+ return new Pair<>(result.getStatus(), Collections.emptyList());
+ }
+ final List<TsTableColumnSchema> columns = new ArrayList<>();
+ result
+ .getDeviewViewFieldTypeMap()
+ .forEach(
+ (field, type) -> {
+ if (Objects.isNull(table.getColumnSchema(field))) {
+ columns.add(toFieldColumnSchemaWithOriginName(field, type));
Review Comment:
**[P1] Compare source measurements when deciding which columns are missing**
A field already mapped through `FROM` is treated as missing because this
checks only its table column name. With only `root.db.d.s1` present, creating
`v(device TAG, renamed INT32 FIELD FROM s1) WITH (auto_sync_schema=true) AS
root.db.**` makes the catch-up scan emit another column named `s1`.
`checkColumnSchemaForTree2TableView()` then removes that candidate because its
source is already referenced by `renamed`, and the empty addition fails with
`COLUMN_ALREADY_EXISTS`. The added `AddViewColumnProcedure` is a child of the
create procedure, so this failure triggers rollback of an otherwise valid view
creation; turning auto-sync on has the same problem. Exclude fields whose
source name is already represented before submitting the child, and make an
already-satisfied reconciliation a success.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/protocol/thrift/impl/DataNodeInternalRPCServiceImpl.java:
##########
@@ -2890,6 +2902,24 @@ public TSStatus updateTemplate(final TUpdateTemplateReq
req) {
SchemaEngine.getInstance()
.updateSubtreeMeasurementCountForTemplate(newTemplate.getId(),
delta);
}
+ if (TreeToTableViewCache.getInstance().hasView()) {
+ // addViewColumnsIfNeeded, which collects the views from the index
and asks CN to add the
+ // columns.
+ try {
+ final Map<PartialPath, List<Pair<String, TSDataType>>>
device2Measurements =
+ collectDevicesUsingExtendedTemplate(oldTemplate, newTemplate);
+ TriggerAddViewColumnVisitor.addViewColumnsIfNeeded(
+ device2Measurements, ALTER_DEVICE_TEMPLATE_ID_PREFIX +
newTemplate.getId());
Review Comment:
**[P1] Run template-extension reconciliation after releasing the ConfigNode
schema lock**
For an activated template covered by an auto-sync view, this synchronous
callback creates a lock cycle. `ClusterSchemaManager.extendSchemaTemplate()` is
synchronized and waits for the DataNode `UPDATE_TEMPLATE` responses while
holding its monitor. This DataNode handler calls `updateTree2TableViews` and
waits for `AddViewColumnProcedure`, whose `columnCheck()` calls the
synchronized `tableColumnCheckForColumnExtension()` on the same manager. That
procedure cannot progress until template extension returns, and template
extension cannot return until this callback finishes. A healthy extension
therefore waits for an RPC or the 30-second procedure timeout to break the
cycle, blocking other operations that need the manager lock. Schedule
reconciliation after the update response, or perform it on ConfigNode outside
that monitor.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/table/TreeToTableViewCache.java:
##########
@@ -0,0 +1,465 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.schemaengine.table;
+
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.path.PathPatternUtil;
+import org.apache.iotdb.commons.schema.node.utils.IMNodeFactory;
+import org.apache.iotdb.commons.schema.table.TreeViewSchema;
+import org.apache.iotdb.commons.schema.table.TsTable;
+import org.apache.iotdb.db.i18n.DataNodeSchemaMessages;
+import
org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.mem.mnode.IMemMNode;
+import
org.apache.iotdb.db.schemaengine.schemaregion.mtree.loader.MNodeFactoryLoader;
+import
org.apache.iotdb.db.schemaengine.schemaregion.view.TreeToTableViewReference;
+
+import org.apache.tsfile.utils.Pair;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+import static
org.apache.iotdb.commons.conf.IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD;
+import static org.apache.iotdb.commons.conf.IoTDBConstant.PATH_ROOT;
+
+/**
+ * The index of the tree to table views on DataNode, keyed by the fixed prefix
of each view's tree
+ * path pattern.
+ *
+ * <p>Only DataNodeTableCache writes this index, and it does so while holding
its own write lock, so
+ * every write here also holds the write lock of this cache and the lock order
is always
+ * DataNodeTableCache's write lock and then this cache's write lock. A reader
only takes this
+ * cache's read lock, so the two locks can never form a cycle.
+ */
+public class TreeToTableViewCache implements ITreeToTableViewCache {
+
+ private static final Logger LOGGER =
LoggerFactory.getLogger(TreeToTableViewCache.class);
+
+ private final IMNodeFactory<IMemMNode> nodeFactory =
+ MNodeFactoryLoader.getInstance().getMemMNodeIMNodeFactory();
+ // This index describes path prefixes, not the database/device roles in a
SchemaRegion. clear()
+ // replaces it with a fresh one.
+ private IMemMNode rootNode = nodeFactory.createInternalMNode(null,
PATH_ROOT);
+
+ private final Map<IMemMNode, Set<TreeToTableViewReference>> viewReferences =
+ new IdentityHashMap<>();
+ private final Map<Pair<String, String>, PartialPath> viewPaths = new
HashMap<>();
+ private final ReentrantReadWriteLock readWriteLock = new
ReentrantReadWriteLock();
+
+ /**
+ * Whether any view is registered. A cluster without any view answers "no
view covers this path"
+ * without taking the lock or walking the trie. It is written only under the
write lock, and read
+ * without any lock.
+ */
+ private volatile boolean hasView = false;
+
+ private TreeToTableViewCache() {}
+
+ @Override
+ public void clear() {
+ readWriteLock.writeLock().lock();
+ try {
+ // A node is only reachable from its parent, so a fresh root drops the
whole trie at once.
+ rootNode = nodeFactory.createInternalMNode(null, PATH_ROOT);
+ viewReferences.clear();
+ viewPaths.clear();
+ hasView = false;
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+
+ @Override
+ public void init(final Map<String, List<TsTable>> initialTables) {
+ readWriteLock.writeLock().lock();
+ try {
+ // The index is rebuilt from scratch, so the entry of a view which is
gone does not survive.
+ clear();
+ initialTables.forEach(
+ (database, tables) -> tables.forEach(table -> registerView(database,
table)));
+ } finally {
+ readWriteLock.writeLock().unlock();
+ }
+ }
+
+ /**
+ * Registers the view at the path of its tree path pattern. The caller has
already picked the
+ * tables which belong in the index, see
DataNodeTableCache#collectInitialViews.
+ */
+ private void registerView(final String database, final TsTable table) {
+ final Optional<String> treePath =
table.getPropValue(TreeViewSchema.TREE_PATH_PATTERN);
+ if (treePath.isEmpty()) {
+ return;
+ }
+ try {
+ addView(
+ database,
+ table.getTableName(),
+ TreeViewSchema.isRestrict(table),
+ table.getTagNum(),
+ new PartialPath(treePath.get()));
+ } catch (final IllegalPathException e) {
+ LOGGER.warn(
+ DataNodeSchemaMessages
+
.LOG_FAILED_TO_REGISTER_THE_TREE_TO_TABLE_VIEW_ARG_ARG_DURING_INIT_ILLEGAL_PATH_E9D65F63,
+ database,
+ table.getTableName(),
+ e);
+ }
+ }
+
+ /**
+ * Register a view at a fixed prefix path.
+ *
+ * @parm Partial path, eg: root.db.dev01.**;
+ * @return true if a new reference was registered; false if it was already
present
+ */
+ // todo, log;
+ @Override
+ public boolean addView(
+ final String database,
+ final String tableName,
+ final boolean isRestrict,
+ final int numOfTags,
+ final PartialPath path)
+ throws IllegalPathException {
+ int prefixLength = checkPartialPath(path);
+ final String[] nodes = path.getNodes();
+ readWriteLock.writeLock().lock();
+ try {
+ final PartialPath registeredPath = viewPaths.get(new Pair<>(database,
tableName));
+ if (Objects.nonNull(registeredPath) && !Objects.equals(registeredPath,
path)) {
+ // A view is registered at exactly one path, so the entry left at the
previous path has to
+ // go, or removing this view by name later would miss it.
+ removeView(database, tableName);
+ }
+ viewPaths.put(new Pair<>(database, tableName), path);
+
+ IMemMNode current = rootNode;
+ for (int i = 1; i < prefixLength; i++) {
+ IMemMNode child = current.getChild(nodes[i]);
+ if (child == null) {
+ child = current.addChild(nodes[i],
nodeFactory.createInternalMNode(current, nodes[i]));
+ }
+ current = child;
+ }
+
+ final boolean added =
+ viewReferences
+ .computeIfAbsent(current, node -> new HashSet<>())
+ .add(new TreeToTableViewReference(database, tableName,
isRestrict, numOfTags));
Review Comment:
**[P2] Replace matching metadata when a view is re-registered at the same
path**
`TreeToTableViewReference.equals()` compares only database and table names,
so `HashSet.add()` keeps the old reference when `CREATE OR REPLACE VIEW`
registers the same view at the same tree path. The old entry is removed above
only if the path changed; changes to TAG count or restrict therefore never
reach this index. For example, replacing a restricted one-TAG view at
`root.db.**` with a restricted two-TAG view still leaves `root.db.site.device`
uncovered, so subsequent measurements are never synchronized. Changing restrict
can also leave devices incorrectly covered. Both cases fail in regression tests
against this commit. Replace the existing reference for that identity even when
the path is unchanged.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/consensus/statemachine/schemaregion/SchemaExecutionVisitor.java:
##########
@@ -105,6 +112,8 @@ public class SchemaExecutionVisitor implements
PlanVisitor<TSStatus, ISchemaRegi
@Override
public TSStatus visitCreateTimeSeries(
final CreateTimeSeriesNode node, final ISchemaRegion schemaRegion) {
+ triggerAddViewColumn(node, null);
+
try {
schemaRegion.createTimeSeries(node, -1);
Review Comment:
**[P2] Add view columns only for schema changes that actually succeeded**
The trigger commits the view column through ConfigNode before
`schemaRegion.createTimeSeries()` runs. The tree creation can still fail here,
for example with `SeriesOverflowException` from `SchemaRegionMemoryImpl` or a
late alias/path conflict, but the view addition has already committed and is
never rolled back. An unsuccessful attempt to create `s2 INT32` can therefore
leave an `INT32` column even though `s2` does not exist; a later successful
creation of `s2 FLOAT` is ignored because column validation removes existing
names, leaving the view with the wrong type. The same ordering exists in the
aligned, multi-series and template activation paths. Trigger reconciliation
after successful creation and use only the measurements that succeeded in a
partially successful batch.
##########
iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ProcedureMessages.java:
##########
@@ -1511,8 +1511,17 @@ private ProcedureMessages() {}
public static final String MESSAGE_ROLLING_BACK_AFTER_FAILURE_ARG_474DF456 =
"正在回滚,失败原因:%s。";
public static final String
MESSAGE_ROLLING_BACK_AFTER_AN_EARLIER_FAILURE_850D0AF5 =
"正在回滚此前发生的失败。";
+<<<<<<< HEAD
public static final String
EXCEPTION_DATANODE_SETSYSTEMSTATUS_RPC_FAILED_ARG_ARG_ARG_7E8E5F9F =
Review Comment:
**[P1] Resolve the merge conflict and duplicate constant before building the
zh locale**
This Java source still contains `<<<<<<< HEAD`, `=======`, and `>>>>>>>`
markers. The `with-zh-locale` profile includes this file, so the Chinese build
cannot compile: `javac -encoding UTF-8` reports seven syntax errors in this
block. Removing only the markers still fails because the commit-release logging
constant introduced at line 1521 is already declared at line 1141. Please
resolve the block while retaining the required constants once each, then verify
the Chinese-locale build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]