date:20240423

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577310909


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java:
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import 
org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Modification;
+import 
org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex;
+import org.apache.iotdb.tsfile.file.metadata.IDeviceID;
+import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import static 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED;
+
+public class SettleSelectorImpl implements ISettleSelector {
+  private static final Logger LOGGER =
+  LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
+
+  private final boolean heavySelect;
+  private final String storageGroupName;
+  private final String dataRegionId;
+  private final long timePartition;
+  private final TsFileManager tsFileManager;
+  private boolean isSeq;
+
+  public SettleSelectorImpl(
+  boolean heavySelect,
+  String storageGroupName,
+  String dataRegionId,
+  long timePartition,
+  TsFileManager tsFileManager) {
+this.heavySelect = heavySelect;
+this.storageGroupName = storageGroupName;
+this.dataRegionId = dataRegionId;
+this.timePartition = timePartition;
+this.tsFileManager = tsFileManager;
+  }
+
+  static class AllDirtyResource {
+List resources = new ArrayList<>();
+
+public void add(TsFileResource resource) {
+  resources.add(resource);
+}
+
+public List getResources() {
+  return resources;
+}
+  }
+
+  static class PartialDirtyResource {
+List resources = new ArrayList<>();
+long totalFileSize = 0;
+
+public boolean add(TsFileResource resource, long dirtyDataSize) {
+  resources.add(resource);
+  totalFileSize += resource.getTsFileSize();
+  totalFileSize -= dirtyDataSize;
+  return checkHasReachedThreshold();
+}
+
+public List getResources() {
+  return resources;
+}
+
+public boolean checkHasReachedThreshold() {
+  return resources.size() >= config.getFileLimitPerInnerTask()
+  || totalFileSize >= config.getTargetCompactionFileSize();
+}
+  }
+
+  @Override
+  public List selectSettleTask(List 
tsFileResources) {
+if

Re: [PR] Try to fix timezone [iotdb]

2024-04-23 Thread via GitHub



sonarcloud[bot] commented on PR #12401:
URL: https://github.com/apache/iotdb/pull/12401#issuecomment-2074096303

   ## [![Quality Gate 
Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png
 'Quality Gate 
Passed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12401) 
**Quality Gate passed**  
   Issues  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 New 
issues](https://sonarcloud.io/project/issues?id=apache_iotdb=12401=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png
 '') [0 Accepted 
issues](https://sonarcloud.io/component_measures?id=apache_iotdb=12401=new_accepted_issues=list)
   
   Measures  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png
 '') [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_iotdb=12401=false=true)
  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Coverage  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png
 '') No data about Duplication  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12401)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Pipe: Added batch to schema snapshot execution in template activation and timeseries creation & Fixed the bug that the "CreateMultiTimeSeries" group with alias is not idempotent [iotdb]

2024-04-23 Thread via GitHub



SteveYurongSu merged PR #12380:
URL: https://github.com/apache/iotdb/pull/12380


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577222419


##
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/CompactionSchedulerWithFastPerformerTest.java:
##
@@ -154,7 +154,7 @@ public void test1() throws IOException, MetadataException, 
InterruptedException
 .setTargetCompactionFileSize(2L * 1024L * 1024L * 1024L);
 String sgName = COMPACTION_TEST_SG + "test1";
 try {
-  CompactionTaskManager.getInstance().restart();
+  // CompactionTaskManager.getInstance().restart();

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Pipe: Two stage aggregate counter [iotdb]

2024-04-23 Thread via GitHub



sonarcloud[bot] commented on PR #12328:
URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2073942021

   ## [![Quality Gate 
Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png
 'Quality Gate 
Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) 
**Quality Gate failed**  
   Failed conditions  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [0.0% Coverage on New 
Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12328=new_coverage=list)
 (required ≥ 80%)  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [B Reliability Rating on New 
Code](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) 
(required ≥ A)  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12328)
   
   ##   
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png
 '') Catch issues before they fail your Quality Gate with our IDE extension 
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png
 '') 
[SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577180788


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java:
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.schema.ttl;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+@NotThreadSafe
+public class TTLCache {
+
+  private final CacheNode ttlCacheTree;
+  public static final long NULL_TTL = -1;
+
+  private int ttlCount;
+
+  public TTLCache() {
+ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT);
+long defaultTTL =
+CommonDateTimeUtils.convertMilliTimeWithPrecision(
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(),
+
CommonDescriptor.getInstance().getConfig().getTimestampPrecision());
+defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL;
+ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL);
+ttlCount = 1;
+  }
+
+  /**
+   * Put ttl into cache tree.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+  public void setTTL(String[] nodes, long ttl) {
+if (nodes.length < 2 || ttl <= 0) {
+  return;
+}
+CacheNode parent = ttlCacheTree;
+for (int i = 1; i < nodes.length; i++) {
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+parent.addChild(nodes[i], NULL_TTL);
+child = parent.getChild(nodes[i]);
+  }
+  parent = child;
+}
+if (parent.ttl == NULL_TTL) {
+  ttlCount++;
+}
+parent.ttl = ttl;
+  }
+
+  /**
+   * Unset ttl and remove all useless nodes. If the path to be removed is 
internal node, then just
+   * reset its ttl. Else, find the sub path and remove it.
+   *
+   * @param nodes path to be removed
+   */
+  public void unsetTTL(String[] nodes) {
+if (nodes.length < 2) {
+  return;
+} else if (nodes.length == 2) {
+  // if path equals to root.**, then unset it to configured ttl
+  if (nodes[0].equals(IoTDBConstant.PATH_ROOT)
+  && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) {
+ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl =
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs();
+return;
+  }
+}
+CacheNode parent = ttlCacheTree;
+int index = 0;
+boolean usefulFlag;
+CacheNode parentOfSubPathToBeRemoved = null;
+for (int i = 1; i < nodes.length; i++) {
+  usefulFlag = !parent.getChildren().isEmpty() || parent.ttl != NULL_TTL;
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+// there is no matching path on ttl cache tree
+return;
+  }
+  if (usefulFlag) {
+parentOfSubPathToBeRemoved = parent;
+index = i;
+  }
+  parent = child;
+}
+// currently, parent is the leaf node of the path to be removed
+if (parent.ttl != NULL_TTL) {
+  ttlCount--;
+}
+
+if (!parent.getChildren().isEmpty()) {
+  // node to be removed is internal node, then just reset its ttl
+  parent.ttl = NULL_TTL;
+  return;
+}
+
+// node to be removed is leaf node, then remove corresponding node of this 
path from cache tree
+if (parentOfSubPathToBeRemoved != null) {
+  parentOfSubPathToBeRemoved.removeChild(nodes[index]);
+}
+  }
+
+  /**
+   * Get ttl from cache tree. Return the TTL of the node closest to the path 
leaf node that has a
+   * TTL which is not NULL_TTL.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577175083


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java:
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.schema.ttl;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+@NotThreadSafe
+public class TTLCache {
+
+  private final CacheNode ttlCacheTree;
+  public static final long NULL_TTL = -1;
+
+  private int ttlCount;
+
+  public TTLCache() {
+ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT);
+long defaultTTL =
+CommonDateTimeUtils.convertMilliTimeWithPrecision(
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(),
+
CommonDescriptor.getInstance().getConfig().getTimestampPrecision());
+defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL;
+ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL);
+ttlCount = 1;
+  }
+
+  /**
+   * Put ttl into cache tree.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+  public void setTTL(String[] nodes, long ttl) {
+if (nodes.length < 2 || ttl <= 0) {
+  return;
+}
+CacheNode parent = ttlCacheTree;
+for (int i = 1; i < nodes.length; i++) {
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+parent.addChild(nodes[i], NULL_TTL);
+child = parent.getChild(nodes[i]);
+  }
+  parent = child;
+}
+if (parent.ttl == NULL_TTL) {
+  ttlCount++;
+}
+parent.ttl = ttl;
+  }
+
+  /**
+   * Unset ttl and remove all useless nodes. If the path to be removed is 
internal node, then just
+   * reset its ttl. Else, find the sub path and remove it.
+   *
+   * @param nodes path to be removed
+   */
+  public void unsetTTL(String[] nodes) {
+if (nodes.length < 2) {
+  return;
+} else if (nodes.length == 2) {
+  // if path equals to root.**, then unset it to configured ttl
+  if (nodes[0].equals(IoTDBConstant.PATH_ROOT)
+  && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) {
+ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl =
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs();
+return;
+  }
+}
+CacheNode parent = ttlCacheTree;
+int index = 0;
+boolean usefulFlag;

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577172088


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java:
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.schema.ttl;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+@NotThreadSafe
+public class TTLCache {
+
+  private final CacheNode ttlCacheTree;
+  public static final long NULL_TTL = -1;
+
+  private int ttlCount;
+
+  public TTLCache() {
+ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT);
+long defaultTTL =
+CommonDateTimeUtils.convertMilliTimeWithPrecision(
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(),
+
CommonDescriptor.getInstance().getConfig().getTimestampPrecision());
+defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL;
+ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL);
+ttlCount = 1;
+  }
+
+  /**
+   * Put ttl into cache tree.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+  public void setTTL(String[] nodes, long ttl) {
+if (nodes.length < 2 || ttl <= 0) {
+  return;
+}
+CacheNode parent = ttlCacheTree;
+for (int i = 1; i < nodes.length; i++) {
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+parent.addChild(nodes[i], NULL_TTL);
+child = parent.getChild(nodes[i]);
+  }
+  parent = child;
+}
+if (parent.ttl == NULL_TTL) {
+  ttlCount++;
+}
+parent.ttl = ttl;
+  }
+
+  /**
+   * Unset ttl and remove all useless nodes. If the path to be removed is 
internal node, then just
+   * reset its ttl. Else, find the sub path and remove it.

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1577167260


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java:
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.schema.ttl;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+@NotThreadSafe
+public class TTLCache {
+
+  private final CacheNode ttlCacheTree;
+  public static final long NULL_TTL = -1;
+
+  private int ttlCount;
+
+  public TTLCache() {
+ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT);
+long defaultTTL =
+CommonDateTimeUtils.convertMilliTimeWithPrecision(
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(),
+
CommonDescriptor.getInstance().getConfig().getTimestampPrecision());
+defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL;
+ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL);
+ttlCount = 1;
+  }
+
+  /**
+   * Put ttl into cache tree.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+  public void setTTL(String[] nodes, long ttl) {
+if (nodes.length < 2 || ttl <= 0) {
+  return;
+}
+CacheNode parent = ttlCacheTree;
+for (int i = 1; i < nodes.length; i++) {
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+parent.addChild(nodes[i], NULL_TTL);
+child = parent.getChild(nodes[i]);
+  }
+  parent = child;
+}
+if (parent.ttl == NULL_TTL) {
+  ttlCount++;
+}
+parent.ttl = ttl;
+  }
+
+  /**
+   * Unset ttl and remove all useless nodes. If the path to be removed is 
internal node, then just
+   * reset its ttl. Else, find the sub path and remove it.
+   *
+   * @param nodes path to be removed
+   */
+  public void unsetTTL(String[] nodes) {
+if (nodes.length < 2) {
+  return;
+} else if (nodes.length == 2) {
+  // if path equals to root.**, then unset it to configured ttl
+  if (nodes[0].equals(IoTDBConstant.PATH_ROOT)
+  && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) {
+ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl =
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs();

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Pipe: Fixed the bug that pure data transfer is forbidden when SchemaRegion is in Simple consensus [iotdb]

2024-04-23 Thread via GitHub



SteveYurongSu merged PR #12397:
URL: https://github.com/apache/iotdb/pull/12397


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] update Export/ImportDataTestIT [iotdb]

2024-04-23 Thread via GitHub



2b3c511 opened a new pull request, #12399:
URL: https://github.com/apache/iotdb/pull/12399

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Enhance estimate of FI memory usage [iotdb]

2024-04-23 Thread via GitHub



JackieTien97 commented on code in PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#discussion_r1577089745


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/PlanNodeType.java:
##
@@ -209,6 +209,8 @@ public enum PlanNodeType {
   EXPLAIN_ANALYZE((short) 90),
 
   PIPE_OPERATE_SCHEMA_QUEUE_REFERENCE((short) 91),
+
+  FULL_OUTER_TIME_JOIN_REFERENCE((short) 92),

Review Comment:
   what's this for?



##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/LocalExecutionPlanContext.java:
##
@@ -91,6 +98,10 @@ public class LocalExecutionPlanContext {
   // use AtomicReference not for thread-safe, just for updating same field in 
different pipeline
   private AtomicReference> timePartitions = new AtomicReference<>();
 
+  /** Records the parent of each pipeline. The order of each list does not 
matter for now. */
+  private Map> 
parentPlanNodeIdToMemoryEstimator =

Review Comment:
   remember to set this map to null, after you check the memory, this Map may 
occupy huge memory when there are many devices;



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Implement new plan optimizer: AggregationPushDown [iotdb]

2024-04-23 Thread via GitHub



liuminghui233 opened a new pull request, #12398:
URL: https://github.com/apache/iotdb/pull/12398

   see doc https://apache-iotdb.feishu.cn/docx/JiREdjYz3oBjrJxVozXcgs9inS1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Pipe: Two stage aggregate counter [iotdb]

2024-04-23 Thread via GitHub



sonarcloud[bot] commented on PR #12328:
URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2072746986

   ## [![Quality Gate 
Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png
 'Quality Gate 
Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) 
**Quality Gate failed**  
   Failed conditions  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [0.0% Coverage on New 
Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12328=new_coverage=list)
 (required ≥ 80%)  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [B Reliability Rating on New 
Code](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) 
(required ≥ A)  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12328)
   
   ##   
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png
 '') Catch issues before they fail your Quality Gate with our IDE extension 
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png
 '') 
[SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576397733


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java:
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.schema.ttl;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+@NotThreadSafe
+public class TTLCache {
+
+  private final CacheNode ttlCacheTree;
+  public static final long NULL_TTL = -1;
+
+  private int ttlCount;
+
+  public TTLCache() {
+ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT);
+long defaultTTL =
+CommonDateTimeUtils.convertMilliTimeWithPrecision(
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(),
+
CommonDescriptor.getInstance().getConfig().getTimestampPrecision());
+defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL;
+ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL);
+ttlCount = 1;
+  }
+
+  /**
+   * Put ttl into cache tree.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+  public void setTTL(String[] nodes, long ttl) {
+if (nodes.length < 2 || ttl <= 0) {
+  return;
+}
+CacheNode parent = ttlCacheTree;
+for (int i = 1; i < nodes.length; i++) {
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+parent.addChild(nodes[i], NULL_TTL);
+child = parent.getChild(nodes[i]);
+  }
+  parent = child;
+}
+if (parent.ttl == NULL_TTL) {
+  ttlCount++;
+}
+parent.ttl = ttl;
+  }
+
+  /**
+   * Unset ttl and remove all useless nodes. If the path to be removed is 
internal node, then just
+   * reset its ttl. Else, find the sub path and remove it.
+   *
+   * @param nodes path to be removed
+   */
+  public void unsetTTL(String[] nodes) {
+if (nodes.length < 2) {
+  return;
+} else if (nodes.length == 2) {
+  // if path equals to root.**, then unset it to configured ttl
+  if (nodes[0].equals(IoTDBConstant.PATH_ROOT)
+  && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) {
+ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl =
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs();
+return;
+  }
+}
+CacheNode parent = ttlCacheTree;
+int index = 0;
+boolean usefulFlag;
+CacheNode parentOfSubPathToBeRemoved = null;
+for (int i = 1; i < nodes.length; i++) {
+  usefulFlag = !parent.getChildren().isEmpty() || parent.ttl != NULL_TTL;
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+// there is no matching path on ttl cache tree
+return;
+  }
+  if (usefulFlag) {
+parentOfSubPathToBeRemoved = parent;
+index = i;
+  }
+  parent = child;
+}
+// currently, parent is the leaf node of the path to be removed
+if (parent.ttl != NULL_TTL) {
+  ttlCount--;
+}
+
+if (!parent.getChildren().isEmpty()) {
+  // node to be removed is internal node, then just reset its ttl
+  parent.ttl = NULL_TTL;
+  return;
+}
+
+// node to be removed is leaf node, then remove corresponding node of this 
path from cache tree
+if (parentOfSubPathToBeRemoved != null) {
+  parentOfSubPathToBeRemoved.removeChild(nodes[index]);
+}
+  }
+
+  /**
+   * Get ttl from cache tree. Return the TTL of the node closest to the path 
leaf node that has a
+   * TTL which is not NULL_TTL.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+

[PR] Pipe: Fixed the bug that pure data transfer is forbidden when SchemaRegion is in Simple consensus [iotdb]

2024-04-23 Thread via GitHub



Caideyipi opened a new pull request, #12397:
URL: https://github.com/apache/iotdb/pull/12397

   ## Description
   As the title said.
   
   
   
   
   
   
   
   
   
   
   
   
   This PR has:
   - [ ] been self-reviewed.
   - [ ] concurrent read
   - [ ] concurrent write
   - [ ] concurrent read and write 
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. 
   - [ ] added or updated version, __license__, or notice information
   - [ ] added comments explaining the "why" and the intent of the code 
wherever would not be obvious 
 for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, 
ensuring the threshold 
 for code coverage.
   - [ ] added integration tests.
   - [ ] been tested in a test IoTDB cluster.
   
   
   
   
   
   # Key changed/added classes (or packages if there are too many classes) 
in this PR
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Pipe: Two stage aggregate counter [iotdb]

2024-04-23 Thread via GitHub



sonarcloud[bot] commented on PR #12328:
URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2072023828

   ## [![Quality Gate 
Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png
 'Quality Gate 
Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) 
**Quality Gate failed**  
   Failed conditions  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [0.0% Coverage on New 
Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12328=new_coverage=list)
 (required ≥ 80%)  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [B Reliability Rating on New 
Code](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) 
(required ≥ A)  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12328)
   
   ##   
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png
 '') Catch issues before they fail your Quality Gate with our IDE extension 
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png
 '') 
[SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Fixed backup data backend running issues [iotdb]

2024-04-23 Thread via GitHub



CloudWise-Lukemiao merged PR #12396:
URL: https://github.com/apache/iotdb/pull/12396


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Fix effective ratio calculation method [iotdb]

2024-04-23 Thread via GitHub



OneSizeFitsQuorum merged PR #12394:
URL: https://github.com/apache/iotdb/pull/12394


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576047456


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java:
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.schema.ttl;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+@NotThreadSafe
+public class TTLCache {
+
+  private final CacheNode ttlCacheTree;
+  public static final long NULL_TTL = -1;
+
+  private int ttlCount;
+
+  public TTLCache() {
+ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT);
+long defaultTTL =
+CommonDateTimeUtils.convertMilliTimeWithPrecision(
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(),
+
CommonDescriptor.getInstance().getConfig().getTimestampPrecision());
+defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL;
+ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL);
+ttlCount = 1;
+  }
+
+  /**
+   * Put ttl into cache tree.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+  public void setTTL(String[] nodes, long ttl) {
+if (nodes.length < 2 || ttl <= 0) {
+  return;
+}
+CacheNode parent = ttlCacheTree;
+for (int i = 1; i < nodes.length; i++) {
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+parent.addChild(nodes[i], NULL_TTL);
+child = parent.getChild(nodes[i]);
+  }
+  parent = child;
+}
+if (parent.ttl == NULL_TTL) {
+  ttlCount++;
+}
+parent.ttl = ttl;
+  }
+
+  /**
+   * Unset ttl and remove all useless nodes. If the path to be removed is 
internal node, then just
+   * reset its ttl. Else, find the sub path and remove it.
+   *
+   * @param nodes path to be removed
+   */
+  public void unsetTTL(String[] nodes) {
+if (nodes.length < 2) {
+  return;
+} else if (nodes.length == 2) {
+  // if path equals to root.**, then unset it to configured ttl
+  if (nodes[0].equals(IoTDBConstant.PATH_ROOT)
+  && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) {
+ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl =
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs();
+return;
+  }
+}
+CacheNode parent = ttlCacheTree;
+int index = 0;
+boolean usefulFlag;
+CacheNode parentOfSubPathToBeRemoved = null;
+for (int i = 1; i < nodes.length; i++) {
+  usefulFlag = !parent.getChildren().isEmpty() || parent.ttl != NULL_TTL;
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+// there is no matching path on ttl cache tree
+return;
+  }
+  if (usefulFlag) {
+parentOfSubPathToBeRemoved = parent;
+index = i;
+  }
+  parent = child;
+}
+// currently, parent is the leaf node of the path to be removed
+if (parent.ttl != NULL_TTL) {
+  ttlCount--;
+}
+
+if (!parent.getChildren().isEmpty()) {
+  // node to be removed is internal node, then just reset its ttl
+  parent.ttl = NULL_TTL;
+  return;
+}
+
+// node to be removed is leaf node, then remove corresponding node of this 
path from cache tree
+if (parentOfSubPathToBeRemoved != null) {
+  parentOfSubPathToBeRemoved.removeChild(nodes[index]);
+}
+  }
+
+  /**
+   * Get ttl from cache tree. Return the TTL of the node closest to the path 
leaf node that has a
+   * TTL which is not NULL_TTL.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576040934


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java:
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.schema.ttl;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+@NotThreadSafe
+public class TTLCache {
+
+  private final CacheNode ttlCacheTree;
+  public static final long NULL_TTL = -1;
+
+  private int ttlCount;
+
+  public TTLCache() {
+ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT);
+long defaultTTL =
+CommonDateTimeUtils.convertMilliTimeWithPrecision(
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(),
+
CommonDescriptor.getInstance().getConfig().getTimestampPrecision());
+defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL;
+ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL);
+ttlCount = 1;
+  }
+
+  /**
+   * Put ttl into cache tree.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+  public void setTTL(String[] nodes, long ttl) {
+if (nodes.length < 2 || ttl <= 0) {
+  return;
+}
+CacheNode parent = ttlCacheTree;
+for (int i = 1; i < nodes.length; i++) {
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+parent.addChild(nodes[i], NULL_TTL);
+child = parent.getChild(nodes[i]);
+  }
+  parent = child;
+}
+if (parent.ttl == NULL_TTL) {
+  ttlCount++;
+}
+parent.ttl = ttl;
+  }
+
+  /**
+   * Unset ttl and remove all useless nodes. If the path to be removed is 
internal node, then just
+   * reset its ttl. Else, find the sub path and remove it.
+   *
+   * @param nodes path to be removed
+   */
+  public void unsetTTL(String[] nodes) {
+if (nodes.length < 2) {
+  return;
+} else if (nodes.length == 2) {
+  // if path equals to root.**, then unset it to configured ttl
+  if (nodes[0].equals(IoTDBConstant.PATH_ROOT)
+  && nodes[1].equals(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)) {
+ttlCacheTree.getChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD).ttl =
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs();
+return;
+  }
+}
+CacheNode parent = ttlCacheTree;

Review Comment:
   Resolved. Rename to `current`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576037933


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java:
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.schema.ttl;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+@NotThreadSafe
+public class TTLCache {
+
+  private final CacheNode ttlCacheTree;
+  public static final long NULL_TTL = -1;
+
+  private int ttlCount;
+
+  public TTLCache() {
+ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT);
+long defaultTTL =
+CommonDateTimeUtils.convertMilliTimeWithPrecision(
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(),
+
CommonDescriptor.getInstance().getConfig().getTimestampPrecision());
+defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE : defaultTTL;
+ttlCacheTree.addChild(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD, defaultTTL);
+ttlCount = 1;
+  }
+
+  /**
+   * Put ttl into cache tree.
+   *
+   * @param nodes should be prefix path or specific device path without 
wildcard
+   */
+  public void setTTL(String[] nodes, long ttl) {
+if (nodes.length < 2 || ttl <= 0) {
+  return;
+}
+CacheNode parent = ttlCacheTree;
+for (int i = 1; i < nodes.length; i++) {
+  CacheNode child = parent.getChild(nodes[i]);
+  if (child == null) {
+parent.addChild(nodes[i], NULL_TTL);
+child = parent.getChild(nodes[i]);

Review Comment:
   Resolved. `addChild` can return the newly added node



##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/schema/ttl/TTLCache.java:
##
@@ -0,0 +1,275 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.schema.ttl;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.tsfile.utils.ReadWriteIOUtils;
+
+import javax.annotation.concurrent.NotThreadSafe;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.Objects;
+
+@NotThreadSafe
+public class TTLCache {
+
+  private final CacheNode ttlCacheTree;
+  public static final long NULL_TTL = -1;
+
+  private int ttlCount;
+
+  public TTLCache() {
+ttlCacheTree = new CacheNode(IoTDBConstant.PATH_ROOT);
+long defaultTTL =
+CommonDateTimeUtils.convertMilliTimeWithPrecision(
+CommonDescriptor.getInstance().getConfig().getDefaultTTLInMs(),
+
CommonDescriptor.getInstance().getConfig().getTimestampPrecision());
+defaultTTL = defaultTTL <= 0 ? Long.MAX_VALUE :

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576029400


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/exception/TTLException.java:
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.commons.exception;
+
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+
+public class TTLException extends Exception {
+
+  public TTLException(String path) {
+super(
+String.format(
+"Illegal pattern path: %s, pattern path should end with **, 
otherwise, it should be a specific database or device path without *",
+path));
+  }
+
+  public TTLException() {
+super(
+String.format(
+"The number of TTL stored in the system has reached threshold %d, 
please increase the ttl_count parameter.",
+
CommonDescriptor.getInstance().getConfig().getTTLCountThreshold()));
+  }
+}

Review Comment:
   Resolved, use `TTLIllegalPathException` and `TTLRuleCountNotEnoughException` 
class instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Fixed backup data backend running issues [iotdb]

2024-04-23 Thread via GitHub



CloudWise-Lukemiao closed pull request #12395: Fixed backup data backend 
running issues
URL: https://github.com/apache/iotdb/pull/12395


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Some region migration related work [iotdb]

2024-04-23 Thread via GitHub



OneSizeFitsQuorum merged PR #12376:
URL: https://github.com/apache/iotdb/pull/12376


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576021262


##
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java:
##
@@ -119,6 +119,9 @@ public class CommonConfig {
*/
   private long[] tierTTLInMs = {Long.MAX_VALUE};
 
+  /** The threshold for the number of TTL stored in the system, the default is 
1000. */
+  private int TTLCountThreshold = 1000;

Review Comment:
   Change to `TTLRuleCount`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576020607


##
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties:
##
@@ -450,6 +450,25 @@ data_replication_factor=1
 # Unit: ms
 # default_ttl_in_ms=-1
 
+# The threshold for the number of TTL stored in the system, the default is 
1000.
+# Negative value means the threshold is unlimited.
+# Datatype: int
+# ttl_count_threshold=1000
+
+# The interval of ttl check task in each database. Default is 2 hours.
+# Notice: It is not recommended to change it too small, as it will affect the 
read and write performance of the system.
+# Unit: ms
+# ttl_check_interval=720
+
+# The max expired time of device set with ttl. If the expired time exceeds 
this value, then the expired data will be cleaned by compaction. Default is 1 
month.
+# Notice: It is not recommended to change it too small, as it will affect the 
read and write performance of the system.
+# Unit: ms
+# max_expired_time=259200
+
+# The expired device rate. If the number of expired devices in one tsfile 
exceeds this value, then expired data of this tsfile will be cleaned by 
compaction.

Review Comment:
   Resolved.



##
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties:
##
@@ -450,6 +450,25 @@ data_replication_factor=1
 # Unit: ms
 # default_ttl_in_ms=-1
 
+# The threshold for the number of TTL stored in the system, the default is 
1000.
+# Negative value means the threshold is unlimited.
+# Datatype: int
+# ttl_count_threshold=1000
+
+# The interval of ttl check task in each database. Default is 2 hours.
+# Notice: It is not recommended to change it too small, as it will affect the 
read and write performance of the system.
+# Unit: ms
+# ttl_check_interval=720
+
+# The max expired time of device set with ttl. If the expired time exceeds 
this value, then the expired data will be cleaned by compaction. Default is 1 
month.
+# Notice: It is not recommended to change it too small, as it will affect the 
read and write performance of the system.
+# Unit: ms
+# max_expired_time=259200
+
+# The expired device rate. If the number of expired devices in one tsfile 
exceeds this value, then expired data of this tsfile will be cleaned by 
compaction.
+# Unit: float

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576019303


##
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties:
##
@@ -450,6 +450,25 @@ data_replication_factor=1
 # Unit: ms
 # default_ttl_in_ms=-1
 
+# The threshold for the number of TTL stored in the system, the default is 
1000.
+# Negative value means the threshold is unlimited.
+# Datatype: int
+# ttl_count_threshold=1000
+
+# The interval of ttl check task in each database. Default is 2 hours.
+# Notice: It is not recommended to change it too small, as it will affect the 
read and write performance of the system.
+# Unit: ms
+# ttl_check_interval=720
+
+# The max expired time of device set with ttl. If the expired time exceeds 
this value, then the expired data will be cleaned by compaction. Default is 1 
month.

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576014872


##
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties:
##
@@ -450,6 +450,25 @@ data_replication_factor=1
 # Unit: ms
 # default_ttl_in_ms=-1
 
+# The threshold for the number of TTL stored in the system, the default is 
1000.
+# Negative value means the threshold is unlimited.
+# Datatype: int
+# ttl_count_threshold=1000
+
+# The interval of ttl check task in each database. Default is 2 hours.
+# Notice: It is not recommended to change it too small, as it will affect the 
read and write performance of the system.

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576011751


##
iotdb-core/node-commons/src/assembly/resources/conf/iotdb-common.properties:
##
@@ -450,6 +450,25 @@ data_replication_factor=1
 # Unit: ms
 # default_ttl_in_ms=-1
 
+# The threshold for the number of TTL stored in the system, the default is 
1000.

Review Comment:
   Resolved. Change to `ttl_rule_count`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1576000190


##
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/settle/SettleCompactionRecoverTest.java:
##
@@ -0,0 +1,1137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.storageengine.dataregion.compaction.settle;
+
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.exception.MetadataException;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import 
org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.AbstractCompactionTest;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.constant.CompactionTaskType;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.impl.FastCompactionPerformer;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.subtask.FastCompactionTaskSummary;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.CompactionLogger;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.SimpleCompactionLogger;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.utils.CompactionFileGeneratorUtils;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.generator.TsFileNameGenerator;
+import org.apache.iotdb.tsfile.common.conf.TSFileDescriptor;
+import org.apache.iotdb.tsfile.exception.write.WriteProcessException;
+import org.apache.iotdb.tsfile.utils.Pair;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static 
org.apache.iotdb.tsfile.common.constant.TsFileConstant.PATH_SEPARATOR;
+
+public class SettleCompactionRecoverTest extends AbstractCompactionTest {
+  @Before
+  public void setUp()
+  throws IOException, WriteProcessException, MetadataException, 
InterruptedException {
+super.setUp();
+IoTDBDescriptor.getInstance().getConfig().setTargetChunkSize(512);
+IoTDBDescriptor.getInstance().getConfig().setTargetChunkPointNum(100);
+TSFileDescriptor.getInstance().getConfig().setMaxNumberOfPointsInPage(10);

Review Comment:
   It has been reset in `AbstractCompactionTest.teardown()`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575992167


##
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/TsFileResourceProgressIndexTest.java:
##
@@ -277,6 +278,7 @@ public void 
testProgressIndexMinimumProgressIndexTopologicalSort() {
 }
   }
 
+  @Ignore

Review Comment:
   I've removed this comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575990673


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java:
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import 
org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Modification;
+import 
org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex;
+import org.apache.iotdb.tsfile.file.metadata.IDeviceID;
+import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import static 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED;
+
+public class SettleSelectorImpl implements ISettleSelector {
+  private static final Logger LOGGER =
+  LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
+
+  private final boolean heavySelect;
+  private final String storageGroupName;
+  private final String dataRegionId;
+  private final long timePartition;
+  private final TsFileManager tsFileManager;
+  private boolean isSeq;
+
+  public SettleSelectorImpl(
+  boolean heavySelect,
+  String storageGroupName,
+  String dataRegionId,
+  long timePartition,
+  TsFileManager tsFileManager) {
+this.heavySelect = heavySelect;
+this.storageGroupName = storageGroupName;
+this.dataRegionId = dataRegionId;
+this.timePartition = timePartition;
+this.tsFileManager = tsFileManager;
+  }
+
+  static class AllDirtyResource {
+List resources = new ArrayList<>();
+
+public void add(TsFileResource resource) {
+  resources.add(resource);
+}
+
+public List getResources() {
+  return resources;
+}
+  }
+
+  static class PartialDirtyResource {
+List resources = new ArrayList<>();
+long totalFileSize = 0;
+
+public boolean add(TsFileResource resource, long dirtyDataSize) {
+  resources.add(resource);
+  totalFileSize += resource.getTsFileSize();
+  totalFileSize -= dirtyDataSize;
+  return checkHasReachedThreshold();
+}
+
+public List getResources() {
+  return resources;
+}
+
+public boolean checkHasReachedThreshold() {
+  return resources.size() >= config.getFileLimitPerInnerTask()
+  || totalFileSize >= config.getTargetCompactionFileSize();
+}
+  }
+
+  @Override
+  public List selectSettleTask(List 
tsFileResources) {
+if

[PR] Fix effective ratio calculation method [iotdb]

2024-04-23 Thread via GitHub



HeimingZ opened a new pull request, #12394:
URL: https://github.com/apache/iotdb/pull/12394

   Use wal node disk size instead of wal manager total size.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575929452


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java:
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import 
org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Modification;
+import 
org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex;
+import org.apache.iotdb.tsfile.file.metadata.IDeviceID;
+import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import static 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED;
+
+public class SettleSelectorImpl implements ISettleSelector {
+  private static final Logger LOGGER =
+  LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
+
+  private final boolean heavySelect;
+  private final String storageGroupName;
+  private final String dataRegionId;
+  private final long timePartition;
+  private final TsFileManager tsFileManager;
+  private boolean isSeq;
+
+  public SettleSelectorImpl(
+  boolean heavySelect,
+  String storageGroupName,
+  String dataRegionId,
+  long timePartition,
+  TsFileManager tsFileManager) {
+this.heavySelect = heavySelect;
+this.storageGroupName = storageGroupName;
+this.dataRegionId = dataRegionId;
+this.timePartition = timePartition;
+this.tsFileManager = tsFileManager;
+  }
+
+  static class AllDirtyResource {
+List resources = new ArrayList<>();
+
+public void add(TsFileResource resource) {
+  resources.add(resource);
+}
+
+public List getResources() {
+  return resources;
+}
+  }
+
+  static class PartialDirtyResource {
+List resources = new ArrayList<>();
+long totalFileSize = 0;

Review Comment:
   Resolved. `class AllDirtyResource` has been removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575918974


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java:
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import 
org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Modification;
+import 
org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex;
+import org.apache.iotdb.tsfile.file.metadata.IDeviceID;
+import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import static 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED;
+
+public class SettleSelectorImpl implements ISettleSelector {
+  private static final Logger LOGGER =
+  LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
+
+  private final boolean heavySelect;
+  private final String storageGroupName;
+  private final String dataRegionId;
+  private final long timePartition;
+  private final TsFileManager tsFileManager;
+  private boolean isSeq;
+
+  public SettleSelectorImpl(
+  boolean heavySelect,

Review Comment:
   Resolved. I've added comments at line 62.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575911991


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/SettleCompactionTask.java:
##
@@ -0,0 +1,389 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task;
+
+import org.apache.iotdb.db.service.metrics.FileMetrics;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.constant.CompactionTaskType;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.exception.CompactionRecoverException;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.CompactionLogAnalyzer;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.CompactionLogger;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.SimpleCompactionLogger;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.log.TsFileIdentifier;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.generator.TsFileNameGenerator;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * This settle task contains all_deleted files and partial_deleted files. The 
partial_deleted files
+ * are divided into several groups, each group may contain one or several 
files. This task will do
+ * the following two things respectively: 1. Settle all all_deleted files by 
deleting them directly.
+ * 2. Settle partial_deleted files: put the files of each partial_deleted 
group into an invisible
+ * innerCompactionTask, and then perform the cleanup work. The source files in 
a file group will be
+ * compacted into a target file.
+ */
+public class SettleCompactionTask extends InnerSpaceCompactionTask {
+  private List allDeletedFiles;
+  private double allDeletedFileSize = 0;
+  private double partialDeletedFileSize = 0;
+
+  private int allDeletedSuccessNum = 0;
+
+  private long totalModsFileSize;
+
+  public SettleCompactionTask(
+  long timePartition,
+  TsFileManager tsFileManager,
+  List allDeletedFiles,
+  List partialDeletedFiles,

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] iotdb 压缩线程处理异常之后，会无限处理错误数据。手动删除对应的序列也没用 [iotdb]

2024-04-23 Thread via GitHub



lintao666 commented on issue #12360:
URL: https://github.com/apache/iotdb/issues/12360#issuecomment-2071687177

   升级到1.3.1之后，这个压缩路径处理 还是有些问题。路径中包含root 无法压缩，但是对应的时间序列是存在的，也可以查询到数据。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Enhance estimatation of FI memory usage [iotdb]

2024-04-23 Thread via GitHub



codecov[bot] commented on PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2071684287

   ## 
[Codecov](https://app.codecov.io/gh/apache/iotdb/pull/12393?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 Report
   Attention: Patch coverage is `96.51163%` with `3 lines` in your changes are 
missing coverage. Please review.
   > Project coverage is 43.19%. Comparing base 
[(`f4804a4`)](https://app.codecov.io/gh/apache/iotdb/commit/f4804a4144076397e254208a150d2762ce716f31?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 to head 
[(`1b63668`)](https://app.codecov.io/gh/apache/iotdb/pull/12393?dropdown=coverage=pr=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache).
   > Report is 44 commits behind head on master.
   
   | 
[Files](https://app.codecov.io/gh/apache/iotdb/pull/12393?dropdown=coverage=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 | Patch % | Lines |
   |---|---|---|
   | 
[...ueryengine/plan/planner/LocalExecutionPlanner.java](https://app.codecov.io/gh/apache/iotdb/pull/12393?src=pr=tree=iotdb-core%2Fdatanode%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fiotdb%2Fdb%2Fqueryengine%2Fplan%2Fplanner%2FLocalExecutionPlanner.java_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache#diff-aW90ZGItY29yZS9kYXRhbm9kZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaW90ZGIvZGIvcXVlcnllbmdpbmUvcGxhbi9wbGFubmVyL0xvY2FsRXhlY3V0aW9uUGxhbm5lci5qYXZh)
 | 57.14% | [3 Missing :warning: 
](https://app.codecov.io/gh/apache/iotdb/pull/12393?src=pr=tree_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache)
 |
   
   Additional details and impacted files
   
   
   ```diff
   @@ Coverage Diff  @@
   ## master   #12393  +/-   ##
   
   - Coverage 46.40%   43.19%   -3.22% 
 Complexity   69   69  
   
 Files  3378 3088 -290 
 Lines207497   190358   -17139 
 Branches  2494822672-2276 
   
   - Hits  9629182221   -14070 
   + Misses   111206   108137-3069 
   ```
   
   
   
   
   
   [:umbrella: View full report in Codecov by 
Sentry](https://app.codecov.io/gh/apache/iotdb/pull/12393?dropdown=coverage=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache).
   
   :loudspeaker: Have feedback on the report? [Share it 
here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575827285


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/InnerSpaceCompactionTask.java:
##
@@ -283,30 +227,87 @@ protected boolean doCompaction() {
 String.format("%.2f", costTime),
 String.format("%.2f", selectedFileSize / 1024.0d / 1024.0d / 
costTime),
 summary);
-  } finally {
-Files.deleteIfExists(logFile.toPath());
-// may failed to set status if the status of target resource is DELETED
-targetTsFileResource.setStatus(TsFileResourceStatus.NORMAL);
   }
 } catch (Exception e) {
   isSuccess = false;
   handleException(LOGGER, e);
   recover();
 } finally {
   releaseAllLocks();
+  try {
+if (logFile != null) {
+  Files.deleteIfExists(logFile.toPath());
+}
+  } catch (IOException e) {
+handleException(LOGGER, e);
+  }
+  // may fail to set status if the status of target resource is DELETED
+  targetTsFileResource.setStatus(TsFileResourceStatus.NORMAL);
 }
 return isSuccess;
   }
 
+  protected void compact(SimpleCompactionLogger compactionLogger) throws 
Exception {
+// carry out the compaction
+targetTsFileList = new 
ArrayList<>(Collections.singletonList(targetTsFileResource));
+performer.setSourceFiles(selectedTsFileResourceList);
+// As elements in targetFiles may be removed in performer, we should use a 
mutable list
+// instead of Collections.singletonList()
+performer.setTargetFiles(targetTsFileList);
+performer.setSummary(summary);
+performer.perform();
+
+prepareTargetFiles();
+
+if (Thread.currentThread().isInterrupted() || summary.isCancel()) {
+  throw new InterruptedException(
+  String.format("%s-%s [Compaction] abort", storageGroupName, 
dataRegionId));
+}
+
+validateCompactionResult(
+sequence ? selectedTsFileResourceList : Collections.emptyList(),
+sequence ? Collections.emptyList() : selectedTsFileResourceList,
+targetTsFileList);
+
+// replace the old files with new file, the new is in same position as the 
old
+tsFileManager.replace(
+sequence ? selectedTsFileResourceList : Collections.emptyList(),
+sequence ? Collections.emptyList() : selectedTsFileResourceList,
+targetTsFileList,
+timePartition);
+
+// release the read lock of all source files, and get the write lock of 
them to delete them
+for (int i = 0; i < selectedTsFileResourceList.size(); ++i) {
+  selectedTsFileResourceList.get(i).writeLock();
+  isHoldingWriteLock[i] = true;
+}

Review Comment:
   Comments have been updated. This is old code, compaction now no longer 
requires adding and releasing read locks of source files.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575818839


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/analyze/cache/schema/DataNodeTTLCache.java:
##
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.queryengine.plan.analyze.cache.schema;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.schema.ttl.TTLCache;
+import org.apache.iotdb.commons.utils.TestOnly;
+
+import java.util.Map;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+public class DataNodeTTLCache {
+  private final TTLCache ttlCache;
+
+  private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
+
+  private DataNodeTTLCache() {
+ttlCache = new TTLCache();
+  }
+
+  public static DataNodeTTLCache getInstance() {
+return DataNodeTTLCacheHolder.INSTANCE;
+  }
+
+  private static class DataNodeTTLCacheHolder {
+private static final DataNodeTTLCache INSTANCE = new DataNodeTTLCache();
+  }
+
+  public void setTTL(String path, long ttl) {
+lock.writeLock().lock();
+try {
+  ttlCache.setTTL(path.split("\\" + IoTDBConstant.PATH_SEPARATOR), ttl);

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575815531


##
iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/persistence/TTLInfo.java:
##
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *  http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.confignode.persistence;
+
+import org.apache.iotdb.common.rpc.thrift.TSStatus;
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.schema.ttl.TTLCache;
+import org.apache.iotdb.commons.snapshot.SnapshotProcessor;
+import org.apache.iotdb.confignode.consensus.request.write.database.SetTTLPlan;
+import org.apache.iotdb.confignode.consensus.response.ttl.ShowTTLResp;
+import org.apache.iotdb.rpc.RpcUtils;
+import org.apache.iotdb.rpc.TSStatusCode;
+
+import org.apache.thrift.TException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.BufferedInputStream;
+import java.io.BufferedOutputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.locks.ReadWriteLock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+
+public class TTLInfo implements SnapshotProcessor {
+  private static final String SNAPSHOT_FILENAME = "ttl_info.bin";
+  private static final Logger LOGGER = LoggerFactory.getLogger(TTLInfo.class);
+
+  private final TTLCache ttlCache;
+
+  private final ReadWriteLock lock;
+
+  public TTLInfo() {
+ttlCache = new TTLCache();
+lock = new ReentrantReadWriteLock();
+  }
+
+  public TSStatus setTTL(SetTTLPlan plan) {
+lock.writeLock().lock();
+try {
+  ttlCache.setTTL(plan.getPathPattern(), plan.getTTL());
+  if (plan.isDataBase()) {
+// set ttl to path.**
+ttlCache.setTTL(
+new PartialPath(plan.getPathPattern())
+.concatNode(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)
+.getNodes(),
+plan.getTTL());
+  }
+} finally {
+  lock.writeLock().unlock();
+}
+return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS);
+  }
+
+  public TSStatus unsetTTL(SetTTLPlan plan) {
+lock.writeLock().lock();
+try {
+  ttlCache.unsetTTL(plan.getPathPattern());
+  if (plan.isDataBase()) {
+// unset ttl to path.**
+ttlCache.unsetTTL(
+new PartialPath(plan.getPathPattern())
+.concatNode(IoTDBConstant.MULTI_LEVEL_PATH_WILDCARD)
+.getNodes());
+  }
+} finally {
+  lock.writeLock().unlock();
+}
+return RpcUtils.getStatus(TSStatusCode.SUCCESS_STATUS);
+  }
+
+  public ShowTTLResp showAllTTL() {
+lock.readLock().lock();
+ShowTTLResp resp = new ShowTTLResp();
+try {
+  Map pathTTLMap = ttlCache.getAllPathTTL();
+  resp.setPathTTLMap(pathTTLMap);
+  resp.setStatus(new 
TSStatus(TSStatusCode.SUCCESS_STATUS.getStatusCode()));
+} finally {
+  lock.readLock().unlock();
+}
+return resp;
+  }

Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [rel/1.3] Fixed rest insertTablet http api lost data event [iotdb]

2024-04-23 Thread via GitHub



CloudWise-Lukemiao commented on PR #12386:
URL: https://github.com/apache/iotdb/pull/12386#issuecomment-2071648982

   http://127.0.0.1:18080/rest/v2/insertTablet
   When inserting with the following parameters, when there are duplicates in 
the timestamp and more than half of the values in the values are null
   
   
{"is_aligned":false,"device":"root.mqtt.topic.abc454","data_types":["TEXT","INT32","FLOAT"],"measurements":["foo","temp","weight"],"timestamps":[1713495895809,1713499707255,1713495893809,1713495892809,1713495891809,1713495890809,1713495889809,171349509,1713495887809,1713495893809,1713495892809,1713495891809,1713495890809,1713495889809,171349509],"values":[["bar",null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,41,32,33,34,39,38,37,36,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,87.0,87.3,87.3,87.0,87.3,87.3]]}
   
   
![image](https://github.com/apache/iotdb/assets/76942485/d6c24e27-e65c-4623-a01e-0382088758b0)
   
   
   After repair, as shown in the following figure：
   
![image](https://github.com/apache/iotdb/assets/76942485/ab2a1f10-c175-46a5-8879-ca28c1da0b5b)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575802400


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java:
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import 
org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Modification;
+import 
org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex;
+import org.apache.iotdb.tsfile.file.metadata.IDeviceID;
+import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import static 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED;
+
+public class SettleSelectorImpl implements ISettleSelector {
+  private static final Logger LOGGER =
+  LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
+
+  private final boolean heavySelect;
+  private final String storageGroupName;
+  private final String dataRegionId;
+  private final long timePartition;
+  private final TsFileManager tsFileManager;
+  private boolean isSeq;
+
+  public SettleSelectorImpl(
+  boolean heavySelect,
+  String storageGroupName,
+  String dataRegionId,
+  long timePartition,
+  TsFileManager tsFileManager) {
+this.heavySelect = heavySelect;
+this.storageGroupName = storageGroupName;
+this.dataRegionId = dataRegionId;
+this.timePartition = timePartition;
+this.tsFileManager = tsFileManager;
+  }
+
+  static class AllDirtyResource {
+List resources = new ArrayList<>();
+
+public void add(TsFileResource resource) {
+  resources.add(resource);
+}
+
+public List getResources() {
+  return resources;
+}
+  }
+
+  static class PartialDirtyResource {
+List resources = new ArrayList<>();
+long totalFileSize = 0;
+
+public boolean add(TsFileResource resource, long dirtyDataSize) {
+  resources.add(resource);
+  totalFileSize += resource.getTsFileSize();
+  totalFileSize -= dirtyDataSize;
+  return checkHasReachedThreshold();
+}
+
+public List getResources() {
+  return resources;
+}
+
+public boolean checkHasReachedThreshold() {
+  return resources.size() >= config.getFileLimitPerInnerTask()
+  || totalFileSize >= config.getTargetCompactionFileSize();
+}
+  }
+
+  @Override
+  public List selectSettleTask(List 
tsFileResources) {
+if

Re: [PR] Enhance estimatation of FI memory usage [iotdb]

2024-04-23 Thread via GitHub



sonarcloud[bot] commented on PR #12393:
URL: https://github.com/apache/iotdb/pull/12393#issuecomment-2071616292

   ## [![Quality Gate 
Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png
 'Quality Gate 
Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12393) 
**Quality Gate failed**  
   Failed conditions  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [0.0% Coverage on New 
Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12393=new_coverage=list)
 (required ≥ 80%)  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12393)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575781065


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/impl/SettleSelectorImpl.java:
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl;
+
+import org.apache.iotdb.commons.conf.IoTDBConstant;
+import org.apache.iotdb.commons.exception.IllegalPathException;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.utils.CommonDateTimeUtils;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import 
org.apache.iotdb.db.queryengine.plan.analyze.cache.schema.DataNodeTTLCache;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.performer.ICompactionPerformer;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.task.SettleCompactionTask;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.execute.utils.CompactionUtils;
+import 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.ISettleSelector;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Deletion;
+import org.apache.iotdb.db.storageengine.dataregion.modification.Modification;
+import 
org.apache.iotdb.db.storageengine.dataregion.modification.ModificationFile;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileManager;
+import org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResource;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.TsFileResourceStatus;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.DeviceTimeIndex;
+import 
org.apache.iotdb.db.storageengine.dataregion.tsfile.timeindex.ITimeIndex;
+import org.apache.iotdb.tsfile.file.metadata.IDeviceID;
+import org.apache.iotdb.tsfile.file.metadata.PlainDeviceID;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import static 
org.apache.iotdb.db.storageengine.dataregion.compaction.selector.impl.SettleSelectorImpl.DirtyStatus.PARTIAL_DELETED;
+
+public class SettleSelectorImpl implements ISettleSelector {
+  private static final Logger LOGGER =
+  LoggerFactory.getLogger(IoTDBConstant.COMPACTION_LOGGER_NAME);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
+
+  private final boolean heavySelect;
+  private final String storageGroupName;
+  private final String dataRegionId;
+  private final long timePartition;
+  private final TsFileManager tsFileManager;
+  private boolean isSeq;
+
+  public SettleSelectorImpl(
+  boolean heavySelect,
+  String storageGroupName,
+  String dataRegionId,
+  long timePartition,
+  TsFileManager tsFileManager) {
+this.heavySelect = heavySelect;
+this.storageGroupName = storageGroupName;
+this.dataRegionId = dataRegionId;
+this.timePartition = timePartition;
+this.tsFileManager = tsFileManager;
+  }
+
+  static class AllDirtyResource {
+List resources = new ArrayList<>();
+
+public void add(TsFileResource resource) {
+  resources.add(resource);
+}
+
+public List getResources() {
+  return resources;
+}
+  }
+
+  static class PartialDirtyResource {
+List resources = new ArrayList<>();
+long totalFileSize = 0;
+
+public boolean add(TsFileResource resource, long dirtyDataSize) {
+  resources.add(resource);
+  totalFileSize += resource.getTsFileSize();
+  totalFileSize -= dirtyDataSize;
+  return checkHasReachedThreshold();
+}
+
+public List getResources() {
+  return resources;
+}
+
+public boolean checkHasReachedThreshold() {
+  return resources.size() >= config.getFileLimitPerInnerTask()
+  || totalFileSize >= config.getTargetCompactionFileSize();
+}
+  }
+
+  @Override
+  public List selectSettleTask(List 
tsFileResources) {
+if

Re: [PR] Pipe: Two stage aggregate counter [iotdb]

2024-04-23 Thread via GitHub



sonarcloud[bot] commented on PR #12328:
URL: https://github.com/apache/iotdb/pull/12328#issuecomment-2071612225

   ## [![Quality Gate 
Failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-failed-20px.png
 'Quality Gate 
Failed')](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) 
**Quality Gate failed**  
   Failed conditions  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [0.0% Coverage on New 
Code](https://sonarcloud.io/component_measures?id=apache_iotdb=12328=new_coverage=list)
 (required ≥ 80%)  
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/failed-16px.png
 '') [B Reliability Rating on New 
Code](https://sonarcloud.io/dashboard?id=apache_iotdb=12328) 
(required ≥ A)  
 
   [See analysis details on 
SonarCloud](https://sonarcloud.io/dashboard?id=apache_iotdb=12328)
   
   ##   
   
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/light_bulb-16px.png
 '') Catch issues before they fail your Quality Gate with our IDE extension 
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/sonarlint-16px.png
 '') 
[SonarLint](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] Enhance estimatation of FI memory usage [iotdb]

2024-04-23 Thread via GitHub



lancelly opened a new pull request, #12393:
URL: https://github.com/apache/iotdb/pull/12393

   Pipelines can be organized into a tree structure. This PR is used to 
estimate the memory usage during the execution of a pipeline. Currently, it is 
assumed that in the most extreme cases, the child pipelines under a 
ConsumeAllChildren type pipeline will all run simultaneously. Therefore, when 
considering memory usage, all children must be accounted for. For 
ConsumeChildrenOneByOne type pipelines, only some of the child pipelines will 
run at the same time. Therefore, only the memory usage of these active children 
needs to be considered, along with the static memory occupied by the inactive 
children.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] feature add iotdb backup [iotdb]

2024-04-23 Thread via GitHub



HTHou merged PR #12365:
URL: https://github.com/apache/iotdb/pull/12365


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] fix: 下载换行样式 [iotdb-docs]

2024-04-23 Thread via GitHub



qiaojialin merged PR #225:
URL: https://github.com/apache/iotdb-docs/pull/225


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Fixed rest service insertTablet lost data event return SUCCESS_STATUS [iotdb]

2024-04-23 Thread via GitHub



CloudWise-Lukemiao commented on PR #12387:
URL: https://github.com/apache/iotdb/pull/12387#issuecomment-2071548937

   http://127.0.0.1:18080/rest/v2/insertTablet
   When inserting with the following parameters, when there are duplicates in 
the timestamp and more than half of the values in the values are null
   
   
{"is_aligned":false,"device":"root.mqtt.topic.abc454","data_types":["TEXT","INT32","FLOAT"],"measurements":["foo","temp","weight"],"timestamps":[1713495895809,1713499707255,1713495893809,1713495892809,1713495891809,1713495890809,1713495889809,171349509,1713495887809,1713495893809,1713495892809,1713495891809,1713495890809,1713495889809,171349509],"values":[["bar",null,null,null,null,null,null,null,null,null,null,null,null,null,null],[null,41,32,33,34,39,38,37,36,null,null,null,null,null,null],[null,null,null,null,null,null,null,null,null,87.0,87.3,87.3,87.0,87.3,87.3]]}
   
   After repair, as shown in the following figure：
   https://github.com/apache/iotdb/assets/76942485/e9688829-cbca-4d2f-9903-c7487bf39aa6;>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Provide more precise info for exception during conf loading [iotdb]

2024-04-23 Thread via GitHub



OneSizeFitsQuorum merged PR #12382:
URL: https://github.com/apache/iotdb/pull/12382


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] fix: 下载换行样式 [iotdb-docs]

2024-04-23 Thread via GitHub



tongshs opened a new pull request, #225:
URL: https://github.com/apache/iotdb-docs/pull/225

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [I] [Bug] insertTablet http api lost data event return SUCCESS_STATUS [iotdb]

2024-04-23 Thread via GitHub



CritasWang commented on issue #12375:
URL: https://github.com/apache/iotdb/issues/12375#issuecomment-2071513726

   https://github.com/apache/iotdb/pull/12387


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575714841


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/schedule/CompactionScheduleTaskManager.java:
##


Review Comment:
   This is reasonable, equivalent to stopping all compaction selector threads 
and TTL checker threads when starting to repair data, and then resuming them 
after the repair is complete. Moreover, users are unaware of the number of 
threads involved in the repair.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



choubenson commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1575710663


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/memory/SystemInfo.java:
##


Review Comment:
   Resolved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Implementation of device TTL [iotdb]

2024-04-23 Thread via GitHub



shuwenwei commented on code in PR #12122:
URL: https://github.com/apache/iotdb/pull/12122#discussion_r1565391622


##
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/rescon/memory/SystemInfo.java:
##


Review Comment:
   Method `addCompactionMemoryCost(
 CompactionTaskType taskType, long memoryCost, long timeOutInSecond)` 
is no longer used after a past change. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

57 matches

Mail list logo