qiaojialin commented on a change in pull request #1384:
URL: https://github.com/apache/incubator-iotdb/pull/1384#discussion_r443338591
##########
File path: docs/SystemDesign/SchemaManager/SchemaManager.md
##########
@@ -175,6 +175,16 @@ The root node exists by default. Creating storage groups,
deleting storage group
* Deleting a storage group is similar to deleting a time series. That is, the
storage group or time series node is deleted in its parent node. The time
series node also needs to delete its alias in the parent node; if in the
deletion process, a node is found not to have any child node, needs to be
deleted recursively.
+## MTree checkpoint
+
+To speed up restarting of IoTDB, we set checkpoint for MTree. Every several
minutes, background thread checks the last modified time of MTree. If users
haven’t modified MTree for more than 1 hour (which means `mlog.txt` hasn’t been
updated for more than 1 hour), and `mlog.txt` has reached the threshold line
number of user configuration, MTree snapshot is created. In this way, we could
avoid reading mlog.txt and executing the commands line by line.
+
+The serialization of MTree is depth-first from children to parent. Information
of nodes are converted into String according to different node types, which is
convenient for deserialization.
+
+* Common MNode: 0,name,children size
Review comment:
```suggestion
* MNode: 0,name,children size
```
##########
File path: server/src/main/java/org/apache/iotdb/db/metadata/MManager.java
##########
@@ -1702,4 +1766,31 @@ public void cacheSchema(String path, MeasurementSchema
schema) {
mRemoteSchemaCache.put(path, schema);
}
}
+
+ private void checkMTreeModified() {
+ if (System.currentTimeMillis() - logFile.lastModified() < 60 * 60 * 1000) {
Review comment:
replace this by config
##########
File path: server/src/main/java/org/apache/iotdb/db/metadata/MManager.java
##########
@@ -1702,4 +1766,31 @@ public void cacheSchema(String path, MeasurementSchema
schema) {
mRemoteSchemaCache.put(path, schema);
}
}
+
+ private void checkMTreeModified() {
+ if (System.currentTimeMillis() - logFile.lastModified() < 60 * 60 * 1000) {
+ logger.info("MTree snapshot is not created because of active
modification");
+ } else if (logWriter.getLineNumber() - lastSnapshotLogLineNumber <
mtreeSnapshotInterval) {
+ logger.info("MTree snapshot need not be created");
+ } else {
+ lock.writeLock().lock();
+ logger.info("Start creating MTree snapshot. This may take a while...");
+ try {
+ mtree.serializeTo(mtreeSnapshotTmpPath, logWriter.getLineNumber());
+ lastSnapshotLogLineNumber = logWriter.getLineNumber();
+ File tmpFile =
SystemFileFactory.INSTANCE.getFile(mtreeSnapshotTmpPath);
+ File snapshotFile =
SystemFileFactory.INSTANCE.getFile(mtreeSnapshotPath);
+ if (snapshotFile.exists()) {
+ Files.delete(snapshotFile.toPath());
+ }
+ if (tmpFile.renameTo(snapshotFile)) {
+ logger.info("Finish creating MTree snapshot to {}.",
mtreeSnapshotPath);
+ }
+ } catch (IOException e) {
+ logger.warn("Failed to create MTree snapshot to {}",
mtreeSnapshotPath, e);
Review comment:
remove tmp file
##########
File path: server/src/main/java/org/apache/iotdb/db/metadata/MTree.java
##########
@@ -72,27 +80,34 @@
public class MTree implements Serializable {
private static final long serialVersionUID = -4200394435237291964L;
+
private MNode root;
+ private int snapshotLineNumber;
Review comment:
snapshot is ok
##########
File path: server/src/main/java/org/apache/iotdb/db/metadata/MTree.java
##########
@@ -899,6 +918,69 @@ private void findNodes(MNode node, String path,
List<String> res, int targetLeve
}
}
+ public int getSnapshotLineNumber() {
+ return snapshotLineNumber;
+ }
+
+ public void serializeTo(String snapshotPath, int lineNumber) throws
IOException {
+ try (BufferedWriter bw = new BufferedWriter(
+ new FileWriter(SystemFileFactory.INSTANCE.getFile(snapshotPath)))) {
+ bw.write(String.valueOf(lineNumber));
+ bw.newLine();
+ root.serializeTo(bw);
+ }
+ }
+
+ public static MTree deserializeFrom(File mtreeSnapshot) {
+ try (BufferedReader br = new BufferedReader(new
FileReader(mtreeSnapshot))) {
+ int snapshotLineNumber = Integer.parseInt(br.readLine());
+ String s;
+ Deque<MNode> nodeStack = new ArrayDeque<>();
+ MNode node = null;
+
+ while ((s = br.readLine()) != null) {
+ String[] nodeInfo = s.split(",");
+ short nodeType = Short.parseShort(nodeInfo[0]);
+ if (nodeType == MetadataConstant.STORAGE_GROUP_MNODE_TYPE) {
+ node = StorageGroupMNode.deserializeFrom(nodeInfo);
+ } else if (nodeType == MetadataConstant.MEASUREMENT_MNODE_TYPE) {
+ node = MeasurementMNode.deserializeFrom(nodeInfo);
+ } else {
+ node = new MNode(null, nodeInfo[1]);
+ }
+
+ int childrenSize = Integer.parseInt(nodeInfo[nodeInfo.length - 1]);
+ if (childrenSize == 0) {
+ nodeStack.push(node);
+ } else {
+ Map<String, MNode> childrenMap = new LinkedHashMap<>();
+ for (int i = 0; i < childrenSize; i++) {
+ MNode child = nodeStack.removeFirst();
+ child.setParent(node);
+ childrenMap.put(child.getName(), child);
+ if (child instanceof MeasurementMNode) {
+ String alias = ((MeasurementMNode) child).getAlias();
+ if (alias != null) {
+ node.addAlias(alias, child);
+ }
+ }
+ }
+ node.setChildren(childrenMap);
+ nodeStack.push(node);
+ }
+ }
+ return new MTree(node, snapshotLineNumber);
+ } catch (IOException e) {
+ logger.warn("Failed to deserialize from {}. Use a new MTree.",
mtreeSnapshot.getPath());
+ return new MTree();
Review comment:
just remind, make sure if the snapshot is crashed, we can still recover
the MTree from mlog
##########
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
##########
@@ -568,11 +568,21 @@
private int primitiveArraySize = 64;
/**
- * whether enable data partition
- * if disabled, all data belongs to partition 0
+ * whether enable data partition. If disabled, all data belongs to partition 0
*/
private boolean enablePartition = false;
+ /**
+ * Interval line number of mlog.txt when creating a checkpoint and saving
snapshot of mtree
+ */
+ private int mtreeSnapshotInterval = 1000000;
Review comment:
```suggestion
private int mtreeSnapshotInterval = 100000;
```
##########
File path: server/src/main/java/org/apache/iotdb/db/metadata/MManager.java
##########
@@ -1702,4 +1766,31 @@ public void cacheSchema(String path, MeasurementSchema
schema) {
mRemoteSchemaCache.put(path, schema);
}
}
+
+ private void checkMTreeModified() {
+ if (System.currentTimeMillis() - logFile.lastModified() < 60 * 60 * 1000) {
+ logger.info("MTree snapshot is not created because of active
modification");
+ } else if (logWriter.getLineNumber() - lastSnapshotLogLineNumber <
mtreeSnapshotInterval) {
+ logger.info("MTree snapshot need not be created");
+ } else {
+ lock.writeLock().lock();
Review comment:
use read lock
##########
File path: server/src/main/java/org/apache/iotdb/db/metadata/MTree.java
##########
@@ -72,27 +80,34 @@
public class MTree implements Serializable {
private static final long serialVersionUID = -4200394435237291964L;
+
private MNode root;
+ private int snapshotLineNumber;
Review comment:
add some comment
##########
File path: server/src/main/java/org/apache/iotdb/db/metadata/MManager.java
##########
@@ -1702,4 +1766,31 @@ public void cacheSchema(String path, MeasurementSchema
schema) {
mRemoteSchemaCache.put(path, schema);
}
}
+
+ private void checkMTreeModified() {
+ if (System.currentTimeMillis() - logFile.lastModified() < 60 * 60 * 1000) {
+ logger.info("MTree snapshot is not created because of active
modification");
+ } else if (logWriter.getLineNumber() - lastSnapshotLogLineNumber <
mtreeSnapshotInterval) {
+ logger.info("MTree snapshot need not be created");
Review comment:
log the line number
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
