[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524: URL: https://github.com/apache/iotdb/pull/1524#discussion_r515760463 ## File path: server/src/assembly/resources/conf/iotdb-engine.properties ## @@ -218,28 +218,35 @@ mtree_snapshot_threshold_time=3600 ### Memory Control Configuration -# This adapter could adjust the system memory to avoid OOM. -# It will refuse to create time series or add storage groups under high system load. -# -# Attention!!! -# If disable this parameter, you need to set some parameters according to your system load: -# Normal scenario: 1-50 storage groups. <100k devices, <10M time series -# memtable_size_threshold = tsfile_size_threshold = IoTDB memory allocation in byte / 2 / num of storage group / 4 -# Make sure the total num of time series in system * primitive_array_size * 16 <= IoTDB memory allocation in byte / 2 / 4 -# you could reduce the primitive_array_size in very high workload. -enable_parameter_adapter=true - -# Memory Allocation Ratio: Write, Read, and Free Memory. -# The parameter form is a:b:c, where a, b and c are integers. for example: 1:1:1 , 6:3:1 -write_read_free_memory_proportion=6:3:1 +# Whether to enable memory control +enable_mem_control=true + +# Memory Allocation Ratio: Write, Read, Schema and Free Memory. +# The parameter form is a:b:c:d, where a, b, c and d are integers. for example: 1:1:1:1 , 6:2:1:1 +write_read_schema_free_memory_proportion=4:3:1:2 # primitive array size (length of each array) in array pool primitive_array_size=128 +# Ratio of memory allocated for buffered arrays, 0.8 by default +buffered_arrays_memory_proportion=0.8 Review comment: 0.8 of write memory ## File path: server/src/assembly/resources/conf/iotdb-engine.properties ## @@ -179,7 +179,7 @@ tsfile_size_threshold=536870912 # When a memTable's size (in byte) exceeds this, the memtable is flushed to disk. The default threshold is 1 GB. memtable_size_threshold=1073741824 -avg_series_point_number_threshold=10 +avg_series_point_number_threshold=1 Review comment: sure ## File path: server/src/assembly/resources/conf/iotdb-engine.properties ## @@ -218,28 +218,35 @@ mtree_snapshot_threshold_time=3600 ### Memory Control Configuration -# This adapter could adjust the system memory to avoid OOM. -# It will refuse to create time series or add storage groups under high system load. -# -# Attention!!! -# If disable this parameter, you need to set some parameters according to your system load: -# Normal scenario: 1-50 storage groups. <100k devices, <10M time series -# memtable_size_threshold = tsfile_size_threshold = IoTDB memory allocation in byte / 2 / num of storage group / 4 -# Make sure the total num of time series in system * primitive_array_size * 16 <= IoTDB memory allocation in byte / 2 / 4 -# you could reduce the primitive_array_size in very high workload. -enable_parameter_adapter=true - -# Memory Allocation Ratio: Write, Read, and Free Memory. -# The parameter form is a:b:c, where a, b and c are integers. for example: 1:1:1 , 6:3:1 -write_read_free_memory_proportion=6:3:1 +# Whether to enable memory control +enable_mem_control=true + +# Memory Allocation Ratio: Write, Read, Schema and Free Memory. +# The parameter form is a:b:c:d, where a, b, c and d are integers. for example: 1:1:1:1 , 6:2:1:1 +write_read_schema_free_memory_proportion=4:3:1:2 # primitive array size (length of each array) in array pool primitive_array_size=128 +# Ratio of memory allocated for buffered arrays, 0.8 by default +buffered_arrays_memory_proportion=0.8 + +# Flush proportion for system, 0.3 by default +flush_proportion=0.3 + +# Reject proportion for system, 0.8 by default +reject_proportion=0.8 + +# If storage group increased more than this threshold, report to system. The default value is 16MB +storage_group_report_threshold=16777216 + # allowed max numbers of deduplicated path in one query # it's just an advised value, the real limitation will be the smaller one between this and the one we calculated max_deduplicated_path_num=1000 +# estimated metadata size of one timeseries in Mtree +estimated_series_size=300 Review comment: Fixed ## File path: server/src/assembly/resources/conf/iotdb-engine.properties ## @@ -218,28 +218,35 @@ mtree_snapshot_threshold_time=3600 ### Memory Control Configuration -# This adapter could adjust the system memory to avoid OOM. -# It will refuse to create time series or add storage groups under high system load. -# -# Attention!!! -# If disable this parameter, you need to set some parameters according to your system load: -# Normal scenario: 1-50 storage groups. <100k devices, <10M time series -# memtable_size_threshold = tsfile_size_threshold = IoTDB memory allocation in byte / 2 / num of storage group / 4 -# Make sure the total num of time series in system *
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r512698662
##
File path:
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessorInfo.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+
+/**
+ * The TsFileProcessorInfo records the memory cost of this TsFileProcessor.
+ */
+public class TsFileProcessorInfo {
+
+ /**
+ * Once tspInfo updated, report to storageGroupInfo that this TSP belongs to.
+ */
+ private StorageGroupInfo storageGroupInfo;
+
+ /**
+ * The memory cost of the unsealed TsFileResources of this TSP
+ */
+ private long unsealedResourceMemCost;
+
+ /**
+ * The memory cost of memTable of this TSP
+ */
+ private long memTableCost;
+
+ /**
+ * The memory cost of ChunkMetadata of this TSP
+ */
+ private long chunkMetadataMemCost;
+
+ /**
+ * The memory cost of WAL of this TSP
+ */
+ private long walMemCost;
+
+ public TsFileProcessorInfo(StorageGroupInfo storageGroupInfo) {
+this.storageGroupInfo = storageGroupInfo;
+this.unsealedResourceMemCost = 0;
+this.memTableCost = 0;
+this.chunkMetadataMemCost = 0;
+this.walMemCost =
IoTDBDescriptor.getInstance().getConfig().getWalBufferSize();
+ }
+
+ public void addUnsealedResourceMemCost(long cost) {
+unsealedResourceMemCost += cost;
+storageGroupInfo.addUnsealedResourceMemCost(cost);
+ }
+
+ public void addChunkMetadataMemCost(long cost) {
+chunkMetadataMemCost += cost;
+storageGroupInfo.addChunkMetadataMemCost(cost);
+ }
+
+ public void addMemTableCost(long cost) {
+memTableCost += cost;
+storageGroupInfo.addMemTableCost(cost);
+ }
+
+ /**
+ * call this method when closing TSP
+ */
+ public void clear() {
+storageGroupInfo.resetUnsealedResourceMemCost(unsealedResourceMemCost);
+storageGroupInfo.resetChunkMetadataMemCost(chunkMetadataMemCost);
+storageGroupInfo.resetWalMemCost(walMemCost);
Review comment:
Fixed~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r512505936
##
File path:
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/enums/TSDataType.java
##
@@ -98,6 +99,23 @@ public short serialize() {
return enumToByte();
}
+ public int getDataTypeSize() {
+switch (this) {
+ case BOOLEAN:
+return 1;
+ case INT32:
+ case FLOAT:
+// For text: return the size of reference here
Review comment:
Thanks, update soon~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r512454774
##
File path:
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessorInfo.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+
+/**
+ * The TsFileProcessorInfo records the memory cost of this TsFileProcessor.
+ */
+public class TsFileProcessorInfo {
+
+ /**
+ * Once tspInfo updated, report to storageGroupInfo that this TSP belongs to.
+ */
+ private StorageGroupInfo storageGroupInfo;
+
+ /**
+ * The memory cost of the unsealed TsFileResources of this TSP
+ */
+ private long unsealedResourceMemCost;
+
+ /**
+ * The memory cost of memTable of this TSP
+ */
+ private long memTableCost;
+
+ /**
+ * The memory cost of ChunkMetadata of this TSP
+ */
+ private long chunkMetadataMemCost;
+
+ /**
+ * The memory cost of WAL of this TSP
+ */
+ private long walMemCost;
+
+ public TsFileProcessorInfo(StorageGroupInfo storageGroupInfo) {
+this.storageGroupInfo = storageGroupInfo;
+this.unsealedResourceMemCost = 0;
+this.memTableCost = 0;
+this.chunkMetadataMemCost = 0;
+this.walMemCost =
IoTDBDescriptor.getInstance().getConfig().getWalBufferSize();
+ }
+
+ public void addUnsealedResourceMemCost(long cost) {
+unsealedResourceMemCost += cost;
+storageGroupInfo.addUnsealedResourceMemCost(cost);
+ }
+
+ public void addChunkMetadataMemCost(long cost) {
+chunkMetadataMemCost += cost;
+storageGroupInfo.addChunkMetadataMemCost(cost);
+ }
+
+ public void addMemTableCost(long cost) {
+memTableCost += cost;
+storageGroupInfo.addMemTableCost(cost);
+ }
+
+ /**
+ * call this method when closing TSP
+ */
+ public void clear() {
+storageGroupInfo.resetUnsealedResourceMemCost(unsealedResourceMemCost);
+storageGroupInfo.resetChunkMetadataMemCost(chunkMetadataMemCost);
+storageGroupInfo.resetWalMemCost(walMemCost);
Review comment:
This method is only called by `endFile()` in TsFileProcessor. Can you
let me know why it has a concurrent issue?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r511969314
##
File path:
server/src/main/java/org/apache/iotdb/db/rescon/PrimitiveArrayManager.java
##
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.rescon;
+
+import java.util.ArrayDeque;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.tsfile.exception.write.UnSupportedDataTypeException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.utils.Binary;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manage all primitive data list in memory, including get and release
operation.
+ */
+public class PrimitiveArrayManager {
+
+ /**
+ * data type -> ArrayDeque
+ */
+ private static final Map> bufferedArraysMap =
new EnumMap<>(
+ TSDataType.class);
+
+ /**
+ * data type -> current number of buffered arrays
+ */
+ private static final Map bufferedArraysNumMap = new
EnumMap<>(
+ TSDataType.class);
+
+ /**
+ * data type -> ratio of data type in schema, which could be seen as
recommended ratio
+ */
+ private static final Map bufferedArraysNumRatio = new
EnumMap<>(
+ TSDataType.class);
+
+ private static final Logger logger =
LoggerFactory.getLogger(PrimitiveArrayManager.class);
+
+ private static final IoTDBConfig config =
IoTDBDescriptor.getInstance().getConfig();
+
+ public static final int ARRAY_SIZE = config.getPrimitiveArraySize();
+
+ /**
+ * threshold total size of arrays for all data types
+ */
+ private static final double BUFFERED_ARRAY_SIZE_THRESHOLD =
+ config.getAllocateMemoryForWrite() *
config.getBufferedArraysMemoryProportion();
+
+ /**
+ * total size of buffered arrays
+ */
+ private static AtomicInteger bufferedArraysSize = new AtomicInteger();
+
+ /**
+ * total size of out of buffer arrays
+ */
+ private static AtomicInteger outOfBufferArraysSize = new AtomicInteger();
+
+ static {
+bufferedArraysMap.put(TSDataType.BOOLEAN, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.INT32, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.INT64, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.FLOAT, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.DOUBLE, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.TEXT, new ArrayDeque<>());
+ }
+
+ private PrimitiveArrayManager() {
+ }
+
+ /**
+ * Get primitive data lists according to type
+ *
+ * @param dataType data type
+ * @return an array
+ */
+ public static Object getDataListByType(TSDataType dataType) {
+// check buffered array num
+if (bufferedArraysSize.get() + ARRAY_SIZE * dataType.getDataTypeSize()
+> BUFFERED_ARRAY_SIZE_THRESHOLD) {
+ // return an out of buffer array
+ outOfBufferArraysSize.addAndGet(ARRAY_SIZE * dataType.getDataTypeSize());
+ return getDataList(dataType);
+}
+
+synchronized (bufferedArraysMap.get(dataType)) {
+ // return a buffered array
+ bufferedArraysNumMap.put(dataType,
bufferedArraysNumMap.getOrDefault(dataType, 0) + 1);
+ bufferedArraysSize.addAndGet(ARRAY_SIZE * dataType.getDataTypeSize());
+ Object dataArray = bufferedArraysMap.get(dataType).poll();
+ if (dataArray != null) {
+return dataArray;
+ }
+}
+return getDataList(dataType);
Review comment:
I'm confused why should it return null here. If change it, the tests
will failed...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r511969466
##
File path: server/src/main/java/org/apache/iotdb/db/metadata/MManager.java
##
@@ -608,13 +644,28 @@ public void deleteStorageGroups(List
storageGroups) throws Metadata
logWriter.deleteStorageGroup(storageGroup.getFullPath());
}
}
-} catch (ConfigAdjusterException e) {
- throw new MetadataException(e);
} catch (IOException e) {
throw new MetadataException(e.getMessage());
}
}
+ /**
+ * update statistics in schemaDataTypeNumMap
+ *
+ * @param type data type
+ * @param num 1 for creating timeseries and -1 for deleting timeseries
+ */
+ private void updateSchemaDataTypeNumMap(TSDataType type, int num) {
+schemaDataTypeNumMap.put(type, schemaDataTypeNumMap.getOrDefault(type, 0)
+ num);
+schemaDataTypeNumMap.put(TSDataType.INT64,
+schemaDataTypeNumMap.getOrDefault(TSDataType.INT64, 0) + num);
+int currentDataTypeTotalNum = schemaDataTypeNumMap.values().size();
+if (num > 0 && currentDataTypeTotalNum >= reportedDataTypeTotalNum * 1.1) {
Review comment:
Fixed~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r511961387
##
File path:
tsfile/src/main/java/org/apache/iotdb/tsfile/file/metadata/enums/TSDataType.java
##
@@ -98,6 +99,23 @@ public short serialize() {
return enumToByte();
}
+ public int getDataTypeSize() {
+switch (this) {
+ case BOOLEAN:
+return 1;
+ case INT32:
+ case FLOAT:
+// For text: return the size of reference here
Review comment:
This method is for calculating the array sizes in array pool. For text
data, it should return the reference size of a Binary object. And the actual
size of the Binary object won't be considered here.
So, should it return 8?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r511814097
##
File path:
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/TsFileProcessorInfo.java
##
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.storagegroup;
+
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+
+/**
+ * The TsFileProcessorInfo records the memory cost of this TsFileProcessor.
+ */
+public class TsFileProcessorInfo {
+
+ /**
+ * Once tspInfo updated, report to storageGroupInfo that this TSP belongs to.
+ */
+ private StorageGroupInfo storageGroupInfo;
+
+ /**
+ * The memory cost of the unsealed TsFileResources of this TSP
+ */
+ private long unsealedResourceMemCost;
+
+ /**
+ * The memory cost of memTable of this TSP
+ */
+ private long memTableCost;
+
+ /**
+ * The memory cost of ChunkMetadata of this TSP
+ */
+ private long chunkMetadataMemCost;
+
+ /**
+ * The memory cost of WAL of this TSP
+ */
+ private long walMemCost;
+
+ public TsFileProcessorInfo(StorageGroupInfo storageGroupInfo) {
+this.storageGroupInfo = storageGroupInfo;
+this.unsealedResourceMemCost = 0;
+this.memTableCost = 0;
+this.chunkMetadataMemCost = 0;
Review comment:
Fixed~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524: URL: https://github.com/apache/iotdb/pull/1524#discussion_r511812257 ## File path: server/src/main/java/org/apache/iotdb/db/engine/memtable/IMemTable.java ## @@ -45,7 +45,7 @@ void write(String deviceId, String measurement, MeasurementSchema schema, long insertTime, Object objectValue); - void write(InsertTabletPlan insertTabletPlan, int start, int end); + void write(InsertTabletPlan insertTabletPlan, int start, int end) throws WriteProcessException; Review comment: Fixed~ ## File path: server/src/test/java/org/apache/iotdb/db/engine/memtable/PrimitiveMemTableTest.java ## @@ -22,7 +22,9 @@ import java.util.Arrays; import java.util.Collections; import java.util.Random; + import org.apache.iotdb.db.engine.querycontext.ReadOnlyMemChunk; +import org.apache.iotdb.db.exception.WriteProcessException; Review comment: Thanks, fixed~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r511810288
##
File path:
server/src/main/java/org/apache/iotdb/db/engine/flush/FlushManager.java
##
@@ -96,6 +96,13 @@ public void runMayThrow() {
}
}
+ /**
+ * get the current size of flush queue
+ */
+ public int getTsFileProcessorQueueSize() {
Review comment:
This method is no longer useful, I'll remove it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r511803707
##
File path:
server/src/main/java/org/apache/iotdb/db/rescon/PrimitiveArrayManager.java
##
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.rescon;
+
+import java.util.ArrayDeque;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.tsfile.exception.write.UnSupportedDataTypeException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.utils.Binary;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manage all primitive data list in memory, including get and release
operation.
+ */
+public class PrimitiveArrayManager {
+
+ /**
+ * data type -> ArrayDeque
+ */
+ private static final Map> bufferedArraysMap =
new EnumMap<>(
+ TSDataType.class);
+
+ /**
+ * data type -> current number of buffered arrays
+ */
+ private static final Map bufferedArraysNumMap = new
EnumMap<>(
+ TSDataType.class);
+
+ /**
+ * data type -> ratio of data type in schema, which could be seen as
recommended ratio
+ */
+ private static final Map bufferedArraysNumRatio = new
EnumMap<>(
+ TSDataType.class);
+
+ private static final Logger logger =
LoggerFactory.getLogger(PrimitiveArrayManager.class);
+
+ private static final IoTDBConfig config =
IoTDBDescriptor.getInstance().getConfig();
+
+ public static final int ARRAY_SIZE = config.getPrimitiveArraySize();
+
+ /**
+ * threshold total size of arrays for all data types
+ */
+ private static final double BUFFERED_ARRAY_SIZE_THRESHOLD =
+ config.getAllocateMemoryForWrite() *
config.getBufferedArraysMemoryProportion();
+
+ /**
+ * total size of buffered arrays
+ */
+ private static AtomicInteger bufferedArraysSize = new AtomicInteger();
+
+ /**
+ * total size of out of buffer arrays
+ */
+ private static AtomicInteger outOfBufferArraysSize = new AtomicInteger();
+
+ static {
+bufferedArraysMap.put(TSDataType.BOOLEAN, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.INT32, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.INT64, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.FLOAT, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.DOUBLE, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.TEXT, new ArrayDeque<>());
+ }
+
+ private PrimitiveArrayManager() {
+ }
+
+ /**
+ * Get primitive data lists according to type
+ *
+ * @param dataType data type
+ * @return an array
+ */
+ public static Object getDataListByType(TSDataType dataType) {
+// check buffered array num
+if (bufferedArraysSize.get() + ARRAY_SIZE * dataType.getDataTypeSize()
+> BUFFERED_ARRAY_SIZE_THRESHOLD) {
+ // return an out of buffer array
+ outOfBufferArraysSize.addAndGet(ARRAY_SIZE * dataType.getDataTypeSize());
+ return getDataList(dataType);
+}
+
+synchronized (bufferedArraysMap.get(dataType)) {
+ // return a buffered array
+ bufferedArraysNumMap.put(dataType,
bufferedArraysNumMap.getOrDefault(dataType, 0) + 1);
+ bufferedArraysSize.addAndGet(ARRAY_SIZE * dataType.getDataTypeSize());
+ Object dataArray = bufferedArraysMap.get(dataType).poll();
+ if (dataArray != null) {
+return dataArray;
+ }
+}
+return getDataList(dataType);
Review comment:
> It's my fault Could you please help add some comments @HTHou ?
This return statement should not reached..
Sure, Let me fix it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r511803707
##
File path:
server/src/main/java/org/apache/iotdb/db/rescon/PrimitiveArrayManager.java
##
@@ -0,0 +1,329 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.rescon;
+
+import java.util.ArrayDeque;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.tsfile.exception.write.UnSupportedDataTypeException;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.utils.Binary;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Manage all primitive data list in memory, including get and release
operation.
+ */
+public class PrimitiveArrayManager {
+
+ /**
+ * data type -> ArrayDeque
+ */
+ private static final Map> bufferedArraysMap =
new EnumMap<>(
+ TSDataType.class);
+
+ /**
+ * data type -> current number of buffered arrays
+ */
+ private static final Map bufferedArraysNumMap = new
EnumMap<>(
+ TSDataType.class);
+
+ /**
+ * data type -> ratio of data type in schema, which could be seen as
recommended ratio
+ */
+ private static final Map bufferedArraysNumRatio = new
EnumMap<>(
+ TSDataType.class);
+
+ private static final Logger logger =
LoggerFactory.getLogger(PrimitiveArrayManager.class);
+
+ private static final IoTDBConfig config =
IoTDBDescriptor.getInstance().getConfig();
+
+ public static final int ARRAY_SIZE = config.getPrimitiveArraySize();
+
+ /**
+ * threshold total size of arrays for all data types
+ */
+ private static final double BUFFERED_ARRAY_SIZE_THRESHOLD =
+ config.getAllocateMemoryForWrite() *
config.getBufferedArraysMemoryProportion();
+
+ /**
+ * total size of buffered arrays
+ */
+ private static AtomicInteger bufferedArraysSize = new AtomicInteger();
+
+ /**
+ * total size of out of buffer arrays
+ */
+ private static AtomicInteger outOfBufferArraysSize = new AtomicInteger();
+
+ static {
+bufferedArraysMap.put(TSDataType.BOOLEAN, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.INT32, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.INT64, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.FLOAT, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.DOUBLE, new ArrayDeque<>());
+bufferedArraysMap.put(TSDataType.TEXT, new ArrayDeque<>());
+ }
+
+ private PrimitiveArrayManager() {
+ }
+
+ /**
+ * Get primitive data lists according to type
+ *
+ * @param dataType data type
+ * @return an array
+ */
+ public static Object getDataListByType(TSDataType dataType) {
+// check buffered array num
+if (bufferedArraysSize.get() + ARRAY_SIZE * dataType.getDataTypeSize()
+> BUFFERED_ARRAY_SIZE_THRESHOLD) {
+ // return an out of buffer array
+ outOfBufferArraysSize.addAndGet(ARRAY_SIZE * dataType.getDataTypeSize());
+ return getDataList(dataType);
+}
+
+synchronized (bufferedArraysMap.get(dataType)) {
+ // return a buffered array
+ bufferedArraysNumMap.put(dataType,
bufferedArraysNumMap.getOrDefault(dataType, 0) + 1);
+ bufferedArraysSize.addAndGet(ARRAY_SIZE * dataType.getDataTypeSize());
+ Object dataArray = bufferedArraysMap.get(dataType).poll();
+ if (dataArray != null) {
+return dataArray;
+ }
+}
+return getDataList(dataType);
Review comment:
> It's my fault Could you please help add some comments @HTHou ?
This return statement should not reached..
Sure, Let me fix it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r511687389
##
File path: server/src/main/java/org/apache/iotdb/db/metadata/MManager.java
##
@@ -529,23 +569,22 @@ private PartialPath
deleteOneTimeseriesAndUpdateStatistics(PartialPath path)
removeFromTagInvertedIndex(pair.right);
PartialPath storageGroupPath = pair.left;
-// TODO: delete the path node and all its ancestors
-mNodeCache.clear();
-try {
- IoTDBConfigDynamicAdapter.getInstance().addOrDeleteTimeSeries(-1);
-} catch (ConfigAdjusterException e) {
- throw new MetadataException(e);
-}
+ // update statistics in schemaDataTypeNumMap
+ updateSchemaDataTypeNumMap(pair.right.getSchema().getType(), -1);
+
+ // TODO: delete the path node and all its ancestors
+ mNodeCache.clear();
-if (config.isEnableParameterAdapter()) {
- PartialPath storageGroup = getStorageGroupPath(path);
- int size = seriesNumberInStorageGroups.get(storageGroup.getFullPath());
- seriesNumberInStorageGroups.put(storageGroup.getFullPath(), size - 1);
- if (size == maxSeriesNumberAmongStorageGroup) {
-seriesNumberInStorageGroups.values().stream().max(Integer::compareTo)
-.ifPresent(val -> maxSeriesNumberAmongStorageGroup = val);
+ if (config.isEnableActiveTimeseriesCounter()) {
+PartialPath storageGroup = getStorageGroupPath(path);
+int size = seriesNumberInStorageGroups.get(storageGroup.getFullPath());
+seriesNumberInStorageGroups.put(storageGroup.getFullPath(), size - 1);
+totalSeriesNumber--;
+if (size == maxSeriesNumberAmongStorageGroup) {
+ seriesNumberInStorageGroups.values().stream().max(Integer::compareTo)
+ .ifPresent(val -> maxSeriesNumberAmongStorageGroup = val);
+}
}
Review comment:
Nice catch! Fixed~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524: URL: https://github.com/apache/iotdb/pull/1524#discussion_r511687043 ## File path: server/src/main/java/org/apache/iotdb/db/metadata/MManager.java ## @@ -119,15 +118,41 @@ private RandomDeleteCache mNodeCache; // tag key -> tag value -> LeafMNode private Map>> tagIndex = new HashMap<>(); + + // data type -> number + private Map schemaDataTypeNumMap = new ConcurrentHashMap<>(); + // reported total series number + private int reportedDataTypeTotalNum; Review comment: Fixed~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r511686584
##
File path: server/src/main/java/org/apache/iotdb/db/utils/MemUtils.java
##
@@ -45,8 +45,11 @@ private MemUtils(){
/**
* function for getting the value size.
+ * If mem control enabled, do not get text data size here, the size will add
to memtable
Review comment:
Yes, it's better. Fixed~
##
File path: server/src/main/java/org/apache/iotdb/db/utils/MemUtils.java
##
@@ -70,7 +73,12 @@ public static long getBinarySize(Binary value) {
.sizeOf(value.getValues());
}
- public static long getRecordSize(InsertTabletPlan insertTabletPlan, int
start, int end) {
+ /**
+ * If mem control enabled, do not get text data size here, the size will add
to memtable
Review comment:
Fixed~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r510556524
##
File path:
server/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java
##
@@ -100,15 +107,16 @@ private IWritableMemChunk createIfNotExistAndGet(String
deviceId, String measure
protected abstract IWritableMemChunk genMemSeries(MeasurementSchema schema);
@Override
- public void insert(InsertRowPlan insertRowPlan) {
+ public void insert(InsertRowPlan insertRowPlan) throws WriteProcessException
{
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r510556332
##
File path:
server/src/test/java/org/apache/iotdb/db/engine/memtable/MemTableFlushTaskTest.java
##
@@ -62,7 +63,7 @@ public void tearDown() throws Exception {
}
@Test
- public void testFlushMemTable() throws ExecutionException,
InterruptedException, IOException {
+ public void testFlushMemTable() throws ExecutionException,
InterruptedException, IOException, WriteProcessException {
Review comment:
Fixed
##
File path:
server/src/test/java/org/apache/iotdb/db/engine/memtable/MemTableTestUtils.java
##
@@ -40,7 +41,7 @@
}
public static void produceData(IMemTable iMemTable, long startTime, long
endTime, String deviceId,
- String measurementId, TSDataType dataType) {
+ String measurementId, TSDataType dataType) throws WriteProcessException {
Review comment:
Fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524: URL: https://github.com/apache/iotdb/pull/1524#discussion_r510555743 ## File path: server/src/main/java/org/apache/iotdb/db/engine/memtable/IMemTable.java ## @@ -43,9 +43,9 @@ Map> getMemTableMap(); void write(String deviceId, String measurement, MeasurementSchema schema, - long insertTime, Object objectValue); + long insertTime, Object objectValue) throws WriteProcessException; Review comment: Fixed This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r510555473
##
File path:
server/src/test/java/org/apache/iotdb/db/engine/memtable/PrimitiveMemTableTest.java
##
@@ -49,7 +51,7 @@ public void setUp() {
}
@Test
- public void memSeriesSortIteratorTest() throws IOException {
+ public void memSeriesSortIteratorTest() throws IOException,
WriteProcessException {
Review comment:
Fixed~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r510555368
##
File path:
server/src/main/java/org/apache/iotdb/db/engine/memtable/AbstractMemTable.java
##
@@ -122,24 +130,24 @@ public void insertTablet(InsertTabletPlan
insertTabletPlan, int start, int end)
throws WriteProcessException {
try {
write(insertTabletPlan, start, end);
- memSize += MemUtils.getRecordSize(insertTabletPlan, start, end);
+ memSize += MemUtils.getRecordSize(insertTabletPlan, start, end,
!enableMemControl);
totalPointsNum += (insertTabletPlan.getMeasurements().length -
insertTabletPlan.getFailedMeasurementNumber())
* (end - start);
} catch (RuntimeException e) {
- throw new WriteProcessException(e.getMessage());
+ throw new WriteProcessException(e);
}
}
@Override
public void write(String deviceId, String measurement, MeasurementSchema
schema, long insertTime,
- Object objectValue) {
+ Object objectValue) throws WriteProcessException {
Review comment:
Fixed~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [iotdb] HTHou commented on a change in pull request #1524: [IOTDB-776] Control the memory usage of flushing the memtable
HTHou commented on a change in pull request #1524:
URL: https://github.com/apache/iotdb/pull/1524#discussion_r510549586
##
File path: server/src/assembly/resources/conf/iotdb-engine.properties
##
@@ -232,14 +232,16 @@ write_read_free_memory_proportion=4:3:1:2
primitive_array_size=32
# Ratio of memory allocated for buffered arrays, 0.6 by default
-buffered_arrays_memory_proportion=0.4
+buffered_arrays_memory_proportion=0.6
# Flush proportion for system, 0.5 by default
flush_proportion=0.5
# Reject proportion for system, 0.8 by default
reject_proportion=0.8
+# If storage group increased more than this threshold, report to system. The
default value is 16777216
Review comment:
Fixed~
##
File path:
server/src/test/java/org/apache/iotdb/db/utils/datastructure/LongTVListTest.java
##
@@ -30,7 +32,7 @@
@Test
- public void testLongTVList1() {
+ public void testLongTVList1() throws WriteProcessException {
Review comment:
Fixed~
##
File path:
server/src/test/java/org/apache/iotdb/db/utils/datastructure/LongTVListTest.java
##
@@ -21,6 +21,8 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Random;
+
+import org.apache.iotdb.db.exception.WriteProcessException;
Review comment:
Fix~
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
