fanhualta commented on a change in pull request #664: [IOTDB-333] Storage group
divide by time range
URL: https://github.com/apache/incubator-iotdb/pull/664#discussion_r361043320
##########
File path:
server/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
##########
@@ -514,7 +572,70 @@ private TsFileProcessor
getOrCreateTsFileProcessor(boolean sequence) {
return tsFileProcessor;
}
- private TsFileProcessor createTsFileProcessor(boolean sequence)
+ /**
+ * get processor from hashmap, flush oldest processor is necessary
+ *
+ * @param timeRangeId time partition range
+ * @param tsFileProcessorHashMap tsFileProcessorHashMap
+ * @param tsfileProcessorLastUseTime last use time of this processor map
+ * @param fileList file list to add new processor
+ * @param sequence whether is sequence or not
+ */
+ private TsFileProcessor getOrCreateTsFileProcessorIntern(long timeRangeId,
+ HashMap<Long, TsFileProcessor> tsFileProcessorHashMap,
+ HashMap<Long, Long> tsfileProcessorLastUseTime,
+ Collection<TsFileResource> fileList,
+ boolean sequence)
+ throws IOException, DiskSpaceInsufficientException {
+
+ TsFileProcessor res = null;
+ // we have to ensure only one thread can change
workSequenceTsFileProcessors
+ writeLock();
+ try {
+ if (!tsFileProcessorHashMap.containsKey(timeRangeId)) {
+ // we have to remove oldest processor to control the num of the
memtables
+ if (tsFileProcessorHashMap.size()
+ >= IoTDBConstant.MEMTABLE_NUM_IN_EACH_STORAGE_GROUP / 2) {
+ long oldestTimeRange = -1;
+ long oldestUseTime = Long.MAX_VALUE;
+ for (long curTimeRange : tsFileProcessorHashMap.keySet()) {
+ long curUseTime = tsfileProcessorLastUseTime.get(curTimeRange);
+ if (oldestUseTime > curUseTime) {
+ oldestUseTime = curUseTime;
+ oldestTimeRange = curTimeRange;
+ }
+ }
+ if (oldestTimeRange == -1) {
+ throw new
IllegalStateException("MEMTABLE_NUM_IN_EACH_STORAGE_GROUP size is too small: "
+ + IoTDBConstant.MEMTABLE_NUM_IN_EACH_STORAGE_GROUP);
+ }
+
+ moveOneWorkProcessorToClosingList(sequence,
tsFileProcessorHashMap.get(oldestTimeRange));
Review comment:
Why you put the logic here? After you classify the batch insert and before
you finish all operations, you may force a tsfile to close. In this case, the
result of classifying batch insert is still right? I don't think so. I think it
may be a severe bug.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
