[GitHub] [incubator-iotdb] samperson1997 commented on issue #279: Spark connector xuekaifeng

2019-07-23 Thread GitBox
samperson1997 commented on issue #279: Spark connector xuekaifeng
URL: https://github.com/apache/incubator-iotdb/pull/279#issuecomment-514490868
 
 
   Hi @SilverNarcissus, this PR could be closed as a newer PR is created 
[here](https://github.com/apache/incubator-iotdb/pull/286).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] samperson1997 commented on issue #286: transform old table form to new table form

2019-07-23 Thread GitBox
samperson1997 commented on issue #286: transform old table form to new table 
form
URL: https://github.com/apache/incubator-iotdb/pull/286#issuecomment-514490401
 
 
   This PR is quite useful to reduce the number of columns when there are too 
many devices!
   
   I have tried this transformation feature by using 
`Transformer.toNewForm(spark, df)` and `Transformer.toOldForm(spark, df)` in 
spark shell.
   Here is an example:
   https://user-images.githubusercontent.com/19167280/61768493-36021680-ae1a-11e9-97af-d857b53978f4.png;>
   https://user-images.githubusercontent.com/19167280/61768501-39959d80-ae1a-11e9-999b-79bd503fa163.png;>
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb-website] kshithijiyer opened a new pull request #28: Adding LICENSE file to the project

2019-07-23 Thread GitBox
kshithijiyer opened a new pull request #28: Adding LICENSE file to the project
URL: https://github.com/apache/incubator-iotdb-website/pull/28
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306612178
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving merged data of {} to the old file {} merged 
chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveMergedToOld(seqFile);
+  }
+  cnt ++;
+  if (logger.isInfoEnabled()) {
+logger.debug("{} has merged {}/{} files", taskName, cnt, 
unmergedFiles.size());
+  }
+}
+if (logger.isInfoEnabled()) {
+  logger.info("{} has merged all files after {}ms", taskName, 
System.currentTimeMillis() - startTime);
+}
+mergeLogger.logMergeEnd();
+  }
+
+  private void moveMergedToOld(TsFileResource seqFile) throws IOException {
+int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 0);
+if (mergedChunkNum == 

[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306612122
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
 
 Review comment:
   I know, but those tsfiles who have useless one-third chunks may not be 
merged anymore. Then the waste will persist, the full merge is just useful for 
new merge files.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306611604
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeManager.java
 ##
 @@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.manage;
+
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.merge.task.MergeTask;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.ServiceType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeManager provides a ThreadPool to queue and run all merge tasks to 
restrain the total
+ * resources occupied by merge and manages a Timer to periodically issue a 
global merge.
+ */
+public class MergeManager implements IService {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeManager.class);
+  private static final MergeManager INSTANCE = new MergeManager();
+
+  private AtomicInteger threadCnt = new AtomicInteger();
+  private ThreadPoolExecutor mergeTaskPool;
+  private ScheduledExecutorService timedMergeThreadPool;
+
+  private MergeManager() {
+  }
+
+  public static MergeManager getINSTANCE() {
+return INSTANCE;
+  }
+
+  public void submit(MergeTask mergeTask) {
+mergeTaskPool.submit(mergeTask);
+  }
+
+  @Override
+  public void start() {
+if (mergeTaskPool == null) {
+  int threadNum = 
IoTDBDescriptor.getInstance().getConfig().getMergeConcurrentThreads();
+  if (threadNum <= 0) {
+threadNum = 1;
+  }
+  mergeTaskPool =
+  (ThreadPoolExecutor) Executors.newFixedThreadPool(threadNum,
+  r -> new Thread(r, "MergeThread-" + 
threadCnt.getAndIncrement()));
+  long mergeInterval = 
IoTDBDescriptor.getInstance().getConfig().getMergeIntervalSec();
+  if (mergeInterval > 0) {
+timedMergeThreadPool = Executors.newSingleThreadScheduledExecutor( r 
-> new Thread(r,
+"TimedMergeThread"));
+timedMergeThreadPool.scheduleAtFixedRate(this::flushAll, mergeInterval,
+mergeInterval, TimeUnit.SECONDS);
+  }
+  logger.info("MergeManager started");
+}
+  }
+
+  @Override
+  public void stop() {
+if (mergeTaskPool != null) {
+  if (timedMergeThreadPool != null) {
+timedMergeThreadPool.shutdownNow();
+timedMergeThreadPool = null;
+  }
+  mergeTaskPool.shutdownNow();
+  logger.info("Waiting for task pool to shut down");
+  while (!mergeTaskPool.isShutdown()) {
 
 Review comment:
   It's useless to judge the condition, because it will false true for 
`mergeTaskPool.isShutdown()` after `shutdownNow `. I think you need to use 
`mergeTaskPool.isTerminated()`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306610686
 
 

 ##
 File path: server/src/main/java/org/apache/iotdb/db/service/IoTDB.java
 ##
 @@ -97,6 +98,7 @@ private void setUp() throws StartupException {
 registerManager.register(Measurement.INSTANCE);
 registerManager.register(SyncServerManager.getInstance());
 registerManager.register(TVListAllocator.getInstance());
+registerManager.register(MergeManager.getINSTANCE());
 
 Review comment:
   I think that registering MergeManager need before registering StorageEngine, 
because you execute RecoverMergeTask in StorageEngine.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306611670
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeManager.java
 ##
 @@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.manage;
+
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.merge.task.MergeTask;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.ServiceType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeManager provides a ThreadPool to queue and run all merge tasks to 
restrain the total
+ * resources occupied by merge and manages a Timer to periodically issue a 
global merge.
+ */
+public class MergeManager implements IService {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeManager.class);
+  private static final MergeManager INSTANCE = new MergeManager();
+
+  private AtomicInteger threadCnt = new AtomicInteger();
+  private ThreadPoolExecutor mergeTaskPool;
+  private ScheduledExecutorService timedMergeThreadPool;
+
+  private MergeManager() {
+  }
 
 Review comment:
   If you don't init pool here, I think it may be a mistake in current register 
order of StroageGroup and MergeManager.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306609546
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/selector/MaxSeriesMergeFileSelector.java
 ##
 @@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.selector;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.MergeException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MaxSeriesMergeFileSelector is an extension of MergeFileSelector which tries 
to maximize the
+ * number of timeseries that can be merged at the same time.
+ */
+public class MaxSeriesMergeFileSelector extends MergeFileSelector {
+
+  public static final int MAX_SERIES_NUM = 1024;
+  private static final Logger logger = 
LoggerFactory.getLogger(MaxSeriesMergeFileSelector.class);
+
+  private List lastSelectedSeqFiles;
+  private List lastSelectedUnseqFiles;
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306607986
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving merged data of {} to the old file {} merged 
chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveMergedToOld(seqFile);
+  }
+  cnt ++;
+  if (logger.isInfoEnabled()) {
+logger.debug("{} has merged {}/{} files", taskName, cnt, 
unmergedFiles.size());
+  }
+}
+if (logger.isInfoEnabled()) {
+  logger.info("{} has merged all files after {}ms", taskName, 
System.currentTimeMillis() - startTime);
+}
+mergeLogger.logMergeEnd();
+  }
+
+  private void moveMergedToOld(TsFileResource seqFile) throws IOException {
+int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 0);
+if (mergedChunkNum == 

[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306607986
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving merged data of {} to the old file {} merged 
chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveMergedToOld(seqFile);
+  }
+  cnt ++;
+  if (logger.isInfoEnabled()) {
+logger.debug("{} has merged {}/{} files", taskName, cnt, 
unmergedFiles.size());
+  }
+}
+if (logger.isInfoEnabled()) {
+  logger.info("{} has merged all files after {}ms", taskName, 
System.currentTimeMillis() - startTime);
+}
+mergeLogger.logMergeEnd();
+  }
+
+  private void moveMergedToOld(TsFileResource seqFile) throws IOException {
+int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 0);
+if (mergedChunkNum == 

[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306607818
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
 
 Review comment:
   I have provided a full merge option and you may use that if your disk space 
is not that much.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306607529
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
 
 Review comment:
   They are not always the same in a RecoverMergeTask and we may have already 
merged some files before the system was down.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306607193
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeContext.java
 ##
 @@ -0,0 +1,60 @@
+package org.apache.iotdb.db.engine.merge.manage;
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306607041
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeTask.java
 ##
 @@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.utils.MergeUtils;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeTask merges given seqFiles and unseqFiles into a new one, which 
basically consists of three
+ * steps: 1. rewrite overflowed, modified or small-sized chunks into temp 
merge files
+ *2. move the merged chunks in the temp files back to the seqFiles or 
move the unmerged
+ *chunks in the seqFiles int temp files and replace the seqFiles with 
the temp files.
+ *3. remove unseqFiles
+ */
+public class MergeTask implements Callable {
+
+  public static final String MERGE_SUFFIX = ".merge";
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeTask.class);
+
+  MergeResource resource;
+  String storageGroupDir;
+  MergeLogger mergeLogger;
+  MergeContext mergeContext = new MergeContext();
+
+  private MergeCallback callback;
+  int concurrentMergeSeriesNum;
+  String taskName;
+  boolean fullMerge;
+
+  public MergeTask(List seqFiles,
+  List unseqFiles, String storageGroupDir, MergeCallback 
callback,
+  String taskName, boolean fullMerge) {
+this.resource = new MergeResource(seqFiles, unseqFiles);
+this.storageGroupDir = storageGroupDir;
+this.callback = callback;
+this.taskName = taskName;
+this.fullMerge = fullMerge;
+this.concurrentMergeSeriesNum = 1;
+  }
+
+  public MergeTask(List seqFiles,
+  List unseqFiles, String storageGroupDir, MergeCallback 
callback,
+  String taskName, boolean fullMerge, int concurrentMergeSeriesNum) {
+this.resource = new MergeResource(seqFiles, unseqFiles);
+this.storageGroupDir = storageGroupDir;
+this.callback = callback;
+this.taskName = taskName;
+this.fullMerge = fullMerge;
+this.concurrentMergeSeriesNum = concurrentMergeSeriesNum;
+  }
+
+  @Override
+  public Void call() throws Exception {
+try  {
+  doMerge();
+} catch (Exception e) {
+  logger.error("Runtime exception in merge {}", taskName, e);
+  cleanUp(false);
+  // call the callback to make sure the StorageGroup exit merging status, 
but passing 2
+  // empty file lists to avoid files being deleted.
+  callback.call(Collections.emptyList(), Collections.emptyList(), new 
File(storageGroupDir, MergeLogger.MERGE_LOG_NAME));
+  throw e;
+}
+return null;
+  }
+
+  private void doMerge() throws IOException {
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} seqFiles, {} unseqFiles", taskName,
+  resource.getSeqFiles().size(), resource.getUnseqFiles().size());
+}
+long startTime = System.currentTimeMillis();
+long totalFileSize = MergeUtils.collectFileSizes(resource.getSeqFiles(),
+resource.getUnseqFiles());
+mergeLogger = new MergeLogger(storageGroupDir);
+
+mergeLogger.logFiles(resource);
+
+List unmergedSeries = MergeUtils.collectPaths(resource);
+unmergedSeries.sort(Comparator.comparing(Path::getFullPath));
 
 Review comment:
   I removed the sort here, but I prefer to explain it in the method document.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this 

[GitHub] [incubator-iotdb] LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache 
for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306588735
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/LruLinkedHashMap.java
 ##
 @@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.cache;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+/**
+ * This class is an LRU cache. Note: It's not thread safe.
+ */
+public abstract class LruLinkedHashMap extends LinkedHashMap {
 
 Review comment:
   @qiaojialin actually I don't like the function `suggested change`. Look 
here, you changed the class name but forget to change the file name, which 
Suyue doesn't notice too.
   I personally prefer you just point out where the problem is instead of 
changing it for her.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jixuan1989 opened a new pull request #296: Cherry pick build module from 0.8

2019-07-23 Thread GitBox
jixuan1989 opened a new pull request #296: Cherry pick build module from 0.8
URL: https://github.com/apache/incubator-iotdb/pull/296
 
 
   cherry pick from rel/0.8 and set the version as 0.9-SNAPSHOT.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer merged pull request #295: add build module

2019-07-23 Thread GitBox
JulianFeinauer merged pull request #295: add build module
URL: https://github.com/apache/incubator-iotdb/pull/295
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer commented on issue #295: add build module

2019-07-23 Thread GitBox
JulianFeinauer commented on issue #295: add build module
URL: https://github.com/apache/incubator-iotdb/pull/295#issuecomment-514239099
 
 
   Thank you @jixuan1989 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jixuan1989 commented on issue #295: add build module

2019-07-23 Thread GitBox
jixuan1989 commented on issue #295: add build module
URL: https://github.com/apache/incubator-iotdb/pull/295#issuecomment-514228290
 
 
   It is because that two new xml files lacking the Apache Rat... I have fixed 
them.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer commented on issue #295: add build module

2019-07-23 Thread GitBox
JulianFeinauer commented on issue #295: add build module
URL: https://github.com/apache/incubator-iotdb/pull/295#issuecomment-514221940
 
 
   Hey @qiaojialin, I tested it locally and it works for me but Travis seems to 
fail. Could you check that? All changes look fine for me, so when travis passes 
I would merge it.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
little-emotion commented on a change in pull request #262: [IOTDB-144]meta data 
cache for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306319203
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/LruLinkedHashMap.java
 ##
 @@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.cache;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+/**
+ * This class is an LRU cache. Note: It's not thread safe.
+ */
+public abstract class LruLinkedHashMap extends LinkedHashMap {
 
 Review comment:
   ok


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
little-emotion commented on a change in pull request #262: [IOTDB-144]meta data 
cache for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306319157
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/RamUsageEstimator.java
 ##
 @@ -0,0 +1,1019 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.cache;
+
+import java.lang.management.ManagementFactory;
+import java.lang.reflect.Array;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.text.DecimalFormat;
+import java.text.DecimalFormatSymbols;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.IdentityHashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Map;
+import java.util.NoSuchElementException;
+
+/**
+ * This class is copied from apache lucene, version 4.6.1. Estimates the 
size(memory representation)
+ * of Java objects. 
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/4.6.1/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin opened a new pull request #295: add build module

2019-07-23 Thread GitBox
qiaojialin opened a new pull request #295: add build module
URL: https://github.com/apache/incubator-iotdb/pull/295
 
 
   You can build a "build-0.8.0-release.tar.gz" under build/target by executing 
"mvn clean package"


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
little-emotion commented on a change in pull request #262: [IOTDB-144]meta data 
cache for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306308298
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/query/reader/resourceRelated/UnseqResourceMergeReader.java
 ##
 @@ -84,12 +81,12 @@ public UnseqResourceMergeReader(Path seriesPath, 
List unseqResou
 continue;
   }
 }
-tsFileReader = FileReaderManager.getInstance()
-.get(tsFileResource.getFile().getPath(), false);
 metaDataList = tsFileResource.getChunkMetaDatas();
   }
 
   // create and add ChunkReader with priority
+  TsFileSequenceReader tsFileReader = FileReaderManager.getInstance()
+  .get(tsFileResource.getFile().getPath(), tsFileResource.isClosed());
   ChunkLoaderImpl chunkLoader = new ChunkLoaderImpl(tsFileReader);
   for (ChunkMetaData chunkMetaData : metaDataList) {
 
 Review comment:
   ok


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306274658
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving merged data of {} to the old file {} merged 
chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveMergedToOld(seqFile);
+  }
+  cnt ++;
+  if (logger.isInfoEnabled()) {
+logger.debug("{} has merged {}/{} files", taskName, cnt, 
unmergedFiles.size());
+  }
+}
+if (logger.isInfoEnabled()) {
+  logger.info("{} has merged all files after {}ms", taskName, 
System.currentTimeMillis() - startTime);
+}
+mergeLogger.logMergeEnd();
+  }
+
+  private void moveMergedToOld(TsFileResource seqFile) throws IOException {
+int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 0);
+if (mergedChunkNum == 

[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306269721
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
 
 Review comment:
   `unmergedSeqFiles` is in `MergeResource`, so it's unnecessary to be an 
independent parameter.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306149465
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeContext.java
 ##
 @@ -0,0 +1,60 @@
+package org.apache.iotdb.db.engine.merge.manage;
 
 Review comment:
   Please add ASF header


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306271414
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
 
 Review comment:
   Well´╝îin the worst case, each tsfile have nearly one-third of the chunks 
useless, which just occupy space. As a result, ultimately only two-thirds of 
the actual utilization of disks, so whether this mode is desirable remains to 
be discussed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306290969
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/selector/MaxSeriesMergeFileSelector.java
 ##
 @@ -0,0 +1,100 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.selector;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.exception.MergeException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MaxSeriesMergeFileSelector is an extension of MergeFileSelector which tries 
to maximize the
+ * number of timeseries that can be merged at the same time.
+ */
+public class MaxSeriesMergeFileSelector extends MergeFileSelector {
+
+  public static final int MAX_SERIES_NUM = 1024;
+  private static final Logger logger = 
LoggerFactory.getLogger(MaxSeriesMergeFileSelector.class);
+
+  private List lastSelectedSeqFiles;
+  private List lastSelectedUnseqFiles;
 
 Review comment:
   It seems that these two fields are useless, please check.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] fanhualta commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r306149187
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeTask.java
 ##
 @@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.List;
+import java.util.concurrent.Callable;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.db.utils.MergeUtils;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeTask merges given seqFiles and unseqFiles into a new one, which 
basically consists of three
+ * steps: 1. rewrite overflowed, modified or small-sized chunks into temp 
merge files
+ *2. move the merged chunks in the temp files back to the seqFiles or 
move the unmerged
+ *chunks in the seqFiles int temp files and replace the seqFiles with 
the temp files.
+ *3. remove unseqFiles
+ */
+public class MergeTask implements Callable {
+
+  public static final String MERGE_SUFFIX = ".merge";
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeTask.class);
+
+  MergeResource resource;
+  String storageGroupDir;
+  MergeLogger mergeLogger;
+  MergeContext mergeContext = new MergeContext();
+
+  private MergeCallback callback;
+  int concurrentMergeSeriesNum;
+  String taskName;
+  boolean fullMerge;
+
+  public MergeTask(List seqFiles,
+  List unseqFiles, String storageGroupDir, MergeCallback 
callback,
+  String taskName, boolean fullMerge) {
+this.resource = new MergeResource(seqFiles, unseqFiles);
+this.storageGroupDir = storageGroupDir;
+this.callback = callback;
+this.taskName = taskName;
+this.fullMerge = fullMerge;
+this.concurrentMergeSeriesNum = 1;
+  }
+
+  public MergeTask(List seqFiles,
+  List unseqFiles, String storageGroupDir, MergeCallback 
callback,
+  String taskName, boolean fullMerge, int concurrentMergeSeriesNum) {
+this.resource = new MergeResource(seqFiles, unseqFiles);
+this.storageGroupDir = storageGroupDir;
+this.callback = callback;
+this.taskName = taskName;
+this.fullMerge = fullMerge;
+this.concurrentMergeSeriesNum = concurrentMergeSeriesNum;
+  }
+
+  @Override
+  public Void call() throws Exception {
+try  {
+  doMerge();
+} catch (Exception e) {
+  logger.error("Runtime exception in merge {}", taskName, e);
+  cleanUp(false);
+  // call the callback to make sure the StorageGroup exit merging status, 
but passing 2
+  // empty file lists to avoid files being deleted.
+  callback.call(Collections.emptyList(), Collections.emptyList(), new 
File(storageGroupDir, MergeLogger.MERGE_LOG_NAME));
+  throw e;
+}
+return null;
+  }
+
+  private void doMerge() throws IOException {
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} seqFiles, {} unseqFiles", taskName,
+  resource.getSeqFiles().size(), resource.getUnseqFiles().size());
+}
+long startTime = System.currentTimeMillis();
+long totalFileSize = MergeUtils.collectFileSizes(resource.getSeqFiles(),
+resource.getUnseqFiles());
+mergeLogger = new MergeLogger(storageGroupDir);
+
+mergeLogger.logFiles(resource);
+
+List unmergedSeries = MergeUtils.collectPaths(resource);
+unmergedSeries.sort(Comparator.comparing(Path::getFullPath));
 
 Review comment:
   You have sorted the list twice, please remove one of them: this line or in 
MergeUtils.collectPaths. By the way, if remove this line, it's better to rename 
`collectPaths ` to `collectPathsAndSort`


This is an automated message from the Apache Git Service.
To respond to 

[GitHub] [incubator-iotdb] fanhualta commented on issue #258: [IOTDB-143]Development of merge

2019-07-23 Thread GitBox
fanhualta commented on issue #258: [IOTDB-143]Development of merge
URL: https://github.com/apache/incubator-iotdb/pull/258#issuecomment-514190364
 
 
   Excellent job. The code structure is graceful. I can learn a lot from it. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
little-emotion commented on a change in pull request #262: [IOTDB-144]meta data 
cache for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306284921
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/DeviceMetaDataCache.java
 ##
 @@ -19,77 +19,144 @@
 package org.apache.iotdb.db.engine.cache;
 
 import java.io.IOException;
-import java.util.LinkedHashMap;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
 import java.util.Map;
-import java.util.Objects;
+import java.util.Set;
 import java.util.concurrent.atomic.AtomicLong;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
 import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadata;
 import org.apache.iotdb.tsfile.file.metadata.TsFileMetaData;
+import org.apache.iotdb.tsfile.read.common.Path;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * This class is used to cache DeviceMetaDataCache of tsfile in 
IoTDB.
+ * This class is used to cache List of tsfile in 
IoTDB. The caching
+ * strategy is LRU.
  */
 public class DeviceMetaDataCache {
 
   private static final Logger logger = 
LoggerFactory.getLogger(DeviceMetaDataCache.class);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
 
-  private static final int CACHE_SIZE = 100;
+  private static StorageEngine storageEngine = StorageEngine.getInstance();
+
+  private static final long MEMORY_THRESHOLD_IN_B = (long) (0.3 * config
+  .getAllocateMemoryForRead());
   /**
-   * key: the file path + deviceId.
+   * key: file path dot deviceId dot sensorId.
+   * 
+   * value: chunkMetaData list of one timeseries in the file.
*/
-  private LinkedHashMap lruCache;
+  private LruLinkedHashMap> lruCache;
 
   private AtomicLong cacheHintNum = new AtomicLong();
   private AtomicLong cacheRequestNum = new AtomicLong();
 
-  private DeviceMetaDataCache(int cacheSize) {
-lruCache = new LruLinkedHashMap(cacheSize, true);
+  /**
+   * approximate estimation of chunkMetaData size
+   */
+  private long chunkMetaDataSize = 0;
+
+  private DeviceMetaDataCache(long memoryThreshold) {
+lruCache = new LruLinkedHashMap>(memoryThreshold, true) {
+  @Override
+  protected long calEntrySize(String key, List value) {
+if (chunkMetaDataSize == 0 && !value.isEmpty()) {
+  chunkMetaDataSize = RamUsageEstimator.sizeOf(value.get(0));
+}
+return value.size() * chunkMetaDataSize + key.length() * 2;
+  }
+};
   }
 
   public static DeviceMetaDataCache getInstance() {
 return RowGroupBlockMetaDataCacheSingleton.INSTANCE;
   }
 
   /**
-   * get {@link TsDeviceMetadata}. THREAD SAFE.
+   * get {@link ChunkMetaData}. THREAD SAFE.
*/
-  public TsDeviceMetadata get(String filePath, String deviceId, TsFileMetaData 
fileMetaData)
+  public List get(String filePath, Path seriesPath)
   throws IOException {
-// The key(the tsfile path and deviceId) for the lruCache
+StringBuilder builder = new 
StringBuilder(filePath).append(".").append(seriesPath.getDevice());
+String pathDeviceStr = builder.toString();
+String key = 
builder.append(".").append(seriesPath.getMeasurement()).toString();
+Object devicePathObject = pathDeviceStr.intern();
 
-String jointPath = filePath + deviceId;
-Object jointPathObject = jointPath.intern();
 synchronized (lruCache) {
   cacheRequestNum.incrementAndGet();
-  if (lruCache.containsKey(jointPath)) {
+  if (lruCache.containsKey(key)) {
 cacheHintNum.incrementAndGet();
 if (logger.isDebugEnabled()) {
   logger.debug(
-  "Cache hint: the number of requests for cache is {}, "
+  "Cache hit: the number of requests for cache is {}, "
   + "the number of hints for cache is {}",
   cacheRequestNum.get(), cacheHintNum.get());
 }
-return lruCache.get(jointPath);
+return new ArrayList<>(lruCache.get(key));
   }
 }
-synchronized (jointPathObject) {
+synchronized (devicePathObject) {
   synchronized (lruCache) {
-if (lruCache.containsKey(jointPath)) {
-  return lruCache.get(jointPath);
+if (lruCache.containsKey(key)) {
+  cacheHintNum.incrementAndGet();
+  return new ArrayList<>(lruCache.get(key));
 }
   }
   if (logger.isDebugEnabled()) {
-logger.debug("Cache didn't hint: the number of requests for cache is 
{}",
+logger.debug("Cache didn't hit: the number of requests for cache is 
{}",
 cacheRequestNum.get());
   }
+  TsFileMetaData fileMetaData = 
TsFileMetaDataCache.getInstance().get(filePath);
   TsDeviceMetadata blockMetaData = TsFileMetadataUtils
-  

[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
little-emotion commented on a change in pull request #262: [IOTDB-144]meta data 
cache for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306279001
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/TsFileMetaDataCache.java
 ##
 @@ -31,67 +32,113 @@
 public class TsFileMetaDataCache {
 
   private static final Logger logger = 
LoggerFactory.getLogger(TsFileMetaDataCache.class);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
+
+  private static final long MEMORY_THRESHOLD_IN_B = (long) (0.25 * config
+  .getAllocateMemoryForRead());
   /**
-   * key: The file seriesPath of tsfile.
+   * key: Tsfile path. value: TsFileMetaData
*/
-  private ConcurrentHashMap cache;
+  private LruLinkedHashMap cache;
   private AtomicLong cacheHintNum = new AtomicLong();
 
 Review comment:
   ok


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
little-emotion commented on a change in pull request #262: [IOTDB-144]meta data 
cache for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306278898
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/DeviceMetaDataCache.java
 ##
 @@ -19,77 +19,144 @@
 package org.apache.iotdb.db.engine.cache;
 
 import java.io.IOException;
-import java.util.LinkedHashMap;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
 import java.util.Map;
-import java.util.Objects;
+import java.util.Set;
 import java.util.concurrent.atomic.AtomicLong;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
 import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadata;
 import org.apache.iotdb.tsfile.file.metadata.TsFileMetaData;
+import org.apache.iotdb.tsfile.read.common.Path;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * This class is used to cache DeviceMetaDataCache of tsfile in 
IoTDB.
+ * This class is used to cache List of tsfile in 
IoTDB. The caching
+ * strategy is LRU.
  */
 public class DeviceMetaDataCache {
 
   private static final Logger logger = 
LoggerFactory.getLogger(DeviceMetaDataCache.class);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
 
-  private static final int CACHE_SIZE = 100;
+  private static StorageEngine storageEngine = StorageEngine.getInstance();
+
+  private static final long MEMORY_THRESHOLD_IN_B = (long) (0.3 * config
+  .getAllocateMemoryForRead());
   /**
-   * key: the file path + deviceId.
+   * key: file path dot deviceId dot sensorId.
+   * 
+   * value: chunkMetaData list of one timeseries in the file.
*/
-  private LinkedHashMap lruCache;
+  private LruLinkedHashMap> lruCache;
 
   private AtomicLong cacheHintNum = new AtomicLong();
 
 Review comment:
   hit.It's a typo.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
little-emotion commented on a change in pull request #262: [IOTDB-144]meta data 
cache for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306277876
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
 ##
 @@ -147,6 +150,12 @@
*/
   private ModificationFile mergingModification;
 
+  /**
+   * This linked hash set records the access order of sensors used by query.
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #294: [IOTDB-152]return data type in TSExecuteStatementResp

2019-07-23 Thread GitBox
qiaojialin commented on a change in pull request #294: [IOTDB-152]return data 
type in TSExecuteStatementResp
URL: https://github.com/apache/incubator-iotdb/pull/294#discussion_r306273947
 
 

 ##
 File path: service-rpc/src/main/thrift/rpc.thrift
 ##
 @@ -74,6 +74,9 @@ struct TSExecuteStatementResp {
3: optional list columns
4: optional string operationType
5: optional bool ignoreTimeStamp
+
+  // Data type list of columns in select statement of SQL
+  6: optional list dataTypeList
 
 Review comment:
   format


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] little-emotion opened a new pull request #294: [IOTDB-152]return data type in TSExecuteStatementResp

2019-07-23 Thread GitBox
little-emotion opened a new pull request #294: [IOTDB-152]return data type in 
TSExecuteStatementResp
URL: https://github.com/apache/incubator-iotdb/pull/294
 
 
   When executing the query statement, after sending executeStatement request 
to server and return the execResp, the client  send several TSFetchMetadata  
requests to server for fetching data types of the query columns. So we can 
return dataTypes in  executeStatement request to avoid the following 
fetchmetaData requests.
   Thank Rui Lei for discovering this problem.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
qiaojialin commented on a change in pull request #262: [IOTDB-144]meta data 
cache for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306268289
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/RamUsageEstimator.java
 ##
 @@ -0,0 +1,1019 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.engine.cache;
+
+import java.lang.management.ManagementFactory;
+import java.lang.reflect.Array;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.lang.reflect.Modifier;
+import java.text.DecimalFormat;
+import java.text.DecimalFormatSymbols;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.IdentityHashMap;
+import java.util.Iterator;
+import java.util.Locale;
+import java.util.Map;
+import java.util.NoSuchElementException;
+
+/**
+ * This class is copied from apache lucene, version 4.6.1. Estimates the 
size(memory representation)
+ * of Java objects. 
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/4.6.1/lucene/core/src/java/org/apache/lucene/util/RamUsageEstimator.java
 
 Review comment:
   This should be listed in the License file


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
qiaojialin commented on a change in pull request #262: [IOTDB-144]meta data 
cache for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306267546
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/LruLinkedHashMap.java
 ##
 @@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.cache;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+/**
+ * This class is an LRU cache. Note: It's not thread safe.
+ */
+public abstract class LruLinkedHashMap extends LinkedHashMap {
 
 Review comment:
   ```suggestion
   public abstract class LRULinkedHashMap extends LinkedHashMap {
   ```


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #285: Add micro and nano timestamp precision

2019-07-23 Thread GitBox
qiaojialin merged pull request #285: Add micro and nano timestamp precision
URL: https://github.com/apache/incubator-iotdb/pull/285
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache 
for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306205394
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/query/reader/resourceRelated/UnseqResourceMergeReader.java
 ##
 @@ -84,12 +81,12 @@ public UnseqResourceMergeReader(Path seriesPath, 
List unseqResou
 continue;
   }
 }
-tsFileReader = FileReaderManager.getInstance()
-.get(tsFileResource.getFile().getPath(), false);
 metaDataList = tsFileResource.getChunkMetaDatas();
   }
 
   // create and add ChunkReader with priority
+  TsFileSequenceReader tsFileReader = FileReaderManager.getInstance()
+  .get(tsFileResource.getFile().getPath(), tsFileResource.isClosed());
   ChunkLoaderImpl chunkLoader = new ChunkLoaderImpl(tsFileReader);
   for (ChunkMetaData chunkMetaData : metaDataList) {
 
 Review comment:
   If metaDataList size equals 0, then the creation of TsFileSequenceReader can 
be skipped.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #278: [IOTDB-149] Fix wrong Windows path.

2019-07-23 Thread GitBox
qiaojialin merged pull request #278: [IOTDB-149]  Fix wrong Windows path.
URL: https://github.com/apache/incubator-iotdb/pull/278
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer commented on issue #293: move artifacts to target folder

2019-07-23 Thread GitBox
JulianFeinauer commented on issue #293: move artifacts to target folder
URL: https://github.com/apache/incubator-iotdb/pull/293#issuecomment-514114843
 
 
   Looks good to me!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer merged pull request #293: move artifacts to target folder

2019-07-23 Thread GitBox
JulianFeinauer merged pull request #293: move artifacts to target folder
URL: https://github.com/apache/incubator-iotdb/pull/293
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache 
for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306198470
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/DeviceMetaDataCache.java
 ##
 @@ -19,77 +19,144 @@
 package org.apache.iotdb.db.engine.cache;
 
 import java.io.IOException;
-import java.util.LinkedHashMap;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
 import java.util.Map;
-import java.util.Objects;
+import java.util.Set;
 import java.util.concurrent.atomic.AtomicLong;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
 import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadata;
 import org.apache.iotdb.tsfile.file.metadata.TsFileMetaData;
+import org.apache.iotdb.tsfile.read.common.Path;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * This class is used to cache DeviceMetaDataCache of tsfile in 
IoTDB.
+ * This class is used to cache List of tsfile in 
IoTDB. The caching
+ * strategy is LRU.
  */
 public class DeviceMetaDataCache {
 
   private static final Logger logger = 
LoggerFactory.getLogger(DeviceMetaDataCache.class);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
 
-  private static final int CACHE_SIZE = 100;
+  private static StorageEngine storageEngine = StorageEngine.getInstance();
+
+  private static final long MEMORY_THRESHOLD_IN_B = (long) (0.3 * config
+  .getAllocateMemoryForRead());
   /**
-   * key: the file path + deviceId.
+   * key: file path dot deviceId dot sensorId.
+   * 
+   * value: chunkMetaData list of one timeseries in the file.
*/
-  private LinkedHashMap lruCache;
+  private LruLinkedHashMap> lruCache;
 
   private AtomicLong cacheHintNum = new AtomicLong();
   private AtomicLong cacheRequestNum = new AtomicLong();
 
-  private DeviceMetaDataCache(int cacheSize) {
-lruCache = new LruLinkedHashMap(cacheSize, true);
+  /**
+   * approximate estimation of chunkMetaData size
+   */
+  private long chunkMetaDataSize = 0;
+
+  private DeviceMetaDataCache(long memoryThreshold) {
+lruCache = new LruLinkedHashMap>(memoryThreshold, true) {
+  @Override
+  protected long calEntrySize(String key, List value) {
+if (chunkMetaDataSize == 0 && !value.isEmpty()) {
+  chunkMetaDataSize = RamUsageEstimator.sizeOf(value.get(0));
+}
+return value.size() * chunkMetaDataSize + key.length() * 2;
+  }
+};
   }
 
   public static DeviceMetaDataCache getInstance() {
 return RowGroupBlockMetaDataCacheSingleton.INSTANCE;
   }
 
   /**
-   * get {@link TsDeviceMetadata}. THREAD SAFE.
+   * get {@link ChunkMetaData}. THREAD SAFE.
*/
-  public TsDeviceMetadata get(String filePath, String deviceId, TsFileMetaData 
fileMetaData)
+  public List get(String filePath, Path seriesPath)
   throws IOException {
-// The key(the tsfile path and deviceId) for the lruCache
+StringBuilder builder = new 
StringBuilder(filePath).append(".").append(seriesPath.getDevice());
+String pathDeviceStr = builder.toString();
+String key = 
builder.append(".").append(seriesPath.getMeasurement()).toString();
+Object devicePathObject = pathDeviceStr.intern();
 
-String jointPath = filePath + deviceId;
-Object jointPathObject = jointPath.intern();
 synchronized (lruCache) {
   cacheRequestNum.incrementAndGet();
-  if (lruCache.containsKey(jointPath)) {
+  if (lruCache.containsKey(key)) {
 cacheHintNum.incrementAndGet();
 if (logger.isDebugEnabled()) {
   logger.debug(
-  "Cache hint: the number of requests for cache is {}, "
+  "Cache hit: the number of requests for cache is {}, "
   + "the number of hints for cache is {}",
   cacheRequestNum.get(), cacheHintNum.get());
 }
-return lruCache.get(jointPath);
+return new ArrayList<>(lruCache.get(key));
   }
 }
-synchronized (jointPathObject) {
+synchronized (devicePathObject) {
   synchronized (lruCache) {
-if (lruCache.containsKey(jointPath)) {
-  return lruCache.get(jointPath);
+if (lruCache.containsKey(key)) {
+  cacheHintNum.incrementAndGet();
+  return new ArrayList<>(lruCache.get(key));
 }
   }
   if (logger.isDebugEnabled()) {
-logger.debug("Cache didn't hint: the number of requests for cache is 
{}",
+logger.debug("Cache didn't hit: the number of requests for cache is 
{}",
 cacheRequestNum.get());
   }
+  TsFileMetaData fileMetaData = 
TsFileMetaDataCache.getInstance().get(filePath);
   TsDeviceMetadata blockMetaData = TsFileMetadataUtils
-  .getTsRowGroupBlockMetaData(filePath, 

[GitHub] [incubator-iotdb] LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache 
for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306186002
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/TsFileMetaDataCache.java
 ##
 @@ -31,67 +32,113 @@
 public class TsFileMetaDataCache {
 
   private static final Logger logger = 
LoggerFactory.getLogger(TsFileMetaDataCache.class);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
+
+  private static final long MEMORY_THRESHOLD_IN_B = (long) (0.25 * config
+  .getAllocateMemoryForRead());
   /**
-   * key: The file seriesPath of tsfile.
+   * key: Tsfile path. value: TsFileMetaData
*/
-  private ConcurrentHashMap cache;
+  private LruLinkedHashMap cache;
   private AtomicLong cacheHintNum = new AtomicLong();
 
 Review comment:
   Hit


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin opened a new pull request #293: move artifacts to target folder

2019-07-23 Thread GitBox
qiaojialin opened a new pull request #293: move artifacts to target folder
URL: https://github.com/apache/incubator-iotdb/pull/293
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #292: Fix latest url

2019-07-23 Thread GitBox
qiaojialin merged pull request #292: Fix latest url
URL: https://github.com/apache/incubator-iotdb/pull/292
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache 
for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306152494
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/storagegroup/StorageGroupProcessor.java
 ##
 @@ -147,6 +150,12 @@
*/
   private ModificationFile mergingModification;
 
+  /**
+   * This linked hash set records the access order of sensors used by query.
 
 Review comment:
   linked list 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache for query

2019-07-23 Thread GitBox
LeiRui commented on a change in pull request #262: [IOTDB-144]meta data cache 
for query
URL: https://github.com/apache/incubator-iotdb/pull/262#discussion_r306152636
 
 

 ##
 File path: 
iotdb/src/main/java/org/apache/iotdb/db/engine/cache/DeviceMetaDataCache.java
 ##
 @@ -19,77 +19,144 @@
 package org.apache.iotdb.db.engine.cache;
 
 import java.io.IOException;
-import java.util.LinkedHashMap;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
 import java.util.Map;
-import java.util.Objects;
+import java.util.Set;
 import java.util.concurrent.atomic.AtomicLong;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
 import org.apache.iotdb.tsfile.file.metadata.TsDeviceMetadata;
 import org.apache.iotdb.tsfile.file.metadata.TsFileMetaData;
+import org.apache.iotdb.tsfile.read.common.Path;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 /**
- * This class is used to cache DeviceMetaDataCache of tsfile in 
IoTDB.
+ * This class is used to cache List of tsfile in 
IoTDB. The caching
+ * strategy is LRU.
  */
 public class DeviceMetaDataCache {
 
   private static final Logger logger = 
LoggerFactory.getLogger(DeviceMetaDataCache.class);
+  private static final IoTDBConfig config = 
IoTDBDescriptor.getInstance().getConfig();
 
-  private static final int CACHE_SIZE = 100;
+  private static StorageEngine storageEngine = StorageEngine.getInstance();
+
+  private static final long MEMORY_THRESHOLD_IN_B = (long) (0.3 * config
+  .getAllocateMemoryForRead());
   /**
-   * key: the file path + deviceId.
+   * key: file path dot deviceId dot sensorId.
+   * 
+   * value: chunkMetaData list of one timeseries in the file.
*/
-  private LinkedHashMap lruCache;
+  private LruLinkedHashMap> lruCache;
 
   private AtomicLong cacheHintNum = new AtomicLong();
 
 Review comment:
   Hit or hint?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin opened a new pull request #292: Fix latest url

2019-07-23 Thread GitBox
qiaojialin opened a new pull request #292: Fix latest url
URL: https://github.com/apache/incubator-iotdb/pull/292
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #291: Cherry pick rel

2019-07-23 Thread GitBox
qiaojialin merged pull request #291: Cherry pick rel
URL: https://github.com/apache/incubator-iotdb/pull/291
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services