Hi Gaofei, I suggest that we open a new mail subject to discuss about this. It is friendly for those who wants to search mails by subject.
Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Gaofei Cao <[email protected]> 于2019年1月4日周五 下午11:22写道： > kill_thanos branch in thulab/iotdb has refactored most features as below. > > > 1. Replacing the single point calculation logic with a batch data load > behavior. > In previous branch, the most important two methods in the `Reader` of > IoTDB are `hasNext` and `next` methods, which examine that whether the > given query series has next point and calculate next point. Multiple > invoking of these two methods decreasing the performance of query, so we > added two new methods `hasNextBatch` and `nextBatch`. As a result, we will > load and transfer data in batch rather than a single point. These two > methods are friendly to CPU. > > > 2. Using nio. > In this branch, we replaced ByteArrayInputStream with NIO, taking the > advantage of java NIO. We used `Channel`, `Buffer`, `MMap` more frequently. > > > 3. Adding file stream manager. > In a query of IoTDB, multiple series may be queried, such as a sql `select > * from root.vehicle`. To avoid opened one tsfile multiple times, we > adopting a file stream manager, which ensure that one file will be opened > at most once in IoTDB queries. We adopt an `ExpiredTimeMap` to manage > opened file streams, and close some files when they are not used for a > given expired time. Maybe there are better file stream reader management > methods, I will keep trace it. > > > 4. Optimizing filter efficiency. > Firstly, we removed the previously `Visitor Pattern` implementation of > filter, and adopted an intuitive implementation. > Secondly, we optimized some filter logic to promote performance. For > example, in a sql `select sensor_0, sensor_1 from device_0 where sensor_1 > > 10`, we did some optimization to avoid the duplicate data calculation of > `sensor_1`. > > > 5. Others, such as removing serialization of thrift, changing the file > format of TsFile, maybe someone else can make a supplement. > > > I suggest that merging it into master branch in the next week. > > > Experimental results show that the query test in kill_thanos branch has > approximately 30% ~ 60% performance promotion. > > > By the way, I am considering that how to get a standard, convincing test > data (in IoT domain) to test the writing and querying performance of > IoTDB. Currently, we just use the data generated by `IoTDB Benchmark` > (another project, also available on github.com/thulab/iotdb-benchmark), > which generated 10w row records of 100device * 100sensor. > > > Thanks & Best Regards > > > ----------------------------------- > Cao Gaofei (曹高飞) > School of Software, > Tsinghua University > ----------------------------------- > > > > > > > > ------------------ Original ------------------ > From: "黄向东"<[email protected]>; > Date: Tue, Jan 1, 2019 11:53 PM > To: "dev"<[email protected]>; > > Subject: Plans of January of 2019 > > > > Hi all, > > Happy New Year! Thank all developers and Mentors (and the Champion) of > IoTDB for your efforts in 2018. > > I think the most important things in January of 2019 is migrating all codes > to Apache repository and finishing the website of IoTDB. > > Tasks of the website: > - @Xinyi Zhao and @Yi Xu, can you organize contributors to finish the > website, as well as the English documents? > - By the way, I suggest that we write more documents/JavaDocs to help new > developers to join our project. > > Tasks of migration to Apache repository: > - There are 4 steps: > (1) solving the problem of the IO speed of flushing WAL on disk. @Rui Liu, > can you organize developers to solve it? > (2) Merge the branch `kill_Thanos` into the master on thulab > repository. @Gaofei Cao. > (3) Add Apache header, change all package names, and add JavaDocs for each > java file. > (4) Migrating codes from thulab to Apache repository. > - By the way, don't forget that there are more than 100 issues on > github.com/thulab/iotdb/issues. > > To finish all the above, we need the effort of everyone in our community! > > Here's to a great 2019! > > Thanks, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院
