I’m not sure unit tests are fully passed especially on windows os. Need to re-check.
iPhoneから送信 2019/01/05 11:54、乔嘉林 Jialin Qiao <[email protected]>のメール: > Hi, glad to kill Thanos :) > > I agree to merge kill_thanos into master. We have already made an unofficial > release version(v0.7.1) in Github for master as backups. > > As a reminder, except for these optimizations, some features are unavailable > compared to the master branch: > > 1. Update and Delete operations. > > 2. Advanced queries: Aggregation, GroupByTime and Fill. > > Besides, the ‘hasNextBatch' and ‘nextBatch’ methods are implemented in > TsFile, but most remain to be done in IoTDB engine. > > The kill_thanos changes too much... We can add these features and further > optimize the code with other PRs. > > Best. > > -- > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > >> -----原始邮件----- >> 发件人: "Xiangdong Huang" <[email protected]> >> 发送时间: 2019-01-05 11:36:07 (星期六) >> 收件人: [email protected] >> 抄送: >> 主题: Re: merge kill_thanos branch to the master branch >> >> I think the biggest issue of the current master is that the package >> structures are chaotic. >> The issue prevents new developers to understand the project. >> It is a villain like Thanos in the Marvel Universe. That's why the new >> branch is called kill_Thanos. >> >> Except what Gaofei mentioned, the storage module, TsFile, is also >> refactored, and the file format has some changes. >> A brief introduction is at >> https://github.com/thulab/iotdb/wiki/%5BTsFile%5D-What-is-new-from-v0.7.0--to-Kill_Thanos >> >> >> In the kill_Thanos branch, the package structure is more clear, but there >> are still many source codes can be refactored better. >> However, it brings extra works to merge the modifications from master into >> the kill_Thanos. >> >> Because all UT and IT in current kill_Thanos are passed, and the >> performance is better, I agree to merge the branches as soon as possible. >> >> Best, >> ----------------------------------- >> Xiangdong Huang >> School of Software, Tsinghua University >> >> 黄向东 >> 清华大学 软件学院 >> >> >> Gaofei Cao <[email protected]> 于2019年1月5日周六 上午12:23写道： >> >>> kill_thanos branch in thulab/iotdb has refactored most features as below. >>> >>> >>> 1. Replacing the single point calculation logic with a batch data load >>> behavior. >>> In previous branch, the most important two methods in the `Reader` of >>> IoTDB are `hasNext` and `next` methods, which examine that whether the >>> given query series has next point and calculate next point. Multiple >>> invoking of these two methods decreasing the performance of query, so we >>> added two new methods `hasNextBatch` and `nextBatch`. As a result, we will >>> load and transfer data in batch rather than a single point. These two >>> methods are friendly to CPU. >>> >>> >>> 2. Using nio. >>> In this branch, we replaced ByteArrayInputStream with NIO, taking the >>> advantage of java NIO. We used `Channel`, `Buffer`, `MMap` more frequently. >>> >>> >>> 3. Adding file stream manager. >>> In a query of IoTDB, multiple series may be queried, such as a sql `select >>> * from root.vehicle`. To avoid opened one tsfile multiple times, we >>> adopting a file stream manager, which ensure that one file will be opened >>> at most once in IoTDB queries. We adopt an `ExpiredTimeMap` to manage >>> opened file streams, and close some files when they are not used for a >>> given expired time. Maybe there are better file stream reader management >>> methods, I will keep trace it. >>> >>> >>> 4. Optimizing filter efficiency. >>> Firstly, we removed the previously `Visitor Pattern` implementation of >>> filter, and adopted an intuitive implementation. >>> Secondly, we optimized some filter logic to promote performance. For >>> example, in a sql `select sensor_0, sensor_1 from device_0 where sensor_1 > >>> 10`, we did some optimization to avoid the duplicate data calculation of >>> `sensor_1`. >>> >>> >>> 5. Others, such as removing serialization of thrift, changing the file >>> format of TsFile, maybe someone else can make a supplement. >>> >>> >>> I suggest that merging it into master branch in the next week. >>> >>> >>> Experimental results show that the query test in kill_thanos branch has >>> approximately 30% ~ 60% performance promotion. >>> >>> >>> By the way, I am considering that how to get a standard, convincing test >>> data (in IoT domain) to test the writing and querying performance of >>> IoTDB. Currently, we just use the data generated by `IoTDB Benchmark` >>> (another project, also available on github.com/thulab/iotdb-benchmark), >>> which generated 10w row records of 100device * 100sensor. >>> >>> >>> Thanks & Best Regards >>> >>> >>> ----------------------------------- >>> Cao Gaofei (曹高飞) >>> School of Software, >>> Tsinghua University >>> -----------------------------------
