Hi, Following is a summary of SDT's encoding and decoding implementation in IoTDB.
- SDT is mainly to calculate the up and down slopes of the data to the segment starting point. If it is within the compression deviation CD range, discard the data. If it exceeds the CD, the original data is stored - In IoTDB, the SDT can act as a new Encoding method. It works inside each Page (PageWriter and PageReader). - Will support with and without timestamp encoding. - For without timestamps encoding, we will record the count of data points in each segment in the page header. Page header needs to maintain a Map<segmentStartTime, count> Encoder will be changed to encode(long time, long value) Data buffer will be stored in each Encoder Decoder will be changed to getTime(), getXXValue() Encoder, Decoder will take both a data column and a series. If you have any question or comment, you are more than welcome to reply! Thank you, Haimei On Mon, Sep 28, 2020 at 1:20 PM Jialin Qiao <[email protected]> wrote: > Hi Haimei, > > Good work! This doc is comprehensive :) > > As for the implementation in IoTDB, here are some points: > > (1) First, SDT could act as a new Encoding method in IoTDB. It works > inside each Page (PageWriter and PageReader). > (2) The interface of Encoder could be changed to encode(long time, XX > value). The interface of Decoder could be change to getTime(), > getXXValue(). Which is, the encoder and decoder is not only responsible for > one data column but a series. This involves some reconstruction of the > Encoder and Decoder, the data buffers should be stored inside each encoder. > (3) For the SDT without timestamps, we need to record the count of each > segment. > (4) We could offer two encodings, SDT with timestamps and SDT without > timestamps. > > Thanks, > -- > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > > > -----原始邮件----- > > 发件人: "[email protected]" <[email protected]> > > 发送时间: 2020-09-25 11:56:16 (星期五) > > 收件人: dev <[email protected]> > > 抄送: > > 主题: Re: Support SDT compression > > > > Great work！ > > > > > > > > Thanks. > > > > Chao Wang > > BONC Ltd > > > > > > From: Eileen Guo > > Date: 2020-09-25 11:47 > > To: dev > > Subject: Support SDT compression > > Hi all, > > > > I've completed a design draft for supporting swinging door compression. > > > > Jira: jira SDT link > > < > https://issues.apache.org/jira/browse/IOTDB-890?filter=-4&jql=assignee%20in%20(haimeiguo)%20order%20by%20created%20DESC > > > > design doc: SDT design doc link > > < > https://docs.google.com/document/d/1VeTwVsm4CkQSVR65bWw9pKg6gRdiDYUz0lBBhWXHl5A/edit?usp=sharing > > > > > > The doc explains SDT algorithm, compression and decompression process, > > performance tests and SDT + IoTDB implementation and usage. > > > > There is still some question about where to use this algorithm. If you > have > > any idea, welcome to comment. > > > > Thank you! > > Haimei Guo >
