Hi,
This is not the current implementation... We do not have a partition folder on
disk now.
By adding a partition folder, there is no need to store all TsFileResources in
the memory, and the device index will not hurt us.
Thanks,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -----原始邮件-----
> 发件人: "Xiangdong Huang" <saint...@gmail.com>
> 发送时间: 2020-07-21 18:46:31 (星期二)
> 收件人: dev <dev@iotdb.apache.org>
> 抄送:
> 主题: Re: [Discuss] How to delivery the device concept to users
>
> Hi Jialin,
>
> Yes it is current logic. But I do not know the relation between what you
> said and this discussion...
>
> Best,
> -----------------------------------
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Jialin Qiao <qj...@mails.tsinghua.edu.cn> 于2020年7月21日周二 下午4:47写道：
>
> > Hi,
> >
> > I would like to give a vision about managing the data files according to
> > time partition.
> >
> > After we introduce the time partition (data is partitioned by time
> > interval), we do split them in memory and different TsFiles. But we may
> > lake a partition folder layer on top of the TsFiles.
> >
> > Maybe it should work as follows:
> >
> > E.g., we insert data into storage group root.sg from 2020-07-19 to
> > 2020-07-21 and the partition interval is 1 day.
> > First, we create three folders (2020-07-19, 2020-07-20, 2020-07-21) under
> > root.sg that belongs to each partition.
> > Then, we store TsFiles to its related partition folder.
> >
> > An example of TsFiles on disk is as follows:
> >
> > sequence
> > ├── root.sg
> > │ ├── 2020-07-19
> > │ │ └── timestamp1-version1-merge.tsfile
> > │ │ └── timestamp1-version1-merge.tsfile.resource
> > │ │ └── ...
> > │ ├── 2020-07-20
> > │ ├── 2020-07-21
> >
> >
> > unsequence(similar with sequence folder)
> > ├── root.sg
> > │ ├── 2020-07-19
> > │ ├── 2020-07-19
> > │ ├── 2020-07-19
> >
> > We only need to store the whole partition folders in memory as a
> > List<String>, this memory consumption is negligible.
> >
> > For the hot partition, e.g., recent 10 days' partition, we could cache
> > their TsFileResources in memory to accelerate
> > queries.
> >
> > Then, how to do a query?
> >
> > Suppose we receive a query: select * from root.sg where time >=
> > 2020-07-20 and time <= 2020-07-21
> >
> > - We could locate two partitions under root.sg that may contains the
> > results: 2020-07-20, 2020-07-21
> > - Then we traverse the partition folder to get all TsFileResources in this
> > partition.
> > - Finally, we do queries.
> >
> > Is this feasible?
> >
> > Thanks,
> > --
> > Jialin Qiao
> > School of Software, Tsinghua University
> >
> > 乔嘉林
> > 清华大学 软件学院
> >
> > > -----原始邮件-----
> > > 发件人: "Xiangdong Huang" <saint...@gmail.com>
> > > 发送时间: 2020-07-20 20:03:34 (星期一)
> > > 收件人: dev <dev@iotdb.apache.org>
> > > 抄送:
> > > 主题: Re: Re: [Discuss] How to delivery the device concept to users
> > >
> > > Hi,
> > >
> > > > I wonder whether we could index the file by its name. (naming the
> > tsfile
> > > by date)
> > >
> > > I think it is a good idea, but maybe not very easy to implement. If we
> > can
> > > organize the data like this, then it is very very regular and very easy
> > to
> > > access or delete expired data...
> > >
> > > > we would need is a tree strucutre where each node has start time / end
> > > time for "everything" in the file.
> > >
> > > This is also a good idea.
> > >
> > > When we are discussing the granularity of "device", we are worrying about
> > > the size of the index, actually.
> > > So, we do not care whether there is a so called "sub device", we just
> > care
> > > how many entities will be indexed.
> > >
> > > Suppose an IoTDB instance can bear 1 million index entries <some_id ->
> > > (start time, end time)>, and given a tree schema, if there are about 1
> > > million nodes from level 0 to level 3, then we can index the nodes on
> > > level3 (so level 3 is so-called "device" in current version).
> > >
> > > Meantime, index the nodes from level0 to level2, as Julian proposed, is
> > > also beneficial.
> > >
> > > The nature of the above idea is letting IoTDB decides which are "devices"
> > > automatically.
> > >
> > > At the beginning of this discussion, I just want to let user claim which
> > > are "devices" (or, which prefixes of Paths have time indexes.. but this
> > > kind of description may be not user friendly..). As it is more easy....
> > but
> > > may carry risk if the user set too many devices.
> > >
> > > Best,
> > > -----------------------------------
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > > 黄向东
> > > 清华大学 软件学院
> > >
> > >
> > > runhus...@foxmail.com <runhus...@foxmail.com> 于2020年7月20日周一 下午7:47写道：
> > >
> > > > Hi，
> > > >
> > > > > I wonder whether we could index the file by its name. (naming the
> > tsfile
> > > > by date) E.g., we store each day's data in one file and name it as
> > > > sg-2020-07-20.TsFile. Then, we do not need to maintain the index in
> > memory,
> > > > we just need to check whether the file exist in the queried interval.
> > > >
> > > > So, how to deal with the out of order data? Could you give more
> > details.
> > > >
> > > >
> > > >
> > > > Thanks!
> > > >
> > > > runhus...@foxmail.com
> > > >
> > > >
> > > > From: Jialin Qiao
> > > > Date: 2020-07-20 18:21
> > > > To: dev
> > > > Subject: Re: [Discuss] How to delivery the device concept to users
> > > > Hi,
> > > >
> > > > > The question I would ask is why "devices" hurt us.
> > > >
> > > > I'd like to introduce this a bit. For each storage group, we flush the
> > > > memtable into TsFiles one by one. For each TsFile, we maintain a
> > temporal
> > > > index on device level in memory. Suppose there are 3 devices in one
> > TsFile,
> > > > the index is like this:
> > > >
> > > > start time array: long[3] = {1, 1, 2}
> > > > end time array: long[3] = {5, 6, 10}
> > > > devicesToIndexInArray: Map<String, Integer> = {"root.sg.d1" -> 0,
> > > > "root.sg.d2" -> 1, "root.sg.d3" -> 2}
> > > >
> > > > If we have millions of devices, for each TsFile, this index will reach
> > > > dozens of MB in memory. Although we could introduce the persistence of
> > the
> > > > index. It is still recommended to decrease the number of devices.
> > > >
> > > > I wonder whether we could index the file by its name. (naming the
> > tsfile
> > > > by date) E.g., we store each day's data in one file and name it as
> > > > sg-2020-07-20.TsFile. Then, we do not need to maintain the index in
> > memory,
> > > > we just need to check whether the file exist in the queried interval.
> > > >
> > > > Thanks,
> > > > --
> > > > Jialin Qiao
> > > > School of Software, Tsinghua University
> > > >
> > > > 乔嘉林
> > > > 清华大学 软件学院
> > > >
> > > > > -----原始邮件-----
> > > > > 发件人: "Julian Feinauer" <j.feina...@pragmaticminds.de>
> > > > > 发送时间: 2020-07-20 17:34:40 (星期一)
> > > > > 收件人: "dev@iotdb.apache.org" <dev@iotdb.apache.org>
> > > > > 抄送:
> > > > > 主题: Re: [Discuss] How to delivery the device concept to users
> > > > >
> > > > > Hey Jialin, xinagdong,
> > > > >
> > > > > very good question!
> > > > >
> > > > > And I tend to agree with Xiangdong.
> > > > > If the users do it that way it probably makes most sense for them.
> > > > > The question I would ask is why "devices" hurt us (I know a bit about
> > > > the implementation of course but probably we have to adopt our
> > datamodel
> > > > also a bit in the future).
> > > > >
> > > > > Generally speaking, form e it also makes sense tob e allowed to have
> > > > "subcategories" below my devices as my devices usually are "big".
> > > > > And technically speaking in the current version this is totally
> > possible
> > > > to have nested structures below devices or measurements (but these will
> > > > then again be devices).
> > > > >
> > > > > So my question is:
> > > > > - Do we really need the static construct of a "device" or can we
> > > > probably use a different datastructure where I "select" my device only
> > at
> > > > query time and we just select everything under that tree as ist
> > > > measurements or "sub-measurements" in cases of nesting.
> > > > >
> > > > > WDYT?
> > > > >
> > > > > Julian
> > > > >
> > > > > Am 20.07.20, 09:34 schrieb "Xiangdong Huang" <saint...@gmail.com>:
> > > > >
> > > > > Hi,
> > > > >
> > > > > This is a quite good topic!
> > > > >
> > > > > 1. maybe we should hear more users opinions.
> > > > >
> > > > > For me, I think emphasize the concept of "device" is good. We can
> > > > even
> > > > > expose the concept in our APIs.
> > > > >
> > > > > 2.
> > > > >
> > > > > > A more efficient way is
> > > > > > root.sg.device1.measurement1_int0
> > > > > > root.sg.device1.measurement1_int1
> > > > > > root.sg.device1.measurement1_int2
> > > > > > root.sg.device1.measurement2_long
> > > > >
> > > > > I think the more efficient way is:
> > > > >
> > > > > root.sg.device1.measurement1.0
> > > > > root.sg.device1.measurement1.1
> > > > > root.sg.device1.measurement1.2
> > > > > root.sg.device1.measurement2
> > > > >
> > > > > And, as you said "a device has a sensor that collects some data
> > in
> > > > array
> > > > > format (int[3]) and some in long type",
> > > > > will the user query just one element from the int[3]? If not, a
> > > > better
> > > > > schema is:
> > > > >
> > > > > root.sg.device1.measurement1 (the dataType is int[])
> > > > > root.sg.device1.measurement2 (the dataType is long)
> > > > >
> > > > > Best,
> > > > > -----------------------------------
> > > > > Xiangdong Huang
> > > > > School of Software, Tsinghua University
> > > > >
> > > > > 黄向东
> > > > > 清华大学 软件学院
> > > > >
> > > > >
> > > > > Jialin Qiao <qj...@mails.tsinghua.edu.cn> 于2020年7月20日周一
> > 下午3:28写道：
> > > > >
> > > > > > Hi
> > > > > >
> > > > > > Recently, I find that some users create timeseries do not
> > > > following the
> > > > > > real world semantic of device
> > > > > >
> > > > > >
> > > > > > E.g., a device has a sensor that collects some data in array
> > format
> > > > > > (int[3]) and some in long type.
> > > > > >
> > > > > >
> > > > > > Many users will create timeseries like this:
> > > > > >
> > > > > >
> > > > > > root.sg.device1.measurement1.int0
> > > > > > root.sg.device1.measurement1.int1
> > > > > > root.sg.device1.measurement1.int2
> > > > > > root.sg.device1.measurement2.long
> > > > > >
> > > > > >
> > > > > > As a consequence, there will be two devices instead of one
> > device.
> > > > This
> > > > > > will cause the real number of devices is much bigger than the
> > real
> > > > devices
> > > > > > they thought. The drawback is: more devices leads to more
> > memory
> > > > > > consumption.
> > > > > >
> > > > > >
> > > > > > A more efficient way is
> > > > > >
> > > > > >
> > > > > > root.sg.device1.measurement1_int0
> > > > > > root.sg.device1.measurement1_int1
> > > > > > root.sg.device1.measurement1_int2
> > > > > > root.sg.device1.measurement2_long
> > > > > >
> > > > > >
> > > > > > In this schema, there will be only one device and 4
> > measurements.
> > > > > >
> > > > > >
> > > > > > The problem is we extract the device id automatically. Users
> > > > usually do
> > > > > > not have a clear concept about "device". Should we emphasize
> > the
> > > > concept of
> > > > > > device by letting users create device manually?
> > > > > >
> > > > > >
> > > > > > What do you think?
> > > > > >
> > > > > > Thanks,
> > > > > > --
> > > > > > Jialin Qiao
> > > > > > School of Software, Tsinghua University
> > > > > >
> > > > > > 乔嘉林
> > > > > > 清华大学 软件学院
> > > > >
> > > >
> >
