Re: Re: [Discuss] How to delivery the device concept to users
Good idea! I think it's ok to seperate the tsfiles to different folders according to the time range if the time range is small enough. Do we need to deal with the query who's time range is very large(historied query)? Like one hundred days, we have to read all tsFiles of the days. Thanks! runhus...@foxmail.com From: Jialin Qiao Date: 2020-07-21 20:28 To: dev Subject: Re: [Discuss] How to delivery the device concept to users Hi, This is not the current implementation... We do not have a partition folder on disk now. By adding a partition folder, there is no need to store all TsFileResources in the memory, and the device index will not hurt us. Thanks, -- Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 > -原始邮件- > 发件人: "Xiangdong Huang" > 发送时间: 2020-07-21 18:46:31 (星期二) > 收件人: dev > 抄送: > 主题: Re: [Discuss] How to delivery the device concept to users > > Hi Jialin, > > Yes it is current logic. But I do not know the relation between what you > said and this discussion... > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Jialin Qiao 于2020年7月21日周二 下午4:47写道： > > > Hi, > > > > I would like to give a vision about managing the data files according to > > time partition. > > > > After we introduce the time partition (data is partitioned by time > > interval), we do split them in memory and different TsFiles. But we may > > lake a partition folder layer on top of the TsFiles. > > > > Maybe it should work as follows: > > > > E.g., we insert data into storage group root.sg from 2020-07-19 to > > 2020-07-21 and the partition interval is 1 day. > > First, we create three folders (2020-07-19, 2020-07-20, 2020-07-21) under > > root.sg that belongs to each partition. > > Then, we store TsFiles to its related partition folder. > > > > An example of TsFiles on disk is as follows: > > > > sequence > > ├── root.sg > > │ ├── 2020-07-19 > > │ │ └── timestamp1-version1-merge.tsfile > > │ │ └── timestamp1-version1-merge.tsfile.resource > > │ │ └── ... > > │ ├── 2020-07-20 > > │ ├── 2020-07-21 > > > > > > unsequence(similar with sequence folder) > > ├── root.sg > > │ ├── 2020-07-19 > > │ ├── 2020-07-19 > > │ ├── 2020-07-19 > > > > We only need to store the whole partition folders in memory as a > > List, this memory consumption is negligible. > > > > For the hot partition, e.g., recent 10 days' partition, we could cache > > their TsFileResources in memory to accelerate > > queries. > > > > Then, how to do a query? > > > > Suppose we receive a query: select * from root.sg where time >= > > 2020-07-20 and time <= 2020-07-21 > > > > - We could locate two partitions under root.sg that may contains the > > results: 2020-07-20, 2020-07-21 > > - Then we traverse the partition folder to get all TsFileResources in this > > partition. > > - Finally, we do queries. > > > > Is this feasible? > > > > Thanks, > > -- > > Jialin Qiao > > School of Software, Tsinghua University > > > > 乔嘉林 > > 清华大学 软件学院 > > > > > -原始邮件- > > > 发件人: "Xiangdong Huang" > > > 发送时间: 2020-07-20 20:03:34 (星期一) > > > 收件人: dev > > > 抄送: > > > 主题: Re: Re: [Discuss] How to delivery the device concept to users > > > > > > Hi, > > > > > > > I wonder whether we could index the file by its name. (naming the > > tsfile > > > by date) > > > > > > I think it is a good idea, but maybe not very easy to implement. If we > > can > > > organize the data like this, then it is very very regular and very easy > > to > > > access or delete expired data... > > > > > > > we would need is a tree strucutre where each node has start time / end > > > time for "everything" in the file. > > > > > > This is also a good idea. > > > > > > When we are discussing the granularity of "device", we are worrying about > > > the size of the index, actually. > > > So, we do not care whether there is a so called "sub device", we just > > care > > > how many entities will be indexed. > > > > > > Suppose an IoTDB instance can bear 1 million index entries > > > (start time, end time)>, and given a tree schema, if there are about 1 > > > million nodes from level 0 to level 3, then we can index the nodes on > > > level3 (so level 3 is so-called "device" in current version). > > > > > > Meantime, index the nodes from level0 to level2, as Julian proposed, is > > > also beneficial. > > > > > > The nature of the above idea is letting IoTDB decides which are "devices" > > > automatically. > > > > > > At the beginning of this discussion, I just want to let user claim which > > > are "devices" (or, which prefixes of Paths have time indexes.. but this > > > kind of description may be not user friendly..). As it is more easy > > but > > > may carry risk if the user set too many devices. > > > > > > Best, > > > --- > > > Xiangdong Huang > > > School of Software, Tsinghua University > > > > > > 黄向东 > > > 清华大学 软件学院
Re: [Discuss] How to delivery the device concept to users
Ah, Don't we have? we have the time partition folder from v0.10 on.. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2020年7月21日周二 下午8:37写道： > Hi, > > This is not the current implementation... We do not have a partition > folder on disk now. > By adding a partition folder, there is no need to store all > TsFileResources in the memory, and the device index will not hurt us. > > Thanks, > -- > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院 > > > -原始邮件- > > 发件人: "Xiangdong Huang" > > 发送时间: 2020-07-21 18:46:31 (星期二) > > 收件人: dev > > 抄送: > > 主题: Re: [Discuss] How to delivery the device concept to users > > > > Hi Jialin, > > > > Yes it is current logic. But I do not know the relation between what you > > said and this discussion... > > > > Best, > > --- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > 黄向东 > > 清华大学 软件学院 > > > > > > Jialin Qiao 于2020年7月21日周二 下午4:47写道： > > > > > Hi, > > > > > > I would like to give a vision about managing the data files according > to > > > time partition. > > > > > > After we introduce the time partition (data is partitioned by time > > > interval), we do split them in memory and different TsFiles. But we may > > > lake a partition folder layer on top of the TsFiles. > > > > > > Maybe it should work as follows: > > > > > > E.g., we insert data into storage group root.sg from 2020-07-19 to > > > 2020-07-21 and the partition interval is 1 day. > > > First, we create three folders (2020-07-19, 2020-07-20, 2020-07-21) > under > > > root.sg that belongs to each partition. > > > Then, we store TsFiles to its related partition folder. > > > > > > An example of TsFiles on disk is as follows: > > > > > > sequence > > > ├── root.sg > > > │ ├── 2020-07-19 > > > │ │ └── timestamp1-version1-merge.tsfile > > > │ │ └── timestamp1-version1-merge.tsfile.resource > > > │ │ └── ... > > > │ ├── 2020-07-20 > > > │ ├── 2020-07-21 > > > > > > > > > unsequence(similar with sequence folder) > > > ├── root.sg > > > │ ├── 2020-07-19 > > > │ ├── 2020-07-19 > > > │ ├── 2020-07-19 > > > > > > We only need to store the whole partition folders in memory as a > > > List, this memory consumption is negligible. > > > > > > For the hot partition, e.g., recent 10 days' partition, we could cache > > > their TsFileResources in memory to accelerate > > > queries. > > > > > > Then, how to do a query? > > > > > > Suppose we receive a query: select * from root.sg where time >= > > > 2020-07-20 and time <= 2020-07-21 > > > > > > - We could locate two partitions under root.sg that may contains the > > > results: 2020-07-20, 2020-07-21 > > > - Then we traverse the partition folder to get all TsFileResources in > this > > > partition. > > > - Finally, we do queries. > > > > > > Is this feasible? > > > > > > Thanks, > > > -- > > > Jialin Qiao > > > School of Software, Tsinghua University > > > > > > 乔嘉林 > > > 清华大学 软件学院 > > > > > > > -原始邮件- > > > > 发件人: "Xiangdong Huang" > > > > 发送时间: 2020-07-20 20:03:34 (星期一) > > > > 收件人: dev > > > > 抄送: > > > > 主题: Re: Re: [Discuss] How to delivery the device concept to users > > > > > > > > Hi, > > > > > > > > > I wonder whether we could index the file by its name. (naming the > > > tsfile > > > > by date) > > > > > > > > I think it is a good idea, but maybe not very easy to implement. If > we > > > can > > > > organize the data like this, then it is very very regular and very > easy > > > to > > > > access or delete expired data... > > > > > > > > > we would need is a tree strucutre where each node has start time / > end > > > > time for "everything" in the file. > > > > > > > > This is also a good idea. > > > > > > > > When we are discussing the granularity of "device", we are worrying > about > > > > the size of the index, actually. > > > > So, we do not care whether there is a so called "sub device", we just > > > care > > > > how many entities will be indexed. > > > > > > > > Suppose an IoTDB instance can bear 1 million index entries -> > > > > (start time, end time)>, and given a tree schema, if there are > about 1 > > > > million nodes from level 0 to level 3, then we can index the nodes on > > > > level3 (so level 3 is so-called "device" in current version). > > > > > > > > Meantime, index the nodes from level0 to level2, as Julian proposed, > is > > > > also beneficial. > > > > > > > > The nature of the above idea is letting IoTDB decides which are > "devices" > > > > automatically. > > > > > > > > At the beginning of this discussion, I just want to let user claim > which > > > > are "devices" (or, which prefixes of Paths have time indexes.. but > this > > > > kind of description may be not user friendly..). As it is more > easy > > > but > > > > may carry risk if the user set too many devices. > > > > > > > > Best, > > > > --- > > > >
Re: [Discuss] How to delivery the device concept to users
Hi, This is not the current implementation... We do not have a partition folder on disk now. By adding a partition folder, there is no need to store all TsFileResources in the memory, and the device index will not hurt us. Thanks, -- Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 > -原始邮件- > 发件人: "Xiangdong Huang" > 发送时间: 2020-07-21 18:46:31 (星期二) > 收件人: dev > 抄送: > 主题: Re: [Discuss] How to delivery the device concept to users > > Hi Jialin, > > Yes it is current logic. But I do not know the relation between what you > said and this discussion... > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Jialin Qiao 于2020年7月21日周二 下午4:47写道： > > > Hi, > > > > I would like to give a vision about managing the data files according to > > time partition. > > > > After we introduce the time partition (data is partitioned by time > > interval), we do split them in memory and different TsFiles. But we may > > lake a partition folder layer on top of the TsFiles. > > > > Maybe it should work as follows: > > > > E.g., we insert data into storage group root.sg from 2020-07-19 to > > 2020-07-21 and the partition interval is 1 day. > > First, we create three folders (2020-07-19, 2020-07-20, 2020-07-21) under > > root.sg that belongs to each partition. > > Then, we store TsFiles to its related partition folder. > > > > An example of TsFiles on disk is as follows: > > > > sequence > > ├── root.sg > > │ ├── 2020-07-19 > > │ │ └── timestamp1-version1-merge.tsfile > > │ │ └── timestamp1-version1-merge.tsfile.resource > > │ │ └── ... > > │ ├── 2020-07-20 > > │ ├── 2020-07-21 > > > > > > unsequence(similar with sequence folder) > > ├── root.sg > > │ ├── 2020-07-19 > > │ ├── 2020-07-19 > > │ ├── 2020-07-19 > > > > We only need to store the whole partition folders in memory as a > > List, this memory consumption is negligible. > > > > For the hot partition, e.g., recent 10 days' partition, we could cache > > their TsFileResources in memory to accelerate > > queries. > > > > Then, how to do a query? > > > > Suppose we receive a query: select * from root.sg where time >= > > 2020-07-20 and time <= 2020-07-21 > > > > - We could locate two partitions under root.sg that may contains the > > results: 2020-07-20, 2020-07-21 > > - Then we traverse the partition folder to get all TsFileResources in this > > partition. > > - Finally, we do queries. > > > > Is this feasible? > > > > Thanks, > > -- > > Jialin Qiao > > School of Software, Tsinghua University > > > > 乔嘉林 > > 清华大学 软件学院 > > > > > -原始邮件- > > > 发件人: "Xiangdong Huang" > > > 发送时间: 2020-07-20 20:03:34 (星期一) > > > 收件人: dev > > > 抄送: > > > 主题: Re: Re: [Discuss] How to delivery the device concept to users > > > > > > Hi, > > > > > > > I wonder whether we could index the file by its name. (naming the > > tsfile > > > by date) > > > > > > I think it is a good idea, but maybe not very easy to implement. If we > > can > > > organize the data like this, then it is very very regular and very easy > > to > > > access or delete expired data... > > > > > > > we would need is a tree strucutre where each node has start time / end > > > time for "everything" in the file. > > > > > > This is also a good idea. > > > > > > When we are discussing the granularity of "device", we are worrying about > > > the size of the index, actually. > > > So, we do not care whether there is a so called "sub device", we just > > care > > > how many entities will be indexed. > > > > > > Suppose an IoTDB instance can bear 1 million index entries > > > (start time, end time)>, and given a tree schema, if there are about 1 > > > million nodes from level 0 to level 3, then we can index the nodes on > > > level3 (so level 3 is so-called "device" in current version). > > > > > > Meantime, index the nodes from level0 to level2, as Julian proposed, is > > > also beneficial. > > > > > > The nature of the above idea is letting IoTDB decides which are "devices" > > > automatically. > > > > > > At the beginning of this discussion, I just want to let user claim which > > > are "devices" (or, which prefixes of Paths have time indexes.. but this > > > kind of description may be not user friendly..). As it is more easy > > but > > > may carry risk if the user set too many devices. > > > > > > Best, > > > --- > > > Xiangdong Huang > > > School of Software, Tsinghua University > > > > > > 黄向东 > > > 清华大学 软件学院 > > > > > > > > > runhus...@foxmail.com 于2020年7月20日周一 下午7:47写道： > > > > > > > Hi， > > > > > > > > > I wonder whether we could index the file by its name. (naming the > > tsfile > > > > by date) E.g., we store each day's data in one file and name it as > > > > sg-2020-07-20.TsFile. Then, we do not need to maintain the index in > > memory, > > > > we just need to check whether the file exist in the queried interval. > > > > > > > > So, how to
Re: [Discuss] How to delivery the device concept to users
Hi Jialin,
Yes it is current logic. But I do not know the relation between what you
said and this discussion...
Best,
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Jialin Qiao 于2020年7月21日周二 下午4:47写道：
> Hi,
>
> I would like to give a vision about managing the data files according to
> time partition.
>
> After we introduce the time partition (data is partitioned by time
> interval), we do split them in memory and different TsFiles. But we may
> lake a partition folder layer on top of the TsFiles.
>
> Maybe it should work as follows:
>
> E.g., we insert data into storage group root.sg from 2020-07-19 to
> 2020-07-21 and the partition interval is 1 day.
> First, we create three folders (2020-07-19, 2020-07-20, 2020-07-21) under
> root.sg that belongs to each partition.
> Then, we store TsFiles to its related partition folder.
>
> An example of TsFiles on disk is as follows:
>
> sequence
> ├── root.sg
> │ ├── 2020-07-19
> │ │ └── timestamp1-version1-merge.tsfile
> │ │ └── timestamp1-version1-merge.tsfile.resource
> │ │ └── ...
> │ ├── 2020-07-20
> │ ├── 2020-07-21
>
>
> unsequence(similar with sequence folder)
> ├── root.sg
> │ ├── 2020-07-19
> │ ├── 2020-07-19
> │ ├── 2020-07-19
>
> We only need to store the whole partition folders in memory as a
> List, this memory consumption is negligible.
>
> For the hot partition, e.g., recent 10 days' partition, we could cache
> their TsFileResources in memory to accelerate
> queries.
>
> Then, how to do a query?
>
> Suppose we receive a query: select * from root.sg where time >=
> 2020-07-20 and time <= 2020-07-21
>
> - We could locate two partitions under root.sg that may contains the
> results: 2020-07-20, 2020-07-21
> - Then we traverse the partition folder to get all TsFileResources in this
> partition.
> - Finally, we do queries.
>
> Is this feasible?
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -原始邮件-
> > 发件人: "Xiangdong Huang"
> > 发送时间: 2020-07-20 20:03:34 (星期一)
> > 收件人: dev
> > 抄送:
> > 主题: Re: Re: [Discuss] How to delivery the device concept to users
> >
> > Hi,
> >
> > > I wonder whether we could index the file by its name. (naming the
> tsfile
> > by date)
> >
> > I think it is a good idea, but maybe not very easy to implement. If we
> can
> > organize the data like this, then it is very very regular and very easy
> to
> > access or delete expired data...
> >
> > > we would need is a tree strucutre where each node has start time / end
> > time for "everything" in the file.
> >
> > This is also a good idea.
> >
> > When we are discussing the granularity of "device", we are worrying about
> > the size of the index, actually.
> > So, we do not care whether there is a so called "sub device", we just
> care
> > how many entities will be indexed.
> >
> > Suppose an IoTDB instance can bear 1 million index entries
> > (start time, end time)>, and given a tree schema, if there are about 1
> > million nodes from level 0 to level 3, then we can index the nodes on
> > level3 (so level 3 is so-called "device" in current version).
> >
> > Meantime, index the nodes from level0 to level2, as Julian proposed, is
> > also beneficial.
> >
> > The nature of the above idea is letting IoTDB decides which are "devices"
> > automatically.
> >
> > At the beginning of this discussion, I just want to let user claim which
> > are "devices" (or, which prefixes of Paths have time indexes.. but this
> > kind of description may be not user friendly..). As it is more easy
> but
> > may carry risk if the user set too many devices.
> >
> > Best,
> > ---
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> >
> > runhus...@foxmail.com 于2020年7月20日周一 下午7:47写道：
> >
> > > Hi，
> > >
> > > > I wonder whether we could index the file by its name. (naming the
> tsfile
> > > by date) E.g., we store each day's data in one file and name it as
> > > sg-2020-07-20.TsFile. Then, we do not need to maintain the index in
> memory,
> > > we just need to check whether the file exist in the queried interval.
> > >
> > > So, how to deal with the out of order data? Could you give more
> details.
> > >
> > >
> > >
> > > Thanks!
> > >
> > > runhus...@foxmail.com
> > >
> > >
> > > From: Jialin Qiao
> > > Date: 2020-07-20 18:21
> > > To: dev
> > > Subject: Re: [Discuss] How to delivery the device concept to users
> > > Hi,
> > >
> > > > The question I would ask is why "devices" hurt us.
> > >
> > > I'd like to introduce this a bit. For each storage group, we flush the
> > > memtable into TsFiles one by one. For each TsFile, we maintain a
> temporal
> > > index on device level in memory. Suppose there are 3 devices in one
> TsFile,
> > > the index is like this:
> > >
> > > start time array: long[3] = {1, 1, 2}
> > > end time array: long[3] = {5, 6, 10}
> > > devicesToIndexInArray: Map =
Re: [Discuss] How to delivery the device concept to users
Hi,
I would like to give a vision about managing the data files according to time
partition.
After we introduce the time partition (data is partitioned by time interval),
we do split them in memory and different TsFiles. But we may lake a partition
folder layer on top of the TsFiles.
Maybe it should work as follows:
E.g., we insert data into storage group root.sg from 2020-07-19 to 2020-07-21
and the partition interval is 1 day.
First, we create three folders (2020-07-19, 2020-07-20, 2020-07-21) under
root.sg that belongs to each partition.
Then, we store TsFiles to its related partition folder.
An example of TsFiles on disk is as follows:
sequence
├── root.sg
│ ├── 2020-07-19
│ │ └── timestamp1-version1-merge.tsfile
│ │ └── timestamp1-version1-merge.tsfile.resource
│ │ └── ...
│ ├── 2020-07-20
│ ├── 2020-07-21
unsequence(similar with sequence folder)
├── root.sg
│ ├── 2020-07-19
│ ├── 2020-07-19
│ ├── 2020-07-19
We only need to store the whole partition folders in memory as a List,
this memory consumption is negligible.
For the hot partition, e.g., recent 10 days' partition, we could cache their
TsFileResources in memory to accelerate
queries.
Then, how to do a query?
Suppose we receive a query: select * from root.sg where time >= 2020-07-20 and
time <= 2020-07-21
- We could locate two partitions under root.sg that may contains the results:
2020-07-20, 2020-07-21
- Then we traverse the partition folder to get all TsFileResources in this
partition.
- Finally, we do queries.
Is this feasible?
Thanks,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -原始邮件-
> 发件人: "Xiangdong Huang"
> 发送时间: 2020-07-20 20:03:34 (星期一)
> 收件人: dev
> 抄送:
> 主题: Re: Re: [Discuss] How to delivery the device concept to users
>
> Hi,
>
> > I wonder whether we could index the file by its name. (naming the tsfile
> by date)
>
> I think it is a good idea, but maybe not very easy to implement. If we can
> organize the data like this, then it is very very regular and very easy to
> access or delete expired data...
>
> > we would need is a tree strucutre where each node has start time / end
> time for "everything" in the file.
>
> This is also a good idea.
>
> When we are discussing the granularity of "device", we are worrying about
> the size of the index, actually.
> So, we do not care whether there is a so called "sub device", we just care
> how many entities will be indexed.
>
> Suppose an IoTDB instance can bear 1 million index entries
> (start time, end time)>, and given a tree schema, if there are about 1
> million nodes from level 0 to level 3, then we can index the nodes on
> level3 (so level 3 is so-called "device" in current version).
>
> Meantime, index the nodes from level0 to level2, as Julian proposed, is
> also beneficial.
>
> The nature of the above idea is letting IoTDB decides which are "devices"
> automatically.
>
> At the beginning of this discussion, I just want to let user claim which
> are "devices" (or, which prefixes of Paths have time indexes.. but this
> kind of description may be not user friendly..). As it is more easy but
> may carry risk if the user set too many devices.
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> runhus...@foxmail.com 于2020年7月20日周一 下午7:47写道：
>
> > Hi，
> >
> > > I wonder whether we could index the file by its name. (naming the tsfile
> > by date) E.g., we store each day's data in one file and name it as
> > sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> > we just need to check whether the file exist in the queried interval.
> >
> > So, how to deal with the out of order data? Could you give more details.
> >
> >
> >
> > Thanks!
> >
> > runhus...@foxmail.com
> >
> >
> > From: Jialin Qiao
> > Date: 2020-07-20 18:21
> > To: dev
> > Subject: Re: [Discuss] How to delivery the device concept to users
> > Hi,
> >
> > > The question I would ask is why "devices" hurt us.
> >
> > I'd like to introduce this a bit. For each storage group, we flush the
> > memtable into TsFiles one by one. For each TsFile, we maintain a temporal
> > index on device level in memory. Suppose there are 3 devices in one TsFile,
> > the index is like this:
> >
> > start time array: long[3] = {1, 1, 2}
> > end time array: long[3] = {5, 6, 10}
> > devicesToIndexInArray: Map = {"root.sg.d1" -> 0,
> > "root.sg.d2" -> 1, "root.sg.d3" -> 2}
> >
> > If we have millions of devices, for each TsFile, this index will reach
> > dozens of MB in memory. Although we could introduce the persistence of the
> > index. It is still recommended to decrease the number of devices.
> >
> > I wonder whether we could index the file by its name. (naming the tsfile
> > by date) E.g., we store each day's data in one file and name it as
> > sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> > we just need
Re: Re: [Discuss] How to delivery the device concept to users
Hi,
> I wonder whether we could index the file by its name. (naming the tsfile
by date)
I think it is a good idea, but maybe not very easy to implement. If we can
organize the data like this, then it is very very regular and very easy to
access or delete expired data...
> we would need is a tree strucutre where each node has start time / end
time for "everything" in the file.
This is also a good idea.
When we are discussing the granularity of "device", we are worrying about
the size of the index, actually.
So, we do not care whether there is a so called "sub device", we just care
how many entities will be indexed.
Suppose an IoTDB instance can bear 1 million index entries
(start time, end time)>, and given a tree schema, if there are about 1
million nodes from level 0 to level 3, then we can index the nodes on
level3 (so level 3 is so-called "device" in current version).
Meantime, index the nodes from level0 to level2, as Julian proposed, is
also beneficial.
The nature of the above idea is letting IoTDB decides which are "devices"
automatically.
At the beginning of this discussion, I just want to let user claim which
are "devices" (or, which prefixes of Paths have time indexes.. but this
kind of description may be not user friendly..). As it is more easy but
may carry risk if the user set too many devices.
Best,
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
runhus...@foxmail.com 于2020年7月20日周一 下午7:47写道：
> Hi，
>
> > I wonder whether we could index the file by its name. (naming the tsfile
> by date) E.g., we store each day's data in one file and name it as
> sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> we just need to check whether the file exist in the queried interval.
>
> So, how to deal with the out of order data? Could you give more details.
>
>
>
> Thanks!
>
> runhus...@foxmail.com
>
>
> From: Jialin Qiao
> Date: 2020-07-20 18:21
> To: dev
> Subject: Re: [Discuss] How to delivery the device concept to users
> Hi,
>
> > The question I would ask is why "devices" hurt us.
>
> I'd like to introduce this a bit. For each storage group, we flush the
> memtable into TsFiles one by one. For each TsFile, we maintain a temporal
> index on device level in memory. Suppose there are 3 devices in one TsFile,
> the index is like this:
>
> start time array: long[3] = {1, 1, 2}
> end time array: long[3] = {5, 6, 10}
> devicesToIndexInArray: Map = {"root.sg.d1" -> 0,
> "root.sg.d2" -> 1, "root.sg.d3" -> 2}
>
> If we have millions of devices, for each TsFile, this index will reach
> dozens of MB in memory. Although we could introduce the persistence of the
> index. It is still recommended to decrease the number of devices.
>
> I wonder whether we could index the file by its name. (naming the tsfile
> by date) E.g., we store each day's data in one file and name it as
> sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> we just need to check whether the file exist in the queried interval.
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -原始邮件-
> > 发件人: "Julian Feinauer"
> > 发送时间: 2020-07-20 17:34:40 (星期一)
> > 收件人: "dev@iotdb.apache.org"
> > 抄送:
> > 主题: Re: [Discuss] How to delivery the device concept to users
> >
> > Hey Jialin, xinagdong,
> >
> > very good question!
> >
> > And I tend to agree with Xiangdong.
> > If the users do it that way it probably makes most sense for them.
> > The question I would ask is why "devices" hurt us (I know a bit about
> the implementation of course but probably we have to adopt our datamodel
> also a bit in the future).
> >
> > Generally speaking, form e it also makes sense tob e allowed to have
> "subcategories" below my devices as my devices usually are "big".
> > And technically speaking in the current version this is totally possible
> to have nested structures below devices or measurements (but these will
> then again be devices).
> >
> > So my question is:
> > - Do we really need the static construct of a "device" or can we
> probably use a different datastructure where I "select" my device only at
> query time and we just select everything under that tree as ist
> measurements or "sub-measurements" in cases of nesting.
> >
> > WDYT?
> >
> > Julian
> >
> > Am 20.07.20, 09:34 schrieb "Xiangdong Huang" :
> >
> > Hi,
> >
> > This is a quite good topic!
> >
> > 1. maybe we should hear more users opinions.
> >
> > For me, I think emphasize the concept of "device" is good. We can
> even
> > expose the concept in our APIs.
> >
> > 2.
> >
> > > A more efficient way is
> > > root.sg.device1.measurement1_int0
> > > root.sg.device1.measurement1_int1
> > > root.sg.device1.measurement1_int2
> > > root.sg.device1.measurement2_long
> >
> > I think the more efficient way is:
> >
> > root.sg.device1.measurement1.0
> >
Re: Re: [Discuss] How to delivery the device concept to users
Hi，
> I wonder whether we could index the file by its name. (naming the tsfile by
> date) E.g., we store each day's data in one file and name it as
> sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> we just need to check whether the file exist in the queried interval.
So, how to deal with the out of order data? Could you give more details.
Thanks!
runhus...@foxmail.com
From: Jialin Qiao
Date: 2020-07-20 18:21
To: dev
Subject: Re: [Discuss] How to delivery the device concept to users
Hi,
> The question I would ask is why "devices" hurt us.
I'd like to introduce this a bit. For each storage group, we flush the memtable
into TsFiles one by one. For each TsFile, we maintain a temporal index on
device level in memory. Suppose there are 3 devices in one TsFile, the index is
like this:
start time array: long[3] = {1, 1, 2}
end time array: long[3] = {5, 6, 10}
devicesToIndexInArray: Map = {"root.sg.d1" -> 0, "root.sg.d2"
-> 1, "root.sg.d3" -> 2}
If we have millions of devices, for each TsFile, this index will reach dozens
of MB in memory. Although we could introduce the persistence of the index. It
is still recommended to decrease the number of devices.
I wonder whether we could index the file by its name. (naming the tsfile by
date) E.g., we store each day's data in one file and name it as
sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory, we
just need to check whether the file exist in the queried interval.
Thanks,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -原始邮件-
> 发件人: "Julian Feinauer"
> 发送时间: 2020-07-20 17:34:40 (星期一)
> 收件人: "dev@iotdb.apache.org"
> 抄送:
> 主题: Re: [Discuss] How to delivery the device concept to users
>
> Hey Jialin, xinagdong,
>
> very good question!
>
> And I tend to agree with Xiangdong.
> If the users do it that way it probably makes most sense for them.
> The question I would ask is why "devices" hurt us (I know a bit about the
> implementation of course but probably we have to adopt our datamodel also a
> bit in the future).
>
> Generally speaking, form e it also makes sense tob e allowed to have
> "subcategories" below my devices as my devices usually are "big".
> And technically speaking in the current version this is totally possible to
> have nested structures below devices or measurements (but these will then
> again be devices).
>
> So my question is:
> - Do we really need the static construct of a "device" or can we probably use
> a different datastructure where I "select" my device only at query time and
> we just select everything under that tree as ist measurements or
> "sub-measurements" in cases of nesting.
>
> WDYT?
>
> Julian
>
> Am 20.07.20, 09:34 schrieb "Xiangdong Huang" :
>
> Hi,
>
> This is a quite good topic!
>
> 1. maybe we should hear more users opinions.
>
> For me, I think emphasize the concept of "device" is good. We can even
> expose the concept in our APIs.
>
> 2.
>
> > A more efficient way is
> > root.sg.device1.measurement1_int0
> > root.sg.device1.measurement1_int1
> > root.sg.device1.measurement1_int2
> > root.sg.device1.measurement2_long
>
> I think the more efficient way is:
>
> root.sg.device1.measurement1.0
> root.sg.device1.measurement1.1
> root.sg.device1.measurement1.2
> root.sg.device1.measurement2
>
> And, as you said "a device has a sensor that collects some data in array
> format (int[3]) and some in long type",
> will the user query just one element from the int[3]? If not, a better
> schema is:
>
> root.sg.device1.measurement1 (the dataType is int[])
> root.sg.device1.measurement2 (the dataType is long)
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Jialin Qiao 于2020年7月20日周一 下午3:28写道：
>
> > Hi
> >
> > Recently, I find that some users create timeseries do not following the
> > real world semantic of device
> >
> >
> > E.g., a device has a sensor that collects some data in array format
> > (int[3]) and some in long type.
> >
> >
> > Many users will create timeseries like this:
> >
> >
> > root.sg.device1.measurement1.int0
> > root.sg.device1.measurement1.int1
> > root.sg.device1.measurement1.int2
> > root.sg.device1.measurement2.long
> >
> >
> > As a consequence, there will be two devices instead of one device. This
> > will cause the real number of devices is much bigger than the real
> devices
> > they thought. The drawback is: more devices leads to more memory
> > consumption.
> >
> >
> > A more efficient way is
> >
> >
> > root.sg.device1.measurement1_int0
> > root.sg.device1.measurement1_int1
> > root.sg.device1.measurement1_int2
> >
Re: [Discuss] How to delivery the device concept to users
Thanks fort he clear explanation, yes I remember that there were also reported
performance issues with that.
But to generalize the concept of a device all we would need is a tree strucutre
where each node has start time / end time for "everything" in the file.
Like in your example:
Root (1, 10)
sg (1, 10)
d1 (1, 5)
d2 (1, 6)
d3 (2, 10)
This would allow us to fetch the necessary information on each level.
What about using some kind of simple KV Store which is ofc disk based but does
its own in memory caching and optimization such that frequent accesses ("hot
devices" or more generally "paths") are fast.
Is this a valid idea?
Julian
Am 20.07.20, 11:53 schrieb "Jialin Qiao" :
Hi,
> The question I would ask is why "devices" hurt us.
I'd like to introduce this a bit. For each storage group, we flush the
memtable into TsFiles one by one. For each TsFile, we maintain a temporal index
on device level in memory. Suppose there are 3 devices in one TsFile, the index
is like this:
start time array: long[3] = {1, 1, 2}
end time array: long[3] = {5, 6, 10}
devicesToIndexInArray: Map = {"root.sg.d1" -> 0,
"root.sg.d2" -> 1, "root.sg.d3" -> 2}
If we have millions of devices, for each TsFile, this index will reach
dozens of MB in memory. Although we could introduce the persistence of the
index. It is still recommended to decrease the number of devices.
I wonder whether we could index the file by its name. (naming the tsfile by
date) E.g., we store each day's data in one file and name it as
sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory, we
just need to check whether the file exist in the queried interval.
Thanks,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -原始邮件-
> 发件人: "Julian Feinauer"
> 发送时间: 2020-07-20 17:34:40 (星期一)
> 收件人: "dev@iotdb.apache.org"
> 抄送:
> 主题: Re: [Discuss] How to delivery the device concept to users
>
> Hey Jialin, xinagdong,
>
> very good question!
>
> And I tend to agree with Xiangdong.
> If the users do it that way it probably makes most sense for them.
> The question I would ask is why "devices" hurt us (I know a bit about the
implementation of course but probably we have to adopt our datamodel also a bit
in the future).
>
> Generally speaking, form e it also makes sense tob e allowed to have
"subcategories" below my devices as my devices usually are "big".
> And technically speaking in the current version this is totally possible
to have nested structures below devices or measurements (but these will then
again be devices).
>
> So my question is:
> - Do we really need the static construct of a "device" or can we probably
use a different datastructure where I "select" my device only at query time and
we just select everything under that tree as ist measurements or
"sub-measurements" in cases of nesting.
>
> WDYT?
>
> Julian
>
> Am 20.07.20, 09:34 schrieb "Xiangdong Huang" :
>
> Hi,
>
> This is a quite good topic!
>
> 1. maybe we should hear more users opinions.
>
> For me, I think emphasize the concept of "device" is good. We can even
> expose the concept in our APIs.
>
> 2.
>
> > A more efficient way is
> > root.sg.device1.measurement1_int0
> > root.sg.device1.measurement1_int1
> > root.sg.device1.measurement1_int2
> > root.sg.device1.measurement2_long
>
> I think the more efficient way is:
>
> root.sg.device1.measurement1.0
> root.sg.device1.measurement1.1
> root.sg.device1.measurement1.2
> root.sg.device1.measurement2
>
> And, as you said "a device has a sensor that collects some data in
array
> format (int[3]) and some in long type",
> will the user query just one element from the int[3]? If not, a better
> schema is:
>
> root.sg.device1.measurement1 (the dataType is int[])
> root.sg.device1.measurement2 (the dataType is long)
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Jialin Qiao 于2020年7月20日周一 下午3:28写道：
>
> > Hi
> >
> > Recently, I find that some users create timeseries do not following
the
> > real world semantic of device
> >
> >
> > E.g., a device has a sensor that collects some data in array format
> > (int[3]) and some in long type.
> >
> >
> > Many users will create timeseries like this:
> >
> >
> > root.sg.device1.measurement1.int0
> >
Re: [Discuss] How to delivery the device concept to users
Hi,
> The question I would ask is why "devices" hurt us.
I'd like to introduce this a bit. For each storage group, we flush the memtable
into TsFiles one by one. For each TsFile, we maintain a temporal index on
device level in memory. Suppose there are 3 devices in one TsFile, the index is
like this:
start time array: long[3] = {1, 1, 2}
end time array: long[3] = {5, 6, 10}
devicesToIndexInArray: Map = {"root.sg.d1" -> 0, "root.sg.d2"
-> 1, "root.sg.d3" -> 2}
If we have millions of devices, for each TsFile, this index will reach dozens
of MB in memory. Although we could introduce the persistence of the index. It
is still recommended to decrease the number of devices.
I wonder whether we could index the file by its name. (naming the tsfile by
date) E.g., we store each day's data in one file and name it as
sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory, we
just need to check whether the file exist in the queried interval.
Thanks,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -原始邮件-
> 发件人: "Julian Feinauer"
> 发送时间: 2020-07-20 17:34:40 (星期一)
> 收件人: "dev@iotdb.apache.org"
> 抄送:
> 主题: Re: [Discuss] How to delivery the device concept to users
>
> Hey Jialin, xinagdong,
>
> very good question!
>
> And I tend to agree with Xiangdong.
> If the users do it that way it probably makes most sense for them.
> The question I would ask is why "devices" hurt us (I know a bit about the
> implementation of course but probably we have to adopt our datamodel also a
> bit in the future).
>
> Generally speaking, form e it also makes sense tob e allowed to have
> "subcategories" below my devices as my devices usually are "big".
> And technically speaking in the current version this is totally possible to
> have nested structures below devices or measurements (but these will then
> again be devices).
>
> So my question is:
> - Do we really need the static construct of a "device" or can we probably use
> a different datastructure where I "select" my device only at query time and
> we just select everything under that tree as ist measurements or
> "sub-measurements" in cases of nesting.
>
> WDYT?
>
> Julian
>
> Am 20.07.20, 09:34 schrieb "Xiangdong Huang" :
>
> Hi,
>
> This is a quite good topic!
>
> 1. maybe we should hear more users opinions.
>
> For me, I think emphasize the concept of "device" is good. We can even
> expose the concept in our APIs.
>
> 2.
>
> > A more efficient way is
> > root.sg.device1.measurement1_int0
> > root.sg.device1.measurement1_int1
> > root.sg.device1.measurement1_int2
> > root.sg.device1.measurement2_long
>
> I think the more efficient way is:
>
> root.sg.device1.measurement1.0
> root.sg.device1.measurement1.1
> root.sg.device1.measurement1.2
> root.sg.device1.measurement2
>
> And, as you said "a device has a sensor that collects some data in array
> format (int[3]) and some in long type",
> will the user query just one element from the int[3]? If not, a better
> schema is:
>
> root.sg.device1.measurement1 (the dataType is int[])
> root.sg.device1.measurement2 (the dataType is long)
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Jialin Qiao 于2020年7月20日周一 下午3:28写道：
>
> > Hi
> >
> > Recently, I find that some users create timeseries do not following the
> > real world semantic of device
> >
> >
> > E.g., a device has a sensor that collects some data in array format
> > (int[3]) and some in long type.
> >
> >
> > Many users will create timeseries like this:
> >
> >
> > root.sg.device1.measurement1.int0
> > root.sg.device1.measurement1.int1
> > root.sg.device1.measurement1.int2
> > root.sg.device1.measurement2.long
> >
> >
> > As a consequence, there will be two devices instead of one device. This
> > will cause the real number of devices is much bigger than the real
> devices
> > they thought. The drawback is: more devices leads to more memory
> > consumption.
> >
> >
> > A more efficient way is
> >
> >
> > root.sg.device1.measurement1_int0
> > root.sg.device1.measurement1_int1
> > root.sg.device1.measurement1_int2
> > root.sg.device1.measurement2_long
> >
> >
> > In this schema, there will be only one device and 4 measurements.
> >
> >
> > The problem is we extract the device id automatically. Users usually do
> > not have a clear concept about "device". Should we emphasize the
> concept of
> > device by letting users create device manually?
> >
> >
> > What do you think?
> >
> > Thanks,
> > --
> > Jialin Qiao
> > School of Software, Tsinghua University
> >
> > 乔嘉林
> > 清华大学 软件学院
>
Re: [Discuss] How to delivery the device concept to users
Hey Jialin, xinagdong, very good question! And I tend to agree with Xiangdong. If the users do it that way it probably makes most sense for them. The question I would ask is why "devices" hurt us (I know a bit about the implementation of course but probably we have to adopt our datamodel also a bit in the future). Generally speaking, form e it also makes sense tob e allowed to have "subcategories" below my devices as my devices usually are "big". And technically speaking in the current version this is totally possible to have nested structures below devices or measurements (but these will then again be devices). So my question is: - Do we really need the static construct of a "device" or can we probably use a different datastructure where I "select" my device only at query time and we just select everything under that tree as ist measurements or "sub-measurements" in cases of nesting. WDYT? Julian Am 20.07.20, 09:34 schrieb "Xiangdong Huang" : Hi, This is a quite good topic! 1. maybe we should hear more users opinions. For me, I think emphasize the concept of "device" is good. We can even expose the concept in our APIs. 2. > A more efficient way is > root.sg.device1.measurement1_int0 > root.sg.device1.measurement1_int1 > root.sg.device1.measurement1_int2 > root.sg.device1.measurement2_long I think the more efficient way is: root.sg.device1.measurement1.0 root.sg.device1.measurement1.1 root.sg.device1.measurement1.2 root.sg.device1.measurement2 And, as you said "a device has a sensor that collects some data in array format (int[3]) and some in long type", will the user query just one element from the int[3]? If not, a better schema is: root.sg.device1.measurement1 (the dataType is int[]) root.sg.device1.measurement2 (the dataType is long) Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2020年7月20日周一 下午3:28写道： > Hi > > Recently, I find that some users create timeseries do not following the > real world semantic of device > > > E.g., a device has a sensor that collects some data in array format > (int[3]) and some in long type. > > > Many users will create timeseries like this: > > > root.sg.device1.measurement1.int0 > root.sg.device1.measurement1.int1 > root.sg.device1.measurement1.int2 > root.sg.device1.measurement2.long > > > As a consequence, there will be two devices instead of one device. This > will cause the real number of devices is much bigger than the real devices > they thought. The drawback is: more devices leads to more memory > consumption. > > > A more efficient way is > > > root.sg.device1.measurement1_int0 > root.sg.device1.measurement1_int1 > root.sg.device1.measurement1_int2 > root.sg.device1.measurement2_long > > > In this schema, there will be only one device and 4 measurements. > > > The problem is we extract the device id automatically. Users usually do > not have a clear concept about "device". Should we emphasize the concept of > device by letting users create device manually? > > > What do you think? > > Thanks, > -- > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院
Re: [Discuss] How to delivery the device concept to users
Hi, This is a quite good topic! 1. maybe we should hear more users opinions. For me, I think emphasize the concept of "device" is good. We can even expose the concept in our APIs. 2. > A more efficient way is > root.sg.device1.measurement1_int0 > root.sg.device1.measurement1_int1 > root.sg.device1.measurement1_int2 > root.sg.device1.measurement2_long I think the more efficient way is: root.sg.device1.measurement1.0 root.sg.device1.measurement1.1 root.sg.device1.measurement1.2 root.sg.device1.measurement2 And, as you said "a device has a sensor that collects some data in array format (int[3]) and some in long type", will the user query just one element from the int[3]? If not, a better schema is: root.sg.device1.measurement1 (the dataType is int[]) root.sg.device1.measurement2 (the dataType is long) Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2020年7月20日周一 下午3:28写道： > Hi > > Recently, I find that some users create timeseries do not following the > real world semantic of device > > > E.g., a device has a sensor that collects some data in array format > (int[3]) and some in long type. > > > Many users will create timeseries like this: > > > root.sg.device1.measurement1.int0 > root.sg.device1.measurement1.int1 > root.sg.device1.measurement1.int2 > root.sg.device1.measurement2.long > > > As a consequence, there will be two devices instead of one device. This > will cause the real number of devices is much bigger than the real devices > they thought. The drawback is: more devices leads to more memory > consumption. > > > A more efficient way is > > > root.sg.device1.measurement1_int0 > root.sg.device1.measurement1_int1 > root.sg.device1.measurement1_int2 > root.sg.device1.measurement2_long > > > In this schema, there will be only one device and 4 measurements. > > > The problem is we extract the device id automatically. Users usually do > not have a clear concept about "device". Should we emphasize the concept of > device by letting users create device manually? > > > What do you think? > > Thanks, > -- > Jialin Qiao > School of Software, Tsinghua University > > 乔嘉林 > 清华大学 软件学院
