Re: Re: [Discuss] How to delivery the device concept to users
Good idea! I think it's ok to seperate the tsfiles to different folders according to the time range if the time range is small enough. Do we need to deal with the query who's time range is very large(historied query)? Like one hundred days, we have to read all tsFiles of the days. Thanks! runhus...@foxmail.com From: Jialin Qiao Date: 2020-07-21 20:28 To: dev Subject: Re: [Discuss] How to delivery the device concept to users Hi, This is not the current implementation... We do not have a partition folder on disk now. By adding a partition folder, there is no need to store all TsFileResources in the memory, and the device index will not hurt us. Thanks, -- Jialin Qiao School of Software, Tsinghua University 乔嘉林 清华大学 软件学院 > -原始邮件- > 发件人: "Xiangdong Huang" > 发送时间: 2020-07-21 18:46:31 (星期二) > 收件人: dev > 抄送: > 主题: Re: [Discuss] How to delivery the device concept to users > > Hi Jialin, > > Yes it is current logic. But I do not know the relation between what you > said and this discussion... > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Jialin Qiao 于2020年7月21日周二 下午4:47写道： > > > Hi, > > > > I would like to give a vision about managing the data files according to > > time partition. > > > > After we introduce the time partition (data is partitioned by time > > interval), we do split them in memory and different TsFiles. But we may > > lake a partition folder layer on top of the TsFiles. > > > > Maybe it should work as follows: > > > > E.g., we insert data into storage group root.sg from 2020-07-19 to > > 2020-07-21 and the partition interval is 1 day. > > First, we create three folders (2020-07-19, 2020-07-20, 2020-07-21) under > > root.sg that belongs to each partition. > > Then, we store TsFiles to its related partition folder. > > > > An example of TsFiles on disk is as follows: > > > > sequence > > ├── root.sg > > │ ├── 2020-07-19 > > │ │ └── timestamp1-version1-merge.tsfile > > │ │ └── timestamp1-version1-merge.tsfile.resource > > │ │ └── ... > > │ ├── 2020-07-20 > > │ ├── 2020-07-21 > > > > > > unsequence(similar with sequence folder) > > ├── root.sg > > │ ├── 2020-07-19 > > │ ├── 2020-07-19 > > │ ├── 2020-07-19 > > > > We only need to store the whole partition folders in memory as a > > List, this memory consumption is negligible. > > > > For the hot partition, e.g., recent 10 days' partition, we could cache > > their TsFileResources in memory to accelerate > > queries. > > > > Then, how to do a query? > > > > Suppose we receive a query: select * from root.sg where time >= > > 2020-07-20 and time <= 2020-07-21 > > > > - We could locate two partitions under root.sg that may contains the > > results: 2020-07-20, 2020-07-21 > > - Then we traverse the partition folder to get all TsFileResources in this > > partition. > > - Finally, we do queries. > > > > Is this feasible? > > > > Thanks, > > -- > > Jialin Qiao > > School of Software, Tsinghua University > > > > 乔嘉林 > > 清华大学 软件学院 > > > > > -原始邮件- > > > 发件人: "Xiangdong Huang" > > > 发送时间: 2020-07-20 20:03:34 (星期一) > > > 收件人: dev > > > 抄送: > > > 主题: Re: Re: [Discuss] How to delivery the device concept to users > > > > > > Hi, > > > > > > > I wonder whether we could index the file by its name. (naming the > > tsfile > > > by date) > > > > > > I think it is a good idea, but maybe not very easy to implement. If we > > can > > > organize the data like this, then it is very very regular and very easy > > to > > > access or delete expired data... > > > > > > > we would need is a tree strucutre where each node has start time / end > > > time for "everything" in the file. > > > > > > This is also a good idea. > > > > > > When we are discussing the granularity of "device", we are worrying about > > > the size of the index, actually. > > > So, we do not care whether there is a so called "sub device", we just > > care > > > how many entities will be indexed. > > > > > > Suppose an IoTDB instance can bear 1 million index entries > > > (start time, end time)>, and given a tree schema, if there are about 1 > > > million nodes from
Re: Re: [Discuss] How to delivery the device concept to users
Hi,
> I wonder whether we could index the file by its name. (naming the tsfile
by date)
I think it is a good idea, but maybe not very easy to implement. If we can
organize the data like this, then it is very very regular and very easy to
access or delete expired data...
> we would need is a tree strucutre where each node has start time / end
time for "everything" in the file.
This is also a good idea.
When we are discussing the granularity of "device", we are worrying about
the size of the index, actually.
So, we do not care whether there is a so called "sub device", we just care
how many entities will be indexed.
Suppose an IoTDB instance can bear 1 million index entries
(start time, end time)>, and given a tree schema, if there are about 1
million nodes from level 0 to level 3, then we can index the nodes on
level3 (so level 3 is so-called "device" in current version).
Meantime, index the nodes from level0 to level2, as Julian proposed, is
also beneficial.
The nature of the above idea is letting IoTDB decides which are "devices"
automatically.
At the beginning of this discussion, I just want to let user claim which
are "devices" (or, which prefixes of Paths have time indexes.. but this
kind of description may be not user friendly..). As it is more easy but
may carry risk if the user set too many devices.
Best,
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
runhus...@foxmail.com 于2020年7月20日周一 下午7:47写道：
> Hi，
>
> > I wonder whether we could index the file by its name. (naming the tsfile
> by date) E.g., we store each day's data in one file and name it as
> sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> we just need to check whether the file exist in the queried interval.
>
> So, how to deal with the out of order data? Could you give more details.
>
>
>
> Thanks!
>
> runhus...@foxmail.com
>
>
> From: Jialin Qiao
> Date: 2020-07-20 18:21
> To: dev
> Subject: Re: [Discuss] How to delivery the device concept to users
> Hi,
>
> > The question I would ask is why "devices" hurt us.
>
> I'd like to introduce this a bit. For each storage group, we flush the
> memtable into TsFiles one by one. For each TsFile, we maintain a temporal
> index on device level in memory. Suppose there are 3 devices in one TsFile,
> the index is like this:
>
> start time array: long[3] = {1, 1, 2}
> end time array: long[3] = {5, 6, 10}
> devicesToIndexInArray: Map = {"root.sg.d1" -> 0,
> "root.sg.d2" -> 1, "root.sg.d3" -> 2}
>
> If we have millions of devices, for each TsFile, this index will reach
> dozens of MB in memory. Although we could introduce the persistence of the
> index. It is still recommended to decrease the number of devices.
>
> I wonder whether we could index the file by its name. (naming the tsfile
> by date) E.g., we store each day's data in one file and name it as
> sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> we just need to check whether the file exist in the queried interval.
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -原始邮件-
> > 发件人: "Julian Feinauer"
> > 发送时间: 2020-07-20 17:34:40 (星期一)
> > 收件人: "dev@iotdb.apache.org"
> > 抄送:
> > 主题: Re: [Discuss] How to delivery the device concept to users
> >
> > Hey Jialin, xinagdong,
> >
> > very good question!
> >
> > And I tend to agree with Xiangdong.
> > If the users do it that way it probably makes most sense for them.
> > The question I would ask is why "devices" hurt us (I know a bit about
> the implementation of course but probably we have to adopt our datamodel
> also a bit in the future).
> >
> > Generally speaking, form e it also makes sense tob e allowed to have
> "subcategories" below my devices as my devices usually are "big".
> > And technically speaking in the current version this is totally possible
> to have nested structures below devices or measurements (but these will
> then again be devices).
> >
> > So my question is:
> > - Do we really need the static construct of a "device" or can we
> probably use a different datastructure where I "select" my device only at
> query time and we just select everything under that tree as ist
> measurements or "sub-measurements" in cases of nesting.
> >
> > WDYT?
> >
> > Julian
> >
> > Am 20.07.20, 09:34 schrieb "Xiangdong Huang" :
> >
> > Hi,
> &
Re: Re: [Discuss] How to delivery the device concept to users
Hi，
> I wonder whether we could index the file by its name. (naming the tsfile by
> date) E.g., we store each day's data in one file and name it as
> sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory,
> we just need to check whether the file exist in the queried interval.
So, how to deal with the out of order data? Could you give more details.
Thanks!
runhus...@foxmail.com
From: Jialin Qiao
Date: 2020-07-20 18:21
To: dev
Subject: Re: [Discuss] How to delivery the device concept to users
Hi,
> The question I would ask is why "devices" hurt us.
I'd like to introduce this a bit. For each storage group, we flush the memtable
into TsFiles one by one. For each TsFile, we maintain a temporal index on
device level in memory. Suppose there are 3 devices in one TsFile, the index is
like this:
start time array: long[3] = {1, 1, 2}
end time array: long[3] = {5, 6, 10}
devicesToIndexInArray: Map = {"root.sg.d1" -> 0, "root.sg.d2"
-> 1, "root.sg.d3" -> 2}
If we have millions of devices, for each TsFile, this index will reach dozens
of MB in memory. Although we could introduce the persistence of the index. It
is still recommended to decrease the number of devices.
I wonder whether we could index the file by its name. (naming the tsfile by
date) E.g., we store each day's data in one file and name it as
sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory, we
just need to check whether the file exist in the queried interval.
Thanks,
--
Jialin Qiao
School of Software, Tsinghua University
乔嘉林
清华大学 软件学院
> -原始邮件-
> 发件人: "Julian Feinauer"
> 发送时间: 2020-07-20 17:34:40 (星期一)
> 收件人: "dev@iotdb.apache.org"
> 抄送:
> 主题: Re: [Discuss] How to delivery the device concept to users
>
> Hey Jialin, xinagdong,
>
> very good question!
>
> And I tend to agree with Xiangdong.
> If the users do it that way it probably makes most sense for them.
> The question I would ask is why "devices" hurt us (I know a bit about the
> implementation of course but probably we have to adopt our datamodel also a
> bit in the future).
>
> Generally speaking, form e it also makes sense tob e allowed to have
> "subcategories" below my devices as my devices usually are "big".
> And technically speaking in the current version this is totally possible to
> have nested structures below devices or measurements (but these will then
> again be devices).
>
> So my question is:
> - Do we really need the static construct of a "device" or can we probably use
> a different datastructure where I "select" my device only at query time and
> we just select everything under that tree as ist measurements or
> "sub-measurements" in cases of nesting.
>
> WDYT?
>
> Julian
>
> Am 20.07.20, 09:34 schrieb "Xiangdong Huang" :
>
> Hi,
>
> This is a quite good topic!
>
> 1. maybe we should hear more users opinions.
>
> For me, I think emphasize the concept of "device" is good. We can even
> expose the concept in our APIs.
>
> 2.
>
> > A more efficient way is
> > root.sg.device1.measurement1_int0
> > root.sg.device1.measurement1_int1
> > root.sg.device1.measurement1_int2
> > root.sg.device1.measurement2_long
>
> I think the more efficient way is:
>
> root.sg.device1.measurement1.0
> root.sg.device1.measurement1.1
> root.sg.device1.measurement1.2
> root.sg.device1.measurement2
>
> And, as you said "a device has a sensor that collects some data in array
> format (int[3]) and some in long type",
> will the user query just one element from the int[3]? If not, a better
> schema is:
>
> root.sg.device1.measurement1 (the dataType is int[])
> root.sg.device1.measurement2 (the dataType is long)
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
>
> Jialin Qiao 于2020年7月20日周一 下午3:28写道：
>
> > Hi
> >
> > Recently, I find that some users create timeseries do not following the
> > real world semantic of device
> >
> >
> > E.g., a device has a sensor that collects some data in array format
> > (int[3]) and some in long type.
> >
> >
> > Many users will create timeseries like this:
> >
> >
> > root.sg.device1.measurement1.int0
> > root.sg.device1.measurement1.int1
> > root.sg.device1.measurement1.int2
> >
