Re: New branch rel/0.13 checked out

[Julian Feinauer]

Fully agree. 0.13.x has soo many cool features that I cant wait to get my hands 
on it : )
Awesome work everybody!

Julian

From: Jialin Qiao 
Date: Thursday, 20. January 2022 at 03:11
To: dev@iotdb.apache.org 
Subject: Re: New branch rel/0.13 checked out
Looking forward to 0.13.0!

―
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院


Haonan Hou  于2022年1月19日周三 14:52写道：

> Hi all,
>
> IoTDB v0.13.0 is coming. Therefore, we checked out a new branch rel/0.13
> for releasing.
> Generally, we don’t accept new features to this branch any more. But bug
> fixes and improvements are acceptable.
> The unmerged new features will be move to the next major version(v0.14).
>
> By the way, the major features of 0.13 are listed in our confluence
> site[1]. Please check it.
>
> [1] https://cwiki.apache.org/confluence/display/IOTDB/v0.13 <
> https://cwiki.apache.org/confluence/display/IOTDB/v0.13>
>
>
> Best,
> Haonan Hou
>
>

AW: Usage of Relation Algebra, Calcite and Query Optimization in IoTDB

[Julian Feinauer]

Hi Xiangdong,
hi Giorgio,

I will share my code soon but currently the code is heavy WIP.
You can find it here in this branch: 
https://github.com/JulianFeinauer/incubator-iotdb/tree/experimental/typeless-backend

Places to look at are here:
https://github.com/JulianFeinauer/incubator-iotdb/tree/experimental/typeless-backend/server/src/main/java/org/apache/iotdb/db/query/executor/calcite
and 
https://github.com/JulianFeinauer/incubator-iotdb/blob/experimental/typeless-backend/server/src/main/java/org/apache/iotdb/db/query/executor/CalciteExecutor.java

@Xiangdong: Calcite has support for Materialized views but basically there is 
(AFAIK!) no special support for indexes.
But in some databases (e.g. Postgres) an Index is physically just a table (but 
that cannot be queried explicitly).
So we could have e.g. a table that contains a previous result of a group by 
operation like a “materialized view” and the optimizer can then integrate it in 
the current plan, if it reduces the query costs.
I also had a short discussion with Yuan Tian how this could look like, exactly.

But I am happy for a wider discussion, of course : )

Best
Julian

Von: Xiangdong Huang 
Datum: Montag, 17. Januar 2022 um 13:54
An: dev 
Betreff: Re: Usage of Relation Algebra, Calcite and Query Optimization in IoTDB
Hi,

This is a good way to make the query processing clear.

> And in the next step, we could introduce INDEXES which in this scenario
could be simply “materialized views” with pre-cached results that were
again stored as a table.

Does Calcite has frameworks for INDEXES? it beyonds my knowledge.

BTW, we will have "SELECT INTO" in v0.13, which can store a query result
into another series.
And, we have "Trigger" which may allow users to generate "materialized
views" if they want.
And, it seems some guys in Alibaba introduced some related concept about
"materialized views". Welcome to share.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Giorgio Zoppi  于2022年1月11日周二 21:16写道：

> Hello Julian,
> could you point out the branch? What about just including what you need of
> Calcite?
> Best Regards,
> Giorgio.
>

Usage of Relation Algebra, Calcite and Query Optimization in IoTDB

[Julian Feinauer]

Hi folks,

I did some discussions already with others but wanted to bring the topic now to 
the list.
Currently, we have no “real” Query Optimizer in IoTDB.
We do some heuristics to optimize queries but not in the way a regular RDBMS 
does things.

Some of you may have heard of the Apache Calcite Framework which is basically a 
complete SQL Frontend which does Query parsing, generation of a Relational Node 
Tree and Query Optimization.
As we are not using regular SQL, we cannot rely on the Query parser and other 
features of the Framework.
But we could use the Optimizer, if we are able to represent our Queries in a 
“relational” way.

I did several experiments on how to integrate Calcite in IoTDB in a way that it 
gets a QueryPlan from IoTDB and later returns a QuerySet which makes it totally 
transparent for the IoTDB server and simply replaces the Logic in the current 
QueryRouter.

To give you a concrete example, take a simple (IoTDB) query like:

select vehicle.d0.s1 from root where root.vehicle.d0.s0 > 100

To represent this as a Relational Expression, we must first consider how the 
tables look. In Database Language we have two input tables (the timeseries 
root.vehicle.d0.s0 and root.vehicle.d0.s0) which can both be considered as 
relational tables of type .

The Logical Query Plan, if we would do aboves operation on Tables then looks 
like that:

LogicalProject(time=[$0], s1=[$1])
  LogicalFilter(condition=[>($2, 100:JavaType(long))])
LogicalSort(sort0=[$0], dir0=[ASC])
  LogicalProject(time=[COALESCE($0, $2)], s1=[$1], s0=[$3])
LogicalJoin(condition=[=($0, $2)], joinType=[left])
  LogicalTableScan(table=[[root, root.test.d0.s1]])
  LogicalTableScan(table=[[root, root.vehicle.d0.s0]])

Of course this could be highly optimized with specific rules (as both tables 
are sorted the join is way easier than a regular join, etc).
If we throw that now into Calcites Query Optimizer (without any IoTDB specific 
rules), we get:

BindableProject(time=[$0], s1=[$1])
  BindableFilter(condition=[>($2, 100:JavaType(long))])
BindableSort(sort0=[$0], dir0=[ASC])
  BindableProject(time=[COALESCE($0, $2)], s1=[$1], s0=[$3])
BindableJoin(condition=[=($0, $2)], joinType=[left])
  BindableTableScan(table=[[root, root.test.d0.s1]])
  BindableTableScan(table=[[root, root.vehicle.d0.s0]])

Which then generates the code we can execute.

So what did we win?
Nothing, so far.

BUT we could now add specific rules e.g. to push the filter down, or to do a 
more effective Join to get en-par with our current performance.
And in the next step, we could introduce INDEXES which in this scenario could 
be simply “materialized views” with pre-cached results that were again stored 
as a table.

So I know that this is a bit theoretic but if people are interested in this 
approach I can go into more details or even suggest to do a design doc or a 
video discussion about the pros and cons of this approach.

I do have a branch with a draft implementation where this feature can simply be 
switched on and off, so we can easily experiment with it without the danger of 
degrading our performance.

Best!
Julian

Usage of Relation Algebra, Calcite and Query Optimization in IoTDB

[Julian Feinauer]

Hi folks,

I did some discussions already with others but wanted to bring the topic now to 
the list.
Currently, we have no “real” Query Optimizer in IoTDB.
We do some heuristics to optimize queries but not in the way a regular RDBMS 
does things.

Some of you may have heard of the Apache Calcite Framework which is basically a 
complete SQL Frontend which does Query parsing, generation of a Relational Node 
Tree and Query Optimization.
As we are not using regular SQL, we cannot rely on the Query parser and other 
features of the Framework.
But we could use the Optimizer, if we are able to represent our Queries in a 
“relational” way.

I did several experiments on how to integrate Calcite in IoTDB in a way that it 
gets a QueryPlan from IoTDB and later returns a QuerySet which makes it totally 
transparent for the IoTDB server and simply replaces the Logic in the current 
QueryRouter.

To give you a concrete example, take a simple (IoTDB) query like:

select vehicle.d0.s1 from root where root.vehicle.d0.s0 > 100

To represent this as a Relational Expression, we must first consider how the 
tables look. In Database Language we have two input tables (the timeseries 
root.vehicle.d0.s0 and root.vehicle.d0.s0) which can both be considered as 
relational tables of type .

The Logical Query Plan, if we would do aboves operation on Tables then looks 
like that:

LogicalProject(time=[$0], s1=[$1])
  LogicalFilter(condition=[>($2, 100:JavaType(long))])
LogicalSort(sort0=[$0], dir0=[ASC])
  LogicalProject(time=[COALESCE($0, $2)], s1=[$1], s0=[$3])
LogicalJoin(condition=[=($0, $2)], joinType=[left])
  LogicalTableScan(table=[[root, root.test.d0.s1]])
  LogicalTableScan(table=[[root, root.vehicle.d0.s0]])

Of course this could be highly optimized with specific rules (as both tables 
are sorted the join is way easier than a regular join, etc).
If we throw that now into Calcites Query Optimizer (without any IoTDB specific 
rules), we get:

BindableProject(time=[$0], s1=[$1])
  BindableFilter(condition=[>($2, 100:JavaType(long))])
BindableSort(sort0=[$0], dir0=[ASC])
  BindableProject(time=[COALESCE($0, $2)], s1=[$1], s0=[$3])
BindableJoin(condition=[=($0, $2)], joinType=[left])
  BindableTableScan(table=[[root, root.test.d0.s1]])
  BindableTableScan(table=[[root, root.vehicle.d0.s0]])

Which then generates the code we can execute.

So what did we win?
Nothing, so far.

BUT we could now add specific rules e.g. to push the filter down, or to do a 
more effective Join to get en-par with our current performance.
And in the next step, we could introduce INDEXES which in this scenario could 
be simply “materialized views” with pre-cached results that were again stored 
as a table.

So I know that this is a bit theoretic but if people are interested in this 
approach I can go into more details or even suggest to do a design doc or a 
video discussion about the pros and cons of this approach.

I do have a branch with a draft implementation where this feature can simply be 
switched on and off, so we can easily experiment with it without the danger of 
degrading our performance.

Best!
Julian

Performance Improvements with Code Generation

[Julian Feinauer]

Hi all,

after a short discussion I had today with Jialin I took some minutes to see if 
/ how code generation could help us to impove the performance on Queries.

The first thing I looked at was the Filterin in the SeriesReader.
Currently many Filters have if conditions / branches in their `satisfy` method.
As this method is called on every datapoint (that is not filtered out on 
another level) there is lot of potential for improvement.

I pushed a branch with my code changes (which are not that much, actually), see 
[1] for all technical details.

What did i do?

Basically I changed the existing filters to also return an AST of their "filter 
operation". So I can visit all filters and dsynamically generate the Java Code 
for the "perfect" Filter in that situation.
This is then dynamically compiled in a Java class and live loaded.

Of course this process takes some time (about 100 ms or a bit below, I think) 
but then improves the speed on every data point.

What are my results so far?

I generated an example file with about 50 Mio. datapoints and three different 
types of filters (simple, standard, complex).

And so far, my results show that the performance can be improved to up to 20%!!!

For simple queries its not beneficial and can make it even a bit slower (see 
overhead above).

For standard queries I saw about 5% improvement in query times.

For complex queries i saw between 10% and 20% improvement in query time.

I will do some more benchmarks the next days and would love tow ork together 
with other devs from the community to get this feature in the next release!

Best!
Julian

PS.: Happy new year for everybody

[1] Branch: https://github.com/apache/iotdb/tree/experimental/code-generation
[https://opengraph.githubassets.com/4a5319a07e99ead0a4adb97783f0ba1682e78fa6c074c1a9204c2ea82a2a347b/apache/iotdb]
GitHub - apache/iotdb at 
experimental/code-generation
Apache IoTDB. Contribute to apache/iotdb development by creating an account on 
GitHub.
github.com

AW: Quarterly report of IoTDB

[Julian Feinauer]

Fine, from my side!

Julian

Von: Xiangdong Huang 
Datum: Mittwoch, 8. Dezember 2021 um 14:22
An: dev 
Betreff: Quarterly report of IoTDB
Hi all,

today is the last day to submit the quarterly report of IoTDB to the Board.
It is my fault that not preparing it earlier.

The following is the draft:

## Description:
The mission of Apache IoTDB is the creation and maintenance of software related
to an IoT native database with high performance for data management and analysis

## Issues:
No

## Membership Data:
Apache IoTDB was founded 2020-09-16 (a year ago)
There are currently 41 committers and 24 PMC members in this project.
The Committer-to-PMC ratio is roughly 2:1.

Community changes, past quarter:
- No new PMC members. Last addition was Chao Wang on 2021-06-02.
- No new committers. Last addition was Chao Wang on 2021-07-03.

## Project Activity:
- IoTDB-0.12.3 was released on 2021-11-18.
- The community is discussing about the new architecture of IoTDB's
 cluster architecture.
- v0.13 is in development, and we will introduce new aligned timeseries feature.
- the project is introduced in China Open Source conference.
- more new contributors are developing IoTDB's ecosystem softwares,
and we are trying to discuss with them whether they will donate to the project.
- a subproject (iotdb-quality) is voted and approved by IoTDB PMC.

## Community Health:
We can see more new contributors joining the community,
and the dev@ mailing list traffic increases.
However, the Chi of the project decreased, and we will investigate why.

If you have any suggestions or something that you think should be
reported to the Board, let me know.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

AW: IoTDB-quality applying for subproject

[Julian Feinauer]

Hi Xiangdong,

we basically have to do a vote here AFAIR and also do the incubation process 
for IP CLEAREANCE and then do a vote there (but can also be lazy consensus).
And we need the ILCA / CLA of all the contributors to the subproject.

Best
Julian

Von: Xiangdong Huang 
Datum: Freitag, 26. November 2021 um 04:17
An: dev 
Betreff: Re: IoTDB-quality applying for subproject
Hi Julian,

do you know if we need some additional processes if we want to make
the donation as a subproject?
Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Julian Feinauer  于2021年11月26日周五 上午3:07写道：
>
> Hey,
>
> thats really good news and I did not yet know about the package but we will 
> certainly have a look as that’s exactly what we need at the moment!
>
> Great Job!
> Julian
>
> Von: Houliang Qi 
> Datum: Donnerstag, 25. November 2021 um 03:57
> An: dev@iotdb.apache.org 
> Betreff: Re: IoTDB-quality applying for subproject
> Great!
> It’s really good news, and the IoTDB-quality is really useful!
> Hope to contribute to it!
>
>
> Thanks,
> ---
> Houliang Qi
> BONC, Ltd
>
>
> On 11/25/2021 10:48，Chao Wang wrote：
> Great! Thank you for your contribution!
>
>
>
>
> Thanks!
>
>
> Chao Wang
> BONC ltd
> ccgow...@163.com
> On 11/25/2021 10:16，Trevor Hart wrote：
> Great news as I would love to contribute to it!
>
>
>
> Thanks
>
> Trevor
>
>
>
>
>
>
>  On Thu, 25 Nov 2021 15:15:03 +1300 陈 鹏宇  wrote 
>
>
>
> Hi everyone,
> I'm on behalf of IoTDB-quality developers. We have recently decided to make 
> our project open source, and formally apply it for a subproject of Apache 
> IoTDB.
> IoTDB-quality is a collection of IoTDB UDFs designed for data quality 
> diagnosis, including common statistics, anomaly detection, frequency 
> analysis, regular expression processing, data quality analysis and data 
> repairing.
> For more documentation, please visit the homepage of IoTDB-quality.
> https://thulab.github.io/iotdb-quality/
> We are also hoping for contributions from community.
>
> Pengyu Chen
> Nov 25 2021

AW: IoTDB-quality applying for subproject

[Julian Feinauer]

Hey,

thats really good news and I did not yet know about the package but we will 
certainly have a look as that’s exactly what we need at the moment!

Great Job!
Julian

Von: Houliang Qi 
Datum: Donnerstag, 25. November 2021 um 03:57
An: dev@iotdb.apache.org 
Betreff: Re: IoTDB-quality applying for subproject
Great!
It’s really good news, and the IoTDB-quality is really useful!
Hope to contribute to it!


Thanks,
---
Houliang Qi
BONC, Ltd


On 11/25/2021 10:48，Chao Wang wrote：
Great! Thank you for your contribution!




Thanks!


Chao Wang
BONC ltd
ccgow...@163.com
On 11/25/2021 10:16，Trevor Hart wrote：
Great news as I would love to contribute to it!



Thanks

Trevor






 On Thu, 25 Nov 2021 15:15:03 +1300 陈 鹏宇  wrote 



Hi everyone,
I'm on behalf of IoTDB-quality developers. We have recently decided to make our 
project open source, and formally apply it for a subproject of Apache IoTDB.
IoTDB-quality is a collection of IoTDB UDFs designed for data quality 
diagnosis, including common statistics, anomaly detection, frequency analysis, 
regular expression processing, data quality analysis and data repairing.
For more documentation, please visit the homepage of IoTDB-quality.
https://thulab.github.io/iotdb-quality/
We are also hoping for contributions from community.

Pengyu Chen
Nov 25 2021

AW: refactor cluster discussion

[Julian Feinauer]

Hi all,

yes, sorry fort he confusion.
The Benchmark was something from the outside and I found its okay to share it 
here as mayn people from the community can read everything and evaluate if its 
done “properly” and stuff.

Best
Julian

Von: Willem Jiang 
Datum: Donnerstag, 25. November 2021 um 01:15
An: dev 
Betreff: Re: refactor cluster discussion
Oh，I think IoTDBers are still following the rule of having a
disucssion in English.
I just had a quick look at the thread and found out Julian just
provides some links of RPC benchmarks which were written in Chinese.
Even though there was a discussion meeting in Chinese, XiangDong wrote
the minutes in English.
https://cwiki.apache.org/confluence/display/IOTDB/%5BChinese%5Diotdb+cluster+discussion+2021.11

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Thu, Nov 25, 2021 at 3:36 AM Giorgio Zoppi  wrote:
>
> Hello IOTDBers,
> it's important for us US/EU people to have all in English..it's a
> communication/learning thing.
> BR,
> Giorgio.
>
> Il giorno mar 23 nov 2021 alle ore 10:54 Xiangdong Huang 
> ha scritto:
>
> > Hi,
> >
> > yes you raise up another existing topic: whether and how to decouple
> > and smoothly change to another RPC framework?
> >
> > Besides, I wonder how do you find these Chinese materials...
> >
> > Best,
> >
> > ---
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  黄向东
> > 清华大学 软件学院
> >
> > Julian Feinauer  于2021年11月23日周二 下午4:53写道：
> > >
> > > Hey Xiangdong,
> > >
> > > that makes totally sense and its good that we „can“ always do CP but
> > switch more towards AP if wanted.
> > > Regarding RPC I just found this interesting benchmark (you may be better
> > able to read everything…):
> > >
> > > https://github.com/eisig/rpc-benchmark
> > > https://www.jianshu.com/p/cdd94d0853c3
> > >
> > > Best
> > > Julian
> > >
> > > Von: Xiangdong Huang 
> > > Datum: Dienstag, 23. November 2021 um 09:46
> > > An: dev 
> > > Betreff: Re: refactor cluster discussion
> > > Hi Julian,
> > >
> > > Well, in my view, it is AP by default, and can switch to CP (but for
> > > some operations, it only supports CP).
> > > 1. As we used cache for reducing RPC, it is AP. But if we mandatorily
> > > requires check whether the cache is the latest, then it is CP but has
> > > a larger latency.
> > > 2. For some un-frequent operations (e.g., delete timeseries, delete
> > > storage group), we use CP, and requires all nodes to execute the
> > > operations successfully.
> > >
> > > For RPC times, yes the less, the better.  If some RPC requests can not
> > > be avoid, then the other way is reduce the message size of the request
> > > and response.
> > >
> > > Best,
> > > ---
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> > >
> > > Julian Feinauer  于2021年11月23日周二 下午4:14写道：
> > >
> > > >
> > > > Hi Community,
> > > >
> > > > I really like the discussions here and although I do not find enough
> > time (and language skills in Chinese) to participate in the meetups I think
> > we are on a very good way.
> > > >
> > > > While reading through the docs provided I just quickly had two
> > thoughts that I wanted to share here
> > > >
> > > >
> > > >   *   Regarding the CAP Theorem, where do we (want to) place
> > ourselves? The current cluster module is, from my understanding CP but many
> > MPP Databases rather go for AP (I think). I am not sure whats the best
> > approach for timeseries and perhaps theres even the chance to switch
> > between both modes in certain scenarios (people usually call it something
> > like almost certainly consistent or something.. but math is clear, you are
> > either CP or AP nothing in between)
> > > >   *   If we do have a lot of communication over sockets we should
> > perhaps re-evaluate our RPC mechanism. I know that it’s a razors edge
> > between “preliminary optimization” and doing “the right”. But I think it
> > would be good to reconsider or check a bit on how much time we loose on RPC
> > because that’s a latency we always have to pay and probably multiple times
> > during a single call depending on the situation
> > > >
> > > > If that’s already well discu

AW: share a rust based IoTDB-client

[Julian Feinauer]

Francis,

let me give you a very warm welcome here!
The work you did so far is simply awesome and I thank you very much!

Would be cool to have you here in the community and maybe contribute some of 
your work to the project itself.

Greetings!
Julian

Von: Francis Du 
Datum: Dienstag, 23. November 2021 um 12:48
An: dev@iotdb.apache.org 
Betreff: Re: share a rust based IoTDB-client
Hi guys, I'm Francis. The author of iotdb-rs. And welcome to join in and
contribute this.There is also an IotDB cli [1] client built by iotdb-rs,
and everyone is welcome to experience and feedback.

I also plan to donate them to the community. There are still some work to
be improved. I hope to get you guys  guidance.

[1]. https://github.com/francis-du/iotdb-rs

On Tue, Nov 23, 2021, 18:52 Jialin Qiao  wrote:

> Good work!
>
> Best,
> Jialin Qiao
>
> Xiangdong Huang  于2021年11月23日周二 下午4:36写道：
>
> > Yes, I have contacted him, and he has already subscribed to the mailing
> > list.
> >
> > ---
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  黄向东
> > 清华大学 软件学院
> >
> > Julian Feinauer  于2021年11月23日周二 下午4:20写道：
> > >
> > > Thats pretty cool work.
> > > Did you contact the author @Xiangdong?
> > > I think it would be cool to invite him here to this community if hes
> not
> > already in?
> > >
> > > Best
> > > Julian
> > >
> > > Von: Xiangdong Huang 
> > > Datum: Mittwoch, 17. November 2021 um 03:42
> > > An: dev 
> > > Betreff: share a rust based IoTDB-client
> > > Hi,
> > >
> > > to whom may be interested in:
> > >
> > > I find a rust based iotdb-client when I explore Twitter :D
> > >
> > > [1] https://twitter.com/francis_run/status/1457699833981046790?s=20
> > > [2] https://github.com/francis-du/iotdb-rs
> > >
> > > Best,
> > > ---
> > > Xiangdong Huang
> > > School of Software, Tsinghua University
> > >
> > >  黄向东
> > > 清华大学 软件学院
> >
>

AW: refactor cluster discussion

[Julian Feinauer]

Hey Xiangdong,

that makes totally sense and its good that we „can“ always do CP but switch 
more towards AP if wanted.
Regarding RPC I just found this interesting benchmark (you may be better able 
to read everything…):

https://github.com/eisig/rpc-benchmark
https://www.jianshu.com/p/cdd94d0853c3

Best
Julian

Von: Xiangdong Huang 
Datum: Dienstag, 23. November 2021 um 09:46
An: dev 
Betreff: Re: refactor cluster discussion
Hi Julian,

Well, in my view, it is AP by default, and can switch to CP (but for
some operations, it only supports CP).
1. As we used cache for reducing RPC, it is AP. But if we mandatorily
requires check whether the cache is the latest, then it is CP but has
a larger latency.
2. For some un-frequent operations (e.g., delete timeseries, delete
storage group), we use CP, and requires all nodes to execute the
operations successfully.

For RPC times, yes the less, the better.  If some RPC requests can not
be avoid, then the other way is reduce the message size of the request
and response.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Julian Feinauer  于2021年11月23日周二 下午4:14写道：

>
> Hi Community,
>
> I really like the discussions here and although I do not find enough time 
> (and language skills in Chinese) to participate in the meetups I think we are 
> on a very good way.
>
> While reading through the docs provided I just quickly had two thoughts that 
> I wanted to share here
>
>
>   *   Regarding the CAP Theorem, where do we (want to) place ourselves? The 
> current cluster module is, from my understanding CP but many MPP Databases 
> rather go for AP (I think). I am not sure whats the best approach for 
> timeseries and perhaps theres even the chance to switch between both modes in 
> certain scenarios (people usually call it something like almost certainly 
> consistent or something.. but math is clear, you are either CP or AP nothing 
> in between)
>   *   If we do have a lot of communication over sockets we should perhaps 
> re-evaluate our RPC mechanism. I know that it’s a razors edge between 
> “preliminary optimization” and doing “the right”. But I think it would be 
> good to reconsider or check a bit on how much time we loose on RPC because 
> that’s a latency we always have to pay and probably multiple times during a 
> single call depending on the situation
>
> If that’s already well discussed and written in the documents than please 
> excuse me missing it.
>
> Best!
> Julian
>
> Von: Xiangdong Huang 
> Datum: Montag, 22. November 2021 um 03:04
> An: dev 
> Betreff: refactor cluster discussion
> Hi all,
>
> In the brainstorming last week [1], we tried to introduce a more clear
> code decoupling design, which embraced MPP architecture, and shrinked
> the raft protocol only into a small scope in the codebase (then it is
> easy to replace the implementation of raft, and even replace raft to
> other replica mechanism).
>
> Some detailed discussion is on [2], and welcome to supply the doc.
>
> [1] 
> https://cwiki.apache.org/confluence/display/IOTDB/%5BChinese%5Diotdb+cluster+discussion+2021.11
>
> [2] 
> https://cwiki.apache.org/confluence/display/IOTDB/refactor-2021-MPP-decoupling
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院

AW: share a rust based IoTDB-client

[Julian Feinauer]

Thats pretty cool work.
Did you contact the author @Xiangdong?
I think it would be cool to invite him here to this community if hes not 
already in?

Best
Julian

Von: Xiangdong Huang 
Datum: Mittwoch, 17. November 2021 um 03:42
An: dev 
Betreff: share a rust based IoTDB-client
Hi,

to whom may be interested in:

I find a rust based iotdb-client when I explore Twitter :D

[1] https://twitter.com/francis_run/status/1457699833981046790?s=20
[2] https://github.com/francis-du/iotdb-rs

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

AW: refactor cluster discussion

[Julian Feinauer]

Hi Community,

I really like the discussions here and although I do not find enough time (and 
language skills in Chinese) to participate in the meetups I think we are on a 
very good way.

While reading through the docs provided I just quickly had two thoughts that I 
wanted to share here


  *   Regarding the CAP Theorem, where do we (want to) place ourselves? The 
current cluster module is, from my understanding CP but many MPP Databases 
rather go for AP (I think). I am not sure whats the best approach for 
timeseries and perhaps theres even the chance to switch between both modes in 
certain scenarios (people usually call it something like almost certainly 
consistent or something.. but math is clear, you are either CP or AP nothing in 
between)
  *   If we do have a lot of communication over sockets we should perhaps 
re-evaluate our RPC mechanism. I know that it’s a razors edge between 
“preliminary optimization” and doing “the right”. But I think it would be good 
to reconsider or check a bit on how much time we loose on RPC because that’s a 
latency we always have to pay and probably multiple times during a single call 
depending on the situation

If that’s already well discussed and written in the documents than please 
excuse me missing it.

Best!
Julian

Von: Xiangdong Huang 
Datum: Montag, 22. November 2021 um 03:04
An: dev 
Betreff: refactor cluster discussion
Hi all,

In the brainstorming last week [1], we tried to introduce a more clear
code decoupling design, which embraced MPP architecture, and shrinked
the raft protocol only into a small scope in the codebase (then it is
easy to replace the implementation of raft, and even replace raft to
other replica mechanism).

Some detailed discussion is on [2], and welcome to supply the doc.

[1] 
https://cwiki.apache.org/confluence/display/IOTDB/%5BChinese%5Diotdb+cluster+discussion+2021.11

[2] 
https://cwiki.apache.org/confluence/display/IOTDB/refactor-2021-MPP-decoupling

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

AW: please add JIRA ticket and ping me/jialin to add assignment permission if you want to assign some issues to yourself

[Julian Feinauer]

Hey Giorgio,

I have this little site setup where you can easily and freely register a demo 
site: https://iotdb.demo.service.pragmaticindustries.com/

Here is my pypi with TestContainers support and Pandas integration: 
https://pypi.org/project/apache-iotdb-nightly/

Best
Julian


Von: Giorgio Zoppi 
Datum: Dienstag, 26. Oktober 2021 um 12:19
An: dev 
Betreff: Re: please add JIRA ticket and ping me/jialin to add assignment 
permission if you want to assign some issues to yourself
Hello Julian,
the community should grow a bit. If we've a bigger community, this product
become marketable. Do you have any slides to share or any python code,
otherwise i will sort out.
Also do we have a public demo, i know that microsoft is offering Azure for
free at open source project. It would be nice to have there a demo
site.Feel free to share the repo for testing and code review.

Best Regards,
Giorgio.

AW: please add JIRA ticket and ping me/jialin to add assignment permission if you want to assign some issues to yourself

[Julian Feinauer]

Hi Giorgio,

you can also ping me with further questions as I also did a integration of 
pandas and IoTDB to make python more convenient.
Currently I am preparing a Django integration as well.

Julian

Von: Giorgio Zoppi 
Datum: Montag, 25. Oktober 2021 um 18:29
An: dev 
Betreff: Re: please add JIRA ticket and ping me/jialin to add assignment 
permission if you want to assign some issues to yourself
Hello IOTDBers,
do we have a python client for a demo? I would like to present IOTDB at
PyBCN.
Best Regards,
Giorgio.

AW: [DISCUSS] use different docker image name for iotdb modules

[Julian Feinauer]

Hey,

good that we have a discussion here. I am unemotional about the outcome and I 
see advantages and disadvantages for both (3 releases, 3 readmes, …) in the 
first scenario but all the points below in the other one.

So its good to have many opinions here : )

Julian

Von: gaoyang <453935...@qq.com.INVALID>
Datum: Donnerstag, 21. Oktober 2021 um 08:58
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] use different docker image name for iotdb modules
1. Docker use a UFS file system, different images can share the same file 
layer, such as JRE…,  so, If I have two both iot-granafa and iot images in my 
server, actually, I have only one JRE layer.

2. Say I am a user, I only want to use the grafana module, then I have to pass 
a ‘grafana’ param when I run a container, it’s a bit of annoying…

3. In some case, say I am using a k8s, if I want write my own entrypoint 
script, I have to pass a ‘grafana’ param to the iot entrypoint script…

4. Someone hope the image as small as possible, if I only want a grafana, why 
give me an additional iotdb-cluster?

In Summary, I prefer there standalone images to a three-in-one image.

> 2021年10月20日 下午9:47，Xiangdong Huang  写道：
>
>> Downside is that the image would be a bit bigger.
>
> Well, I think it is fine, as current image is huge because of JRE...
> and the 3 modules have many same files..
>
> I think `docker run image command` is also a good solution and maybe
> better than my proposal :D
>
> But can docker-compose also use different command using the same docker image?
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
> Julian Feinauer  于2021年10月20日周三 下午9:25写道：
>>
>> Hey,
>>
>> I agree with it (and already stumbled upon it).
>>
>> Another alternative would be to have only one image containing data for all 
>> three modules and an entrypoint script which distinguishes what process 
>> would really start.
>>
>> So you would just do
>>
>> ```
>> docker run iotdb <-- regular
>> docker run iotdb cluster <-- cluster mode
>> docker run iotdb Grafana <-- grafana bridge
>> ```
>>
>> Downside is that the image would be a bit bigger.
>> Advantage is that we cannot miss once but simply ship one image and that’s 
>> it.
>>
>> WDYT?
>>
>> Julian
>>
>> Von: Xiangdong Huang 
>> Datum: Mittwoch, 20. Oktober 2021 um 15:13
>> An: dev 
>> Betreff: [DISCUSS] use different docker image name for iotdb modules
>> Hi,
>>
>> Now all iotdb's docker images are under apache/iotdb, e.g.,
>>
>> - apache/iotdb:0.12.2-node for the single node module
>> - apache/iotdb:0.12.2-cluster for the cluster module
>>
>> and
>> - apache/iotdb:0.12.2-grafana for the grafana connector module
>>
>> In this way, we actually use the version of the images to distinguish
>> different modules.
>>
>> The advantage is that users can find all images in one webpage [1]
>>
>> The disadvantage is that it is hard to define the tag of "latest".
>>
>> The problem will be more serious when using docker-compose, for example,
>> we may write a docker-compose file for starting iotdb server, grafana,
>> iotdb-grafana-connector:
>>
>> ```
>> services:
>>iotdb:
>>  image: apache/iotdb:0.12.2-node
>>grafana-connector:
>>  image: apache/iotdb:0.12.2-grafana
>> ```
>>
>> Then we can not use "latest" and we have to maintain docker-compose
>> file for each version.
>> If we separate the docker repo into individuals, then we never do not
>> need to maintain the docker compose file:
>>
>> ```
>> services:
>>iotdb:
>>  image: apache/iotdb
>>grafana-connector:
>>  image: apache/iotdb-grafana
>> ```
>> and the docker compose file will use the "latest" version directly.
>>
>>
>> So, my idea is, apply 3 repos for iotdb docker images:
>> - iotdb, for iotdb-node
>> - iotdb-cluster, for the cluster module
>> - iotdb-grafana, for iotdb-grafana-connecotr module.
>>
>> How do you think?
>>
>>
>> [1] https://hub.docker.com/r/apache/iotdb
>>
>> Best,
>> ---
>> Xiangdong Huang
>> School of Software, Tsinghua University
>>
>> 黄向东
>> 清华大学 软件学院

AW: [DISCUSS] use different docker image name for iotdb modules

[Julian Feinauer]

Hey,

of course this works with docker compose.
You just have to use the “command” option in the docker-compose yaml file (you 
can state that for each separate service).

See e.g. here: 
https://stackoverflow.com/questions/40210067/how-to-use-command-in-docker-compose-yml

Best
J

Von: Xiangdong Huang 
Datum: Mittwoch, 20. Oktober 2021 um 15:47
An: dev 
Betreff: Re: [DISCUSS] use different docker image name for iotdb modules
> Downside is that the image would be a bit bigger.

Well, I think it is fine, as current image is huge because of JRE...
and the 3 modules have many same files..

I think `docker run image command` is also a good solution and maybe
better than my proposal :D

But can docker-compose also use different command using the same docker image?

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Julian Feinauer  于2021年10月20日周三 下午9:25写道：
>
> Hey,
>
> I agree with it (and already stumbled upon it).
>
> Another alternative would be to have only one image containing data for all 
> three modules and an entrypoint script which distinguishes what process would 
> really start.
>
> So you would just do
>
> ```
> docker run iotdb <-- regular
> docker run iotdb cluster <-- cluster mode
> docker run iotdb Grafana <-- grafana bridge
> ```
>
> Downside is that the image would be a bit bigger.
> Advantage is that we cannot miss once but simply ship one image and that’s it.
>
> WDYT?
>
> Julian
>
> Von: Xiangdong Huang 
> Datum: Mittwoch, 20. Oktober 2021 um 15:13
> An: dev 
> Betreff: [DISCUSS] use different docker image name for iotdb modules
> Hi,
>
> Now all iotdb's docker images are under apache/iotdb, e.g.,
>
> - apache/iotdb:0.12.2-node for the single node module
> - apache/iotdb:0.12.2-cluster for the cluster module
>
> and
> - apache/iotdb:0.12.2-grafana for the grafana connector module
>
> In this way, we actually use the version of the images to distinguish
> different modules.
>
> The advantage is that users can find all images in one webpage [1]
>
> The disadvantage is that it is hard to define the tag of "latest".
>
> The problem will be more serious when using docker-compose, for example,
> we may write a docker-compose file for starting iotdb server, grafana,
> iotdb-grafana-connector:
>
> ```
> services:
> iotdb:
>   image: apache/iotdb:0.12.2-node
> grafana-connector:
>   image: apache/iotdb:0.12.2-grafana
> ```
>
> Then we can not use "latest" and we have to maintain docker-compose
> file for each version.
> If we separate the docker repo into individuals, then we never do not
> need to maintain the docker compose file:
>
> ```
> services:
> iotdb:
>   image: apache/iotdb
> grafana-connector:
>   image: apache/iotdb-grafana
> ```
> and the docker compose file will use the "latest" version directly.
>
>
> So, my idea is, apply 3 repos for iotdb docker images:
> - iotdb, for iotdb-node
> - iotdb-cluster, for the cluster module
> - iotdb-grafana, for iotdb-grafana-connecotr module.
>
> How do you think?
>
>
> [1] https://hub.docker.com/r/apache/iotdb
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院

AW: [DISCUSS] use different docker image name for iotdb modules

[Julian Feinauer]

Hey,

I agree with it (and already stumbled upon it).

Another alternative would be to have only one image containing data for all 
three modules and an entrypoint script which distinguishes what process would 
really start.

So you would just do

```
docker run iotdb <-- regular
docker run iotdb cluster <-- cluster mode
docker run iotdb Grafana <-- grafana bridge
```

Downside is that the image would be a bit bigger.
Advantage is that we cannot miss once but simply ship one image and that’s it.

WDYT?

Julian

Von: Xiangdong Huang 
Datum: Mittwoch, 20. Oktober 2021 um 15:13
An: dev 
Betreff: [DISCUSS] use different docker image name for iotdb modules
Hi,

Now all iotdb's docker images are under apache/iotdb, e.g.,

- apache/iotdb:0.12.2-node for the single node module
- apache/iotdb:0.12.2-cluster for the cluster module

and
- apache/iotdb:0.12.2-grafana for the grafana connector module

In this way, we actually use the version of the images to distinguish
different modules.

The advantage is that users can find all images in one webpage [1]

The disadvantage is that it is hard to define the tag of "latest".

The problem will be more serious when using docker-compose, for example,
we may write a docker-compose file for starting iotdb server, grafana,
iotdb-grafana-connector:

```
services:
iotdb:
  image: apache/iotdb:0.12.2-node
grafana-connector:
  image: apache/iotdb:0.12.2-grafana
```

Then we can not use "latest" and we have to maintain docker-compose
file for each version.
If we separate the docker repo into individuals, then we never do not
need to maintain the docker compose file:

```
services:
iotdb:
  image: apache/iotdb
grafana-connector:
  image: apache/iotdb-grafana
```
and the docker compose file will use the "latest" version directly.


So, my idea is, apply 3 repos for iotdb docker images:
- iotdb, for iotdb-node
- iotdb-cluster, for the cluster module
- iotdb-grafana, for iotdb-grafana-connecotr module.

How do you think?


[1] https://hub.docker.com/r/apache/iotdb

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Tags for Apache IoTDB on Stackoverflow

[Julian Feinauer]

Hi folks,

thanks to Chris we have now two dedicated tags on Stackoverflow: iotdb and 
apache-iotdb where you can tag respective questions or watch them tob e able to 
answer them better!

https://stackoverflow.com/questions/tagged/iotdb
https://stackoverflow.com/questions/tagged/apache-iotdb

Best
Julian

AW: Use of generics.

[Julian Feinauer]

Hey,

that is, although not very elegant a very common pattern in Java and is e.g. 
also used by Apache Calcite internally. There a "Table Scan" is roughly 
representat as an Object[][]. It would be more elegant to use Generics but as 
Yuan states the first dimension of the array dictates the rows and the second 
the columns. So we could only end up with something like
TupleN[] where we would need to have 
classes for all these Tuple_N's.
And as they all have a different amount of dependant generic parameters we 
could not abstract that type away.
So you go back to Object[][] and basically tell the computer to not care about 
types here as you know what you are doing (and you better do...).

Best
Julian

Von: Yuan Tian 
Gesendet: Mittwoch, 29. September 2021 09:25
An: dev@iotdb.apache.org 
Betreff: Re: Use of generics.

Hi,
In Java, array is also an object, so the` Object[] values` in Tablet.java
is a two-dimension array. Object represents arrays of different type.

Java doesn't have class member function template, however maybe in C++, you
can use that feature to improve the performance. But even in this way, we
cannot only contains 'primitive" types array in tablet, because different
sensors in this tablet may have different types and the number of the
sensors are also unknown.

But you can use variadic template feature introduced since C++11 to fix
that. You can refer to the implementation of tuple in C++, the number of
which is also unknown.

On Wed, Sep 29, 2021 at 9:17 AM Giorgio Zoppi 
wrote:

> Hello again,
> i am in learning mode. I've seen in the Tablet.java an array of Object[]
> and i want to know the rational of this. We know that in Java generics are
> "fake" due to the type erasure, but is there any possibility to replace
> that:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *** deviceId of this tablet */  public String prefixPath;  /** the list of
> measurement schemas for creating the tablet */  private
> List schemas;  /**
> measurementId->indexOf(measurementSchema) */  private Map
> measurementIndex;  /** timestamps in this tablet */  public long[]
> timestamps;  /** each object is a primitive type array, which represents
> values of one measurement */  public Object[] values;  /** each bitmap
> represents the existence of each value in the current column. */  public
> BitMap[] bitMaps;  /** the number of rows to include in this tablet */
> public int rowSize;  /** the maximum number of rows for this tablet */
> private int maxRowNumber;  /** whether this tablet store data of aligned
> timeseries or not */  private boolean isAligned;*
>
>
> With restrictions, i mean a primitive type array  wrapper that it doesn't
> accept virtually everything
> in
> - addValue(String measurementId, int rowIndex, Object value)
>
>
> And replacing addValue with something like
>
> -addValue ( String measurementId, int rowIndex, T
> Object);
> -addValue ( String measurementId, int rowIndex, T
> Object);
>
> In this way we could have in the tablet only the 'primitive" types and
> using boxing as well.
> What do you think?
> Best Regards,
> Giorgio.
>
>
>
>
>
> --
> Life is a chess game - Anonymous.
>

AW: IOTDB-Lite as sqllite approach

[Julian Feinauer]

Hi Giorgio and Xiangdong,

I agree with both of you.
And as initially indicated I think the long term perfect way would be to have a 
C based implementation for the whole "process" with wrappers for many languages 
and ecosystems (as SQLITE does it).

And I agree that the advantage in a Java setup is not much although I sometimes 
see the advantage of having something running in the main process of an 
application instead of having a separate service (Android is a good example, 
indeed).

Best
Julian


Von: Giorgio Zoppi 
Gesendet: Mittwoch, 29. September 2021 03:20
An: dev 
Betreff: Re: IOTDB-Lite as sqllite approach

We might want to support TsFile in different languages: Go, Rust, C and C++
as parquet/arrow people are doing.

AW: Endianess.

[Julian Feinauer]

Big +1 for fixed Endianness!

I think a C implementation would also be beneficial for something like a sqlite 
interface (see my thread here 
https://lists.apache.org/thread.html/rd4d0296e0fef9c1c822f605f641d829047e0932429499a5e05667b8e%40%3Cdev.iotdb.apache.org%3E)
 as it could be wrapped to whatever language we like basically (again 
sqlite...).

Best
Julian

Von: Giorgio Zoppi 
Datum: Montag, 27. September 2021 um 13:06
An: dev 
Betreff: Re: Endianess.
Hello IOTDBers,
Chris I agree with you, everything shall be Network Byte Order(
https://www.ibm.com/docs/en/zos/2.4.0?topic=hosts-network-byte-order) by
the first writing. Thank you. I will look at your C code for review,
The first stage is to support natively TsFile.  At this moment we've just
unmolded code however looking at Apache Arrow codebase is useful for the
progress.  After the ts file we'll look together at a server, my idea is to
have in the long term a lower latency product that scales better than Java
ones (iotdb-native). But for the moment it's just a dream, so baby steps
and hard work as usual.
Thank you!

Best Regards,
Giorgio.

IOTDB-Lite as sqllite approach

[Julian Feinauer]

Hi folks,

I just came up with a little idea to improve the usage of TSFiles at the 
„edge“. What if we create an access layer (besides the Readers /Writers) to 
access the TSFiles with SQL as if they were in the server.
Thats basically the same that sqlite does with extreme success 
(https://sqlite.org/index.html).

For Java I created a very minimal sketch based on Apache Calcite (so with a 
different SQL Dialect than the server but this could be adapted), here: 
https://github.com/JulianFeinauer/incubator-iotdb/tree/feature/iotdb-lite/iotdb-lite

What do you think about that?

Happy to hear many ideas and feedback : )

Julian

Re: TsFile Native Library.

[Julian Feinauer]

Thank you Chris!

I personally also would love to try this out and would be happy to join any 
efforts in this regard.
We already discussed tsfile in go with Chris and Niklas.

Julian

Holen Sie sich Outlook für Android

From: Christofer Dutz 
Sent: Thursday, September 23, 2021 6:04:52 PM
To: dev@iotdb.apache.org 
Subject: AW: TsFile Native Library.

Hi all,

just reading this ... thought I might also jump in on this.
Julian and I have recently been discussing how we might be able to use our 
PLC4X Code Generation framework to help bring the TSFile functionality to all 
languages PLC4X currently supports.

We were thinking of specifying the TSFile format in our mspec format, where we 
specify the datastructures and the logic for parsing and serializing.

If this works, we would be able to generate the core functionality fort he 
follwing languages:

- Java
- Go
- C
- C#
And possibly soon:
- Python
- Rust
... C++ shouldn't be too much work and we also thought of that but in general 
we thought it would make more sense to go down the Rust route

Just tell me if you folks are interested in working on this together.
I'm super-fit with the code generation and specification part, but making a 
library from that still would require some manual work.

Chris


-Ursprüngliche Nachricht-
Von: Xiangdong Huang 
Gesendet: Donnerstag, 23. September 2021 13:55
An: dev 
Betreff: Re: TsFile Native Library.

It is really good news, and look forward to hearing its progress.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Yuan Tian  于2021年9月23日周四 下午4:45写道：
>
> Hi,
>
> I'm glad to see that. I'm also trying to learn C++ these days, so
> maybe we can do this work together. And if you have any problems about
> Tsfile module in java, feel free to contact me!
>
> Best,
> ---
> Yuan Tian
>
>
> On Thu, Sep 23, 2021 at 12:13 AM Giorgio Zoppi
> 
> wrote:
>
> > Hello everybody,
> > after a while today i had some time to write code about ts file
> > parsing in native code (C++20).
> > There is still nothing, but if you can take a look
> > https://github.com/giorgiozoppi/tsfile
> > Best Regards,
> > Giorgio.
> >

Running IoTDB in Kubernetes

[Julian Feinauer]

Hi Folks,

I finally found some time to continue working on running IoTDB in Kubernetes.

Some may remember that I posted a draft Helm Chart 
(https://artifacthub.io/packages/helm/apache-iotdb-single-node/apache-iotdb).

Now I made some progress on running the Cluster Module in Kubernetes as well as 
the single node instance with an Operator. An Operator in Kubernetes means that 
you can declaratively just add a so called “Custom Ressource” and the Operator 
in the background will take care of “running” the thing.

The crucial part is that I needed to do two minor code changes to IoTDB to get 
it running:

* No IP resolution at startup
* if the string "hostname" is set as "internal_ip" in iotdb-cluster.properties 
it will resolve the hostname at startup (FQDN) and set as internal_ip parameter

Although the latter is only for comfort and could also be solved differently.
Both changes are in the branch 
https://github.com/apache/iotdb/commits/experimental/0.12.2-cluster-for-k8s.

Xiangdong already started a discussion about the first point: 
https://lists.apache.org/thread.html/r30d1328da57b2ac22792320e37e9cefa2a6127d9d4c64db8b6bbe332%40%3Cdev.iotdb.apache.org%3E

You can find all my work so far and a draft Operator (that works!) in my repo 
https://github.com/JulianFeinauer/iotdb-operator.

I would be very grateful for any discussion or remarks to drive this topic 
further.

And please feel free to contact me if you have questions getting the 
infrastructure up and running!

Best
Julian

Re: [DISCUSS] anniversary celebration of Apache IoTDB to TLP

[Julian Feinauer]

Hey,

First of big thanks to Prof Wang for this gift to the project!

I would really love to have an IoTDB shirt (i have some db shirts already but 
none from the coolest tsdb...). And we should send one to Andy Pavlo then as 
well as he collects shirts from Databases.

I think it would be cool to install some kind of scholarship with the money 
like googles summer of code to motivate students to work on the project.

Just my ideas :)

But i love swag as well (cups, stickers, hoodies, shirts,...)

Julian

Holen Sie sich Outlook für Android

From: Xiangdong Huang 
Sent: Friday, August 27, 2021 7:38:49 AM
To: dev 
Subject: Re: [DISCUSS] anniversary celebration of Apache IoTDB to TLP

Hi,

I think we can create and share a document.
Anyone can add contributors (if they can provide at least one URL to
show the contribution).

Once getting a name list, we can start voting to know the quota.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Jialin Qiao  于2021年8月26日周四 上午11:41写道：
>
> Hi,
>
> Excellent! Apart from gifts such as T-shirt and moon cake, we could discuss
> about how to assign the quota.
>
> As this is the first year we do this, we could consider all contributors
> (without time limit) to IoTDB this time. Next time we could limit the
> contribution from now on.
>
> If we accept this precondition, then;
>
> - how to define the contributors.
> We could first collect the candidates.
> (1) All contributors who submitted code to IoTDB statisticsed by github are
> added by default.
> (2) Contributors that attend lectures, meetups to introduce IoTDB.
> (3) Contributors that help organize meetups of IoTDB.
>
> Then, we could get a list of contributors.
>
> The next question is: should we list their contribution? When nominating,
> we could list contributions in brief.
> When voting, I prefer only to give a list of names, which may encourage
> contributors to take part in the mail list to share their activities.
>
> - how to assign the quota.
>
> Using voting software is ok, whether anonymous or not is fine from my side.
>
>
> Thanks,
> ―
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
>
> Chao Wang  于2021年8月25日周三 上午9:10写道：
>
> > Wow!   Thanks for the donation of Prof. Jianmin Wang.
> >
> >
> > The products around the community are very good, like Mid-Autumn moon
> > cake.
> >
> >
> > Thanks!
> >
> >
> > Chao Wang
> > BONC ltd
> > ccgow...@163.com
> > On 8/24/2021 21:06，Xiangdong Huang wrote：
> > Hi Willem,
> >
> > Thanks. Yes, it is a good suggestion, I think we can do it in parallel.
> >
> > Best,
> > ---
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> > 黄向东
> > 清华大学 软件学院
> >
> > Willem Jiang  于2021年8月24日周二 下午9:04写道：
> >
> > Wo, thanks for the donation of Prof. Jianmin Wang.
> >
> > I think it could be meaningful if we can send a T-shirt of IoTDB with
> > the logo of the anniversary celebration to the contributor instead of
> > just send out the money.
> > In this way we can get touch with all kinds of contributor.
> >
> > Just my 2 cents.
> >
> > Willem Jiang
> >
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> > On Tue, Aug 24, 2021 at 8:32 PM Xiangdong Huang 
> > wrote:
> >
> > Hi all,
> >
> > IoTDB graduated from the incubator last year.
> > And next month, it is the first anniversary of Apache IoTDB (Apache
> > announced IoTDB's graduation on 23th, Sep, 2020).
> >
> > This year, we have:
> > - 1277 discussions on the mailing list.
> > - 740 Jira issues
> > - 272 Github issues
> > - 1767 pull requests
> > - we enabled Github Discussion, created Apache IoTDB Slack, etc..
> > - 3 major versions are released
> >
> > In total, we have:
> > - 138 contributors that contributing codes (statistics by Github)
> > -  24 PMCs and 41 Apache committers
> > - GitHub starts: 1.5k
> >
> > We can see the community grows larger and larger. The achievement
> > belongs to everyone in the community, including the committers, but
> > also all users, evangelists, and mentors, etc..
> >
> > Thank everyone for your contributions.
> >
> > As you know, Apache IoTDB is initially donated from Tsinghua University in
> > 2018.
> > After a short discussion with Prof. Jianmin Wang, the dean of the
> > School of Software, Tsinghua University, he decides to donate 100K
> > Chinese Yuans to the contributors in the community, as the gift of
> > Apache IoTDB's anniversary celebration. Everyone who is in the
> > community can get a part of the money (With regard to how to pay
> > people around the world, let's put it aside now). Maybe it is not too
> > much, but I think it will be very meaningful.
> >
> > So, I'd like to start a discussion, to decide,
> > - how to define the contributors.
> > Maybe: All persons that did one of the following things: submit
> > PRs, have public talks to introduce IoTDB, other ways to 

Re: discuss why we have to replace hostname to ip in the cluster module

[Julian Feinauer]

+1 from me (stumbled across this while setup of cluster mode in k8s) i think we 
could leave the resolution or dns caching to the underlying os.

Julian

Holen Sie sich Outlook für Android

From: Xiangdong Huang 
Sent: Friday, August 27, 2021 7:30:22 AM
To: dev 
Subject: Re: discuss why we have to replace hostname to ip in the cluster module

yes, my viewpoint is the same, hostname or IP has no difference for IoTDB.

The core is, we need to use node identifiers to  distinguish nodes.

Best,

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Houliang Qi  于2021年8月27日周五 下午1:23写道：
>
> Hi, Xiangdong
>
>
> > I do not think we will abandon using IP. Either using IP or hostname 
> > depends on users. If they are sure they have a good DNS, they can use DNS; 
> > If they like IP or the nodes do not set the DNS, they can use IP. If they 
> > like, they can also mix-use the hostname and IP addresses. We do not need 
> > to care…..
>
>
> As mentioned above, since users can use either IP or hostname, I don't 
> realize what the focus of this discussion is. I think the focus is on what 
> the user manual is exposed to the user? What are the functions provided? As 
> for the implementation of the code, I think it is OK to use hostname or IP.
>
>
> In the k8s cluster, the hostname of some pods is also changed. If we only 
> replace IP with hostname, it will not make our iotdb run perfectly on k8s. 
> Therefore, I think it is not important to use hostname or IP. The important 
> thing is to realize the function of automatic service maintenance that can be 
> deployed on k8s.
>
>
> Thanks,
> ---
> Houliang Qi
> BONC, Ltd
> On 08/26/2021 18:29，Xiangdong Huang wrote：
> Hi,
>
> DNS request is an additional cost in nodes communication.
>
> I do not think we will abandon using IP. Either using IP or hostname
> depends on users.
> If they are sure they have a good DNS, they can use DNS;
> If they like IP or the nodes do not set the DNS, they can use IP.
> If they like, they can also mix-use the hostname and IP addresses. We
> do not need to care.
>
> We should make the relationship between node IP, node hostname and node 
> identifier be clear.
>
> IMO, node identifier is the only option.
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
> Eric Pai  于2021年8月26日周四 下午4:46写道：
>
> Hi, Xiangdong,
>
> Supporting hostname is really a good feature! However, we should consider 
> these cons carefully:
>
> 1. DNS request is an additional cost in nodes communication. In cloud 
> environment, the latency may be as large as hundreds of milliseconds(e.g. 
> across available regions), and local DNS cache is also an issue in some 
> extreme scenarios.
> 2. We should make the relationship between node IP, node hostname and node 
> identifier be clear. A node should not be treated as a new one if it changes 
> its IP or hostname. Or allowing to change IP only.
>
> 在 2021/8/25 下午11:15，“Xiangdong Huang” 写入:
>
> Hi,
>
> Today Julian and I discussed some logic of the cluster module.
> Both of us feel it strange that we replace the hostname to IP before a
> cluster node starts up.
> I think it is not a very common way.
>
> hostname sometimes is very very helpful. For example, in K8S
> environment, a node's hostname never changes, but its IP may change.
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>

AW: Do we need a new Query Engine?

[Julian Feinauer]

Hi,

sorry to be so unspecific. Prometheus has quite a nice Documentation as 
reference [1].

Thinks I like there are for example:

Math on Timeseries:
* timeseriesA +(, -, *, /) timeseriesB as a new series

Reuse Aggregations as new timeseries:
Avg(timeseriesA)/avg(timeseriesB) in a group by context

Best
Julian

[1] https://prometheus.io/docs/prometheus/latest/querying/basics/


Von: Xiangdong Huang 
Datum: Dienstag, 23. Februar 2021 um 09:51
An: dev 
Betreff: Re: Do we need a new Query Engine?
Hi,

Welcome to more discussion about this topic.

>  I really love there is the power of their query language. Lots of
functions and the possibility to do math on timeseries

Sharing some examples may be the first step to open our mind :D

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Julian Feinauer  于2021年2月23日周二 上午4:36写道：

> Hi folks,
>
> the title is a bit drastic but what I wanted to bring up as a discussion
> is what we plan to allow in our Query Language.
>
> I worked a bit with Prometheus recently and what I really love there is
> the power of their query language. Lots of functions and the possibility to
> do math on timeseries. So not only scalar and timeseries but also
> timeseries and timeseries.
>
> When I though about how to integrate that seamlessly into IoTDB I
> questioned myself if the current state of the “Query Engine” or “Execution
> Engine” is setup right for that or if there would be the need to “rewrite”
> it.
> My gut feeling is that it started off well and just grew over and over
> with many more features and branches for all the different options.
>
> What are other peoples feelings?
> I really would love to have some more powerful math!
>
> Julian
>

Do we need a new Query Engine?

[Julian Feinauer]

Hi folks,

the title is a bit drastic but what I wanted to bring up as a discussion is 
what we plan to allow in our Query Language.

I worked a bit with Prometheus recently and what I really love there is the 
power of their query language. Lots of functions and the possibility to do math 
on timeseries. So not only scalar and timeseries but also timeseries and 
timeseries.

When I though about how to integrate that seamlessly into IoTDB I questioned 
myself if the current state of the “Query Engine” or “Execution Engine” is 
setup right for that or if there would be the need to “rewrite” it.
My gut feeling is that it started off well and just grew over and over with 
many more features and branches for all the different options.

What are other peoples feelings?
I really would love to have some more powerful math!

Julian

Re: Update for Python Lib for 0.10.1

[Julian Feinauer]

Hi Xiangdong,

thanks for your work.
Indeed your module looks good now (init.py everywhere and such things).

I will also add the Support for python-testcontainers in the official code as 
ist quite neat for tests.

Julian

Am 26.11.20, 11:09 schrieb "Xiangdong Huang" :

By the way, tests are not contained in the PR.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Xiangdong Huang  于2020年11月26日周四 下午6:07写道：

> Hi Julian,
>
> Thanks!
> Actually, I occur some problems when release iotdb-py-client 0.11.0 to
> pypi.
>
> After checking your modifications and under the help of Kaifeng Xue,
> I think I fixed them. see PR[1] and [2].
>
> (I uniform the package name as `iotdb`)
>
> [1] https://github.com/apache/iotdb/pull/2127
> [2] https://github.com/apache/iotdb/pull/2128
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
    >
>  黄向东
> 清华大学 软件学院
>
>
> Julian Feinauer  于2020年11月26日周四 上午3:54写道：
>
>> I would like to merge that soon into iotdb.
>>
>> But yes, generally it should be.
>>
>> Julian
>>
>> Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
>>
>> 
>> From: Xiangdong Huang 
>> Sent: Wednesday, November 25, 2020 4:10:30 PM
>> To: dev 
>> Subject: Re: Update for Python Lib for 0.10.1
>>
>> Hi,
>>
>> Is that possible to enable it on Github Action?
>>
>> ---
>> Xiangdong Huang
>> School of Software, Tsinghua University
>>
>>  黄向东
>> 清华大学 软件学院
>>
>>
>> Giorgio Zoppi  于2020年11月25日周三 下午9:33写道：
>>
>> > Hey Julian,
>> > do you have as well a github to share?
>> > Best Regards,
>> > Giorgio
>> >
>>
>

Update for Python Lib for 0.10.1

[Julian Feinauer]

Hi Folks,

as already discussed before I made a small pypi package to make it easier for 
us to use IoTDB in Python with Release 0.10.1 (as the python stuff there was 
not straightforward to include).
I recently pushed a new version with support fort he awesome Python 
Testcontainers Project: 

 
https://testcontainers-python.readthedocs.io/en/latest/index.html

You can now do a (Docker) based IoTDB Test like that:

class MyTestCase(unittest.TestCase):

def test_something(self):
with IoTDBContainer() as c:
session = Session('localhost', c.get_exposed_port(6667), 'root', 
'root')
session.open(False)
result = session.execute_query_statement("SHOW TIMESERIES")
print(result)
session.close()

This will start a IoTDB Container and call your code as soon as ist ready (and 
take it down afterwards).

You find the lib here: https://pypi.org/project/iotdb-session-0.10.1/
And the Code is here: https://github.com/JulianFeinauer/iotdb-session-py

Best
Julian

Re: grafana plugin

[Julian Feinauer]

Hi Boris,

this looks awesome (at least the screenshots).
I will have to try a look with a translator as you know 我聽不懂中文

Best
Julian

Am 20.11.20, 09:07 schrieb "Boris Zhu" :

Hi all


here is the design of the new grafana plugin. Here is the doc link [1].


[1]https://cwiki.apache.org/confluence/display/IOTDB/Grafana+plugin+design

Thanks,
Tianci Zhu

Re: A trial of compiling IoTDB to native binary image

[Julian Feinauer]

Wow, this is awesome news!
Is your setup based on graalvm?
I tried the same with plc4x but had issues with netty and its heavy usage of 
reflections.

Julian

Holen Sie sich Outlook für Android


From: Xiangdong Huang 
Sent: Wednesday, November 11, 2020 4:06:52 AM
To: dev 
Subject: A trial of compiling IoTDB to native binary image

Hi,

Just an update about my recent experiment.

I used LLVM to compile IoTDB to a native image on Mac and got a
native-image (51MB size).
The image can be run directly without JRE.

I have not tested the performance and the stability, but I think it may be
a choice to use IoTDB on those which have no JRE.
(Not sure whether we can do it on ARM architectures. If we could, it will
be an amazing thing).

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Re: Embdedded IoTDB Grafana Endpoint

[Julian Feinauer]

Hey,

yes, to do it clean we could do it directly without Spring. Then ist just a 
very small handler.
Similar to what we already use optionally in the prometheus module.

I dislike the idea of having spring in the server as well.

Julian

Am 06.11.20, 12:03 schrieb "Xiangdong Huang" :

Hi,

> what do you think of an option to run the SimpleJson Grafana Connector
directly in IoTDBs Server Context (could be disbaled by default).
it is ok.

But I think we need to prune some dependencies of Spring... It it too large
now.. (I remember it is ~ 20MB to add such a module)

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Julian Feinauer  于2020年11月6日周五 下午6:13写道：

> Hi folks,
>
> as we just discussed the prometheus integration, what do you think of an
> option to run the SimpleJson Grafana Connector directly in IoTDBs Server
> Context (could be disbaled by default).
> This would make it way easier for users to just „look“ aat their IoTDB
> Data.
>
> WDYT?
>
> Julian
>

Embdedded IoTDB Grafana Endpoint

[Julian Feinauer]

Hi folks,

as we just discussed the prometheus integration, what do you think of an option 
to run the SimpleJson Grafana Connector directly in IoTDBs Server Context 
(could be disbaled by default).
This would make it way easier for users to just „look“ aat their IoTDB Data.

WDYT?

Julian

Re: Grafana Docker

[Julian Feinauer]

Hi folks,

Simon and I sat together and released such a container yesterday as 
jfeinauer/iotdb-grafana:0.10.1.
Just in case anybody needs one.

When our docker setup is final and well working I hope we can provide one or 
two PRs.

Julian

Am 05.11.20, 03:30 schrieb "Xiangdong Huang" :

Hi Simon,

Indeed we did not release iotdb-grafana-connector on DockerHub..

It is easy to do such an image, as it is just a Spring application.

Let me know if you have any questions.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


    Julian Feinauer  于2020年11月4日周三 下午8:14写道：

> Hi Simon,
>
> gratulations to your first mail here : )
> Indeed it is quite simple and there also are docker containers somewhere
> from Xiangdong or myself even.
> Just do it exactly similar to install the plugin.
>
> Let me see if I find my dockerfiles somewhere.
>
> Best
> Julian
>
> Von: Simon Mayer 
> Antworten an: "dev@iotdb.apache.org" 
> Datum: Mittwoch, 4. November 2020 um 13:11
> An: "dev@iotdb.apache.org" 
> Betreff: WG: Grafana Docker
>
>
> Hi Folks,
> Some of you maybe know me.
> Im a colleague of Julian. I'm using IOTB Docker and want to connect
> Grafana to it.
> I've found an manual for it but only on local computer not for Docker
> Containers. Is it the same if I want to conenct an Grafana Docker to
> Datasource IOTDB Docker?
> By the way I guess there is an letter missing in Attachmend you can see 
it.
> It should be iotdb/grafana/src/main/java/org/apache/iotdb not
> iotdb/grafana/rc/main/java/org/apache/iotdb
> found at https://github.com/apache/iotdb/tree/master/grafana
>
>
> Softwareentwicklung
>
> T  +49 7022 2493301
>
> E  s<mailto:f.koe...@pragmaticminds.de>.ma...@pragmaticminds.de
>
>
>
> pragmatic minds GmbH
>
> Jesingerstraße 57
>
> 73230 Kirchheim unter Teck
>
> www.pragmaticminds.de<http://www.pragmaticminds.de/>
>
>
>
> pragmatic minds GmbH, Sitz: Kirchheim unter Teck, Amtsgericht Stuttgart -
> HRB 757136, Geschäftsführer: Dr. Julian Feinauer
>
>

Re: Grafana Docker

[Julian Feinauer]

Hi Simon,

gratulations to your first mail here : )
Indeed it is quite simple and there also are docker containers somewhere from 
Xiangdong or myself even.
Just do it exactly similar to install the plugin.

Let me see if I find my dockerfiles somewhere.

Best
Julian

Von: Simon Mayer 
Antworten an: "dev@iotdb.apache.org" 
Datum: Mittwoch, 4. November 2020 um 13:11
An: "dev@iotdb.apache.org" 
Betreff: WG: Grafana Docker


Hi Folks,
Some of you maybe know me.
Im a colleague of Julian. I'm using IOTB Docker and want to connect Grafana to 
it.
I've found an manual for it but only on local computer not for Docker 
Containers. Is it the same if I want to conenct an Grafana Docker to Datasource 
IOTDB Docker?
By the way I guess there is an letter missing in Attachmend you can see it.
It should be iotdb/grafana/src/main/java/org/apache/iotdb not 
iotdb/grafana/rc/main/java/org/apache/iotdb
found at https://github.com/apache/iotdb/tree/master/grafana


Softwareentwicklung

T  +49 7022 2493301

E  s<mailto:f.koe...@pragmaticminds.de>.ma...@pragmaticminds.de



pragmatic minds GmbH

Jesingerstraße 57

73230 Kirchheim unter Teck

www.pragmaticminds.de<http://www.pragmaticminds.de/>



pragmatic minds GmbH, Sitz: Kirchheim unter Teck, Amtsgericht Stuttgart - HRB 
757136, Geschäftsführer: Dr. Julian Feinauer

Re: Refactor Docker Package

[Julian Feinauer]

Oh, thank you Xiangdong <3

Will have a look later : )

Julian

Am 03.11.20, 20:56 schrieb "Xiangdong Huang" :

Hi Julian,

I copied your package structure and use maven to generate it.

see https://github.com/apache/iotdb/pull/1939

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Julian Feinauer  于2020年10月26日周一 下午9:28写道：

> Hi,
>
> in fact so far I only worked in a separate github to provide it fort he
> old 0.10.1 release.
>
> But ideally we would also do it in the release phase or at least very
> automated (again, see my repo there is a .sh to make the release):
> https://github.com/JulianFeinauer/iotdb-session-py
>
> Julian
>
> Am 26.10.20, 14:19 schrieb "Xiangdong Huang" :
>
> Hi Julian,
>
> Nice, it is great if someone can maintain the python module in the
> community.
>
> Does your branch support calling uploading the python module to Pip
> when
> calling `mvn release:perform`?
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
> Julian Feinauer  于2020年10月26日周一
> 下午8:56写道：
>
> > Hi,
> >
> > short update.
> > I uploaded my project also to pypi to make it easy to use Python 
with
> > IoTDB Release 0.10.1.
> > You can find the package here:
> > https://pypi.org/project/iotdb-session-0.10.1/
> >
> > A small example is attached.
> >
> > Julian
> >
> > Am 26.10.20, 11:42 schrieb "Julian Feinauer" <
> > j.feina...@pragmaticminds.de>:
> >
> > Hi folks,
> >
> > currently the Python support (in 0.10.1) is... well...
> improveable : )
> >
> > As we started to work with IoTDB in Python I would like to
> refactor
> > and improve the package in the master branch for the next release.
> > I already created an example (based on 0.10.1) branch which I
> just
> > pushed to github here:
> https://github.com/JulianFeinauer/iotdb-session-py
> >
> > The shell script shows how to build the package.
> >
> > I would like to automate the setup oft he package with maven,
> then we
> > could push it way easier to pypi.
> >
> > Note, I had to name the package iotdb_session as the current
> package
> > (thrift) is already called iotdb.
> >
> > WDYT? Is anyone working at this at the moment?
> >
> > Best
> > Julian
> >
> >
>
>

Re: back and considering releasing v0.11

[Julian Feinauer]

Haha, thats a tough Shedule.
Then do the release and we will do the python pip release later __

Julian

Am 03.11.20, 09:11 schrieb "Jialin Qiao" :

Hi,

Today :D

--
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

 -原始邮件-
 发件人: "Julian Feinauer" 
 发送时间: 2020-11-03 16:02:50 (星期二)
 收件人: "dev@iotdb.apache.org" 
 抄送: 
 主题: Re: back and considering releasing v0.11
 
 Hi,
 
 +1 for Release 0.11 ASAP but I probably like to do the python 
"makeover" for it as the current version in 0.10.1 is a bit complicated to use.
 
 When do you like to do the release?
 
 Julian
 
 Am 02.11.20, 11:24 schrieb "Xiangwei Wei" :
 
 Hi,
 
 
 +1 :D
 
 Jialin Qiao  于2020年11月1日周日 下午8:03写道：
 
  Hi,
 
  Enable level compaction by default:
  https://github.com/apache/iotdb/pull/1841
  New memory control: https://github.com/apache/iotdb/pull/1524
 
  Thanks,
  --
  Jialin Qiao
  School of Software, Tsinghua University
 
  乔嘉林
  清华大学 软件学院
 
   -原始邮件-
   发件人: "Xiangdong Huang" 
   发送时间: 2020-11-01 02:09:59 (星期日)
   收件人: dev 
   抄送:
   主题: Re: back and considering releasing v0.11
  
   Hi,
  
   If so, I'd like to know how many PR that can be merged 
before we
  freeze the
   v0.11 branch.
  
   Can someone help me to list that?
  
   Best,
   ---
   Xiangdong Huang
   School of Software, Tsinghua University
  
黄向东
   清华大学 软件学院
  
  
   Haonan Hou  于2020年10月31日周六 下午4:19写道：
  
+1
   
Thanks,
Haonan Hou
   
 On Oct 31, 2020, at 4:15 PM, 孙泽嵩 
  sz...@mails.tsinghua.edu.cn wrote:

 +1 for releasing 0.11


 Best,
 ---
 Zesong Sun
 School of Software, Tsinghua University

 孙泽嵩
 清华大学 软件学院

 2020年10月31日 16:11，Jialin Qiao 
  qj...@mails.tsinghua.edu.cn 写道：

 Hi,

 Welcome back!
 +1 for releasing 0.11 asap.

 Thanks,
 --
 Jialin Qiao
 School of Software, Tsinghua University

 乔嘉林
 清华大学 软件学院

  -原始邮件-
  发件人: "Xiangdong Huang" 

  发送时间: 2020-10-31 15:59:28 (星期六)
  收件人: dev 
  抄送:
  主题: back and considering releasing 
v0.11
 
  Hi,
 
  I almost disappeared for about 2 
weeks because of
  busy work...
  Glad to see there are so many new 
bugfix/features
  that involved
these days.
 
  From now on I will be back and spend 
my most of
  the time to the
community
  and the project.
 
  I checked the goal of v0.11 [1], many 
of them have
  been done but
some of
  them are not.
  However, there is a long time that we 
did not
  release a new major
version.
 
  Shall we move some of the tasks in 
[1] to the next
  version, and
release
  v0.11 asap?
 
  How do you think?
 
  [1]
  https://cwiki.apache.org/confluence/display/IOTDB/v0.11
   ; 
   
  Best.
  ---
  Xiangdong Huang
  School of Software, Tsinghua 
University
 
   黄向东
  清华大学 软件学院
 


   
   
  
  
hhao...@outlook.com
 
 
 
 -- 
 Best,
 Xiangwei Wei
 

Re: back and considering releasing v0.11

[Julian Feinauer]

Hi,

+1 for Release 0.11 ASAP but I probably like to do the python "makeover" for it 
as the current version in 0.10.1 is a bit complicated to use.

When do you like to do the release?

Julian

Am 02.11.20, 11:24 schrieb "Xiangwei Wei" :

Hi,


+1 :D

Jialin Qiao  于2020年11月1日周日 下午8:03写道：

> Hi,
>
> Enable level compaction by default:
> https://github.com/apache/iotdb/pull/1841
> New memory control: https://github.com/apache/iotdb/pull/1524
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
>  -原始邮件-
>  发件人: "Xiangdong Huang" 
>  发送时间: 2020-11-01 02:09:59 (星期日)
>  收件人: dev 
>  抄送:
>  主题: Re: back and considering releasing v0.11
> 
>  Hi,
> 
>  If so, I'd like to know how many PR that can be merged before we
> freeze the
>  v0.11 branch.
> 
>  Can someone help me to list that?
> 
>  Best,
>  ---
>  Xiangdong Huang
>  School of Software, Tsinghua University
> 
>   黄向东
>  清华大学 软件学院
> 
> 
>  Haonan Hou  于2020年10月31日周六 下午4:19写道：
> 
>   +1
>  
>   Thanks,
>   Haonan Hou
>  
>On Oct 31, 2020, at 4:15 PM, 孙泽嵩 <
> sz...@mails.tsinghua.edu.cn> wrote:
>   
>+1 for releasing 0.11
>   
>   
>Best,
>---
>Zesong Sun
>School of Software, Tsinghua University
>   
>孙泽嵩
>清华大学 软件学院
>   
>2020年10月31日 16:11，Jialin Qiao <
> qj...@mails.tsinghua.edu.cn> 写道：
>   
>Hi,
>   
>Welcome back!
>+1 for releasing 0.11 asap.
>   
>Thanks,
>--
>Jialin Qiao
>School of Software, Tsinghua University
>   
>乔嘉林
>清华大学 软件学院
>   
> -原始邮件-
> 发件人: "Xiangdong Huang" 
> 发送时间: 2020-10-31 15:59:28 (星期六)
> 收件人: dev 
> 抄送:
> 主题: back and considering releasing v0.11
>
> Hi,
>
> I almost disappeared for about 2 weeks because of
> busy work...
> Glad to see there are so many new bugfix/features
> that involved
>   these days.
>
> From now on I will be back and spend my most of
> the time to the
>   community
> and the project.
>
> I checked the goal of v0.11 [1], many of them have
> been done but
>   some of
> them are not.
> However, there is a long time that we did not
> release a new major
>   version.
>
> Shall we move some of the tasks in [1] to the next
> version, and
>   release
> v0.11 asap?
>
> How do you think?
>
> [1]
> https://cwiki.apache.org/confluence/display/IOTDB/v0.11
>  ; 
>  
> Best.
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>   
>  
>  
>  hhao...@outlook.com>



-- 
Best,
Xiangwei Wei

Re: About the timezone in IoTDB

[Julian Feinauer]

Hi,

very good addition, indeed.
As one of the first users "far away" from China it was always a bit of math for 
me to get everything together with default settings (UTC+8?) : )

Julian

Am 30.10.20, 02:59 schrieb "Xiangwei Wei" :

In the previous design, the timezone was set in IoTDB server configuration.
When one client gets the connection from the server,* it will keep the same
timezone as server.*

However, since IoTDB is becoming more and more international. Some user may
encounter problems, as their clients are in different timezone from their
servers, and muti clients may be in different timezone too, which lead to
that the time input by format in inserting or querying will be parsed to
the incorrect timestamp. And more, the time result show in client will be
incorrect too.

Actually, the design self before was unreasonable. *The timezone of server
is not important,* *which is only responsible for storing timestamp. On the
contrary, the client is the key*. Because how to parse the formatted time
submitted by client should be decided by the timezone of client itself.

For example, my client is in +08:00, while the server is in +01:00. Insert
a datapoint, "inser into root.sg.d1(time, s1) values (2020-10-30 09:48,
948)". The server will parse the formatted time by +08:00 instead of +01:00.

Therefore, I modified the timezone configuration in IoTDB. *Delete the
timezone configuration in server, and add a timezone parameter while the
client request a connection from server, which will be stored by server as
a client -> timezone map. *Then the server can handle the request from
every client correctly.


You can check it in PR[1]. Thank you :D

[1] https://github.com/apache/iotdb/pull/1846


-- 
Best,
Xiangwei Wei

Re: Hello, everyone. I'm Mingming Zhang

[Julian Feinauer]

Cool to hear : )

Am 27.10.20, 15:11 schrieb "pengwa...@fudan.edu.cn.INVALID" 
:

Hi Jialin,

Thanks for your information.

I just talked with Xiangdong, and we will discuss the index in detail a few 
days later.

Best,

Peng



pengwa...@fudan.edu.cn

From: Jialin Qiao
Date: 2020-10-27 21:43
To: dev
Subject: Re: Hello, everyone. I'm Mingming Zhang
Hi,

As far as I know, Rong Kang is working on an index framework in IoTDB.
However, this module is not ready now...

Thanks,
--
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

> -原始邮件-
> 发件人: "Julian Feinauer" 
> 发送时间: 2020-10-27 15:49:08 (星期二)
> 收件人: "dev@iotdb.apache.org" 
> 抄送: 
> 主题: Re: Hello, everyone. I'm Mingming Zhang
> 
> Hi Mingming,
> 
> cool to have you here and looking forward to the features you may bring : 
)
> 
> @Jialin: Do we have a general setup for using indexes in IoTDB?
> 
> Julian
> 
> Am 27.10.20, 05:52 schrieb "Jialin Qiao" :
> 
> Hi Mingming,
> 
> Welcome! Similarity search is very important in the IoT scenario.
> 
> You may need to bring an index into IoTDB, @Rong Kang is familiar 
with this. He may offer some help :)
> 
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院
> 
> > -原始邮件-
> > 发件人: "Xiangwei Wei" 
> > 发送时间: 2020-10-27 09:32:03 (星期二)
> > 收件人: dev@iotdb.apache.org
> > 抄送: 
> > 主题: Re: Hello, everyone. I'm Mingming Zhang
> > 
> > Hi, Mingming
> > 
> > Welcome to IoTDB!!
> > 
> > You can see the user guide from [1], and system design doc from[2].
> > 
> > [1]
> > 
http://iotdb.apache.org/zh/UserGuide/Master/Get%20Started/QuickStart.html
> > [2] 
http://iotdb.apache.org/zh/SystemDesign/Architecture/Architecture.html
> > 
> > Our issue is managed by jira[3]. You could find some easy-fixed 
issues to
> > start up :)
> > 
> > [3]
> > 
https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-869?filter=allopenissues
> > 
> > 
> > Mingming Zhang  于2020年10月25日周日 
下午8:24写道：
> > 
> > > Hello, everyone ~
> > >
> > >  I’m  Mingming Zhang, a postgraduate from Fudan University. 
It’s a
> > > pleasure to join the IoTDB community and cooperate with you to 
prosper it.
> > >
> > >  My major is software engineering，and  focus on the time 
series data
> > > mining for more than one year. Luckily, IoTDB is in line with my 
research
> > > fields and I ‘m very interested on it ,so I‘d like to make some
> > > contribution to the algorithm design of the time series 
similarity search
> > > and implement it into the IoTDB, I hope I can apply the theories 
I have
> > > learned to the real industrial IoT field, and do some meaningful 
results
> > > for our community.
> > >
> > >  Finally, I‘m appreciate it if you can give me more valuable 
advice,
> > > which will help me a lot !
> > >
> > > Best regards，
> > > Mingming Zhang
> > >
> > 
> > 
> > -- 
> > Best,
> > Xiangwei Wei
> 

Re: Hello, everyone. I'm Mingming Zhang

[Julian Feinauer]

Hi Mingming,

cool to have you here and looking forward to the features you may bring : )

@Jialin: Do we have a general setup for using indexes in IoTDB?

Julian

Am 27.10.20, 05:52 schrieb "Jialin Qiao" :

Hi Mingming,

Welcome! Similarity search is very important in the IoT scenario.

You may need to bring an index into IoTDB, @Rong Kang is familiar with 
this. He may offer some help :)

Thanks,
--
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

> -原始邮件-
> 发件人: "Xiangwei Wei" 
> 发送时间: 2020-10-27 09:32:03 (星期二)
> 收件人: dev@iotdb.apache.org
> 抄送: 
> 主题: Re: Hello, everyone. I'm Mingming Zhang
> 
> Hi, Mingming
> 
> Welcome to IoTDB!!
> 
> You can see the user guide from [1], and system design doc from[2].
> 
> [1]
> http://iotdb.apache.org/zh/UserGuide/Master/Get%20Started/QuickStart.html
> [2] http://iotdb.apache.org/zh/SystemDesign/Architecture/Architecture.html
> 
> Our issue is managed by jira[3]. You could find some easy-fixed issues to
> start up :)
> 
> [3]
> 
https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-869?filter=allopenissues
> 
> 
> Mingming Zhang  于2020年10月25日周日 下午8:24写道：
> 
> > Hello, everyone ~
> >
> >  I’m  Mingming Zhang, a postgraduate from Fudan University. It’s a
> > pleasure to join the IoTDB community and cooperate with you to prosper 
it.
> >
> >  My major is software engineering，and  focus on the time series data
> > mining for more than one year. Luckily, IoTDB is in line with my 
research
> > fields and I ‘m very interested on it ,so I‘d like to make some
> > contribution to the algorithm design of the time series similarity 
search
> > and implement it into the IoTDB, I hope I can apply the theories I have
> > learned to the real industrial IoT field, and do some meaningful results
> > for our community.
> >
> >  Finally, I‘m appreciate it if you can give me more valuable advice,
> > which will help me a lot !
> >
> > Best regards，
> > Mingming Zhang
> >
> 
> 
> -- 
> Best,
> Xiangwei Wei

Re: Refactor Docker Package

[Julian Feinauer]

Hi,

in fact so far I only worked in a separate github to provide it fort he old 
0.10.1 release.

But ideally we would also do it in the release phase or at least very automated 
(again, see my repo there is a .sh to make the release): 
https://github.com/JulianFeinauer/iotdb-session-py

Julian

Am 26.10.20, 14:19 schrieb "Xiangdong Huang" :

Hi Julian,

Nice, it is great if someone can maintain the python module in the
community.

Does your branch support calling uploading the python module to Pip when
calling `mvn release:perform`?

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


    Julian Feinauer  于2020年10月26日周一 下午8:56写道：

> Hi,
>
> short update.
> I uploaded my project also to pypi to make it easy to use Python with
> IoTDB Release 0.10.1.
> You can find the package here:
> https://pypi.org/project/iotdb-session-0.10.1/
>
> A small example is attached.
>
> Julian
    >
    > Am 26.10.20, 11:42 schrieb "Julian Feinauer" <
> j.feina...@pragmaticminds.de>:
>
> Hi folks,
>
> currently the Python support (in 0.10.1) is... well... improveable : )
>
> As we started to work with IoTDB in Python I would like to refactor
> and improve the package in the master branch for the next release.
> I already created an example (based on 0.10.1) branch which I just
> pushed to github here: https://github.com/JulianFeinauer/iotdb-session-py
>
> The shell script shows how to build the package.
>
> I would like to automate the setup oft he package with maven, then we
> could push it way easier to pypi.
>
> Note, I had to name the package iotdb_session as the current package
> (thrift) is already called iotdb.
>
> WDYT? Is anyone working at this at the moment?
>
> Best
> Julian
>
>

Re: Refactor Docker Package

[Julian Feinauer]

Hi,

short update. 
I uploaded my project also to pypi to make it easy to use Python with IoTDB 
Release 0.10.1.
You can find the package here: https://pypi.org/project/iotdb-session-0.10.1/

A small example is attached.

Julian

Am 26.10.20, 11:42 schrieb "Julian Feinauer" :

Hi folks,

currently the Python support (in 0.10.1) is... well... improveable : )

As we started to work with IoTDB in Python I would like to refactor and 
improve the package in the master branch for the next release.
I already created an example (based on 0.10.1) branch which I just pushed 
to github here: https://github.com/JulianFeinauer/iotdb-session-py

The shell script shows how to build the package.

I would like to automate the setup oft he package with maven, then we could 
push it way easier to pypi.

Note, I had to name the package iotdb_session as the current package 
(thrift) is already called iotdb.

WDYT? Is anyone working at this at the moment?

Best
Julian

Refactor Docker Package

[Julian Feinauer]

Hi folks,

currently the Python support (in 0.10.1) is... well... improveable : )

As we started to work with IoTDB in Python I would like to refactor and improve 
the package in the master branch for the next release.
I already created an example (based on 0.10.1) branch which I just pushed to 
github here: https://github.com/JulianFeinauer/iotdb-session-py

The shell script shows how to build the package.

I would like to automate the setup oft he package with maven, then we could 
push it way easier to pypi.

Note, I had to name the package iotdb_session as the current package (thrift) 
is already called iotdb.

WDYT? Is anyone working at this at the moment?

Best
Julian

Re: [DISCUSS] Should we use spark-tsfile or spark-iotdb

[Julian Feinauer]

Hi,

that is very interesting results and indeed I would opt for using iotdb by 
default.
My vision is to enable advanced query optimization and indexing as it is known 
by RDBMS.
Then this could be faster by orders of magnitude if the database has a suitable 
index.

Just my 2 ct : )

Julian

Am 16.10.20, 04:46 schrieb "南京大学软件学院薛恺丰" <827011...@qq.com>:

Hi everyone,
I have done some test about spark-tsfile and spark-iotdb, the result is 
at:https://cwiki.apache.org/confluence/display/IOTDB/Spark-tsfile+and+Spark-iotdb+comparasion


What conclusion we can get from these result?


I think the reason is iotdb query process is paralleled, but tsfile are 
not. And we may not optimize tsfile query process frequently in future. Should 
we just use spark-iotdb in future?


Best Regards,
Kaifeng Xue

Re: Support SDT compression

[Julian Feinauer]

Thanks!

I think it cold be a very cool index. Due to linearity it would be pretty easy 
to translate  regular queries to queries on the sdt signal which had way lower 
data points.

Or in some sense this is a visualisation optimized version of the time series.

Really cool feature!

Julian

Holen Sie sich Outlook für Android<https://aka.ms/ghei36>


From: Jialin Qiao 
Sent: Saturday, October 3, 2020 5:04:20 AM
To: dev@iotdb.apache.org 
Subject: Re: Support SDT compression

Hi,

Yes, it's lossy. Users need to config the tolerant error.

Thanks,
--
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

> -原始邮件-
> 发件人: "Julian Feinauer" 
> 发送时间: 2020-10-03 02:09:44 (星期六)
> 收件人: "dev@iotdb.apache.org" 
> 抄送:
> 主题: Re: Support SDT compression
>
> Hi,
>
> I read the document (which is excellent, good work!) and it sounds very 
> interesting but as far as I understand the algorithm its lossy.
> Is this true?
> Or do I miss something?
>
> Thanks!
> Julian
>
> Am 29.09.20, 12:04 schrieb "Jialin Qiao" :
>
> Hi,
>
> Good summary~
>
> > Page header needs to maintain a Map
>
> It's better to keep the structure of PageHeader the same as 0.10.
> You can store this information in the PageData.
>
> > Encoder, Decoder will take both a data column and a series.
>
> Try to provide interface with primitive data type. So a series should be 
> put by encodeFloatPoint(long time, float value)
>
> Besides, there are some more details need to consider:
>
> - How to store the endpoint of each segment
> - In the SDT with timestamp encoding, how to store the timestamps? (Maybe 
> just using TS2_DIFF is fine)
>
> This is not a small change to IoTDB...
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -原始邮件-
> > 发件人: "Haimei Guo" 
> > 发送时间: 2020-09-29 15:19:49 (星期二)
> > 收件人: dev@iotdb.apache.org
> > 抄送:
> > 主题: Re: Support SDT compression
> >
> > Hi,
> >
> > Following is a summary of SDT's encoding and decoding implementation in
> > IoTDB.
> >
> >-
> >
> >SDT is mainly to calculate the up and down slopes of the data to the
> >segment starting point. If it is within the compression deviation CD 
> range,
> >discard the data. If it exceeds the CD, the original data is stored
> >-
> >
> >In IoTDB, the SDT can act as a new Encoding method. It works inside 
> each
> >Page (PageWriter and PageReader).
> >-
> >
> >Will support with and without timestamp encoding.
> >-
> >
> >For without timestamps encoding, we will record the count of data 
> points
> >in each segment in the page header. Page header needs to maintain a
> >Map
> >
> > Encoder will be changed to encode(long time, long value)
> > Data buffer will be stored in each Encoder
> > Decoder will be changed to getTime(), getXXValue()
> > Encoder, Decoder will take both a data column and a series.
> >
> >
> > If you have any question or comment, you are more than welcome to reply!
> >
> >
> > Thank you,
> >
> > Haimei
> >
> >
> > On Mon, Sep 28, 2020 at 1:20 PM Jialin Qiao 
> 
> > wrote:
> >
> > > Hi Haimei,
> > >
> > > Good work! This doc is comprehensive :)
> > >
> > > As for the implementation in IoTDB, here are some points:
> > >
> > > (1) First, SDT could act as a new Encoding method in IoTDB. It works
> > > inside each Page (PageWriter and PageReader).
> > > (2) The interface of Encoder could be changed to encode(long time, XX
> > > value). The interface of Decoder could be change to getTime(),
> > > getXXValue(). Which is, the encoder and decoder is not only 
> responsible for
> > > one data column but a series. This involves some reconstruction of the
> > > Encoder and Decoder, the data buffers should be stored inside each 
> encoder.
> > > (3) For the SDT without timestamps, we need to record the count of 
> each
> > > segment.
> > > (4) We could offer two encodings, SDT with timestamps and SDT without
> > > timestamps.
> > >
>  

Re: Support SDT compression

[Julian Feinauer]

Hi,

I read the document (which is excellent, good work!) and it sounds very 
interesting but as far as I understand the algorithm its lossy.
Is this true?
Or do I miss something?

Thanks!
Julian

Am 29.09.20, 12:04 schrieb "Jialin Qiao" :

Hi,

Good summary~

> Page header needs to maintain a Map

It's better to keep the structure of PageHeader the same as 0.10.
You can store this information in the PageData. 

> Encoder, Decoder will take both a data column and a series.

Try to provide interface with primitive data type. So a series should be 
put by encodeFloatPoint(long time, float value)

Besides, there are some more details need to consider: 

- How to store the endpoint of each segment
- In the SDT with timestamp encoding, how to store the timestamps? (Maybe 
just using TS2_DIFF is fine)

This is not a small change to IoTDB...

Thanks,
--
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

> -原始邮件-
> 发件人: "Haimei Guo" 
> 发送时间: 2020-09-29 15:19:49 (星期二)
> 收件人: dev@iotdb.apache.org
> 抄送: 
> 主题: Re: Support SDT compression
> 
> Hi,
> 
> Following is a summary of SDT's encoding and decoding implementation in
> IoTDB.
> 
>-
> 
>SDT is mainly to calculate the up and down slopes of the data to the
>segment starting point. If it is within the compression deviation CD 
range,
>discard the data. If it exceeds the CD, the original data is stored
>-
> 
>In IoTDB, the SDT can act as a new Encoding method. It works inside 
each
>Page (PageWriter and PageReader).
>-
> 
>Will support with and without timestamp encoding.
>-
> 
>For without timestamps encoding, we will record the count of data 
points
>in each segment in the page header. Page header needs to maintain a
>Map
> 
> Encoder will be changed to encode(long time, long value)
> Data buffer will be stored in each Encoder
> Decoder will be changed to getTime(), getXXValue()
> Encoder, Decoder will take both a data column and a series.
> 
> 
> If you have any question or comment, you are more than welcome to reply!
> 
> 
> Thank you,
> 
> Haimei
> 
> 
> On Mon, Sep 28, 2020 at 1:20 PM Jialin Qiao 
> wrote:
> 
> > Hi Haimei,
> >
> > Good work! This doc is comprehensive :)
> >
> > As for the implementation in IoTDB, here are some points:
> >
> > (1) First, SDT could act as a new Encoding method in IoTDB. It works
> > inside each Page (PageWriter and PageReader).
> > (2) The interface of Encoder could be changed to encode(long time, XX
> > value). The interface of Decoder could be change to getTime(),
> > getXXValue(). Which is, the encoder and decoder is not only responsible 
for
> > one data column but a series. This involves some reconstruction of the
> > Encoder and Decoder, the data buffers should be stored inside each 
encoder.
> > (3) For the SDT without timestamps, we need to record the count of each
> > segment.
> > (4) We could offer two encodings, SDT with timestamps and SDT without
> > timestamps.
> >
> > Thanks,
> > --
> > Jialin Qiao
> > School of Software, Tsinghua University
> >
> > 乔嘉林
> > 清华大学 软件学院
> >
> > > -原始邮件-
> > > 发件人: "runhus...@foxmail.com" 
> > > 发送时间: 2020-09-25 11:56:16 (星期五)
> > > 收件人: dev 
> > > 抄送:
> > > 主题: Re: Support SDT compression
> > >
> > > Great work！
> > >
> > >
> > >
> > > Thanks.
> > >
> > > Chao Wang
> > > BONC Ltd
> > >
> > >
> > > From: Eileen Guo
> > > Date: 2020-09-25 11:47
> > > To: dev
> > > Subject: Support SDT compression
> > > Hi all,
> > >
> > > I've completed a design draft for supporting swinging door 
compression.
> > >
> > > Jira: jira SDT link
> > > <
> > 
https://issues.apache.org/jira/browse/IOTDB-890?filter=-4=assignee%20in%20(haimeiguo)%20order%20by%20created%20DESC
> > >
> > > design doc: SDT design doc link
> > > <
> > 
https://docs.google.com/document/d/1VeTwVsm4CkQSVR65bWw9pKg6gRdiDYUz0lBBhWXHl5A/edit?usp=sharing
> > >
> > >
> > > The doc explains SDT algorithm, compression and decompression process,
> > > performance tests and SDT + IoTDB implementation and usage.
> > >
> > > There is still some question about where to use this algorithm. If you
> > have
> > > any idea, welcome to comment.
> > >
> > > Thank you!
> > > Haimei Guo
> >

Re: [VOTE] Graduate Apache IoTDB as TLP

[Julian Feinauer]

+1, looking forward to it.

Its a pitty that we cannot have a graduation party alltogether : /

Julian

Am 10.09.20, 09:46 schrieb "Xiangdong Huang" :

Hi all,

I  have started a formal vote for IoTDB's graduation.

Welcome to vote on the thread (general@incubator.a.o)
with binding (IPMC) or non-binding (PPMC, committer, and contributor).

:D

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


-- Forwarded message -
发件人： Xiangdong Huang 
Date: 2020年9月10日周四 下午3:42
Subject: [VOTE] Graduate Apache IoTDB as TLP
To: 


Dear all IPMCs:

Following discussions with great support from our mentors,
committers and community members.
I would like to call for a formal VOTE for graduating Apache IoTDB
(Incubating),
as a Top Level Project.

This is a formal voting thread about Apache IoTDB's graduation, please Vote:
[ ] +1 - Recommend graduation of Apache IoTDB as a TLP
[ ]  0
[ ] -1 - Do not recommend the graduation of Apache IoTDB because...

The VOTE will open for at least 72 hours.

Apache IoTDB (Incubating) entered the incubator in Nov 2018, the community
has grown vibrantly since, with all the design, development happening on
the Apache infrastructure, the "Apache Way".

To list a few of the community's achievements,

- Apache IoTDB name search has been approved
- Accepted > 1300 PRs from 75 contributors
- Migrated developer conversations to the list at dev@iotdb.apache.org
  (more than 3000 mails, without JIRA notifications, and gitbox, are sent
by more than 160 persons)
- There are 9 versions (3 major versions) released successfully conformant
to Apache release policy by 5 release managers;
- Invited 12 new committers (all of them accepted)
- invited 4 of those new committers to join the PMC (all of them accepted)
- Our proposed PMC is diverse and consists of members from more than 10
organizations

Preparations and discussions history about the graduation:

- The maturity assessment is done [2],
- The community agreed on starting the graduation process in a formal vote
[3] (result see [4]).
And, I'd like to note that all our mentors also participated in the vote
with a positive vote!
- The community also decided about a suggestion for the initial VP, a
charter [5, 6] and the initial PMC list [7].
- We also received positive feedback from the incubator general mailing
list [8].


The draft of the resolution:

Establish the Apache IoTDB Project

   WHEREAS, the Board of Directors deems it to be in the best
   interests of the Foundation and consistent with the
   Foundation's purpose to establish a Project Management
   Committee charged with the creation and maintenance of
   open-source software, for distribution at no charge to
   the public, related to an IoT native database with high performance
   for data management and analysis, on the edge and the cloud.

   NOW, THEREFORE, BE IT RESOLVED, that a Project Management
   Committee (PMC), to be known as the "Apache IoTDB Project",
   be and hereby is established pursuant to Bylaws of the
   Foundation; and be it further

   RESOLVED, that the Apache IoTDB Project be and hereby is
   responsible for the creation and maintenance of software
   related to an IoT native database with high performance
   for data management and analysis, on the edge and the cloud.
   and be it further

   RESOLVED, that the office of "Vice President, Apache IoTDB be
   and hereby is created, the person holding such office to
   serve at the direction of the Board of Directors as the chair
   of the Apache IoTDB Project, and to have primary responsibility
   for management of the projects within the scope of
   responsibility of the Apache IoTDB Project; and be it further

   RESOLVED, that the persons listed immediately below be and
   hereby are appointed to serve as the initial members of the
   Apache IoTDB Project:

 * Chen Wang  
 * Christofer Dutz 
 * Dawei Liu 
 * Gaofei Cao 
 * Haonan Hou 
 * Jialin Qiao 
 * Jianmin Wang 
 * Jincheng Sun 
     * Jinrui Zhang 
 * Julian Feinauer 
 * Jun Yuan 
 * Justin Mclean 
 * Kevin A. McGrail 
 * Kun Liu 
 * Lei Rui 
 * Rong Kang 
 * Rui Liu 
 * Shuo Zhang 
 * Stefanie Zhao 
 * Tian Jiang 
 * Tianan Li 
 * Willem Ning Jiang 
 * Xiang

Re: Considering lossy encoding/compression?

[Julian Feinauer]

Hi,

another common pattern is just to downsample the datasets (as Influx does it 
for example). This is technically also a lossless encoding (but not as fancy).

Julian

Am 09.09.20, 05:09 schrieb "Xiangdong Huang" :

Hi,

> I know PI has Spinning Door Transformation compress algorithm which is a
lossy encoding method, and it has been used in many cement factories.

Yes, if we have demand to implement a lossy encoding, I'd like to also
choose SDT...

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


runhus...@foxmail.com  于2020年9月9日周三 上午10:53写道：

> Hi，
>
> I know PI has Spinning Door Transformation compress algorithm which is
> a lossy encoding method, and it has been used in many cement factories.
>
>
>
> Thanks.
>
> Chao Wang
> BONC Ltd
>
>
> From: Xiangdong Huang
> Date: 2020-09-09 10:46
> To: dev
> Subject: Considering lossy encoding/compression?
> Hi,
>
> Now IoTDB only supports lossless encoding method, like RLE, 
delta-of-delta,
> etc..
>
> I'd like to know is there a requirement that using a lossy encoding method
> for storing IoT data?
>
> If so, we can add some lossy encoding options.
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>

Re: [VOTE] Charta and proposal for first VP

[Julian Feinauer]

+1 (binding)

Julian

Am 03.09.20, 13:14 schrieb "Dawei Liu" :

Hi,


+1


Best,
Dawei
On 09/03/2020 18:57, Julian Feinauer wrote:
Hi all,

as the discussion in the thread calmed down quite a bit and there was no 
real controvercy I decided to start a formal Vote on our Charta and who we want 
to suggest to the board as initial VP for Apache IoTDB.

From the discuss thread 
(https://lists.apache.org/thread.html/reeac2c6d33d3b7fe3807ded35165f4ee62c3814448ed7b3272672506%40%3Cdev.iotdb.apache.org%3E)
 there is the clear favorit (and only nomination):

Charta: an IoT native database with high performance for data management 
and analysis, on the edge and the cloud.

VP: Xiangdong Huang

As I already stated in the discuss thread I think both choices are 
excellent.

What does the community think?

+1 for YES, Xiangdong as VP and Charta as above
-1 for NO and please make another suggestion then, so that we can move the 
discussion further

As usual, the Vote will be open for at least 72hrs.

Best
Julian

[VOTE] Charta and proposal for first VP

[Julian Feinauer]

Hi all,

as the discussion in the thread calmed down quite a bit and there was no real 
controvercy I decided to start a formal Vote on our Charta and who we want to 
suggest to the board as initial VP for Apache IoTDB.

From the discuss thread 
(https://lists.apache.org/thread.html/reeac2c6d33d3b7fe3807ded35165f4ee62c3814448ed7b3272672506%40%3Cdev.iotdb.apache.org%3E)
 there is the clear favorit (and only nomination):

Charta: an IoT native database with high performance for data management and 
analysis, on the edge and the cloud.

VP: Xiangdong Huang

As I already stated in the discuss thread I think both choices are excellent.

What does the community think?

+1 for YES, Xiangdong as VP and Charta as above
-1 for NO and please make another suggestion then, so that we can move the 
discussion further

As usual, the Vote will be open for at least 72hrs.

Best
Julian

[RESULT][VOTE] Start the graduation process

[Julian Feinauer]

Hi all,

thanks for the many votes that were cast!
As 72hrs have passed the vote ends.

Results are:
17 +1 Votes
from which
12 are binding +1 Votes
and
3 +1 votes from our mentors (all mentors voted)

Thus, we will continue on with the next steps towards graduation.
Thanks once again to all voters and especially to our mentors that support us!

Julian

Re: [VOTE] Start the graduation process

[Julian Feinauer]

+1

Julian

Am 02.09.20, 08:20 schrieb "Xiangdong Huang" :

Hi,

Very glad to see all our mentors voted +1.

Cheers,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Willem Jiang  于2020年9月2日周三 下午2:12写道：

> My +1 (binding)
> It's good to see iotdb is ready to graduate.
>
> Willem Jiang
>
> Twitter: willemjiang
> Weibo: 姜宁willem
>
> On Sun, Aug 30, 2020 at 8:46 PM Julian Feinauer
>  wrote:
> >
> > Hi folks,
> >
> > as we just had a very positive discussion about the maturity of our
> community [1] I want to formally start the vote if we want to start the
> graduation process.
> >
> > For Information about our maturity assessment please see [2].
> >
> > Do you agree that the IoTDB project is ready for graduation and that we
> should start the graduation process now?
> >
> > As usual, this vote will be open for at least 72hrs.
> >
> > Best
> > Julian
> >
> >
> >
> > [1]
> 
https://lists.apache.org/thread.html/r7607cbf6ff2020dc7e7cc27eb698ce265b768ed2a63d3e353c2dc572%40%3Cdev.iotdb.apache.org%3E
> > [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148645763
> >
>

Re: [DISCUSS] graduation process: Charter and VP of IoTDB discussion

[Julian Feinauer]

Hi,

thanks for bringing that up.
Indeed I think your short proposal is right.

Rergarding the VP, I had a short off-list discussion with @Christofer Dutz and 
we both agreed that it should be someone who speaks Chinese and English as our 
community is kind of bilingual.

Regarding the amount of work and support that Xiangdong put into the community 
development and also his work in translating between English and Chinese 
community parts (his English is pretty good, I hope his Chinese too^^) I like 
to suggest Xiangdong, as first VP of IoTDB.

What do others think about that?

Julian

Am 01.09.20, 06:17 schrieb "Xiangdong Huang" :

Hi,

The vote for start the graduation process has last two days and we have
received many +1.

According to [1], we can begin to discuss the charter (resolution) in
parallel.

The main task is using one or two sentences to define what IoTDB is:

A good resolution is neither too narrow nor too broad. If the project’s
scope is too narrow, then its activities will be unnecessarily constrained.
If a project’s scope is too broad then it may lack focus and suffer from
governance issues.

Besides, we need also to elect someone as the VP of the project.

So, any ideas?

IMO, how about:

Apache IoTDB: a native database for storing and analyzing Internet of
Things data, on the edge and the cloud.

[1]
https://incubator.apache.org/guides/graduation.html#the_graduation_process
Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

AW: Configuring IoTDB when run programatically

[Julian Feinauer]

Hey Chris,

Good idea... What about... A PR? :D

Julian

Von meinem Mobiltelefon gesendet


 Ursprüngliche Nachricht 
Von: Christofer Dutz 
Datum: So., 30. Aug. 2020, 17:19
An: dev@iotdb.apache.org
Betreff: Configuring IoTDB when run programatically
Hi all,

I’m currently doing my first steps with IoTDB and am currently struggling to 
get IoTDB to startup in an embedded mode.
I am setting up IoTDB to be setup as a service in my SpringBoot application. 
For this I have my iotdb-engine.properties in my classpath.
However I want IoTDB to have all of its files in my applications home directory.

Now it seems as if this szenario isn’t supported nicely as If I specify the 
IOTDB_HOME, it wants to load the properties from there too, but I would like to 
leave the properties in my classpath.
The reason for this is that I don’t need any setup steps to prepare the 
directories.

It would be cool if in IoTDBDescriptor line 128 wouldn’t expect the URL to be a 
File URL. I could also have a classpath URL 
“classpath:/iotdb-engine.properties” but that causes trouble ;-)

Chris

Re: IoTDB podling report draft - Sepember 2020

[Julian Feinauer]

Hi,

as we already had a discussion about graduation AND I just started a vote about 
graduation we should just note that in the report, I think.

Julian

Am 28.08.20, 16:29 schrieb "Dawei Liu" :

Hi all,
I finished a draft for the report, pls have a review.




## IoTDB


IoTDB is a data store for managing large amounts of time series data such as
timestamped data from IoT sensors in industrial applications.


IoTDB has been incubating since 2018-11-18.


### Three most important unfinished issues to address before graduating:


  1. attract more contributors 
  2. constantly update the stats in the maturity model assessment
  3. get a consensus on graduation in the community and start a vote


### Are there any issues that the IPMC or ASF Board need to be aware of?
  No.


### How has the community developed since the last report?
  1. 2 new PPMC are elected(Haonan Hou, Dawei Liu)
  2. more new contributors appear in the mailing list.
  3. More people are getting involved in community voting, for 
example(v0.10.0 release vote: 17 votes, 7 votes from committers)
  4. Successfully held an online Meetup(2020-07-11)


### How has the project developed since the last report?
  1. Two version(v0.10.0 and v0.10.1) are released. 
  2. The project keeps active, there are 258 pr has merged and 148 pr into 
the main branch. 
  3. The distributed version is in testing stage..
  4. Many new improvements and features have been addressed, for 
example(RestAPI performance optimization, implemented data level merge, support 
order by time , etc ...)


### How would you assess the podling's maturity?
Please feel free to add your own commentary.


  - [ ] Initial setup
  - [ ] Working towards first release
  - [ ] Community building
  - [x] Nearing graduation
  - [ ] Other:


### Date of last release:


  2020-08-23


### When were the last committers or PPMC members elected?
  2020-07-13 for the last PPMC.


### Have your mentors been helpful and responsive?
  Yes, helpful and responsive.


### Is the PPMC managing the podling's brand / trademarks?
  Required Name Search is done. A google search didn’t show any major 
branding issues that the PPMC needs to deal with.

[VOTE] Start the graduation process

[Julian Feinauer]

Hi folks,

as we just had a very positive discussion about the maturity of our community 
[1] I want to formally start the vote if we want to start the graduation 
process.

For Information about our maturity assessment please see [2].

Do you agree that the IoTDB project is ready for graduation and that we should 
start the graduation process now?

As usual, this vote will be open for at least 72hrs.

Best
Julian



[1] 
https://lists.apache.org/thread.html/r7607cbf6ff2020dc7e7cc27eb698ce265b768ed2a63d3e353c2dc572%40%3Cdev.iotdb.apache.org%3E
[2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148645763

Re: Podling Iotdb Report Reminder - September 2020

[Julian Feinauer]

Thank you very much Dawei.
In fact I directly thought "Probably Dawei could do it, he also writes these 
nice reports" : )

Thanks!
Julian

Am 28.08.20, 04:41 schrieb "Dawei Liu" :

Hi,


I'd like to try write this report.


Can someone give me some guidance?


Best,
——
Dawei Liu




On 08/27/2020 19:47，Xiangdong Huang wrote：
Hi,

Thanks for reminding us.

Does someone want to write the report?

I may have no time before Aug 31. If there is no person writing it, I will
do that on Sep 1.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

黄向东
清华大学 软件学院


 于2020年8月27日周四 上午9:55写道：

Dear podling,

This email was sent by an automated system on behalf of the Apache
Incubator PMC. It is an initial reminder to give you plenty of time to
prepare your quarterly board report.

The board meeting is scheduled for Wed, 16 September 2020.
The report for your podling will form a part of the Incubator PMC
report. The Incubator PMC requires your report to be submitted 2 weeks
before the board meeting, to allow sufficient time for review and
submission (Wed, September 02).

Please submit your report with sufficient time to allow the Incubator
PMC, and subsequently board members to review and digest. Again, the
very latest you should submit your report is 2 weeks prior to the board
meeting.

Candidate names should not be made public before people are actually
elected, so please do not include the names of potential committers or
PPMC members in your report.

Thanks,

The Apache Incubator PMC

Submitting your Report

--

Your report should contain the following:

*   Your project name
*   A brief description of your project, which assumes no knowledge of
the project or necessarily of its field
*   A list of the three most important issues to address in the move
towards graduation.
*   Any issues that the Incubator PMC or ASF Board might wish/need to be
aware of
*   How has the community developed since the last report
*   How has the project developed since the last report.
*   How does the podling rate their own maturity.

This should be appended to the Incubator Wiki page at:

https://cwiki.apache.org/confluence/display/INCUBATOR/September2020

Note: This is manually populated. You may need to wait a little before
this page is created from a template.

Note: The format of the report has changed to use markdown.

Mentors
---

Mentors should review reports for their project(s) and sign them off on
the Incubator wiki page. Signing off reports shows that you are
following the project - projects that are not signed may raise alarms
for the Incubator PMC.

Incubator PMC

Re: Support double quotation in Path and optimize the Path String

[Julian Feinauer]

Ah thanks. If you knowit ist good!

Am 27.08.20, 10:34 schrieb "Boris Zhu" :

WeChat Group

On Thu, Aug 27, 2020 at 4:30 PM Julian Feinauer <
j.feina...@pragmaticminds.de> wrote:

> Hi Boris,
>
> excellent initative!
> I also had a discussion with @Xiangdong Huang and @Jialin Qiao about that
> matter.
>
> Does one of you remember where we documented our discussion?
>
> Julian
>
> Am 27.08.20, 05:44 schrieb "Boris Zhu" :
>
> Hi
> I write a pr about using the detached full path instead of the full
> path
> between SQL analysis and metadata in order to reduce unnecessary path
> split. A new Class extends Path which is called “PartialPath” which
> contains the detached full path. BTW, you can use double quotes in
> name of
> device. here is the pr,
> https://github.com/apache/incubator-iotdb/pull/1627
>
>

Re: Support double quotation in Path and optimize the Path String

[Julian Feinauer]

Hi Boris,

excellent initative!
I also had a discussion with @Xiangdong Huang and @Jialin Qiao about that 
matter.

Does one of you remember where we documented our discussion?

Julian

Am 27.08.20, 05:44 schrieb "Boris Zhu" :

Hi
I write a pr about using the detached full path instead of the full path
between SQL analysis and metadata in order to reduce unnecessary path
split. A new Class extends Path which is called “PartialPath” which
contains the detached full path. BTW, you can use double quotes in name of
device. here is the pr, https://github.com/apache/incubator-iotdb/pull/1627

Re: Restart the status monitor module of iotdb

[Julian Feinauer]

Hi,

indeed there is potential for some more modularity (and I think I already added 
a submodule in that branch).
Currently Xiangdong is improving the collected metrics and I would be happy to 
get this into the master / develop ASAP : )

Julian

Am 26.08.20, 08:10 schrieb "Xiangdong Huang" :

Hi,

I think TOTAL_REQ_SUCCESS is also important in some cases.

By the way, there is a branch "feature/improve-monitoring-metrics" which is
written by Julian

and use Micrometer to collect some performance info.

I think we can maintain that branch in the next step.
(By the way,  I believe that making the project more modularity is
beneficial.
We can extract the implementation as interfaces to let users decide using
which way to get the info)

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Xiangwei Wei  于2020年8月26日周三 下午1:52写道：

> Hi,
>
> Currently, the status monitor module of iotdb is abandoned and not
> maintained now. It's time for us to restart this module.
>
> Before, it was designed including two child modules: 1. the writing data
> status monitor and 2. the file size monitor.
>
> 1. The writing data monitor module is responsible for collecting writing
> data statistics including
> - TOTAL_POINTS (Calculate the total writing points number.)
> - TOTAL_REQ_SUCCESS (Calculate the successful requests number.)
> - TOTAL_REQ_FAIL (Calculate the failed requests number.)
> - TOTAL_POINTS_FAIL (Calculate the failed writing points number.)
> - TOTAL_POINTS_SUCCESS
> etc.
>
> and presenting them to users by statistics time series like
> .root.stats.write.global.TOTAL_POINTS. It's not supported completely now.
>
> Actually, many parts of origin design are redundant and not meaningful for
> users. Therefore, *I want to just keep TOTAL_POINTS here and remove
> others*.
> And i did not come up with some good ideas about *what needs to be added 
to
> this monitor* while writing data. Welcome to any ideas here!
>
>
>
> 2. The file size monitor is concerned about how much space the data file
> takes. It's supported partly and very limited now.
>
> In my opinion, it's easily for users to see how much space the data file
> takes by a visible interface or just a Linux code. And it's more flexible
> for users to decide what to calculate by themselves. Therefore, there is 
no
> sufficient reason for us to redesign and reimplement this child module. *I
> want to just remove it from iotdb.*
>
> The impletation way of status monitor module was designed as setting a
> monitor thread and returning the status info every
> *back_loop_period_in_second* (default 5s), removing outdated data
> every *stat_monitor_retain_interval_in_second
> (default 600s). *It cost a lot and is meaningless if no write was made
> then.
>
> Therefore, we want to collect the data while writing data and keep it in
> memory temperarily, *every time flush() is invoked or user want to select
> the monitor data, we update the data info to the disk and return users the
> lastest data.*
>
> That what need to be monitored which is useful for users except
> TOTAL_POINTS still need to be discussed. Welcome to any ideas.
>
>
>
>
>
> --
> Best,
> Xiangwei Wei
>

Re: [DISCUSS] discuss about graduation of Apache IoTDB (incubating)

[Julian Feinauer]

Hi,

I am no Mentor but rather a community member but speaking in my Role as IPMC 
Member I agree with Chris and Kevin and also think the project and the 
community are in good shape and we are ready for graduation.

Julian

Am 24.08.20, 14:37 schrieb "Willem Jiang" :

+1. I think IotDB is close to graduation :).

Willem Jiang

Twitter: willemjiang
Weibo: 姜宁willem

On Sun, Aug 23, 2020 at 5:04 PM Christofer Dutz
 wrote:
>
> Hi all,
>
> I think the project is in great shape and I would happily support its 
graduation.
>
> Chris
> 
> Von: Xiangdong Huang 
> Gesendet: Sonntag, 23. August 2020 05:45
> An: dev ; Justin Mclean 
> Betreff: Re: [DISCUSS] discuss about graduation of Apache IoTDB 
(incubating)
>
> Hi everyone,
>
> After about one month effort, we finish releasing v0.10.1, which exposing
> some
> new issues of releasing the project and finally we have fixed them.
>
> As about two months have passed since our first discussion about 
graduation,
> I‘d like to recall this discussion and listen to all of your opinions.
>
> Thanks
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
> Dawei Liu  于2020年7月9日周四 上午11:32写道：
>
> > Hi,
> >
> >
> > Time flies, I have been work together with all of you for 8 months.
> >
> >
> > I feel connected with the culture of Apache-way. The culture is of 
benefit
> > to me.
> >
> >
> > I was back to the community end of June after two months of busy work in
> > my company ,
> > happily finding there is a big improvement in query and write 
performance
> > of the Master branch.
> >
> >
> > I see a lot of new faces , fresh-up and energetic,
> > from which I can see the bright future of our IoTDB community.
> >
> >
> > I will be really happy if we reach the demand of graduation.
> >
> >
> > Thanks
> > ——
> > Dawei Liu
> >
> >
> > On 07/7/2020 15:30，Christofer Dutz wrote：
> > I for my part am very happy with the progress the project made.
> >
> > Taking the amount of "pointing out stuff"-emails (TM) I have to write
> > (Which is almost 0) I wouldn't have any issues right now voting +1 on 
your
> > graduation.
> >
> > Chris
> >
> >
> > Am 07.07.20, 08:55 schrieb "Julian Feinauer" <
> > j.feina...@pragmaticminds.de>:
> >
> > Hi,
> >
> > thanks for your feedback Justin and I would also be very interested in
> > hearing your opinion on the maturity.
> > As I'm no mentor and pretty involved in the community know I feel that 
the
> > community is mature enough but as with every young project, there are
> > always things to improve.
> >
> > But I think ist a good point in time now to get some feedback and 
improve
> > specifics that we are still lacking.
> >
> > Julian
> >
> > Am 07.07.20, 06:07 schrieb "Justin Mclean" :
> >
> > Hi,
> >
> > I think the project have made very good progress, but it might be a 
little
> > bit early to graduate. I’ve have a look over things in a little more 
detail
> > over the next couple of days and see if that changes my mind.
> >
> > Thanks,
> > Justin
> >
> >
> >

AW: ApacheCon @Home 2020: Proposal not accepted

[Julian Feinauer]

Hi friends,

I always got confirmation for a talk about our bilingual community and how we 
all came to work that nicely together.

@Giorgio sounds cool!

Julian

Von meinem Mobiltelefon gesendet


 Ursprüngliche Nachricht 
Von: "Kevin A. McGrail" 
Datum: So., 2. Aug. 2020, 13:41
An: dev@iotdb.apache.org, saint...@gmail.com
Betreff: Re: ApacheCon @Home 2020: Proposal not accepted
Always happy to help with slide review!

On 8/1/2020 9:51 PM, Xiangdong Huang wrote:
> Nice!  Maybe you can cooperate with Kevin to prepare the slides and finish
> the talk.

--
Kevin A. McGrail
kmcgr...@apache.org

Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171

Re: [VOTE] Apache IoTDB 0.10.1 (incubating) RC1 release

[Julian Feinauer]

Hi,

yes, big thank to Houliang for his finding and good that you cancel the vote.

I think its good for the community also to see that it doesn’t matter if you 
are committer or PMC or just a user who contributes a bit in his spare time to 
make an impact if you bring up good points!

Julian

Am 27.07.20, 11:51 schrieb "Xiangdong Huang" :

Hi,

Thanks all for voting for this release.

As Houliang mentioned, in iotdb-hive-connector, we use Apache Hive
(hive-serde) 2.8.4 as a dependency.

Sadly, Apache Hive depends on Apache ORC file, v1.3.4, but orc-core v.1.3.4
has a GPL dependency...

ORC-core v1.6 fixed the issue.

We have to exclude orc-core v1.3.4 from hive-serde 2.8.4 and claim
orc-corev1.6 explicitly.

(This will be a good hint for all users/projects who use Apache Hive 2.8.4)

I'd like to terminate this vote and will begin to release RC2.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Houliang Qi  于2020年7月27日周一 下午4:41写道：

> Hi all,
>  -1  from contributor
>
>
> I checked the following operations on macOS 10.15.4 and jdk 1.8
>
>
> Check third party dependencies [Not OK]
> hive-connector’s dependencies(hive-serde -> orc-core ) use GPL 2.0
> license, which incompatible with apache 2.0 license
> (Remove the orc-core dependency in hive-serde and use orc-core 1.6 is ok )
>
>
>
> The source release:
>
> Check third party dependencies [not ok]
>
> Incubating in name [ok]
>
> Has DISCLAIMER [ok]
>
> LICENSE and NOTICE [ok]
>
> Signatures and hashes [ok]
>
> Release candidates match with corresponding tags [ok]
>
> no jar files [ok]
>
> Could compile from source: ./mvnw.sh clean install [ok]
>
>
>
>
> The binary distribution:
>
> Incubating in name [ok]
>
> Has DISCLAIMER [ok]
>
> LICENSE and NOTICE [ok]
>
> Signatures and hashes [ok]
>
> Could run with the following statements [ok]
>
> SET STORAGE GROUP TO root.turbine;
>
> CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE,
> ENCODING=GORILLA;
>
> insert into root.turbine.d1(timestamp,s0) values(1,1);
>
> insert into root.turbine.d1(timestamp,s0) values(2,2);
>
> insert into root.turbine.d1(timestamp,s0) values(3,3);
>
> select * from root;
>
>
>
>
> minor issue :
>
> The mvnw.sh does not have executable permissions
>
>  ——
> Thanks,
> Houliang Qi
> On 07/27/2020 16:37，runhus...@foxmail.com wrote：
> Hi,
>
> +1  from contributor
>
>
> I checked:
>
>
> - Incubating in name
> - Download links
> - Signatures and hashes
> - Compiling from source
> - Start in Windows
> - Run with following statements
>
>
> SET STORAGE GROUP TO root.turbine;
> CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE,
> ENCODING=GORILLA;
> insert into root.turbine.d1(timestamp,s0) values(1,1);
> insert into root.turbine.d1(timestamp,s0) values(2,2);
> insert into root.turbine.d1(timestamp,s0) values(3,3);
> select * from root;
> select * from root.* limit 0 offset 23;
>
>
> minor issue :
> offset and limit only support Int32, maybe it's small.
>
>
>
>
>
>
> Thanks!
>
> runhus...@foxmail.com
>
>
> From: Dawei Liu
> Date: 2020-07-24 22:14
> To: dev@iotdb.apache.org
> CC: dev@iotdb.apache.org
> Subject: Re: [VOTE] Apache IoTDB 0.10.1 (incubating) RC1 release
> Hi,
>
>
> +1
>
>
> I checked:
>
>
> - Incubating in name
> - Download links
> - Start in Mac
> - Run with following statements
>
>
> set storage group to root.release;
> create timeseries root.release.d1.test(status) with DATATYPE=BOOLEAN,
> ENCODING=PLAIN, compression=SNAPPY
> show timeseries;
> insert into root.release.d1(time,status) values(1,false);
> select * from root.release.d1;
>
>
> minor issue :
> The package not include 'README_ZH.md' file
>
>
>
>
> Best
> ——
> Dawei Liu
> On 07/24/2020 21:31，孙泽嵩 wrote：
> Hi,
>
> +1 from committer : )
>
> I checked:
>
> - Incubating in name
> - Download links
> - Signatures and hashes
> - DISCLAIMER
> - Git tag
> - ASF headers in all source files
> - Compiling from source
> - LICENSE and NOTICE
> - Start in Mac
> - Run with following statements
>
> SET STORAGE GROUP TO root.turbine;
> CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE,
> ENCODING=GORILLA;
> insert into root.turbine.d1(timestamp,s0) values(1,1);
> insert into 

Re: Is there any docs on how to run iotdb on android?

[Julian Feinauer]

Hope it helps you!

The only module you may need to reference is:

implementation 'org.apache.iotdb:tsfile:0.10.0'

It  contains everxthing for reading and writing files.

Writing is straightforward, reading is a bit more complex and we should improve 
the docu.
But just ask here if there are things not clear to you!

Julian

Am 23.07.20, 10:16 schrieb "Shi Jinghai" :

Hi Julian,

That’s great. I’ll try the way you mentioned.

Kind Regards,

发件人: Julian Feinauer<mailto:j.feina...@pragmaticminds.de>
发送时间: 2020年7月23日 16:06
收件人: dev@iotdb.apache.org<mailto:dev@iotdb.apache.org>
主题: Re: Is there any docs on how to run iotdb on android?

Hey,

you need no jdk, yes.
Just integrate the jar via grafle (I think android favors gradle) and you 
only need a jre.

Julian

Am 23.07.20, 10:02 schrieb "Shi Jinghai" :

Thanks Julian and Xiangdong for your quick reply.

To Julian,
Would it possible iotdb have such a tool jar, commons-iotdb? Then the 
jdk requirement could be removed, right?



发件人: Julian Feinauer<mailto:j.feina...@pragmaticminds.de>
发送时间: 2020年7月23日 15:18
收件人: dev@iotdb.apache.org<mailto:dev@iotdb.apache.org>
主题: Re: Is there any docs on how to run iotdb on android?

Hey,

another alternative would be to use plain tsfiles and not the whole 
IoTDB Stack.
This should work out oft he box on android, I think.

Julian

Am 23.07.20, 09:03 schrieb "Xiangdong Huang" :

Hi Jinghai,

We just tried Raspberry PI, and did not test on Andorid.
This is an interesting scenario. We can have a try.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Shi Jinghai  于2020年7月23日周四 下午1:21写道：

> Hi List,
>
> Is there any docs on how to run iotdb on android?
>
> My scenario is to log iot equipment info in a home.
>
> TIA,
>
> Shi Jinghai
>

Re: Is there any docs on how to run iotdb on android?

[Julian Feinauer]

Hey,

you need no jdk, yes.
Just integrate the jar via grafle (I think android favors gradle) and you only 
need a jre.

Julian

Am 23.07.20, 10:02 schrieb "Shi Jinghai" :

Thanks Julian and Xiangdong for your quick reply.

To Julian,
Would it possible iotdb have such a tool jar, commons-iotdb? Then the jdk 
requirement could be removed, right?



发件人: Julian Feinauer<mailto:j.feina...@pragmaticminds.de>
发送时间: 2020年7月23日 15:18
收件人: dev@iotdb.apache.org<mailto:dev@iotdb.apache.org>
主题: Re: Is there any docs on how to run iotdb on android?

Hey,

another alternative would be to use plain tsfiles and not the whole IoTDB 
Stack.
This should work out oft he box on android, I think.

Julian

Am 23.07.20, 09:03 schrieb "Xiangdong Huang" :

Hi Jinghai,

We just tried Raspberry PI, and did not test on Andorid.
This is an interesting scenario. We can have a try.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Shi Jinghai  于2020年7月23日周四 下午1:21写道：

> Hi List,
>
> Is there any docs on how to run iotdb on android?
>
> My scenario is to log iot equipment info in a home.
>
> TIA,
>
> Shi Jinghai
>

Re: Is there any docs on how to run iotdb on android?

[Julian Feinauer]

Hey,

another alternative would be to use plain tsfiles and not the whole IoTDB Stack.
This should work out oft he box on android, I think.

Julian

Am 23.07.20, 09:03 schrieb "Xiangdong Huang" :

Hi Jinghai,

We just tried Raspberry PI, and did not test on Andorid.
This is an interesting scenario. We can have a try.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Shi Jinghai  于2020年7月23日周四 下午1:21写道：

> Hi List,
>
> Is there any docs on how to run iotdb on android?
>
> My scenario is to log iot equipment info in a home.
>
> TIA,
>
> Shi Jinghai
>

Re: [Discuss] Release of 0.10.1

[Julian Feinauer]

Cool! Very happy to hear!

Julian

Am 22.07.20, 08:22 schrieb "Xiangdong Huang" :

Hi,

If there are no other issues, I will start to release v0.10.1.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Jialin Qiao  于2020年7月18日周六 下午2:24写道：

> Hi,
>
> Thanks for all of your supplement.
>
> Before releasing, I checked the boudles in the distribution version and
> fixed the LICENSE-binary in [1].
>
> [1] https://github.com/apache/incubator-iotdb/pull/1518
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院
>
> > -原始邮件-
> > 发件人: "Haonan Hou" 
> > 发送时间: 2020-07-17 16:24:02 (星期五)
> > 收件人: "dev@iotdb.apache.org" 
> > 抄送:
> > 主题: Re: [Discuss] Release of 0.10.1
> >
> > +1.
> >
> > Seems like many critical bugs has been fixed. I think it’s necessary to
> release 0.10.1 now.
> >
> > Best,
> > Haonan
> >
> >
> > > On Jul 17, 2020, at 4:16 PM, Julian Feinauer <
> j.feina...@pragmaticminds.de> wrote:
> > >
> > > Release early and often, they say. So you got my +1 : )
> > >
> > > Julian
> > >
> > > Am 17.07.20, 10:13 schrieb "田原" :
> > >
> > >Hi Jialin,
> > >
> > >+1.
> > >
> > >And I make another two improvements that should be included in
> 0.10.1
> > >
> > >* [IOTDB-802] Improve Group By
> > >* Change the default fetch size in session(
> https://github.com/apache/incubator-iotdb/pull/1506)
> > >
> > >Best,
> > >---
> > >Yuan Tian
> > >
> > >
> > >> -原始邮件-
> > >> 发件人: "Jialin Qiao" 
> > >> 发送时间: 2020-07-16 18:02:10 (星期四)
> > >> 收件人: dev-iotdb 
> > >> 抄送:
> > >> 主题: [Discuss] Release of 0.10.1
> > >>
> > >> Hi,
> > >>
> > >>
> > >> Bugs always appear, but we will kill them one by one.
> > >>
> > >>
> > >> I list the fixed bug since 0.10.0, it's time to consider releasing
> 0.10.1
> > >>
> > >>
> > >> * [IOTDB-797] InsertTablet deserialization from WAL error
> > >> * [IOTDB-788] Can not upgrade all storage groups
> > >> * [IOTDB-792] deadlock when insert while show latest timeseries
> > >> * [IOTDB-794] Rename file or delete file Error in start check in
> Windows
> > >> * [IOTDB-795] BufferUnderflowException in Hive-connector
> > >> * [IOTDB-766] Do not release unclosed file reader, a small memory 
leak
> > >> * [IOTDB-796] Concurrent Query throughput is low
> > >> * Query result is not correct when some unsequence data exists
> > >>
> > >>
> > >> Have I missed any? And, is there anyone wants to be the RM of 0.10.1
> :)
> > >>
> > >> Thanks,
> > >> --
> > >> Jialin Qiao
> > >> School of Software, Tsinghua University
> > >>
> > >> 乔嘉林
> > >> 清华大学 软件学院
> > >
> >
>

Re: [Discuss] How to delivery the device concept to users

[Julian Feinauer]

Thanks fort he clear explanation, yes I remember that there were also reported 
performance issues with that.

But to generalize the concept of a device all we would need is a tree strucutre 
where each node has start time / end time for "everything" in the file.
Like in your example:

Root (1, 10) 
sg (1, 10)
  d1 (1, 5)
  d2 (1, 6)
  d3 (2, 10)

This would allow us to fetch the necessary information on each level.

What about using some kind of simple KV Store which is ofc disk based but does 
its own in memory caching and optimization such that frequent accesses ("hot 
devices" or more generally "paths") are fast.

Is this a valid idea?

Julian

Am 20.07.20, 11:53 schrieb "Jialin Qiao" :

Hi,

> The question I would ask is why "devices" hurt us.

I'd like to introduce this a bit. For each storage group, we flush the 
memtable into TsFiles one by one. For each TsFile, we maintain a temporal index 
on device level in memory. Suppose there are 3 devices in one TsFile, the index 
is like this:

start time array: long[3] = {1, 1, 2}
end time array: long[3] = {5, 6, 10}
devicesToIndexInArray: Map = {"root.sg.d1" -> 0, 
"root.sg.d2" -> 1, "root.sg.d3" -> 2}

If we have millions of devices, for each TsFile, this index will reach 
dozens of MB in memory. Although we could introduce the persistence of the 
index. It is still recommended to decrease the number of devices.

I wonder whether we could index the file by its name. (naming the tsfile by 
date) E.g., we store each day's data in one file and name it as 
sg-2020-07-20.TsFile. Then, we do not need to maintain the index in memory, we 
just need to check whether the file exist in the queried interval.

Thanks,
--
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

> -原始邮件-
> 发件人: "Julian Feinauer" 
> 发送时间: 2020-07-20 17:34:40 (星期一)
> 收件人: "dev@iotdb.apache.org" 
> 抄送: 
> 主题: Re: [Discuss] How to delivery the device concept to users
> 
> Hey Jialin, xinagdong,
> 
> very good question!
> 
> And I tend to agree with Xiangdong.
> If the users do it that way it probably makes most sense for them.
> The question I would ask is why "devices" hurt us (I know a bit about the 
implementation of course but probably we have to adopt our datamodel also a bit 
in the future).
> 
> Generally speaking, form e it also makes sense tob e allowed to have 
"subcategories" below my devices as my devices usually are "big".
> And technically speaking in the current version this is totally possible 
to have nested structures below devices or measurements (but these will then 
again be devices).
> 
> So my question is:
> - Do we really need the static construct of a "device" or can we probably 
use a different datastructure where I "select" my device only at query time and 
we just select everything under that tree as ist measurements or 
"sub-measurements" in cases of nesting.
> 
> WDYT?
> 
> Julian
> 
> Am 20.07.20, 09:34 schrieb "Xiangdong Huang" :
> 
> Hi,
> 
> This is a quite good topic!
> 
> 1. maybe we should hear more users opinions.
> 
> For me, I think emphasize the concept of "device" is good. We can even
> expose the concept in our APIs.
> 
> 2.
> 
> > A more efficient way is
> > root.sg.device1.measurement1_int0
> > root.sg.device1.measurement1_int1
> >  root.sg.device1.measurement1_int2
> > root.sg.device1.measurement2_long
> 
> I think the more efficient way is:
> 
> root.sg.device1.measurement1.0
> root.sg.device1.measurement1.1
> root.sg.device1.measurement1.2
> root.sg.device1.measurement2
> 
> And, as you said "a device has a sensor that collects some data in 
array
> format (int[3]) and some in long type",
> will the user query just one element from the int[3]? If not, a better
> schema is:
> 
> root.sg.device1.measurement1 (the dataType is int[])
> root.sg.device1.measurement2 (the dataType is long)
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
>  黄向东
> 清华大学 软件学院
> 
> 
> Jialin Qiao  于2020年7月20日周一 下午3:28写道：
> 
> > Hi
> >
> &

Re: Functional design of quoted paths

[Julian Feinauer]

Hey Xiangwei,

that is an excellent suggestion (I discussed the same already with Xiangdong 
some weeks ago).
We mix a bit between Strings and Paths (and even inside paths we use strings 
and dynamic parsing too often).

So it would make sense to mee to follow your suggested approach with clear 
escape rules so that basically every string can be the name of a path / 
measurement.
This makes especially sense when importing data from external systems.
If you e.g. get device names from another system which allows dots then you 
cannot map it (bidirectional) to IoTDB.

Best
Julian

Am 20.07.20, 10:41 schrieb "Xiangwei Wei" :

Hi, Xiangdong

You are right. Thank you for correction.

Xiangdong Huang  于2020年7月20日周一 下午3:29写道：

> Hi Xiangwei,
>
> > Besides, double quote is used as quote while single quote is for string
> literal.
>
> Are you meaning double quote is used as escape character..
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
> Xiangwei Wei  于2020年7月20日周一 上午11:38写道：
>
> > Hi all,
> >
> > Currently, for paths like ROOT.sg.d1."s.1", we support the double quote
> on
> > measurement only, and the process about quoted paths is not uniform.
> >
> > Therefore, in next implementation, we want to achieve the following
> GOALS:
> >
> > 1. let’s support double quotes on multiple locations of a path (not only
> > the measurement), like ROOT.sg1."d..1"."s_1". Besides, double quote is
> used
> > as quote while single quote is for string literal.
> >
> > 2. let’s uniform the convert between Path and string;
> >
> > 3. maybe we can expose Path object to users in session API.
> >
> > The next is some EXAMPLES:
> >
> > Session
> >
> > String deviceId = "root.sg1.\"d..1\"";List measurements = new
> > ArrayList<>();measurements.add("\"s..1\"");
> >
> > JDBC
> >
> > statement.execute("insert into root.sg1.\"d..1\"(timestamp, \"s..1\")
> > values(1,2)");
> >  statement.execute("select * from root");
> >  headers in
> > result:+-+---+|
> >
> >
> >
> 
Time|root.turbine1.\"d..1\".\"s..1\"|+-+---+|1970-01-01T08:00:00.001+08:00|
> >
> > 2.0|+-+---+
> >
> > TsFile
> >
> > We store the following Strings in file:
> >
> > String devceId = "root.sg1.\"d..1\"";String measurementId = "\"s..1\"";
> >
> > Schema tree in IoTDB
> >
> > root -> sg1 -> “d…1” -> “s…1”
> >
> >
> > Welcome to any suggestions ：）
> >
> >
> > --
> > Best,
> > Xiangwei Wei
> >
>


-- 
Best,
Xiangwei Wei

Re: [Discuss] How to delivery the device concept to users

[Julian Feinauer]

Hey Jialin, xinagdong,

very good question!

And I tend to agree with Xiangdong.
If the users do it that way it probably makes most sense for them.
The question I would ask is why "devices" hurt us (I know a bit about the 
implementation of course but probably we have to adopt our datamodel also a bit 
in the future).

Generally speaking, form e it also makes sense tob e allowed to have 
"subcategories" below my devices as my devices usually are "big".
And technically speaking in the current version this is totally possible to 
have nested structures below devices or measurements (but these will then again 
be devices).

So my question is:
- Do we really need the static construct of a "device" or can we probably use a 
different datastructure where I "select" my device only at query time and we 
just select everything under that tree as ist measurements or 
"sub-measurements" in cases of nesting.

WDYT?

Julian

Am 20.07.20, 09:34 schrieb "Xiangdong Huang" :

Hi,

This is a quite good topic!

1. maybe we should hear more users opinions.

For me, I think emphasize the concept of "device" is good. We can even
expose the concept in our APIs.

2.

> A more efficient way is
> root.sg.device1.measurement1_int0
> root.sg.device1.measurement1_int1
>  root.sg.device1.measurement1_int2
> root.sg.device1.measurement2_long

I think the more efficient way is:

root.sg.device1.measurement1.0
root.sg.device1.measurement1.1
root.sg.device1.measurement1.2
root.sg.device1.measurement2

And, as you said "a device has a sensor that collects some data in array
format (int[3]) and some in long type",
will the user query just one element from the int[3]? If not, a better
schema is:

root.sg.device1.measurement1 (the dataType is int[])
root.sg.device1.measurement2 (the dataType is long)

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Jialin Qiao  于2020年7月20日周一 下午3:28写道：

> Hi
>
> Recently, I find that some users create timeseries do not following the
> real world semantic of device
>
>
> E.g., a device has a sensor that collects some data in array format
> (int[3]) and some in long type.
>
>
> Many users will create timeseries like this:
>
>
> root.sg.device1.measurement1.int0
> root.sg.device1.measurement1.int1
> root.sg.device1.measurement1.int2
> root.sg.device1.measurement2.long
>
>
> As a consequence, there will be two devices instead of one device. This
> will cause the real number of devices is much bigger than the real devices
> they thought. The drawback is: more devices leads to more memory
> consumption.
>
>
> A more efficient way is
>
>
> root.sg.device1.measurement1_int0
> root.sg.device1.measurement1_int1
> root.sg.device1.measurement1_int2
> root.sg.device1.measurement2_long
>
>
> In this schema, there will be only one device and 4 measurements.
>
>
> The problem is we extract the device id automatically. Users usually do
> not have a clear concept about "device". Should we emphasize the concept 
of
> device by letting users create device manually?
>
>
> What do you think?
>
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
>
> 乔嘉林
> 清华大学 软件学院

Re: [Discuss] Release of 0.10.1

[Julian Feinauer]

Release early and often, they say. So you got my +1 : )

Julian

Am 17.07.20, 10:13 schrieb "田原" :

Hi Jialin,

+1.

And I make another two improvements that should be included in 0.10.1

* [IOTDB-802] Improve Group By 
* Change the default fetch size in 
session(https://github.com/apache/incubator-iotdb/pull/1506)

Best,
---
Yuan Tian


> -原始邮件-
> 发件人: "Jialin Qiao" 
> 发送时间: 2020-07-16 18:02:10 (星期四)
> 收件人: dev-iotdb 
> 抄送: 
> 主题: [Discuss] Release of 0.10.1
> 
> Hi,
> 
> 
> Bugs always appear, but we will kill them one by one.
> 
> 
> I list the fixed bug since 0.10.0, it's time to consider releasing 0.10.1
> 
> 
> * [IOTDB-797] InsertTablet deserialization from WAL error
> * [IOTDB-788] Can not upgrade all storage groups
> * [IOTDB-792] deadlock when insert while show latest timeseries
> * [IOTDB-794] Rename file or delete file Error in start check in Windows
> * [IOTDB-795] BufferUnderflowException in Hive-connector 
> * [IOTDB-766] Do not release unclosed file reader, a small memory leak
> * [IOTDB-796] Concurrent Query throughput is low
> * Query result is not correct when some unsequence data exists
> 
> 
> Have I missed any? And, is there anyone wants to be the RM of 0.10.1 :)
> 
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院

Re: [ANNOUNCE] Add Haonan Hou as a new PPMC of IoTDB

[Julian Feinauer]

Congratulations Haonan also from my side : )
Keep up the good work!

Julian

Am 17.07.20, 08:18 schrieb "Haonan Hou" :

Hi,

Thank you everyone for your warm welcome!

BR,
Haonan Hou

> On Jul 17, 2020, at 1:30 PM, Dawei Liu  wrote:
> 
> 
> 
> Congrats, Haonan!
> 
> 
> Best
> ——
> Dawei Liu
> 
> 
> On 07/17/2020 13:29，孙泽嵩 wrote：
> Congrats, Haonan!
> 
> 
> Best,
> ---
> Zesong Sun
> School of Software, Tsinghua University
> 
> 孙泽嵩
> 清华大学 软件学院
> 
> 2020年7月17日 12:08，Houliang Qi  写道：
> 
> Congratulations,  Haonan!
> ——
> Thanks,
> Houliang Qi
> On 07/17/2020 10:38，Jialin Qiao wrote：
> Hi all,
> 
> 
> I am pleased to announce that after discussion and vote among PPMC 
groups, and publicity on IPMC mailing list, Haonan Hou is nominated as a new 
PPMC of IoTDB.
> 
> 
> Haonan has joined the IoTDB community for more than 10 months and is 
always active in the mailing list. He mainly focuses on the new TsFile 
structure, online upgrade tools, documentations. He also contributes a lot for 
the propagation of IoTDB and resolving users' problems in the github issues.
> 
> 
> Now, let's welcome Haonan!
> 
> 
> Best,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院
> 

Re: Courses /Trainings for Open Source Projects

[Julian Feinauer]

Dawei,

sadly you are totally right.
We curently have pretty poor loading times and do not yet know why.
Last week(s) we focused on setup and content but next step is definetly improve 
loading times and performance as ist currently pretty inactceptable.

Julian

Am 10.07.20, 04:12 schrieb "Dawei Liu" :

Hi，


This website opens very, very slowly on my computer. Can others open it?


Best
———
Dawei Liu
On 07/5/2020 20:04，Julian Feinauer wrote：
Hi folks,

as some may have noticed on other lists Chris, myself and some others are 
starting something together (and also with other Partners from the community!).
The project is called https://industrial-opensource.com and we try to form 
a network of (I)IoT experts from different Open Source projects to help the 
industry.

Our next idea was to provide trainings and courses for the open source 
projects we list on the site (or even more) but we are not yet clear about the 
interest and demand.
And, as we are community driven this explicity INCLUDES free content and 
NOT only paid services. Thus I think ist reasonable to share the little survey 
we prepared on this list also.
The survey should help us to see if there is a demand or what exactly the 
community / communities may need and expect.

So we would be happy if you take the time to fill it out : )

Link to the survey: https://www.surveymonkey.de/r/CKZMTDL

Thanks already!
Julian

Re: [Experiment sharing] How chunk size(number of points) impact the query performance

[Julian Feinauer]

Hey,

very interesting experiment.
Did you use something like JMH fort he Benchmark or did you perform it directly?

Julian

Am 09.07.20, 09:29 schrieb "孙泽嵩" :

Hi Jialin,

Great experiment! Thanks for your sharing.

Looking forward to the function of hot compaction.


Best,
---
Zesong Sun
School of Software, Tsinghua University

孙泽嵩
清华大学 软件学院

> 2020年7月8日 16:39，Jialin Qiao  写道：
> 
> Hi,
> 
> 
> I'd like to share with you some experiment results about how chunk size 
impact the query performance. 
> 
> 
> Hardware: 
> MacBook Pro (Retina, 15-inch, Mid 2015)
> CPU: 2.2 GHz Intel Core i7
> Memory: 16 GB 1600 MHz DDR3
> I use a mobile HDD (SEAGATE, 1TB, Model SRD00F1)  as the storage.
> 
> 
> Workload: 1 storage group, 1 device, 100 measurements in long type. 1 
million data points generated randomly for each time series. 
> 
> 
> A background knowledge is the origin flushed chunk size = 
memtable_size_threshold / series number / byte per data point (16 for long data 
points)
> 
> 
> I adjust the memtable_size_threhold to control the chunk size.
> 
> 
> Configurations of IoTDB:
> 
> 
> enable_parameter_adapter=false
> avg_series_point_number_threshold=1000 (to make the 
memtable_size_threshold valid)
> page_size_in_byte=10 (each chunk has one page)
> tsfile_size_threshold = memtable_size_threshold = 
16/160/1600/16000/16
> 
> 
> I use SessionExample.insertTablet to insert data under different 
configurations. Then I got Chunk sizes from 100 to 100.
> 
> 
> Then I use SessionExample.queryByIterator to iterate the result set of 
"select s1 from root.sg1.d1" without constructing other data structures.
> 
> 
> The results are:
> 
> 
> | chunk size | query time cost in ms |
> |   100  | 47620 |
> |   1000| 13984 |
> |   1  | 2416   |
> |   10| 1322   |
> 
> 
> As we could see the chunk size has a dominate impact to the raw data 
query performance. In the current query engine, Chunk is the basic data unit to 
read from the disk. For reading each Chunk, we need one seek + one IO 
operation. A larger chunk size means less Chunks to read. 
> 
> 
> Therefore, it's better to enlarge the memtable_size_threshold for 
accelerate queries. However, enlarging memtable_size_threshold means more 
memory is needed. This is not always satisfied in some scenes. Therefore, we 
need compaction, either hot compaction triggered in flushing or the timed 
compaction strategy, to compact small chunks to a large one.
> 
> 
> Thanks,
> --
> Jialin Qiao
> School of Software, Tsinghua University
> 
> 乔嘉林
> 清华大学 软件学院

Re: Fetch query result iteratively to avoid OOM

[Julian Feinauer]

Hi Jialin,

this is a very good point, indeed.
I remember in Calcite there was the Linq4j Module which was based on an 
extension of an "Iterable / Iterator" which was basically Lazy Loading / 
Computation.
This object was assembled throughout the query and finally given to the Servers 
handler who could then do all the junking or fetching with the client.

Of course some Implementations require some amount of memory but in most 
situations one could always realize it in a "bounded" way.

Here some details: https://calcite.apache.org/docs/

Julian

Am 09.07.20, 09:17 schrieb "Jialin Qiao" :

Hi,


Currently, Haihang and Yuan Tian added an improvement to fetch the results 
of show timeseries iteratively to avoid OOM [1].


For data queries, we use the QueryDataSet which provides the hasNext & next 
iterator. Then the client could fetch result by fetchSize.


However, for most metadata queries, the ListDataSet is used, which caches 
all results at server. This may cause OOM.
"show timeseries" query is just one case whose results may be very large 
and use the ListDataSet. 


We'd better check other metadata queries, and when you add a new query 
type, please consider its memory consumption.


[1] https://github.com/apache/incubator-iotdb/pull/1470

Thanks,
--
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

Re: [DISCUSS] discuss about graduation of Apache IoTDB (incubating)

[Julian Feinauer]

Hi,

thanks for your feedback Justin and I would also be very interested in hearing 
your opinion on the maturity.
As I'm no mentor and pretty involved in the community know I feel that the 
community is mature enough but as with every young project, there are always 
things to improve.

But I think ist a good point in time now to get some feedback and improve 
specifics that we are still lacking.

Julian

Am 07.07.20, 06:07 schrieb "Justin Mclean" :

Hi,

I think the project have made very good progress, but it might be a little 
bit early to graduate. I’ve have a look over things in a little more detail 
over the next couple of days and see if that changes my mind.

Thanks,
Justin

Courses /Trainings for Open Source Projects

[Julian Feinauer]

Hi folks,

as some may have noticed on other lists Chris, myself and some others are 
starting something together (and also with other Partners from the community!).
The project is called https://industrial-opensource.com and we try to form a 
network of (I)IoT experts from different Open Source projects to help the 
industry.

Our next idea was to provide trainings and courses for the open source projects 
we list on the site (or even more) but we are not yet clear about the interest 
and demand.
And, as we are community driven this explicity INCLUDES free content and NOT 
only paid services. Thus I think ist reasonable to share the little survey we 
prepared on this list also.
The survey should help us to see if there is a demand or what exactly the 
community / communities may need and expect.

So we would be happy if you take the time to fill it out : )

Link to the survey: https://www.surveymonkey.de/r/CKZMTDL

Thanks already!
Julian

Re: refactor the modules (for thrift and antlr code generation)

[Julian Feinauer]

Hi,

good point, I agree.

Julian

Am 29.06.20, 09:03 schrieb "Xiangdong Huang" :

Hi,

> we could also consider migrating to protobuf and grpc then as I like
their maven plugin and it really works flawlessly (and downloads the protoc
automagically).

I think we can postpone the discussion about the replacement after the
cluster module is merged... It will be a disaster if we do that now.

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Julian Feinauer  于2020年6月29日周一 下午2:58写道：

> Hi,
>
> we could also consider migrating to protobuf and grpc then as I like their
> maven plugin and it really works flawlessly (and downloads the protoc
> automagically).
>
> But I would also be fine with extracting it.
>
> Julian
>
> Am 29.06.20, 08:30 schrieb "Xiangdong Huang" :
>
> Hi,
>
> I suffer from this issue for a long time:
>
> When I checkout to some branch where the version is  xxx-SNAPSHOT and
> want
> to run the source codes on the branch in IDE (IDEA), I have to run 
`mvn
> generate-sources` first to generate the thrift and antlr4 codes,
> otherwise,
> the IDEA will say some classes are not found.
>
> However, if the xxx-SNAPSHOT.jar is not on my local maven repo, `mvn
> generate-sources` will fail because the dependency `tsfile.jar` is not
> found...
> Then I have to run `mvn package -DskipTests` and waste several minutes
> for
> waiting the compiler.
>
> I'd like to extract the .thrift file and .g4 file separately into two
> new
> modules (without any other module dependencies except for thrift and
> antlr), so that we can run `mvn generate-sources` successfully and 
save
> time..
>
> how do you think?
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>

Re: refactor the modules (for thrift and antlr code generation)

[Julian Feinauer]

Hi,

we could also consider migrating to protobuf and grpc then as I like their 
maven plugin and it really works flawlessly (and downloads the protoc 
automagically).

But I would also be fine with extracting it.

Julian

Am 29.06.20, 08:30 schrieb "Xiangdong Huang" :

Hi,

I suffer from this issue for a long time:

When I checkout to some branch where the version is  xxx-SNAPSHOT and want
to run the source codes on the branch in IDE (IDEA), I have to run `mvn
generate-sources` first to generate the thrift and antlr4 codes, otherwise,
the IDEA will say some classes are not found.

However, if the xxx-SNAPSHOT.jar is not on my local maven repo, `mvn
generate-sources` will fail because the dependency `tsfile.jar` is not
found...
Then I have to run `mvn package -DskipTests` and waste several minutes for
waiting the compiler.

I'd like to extract the .thrift file and .g4 file separately into two new
modules (without any other module dependencies except for thrift and
antlr), so that we can run `mvn generate-sources` successfully and save
time..

how do you think?

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Re: [discuss] Apache IoTDB 0.10.0 (incubating) RC4 release

[Julian Feinauer]

Hi,

I think virtual keysigning is possible if there is some kind of trust.
I, for example, would feel comfortable enough to sign a key from people I know 
via a Websession.
If official documents are prepared and scanned it can be done.

In the end, everybody has to decide which key he wants to sign and which dont.

Julian

Am 24.06.20, 02:12 schrieb "Justin Mclean" :

Hi,

There’s an option the sign a release with multiple keys but few projects do 
that, and it’s usually just the release manager.

Justin

Re: [VOTE] Apache IoTDB 0.10.0 (incubating) RC4 release

[Julian Feinauer]

Hi all,

my Vote is +1 (binding PPMC Vote and IPMC Vote),

I did:
- Checked Signature and Hashes
- Extracted the archive
- Checked LICENSE, NOTICE, README and RELEASE_NOTES and ist content
- No unexpected binaries
- No Snapshot Dependencies
- Compiled sources according to README
- Run `mvn clean install`

Best
Julian


Am 22.06.20, 08:52 schrieb "勾王敏浩" :

Hi,
 +1 from committer 

I have checked:

source release:
1.Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in 
the extracted source bundle (Correct)
2.Running RAT(mvn apache-rat:check) (Correct)
3.Compile and build code with command(./mvnw install)(Correct)
4.Verifying the signature (ASC) (Correct)
5.Verifying the hashes (SHA512) (Correct)

binary version
1.Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in 
the extracted source bundle (Correct)
2.Verifying the signature (ASC) (Correct)
3.Verifying the hashes (SHA512) (Correct)

functional check in binary version:
set storage group to root.demo
CREATE TIMESERIES root.demo.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
insert into root.demo.d0(time,s0) values(1,1)
insert into root.demo.d0(time,s0) values(2,2)
show timeseries root
select * from root

Best,
Wangminhao Gou


> -原始邮件-
> 发件人: "Xiangdong Huang" 
> 发送时间: 2020-06-17 20:04:38 (星期三)
> 收件人: dev 
> 抄送: 
> 主题: [VOTE] Apache IoTDB 0.10.0 (incubating) RC4 release
> 
> Hi all,
> 
> 
> Welcome  (well, once again and again) to verify the release of Apache 
IoTDB
> (incubating) 0.10.0 (RC4).
> 
> 
> Apache IoTDB (Incubating) 0.10.0 has been staged under [1] and it’s time 
to
> vote
> on accepting it for release.  All Maven artifacts are available under [2].
> 
> 
> RC3 fixes the following issues of RC1, an internal bug in 0.10.0-SNAPSHOT
> and issues in RC2:
> 
> - incorrect version info in CLI.
> 
> - add executable permission on sbin/*, tools/* and conf/iotdb-env.sh 
files.
> 
> - fix some incorrect links in documents.
> 
> - fix the version of the site module.
> 
> 
> The release note can be get from [5]. The data file of v0.10.0 is
> incompatible with v0.9.x, but v0.10.0 can automatically upgrade the data
> file to the latest format.
> 
> If approved we will seek final release approval from the IPMC.
> 
> 
> Voting will be open for 72hr.
> A minimum of 3 binding +1 votes and more binding +1 than binding -1
> are required to pass.
> 
> Release tag: release/0.10.0
> Hash for the release tag: f506d9c731231f97d8813528cbf062760d40
> 
> 
> Please follow the steps in [3]:
> 
> "Before voting +1 [P]PMC members are required to download
> the signed source code package, compile it as provided, and test
> the resulting executable on their own platform, along with also
> verifying that the package meets the requirements of the ASF policy
> on releases."
> 
> You can achieve the above by following [4].
> 
> [ ]  +1 accept (indicate what you validated - e.g. performed the non-RM
> items in [4])
> [ ]  -1 reject (explanation required)
> 
> 
> [1] https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.10.0/rc4/
> 
> [2] https://repository.apache.org/content/repositories/orgapacheiotdb-1038
> [3] https://www.apache.org/dev/release.html#approving-a-release
> [4]
> 
https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release
> [5]
> 
https://dist.apache.org/repos/dist/dev/incubator/iotdb/0.10.0/rc4/RELEASE_NOTES.md
> 

> [6] https://dist.apache.org/repos/dist/dev/incubator/iotdb/KEYS
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
>  黄向东
> 清华大学 软件学院

Re: [Discuss][Mentors pay attention] Maturity Evaluation of IoTDB

[Julian Feinauer]

Hi all,

I am no mentor ut answer here as PPMC and IPMC member.
I think the project has done HUGE progress and I personally am happy and a bit 
surprised on how good and fast we got the PPMC into the right mindset (at least 
the active part oft he PPMC as in many podlings).

There still are issues (like e.g. voting, which Chris also brought up) but the 
project is on a very good way, IMHO.

Julian

Am 15.06.20, 04:21 schrieb "Kevin A. McGrail" :

Since I have reviewed this before the posting, I'd like other mentors to
please review first.
--
Kevin A. McGrail
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171


On Sun, Jun 14, 2020 at 9:38 PM Xiangdong Huang  wrote:

> Hi all,
>
> Apache IoTDB has been incubated for one and a half years.
> I still remember that when IoTDB being an incubator project at first, 
Chris
> asked "where is the code?". At that time, the IoTDB community is far from
> being a qualified Apache community.
>
> Fortunately, we have great mentors, committers, users and other
> contributors.
> With the help of the whole community, I think the IoTDB community has a 
big
> growth.
>
> So, I followed the Apache maturity model [1], and finish a draft of 
IoTDB's
> maturity model.
>
> Please have a review of the model [2].
> It is a guideline to let us know where we need to improve and make the
> community better.
> (And we can know how far we are from the graduation :D)
>
> [1]
> http://community.apache.org/apache-way/apache-project-maturity-model.html
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=148645763
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>

Re: [IOTDB-726] CheckPoint of MTree

[Julian Feinauer]

Yes, we could then also use all cores for deserialization (if thats the 
bottleneck) for reloading all of them.
Or generally store only some K in one file and then open another one then we 
could again take care of parallelism.

J

Am 19.06.20, 11:35 schrieb "Xiangdong Huang" :

> Another thing we could consider is to chunk them according to their
namespaces in folders / files or any other struct.

according to the Storage group names, for example.

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Julian Feinauer  于2020年6月19日周五 下午4:54写道：

> Another thing we could consider is to chunk them according to their
> namespaces in folders / files or any other struct. Then we could
> efficiently do lazy loading and only pick what we really need.
>
> WDYT?
>
> Am 19.06.20, 10:36 schrieb "Xiangdong Huang" :
>
> > I did an experiment for 1M timeseries, and the serialization process
> costs 971ms.
>
> 971ms for Serializing 1M timeseries, but 6 seconds for deserializing?
>
> > I didn’t time this … I’ll do an experiment after fixing the 
suggested
> changes in current PR [1]
>
> The problem of current PR is that your snapshot is larger and larger
> along
> with the system running.
> Any idea about this case?
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  黄向东
> 清华大学 软件学院
>
>
> 孙泽嵩  于2020年6月19日周五 下午2:20写道：
>
> > Wow, thanks, Julian!
> >
> > Let me try and do experiments to get the best result : )
> >
> > Best,
> > ---
> > Zesong Sun
> > School of Software, Tsinghua University
> >
> > 孙泽嵩
> > 清华大学 软件学院
> >
> > > 2020年6月19日 14:14，Julian Feinauer 
> 写道：
> > >
> > > Oh and another note. By using a faster serialization Lib than Java
> > default we could ideally speed up the process up to 10x.
> > >
> > > See eg here https://github.com/RuedigerMoeller/fast-serialization
> > >
> > > Julian
> > >
> > > Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
> > >
> > > 
> > > From: Julian Feinauer 
> > > Sent: Friday, June 19, 2020 8:11:56 AM
> > > To: dev@iotdb.apache.org 
> > > Subject: Re: [IOTDB-726] CheckPoint of MTree
> > >
> > > What about using some kind of cache that spills to disk. That way
> we
> > would be up in no time and just lazy load devices when needed.
> > >
> > > I remember that eh cache has such features (
> > https://www.baeldung.com/ehcache) but there are other
> implementations as
> > well.
> > >
> > > Julian
> > >
> > > Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
> > >
> > > 
> > > From: 孙泽嵩 
> > > Sent: Friday, June 19, 2020 7:57:51 AM
> > > To: dev@iotdb.apache.org 
> > > Subject: Re: [IOTDB-726] CheckPoint of MTree
> > >
> > > Hi Jialin,
> > >
> > > I did an experiment for 1M timeseries, and the serialization
> process
> > costs 971ms.
> > >
> > > Maybe we could consider creating a snapshot when the MTree is not
> > changed for a long time (for example, one hour).
> > >
> > > In this way, the client will not be stuck and users may not even
> notice
> > it.
> > >
> > >
> > > Best,
> > > ---
> > > Zesong Sun
> > > School of Software, Tsinghua University
> > >
> > > 孙泽嵩
> > > 清华大学 软件学院
> > >
> > >> 2020年6月18日 16:19，孙泽嵩  写道：
> > >>
> > >> Hi,
> > >>
> > >> Good opinions!
> &

Re: [IOTDB-726] CheckPoint of MTree

[Julian Feinauer]

Another thing we could consider is to chunk them according to their namespaces 
in folders / files or any other struct. Then we could efficiently do lazy 
loading and only pick what we really need.

WDYT?

Am 19.06.20, 10:36 schrieb "Xiangdong Huang" :

> I did an experiment for 1M timeseries, and the serialization process
costs 971ms.

971ms for Serializing 1M timeseries, but 6 seconds for deserializing?

> I didn’t time this … I’ll do an experiment after fixing the suggested
changes in current PR [1]

The problem of current PR is that your snapshot is larger and larger along
with the system running.
Any idea about this case?

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


孙泽嵩  于2020年6月19日周五 下午2:20写道：

> Wow, thanks, Julian!
>
> Let me try and do experiments to get the best result : )
>
> Best,
> ---
> Zesong Sun
> School of Software, Tsinghua University
>
> 孙泽嵩
    > 清华大学 软件学院
>
> > 2020年6月19日 14:14，Julian Feinauer  写道：
> >
> > Oh and another note. By using a faster serialization Lib than Java
> default we could ideally speed up the process up to 10x.
> >
> > See eg here https://github.com/RuedigerMoeller/fast-serialization
> >
> > Julian
> >
> > Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
> >
> > 
> > From: Julian Feinauer 
> > Sent: Friday, June 19, 2020 8:11:56 AM
> > To: dev@iotdb.apache.org 
> > Subject: Re: [IOTDB-726] CheckPoint of MTree
> >
> > What about using some kind of cache that spills to disk. That way we
> would be up in no time and just lazy load devices when needed.
> >
> > I remember that eh cache has such features (
> https://www.baeldung.com/ehcache) but there are other implementations as
> well.
> >
> > Julian
> >
> > Holen Sie sich Outlook für Android<https://aka.ms/ghei36>
> >
> > 
> > From: 孙泽嵩 
> > Sent: Friday, June 19, 2020 7:57:51 AM
> > To: dev@iotdb.apache.org 
> > Subject: Re: [IOTDB-726] CheckPoint of MTree
> >
> > Hi Jialin,
> >
> > I did an experiment for 1M timeseries, and the serialization process
> costs 971ms.
> >
> > Maybe we could consider creating a snapshot when the MTree is not
> changed for a long time (for example, one hour).
> >
> > In this way, the client will not be stuck and users may not even notice
> it.
> >
> >
> > Best,
> > ---
> > Zesong Sun
> > School of Software, Tsinghua University
> >
> > 孙泽嵩
> > 清华大学 软件学院
> >
> >> 2020年6月18日 16:19，孙泽嵩  写道：
> >>
> >> Hi,
> >>
> >> Good opinions!
> >>
> >>> how about adding a "create snapshot for schema" sql to let users
> trigger this manually
> >>
> >> I’ll add this sql in a new PR.
> >>
> >>> how long it takes to recover from a 1M timeseries snapshot.
> >>
> >> Based on my previous experiment, it takes about 6s as you said.
> >>
> >>> how long it takes to create a snapshot for 1M/10M timeseries?
> >>
> >> I didn’t time this … I’ll do an experiment after fixing the suggested
> changes in current PR [1]
> >>
> >>
> >> [1] https://github.com/apache/incubator-iotdb/pull/1384
> >>
> >>
> >> Best,
> >> ---
> >> Zesong Sun
> >> School of Software, Tsinghua University
> >>
> >> 孙泽嵩
> >> 清华大学 软件学院
> >>
> >>> 2020年6月18日 14:39，Jialin Qiao  写道：
> >>>
> >>> Hi,
> >>>
> >>> Currently, the snapshot is triggered every xxx lines in mlog.txt.
> >>> When meeting 20M timeseries, the default 10k lines will cause too many
> snapshot, which will block the creating.
> >>> However, if we enlarge the condition to 1M, the last 1M will take
> about 6s to recover, about 160K per second.
> >>>
> >>> So, my concern is how long it takes to create a sn

Re: [IOTDB-726] CheckPoint of MTree

[Julian Feinauer]

Oh and another note. By using a faster serialization Lib than Java default we 
could ideally speed up the process up to 10x.

See eg here https://github.com/RuedigerMoeller/fast-serialization

Julian

Holen Sie sich Outlook für Android<https://aka.ms/ghei36>


From: Julian Feinauer 
Sent: Friday, June 19, 2020 8:11:56 AM
To: dev@iotdb.apache.org 
Subject: Re: [IOTDB-726] CheckPoint of MTree

What about using some kind of cache that spills to disk. That way we would be 
up in no time and just lazy load devices when needed.

I remember that eh cache has such features (https://www.baeldung.com/ehcache) 
but there are other implementations as well.

Julian

Holen Sie sich Outlook für Android<https://aka.ms/ghei36>


From: 孙泽嵩 
Sent: Friday, June 19, 2020 7:57:51 AM
To: dev@iotdb.apache.org 
Subject: Re: [IOTDB-726] CheckPoint of MTree

Hi Jialin,

I did an experiment for 1M timeseries, and the serialization process costs 
971ms.

Maybe we could consider creating a snapshot when the MTree is not changed for a 
long time (for example, one hour).

In this way, the client will not be stuck and users may not even notice it.


Best,
---
Zesong Sun
School of Software, Tsinghua University

孙泽嵩
清华大学 软件学院

> 2020年6月18日 16:19，孙泽嵩  写道：
>
> Hi,
>
> Good opinions!
>
>> how about adding a "create snapshot for schema" sql to let users trigger 
>> this manually
>
> I’ll add this sql in a new PR.
>
>> how long it takes to recover from a 1M timeseries snapshot.
>
> Based on my previous experiment, it takes about 6s as you said.
>
>> how long it takes to create a snapshot for 1M/10M timeseries?
>
> I didn’t time this … I’ll do an experiment after fixing the suggested changes 
> in current PR [1]
>
>
> [1] https://github.com/apache/incubator-iotdb/pull/1384
>
>
> Best,
> ---
> Zesong Sun
> School of Software, Tsinghua University
>
> 孙泽嵩
> 清华大学 软件学院
>
>> 2020年6月18日 14:39，Jialin Qiao  写道：
>>
>> Hi,
>>
>> Currently, the snapshot is triggered every xxx lines in mlog.txt.
>> When meeting 20M timeseries, the default 10k lines will cause too many 
>> snapshot, which will block the creating.
>> However, if we enlarge the condition to 1M, the last 1M will take about 6s 
>> to recover, about 160K per second.
>>
>> So, my concern is how long it takes to create a snapshot for 1M/10M 
>> timeseries? And how long it takes to recover from a 1M timeseries snapshot.
>>
>> Besides, how about adding a "create snapshot for schema" sql to let users 
>> trigger this manually?
>>
>> Thanks,
>> --
>> Jialin Qiao
>> School of Software, Tsinghua University
>>
>> 乔嘉林
>> 清华大学 软件学院
>>
>>> -原始邮件-
>>> 发件人: "孙泽嵩" 
>>> 发送时间: 2020-06-15 19:14:08 (星期一)
>>> 收件人: dev@iotdb.apache.org
>>> 抄送:
>>> 主题: Re: [IOTDB-726] CheckPoint of MTree
>>>
>>> Hi Julian,
>>>
>>> Currently I’m just using plain text file.
>>>
>>> But I could consider and try with RocksDB : )
>>> I also noticed that there is an issue related to RocksDB integration [1].
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/IOTDB-767
>>>
>>>
>>> Best,
>>> ---
>>> Zesong Sun
>>> School of Software, Tsinghua University
>>>
>>> 孙泽嵩
>>> 清华大学 软件学院
>>>
>>>> 2020年6月15日 19:00，Julian Feinauer  写道：
>>>>
>>>> Hi Zesong,
>>>>
>>>> this is an excellent Idea!
>>>> Do you serialize the snapshot as plain text file?
>>>> Or would it make sense to use something like RocksDB for something like 
>>>> that?
>>>>
>>>> Julian
>>>>
>>>> Am 15.06.20, 12:12 schrieb "孙泽嵩" :
>>>>
>>>>  Greetings,
>>>>
>>>>  I’m currently working on issue [IOTDB-726] CheckPoint of MTree [1]
>>>>
>>>>  In the situation that there exist a large number of timeseries, it would 
>>>> take a long time to restart IoTDB by reading mlog.txt and executing the 
>>>> commands line by line.
>>>>  For example, it takes about 2 minutes to restart with 20M timeseries.
>>>>
>>>>  To solve this problem, “checkpoint” is designed and added to MTree to 
>>>> reduce the time of reading mlog when IoTDB restarts:
>>>>  Generate a snapshot, which includes the seri

Re: [IOTDB-726] CheckPoint of MTree

[Julian Feinauer]

What about using some kind of cache that spills to disk. That way we would be 
up in no time and just lazy load devices when needed.

I remember that eh cache has such features (https://www.baeldung.com/ehcache) 
but there are other implementations as well.

Julian

Holen Sie sich Outlook für Android<https://aka.ms/ghei36>


From: 孙泽嵩 
Sent: Friday, June 19, 2020 7:57:51 AM
To: dev@iotdb.apache.org 
Subject: Re: [IOTDB-726] CheckPoint of MTree

Hi Jialin,

I did an experiment for 1M timeseries, and the serialization process costs 
971ms.

Maybe we could consider creating a snapshot when the MTree is not changed for a 
long time (for example, one hour).

In this way, the client will not be stuck and users may not even notice it.


Best,
---
Zesong Sun
School of Software, Tsinghua University

孙泽嵩
清华大学 软件学院

> 2020年6月18日 16:19，孙泽嵩  写道：
>
> Hi,
>
> Good opinions!
>
>> how about adding a "create snapshot for schema" sql to let users trigger 
>> this manually
>
> I’ll add this sql in a new PR.
>
>> how long it takes to recover from a 1M timeseries snapshot.
>
> Based on my previous experiment, it takes about 6s as you said.
>
>> how long it takes to create a snapshot for 1M/10M timeseries?
>
> I didn’t time this … I’ll do an experiment after fixing the suggested changes 
> in current PR [1]
>
>
> [1] https://github.com/apache/incubator-iotdb/pull/1384
>
>
> Best,
> ---
> Zesong Sun
> School of Software, Tsinghua University
>
> 孙泽嵩
> 清华大学 软件学院
>
>> 2020年6月18日 14:39，Jialin Qiao  写道：
>>
>> Hi,
>>
>> Currently, the snapshot is triggered every xxx lines in mlog.txt.
>> When meeting 20M timeseries, the default 10k lines will cause too many 
>> snapshot, which will block the creating.
>> However, if we enlarge the condition to 1M, the last 1M will take about 6s 
>> to recover, about 160K per second.
>>
>> So, my concern is how long it takes to create a snapshot for 1M/10M 
>> timeseries? And how long it takes to recover from a 1M timeseries snapshot.
>>
>> Besides, how about adding a "create snapshot for schema" sql to let users 
>> trigger this manually?
>>
>> Thanks,
>> --
>> Jialin Qiao
>> School of Software, Tsinghua University
>>
>> 乔嘉林
>> 清华大学 软件学院
>>
>>> -原始邮件-
>>> 发件人: "孙泽嵩" 
>>> 发送时间: 2020-06-15 19:14:08 (星期一)
>>> 收件人: dev@iotdb.apache.org
>>> 抄送:
>>> 主题: Re: [IOTDB-726] CheckPoint of MTree
>>>
>>> Hi Julian,
>>>
>>> Currently I’m just using plain text file.
>>>
>>> But I could consider and try with RocksDB : )
>>> I also noticed that there is an issue related to RocksDB integration [1].
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/IOTDB-767
>>>
>>>
>>> Best,
>>> ---
>>> Zesong Sun
>>> School of Software, Tsinghua University
>>>
>>> 孙泽嵩
>>> 清华大学 软件学院
>>>
>>>> 2020年6月15日 19:00，Julian Feinauer  写道：
>>>>
>>>> Hi Zesong,
>>>>
>>>> this is an excellent Idea!
>>>> Do you serialize the snapshot as plain text file?
>>>> Or would it make sense to use something like RocksDB for something like 
>>>> that?
>>>>
>>>> Julian
>>>>
>>>> Am 15.06.20, 12:12 schrieb "孙泽嵩" :
>>>>
>>>>  Greetings,
>>>>
>>>>  I’m currently working on issue [IOTDB-726] CheckPoint of MTree [1]
>>>>
>>>>  In the situation that there exist a large number of timeseries, it would 
>>>> take a long time to restart IoTDB by reading mlog.txt and executing the 
>>>> commands line by line.
>>>>  For example, it takes about 2 minutes to restart with 20M timeseries.
>>>>
>>>>  To solve this problem, “checkpoint” is designed and added to MTree to 
>>>> reduce the time of reading mlog when IoTDB restarts:
>>>>  Generate a snapshot, which includes the serialization of MTree, every 
>>>> time mlog reaches a certain number of lines.
>>>>  When a new snapshot is generated, the old one is deleted. Snapshot file 
>>>> and mlog.txt are in the same directory.
>>>>
>>>>  Users could configure the threshold number of the mlog lines. By default, 
>>>> a snapshot is generated for every 100k lines.
>>>>
>>>>  I’ve already made a demo and proved that the method could speed up the 
>>>> restarting process.
>>>>  As for the reading mlog.txt and initializing MTree part, it reduces time 
>>>> by 28.3% (16.6s with origin method, 11.9s with new demo, both for 2M 
>>>> timeseries).
>>>>
>>>>  I would like to make a PR afterwards. If you have any suggestions about 
>>>> the design, feel free to discuss with me.
>>>>
>>>>
>>>>  [1] https://issues.apache.org/jira/browse/IOTDB-726
>>>>
>>>>
>>>>  Best,
>>>>  ---
>>>>  Zesong Sun
>>>>  School of Software, Tsinghua University
>>>>
>>>>  孙泽嵩
>>>>  清华大学 软件学院
>>>>
>>>>
>>>
>

Re: Using Sync Tool with IoTDB Server

[Julian Feinauer]

Hi,

the Testcontainers Project solves this extremely efficient und elegant and has 
an nice integration to Junit.
And Controls the complete lifecycle of the containers:

https://www.testcontainers.org/test_framework_integration/junit_4/

Julian

Am 17.06.20, 10:00 schrieb "Xiangdong Huang" :

Hi  Julian,

> Especially with TestContainers it would be very easy to startup an IoTDB
Server in an container and perform some testing against it.

Yes if we can run iotdb instance in containers for test, it could be fine.

But the concern is that can it work together with JUnit and TravisCI?

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Julian Feinauer  于2020年6月17日周三 下午3:28写道：

> Probabl.y we could try to set up integration tests fort hat.
> Especially with TestContainers it would be very easy to startup an IoTDB
> Server in an container and perform some testing against it.
>
> Julian
>
> Am 17.06.20, 09:13 schrieb "李天安" :
>
> Hi,
>
> In the sync tool, the receiver will compare the versions of IoTDB with
> the sender's and rejects the synchronization task if the versions are
> different.
>
> In the load tool, the engine will check the version number of tsfile
> and will reject the load operation if the version doesn't match with the
> engine's.
>
>
> Best Regards,
> —
> Tianan Li
> School of Software, Tsinghua University
>
> 李天安
> 清华大学 软件学院
>
>
> > -原始邮件-
> > 发件人: "Xiangdong Huang" 
> > 发送时间: 2020-06-17 09:15:03 (星期三)
> > 收件人: dev 
> > 抄送:
> > 主题: Re: Using Sync Tool with IoTDB Server
> >
> > Hi,
> >
> > I have a question, is the load or sync function supports
> communication
> > among different versions (e.g., load tsfile of 0.9 to 0.10.0)? If
> not, is
> > there a check?
> >
> > Best,
> > ---
> > Xiangdong Huang
> > School of Software, Tsinghua University
> >
> >  黄向东
> > 清华大学 软件学院
> >
> >
> > 李天安  于2020年6月17日周三 上午1:27写道：
> >
> > > Hi Julian,
> > >
> > > In sync tool, before data synchronization, all schemas will be
> > > synchronized first, so the schema of each tsfile transmitted is
> guaranteed
> > > to exist.
> > > Synchronous schema adopts incremental method.
> > >
> > >
> > > Best Regards,
> > > —
> > > Tianan Li
> > > School of Software, Tsinghua University
> > >
> > > 李天安
> > > 清华大学 软件学院
> > >
> > > > -原始邮件-
> > > > 发件人: "Julian Feinauer" 
> > > > 发送时间: 2020-06-15 18:36:49 (星期一)
> > > > 收件人: "dev@iotdb.apache.org" 
> > > > 抄送:
> > > > 主题: Using Sync Tool with IoTDB Server
> > > >
> > > > Hi folks,
> > > >
> > > > just a short question regarding the Sync Tool from IoTDB.
> > > > If I have a plain tsfile that I sync via the sync tool to the
> server are
> > > then all names from the timeseries picked automatically and
> entered in the
> > > MTree and all?
> > > > Or do I have to configure something upfront?
> > > >
> > > > Thanks!
> > > > Julian
> > >
>
>

Re: Using Sync Tool with IoTDB Server

[Julian Feinauer]

Probabl.y we could try to set up integration tests fort hat.
Especially with TestContainers it would be very easy to startup an IoTDB Server 
in an container and perform some testing against it.

Julian

Am 17.06.20, 09:13 schrieb "李天安" :

Hi,

In the sync tool, the receiver will compare the versions of IoTDB with the 
sender's and rejects the synchronization task if the versions are different.

In the load tool, the engine will check the version number of tsfile and 
will reject the load operation if the version doesn't match with the engine's.


Best Regards,
—
Tianan Li
School of Software, Tsinghua University

李天安
清华大学 软件学院


> -原始邮件-
> 发件人: "Xiangdong Huang" 
> 发送时间: 2020-06-17 09:15:03 (星期三)
> 收件人: dev 
> 抄送: 
> 主题: Re: Using Sync Tool with IoTDB Server
> 
> Hi,
> 
> I have a question, is the load or sync function supports communication
> among different versions (e.g., load tsfile of 0.9 to 0.10.0)? If not, is
> there a check?
> 
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
> 
>  黄向东
> 清华大学 软件学院
> 
> 
> 李天安  于2020年6月17日周三 上午1:27写道：
> 
> > Hi Julian,
> >
> > In sync tool, before data synchronization, all schemas will be
> > synchronized first, so the schema of each tsfile transmitted is 
guaranteed
> > to exist.
> > Synchronous schema adopts incremental method.
> >
> >
> > Best Regards,
> > —
    > > Tianan Li
> > School of Software, Tsinghua University
> >
> > 李天安
> > 清华大学 软件学院
> >
> > > -原始邮件-
> > > 发件人: "Julian Feinauer" 
> > > 发送时间: 2020-06-15 18:36:49 (星期一)
> > > 收件人: "dev@iotdb.apache.org" 
> > > 抄送:
> > > 主题: Using Sync Tool with IoTDB Server
> > >
> > > Hi folks,
> > >
> > > just a short question regarding the Sync Tool from IoTDB.
> > > If I have a plain tsfile that I sync via the sync tool to the server 
are
> > then all names from the timeseries picked automatically and entered in 
the
> > MTree and all?
> > > Or do I have to configure something upfront?
> > >
> > > Thanks!
> > > Julian
> >

Re: [IOTDB-726] CheckPoint of MTree

[Julian Feinauer]

Hey,

I just wonder i fit makes sense to rely on something like RocksDB e.g. for 
inverted Indexes and fast lookups.
What do you currently use to serialize / deserialize the MTree? 
There are different methods with big performance differences, I remember.

Julian

Am 15.06.20, 13:14 schrieb "孙泽嵩" :

Hi Julian,

Currently I’m just using plain text file.

But I could consider and try with RocksDB : )
I also noticed that there is an issue related to RocksDB integration [1].


[1] https://issues.apache.org/jira/browse/IOTDB-767


Best,
---
Zesong Sun
School of Software, Tsinghua University

孙泽嵩
清华大学 软件学院

> 2020年6月15日 19:00，Julian Feinauer  写道：
> 
> Hi Zesong,
> 
> this is an excellent Idea!
> Do you serialize the snapshot as plain text file?
> Or would it make sense to use something like RocksDB for something like 
that?
> 
> Julian
> 
> Am 15.06.20, 12:12 schrieb "孙泽嵩" :
> 
>Greetings,
> 
>I’m currently working on issue [IOTDB-726] CheckPoint of MTree [1]
> 
>In the situation that there exist a large number of timeseries, it 
would take a long time to restart IoTDB by reading mlog.txt and executing the 
commands line by line.
>For example, it takes about 2 minutes to restart with 20M timeseries.
> 
>To solve this problem, “checkpoint” is designed and added to MTree to 
reduce the time of reading mlog when IoTDB restarts: 
>Generate a snapshot, which includes the serialization of MTree, every 
time mlog reaches a certain number of lines.
>When a new snapshot is generated, the old one is deleted. Snapshot 
file and mlog.txt are in the same directory.
> 
>Users could configure the threshold number of the mlog lines. By 
default, a snapshot is generated for every 100k lines.
> 
>I’ve already made a demo and proved that the method could speed up the 
restarting process.
>As for the reading mlog.txt and initializing MTree part, it reduces 
time by 28.3% (16.6s with origin method, 11.9s with new demo, both for 2M 
timeseries).
> 
>I would like to make a PR afterwards. If you have any suggestions 
about the design, feel free to discuss with me.
> 
> 
>[1] https://issues.apache.org/jira/browse/IOTDB-726
> 
> 
>Best,
>---
>Zesong Sun
>School of Software, Tsinghua University
> 
>孙泽嵩
>清华大学 软件学院
> 
> 

Re: [IOTDB-726] CheckPoint of MTree

[Julian Feinauer]

Hi Zesong,

this is an excellent Idea!
Do you serialize the snapshot as plain text file?
Or would it make sense to use something like RocksDB for something like that?

Julian

Am 15.06.20, 12:12 schrieb "孙泽嵩" :

Greetings,

I’m currently working on issue [IOTDB-726] CheckPoint of MTree [1]

In the situation that there exist a large number of timeseries, it would 
take a long time to restart IoTDB by reading mlog.txt and executing the 
commands line by line.
For example, it takes about 2 minutes to restart with 20M timeseries.

To solve this problem, “checkpoint” is designed and added to MTree to 
reduce the time of reading mlog when IoTDB restarts: 
Generate a snapshot, which includes the serialization of MTree, every time 
mlog reaches a certain number of lines.
When a new snapshot is generated, the old one is deleted. Snapshot file and 
mlog.txt are in the same directory.

Users could configure the threshold number of the mlog lines. By default, a 
snapshot is generated for every 100k lines.

I’ve already made a demo and proved that the method could speed up the 
restarting process.
As for the reading mlog.txt and initializing MTree part, it reduces time by 
28.3% (16.6s with origin method, 11.9s with new demo, both for 2M timeseries).

I would like to make a PR afterwards. If you have any suggestions about the 
design, feel free to discuss with me.


[1] https://issues.apache.org/jira/browse/IOTDB-726


Best,
---
Zesong Sun
School of Software, Tsinghua University

孙泽嵩
清华大学 软件学院

Using Sync Tool with IoTDB Server

[Julian Feinauer]

Hi folks,

just a short question regarding the Sync Tool from IoTDB.
If I have a plain tsfile that I sync via the sync tool to the server are then 
all names from the timeseries picked automatically and entered in the MTree and 
all?
Or do I have to configure something upfront?

Thanks!
Julian

Re: JSON Input for IoTDB

[Julian Feinauer]

Hey Jialin,

thanks for your response.
I indeed took the second path you suggested.

Currently I try to implement it like Type Erasure in Java, so at PlanExecutor 
level a mapping is done to "plain IoTDB types" and ist a regular insert. Only 
the PlanExecutor may know that it was a structure before.

This makes it easy to implement.
So we could add another "type" based structure if we like and parse it at that 
level.

See my comments here: 
https://issues.apache.org/jira/browse/IOTDB-742?focusedCommentId=17134154=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17134154

Julian

Am 10.06.20, 10:22 schrieb "Jialin Qiao" :

Hi,

Good idea! This may help IoTDB manage GPS data or other semi-structured 
data.

The mapping for json data looks good to me.

For arrays, as each tuple in the array shares the same timestamp, if we 
create the number of array-length's timeseries, it will store duplicated 
timestamps many times. Maybe we could consider to convert the array into a 
Binary or extend the TSDataType and TsFile to support array natively.

Besides, it also depends on the query pattern. 
How will the users query the array? Will the query like "select array[1] 
from root.sg.d" or "select array from root.sg.d"?

Thanks,
--
Jialin Qiao
School of Software, Tsinghua University

乔嘉林
清华大学 软件学院

> -原始邮件-
> 发件人: "Julian Feinauer" 
> 发送时间: 2020-06-09 15:22:26 (星期二)
> 收件人: "dev@iotdb.apache.org" 
> 抄送: 
> 主题: JSON Input for IoTDB
> 
> Hi folks,
> 
> I already created Issue https://issues.apache.org/jira/browse/IOTDB-742 
in this direction but wanted to discuss a topic.
> Since it is now possible to have measurements and devices below a 
measurement we could do a pretty one to one mapping between JSON (or other 
strucuted data) and IoTDB Representation.
> 
> E.g.
> 
> {
>   „temp“ : 20.0,
>   „speed“: 100,
>   „design“ : {
> „color“: „blue“
>   }
> }
> 
> Could be inserted into a FIELD mycar and would then just be the series
> 
> - root.sg.dev.mycar.temp -> 20.0
> - root.sg.dev.mycar.speed -> 100.0
> - root.sg.dev.mycar.design.color -> „blue“
> 
> This works as long as there are no arrays.
> For arrays I see two possibilities.
> Either store them as „2 series“:
> 
> Root.sg.dev.mycar._idx
> Root.sg.dev.mycar.arrayvalue
> 
> With [1, 2, 4] being represented as
> 
> Root.sg.dev.mycar._idx -> 0, 1, 2
> Root.sg.dev.mycar.arrayvalue -> 1, 2, 4
> (all with equal timestamp)
> 
> Or we have a special naming convention e.g. for an array
> 
> {
>   „a“ : [1, 2, 4]
> }
> 
> We would map it to three series
> 
> Root.sg.dev.mycar.a_0 <- 1
> Root.sg.dev.mycar.a_1 <- 2
> Root.sg.dev.mycar.a_2 <- 4
> 
> What do you think about that?
> 
> Julian
> 
> 
> 
> 

Re: [Discuss]A demo site running iotdb

[Julian Feinauer]

Hi,

I think this ist he best way for sure.
In the meantime or if this does now work out I can also offer to provide 
ressources from our company, if needed.

Julian

Am 11.06.20, 08:28 schrieb "Justin Mclean" :

Hi,

> We have applied a VM already.  iotdb-vm.apache.org.

Excellent plan and sorry I missed that.

Thanks,
Justin

JSON Input for IoTDB

[Julian Feinauer]

Hi folks,

I already created Issue https://issues.apache.org/jira/browse/IOTDB-742 in this 
direction but wanted to discuss a topic.
Since it is now possible to have measurements and devices below a measurement 
we could do a pretty one to one mapping between JSON (or other strucuted data) 
and IoTDB Representation.

E.g.

{
  „temp“ : 20.0,
  „speed“: 100,
  „design“ : {
„color“: „blue“
  }
}

Could be inserted into a FIELD mycar and would then just be the series

- root.sg.dev.mycar.temp -> 20.0
- root.sg.dev.mycar.speed -> 100.0
- root.sg.dev.mycar.design.color -> „blue“

This works as long as there are no arrays.
For arrays I see two possibilities.
Either store them as „2 series“:

Root.sg.dev.mycar._idx
Root.sg.dev.mycar.arrayvalue

With [1, 2, 4] being represented as

Root.sg.dev.mycar._idx -> 0, 1, 2
Root.sg.dev.mycar.arrayvalue -> 1, 2, 4
(all with equal timestamp)

Or we have a special naming convention e.g. for an array

{
  „a“ : [1, 2, 4]
}

We would map it to three series

Root.sg.dev.mycar.a_0 <- 1
Root.sg.dev.mycar.a_1 <- 2
Root.sg.dev.mycar.a_2 <- 4

What do you think about that?

Julian

Re: [Discuss] Release of 0.10.0

[Julian Feinauer]

Hi,

I suggest to always to a release branch as best practice as it always help you 
if you need to create a fix release like 0.10.1 (if you want to keep working on 
features on master branch).
But yes I agree, we should try to merge as many PRs as possible.

Are there any features left that one NEEDS in 0.10 that are currently in work?

Julian

Am 20.05.20, 08:29 schrieb "Xiangdong Huang" :

Hi,

The grafana issue has been fixed.

As we have so many PRs opened now, we need to check when to fork a new
branch rel/0.10.

Actually, only one PR needs to be pay attention [1].

IMO, I want to merge it into the master ASAP to avoid more conflicts.
Otherwise, there will be  too many differences between rel/0.10 and master.
But as it modified many codes and not all of them are tested, I am afraid
whether it brings unstable factors.

[1] https://github.com/apache/incubator-iotdb/pull/1169

Best,
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


    Julian Feinauer  于2020年5月18日周一 下午6:44写道：

> Hey,
>
> that looks pretty cool indeed.
> Has anyone tested the IoTDB Grafana Bridge?
> I had issues with it at some point in master branch, not sure about the
> latest version?
>
> But sounds really cool!
>
> Julian
>
> Am 18.05.20, 12:08 schrieb "Haonan Hou" :
>
> Hi,
>
> Right now the upgrade tool for upgrading IoTDB v0.9.x to v0.10 has
> been developed. We have also finished the tests about it and everything
> works great. I think it’s the right time to release the v0.10.0.
>
> Anyone hope to be the RM of 0.10.0?
>
> Best,
> Haonan Hou
>
>

Re: Demo web site proposal.

[Julian Feinauer]

Hi Giorgio,

this is indeed a neat idea... let me have a look later perhaps we can package 
everything together with a ready to use Grafana Dashboard.

Will try to have a look this evening (CEST).

Julian

Am 18.05.20, 16:13 schrieb "Giorgio Zoppi" :

Hi all,

I have seen the current demo trial website.

It will be nice to add COVID-19 data time series data and show with 
grafana what happens.

Julian, could you help with this?

https://github.com/ohad-israeli/covid19-timeseries

Please let me know if you need some help.

BR,

Giorgio

Re: [Discuss] Release of 0.10.0

[Julian Feinauer]

Hey,

that looks pretty cool indeed.
Has anyone tested the IoTDB Grafana Bridge? 
I had issues with it at some point in master branch, not sure about the latest 
version?

But sounds really cool!

Julian

Am 18.05.20, 12:08 schrieb "Haonan Hou" :

Hi, 

Right now the upgrade tool for upgrading IoTDB v0.9.x to v0.10 has been 
developed. We have also finished the tests about it and everything works great. 
I think it’s the right time to release the v0.10.0. 

Anyone hope to be the RM of 0.10.0?

Best,
Haonan Hou