date:20220622

回复: Release of new cluster version

2022-06-22 Thread 冯庆新

+1 for 0.14.0-preview

发件人: Eric Pai
发送时间: 2022年6月22日 16:56
收件人: dev@iotdb.apache.org
主题: Re: Release of new cluster version

Good news! How about '0.14.0-preview' ? As in general development we always use
*-alpha as an internal test version with all main features present. As the
description of the version, it mainly contains the new cluster.

在 2022/6/22 16:44，“Jialin Qiao” 写入:

Hi all,

It has been 4~5 months since we started the new cluster version. Thanks for
the contribution of all contributors!

It's time to start the topic of releasing :)

The summary status of the master branch is as follows:

(1) The new cluster version has many advantages compared to the old
one(0.12~0.13).

* MPP Query Engine, we use volcano module with operators for better
extendibility in data processing.
* Configurable consensus protocol (Standalone, Raft, Multi-Leader) for
Partition, Schema and Data.
* Controlable partition table, so that we could control which node to
manage data.
* No whole cluster consensus group.
* More flexible and lightweight scale-out, we could add a node in a few
seconds without moving data.
* Extensible load balance strategy.
* Built-in metric framework to monitor the status of the cluster.

(2) We have implement the basic function (read/write/schema/udf) of the new
Cluster/standalone version.

https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fmindnotes%2FbmncnEr7XPzmPvrZgYbcSEvMTIf%23mindmapdata=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=HpaVs90uuGUsA4MhtHGyIltGwexNWcITWmcINs00rN8%3Dreserved=0

There are still some functions that need to be implemented in new
Cluster/standalone.
* Trigger
* Schema Template
* Continuous Query
* Select into
* Sync framework

(3) The performance optimization of the new Cluster is still going.

https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fdocx%2Fdoxcn6xEgTopkvsLsG3TXjwlx0cdata=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=YDO5juXfrC0LZDmYJovmgAHw6LxZJANbLFcyobt4FOM%3Dreserved=0

* Write performance of one replication with standalone/multiLeader in new
cluster. [ok].
* Write performance of one replication with Ratis. [underway]
* Write performance of three replication with Ratis/MultiLeader. [underway]
* Query performance of one replication with standalone/ratis/multileader.
[underway]
* Query performance of three replication with Ratis/MultiLeader. [underway]

I suppose we could get a not-bad number by this weekend for all above
scenaios.

(4) More test needs to be done to improve the stability of the new
cluster/standalone, and the ITs are being added.

As this version does not contains the whole functions of 0.13, and the
system performance and stability need to be improved for some time before
produnction, I suggest retaining the 0.14.0 version number.

So, how about using 【0.14.0-alpha】, and we only release the binary
distribution of new cluster, without new/old standalone version.

Then, for the formal 0.14.0, we could release the new Cluster/Standalone
version and discard the old Standalone version.

Thanks,
—
Jialin Qiao
Apache IoTDB PMC

Re: Re: Support new path pattern: 0 or more layers

2022-06-22 Thread Guanfei Guo

In summary, we can get the following syntax:
*{start, end} - start (included) to end (excluded) layers
*{ , end} - 0 to end (excluded) layers
*{start, } - start (included) or more layers, ** stands for *{1, }
*{ , } - 0 or more layers, maybe we can use a shortcut (such as ***)?
*{n} - exactly n layers, * stands for *{1}



Thanks.
Guanfei Guo
 
From: Eric Pai
Date: 2022-06-21 17:13
To: dev@iotdb.apache.org
Subject: Re: Support new path pattern: 0 or more layers
I think we may meet the requirements of matching 0-1 level in the future, then 
*** may be not a good design.
As a single * stands for exactly 1 level now, we can define *{start, end} to 
[start, end) levels, {x} to exactly x levels and {*} to arbitrary levels, where 
** is just a short representation of *{*}.
So it's easy to represent 0 or more layers using *{0, *}.
 
在 2022/6/21 14:44，“Xiangdong Huang” 写入:
 
Hi,
 
- if only for 0 or more layers, I opt for ***
- if we want to define a clear range of layers, I opt for *{start, end} or
**{start, end}
 
---
Xiangdong Huang
School of Software, Tsinghua University
 
 黄向东
清华大学 软件学院
 
 
guoguan...@qq.com.INVALID  于2022年6月21日周二 10:14写道：
 
> Hi guys!
>  Now IoTDB supports the following path patterns:
> * (1 asterisk) - one layer
> ** (2 asterisks) - one or more layers
>
>We need to support a new path pattern: 0 or one or more layers, there
> are some optional schemes:
>Scheme 1, support it directly with a specific syntax:
>   *** (3 asterisks) -  0 or one or more layers
>
>Scheme 2, use more common syntax:
>  a special symbol (such as $, or someone else) - 0 or 1 layer
> $* - 0 or 1 or more layers
>
>  JIRA:   
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FIOTDB-3450data=05%7C01%7C%7Ccbc22404c72843af731208da53518d0d%7C84df9e7fe9f640afb435%7C1%7C0%7C637913906992358086%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=aiZyCx7gbDsIaiVP37%2BB6vc3s6kymBDiw%2FqmdB3igLQ%3Dreserved=0
>
> Any ideas?
>
>
>
> guoguan...@qq.com
>

回复： Release of new cluster version

2022-06-22 Thread CloudWise-Luke

+1 for '0.14.0-preview'

CloudWiseluke.miao

--原始邮件--
发件人:
"dev"
https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fmindnotes%2FbmncnEr7XPzmPvrZgYbcSEvMTIf%23mindmapamp;data=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Camp;sdata=HpaVs90uuGUsA4MhtHGyIltGwexNWcITWmcINs00rN8%3Damp;reserved=0

There are still some functions that need to
be implemented in new
Cluster/standalone.
* Trigger
* Schema Template
* Continuous Query
* Select into
* Sync framework

(3) The performance optimization of the
new Cluster is still going.

https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fdocx%2Fdoxcn6xEgTopkvsLsG3TXjwlx0camp;data=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Camp;sdata=YDO5juXfrC0LZDmYJovmgAHw6LxZJANbLFcyobt4FOM%3Damp;reserved=0

* Write performance of one replication with
standalone/multiLeader in
new
cluster. [ok].
* Write performance of one replication with
Ratis. [underway]
* Write performance of three replication with
Ratis/MultiLeader.
[underway]
* Query performance of one replication with
standalone/ratis/multileader.
[underway]
* Query performance of three replication with
Ratis/MultiLeader.
[underway]

I suppose we could get a not-bad number by
this weekend for all above
scenaios.

(4) More test needs to be done to improve the
stability of the new
cluster/standalone, and the ITs are being
added.

As this version does not contains the whole
functions of 0.13, and the
system performance and stability need to be
improved for some time
before
produnction, I suggest retaining the 0.14.0
version number.

So, how about using 【0.14.0-alpha】, and we
only release the binary
distribution of new cluster, without new/old
standalone version.

Then, for the formal 0.14.0, we could release
the new
Cluster/Standalone
version and discard the old Standalone
version.

Thanks,
—
Jialin Qiao
Apache IoTDB PMC

Re: Release of new cluster version

2022-06-22 Thread Xiangdong Huang

+1 for  0.14.0-preview, and propose v0.13.1 and v0.12.6 at the same time.

---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院

Jialin Qiao  于2022年6月22日周三 17:12写道：
>
> Hi,
>
> +1 for '0.14.0-preview'
>
> I even want to use -beta previously :D
>
> Thanks,
> —
> Jialin Qiao
> Apache IoTDB PMC
>
>
> Eric Pai  于2022年6月22日周三 16:55写道：
>
> > Good news! How about '0.14.0-preview' ? As in general development we
> > always use *-alpha as an internal test version with all main features
> > present. As the description of the version, it mainly contains the new
> > cluster.
> >
> > 在 2022/6/22 16:44，“Jialin Qiao” 写入:
> >
> > Hi all,
> >
> > It has been 4~5 months since we started the new cluster version.
> > Thanks for
> > the contribution of all contributors!
> >
> > It's time to start the topic of releasing :)
> >
> > The summary status of the master branch is as follows:
> >
> > (1) The new cluster version has many advantages compared to the old
> > one(0.12~0.13).
> >
> > * MPP Query Engine, we use volcano module with operators for better
> > extendibility in data processing.
> > * Configurable consensus protocol (Standalone, Raft, Multi-Leader) for
> > Partition, Schema and Data.
> > * Controlable partition table, so that we could control which node to
> > manage data.
> > * No whole cluster consensus group.
> > * More flexible and lightweight scale-out, we could add a node in a few
> > seconds without moving  data.
> > * Extensible load balance strategy.
> > * Built-in metric framework to monitor the status of the cluster.
> >
> > (2) We have implement the basic function (read/write/schema/udf) of
> > the new
> > Cluster/standalone version.
> >
> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fmindnotes%2FbmncnEr7XPzmPvrZgYbcSEvMTIf%23mindmapdata=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=HpaVs90uuGUsA4MhtHGyIltGwexNWcITWmcINs00rN8%3Dreserved=0
> >
> > There are still some functions that need to be implemented in new
> > Cluster/standalone.
> > * Trigger
> > * Schema Template
> > * Continuous Query
> > * Select into
> > * Sync framework
> >
> > (3) The performance  optimization of the new Cluster is still going.
> >
> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fdocx%2Fdoxcn6xEgTopkvsLsG3TXjwlx0cdata=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=YDO5juXfrC0LZDmYJovmgAHw6LxZJANbLFcyobt4FOM%3Dreserved=0
> >
> > * Write performance of one replication with standalone/multiLeader in
> > new
> > cluster. [ok].
> > * Write performance of one replication with Ratis. [underway]
> > * Write performance of three replication with Ratis/MultiLeader.
> > [underway]
> > * Query performance of one replication with
> > standalone/ratis/multileader.
> > [underway]
> > * Query performance of three replication with Ratis/MultiLeader.
> > [underway]
> >
> > I suppose we could get a not-bad number by this weekend for all above
> > scenaios.
> >
> > (4) More test needs to be done to improve the stability of the new
> > cluster/standalone, and the ITs are being added.
> >
> > As this version does not contains the whole functions of 0.13, and the
> > system performance and stability need to be improved for some time
> > before
> > produnction, I suggest retaining the 0.14.0 version number.
> >
> > So, how about using 【0.14.0-alpha】, and we only release the binary
> > distribution of new cluster, without new/old standalone version.
> >
> > Then, for the formal 0.14.0, we could release the new
> > Cluster/Standalone
> > version and discard the old Standalone version.
> >
> > Thanks,
> > —
> > Jialin Qiao
> > Apache IoTDB PMC
> >
> >

??discuss?? Added "Modify Time Series Encoding and Compression Type" interface/command

2022-06-22 Thread ???????????

Hi,




Want to do:

Added "Modify Time Series Encoding and Compression Type" interface/command




Application scenarios:

In IoTDB application projects, reasonable setting of encoding and compression 
algorithms can effectively reduce disk space occupancy and reduce server costs 
in disguise. Modifying encoding and compression algorithms is an ideal method.




Encoding and compression type data location:

1. Metadata - File

2. Metadata - Memory

3. tsfile file - unsealed

4. tsfile file - sealed

5. Wirter in memory

6. The reader in memory

7. Files and memory in merge

7. Others in memory to be added...




Divided by data segment:

Sealed historical tsfile file data - unsealed tsfile file data - incremental 
data




plan selection:

Option 1: The modification command only names incremental data, and provides 
historical data modification commands at the same time

Advantages: fast execution of real-time commands, little impact on online 
operation

Disadvantage: If the prompt is inappropriate, the user may mistakenly think 
that all data has been changed by executing the first command




Option 2: The modification command can modify incremental data and historical 
data at the same time

Advantages: One command completes all operations, immediate results, and the 
reduction of disk space usage can be directly observed

Disadvantages: The execution time of a single command is too long, which has a 
great impact on the online, and needs to be controlled




List of possible tasks:

1. When querying and reading tsfile, use the encoding/compression type in the 
chunk header (to be investigated)

2. Modify metadata, including memory and files, so that incremental data 
directly uses the latest encoding/compression type

3. Modify the related Decoder Encoder Compressor in memory

4. Modify the historical tsfile file, the original tsfile-temporary tsfile, 
do the switching operation after the writing is completed, and write in the 
positive sequence

5. When the command is executed, the sealing operation is performed 
immediately, and the sealed file will be rewritten when the history file is 
modified (to be discussed)

6. If currently in merge, reject the command? (to be discussed)

7. Interface development, first modify the part that affects incremental data, 
and then modify the historical tsfile file

8. SQL development (does not support the first version?)

9. Service restart command recovery operation requires persistent execution 
process

10. Perform pre- and post-checking, possibly saving a summary of each tsfile 
before and after modification for verification




Possibly noteworthy:

1. Lock control

2. Concurrency control

3. Permission control




question:

1. If this is feasible, which version is better to develop in, 12/13/14




Best, ---

Pengfei Liu

Re: Release of new cluster version

2022-06-22 Thread Jialin Qiao

Hi,

+1 for '0.14.0-preview'

I even want to use -beta previously :D

Thanks,
—
Jialin Qiao
Apache IoTDB PMC


Eric Pai  于2022年6月22日周三 16:55写道：

> Good news! How about '0.14.0-preview' ? As in general development we
> always use *-alpha as an internal test version with all main features
> present. As the description of the version, it mainly contains the new
> cluster.
>
> 在 2022/6/22 16:44，“Jialin Qiao” 写入:
>
> Hi all,
>
> It has been 4~5 months since we started the new cluster version.
> Thanks for
> the contribution of all contributors!
>
> It's time to start the topic of releasing :)
>
> The summary status of the master branch is as follows:
>
> (1) The new cluster version has many advantages compared to the old
> one(0.12~0.13).
>
> * MPP Query Engine, we use volcano module with operators for better
> extendibility in data processing.
> * Configurable consensus protocol (Standalone, Raft, Multi-Leader) for
> Partition, Schema and Data.
> * Controlable partition table, so that we could control which node to
> manage data.
> * No whole cluster consensus group.
> * More flexible and lightweight scale-out, we could add a node in a few
> seconds without moving  data.
> * Extensible load balance strategy.
> * Built-in metric framework to monitor the status of the cluster.
>
> (2) We have implement the basic function (read/write/schema/udf) of
> the new
> Cluster/standalone version.
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fmindnotes%2FbmncnEr7XPzmPvrZgYbcSEvMTIf%23mindmapdata=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=HpaVs90uuGUsA4MhtHGyIltGwexNWcITWmcINs00rN8%3Dreserved=0
>
> There are still some functions that need to be implemented in new
> Cluster/standalone.
> * Trigger
> * Schema Template
> * Continuous Query
> * Select into
> * Sync framework
>
> (3) The performance  optimization of the new Cluster is still going.
>
> https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fdocx%2Fdoxcn6xEgTopkvsLsG3TXjwlx0cdata=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=YDO5juXfrC0LZDmYJovmgAHw6LxZJANbLFcyobt4FOM%3Dreserved=0
>
> * Write performance of one replication with standalone/multiLeader in
> new
> cluster. [ok].
> * Write performance of one replication with Ratis. [underway]
> * Write performance of three replication with Ratis/MultiLeader.
> [underway]
> * Query performance of one replication with
> standalone/ratis/multileader.
> [underway]
> * Query performance of three replication with Ratis/MultiLeader.
> [underway]
>
> I suppose we could get a not-bad number by this weekend for all above
> scenaios.
>
> (4) More test needs to be done to improve the stability of the new
> cluster/standalone, and the ITs are being added.
>
> As this version does not contains the whole functions of 0.13, and the
> system performance and stability need to be improved for some time
> before
> produnction, I suggest retaining the 0.14.0 version number.
>
> So, how about using 【0.14.0-alpha】, and we only release the binary
> distribution of new cluster, without new/old standalone version.
>
> Then, for the formal 0.14.0, we could release the new
> Cluster/Standalone
> version and discard the old Standalone version.
>
> Thanks,
> —
> Jialin Qiao
> Apache IoTDB PMC
>
>

Re: Release of new cluster version

2022-06-22 Thread Eric Pai