AW: version about odbc

[Christofer Dutz]

The ODBC is not really a “compilable” example.
It contains a description how you can use a commercial product to wrap our JDBC 
driver in order to produce an ODBC driver.
Also does it contain a compilable C# project file, that demonstates the usage 
of the ODBC API.

However even if the example would be compiled as part of the build, you would 
still need to manually prepare the ODBC wrapper.

I would consider this module more documentation than “compilable code”.

Chris


Von: Yuan Tian 
Datum: Freitag, 7. Juni 2024 um 12:14
An: dev 
Betreff: version about odbc
Hi all,

I found that in master branch, we've already upgraded the version to 
1.3.3-SNAPSHOT, but the examples about odbc module is missed?

[cid:ii_lx4j3xy50]

Best regards,
-
Yuan Tian

version about odbc

[Yuan Tian]

Hi all,

I found that in master branch, we've already upgraded the version to
1.3.3-SNAPSHOT, but the examples about odbc module is missed?

[image: image.png]

Best regards,
-
Yuan Tian

Re: Prepare releasing 1.3.2?

[Haonan Hou]

Ok, the version of master branch has updated to 1.3.3-SNAPSHOT.
Once Ratis 3.1.0 released, I'll prepare the rc/1.3.2 branch. 

Best,
Haonan


On 2024/06/03 02:07:55 Xinyu Tan wrote:
> Hi haonan
> 
> +1 for bumping the version of master branch to 1.3.3-SNAPSHOT.
> 
> As for the 1.3.2 release, currently ratis is releasing there new version 
> 3.1.0[1], we can rely on the official Ratis 3.1.0 release in the open source 
> version IoTDB 1.3.2 instead of the snapshot release to conform to the 
> guidelines of the Apache community.
> 
> Best
> -
> Xinyu Tan
> 
> [1] https://lists.apache.org/thread/q8gdfz1j1l4qv1v24z4dxcg01j9qs28n
> 
> On 2024/06/03 01:44:17 Haonan Hou wrote:
> > Hi all,
> > 
> > IoTDB v1.3.1 has released over 1 month. In this month, we did a lot over 
> > bug fixes and improvements. I think we can start the process of releasing 
> > new version.
> > 
> > If we all agree, I'm going to checkout the rc/1.3.2 branch and bump the 
> > version of master branch to 1.3.3-SNAPSHOT.
> > 
> > BR,
> > Haonan
> > 
> 

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[ITAMI SHO]

Dear Junzhi Peng and Sicheng Yu,

Thank you for sharing the exciting news about the new consensus protocols, 
FastIoTConsensus and IoTV2Consensus. Your efforts and contributions to 
improving the IoTDB project are greatly appreciated. We look forward to 
exploring these new features and providing feedback.

Best regards,

Itami Sho

2024年6月3日 18:28，Junzhi Peng  写道：

Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng

Re: [Proposal] Device Level TTL for IoTDB

[Xinyu Tan]

Hi Peichen

I'm very excited to see the implementation of device-level TTL. Before your 
work, if users wanted to set different TTLs for different devices, they had to 
model them into many databases, which was very inconvenient. After your work, 
users can set different TTLs within a single database as they wish, greatly 
enhancing the user experience for those users with TTL needs.

I am thrilled to see your work being merged and look forward to it creating 
greater value for users.

Best

Xinyu Tan

On 2024/06/03 14:10:05 BensonChou wrote:
> Hello, everyone.
> 
> I am BensonChou(Peichen Zhou), a committer of IoTDB. Over the past few 
> months, I have implemented a new feature, which is device-level TTL, and I am 
> so excited to share this new feature with you.
> 
> Previously, IoTDB only supported database-level TTL. For devices with 
> different collection frequencies and data validity periods under a single 
> database, we want to use a more granular device-level TTL to manage the data 
> lifecycle and optimize disk space as much as possible.
> 
> The new version of IoTDB will support device-level TTL management, allowing 
> users to set, modify, and unset TTLs in bulk. When setting TTL, the system 
> will apply it to all devices that match the specified path. If multiple TTLs 
> exist on a device path, the TTL closest to the device node will take effect. 
> Once device data expires, it will no longer be queryable. While the data in 
> disk files is not guaranteed to be immediately deleted, it will eventually be 
> deleted.
> 
> The related SQL syntax isas following:
> 
> 
> 
> SET TTL TO path value. Eg: SET TTL TO ROOT.DB 3600;
> 
> UNSET TTL TO path. Eg: UNSET TTL TO ROOT.DB;
> 
> SHOW ALL TTL
> Additionally, we have introduced the keyword `INF` to represent an infinite 
> TTL value.You can achieve more diversified configurations to meet 
> different scenarios and requirements through the configuration parameters 
> `default_ttl_in_ms`, `ttl_check_interval`, `ttl_rule_capacity
> `, `max_expired_time`, and `expired_data_ratio` in the configuration file. 
> 
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> BensonChou
> 
> Reference:
> 
> https://github.com/apache/iotdb/pull/12122

Re: [Proposal] Device Level TTL for IoTDB

[Xinyu Tan]

Hi Peichen

I'm very excited to see the implementation of device-level TTL. Before your 
work, if users wanted to set different TTLs for different devices, they had to 
model them into many databases, which was very inconvenient. After your work, 
users can set different TTLs within a single database as they wish, greatly 
enhancing the user experience for those users with TTL needs.

I am thrilled to see your work being merged and look forward to it creating 
greater value for users.

Best

Xinyu Tan

On 2024/06/03 14:10:05 BensonChou wrote:
> Hello, everyone.
> 
> I am BensonChou(Peichen Zhou), a committer of IoTDB. Over the past few 
> months, I have implemented a new feature, which is device-level TTL, and I am 
> so excited to share this new feature with you.
> 
> Previously, IoTDB only supported database-level TTL. For devices with 
> different collection frequencies and data validity periods under a single 
> database, we want to use a more granular device-level TTL to manage the data 
> lifecycle and optimize disk space as much as possible.
> 
> The new version of IoTDB will support device-level TTL management, allowing 
> users to set, modify, and unset TTLs in bulk. When setting TTL, the system 
> will apply it to all devices that match the specified path. If multiple TTLs 
> exist on a device path, the TTL closest to the device node will take effect. 
> Once device data expires, it will no longer be queryable. While the data in 
> disk files is not guaranteed to be immediately deleted, it will eventually be 
> deleted.
> 
> The related SQL syntax isas following:
> 
> 
> 
> SET TTL TO path value. Eg: SET TTL TO ROOT.DB 3600;
> 
> UNSET TTL TO path. Eg: UNSET TTL TO ROOT.DB;
> 
> SHOW ALL TTL
> Additionally, we have introduced the keyword `INF` to represent an infinite 
> TTL value.You can achieve more diversified configurations to meet 
> different scenarios and requirements through the configuration parameters 
> `default_ttl_in_ms`, `ttl_check_interval`, `ttl_rule_capacity
> `, `max_expired_time`, and `expired_data_ratio` in the configuration file. 
> 
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> BensonChou
> 
> Reference:
> 
> https://github.com/apache/iotdb/pull/12122

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[Xinyu Tan]

Hi, Junzhi

I'm very pleased to see that you and Sicheng Yu are leading the design and 
development of FastIoTConsensus and IoTV2Consensus. From designing a unified 
consensus algorithm framework, integrating the strong consistency consensus 
algorithm Ratis, to self-designing the weak consistency protocol IoTConsensus, 
and now further evolving from operation synchronization to state 
synchronization with FastIoTConsensus and IoTV2Consensus, we have successfully 
explored better consensus solutions in time-series scenarios, creating greater 
value for users with different needs.

I look forward to working together to further refine the consensus layer of 
IoTDB!

Best
-
Xinyu Tan

On 2024/06/03 10:28:18 Junzhi Peng wrote:
> Hello everyone,
> 
> I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
> the committer of IoTDB, have implemented two new consensus protocols on
> DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
> IoTV2Consensus, and we are excited to share with you this new feature.
> 
> As we all know, we have been using our self-developed IoTConsensus for
> DataRegion in most cases. It has excellent performance and can meet our
> business needs well. However, we found that IoTConsensus has availability
> issues in some scenarios. For example, after a node goes down, the WAL logs
> of other nodes may accumulate quickly, thus blocking writing. In addition,
> we also hope to further sacrifice consistency to obtain higher
> synchronization performance. Therefore, we introduced new consensus
> protocols: FastIoTConsensus and IoTV2Consensus.
> 
> FastIoTConsensus synchronizes the TsFile data file to achieve replica
> synchronization, thereby decoupling from WAL and solving the problem of
> IoTConsensus replica accumulation. Compared with IoTConsensus,
> FastIoTConsensus will further sacrifice consistency and have higher
> synchronization latency, but in theory it has higher write throughput
> performance and lower storage and computing resource usage.
> 
> IoTV2Consensus also introduces synchronization of TsFile data files to
> solve the problem of IoTConsensus WAL accumulation and improve performance.
> However, IoTV2Consensus also supports the use of WAL for data
> synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
> carrier of data synchronization according to the real-time load of the
> replica. IoTV2Consensus uses WAL for data synchronization by default. When
> WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
> synchronization.
> 
> At present, we have implemented the above two consensus protocols in this PR
> . However, in the future, we
> will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
> welcome everyone to use and make suggestions, but the relevant
> modifications and iterations will be carried out after FastIoTConsensus is
> polished.
> 
> We hope you are interested in this feature and would like to participate in
> the development and testing. You can easily modify the
> `data_region_consensus_protocol_class` configuration item of current
> `iotdb-system.properties` to use the above two new consensus protocols. You
> can also leave your comments and suggestions in this thread. Appreciate any
> suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> —
> Junzhi Peng
> 

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[Xinyu Tan]

Hi, Junzhi

I'm very pleased to see that you and Sicheng Yu are leading the design and 
development of FastIoTConsensus and IoTV2Consensus. From designing a unified 
consensus algorithm framework, integrating the strong consistency consensus 
algorithm Ratis, to self-designing the weak consistency protocol IoTConsensus, 
and now further evolving from operation synchronization to state 
synchronization with FastIoTConsensus and IoTV2Consensus, we have successfully 
explored better consensus solutions in time-series scenarios, creating greater 
value for users with different needs.

I look forward to working together to further refine the consensus layer of 
IoTDB!

Best
-
Xinyu Tan

On 2024/06/03 10:28:18 Junzhi Peng wrote:
> Hello everyone,
> 
> I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
> the committer of IoTDB, have implemented two new consensus protocols on
> DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
> IoTV2Consensus, and we are excited to share with you this new feature.
> 
> As we all know, we have been using our self-developed IoTConsensus for
> DataRegion in most cases. It has excellent performance and can meet our
> business needs well. However, we found that IoTConsensus has availability
> issues in some scenarios. For example, after a node goes down, the WAL logs
> of other nodes may accumulate quickly, thus blocking writing. In addition,
> we also hope to further sacrifice consistency to obtain higher
> synchronization performance. Therefore, we introduced new consensus
> protocols: FastIoTConsensus and IoTV2Consensus.
> 
> FastIoTConsensus synchronizes the TsFile data file to achieve replica
> synchronization, thereby decoupling from WAL and solving the problem of
> IoTConsensus replica accumulation. Compared with IoTConsensus,
> FastIoTConsensus will further sacrifice consistency and have higher
> synchronization latency, but in theory it has higher write throughput
> performance and lower storage and computing resource usage.
> 
> IoTV2Consensus also introduces synchronization of TsFile data files to
> solve the problem of IoTConsensus WAL accumulation and improve performance.
> However, IoTV2Consensus also supports the use of WAL for data
> synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
> carrier of data synchronization according to the real-time load of the
> replica. IoTV2Consensus uses WAL for data synchronization by default. When
> WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
> synchronization.
> 
> At present, we have implemented the above two consensus protocols in this PR
> . However, in the future, we
> will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
> welcome everyone to use and make suggestions, but the relevant
> modifications and iterations will be carried out after FastIoTConsensus is
> polished.
> 
> We hope you are interested in this feature and would like to participate in
> the development and testing. You can easily modify the
> `data_region_consensus_protocol_class` configuration item of current
> `iotdb-system.properties` to use the above two new consensus protocols. You
> can also leave your comments and suggestions in this thread. Appreciate any
> suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> —
> Junzhi Peng
> 

Re: [DISCUSS] GH Actions optimizations

[Yuan Tian]

ok..., if so, I think it's really nice.

On Wed, Jun 5, 2024 at 8:28 PM Christofer Dutz 
wrote:

> Also is the settings currently done that it only skips any tests in
> non-master branches … so if there’s a PR on a branch, that will be
> optimized, but every commit going back to master should have all tests run.
>
> Chris
>
>
> Von: Christofer Dutz 
> Datum: Mittwoch, 5. Juni 2024 um 14:22
> An: dev@iotdb.apache.org 
> Betreff: AW: [DISCUSS] GH Actions optimizations
> Well even with the re-run option still failures get counted and a
> dashboard like this:
>
> https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
> Is a lot better than having to continuously go back to your PRs and click
> on “re-run failed tests”
>
> Chris
>
> Von: Yuan Tian 
> Datum: Mittwoch, 5. Juni 2024 um 03:06
> An: dev@iotdb.apache.org 
> Betreff: Re: [DISCUSS] GH Actions optimizations
> Hi Chris,
>
> That's really good news. About the `surefire` part, I think we need to
> discuss more about it because some of the previously failed tests have
> indeed helped us discover concurrency issues in IoTDB. I am concerned that
> doing this might reduce the likelihood of exposing such bugs.
>
>
> Best regards,
> 
> Yuan Tian
>

AW: [DISCUSS] GH Actions optimizations

[Christofer Dutz]

Also is the settings currently done that it only skips any tests in non-master 
branches … so if there’s a PR on a branch, that will be optimized, but every 
commit going back to master should have all tests run.

Chris


Von: Christofer Dutz 
Datum: Mittwoch, 5. Juni 2024 um 14:22
An: dev@iotdb.apache.org 
Betreff: AW: [DISCUSS] GH Actions optimizations
Well even with the re-run option still failures get counted and a dashboard 
like this:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
Is a lot better than having to continuously go back to your PRs and click on 
“re-run failed tests”

Chris

Von: Yuan Tian 
Datum: Mittwoch, 5. Juni 2024 um 03:06
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] GH Actions optimizations
Hi Chris,

That's really good news. About the `surefire` part, I think we need to
discuss more about it because some of the previously failed tests have
indeed helped us discover concurrency issues in IoTDB. I am concerned that
doing this might reduce the likelihood of exposing such bugs.


Best regards,

Yuan Tian

AW: [DISCUSS] GH Actions optimizations

[Christofer Dutz]

Well even with the re-run option still failures get counted and a dashboard 
like this:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
Is a lot better than having to continuously go back to your PRs and click on 
“re-run failed tests”

Chris

Von: Yuan Tian 
Datum: Mittwoch, 5. Juni 2024 um 03:06
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] GH Actions optimizations
Hi Chris,

That's really good news. About the `surefire` part, I think we need to
discuss more about it because some of the previously failed tests have
indeed helped us discover concurrency issues in IoTDB. I am concerned that
doing this might reduce the likelihood of exposing such bugs.


Best regards,

Yuan Tian

Re: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Jialin Qiao]

Thank Chris!

Jialin Qiao

Christofer Dutz  于2024年6月4日周二 17:28写道：
>
> Ok … artifacts should be available …
>
> Chris
>
> Von: Christofer Dutz 
> Datum: Dienstag, 4. Juni 2024 um 11:26
> An: dev@iotdb.apache.org 
> Betreff: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache 
> Thrift 0.20.0)
> So, the vote passes with 7 +1 votes at least 5 I could verify as binding 
> votes.
>
> In the future, it would be great if people would add their name/apacheId etc. 
> to the votes to make it easier to verify and count for the RM.
>
> Releasing the maven artifacts right away.
>
> Thanks for all those who voted.
>
> Chris
>
>
> Von: Christofer Dutz 
> Datum: Dienstag, 4. Juni 2024 um 09:37
> An: dev@iotdb.apache.org 
> Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
> 0.20.0)
> Yeah.. Currently at community over code... Was planning on doing this in a 
> break today.
>
> Gesendet von Outlook für Android
> 
> From: Haonan Hou 
> Sent: Monday, June 3, 2024 4:11:04 AM
> To: dev@iotdb.apache.org 
> Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
> 0.20.0)
>
> Hi Chris,
>
> I think 72 hours has passed. These artifacts can be released now?
>
> Thanks,
> Haonan
>
> On 2024/05/28 14:40:39 Christofer Dutz wrote:
> > Hi all,
> >
> > I after fixing some issues in the cmake-maven-plugin, have I updated the 
> > iotdb-bin-resources module to use these.
> > Now we can stage Apache Thrift binaries for all platforms.
> >
> > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible 
> > again), I also updated the Thrift version and staged a release of Thrift 
> > binaries for all of our supported platforms in Nexus.
> >
> > I’ve staged the artifacts at:
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> >
> > Tag Name: iotdb-tools-thrift-v0.20.0.0
> > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> >
> > The source bundle generally consists of a pom.xml and an assembly xml … so 
> > it should be quick to review.
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> >
> > I’m intentionally not staging it on the release-svn, as nobody should be 
> > required to consume this module but us ourselves.
> >
> > The vote stays open for 72hours … assuming enough support, I’ll then finish 
> > changing the main repo to use these artifacts.
> >
> >
> > Chris
> >

[DISCUSS] GH Actions optimizations

[Christofer Dutz]

At today’s sessions at community over code EU in a session about “Apache build 
observability” we got a little spotlight on IoTDB.
The develocity folks pointed out Apache IoTDB as being the one Apache project 
that provides the biggest room for improvement by enabling some of their 
build-optimization tools.

I have created a PR : https://github.com/apache/iotdb/pull/12654

This enables predictive test-selection which should prioritize tests that are 
most probable to be affected by changes and to skip ones that are more 
definitely not going to be affected.

I was actively approached by the develocity folks here at the conference and 
together we created a PR that should enable this for us.
Also did I take the liberty to also enable surefire to automatically re-run 
failed tests for a maximum of 3 times before actually failing a test.

These changes should allow reducing the load we have on GH Actions by 40% 
(according to the dashboards develocity has)

Also did they recommend utilizing build cache extensions, which should even 
more greatly reduce the build time and the load on the GH runners.

If anyone wants to play around with the dashboards for IoTDB …. Please have a 
look here.

https://ge.apache.org/scans?search.rootProjectNames=*IoTDB*

There are very interesting dashboards that show us stuff like flaky tests:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY

Chris

AW: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Ok … artifacts should be available …

Chris

Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 11:26
An: dev@iotdb.apache.org 
Betreff: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache 
Thrift 0.20.0)
So, the vote passes with 7 +1 votes at least 5 I could verify as binding votes.

In the future, it would be great if people would add their name/apacheId etc. 
to the votes to make it easier to verify and count for the RM.

Releasing the maven artifacts right away.

Thanks for all those who voted.

Chris


Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 09:37
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)
Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

[RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

So, the vote passes with 7 +1 votes at least 5 I could verify as binding votes.

In the future, it would be great if people would add their name/apacheId etc. 
to the votes to make it easier to verify and count for the RM.

Releasing the maven artifacts right away.

Thanks for all those who voted.

Chris


Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 09:37
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)
Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

Re: [Proposal] New Consensus Protocals for IoTDB DataRegion

[Xiangpeng Hu]

Hello Junzhi,


I would like to extend my heartfelt thanks to you and Sicheng Yu for your 
FastIoTConsensus and IoTV2Consensus protocols. It is a significant step forward 
for IoTDB.


I am well aware that the existing IoTConsensus has been performing 
exceptionally well in most scenarios, yet it has encountered availability 
issues. I appreciate your insight into the need to further sacrifice 
consistency for higher synchronization performance, which is a forward-thinking 
approach. I am also very excited about the opportunity to participate in the 
development and testing of these new consensus protocols.


Lastly, your hard work and innovation are greatly appreciated, and I look 
forward to seeing the real-world performance of FastIoTConsensus and 
IoTV2Consensus.


Best regards,
Xiangpeng Hu
 Replied Message 
| From | Junzhi Peng |
| Date | 6/3/2024 18:28 |
| To |  |
| Subject | [Proposal] New Consensus Protocals for IoTDB DataRegion |
Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng

Re: [Proposal] Merge configuration files to iotdb-system.properties

[wenwei shu]

Hi,

Most of the changes in this PR are to modify the configuration file itself
and some script tools that will read the configuration file. Codes for
accessing the options are not merged.

Best regards,
Wenwei Shu

Christofer Dutz  于2024年6月1日周六 15:32写道：

> However,
>
> I think it would be good If the code for accessing the options would not
> be merged.
> Because that would move the project to being more of a mono-module type of
> project.
>
> Chris
>
> Von: Jialin Qiao 
> Datum: Samstag, 1. Juni 2024 um 08:07
> An: dev@iotdb.apache.org 
> Betreff: Re: [Proposal] Merge configuration files to
> iotdb-system.properties
> Hi,
>
> Good feature!  This will make it easy for users to set configs.
>
> Jialin Qiao
>
> wenwei shu  于2024年5月31日周五 21:23写道：
> >
> >  Hello everyone,
> >
> > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > working on merging `iotdb-confignode.properties`,
> > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > `iotdb-system.properties` file. For old version users who upgrade to a
> new
> > version, if they don't create the new configuration file themselves
> during
> > the upgrade, IoTDB will generate the new configuration file based on
> > several old configuration files after it is started. You can check this
> PR
> > to get more details: https://github.com/apache/iotdb/pull/12570.
> > Thank you for your reading.
> >
> > Best regards,
> > Wenwei Shu
>

[Proposal] New Consensus Protocals for IoTDB DataRegion

[Junzhi Peng]

Hello everyone,

I'm Junzhi Peng, a new contributor to IoTDB. Recently, I and Sicheng Yu,
the committer of IoTDB, have implemented two new consensus protocols on
DataRegion based on IoTDB Pipe framework: FastIoTConsensus and
IoTV2Consensus, and we are excited to share with you this new feature.

As we all know, we have been using our self-developed IoTConsensus for
DataRegion in most cases. It has excellent performance and can meet our
business needs well. However, we found that IoTConsensus has availability
issues in some scenarios. For example, after a node goes down, the WAL logs
of other nodes may accumulate quickly, thus blocking writing. In addition,
we also hope to further sacrifice consistency to obtain higher
synchronization performance. Therefore, we introduced new consensus
protocols: FastIoTConsensus and IoTV2Consensus.

FastIoTConsensus synchronizes the TsFile data file to achieve replica
synchronization, thereby decoupling from WAL and solving the problem of
IoTConsensus replica accumulation. Compared with IoTConsensus,
FastIoTConsensus will further sacrifice consistency and have higher
synchronization latency, but in theory it has higher write throughput
performance and lower storage and computing resource usage.

IoTV2Consensus also introduces synchronization of TsFile data files to
solve the problem of IoTConsensus WAL accumulation and improve performance.
However, IoTV2Consensus also supports the use of WAL for data
synchronization. Compared with IoTConsensus, IoTV2Consensus will switch the
carrier of data synchronization according to the real-time load of the
replica. IoTV2Consensus uses WAL for data synchronization by default. When
WAL accumulates, IoTV2Consensus will be downgraded to using TsFile for data
synchronization.

At present, we have implemented the above two consensus protocols in this PR
. However, in the future, we
will prioritize the iteration of FastIoTConsensus. For IoTV2Consensus, we
welcome everyone to use and make suggestions, but the relevant
modifications and iterations will be carried out after FastIoTConsensus is
polished.

We hope you are interested in this feature and would like to participate in
the development and testing. You can easily modify the
`data_region_consensus_protocol_class` configuration item of current
`iotdb-system.properties` to use the above two new consensus protocols. You
can also leave your comments and suggestions in this thread. Appreciate any
suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
—
Junzhi Peng

Re: [Proposal] Merge configuration files to iotdb-system.properties

[Jialin Qiao]

Hi,

> I think it would be good If the code for accessing the options would not be 
> merged.

+1, I think the code is not merged in Wenwei's PR.

Jialin Qiao

Caiyin Yang  于2024年6月3日周一 10:27写道：
>
> Hi Xinyu,
>
> If that's the case, then this merge seems to be very user-friendly.
>
> Best
> Caiyin Yang
>
> Xinyu Tan  于2024年6月3日周一 10:22写道：
>
> > Hi, Caiyin
> >
> > Following Wenwei's work, we plan to introduce an
> > iotdb-system.properties.template file that will contain all configuration
> > items, relevant comments, and optional settings, similar to the latest user
> > configuration manual. In the future, the iotdb-system.properties file will
> > only include explicitly configured items by the user (such as IP addresses
> > or manually configured storage engine parameters). Compared to the current
> > method of finding and modifying configurations in the lengthy
> > iotdb-confignode.properties and iotdb-datanode.properties files, this
> > approach seems more convenient. If you have any good ideas, feel free to
> > discuss your preferred method further!
> >
> > Best
> > 
> > Xinyu Tan
> >
> > On 2024/06/01 09:18:48 ycy wi wrote:
> > > Hi wenwei,
> > >
> > > I am curious whether this will ultimately lead to an oversized or
> > redundant configuration file. I might just want to modify some
> > configuration items in the dataNode. In theory, I only need to pay
> > attention to the configuration items in iotdb-datanode.properties. Will the
> > merge cause any confusion due to similar configuration items in
> > iotdb-confignode.properties?
> > >
> > > Best
> > > CaiyinYang
> > >
> > >
> > >
> > >
> > >
> > > > 2024年5月31日 21:20，wenwei shu  写道：
> > > >
> > > > Hello everyone,
> > > >
> > > > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > > > working on merging `iotdb-confignode.properties`,
> > > > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > > > `iotdb-system.properties` file. For old version users who upgrade to a
> > new
> > > > version, if they don't create the new configuration file themselves
> > during
> > > > the upgrade, IoTDB will generate the new configuration file based on
> > > > several old configuration files after it is started. You can check
> > this PR
> > > > to get more details: https://github.com/apache/iotdb/pull/12570.
> > > > Thank you for your reading.
> > > >
> > > > Best regards,
> > > > Wenwei Shu
> > >
> > >
> >

Re: [Proposal] Load Tsfile Functionality Optimisation

[ITAMI SHO]

Hi Xiangzhi,

Thank you for sharing your work on the load tsfile feature. I am currently 
working on the Load function as well, and I have a question for you. Have you 
considered enhancing the Load function directly within the core code, rather 
than relying on the script to load tsfiles one by one? Additionally , do you 
think there would be a significant performance difference between the original 
batch load method and your script of loading files individually?

Your insights on this matter would be very valuable, and I believe it could 
lead to further optimizations!

Best regards,
Itami Sho

> 2024年5月30日 11:11，孟祥志  写道：
> 
> Hello everyone,
> 
> I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share 
> with you a feature that I have been working on for the past few months.
> 
> load tsfile has always been an important feature for IoTDB, but due to the 
> original design, he has some flaws in it, when load a folder fails to import 
> some tsfile, it may cause the whole task fails, which is obviously a not very 
> user-friendly design, so I decided to modify his.
> 
> I'll modify the original load tsfile script to traverse the entire folder 
> first, find all the files, and then use the LOAD function to import 
> individual files; and I've added a few interesting changes along the way, 
> modifying the successful operation parameter to make it easier for people to 
> deal with successful files, and added a parameter for failed files to make it 
> easier for people to find those failed files, and finally added a new number 
> of threads to make the LOAD function work more quickly.
> 
> 
> I hope you are interested in this feature and would like to participate in 
> the development and testing. You can also leave your comments and suggestions 
> in this thread. Appreciate any suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> Xiangzhi Meng
> 
> Reference:
> https://github.com/apache/iotdb/pull/12541

Re: [Proposal] Active metaData Query

[ycy wi]

Hi ITAMI,

‘Active’ refers to the data that can be seen by query in the specified time 
range, which you can also understand as data that has been written in the 
specified time and not deleted later. 

Best
CaiyinYang





> 2024年6月3日 15:40，ITAMI SHO  写道：
> 
> question

Re: [Proposal] Active metaData Query

[ITAMI SHO]

Hi Caiyin,

Thank you for your contribution, I have a question regrading the definition of 
active metadata. Could you please provide more details on what constitutes 
“active metadata” in this context? Understanding this definition better will 
help us utilize this new feature!

Thank you again for your hard work and contribution.

Best regards,
Itami Sho

> 2024年6月3日 11:45，Caiyin Yang  写道：
> 
> Hello everyone, I'm Caiyin Yang, a contributor to IoTDB.
> 
> I have implemented an active metadata query feature in IoTDB in the past
> few months, which is an extension based on the original *SHOW/COUNT
> DEVICES/TIMESERIES *SQL. You only need to specify the time condition in the
> SQL statement (*for example: show timeseries where time > xx and time < xx*)
> to know which metadata exists in that interval.
> 
> The active metadata query first uses the statistical information in memory
> for optimization, and only performs disk scanning when the statistical
> information cannot be used.
> 
> To optimize the performance of disk scanning, I implemented the sequential
> scan of TsFile files called RegionScan.
> 
> In the scenario of full scanning of active data, compared with the original
> series query, there is better performance (preliminary tests have shown an
> improvement of nearly 20% - 30%). I will then carry out more comprehensive
> performance testing scenarios and welcome everyone to provide any feedback.
> 
> *PR Links:*
> https://github.com/apache/iotdb/pull/12446
> https://github.com/apache/iotdb/pull/12539
> https://github.com/apache/iotdb/pull/12607

Re: Canonical the semantics of auto create schema function

[ITAMI SHO]

Hi Yongzao Dan,

Thank you for your recent contribution to the Apache IoTDB community. Your work 
on clarifying the semantics of the auto-create schema function and addressing 
the associated issues has been incredibly valuable to IoTDB. Your dedication 
and attention to detail are greatly appreciated!

Best regards,
Itami Sho

> 2024年6月1日 12:43，Yongzao Dan <532741...@qq.com.INVALID> 写道：
> 
> Hi everyone,
> 
> 
>  I'm Yongzao Dan, a committer in the Apache IoTDB community.
> 
> 
>  The semantics ofauto create schema function has been confused 
> since the community updated the distribution version, where the IoTDB will 
> automatically create both corresponding database and time series schema when 
> enable this function and process write request. But the IoTDB will still 
> create a database when disable this function and process write request. In 
> fact, neither the database nor the time series schema should be created when 
> disable this function.
> 
> 
>  Thus I commited a PR to fix this neglected semantical bug during 
> version iteration. Added a new IT as well to prevent this function being 
> incorrectly modified in future 
> update:https://github.com/apache/iotdb/pull/12590
> 
> 
> Warm regards,
> Yongzao Dan

[Proposal] Active metaData Query

[Caiyin Yang]

Hello everyone, I'm Caiyin Yang, a contributor to IoTDB.

I have implemented an active metadata query feature in IoTDB in the past
few months, which is an extension based on the original *SHOW/COUNT
DEVICES/TIMESERIES *SQL. You only need to specify the time condition in the
SQL statement (*for example: show timeseries where time > xx and time < xx*)
to know which metadata exists in that interval.

The active metadata query first uses the statistical information in memory
for optimization, and only performs disk scanning when the statistical
information cannot be used.

To optimize the performance of disk scanning, I implemented the sequential
scan of TsFile files called RegionScan.

In the scenario of full scanning of active data, compared with the original
series query, there is better performance (preliminary tests have shown an
improvement of nearly 20% - 30%). I will then carry out more comprehensive
performance testing scenarios and welcome everyone to provide any feedback.

*PR Links:*
https://github.com/apache/iotdb/pull/12446
https://github.com/apache/iotdb/pull/12539
https://github.com/apache/iotdb/pull/12607

Re: [Proposal] Add the feature of importing and exporting metadata by CSV file

[Xiangpeng Hu]

Hi Rong Li,


This is a great new feature. Thank you for working on this. Registering 
metadata is usually a time-consuming operation in the case of massive 
sequences. I am curious about the efficiency of importing and exporting 
metadata. Have the scripts been tested on large-scale sequences? If so, could 
you share the relevant data?


Xiangpeng Hu
 Replied Message 
| From | 李蓉 |
| Date | 6/3/2024 10:04 |
| To | dev@iotdb.apache.org |
| Subject | [Proposal] Add the feature of importing and exporting metadata by 
CSV file |
Hello Everyone,
 
I am Rong Li. I am excited to share with you a feature that I have been working 
on for the past few months.

As we all know, IoTDB supports various forms of importing and exporting data, 
but does not support importing and exporting metadata. In practical use, there 
are also many scenarios that require importing and exporting metadata, so I 
added a metadata import and export script to achieve this function.
 
I hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.
 
Thank you for your attention and support.

Best regards,

Rong Li

Reference:
https://github.com/apache/iotdb/pull/12623

Re: [Proposal] Merge configuration files to iotdb-system.properties

[Caiyin Yang]

Hi Xinyu,

If that's the case, then this merge seems to be very user-friendly.

Best
Caiyin Yang

Xinyu Tan  于2024年6月3日周一 10:22写道：

> Hi, Caiyin
>
> Following Wenwei's work, we plan to introduce an
> iotdb-system.properties.template file that will contain all configuration
> items, relevant comments, and optional settings, similar to the latest user
> configuration manual. In the future, the iotdb-system.properties file will
> only include explicitly configured items by the user (such as IP addresses
> or manually configured storage engine parameters). Compared to the current
> method of finding and modifying configurations in the lengthy
> iotdb-confignode.properties and iotdb-datanode.properties files, this
> approach seems more convenient. If you have any good ideas, feel free to
> discuss your preferred method further!
>
> Best
> 
> Xinyu Tan
>
> On 2024/06/01 09:18:48 ycy wi wrote:
> > Hi wenwei,
> >
> > I am curious whether this will ultimately lead to an oversized or
> redundant configuration file. I might just want to modify some
> configuration items in the dataNode. In theory, I only need to pay
> attention to the configuration items in iotdb-datanode.properties. Will the
> merge cause any confusion due to similar configuration items in
> iotdb-confignode.properties?
> >
> > Best
> > CaiyinYang
> >
> >
> >
> >
> >
> > > 2024年5月31日 21:20，wenwei shu  写道：
> > >
> > > Hello everyone,
> > >
> > > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > > working on merging `iotdb-confignode.properties`,
> > > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > > `iotdb-system.properties` file. For old version users who upgrade to a
> new
> > > version, if they don't create the new configuration file themselves
> during
> > > the upgrade, IoTDB will generate the new configuration file based on
> > > several old configuration files after it is started. You can check
> this PR
> > > to get more details: https://github.com/apache/iotdb/pull/12570.
> > > Thank you for your reading.
> > >
> > > Best regards,
> > > Wenwei Shu
> >
> >
>

Re: Canonical the semantics of auto create schema function

[Xinyu Tan]

Hi rongzhao

It's really cool that with this change we're able to provide more explicit 
auto-creation semantics to our users.

Best
--
Xinyu Tan

On 2024/06/01 04:43:05 Yongzao Dan wrote:
> Hi everyone,
> 
> 
>  I'm Yongzao Dan, a committer in the Apache IoTDB community.
> 
> 
>  The semantics ofauto create schema function has been confused 
> since the community updated the distribution version, where the IoTDB will 
> automatically create both corresponding database and time series schema when 
> enable this function and process write request. But the IoTDB will still 
> create a database when disable this function and process write request. In 
> fact, neither the database nor the time series schema should be created when 
> disable this function.
> 
> 
>  Thus I commited a PR to fix this neglected semantical bug during 
> version iteration. Added a new IT as well to prevent this function being 
> incorrectly modified in future 
> update:https://github.com/apache/iotdb/pull/12590
> 
> 
> Warm regards,
> Yongzao Dan

Re: [Proposal] Merge configuration files to iotdb-system.properties

[Xinyu Tan]

Hi, Caiyin

Following Wenwei's work, we plan to introduce an 
iotdb-system.properties.template file that will contain all configuration 
items, relevant comments, and optional settings, similar to the latest user 
configuration manual. In the future, the iotdb-system.properties file will only 
include explicitly configured items by the user (such as IP addresses or 
manually configured storage engine parameters). Compared to the current method 
of finding and modifying configurations in the lengthy 
iotdb-confignode.properties and iotdb-datanode.properties files, this approach 
seems more convenient. If you have any good ideas, feel free to discuss your 
preferred method further!

Best

Xinyu Tan

On 2024/06/01 09:18:48 ycy wi wrote:
> Hi wenwei,
> 
> I am curious whether this will ultimately lead to an oversized or redundant 
> configuration file. I might just want to modify some configuration items in 
> the dataNode. In theory, I only need to pay attention to the configuration 
> items in iotdb-datanode.properties. Will the merge cause any confusion due to 
> similar configuration items in iotdb-confignode.properties?
> 
> Best
> CaiyinYang
> 
> 
> 
> 
> 
> > 2024年5月31日 21:20，wenwei shu  写道：
> > 
> > Hello everyone,
> > 
> > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > working on merging `iotdb-confignode.properties`,
> > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > `iotdb-system.properties` file. For old version users who upgrade to a new
> > version, if they don't create the new configuration file themselves during
> > the upgrade, IoTDB will generate the new configuration file based on
> > several old configuration files after it is started. You can check this PR
> > to get more details: https://github.com/apache/iotdb/pull/12570.
> > Thank you for your reading.
> > 
> > Best regards,
> > Wenwei Shu
> 
> 

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Haonan Hou]

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
> 
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
> 
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
> 
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> 
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> 
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> 
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
> 
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
> 
> 
> Chris
> 

Re: Prepare releasing 1.3.2?

[Xinyu Tan]

Hi haonan

+1 for bumping the version of master branch to 1.3.3-SNAPSHOT.

As for the 1.3.2 release, currently ratis is releasing there new version 
3.1.0[1], we can rely on the official Ratis 3.1.0 release in the open source 
version IoTDB 1.3.2 instead of the snapshot release to conform to the 
guidelines of the Apache community.

Best
-
Xinyu Tan

[1] https://lists.apache.org/thread/q8gdfz1j1l4qv1v24z4dxcg01j9qs28n

On 2024/06/03 01:44:17 Haonan Hou wrote:
> Hi all,
> 
> IoTDB v1.3.1 has released over 1 month. In this month, we did a lot over bug 
> fixes and improvements. I think we can start the process of releasing new 
> version.
> 
> If we all agree, I'm going to checkout the rc/1.3.2 branch and bump the 
> version of master branch to 1.3.3-SNAPSHOT.
> 
> BR,
> Haonan
> 

[Proposal] Add the feature of importing and exporting metadata by CSV file

[李蓉]

Hello Everyone,
 
I am Rong Li. I am excited to share with you a feature that I have been working 
on for the past few months.

As we all know, IoTDB supports various forms of importing and exporting data, 
but does not support importing and exporting metadata. In practical use, there 
are also many scenarios that require importing and exporting metadata, so I 
added a metadata import and export script to achieve this function.
 
I hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.
 
Thank you for your attention and support.

Best regards,

Rong Li

Reference:
https://github.com/apache/iotdb/pull/12623

Re: [Proposal] Merge configuration files to iotdb-system.properties

[wenwei shu]

Hi caiyin,

After merging, there are indeed more configuration items in one file, but i
think it will not cause confusion. Most configuration item names in
`iotdb-confignode.properties` or `iotdb-datanode.properties` begin with
`cn` or `dn`.

Best
Wenwei Shu

ycy wi  于2024年6月1日周六 17:19写道：

> Hi wenwei,
>
> I am curious whether this will ultimately lead to an oversized or
> redundant configuration file. I might just want to modify some
> configuration items in the dataNode. In theory, I only need to pay
> attention to the configuration items in iotdb-datanode.properties. Will the
> merge cause any confusion due to similar configuration items in
> iotdb-confignode.properties?
>
> Best
> CaiyinYang
>
>
>
>
>
> > 2024年5月31日 21:20，wenwei shu  写道：
> >
> > Hello everyone,
> >
> > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> > working on merging `iotdb-confignode.properties`,
> > `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> > `iotdb-system.properties` file. For old version users who upgrade to a
> new
> > version, if they don't create the new configuration file themselves
> during
> > the upgrade, IoTDB will generate the new configuration file based on
> > several old configuration files after it is started. You can check this
> PR
> > to get more details: https://github.com/apache/iotdb/pull/12570.
> > Thank you for your reading.
> >
> > Best regards,
> > Wenwei Shu
>
>

Re: Prepare releasing 1.3.2?

[Yuan Tian]

Hi haonan,


+1(binding), looking forward to exciting v1.3.2

Best regards,

Yuan Tian



On Mon, Jun 3, 2024 at 9:45 AM Haonan Hou  wrote:

> Hi all,
>
> IoTDB v1.3.1 has released over 1 month. In this month, we did a lot over
> bug fixes and improvements. I think we can start the process of releasing
> new version.
>
> If we all agree, I'm going to checkout the rc/1.3.2 branch and bump the
> version of master branch to 1.3.3-SNAPSHOT.
>
> BR,
> Haonan
>

Prepare releasing 1.3.2?

[Haonan Hou]

Hi all,

IoTDB v1.3.1 has released over 1 month. In this month, we did a lot over bug 
fixes and improvements. I think we can start the process of releasing new 
version.

If we all agree, I'm going to checkout the rc/1.3.2 branch and bump the version 
of master branch to 1.3.3-SNAPSHOT.

BR,
Haonan

Re: [PR] chore(deps): Bump spring.version from 5.3.33 to 5.3.34 [iotdb-extras]

[via GitHub]

dependabot[bot] closed pull request #4: chore(deps): Bump spring.version from 
5.3.33 to 5.3.34
URL: https://github.com/apache/iotdb-extras/pull/4


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] chore(deps): Bump spring.version from 5.3.33 to 5.3.36 [iotdb-extras]

[via GitHub]

dependabot[bot] opened a new pull request, #8:
URL: https://github.com/apache/iotdb-extras/pull/8

   Bumps `spring.version` from 5.3.33 to 5.3.36.
   Updates `org.springframework:spring-tx` from 5.3.33 to 5.3.36
   
   Release notes
   Sourced from https://github.com/spring-projects/spring-framework/releases;>org.springframework:spring-tx's
 releases.
   
   v5.3.36
   :lady_beetle: Bug Fixes
   
   Overridden aspect method runs twice https://redirect.github.com/spring-projects/spring-framework/issues/32868;>#32868
   @DateTimeFormat(iso = DateTimeFormat.ISO.DATE\_TIME) cannot 
convert UTC without milliseconds to java.util.Date https://redirect.github.com/spring-projects/spring-framework/issues/32860;>#32860
   Spring AOP fails against registered @Configurable aspect https://redirect.github.com/spring-projects/spring-framework/issues/32840;>#32840
   
   v5.3.35
   :star: New Features
   
   Accept ajc-compiled @Aspect classes for Spring AOP proxy 
usage https://redirect.github.com/spring-projects/spring-framework/issues/32818;>#32818
   
   :lady_beetle: Bug Fixes
   
   DeferredQueryInvocationHandler fails to unwrap QuerySqmImpl class 
outside of transaction https://redirect.github.com/spring-projects/spring-framework/issues/32770;>#32770
   MergedAnnotations search does not find container for 
repeatable annotation https://redirect.github.com/spring-projects/spring-framework/issues/32751;>#32751
   AnnotationConfigWebApplicationContext should propagate 
ApplicationStartup to BeanFactory https://redirect.github.com/spring-projects/spring-framework/issues/32749;>#32749
   Ignore non-String keys in 
PropertiesPropertySource.getPropertyNames() https://redirect.github.com/spring-projects/spring-framework/issues/32744;>#32744
   multiple subscribers not supported when using WebClient 
exchange https://redirect.github.com/spring-projects/spring-framework/issues/32728;>#32728
   Deadlock/Stall in ConcurrentWebSocketSessionDecorator with Undertow 
2.3.10 https://redirect.github.com/spring-projects/spring-framework/issues/32698;>#32698
   
   :notebook_with_decorative_cover: Documentation
   
   Correct documentation on streaming with MockMvcWebTestClient https://redirect.github.com/spring-projects/spring-framework/issues/32723;>#32723
   Update links to HttpOnly documentation at OWASP in 
ResponseCookie https://redirect.github.com/spring-projects/spring-framework/issues/32668;>#32668
   
   :hammer: Dependency Upgrades
   
   Upgrade to Reactor 2020.0.44 https://redirect.github.com/spring-projects/spring-framework/issues/32788;>#32788
   
   v5.3.34
   :star: New Features
   
   Log column type for limited support message in 
JdbcUtils.getResultSetValue https://redirect.github.com/spring-projects/spring-framework/issues/32603;>#32603
   Avoid additional unnecessary Annotation array cloning in 
TypeDescriptor https://redirect.github.com/spring-projects/spring-framework/issues/32477;>#32477
   Avoid cloning empty Annotation array in 
TypeDescriptor https://redirect.github.com/spring-projects/spring-framework/issues/32466;>#32466
   
   :lady_beetle: Bug Fixes
   
   Refine scheme, userinfo, host and port parsing in UriComponentsBuilder 
https://redirect.github.com/spring-projects/spring-framework/issues/32618;>#32618
   MethodIntrospector.selectMethods() fails to detect bridge 
methods across ApplicationContexts https://redirect.github.com/spring-projects/spring-framework/issues/32588;>#32588
   JmsUtils.commitIfNecessary catches and ignores JMS 
IllegalStateException, losing message with ActiveMQ Artemis https://redirect.github.com/spring-projects/spring-framework/issues/32480;>#32480
   Consistently apply TaskDecorator to ManagedExecutorService as well https://redirect.github.com/spring-projects/spring-framework/issues/32457;>#32457
   
   :hammer: Dependency Upgrades
   
   Upgrade to Reactor 2020.0.43 https://redirect.github.com/spring-projects/spring-framework/issues/32594;>#32594
   
   
   
   
   Commits
   
   https://github.com/spring-projects/spring-framework/commit/2399806d5f0a0e5fcecd8dd57fbae493f8b8da3f;>2399806
 Release v5.3.36
   https://github.com/spring-projects/spring-framework/commit/eff9f5b92a0f4daa65f6f6276b0915879b24497c;>eff9f5b
 Select most specific advice method in case of override
   https://github.com/spring-projects/spring-framework/commit/f1fed9c174705aa4f2817c94fc519f17b670976b;>f1fed9c
 Polishing
   https://github.com/spring-projects/spring-framework/commit/d2e7cf439545cac51a464956017f1db433990eba;>d2e7cf4
 Default fallback parsing for UTC without milliseconds
   https://github.com/spring-projects/spring-framework/commit/47a7abee8f223266bfdd012a03b88f2f7b8f5a32;>47a7abe
 Polishing
   https://github.com/spring-projects/spring-framework/commit/ef2c140d3cbb5fe7dd53afd65c6ca061f617911a;>ef2c140
 Defensively catch and log pointcut parsing exceptions
   https://github.com/spring-projects/spring-framework/commit/de6cf845b102cffd81dd52013cf9dc3cfe0d04df;>de6cf84
 Next development 

Re: [PR] chore(deps): Bump spring.version from 5.3.33 to 5.3.34 [iotdb-extras]

[via GitHub]

dependabot[bot] commented on PR #4:
URL: https://github.com/apache/iotdb-extras/pull/4#issuecomment-2143521149

   Superseded by #8.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [Proposal] Merge configuration files to iotdb-system.properties

[ycy wi]

Hi wenwei,

I am curious whether this will ultimately lead to an oversized or redundant 
configuration file. I might just want to modify some configuration items in the 
dataNode. In theory, I only need to pay attention to the configuration items in 
iotdb-datanode.properties. Will the merge cause any confusion due to similar 
configuration items in iotdb-confignode.properties?

Best
CaiyinYang





> 2024年5月31日 21:20，wenwei shu  写道：
> 
> Hello everyone,
> 
> I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> working on merging `iotdb-confignode.properties`,
> `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> `iotdb-system.properties` file. For old version users who upgrade to a new
> version, if they don't create the new configuration file themselves during
> the upgrade, IoTDB will generate the new configuration file based on
> several old configuration files after it is started. You can check this PR
> to get more details: https://github.com/apache/iotdb/pull/12570.
> Thank you for your reading.
> 
> Best regards,
> Wenwei Shu

AW: [Proposal] Merge configuration files to iotdb-system.properties

[Christofer Dutz]

However,

I think it would be good If the code for accessing the options would not be 
merged.
Because that would move the project to being more of a mono-module type of 
project.

Chris

Von: Jialin Qiao 
Datum: Samstag, 1. Juni 2024 um 08:07
An: dev@iotdb.apache.org 
Betreff: Re: [Proposal] Merge configuration files to iotdb-system.properties
Hi,

Good feature!  This will make it easy for users to set configs.

Jialin Qiao

wenwei shu  于2024年5月31日周五 21:23写道：
>
>  Hello everyone,
>
> I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> working on merging `iotdb-confignode.properties`,
> `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> `iotdb-system.properties` file. For old version users who upgrade to a new
> version, if they don't create the new configuration file themselves during
> the upgrade, IoTDB will generate the new configuration file based on
> several old configuration files after it is started. You can check this PR
> to get more details: https://github.com/apache/iotdb/pull/12570.
> Thank you for your reading.
>
> Best regards,
> Wenwei Shu

Re: [Proposal] Merge configuration files to iotdb-system.properties

[Jialin Qiao]

Hi,

Good feature!  This will make it easy for users to set configs.

Jialin Qiao

wenwei shu  于2024年5月31日周五 21:23写道：
>
>  Hello everyone,
>
> I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> working on merging `iotdb-confignode.properties`,
> `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> `iotdb-system.properties` file. For old version users who upgrade to a new
> version, if they don't create the new configuration file themselves during
> the upgrade, IoTDB will generate the new configuration file based on
> several old configuration files after it is started. You can check this PR
> to get more details: https://github.com/apache/iotdb/pull/12570.
> Thank you for your reading.
>
> Best regards,
> Wenwei Shu

[Proposal] Merge configuration files to iotdb-system.properties

[wenwei shu]

 Hello everyone,

I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
working on merging `iotdb-confignode.properties`,
`iotdb-datanode.properties` and `iotdb-common.properties` into a new
`iotdb-system.properties` file. For old version users who upgrade to a new
version, if they don't create the new configuration file themselves during
the upgrade, IoTDB will generate the new configuration file based on
several old configuration files after it is started. You can check this PR
to get more details: https://github.com/apache/iotdb/pull/12570.
Thank you for your reading.

Best regards,
Wenwei Shu

AW: [Proposal] Data Subscription Client on IoTDB

[Christofer Dutz]

Hi Yuchen,

do I understand it correctly, that with this, a client could subscribe to data 
and receive this in TsFile format? This plus the ability of an IoTDB instance 
subscribing to such a TsFile subscription would perfectly align with my plans 
for 2024 allowing TsFile libraries on PLCs sending data to IoTDB as well as 
simplify data-collection gateways.

Chris

Von: Yuchen Ding 
Datum: Freitag, 31. Mai 2024 um 10:14
An: dev@iotdb.apache.org 
Betreff: Re: [Proposal] Data Subscription Client on IoTDB
Hello everyone,

I am VGalaxies. Recently, I have been working on providing TsFile subscription 
support for IoTDB data subscription client. The background of this feature is 
to achieve data file export and backup for multi-replica clusters using data 
subscription. Using the existing data subscription client, the subscribed data 
is in the form of SessionDataSet. The server needs to parse TsFile, and the 
client needs to rewrite TsFile. Raw data is transmitted over the network, 
without leveraging the high compression features of TsFile. Therefore, we hope 
to support exporting TsFile using the TsFile client SDK in data subscriptions.

In terms of functionality, TsFile subscription support for the data 
subscription client includes three steps. First, create a topic with a data 
presentation format of TsFile, by specifying the topic format as TsFileHandler. 
Second, when creating a consumer, specify the directory where the subscribed 
TsFile will be saved using the fileSaveDir parameter. Third, obtain the 
corresponding handler based on the type of SubscriptionMessage, which is 
SubscriptionTsFileHandler. SubscriptionTsFileHandler encapsulates operations 
like cp, mv, and rm from the Java standard library, and can also iterate data 
through the TsFile SDK, i.e., TsFileReader. Users can achieve TsFile 
subscription with minimal configuration.

Technically, we have restructured the Message Payload of the pipeSubscribe RPC 
poll type to support the transmission of data in both SessionDataSet and TsFile 
formats. The transmission of a TsFile file is divided into multiple events, 
including tsfile init event, tsfile piece event, and tsfile seal event. The 
reliable transmission of TsFile files is achieved through the interaction of 
these events between the DN side and the client side.

Currently, TsFile subscription still has some limitations. For example, when 
the format of the topic is TsFileHandler, there are certain constraints on its 
path and time configuration. We will continue to optimize these issues in 
future iterations.

I have initially implemented support for TsFile subscription in this PR[1]. I 
hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
VGalaxies

Reference:
1. https://github.com/apache/iotdb/pull/12326

On 2024/04/08 03:06:59 VGalaxies wrote:
> Hello everyone,
>
> I am VGalaxies, a new contributor to Apache IoTDB. I am excited to
> share with you a new feature that I have been working on for the past
> few months.
>
> The data subscription client is a new way to access data within IoTDB,
> distinct from the traditional method of querying data using SQL-like
> syntax. In scenarios where real-time data, quick response to data
> changes, and building highly event-driven systems are required, data
> subscription has greater advantages over data querying. For example,
> in the following two scenarios:
>
> 1. Replace extensive polling queries for large amounts of data: Avoid
> significant impacts on the performance of existing systems when
> querying frequently or when there are many data points. Also, avoid
> problems with determining the query scope and ensure downstream
> receives accurate full data.
> 2. Facilitate downstream system integration: It's easier to integrate
> with components such as Flink, Spark, Kafka/DataX, Camel/MySQL, etc.
> There's no need to customize the logic of IoTDB's data change capture
> for each big data component separately, simplifying integration
> component design and making it easier for users.
>
> The IoTDB subscription client references some features defined by some
> message queue products like Kafka. It consists of 3 core concepts:
> Topic, Consumer, and Consumer Group.
>
> - Topic is a logical concept used by the IoTDB subscription client to
> classify data, serving as a channel for data publication. Producers
> publish data to specific topics, while consumers subscribe to these
> topics to receive related data. In the IoTDB subscription client,
> topics describe the sequence characteristics, time characteristics,
> presentation forms, and optional custom processing logic of subscribed
> data.
> - Consumer is an application or service in the IoTDB subscription
> client responsible for receiving and processing data published 

Re: [Proposal] Data Subscription Client on IoTDB

[Yuchen Ding]

Hello everyone,

I am VGalaxies. Recently, I have been working on providing TsFile subscription 
support for IoTDB data subscription client. The background of this feature is 
to achieve data file export and backup for multi-replica clusters using data 
subscription. Using the existing data subscription client, the subscribed data 
is in the form of SessionDataSet. The server needs to parse TsFile, and the 
client needs to rewrite TsFile. Raw data is transmitted over the network, 
without leveraging the high compression features of TsFile. Therefore, we hope 
to support exporting TsFile using the TsFile client SDK in data subscriptions.

In terms of functionality, TsFile subscription support for the data 
subscription client includes three steps. First, create a topic with a data 
presentation format of TsFile, by specifying the topic format as TsFileHandler. 
Second, when creating a consumer, specify the directory where the subscribed 
TsFile will be saved using the fileSaveDir parameter. Third, obtain the 
corresponding handler based on the type of SubscriptionMessage, which is 
SubscriptionTsFileHandler. SubscriptionTsFileHandler encapsulates operations 
like cp, mv, and rm from the Java standard library, and can also iterate data 
through the TsFile SDK, i.e., TsFileReader. Users can achieve TsFile 
subscription with minimal configuration.

Technically, we have restructured the Message Payload of the pipeSubscribe RPC 
poll type to support the transmission of data in both SessionDataSet and TsFile 
formats. The transmission of a TsFile file is divided into multiple events, 
including tsfile init event, tsfile piece event, and tsfile seal event. The 
reliable transmission of TsFile files is achieved through the interaction of 
these events between the DN side and the client side.

Currently, TsFile subscription still has some limitations. For example, when 
the format of the topic is TsFileHandler, there are certain constraints on its 
path and time configuration. We will continue to optimize these issues in 
future iterations.

I have initially implemented support for TsFile subscription in this PR[1]. I 
hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
VGalaxies

Reference:
1. https://github.com/apache/iotdb/pull/12326

On 2024/04/08 03:06:59 VGalaxies wrote:
> Hello everyone,
> 
> I am VGalaxies, a new contributor to Apache IoTDB. I am excited to
> share with you a new feature that I have been working on for the past
> few months.
> 
> The data subscription client is a new way to access data within IoTDB,
> distinct from the traditional method of querying data using SQL-like
> syntax. In scenarios where real-time data, quick response to data
> changes, and building highly event-driven systems are required, data
> subscription has greater advantages over data querying. For example,
> in the following two scenarios:
> 
> 1. Replace extensive polling queries for large amounts of data: Avoid
> significant impacts on the performance of existing systems when
> querying frequently or when there are many data points. Also, avoid
> problems with determining the query scope and ensure downstream
> receives accurate full data.
> 2. Facilitate downstream system integration: It's easier to integrate
> with components such as Flink, Spark, Kafka/DataX, Camel/MySQL, etc.
> There's no need to customize the logic of IoTDB's data change capture
> for each big data component separately, simplifying integration
> component design and making it easier for users.
> 
> The IoTDB subscription client references some features defined by some
> message queue products like Kafka. It consists of 3 core concepts:
> Topic, Consumer, and Consumer Group.
> 
> - Topic is a logical concept used by the IoTDB subscription client to
> classify data, serving as a channel for data publication. Producers
> publish data to specific topics, while consumers subscribe to these
> topics to receive related data. In the IoTDB subscription client,
> topics describe the sequence characteristics, time characteristics,
> presentation forms, and optional custom processing logic of subscribed
> data.
> - Consumer is an application or service in the IoTDB subscription
> client responsible for receiving and processing data published to
> specific topics. Consumers retrieve data from the queue and perform
> corresponding processing. The IoTDB subscription client provides two
> types of consumers: pull consumer and push consumer.
> - Consumer Group is a collection of consumers. When different
> consumers in the same consumer group subscribe to the same topic,
> these consumers share the processing progress of data under this
> topic. Each data under this topic can only be processed by one
> consumer within the group, ensuring that 

[Proposal] Support different charsets in IoTDB

[ITAMI SHO]

Hi all,
I’m Itami Sho, a contributor to Apache IoTDB. Currently, IoTDB supports only 
the UTF-8 charset. To enhance the system's flexibility and usability across 
different environments, I have implemented a feature that handles character set 
conversion when the user system's charset does not match IoTDB's default UTF-8.

The main functionality of this feature involves converting input character sets 
to UTF-8 and output character sets to the user-specified charset.

This update will primarily focus on modifying the JDBC interfaces. During the 
initialization of a connection, users can now specify the charset parameter in 
the URL, as demonstrated below:

try (Connection connection = 
DriverManager.getConnection("jdbc:iotdb://127.0.0.1:6667?charset=GB18030", 
"root", "root"); Statement statement = connection.createStatement()) { // Your 
code here }

In future work, I plan to extend support for this feature to other interfaces 
such as Session, REST API, MQTT, and more.

You can check my PR to get more code details: 
https://github.com/apache/iotdb/pull/12618

I look forward to your feedback and suggestions to further improve this 
feature. Your input will be invaluable in ensuring that this enhancement meets 
the needs of the community : )

Best regards,

Itami Sho

[Proposal] Query mode in extraction of Streaming Engine

[??]

Hello everyone,
I'm Caideyipi, and is developing the Streaming Engine module. Here I wish to 
share a new feature I introduced to the "Extraction" phase of Streaming Engine.
Formerly, the Streaming Engine is long restricted to unbounded stream. For 
instance, if users want to extract the historical data only, they can only 
judge the end themselves by querying the receiver side, and drop the pipe 
manually. The inconvenience is apparent in not only the normal scenarios, but 
the data subscription and the tsFile export service, which may need a terminal 
event to halt the entire procedure.
Hence, a bound is introduced in the extractor phase. Now users can use 
"source.mode" in "with source" clause ("source" can be omitted) to decide 
whether to send the terminal event and drop the pipe automatically when the 
transmission is completed. When "source.mode" == "query" (default is 
"subscribe"), when the historical data is all processed/transmitted, a terminal 
event is sent to the subsequent steps. When all the pipe tasks has completed, 
the ConfigNode will collect the information in pipe heartbeats, and drop the 
pipe if all the pipeTasks have completed.
Fervently hope you like the feature, and glad to see any ideas or suggestions.
Thanks for your reading and support.

Sincerely,

Caideyipi

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Xiangdong Huang]

+1 binding
---
Xiangdong Huang


ITAMI SHO  于2024年5月29日周三 16:23写道：
>
> +1
>
> It will increase the stability of our builds!
>
> Itami Sho
>
> > 2024年5月28日 22:40，Christofer Dutz  写道：
> >
> > Hi all,
> >
> > I after fixing some issues in the cmake-maven-plugin, have I updated the 
> > iotdb-bin-resources module to use these.
> > Now we can stage Apache Thrift binaries for all platforms.
> >
> > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible 
> > again), I also updated the Thrift version and staged a release of Thrift 
> > binaries for all of our supported platforms in Nexus.
> >
> > I’ve staged the artifacts at:
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> >
> > Tag Name: iotdb-tools-thrift-v0.20.0.0
> > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> >
> > The source bundle generally consists of a pom.xml and an assembly xml … so 
> > it should be quick to review.
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> >
> > I’m intentionally not staging it on the release-svn, as nobody should be 
> > required to consume this module but us ourselves.
> >
> > The vote stays open for 72hours … assuming enough support, I’ll then finish 
> > changing the main repo to use these artifacts.
> >
> >
> > Chris
>

Re: [Proposal] Load Tsfile Functionality Optimisation

[Xiangpeng Hu]

Hello Xiangzhi,


This is a interesting and helpful new feature. Thank you for your dedication 
and hard work in enhancing Apache IoTDB!


Best regards,
|
Xiangpeng Hu
|
 Replied Message 
| From | ycy wi |
| Date | 5/30/2024 11:47 |
| To |  |
| Subject | Re: [Proposal] Load Tsfile Functionality Optimisation |
Hi Xiangzhi,

Thank you for your work! Load function is something I frequently use when 
testing, and your optimization should greatly enhance my user experience!

By the way, I would like to know what the general causes of failure might be, 
and whether there is an option to retry or handle the failed TsFile?

Best
CaiyinYang



2024年5月30日 11:11，孟祥志  写道：

Hello everyone,

I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share 
with you a feature that I have been working on for the past few months.

load tsfile has always been an important feature for IoTDB, but due to the 
original design, he has some flaws in it, when load a folder fails to import 
some tsfile, it may cause the whole task fails, which is obviously a not very 
user-friendly design, so I decided to modify his.

I'll modify the original load tsfile script to traverse the entire folder 
first, find all the files, and then use the LOAD function to import individual 
files; and I've added a few interesting changes along the way, modifying the 
successful operation parameter to make it easier for people to deal with 
successful files, and added a parameter for failed files to make it easier for 
people to find those failed files, and finally added a new number of threads to 
make the LOAD function work more quickly.


I hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,

Xiangzhi Meng

Reference:
https://github.com/apache/iotdb/pull/12541

Re: [Proposal] Load Tsfile Functionality Optimisation

[ycy wi]

Hi Xiangzhi,

Thank you for your work! Load function is something I frequently use when 
testing, and your optimization should greatly enhance my user experience!

By the way, I would like to know what the general causes of failure might be, 
and whether there is an option to retry or handle the failed TsFile?

Best
CaiyinYang



> 2024年5月30日 11:11，孟祥志  写道：
> 
> Hello everyone,
> 
> I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share 
> with you a feature that I have been working on for the past few months.
> 
> load tsfile has always been an important feature for IoTDB, but due to the 
> original design, he has some flaws in it, when load a folder fails to import 
> some tsfile, it may cause the whole task fails, which is obviously a not very 
> user-friendly design, so I decided to modify his.
> 
> I'll modify the original load tsfile script to traverse the entire folder 
> first, find all the files, and then use the LOAD function to import 
> individual files; and I've added a few interesting changes along the way, 
> modifying the successful operation parameter to make it easier for people to 
> deal with successful files, and added a parameter for failed files to make it 
> easier for people to find those failed files, and finally added a new number 
> of threads to make the LOAD function work more quickly.
> 
> 
> I hope you are interested in this feature and would like to participate in 
> the development and testing. You can also leave your comments and suggestions 
> in this thread. Appreciate any suggestion/feedback & contribution.
> 
> Thank you for your attention and support.
> 
> Best regards,
> 
> Xiangzhi Meng
> 
> Reference:
> https://github.com/apache/iotdb/pull/12541

[Proposal] Load Tsfile Functionality Optimisation

[孟祥志]

Hello everyone,

I am Xiangzhi Meng, a new contributor to Apache IoTDB. I am excited to share 
with you a feature that I have been working on for the past few months.

load tsfile has always been an important feature for IoTDB, but due to the 
original design, he has some flaws in it, when load a folder fails to import 
some tsfile, it may cause the whole task fails, which is obviously a not very 
user-friendly design, so I decided to modify his.

I'll modify the original load tsfile script to traverse the entire folder 
first, find all the files, and then use the LOAD function to import individual 
files; and I've added a few interesting changes along the way, modifying the 
successful operation parameter to make it easier for people to deal with 
successful files, and added a parameter for failed files to make it easier for 
people to find those failed files, and finally added a new number of threads to 
make the LOAD function work more quickly.


I hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,

Xiangzhi Meng

Reference:
https://github.com/apache/iotdb/pull/12541

[Proposal] New Columns in "Show Pipes" on IoTDB

[??]

Hello everyone,
I'm Caideyipi, a contributor to IoTDB, currently working on Streaming Engine. 
Here I wish to share a new feature I introduced to IoTDB "Show Pipes" SQL.
Previously, the "Show Pipes" SQL contains 7 columns, like 
"ID|CreationTime|State|PipeSource|PipeProcessor|PipeSink|ExceptionMessage". By 
that, we can easily see the pipes' definitions, statuses, and exceptions if 
exist. However, the pipe's current progress is left unclear, and users must 
query the receiver or browse the dashboard to acquire the progress. In some 
typical scenarios, not only the dashboard is absent due to resource limitation, 
but the receiver is inaccessible for secrecy concerns. In that circumstance, 
the progress may not be easily known by users.
Hence, two new columns, "RemainingEventCount" and "EstimatedRemainingSeconds", 
are introduced in show pipes result. They are also introduced to the dashboard. 
Detailed illustrations are provided below.
- "RemainingEventCount" is that how many events are left 
unprocessed/untransferred in the pipe framework, consisting of:
   1. The tsfile, insertion, raw tablet, heartbeat and schema events.
   2. Events in the config region and all the schema regions and data regions.
- "EstimatedRemainingSeconds" is the estimated elapsed seconds when the 
Streaming Engine has processed/transferred all the existing data. It's 
calculated by the "RemainingEventCount" of each region, and uses each region's 
average event commit rate in one minute in calculation. The commit rate will 
apply the "single exponential smoothing" method when updated to get more 
precise result, and the largest displayed time is 1 year.
To avoid affecting the cluster's performance, the two values are updated in the 
pipe's heartbeat and may have some delay. As an improvement, in a cluster 
having only one dataNode, if the pipe comprising only data region and schema 
region, it's calculation is done at the dataNode and is refreshed on query.
I heartily hope that you are interested in this feature. Any participation in 
testing, and any comments or suggestions in this thread are welcome.

Thanks for your reading and support.

Sincerely,

Caideyipi

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[ITAMI SHO]

+1

It will increase the stability of our builds!

Itami Sho

> 2024年5月28日 22:40，Christofer Dutz  写道：
> 
> Hi all,
> 
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
> 
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
> 
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> 
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> 
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> 
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
> 
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
> 
> 
> Chris

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[ycy wi]

+1 
It will significantly improved our build stability and cross-platform support.

> 2024年5月28日 22:40，Christofer Dutz  写道：
> 
> Hi all,
> 
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
> 
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
> 
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> 
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> 
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> 
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
> 
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
> 
> 
> Chris

Re: Could anyone please approve this PR?

[Jialin Qiao]

This is merged :)


Jialin Qiao

Christofer Dutz  于2024年5月28日周二 22:23写道：
>
> Hi all,
>
> in the iotdb-bin-resources I have also updated the Cmake version and thrift 
> version in order to be able to deploy new versions of thrift artifacts.
>
> https://github.com/apache/iotdb-bin-resources/pull/61
>
> Would be cool if I could get some approval.
>
> Chris

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Jialin Qiao]

+1 (binding)


Jialin Qiao

Yuan Tian  于2024年5月29日周三 09:07写道：
>
> +1 (binding)
>
> On Tue, May 28, 2024 at 10:40 PM Christofer Dutz 
> wrote:
>
> > Hi all,
> >
> > I after fixing some issues in the cmake-maven-plugin, have I updated the
> > iotdb-bin-resources module to use these.
> > Now we can stage Apache Thrift binaries for all platforms.
> >
> > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible
> > again), I also updated the Thrift version and staged a release of Thrift
> > binaries for all of our supported platforms in Nexus.
> >
> > I’ve staged the artifacts at:
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> >
> > Tag Name: iotdb-tools-thrift-v0.20.0.0
> >
> > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> >
> > The source bundle generally consists of a pom.xml and an assembly xml … so
> > it should be quick to review.
> >
> > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> >
> > I’m intentionally not staging it on the release-svn, as nobody should be
> > required to consume this module but us ourselves.
> >
> > The vote stays open for 72hours … assuming enough support, I’ll then
> > finish changing the main repo to use these artifacts.
> >
> >
> > Chris
> >

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Yuan Tian]

+1 (binding)

On Tue, May 28, 2024 at 10:40 PM Christofer Dutz 
wrote:

> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible
> again), I also updated the Thrift version and staged a release of Thrift
> binaries for all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
>
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so
> it should be quick to review.
>
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then
> finish changing the main repo to use these artifacts.
>
>
> Chris
>

AW: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

+1 (binding)

Chris

[OK] Checked the signatures
[OK] Checked the hashes
[OK] Checked the content of the source bundle
[OK] Checked that the staged binaries work on their target platforms



Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 16:40
An: dev@iotdb.apache.org 
Betreff: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)
Hi all,

I after fixing some issues in the cmake-maven-plugin, have I updated the 
iotdb-bin-resources module to use these.
Now we can stage Apache Thrift binaries for all platforms.

In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), I 
also updated the Thrift version and staged a release of Thrift binaries for all 
of our supported platforms in Nexus.

I’ve staged the artifacts at:
https://repository.apache.org/content/repositories/orgapacheiotdb-1158

Tag Name: iotdb-tools-thrift-v0.20.0.0
https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0

The source bundle generally consists of a pom.xml and an assembly xml … so it 
should be quick to review.
https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip

I’m intentionally not staging it on the release-svn, as nobody should be 
required to consume this module but us ourselves.

The vote stays open for 72hours … assuming enough support, I’ll then finish 
changing the main repo to use these artifacts.


Chris

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Haonan Hou]

+1

Haonan Hou

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
> 
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
> 
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
> 
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
> 
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
> 
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
> 
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
> 
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
> 
> 
> Chris
> 

[DISCUSS] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Hi all,

this vote is not on switching to Thrift 0.20.0. In order to try out Thrift we 
however need to have the Thrift binaries available.
That’s why I’ve started a VOTE on publishing these.

The alternative would be to work with intermediate maven repos, which just 
makes things a lot more difficult.

Chris

[VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Hi all,

I after fixing some issues in the cmake-maven-plugin, have I updated the 
iotdb-bin-resources module to use these.
Now we can stage Apache Thrift binaries for all platforms.

In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), I 
also updated the Thrift version and staged a release of Thrift binaries for all 
of our supported platforms in Nexus.

I’ve staged the artifacts at:
https://repository.apache.org/content/repositories/orgapacheiotdb-1158

Tag Name: iotdb-tools-thrift-v0.20.0.0
https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0

The source bundle generally consists of a pom.xml and an assembly xml … so it 
should be quick to review.
https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip

I’m intentionally not staging it on the release-svn, as nobody should be 
required to consume this module but us ourselves.

The vote stays open for 72hours … assuming enough support, I’ll then finish 
changing the main repo to use these artifacts.


Chris

Could anyone please approve this PR?

[Christofer Dutz]

Hi all,

in the iotdb-bin-resources I have also updated the Cmake version and thrift 
version in order to be able to deploy new versions of thrift artifacts.

https://github.com/apache/iotdb-bin-resources/pull/61

Would be cool if I could get some approval.

Chris

AW: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Ok … and done:
https://repository.apache.org/content/repositories/orgapacheiotdb-1158

Here are all 6 types of binaries.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 11:01
An: dev@iotdb.apache.org 
Betreff: AW: [DISCUSS] Investigate updating to Thrift 0.20.0?
Ok …

So, I’m currently working on staging the last of the binaries (thrift for 
windows x86_64).
However, in order to make this work I think we would also need to update the 
CMake version.
I spent a lot of time over the last weekends figuring out how to support Cmake 
on all of these platforms with the maintainer of the cmake-maven-plugin.
Now we have a version that works on all systems I have.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 08:58
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?
Happy to prepare the binaries for this.

Chris

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Tuesday, May 28, 2024 3:37:30 AM
To: dev-iotdb 
Subject: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
>
> Chris

AW: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Ok …

So, I’m currently working on staging the last of the binaries (thrift for 
windows x86_64).
However, in order to make this work I think we would also need to update the 
CMake version.
I spent a lot of time over the last weekends figuring out how to support Cmake 
on all of these platforms with the maintainer of the cmake-maven-plugin.
Now we have a version that works on all systems I have.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 08:58
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?
Happy to prepare the binaries for this.

Chris

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Tuesday, May 28, 2024 3:37:30 AM
To: dev-iotdb 
Subject: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
>
> Chris

Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Happy to prepare the binaries for this.

Chris

Gesendet von Outlook für Android

From: Haonan Hou 
Sent: Tuesday, May 28, 2024 3:37:30 AM
To: dev-iotdb 
Subject: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
>
> Chris

Re: Missing binaries for Windows on Aarch64 systems

[Christofer Dutz]

Yeah.

But I could also produce a new set of binaries for 0.14.1...we'd have to 
re-release them too.

Chris

Gesendet von Outlook für Android

From: Jialin Qiao 
Sent: Tuesday, May 28, 2024 3:39:59 AM
To: dev@iotdb.apache.org 
Subject: Re: Missing binaries for Windows on Aarch64 systems

Hi,

Is this will also be fixed by upgrading thrift to 0.20?

Jialin Qiao

Christofer Dutz  于2024年5月24日周五 22:58写道：
>
> Hi all,
>
> I was just verifying the build using the most recent cmake-maven-plugin works 
> and stumbled over something.
>
> Currently we don’t have thrift binaries for combinations of systems. It seems 
> we’re missing windows-aarch64.
>
> Should we plan fixing this and doing a release 0.14.1.1?
>
> Chris

Re: Upgrading 0.12 -> 1.3

[Trevor Hart]

Thank you Jialin. I didnt realise the data was not migrated as part of the 
upgrade.



That is good to see there is a resource to migrate. I will use that approach.



Thanks 

Trevor Hart




 On Tue, 28 May 2024 13:35:41 +1200 Jialin Qiao  
wrote ---



Hi, 
 
The architecture of 1.x is different from 0.13. The upgrading can not 
be in situ. You need to deploy an 1.3 instance, then transfer the 
data. 
There are two ways to transfer data: 
1. Using Session to query from the 0.13, then write into 1.3. 
2. Using TsFile API to rewrite the TsFile, then load TsFiles to 1.3. 
 
1. is easier. You could refer to this: 
https://github.com/apache/iotdb/blob/master/example/session/src/main/java/org/apache/iotdb/DataMigrationExample.java
 
 
Jialin Qiao

Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Haonan Hou]

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
> 
> Hi all,
> 
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
> 
> https://github.com/apache/thrift/blob/master/CHANGES.md
> 
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
> 
> 
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
> 
> Chris

Re: Missing binaries for Windows on Aarch64 systems

[Jialin Qiao]

Hi,

Is this will also be fixed by upgrading thrift to 0.20?

Jialin Qiao

Christofer Dutz  于2024年5月24日周五 22:58写道：
>
> Hi all,
>
> I was just verifying the build using the most recent cmake-maven-plugin works 
> and stumbled over something.
>
> Currently we don’t have thrift binaries for combinations of systems. It seems 
> we’re missing windows-aarch64.
>
> Should we plan fixing this and doing a release 0.14.1.1?
>
> Chris

Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Jialin Qiao]

Hi,

Looks good :D
If the upgrading does not bring any change to our users, +1 for upgrading.
You could prepare a PR, then we perform some validations.

Jialin Qiao

Christofer Dutz  于2024年5月28日周二 03:41写道：
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>   *   Mac – aarch64
>   *   Mac – x86_64
>   *   Linux  – aarch64
>   *   Linux  – x86_64
>   *   Windows  – aarch64
>   *   Windows  – x86_64
>
> Chris

Re: Upgrading 0.12 -> 1.3

[Jialin Qiao]

Hi,

The architecture of 1.x is different from 0.13. The upgrading can not
be in situ. You need to deploy an 1.3 instance, then transfer the
data.
There are two ways to transfer data:
1. Using Session to query from the 0.13, then write into 1.3.
2. Using TsFile API to rewrite the TsFile, then load TsFiles to 1.3.

1. is easier. You could refer to this:
https://github.com/apache/iotdb/blob/master/example/session/src/main/java/org/apache/iotdb/DataMigrationExample.java

Jialin Qiao

Trevor Hart  于2024年5月28日周二 06:27写道：
>
> Hello
>
>
>
> I have an application that is running against IoTDB version 0.12 which I 
> would like to upgrade to 1.3
>
>
>
> However I have some timeseries paths that contain numbers (eg below)
>
>
> root.ABC.health.url.17.code
>
>
>
> From the documentation it seems this is not supported beyond 0.13
>
>
>
> To migrate this data I am proposing this process;
>
>
>
> 1. Use "select into" to migrate to a new path ie root.ABC.health.url.17.code 
> -> root.ABC.health.url.U17.code
>
> 2. Delete the old timeseries ie root.ABC.health.url.17.code
>
> 3. Upgrade IoTDB
>
>
> Is this the correct/best approach?
>
>
> Thanks
>
> Trevor Hart

Upgrading 0.12 -> 1.3

[Trevor Hart]

Hello



I have an application that is running against IoTDB version 0.12 which I would 
like to upgrade to 1.3



However I have some timeseries paths that contain numbers (eg below)


root.ABC.health.url.17.code



>From the documentation it seems this is not supported beyond 0.13



To migrate this data I am proposing this process;



1. Use "select into" to migrate to a new path ie root.ABC.health.url.17.code -> 
root.ABC.health.url.U17.code

2. Delete the old timeseries ie root.ABC.health.url.17.code

3. Upgrade IoTDB


Is this the correct/best approach?


Thanks 

Trevor Hart

[DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Hi all,

while doing some experiments over the weekend I came to notice that Thrift 
0.19.0 and above is again compatible with Java 8 … this was the major reason 
for us not updating to versions without CVEs.
I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.

https://github.com/apache/thrift/blob/master/CHANGES.md

Might be worth investigating to update? I’ve already got an up-to-date version 
of the thrift compiler … if we wanted to try it out, I could stage release 
versions for:


  *   Mac – aarch64
  *   Mac – x86_64
  *   Linux  – aarch64
  *   Linux  – x86_64
  *   Windows  – aarch64
  *   Windows  – x86_64

Chris

Missing binaries for Windows on Aarch64 systems

[Christofer Dutz]

Hi all,

I was just verifying the build using the most recent cmake-maven-plugin works 
and stumbled over something.

Currently we don’t have thrift binaries for combinations of systems. It seems 
we’re missing windows-aarch64.

Should we plan fixing this and doing a release 0.14.1.1?

Chris

Develop branch of TsFile for IoTDB

[Yuan Tian]

Hi all,

Just want every developer in the IoTDB community that the branchiotdb of
the TsFile project is used to deploy the SNAPSHOT version for the IoTDB
master branch.

I also submit a pr[1] to add some explanations in README of IoTDB.

[1] https://github.com/apache/iotdb/pull/12581


Best regards,
---
Yuan Tian

Re: Introducing a memory control mechanism during the query planning stage #12573

[谭新宇]

Hi, lanyu and hongying

I am very pleased to see that you have made breakthroughs in memory management, 
and I am so eager to learn memory control techniques from you!

Best
———
Xinyu Tan

> 2024年5月23日 15:24，Yuan Tian  写道：
> 
> Hi hongying,
> 
> sure, feel free to discuss anything about any modules of IoTDB in mail list.
> 
> On Thu, May 23, 2024 at 2:42 PM 张洪胤  wrote:
> 
>> hi, yuan tian, I am investigating the memory management mechanisms of
>> Oracle, PostgreSQL, and other databases, and sorting out the current memory
>> management status of Apache IoTDB. I believe that a unified memory
>> management framework cannot be separated from the discussion and
>> participation of the entire community, as well as the participation of open
>> source workers who specialize in different modules.
>> 
>> 
>> 
>> 
>> Best regards,
>> 
>> Hongyin Zhang
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> At 2024-05-23 14:25:01, "Yuan Tian"  wrote:
>>> Hi hongying,
>>> 
>>> looking forward to your unified memory management framework for IoTDB.
>>> 
>>> On Thu, May 23, 2024 at 12:11 PM Hongyin Zhang 
>> wrote:
>>> 
 +1, brilliant job! This work allowed me to learn a lot, which laid a
>> solid
 foundation for subsequent system memory control and improved the
>> system's
 robustness in query scenarios.
 
 Best regards,
 Hongyin Zhang
 
 
 
 
 
 
 
 
 
 
 
 At 2024-05-23 11:08:35, "Liao Lanyu" <1435078...@qq.com.INVALID> wrote:
> Hi,
> Currently, the IoTDB query engine does not implement memory control at
 the FE (Frontend) stage. In scenarios with massive series queries (e.g.,
 select * from root.**), the query plans generated at the FE stage can
 become excessively large. Roughly estimating, the size of a single
 SeriesScanNode is about 1/2 KB, which means that two million series
 corresponding to two million SeriesScanNodes would occupy 1 GB, posing a
 potential risk of Out-Of-Memory (OOM). In high concurrency scenarios,
>> even
 if a single query plan is not large, the total memory occupied by
>> multiple
 query plans can still lead to OOM.
> Therefore, it is now desired to introduce memory size control for FE
 query plans within the query engine.
> The PR is:https://github.com/apache/iotdb/pull/12573
> 
> 
> 
> 
> 1435078631
> 1435078...@qq.com
> 
> 
> 
> 
 
>> 

Re: Re: Introducing a memory control mechanism during the query planning stage #12573

[Yuan Tian]

Hi hongying,

sure, feel free to discuss anything about any modules of IoTDB in mail list.

On Thu, May 23, 2024 at 2:42 PM 张洪胤  wrote:

> hi, yuan tian, I am investigating the memory management mechanisms of
> Oracle, PostgreSQL, and other databases, and sorting out the current memory
> management status of Apache IoTDB. I believe that a unified memory
> management framework cannot be separated from the discussion and
> participation of the entire community, as well as the participation of open
> source workers who specialize in different modules.
>
>
>
>
> Best regards,
>
> Hongyin Zhang
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2024-05-23 14:25:01, "Yuan Tian"  wrote:
> >Hi hongying,
> >
> >looking forward to your unified memory management framework for IoTDB.
> >
> >On Thu, May 23, 2024 at 12:11 PM Hongyin Zhang 
> wrote:
> >
> >> +1, brilliant job! This work allowed me to learn a lot, which laid a
> solid
> >> foundation for subsequent system memory control and improved the
> system's
> >> robustness in query scenarios.
> >>
> >> Best regards,
> >> Hongyin Zhang
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> At 2024-05-23 11:08:35, "Liao Lanyu" <1435078...@qq.com.INVALID> wrote:
> >> >Hi,
> >> >Currently, the IoTDB query engine does not implement memory control at
> >> the FE (Frontend) stage. In scenarios with massive series queries (e.g.,
> >> select * from root.**), the query plans generated at the FE stage can
> >> become excessively large. Roughly estimating, the size of a single
> >> SeriesScanNode is about 1/2 KB, which means that two million series
> >> corresponding to two million SeriesScanNodes would occupy 1 GB, posing a
> >> potential risk of Out-Of-Memory (OOM). In high concurrency scenarios,
> even
> >> if a single query plan is not large, the total memory occupied by
> multiple
> >> query plans can still lead to OOM.
> >> >Therefore, it is now desired to introduce memory size control for FE
> >> query plans within the query engine.
> >> >The PR is:https://github.com/apache/iotdb/pull/12573
> >> >
> >> >
> >> >
> >> >
> >> >1435078631
> >> >1435078...@qq.com
> >> >
> >> >
> >> >
> >> >
> >>
>

Re:Re: Introducing a memory control mechanism during the query planning stage #12573

[张洪胤]

hi, yuan tian, I am investigating the memory management mechanisms of Oracle, 
PostgreSQL, and other databases, and sorting out the current memory management 
status of Apache IoTDB. I believe that a unified memory management framework 
cannot be separated from the discussion and participation of the entire 
community, as well as the participation of open source workers who specialize 
in different modules.




Best regards,

Hongyin Zhang














At 2024-05-23 14:25:01, "Yuan Tian"  wrote:
>Hi hongying,
>
>looking forward to your unified memory management framework for IoTDB.
>
>On Thu, May 23, 2024 at 12:11 PM Hongyin Zhang  wrote:
>
>> +1, brilliant job! This work allowed me to learn a lot, which laid a solid
>> foundation for subsequent system memory control and improved the system's
>> robustness in query scenarios.
>>
>> Best regards,
>> Hongyin Zhang
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> At 2024-05-23 11:08:35, "Liao Lanyu" <1435078...@qq.com.INVALID> wrote:
>> >Hi,
>> >Currently, the IoTDB query engine does not implement memory control at
>> the FE (Frontend) stage. In scenarios with massive series queries (e.g.,
>> select * from root.**), the query plans generated at the FE stage can
>> become excessively large. Roughly estimating, the size of a single
>> SeriesScanNode is about 1/2 KB, which means that two million series
>> corresponding to two million SeriesScanNodes would occupy 1 GB, posing a
>> potential risk of Out-Of-Memory (OOM). In high concurrency scenarios, even
>> if a single query plan is not large, the total memory occupied by multiple
>> query plans can still lead to OOM.
>> >Therefore, it is now desired to introduce memory size control for FE
>> query plans within the query engine.
>> >The PR is:https://github.com/apache/iotdb/pull/12573
>> >
>> >
>> >
>> >
>> >1435078631
>> >1435078...@qq.com
>> >
>> >
>> >
>> >
>>

Re: Introducing a memory control mechanism during the query planning stage #12573

[Yuan Tian]

Hi hongying,

looking forward to your unified memory management framework for IoTDB.

On Thu, May 23, 2024 at 12:11 PM Hongyin Zhang  wrote:

> +1, brilliant job! This work allowed me to learn a lot, which laid a solid
> foundation for subsequent system memory control and improved the system's
> robustness in query scenarios.
>
> Best regards,
> Hongyin Zhang
>
>
>
>
>
>
>
>
>
>
>
> At 2024-05-23 11:08:35, "Liao Lanyu" <1435078...@qq.com.INVALID> wrote:
> >Hi,
> >Currently, the IoTDB query engine does not implement memory control at
> the FE (Frontend) stage. In scenarios with massive series queries (e.g.,
> select * from root.**), the query plans generated at the FE stage can
> become excessively large. Roughly estimating, the size of a single
> SeriesScanNode is about 1/2 KB, which means that two million series
> corresponding to two million SeriesScanNodes would occupy 1 GB, posing a
> potential risk of Out-Of-Memory (OOM). In high concurrency scenarios, even
> if a single query plan is not large, the total memory occupied by multiple
> query plans can still lead to OOM.
> >Therefore, it is now desired to introduce memory size control for FE
> query plans within the query engine.
> >The PR is:https://github.com/apache/iotdb/pull/12573
> >
> >
> >
> >
> >1435078631
> >1435078...@qq.com
> >
> >
> >
> >
>

Re:Introducing a memory control mechanism during the query planning stage #12573

[Hongyin Zhang]

+1, brilliant job! This work allowed me to learn a lot, which laid a solid 
foundation for subsequent system memory control and improved the system's 
robustness in query scenarios.

Best regards,
Hongyin Zhang











At 2024-05-23 11:08:35, "Liao Lanyu" <1435078...@qq.com.INVALID> wrote:
>Hi,
>Currently, the IoTDB query engine does not implement memory control at the FE 
>(Frontend) stage. In scenarios with massive series queries (e.g., select * 
>from root.**), the query plans generated at the FE stage can become 
>excessively large. Roughly estimating, the size of a single SeriesScanNode is 
>about 1/2 KB, which means that two million series corresponding to two million 
>SeriesScanNodes would occupy 1 GB, posing a potential risk of Out-Of-Memory 
>(OOM). In high concurrency scenarios, even if a single query plan is not 
>large, the total memory occupied by multiple query plans can still lead to OOM.
>Therefore, it is now desired to introduce memory size control for FE query 
>plans within the query engine.
>The PR is:https://github.com/apache/iotdb/pull/12573
>
>
>
>
>1435078631
>1435078...@qq.com
>
>
>
>

Re: Introducing a memory control mechanism during the query planning stage #12573

[William Song]

Hi Lanyu,

Bravo work! Learned a lot from your PR.

Best
Ziyang

> 2024年5月23日 11:08，Liao Lanyu <1435078...@qq.com.INVALID> 写道：
> 
> Hi,
> Currently, the IoTDB query engine does not implement memory control at the FE 
> (Frontend) stage. In scenarios with massive series queries (e.g., select * 
> from root.**), the query plans generated at the FE stage can become 
> excessively large. Roughly estimating, the size of a single SeriesScanNode is 
> about 1/2 KB, which means that two million series corresponding to two 
> million SeriesScanNodes would occupy 1 GB, posing a potential risk of 
> Out-Of-Memory (OOM). In high concurrency scenarios, even if a single query 
> plan is not large, the total memory occupied by multiple query plans can 
> still lead to OOM.
> Therefore, it is now desired to introduce memory size control for FE query 
> plans within the query engine.
> The PR is:https://github.com/apache/iotdb/pull/12573
> 
> 
> 
> 
> 1435078631
> 1435078...@qq.com
> 
> 
> 
> 

Re: Introducing a memory control mechanism during the query planning stage #12573

[ITAMI SHO]

+1,
Nice job

Best regards,
Itami Sho


> 2024年5月23日 11:08，Liao Lanyu <1435078...@qq.com.INVALID> 写道：
> 
> Hi,
> Currently, the IoTDB query engine does not implement memory control at the FE 
> (Frontend) stage. In scenarios with massive series queries (e.g., select * 
> from root.**), the query plans generated at the FE stage can become 
> excessively large. Roughly estimating, the size of a single SeriesScanNode is 
> about 1/2 KB, which means that two million series corresponding to two 
> million SeriesScanNodes would occupy 1 GB, posing a potential risk of 
> Out-Of-Memory (OOM). In high concurrency scenarios, even if a single query 
> plan is not large, the total memory occupied by multiple query plans can 
> still lead to OOM.
> Therefore, it is now desired to introduce memory size control for FE query 
> plans within the query engine.
> The PR is:https://github.com/apache/iotdb/pull/12573
> 
> 
> 
> 
> 1435078631
> 1435078...@qq.com
> 
> 
> 
> 

Re: Introducing a memory control mechanism during the query planning stage #12573

[yanze chen]

+1. I wanted to express my admiration for the remarkable progress you've
made with the IoTDB query engine! Implementing memory control for FE query
plans would significantly enhance our system's robustness and reliability.

Best regards,
Yanze Chen

Liao Lanyu <1435078...@qq.com.invalid> 于2024年5月23日周四 11:08写道：

> Hi,
> Currently, the IoTDB query engine does not implement memory control at the
> FE (Frontend) stage. In scenarios with massive series queries (e.g., select
> * from root.**), the query plans generated at the FE stage can become
> excessively large. Roughly estimating, the size of a single SeriesScanNode
> is about 1/2 KB, which means that two million series corresponding to two
> million SeriesScanNodes would occupy 1 GB, posing a potential risk of
> Out-Of-Memory (OOM). In high concurrency scenarios, even if a single query
> plan is not large, the total memory occupied by multiple query plans can
> still lead to OOM.
> Therefore, it is now desired to introduce memory size control for FE query
> plans within the query engine.
> The PR is:https://github.com/apache/iotdb/pull/12573
>
>
>
>
> 1435078631
> 1435078...@qq.com
>
>
>
> 



-- 
Yanze Chen
School of Software, Tsinghua University

Re: Introducing a memory control mechanism during the query planning stage #12573

[Yuan Tian]

Hi lanyu,

Nice work!

On Thu, May 23, 2024 at 11:09 AM Liao Lanyu <1435078...@qq.com.invalid>
wrote:

> Hi,
> Currently, the IoTDB query engine does not implement memory control at the
> FE (Frontend) stage. In scenarios with massive series queries (e.g., select
> * from root.**), the query plans generated at the FE stage can become
> excessively large. Roughly estimating, the size of a single SeriesScanNode
> is about 1/2 KB, which means that two million series corresponding to two
> million SeriesScanNodes would occupy 1 GB, posing a potential risk of
> Out-Of-Memory (OOM). In high concurrency scenarios, even if a single query
> plan is not large, the total memory occupied by multiple query plans can
> still lead to OOM.
> Therefore, it is now desired to introduce memory size control for FE query
> plans within the query engine.
> The PR is:https://github.com/apache/iotdb/pull/12573
>
>
>
>
> 1435078631
> 1435078...@qq.com
>
>
>
> 

Re: Windows Installer

[Jialin Qiao]

Hi,

That's Great!

If adding this installer, how much will our distribution package increase?

Jialin Qiao

Wang Critas  于2024年5月20日周一 09:57写道：
>
> Hi Trevor,
>
> So cool.
>
> Perhaps you can integrate it into iotdb/distribution. that would be more great
>
> Xuan Wang
>
> 发件人: Trevor Hart 
> 日期: 星期一, 2024年5月20日 09:48
> 收件人: Dev 
> 主题: Windows Installer
> Hello
>
>
>
> I dont know if there is some interest in this but I have built a Windows 
> installer for installing IoTDB on Windows servers.
>
>
>
> The installer includes Open JDK 11 and also does the following;
>
>
>
> 1. Creates a Windows service for the Config Node
>
> 2. Creates a Windows service for the Data Node
>
> 3. Open port 6667 on Windows firewall
>
>
>
> You can see the installer in action here; 
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6BzCd-vAiGc%26t%3D1s%26ab_channel%3DOpeLtd=05%7C02%7C%7Cc4c099493f3e49494ae008dc786ef772%7C84df9e7fe9f640afb435%7C1%7C0%7C638517665208767354%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=seKLvYsiEkBQsjdw6KFaIIUvL1%2FFtv7G36VQC2HVyb8%3D=0
>
>
>
> If there is any interest in this I can make it available.
>
>
> Thanks
>
> Trevor Hart

Re: Handling Duplicate Timestamps

[Jialin Qiao]

Hi Trevor,

Yes, IoTDB cannot handle this scenario currently because our primary
key is Path + Timestamp.

This year we will focus on the table model, a lot work to do :-)

Jialin Qiao

Trevor Hart  于2024年5月20日周一 09:45写道：
>
> Hi Jialin
>
>
>
> Yes the values would be different.
>
>
>
> As as example, these are from a web server log. The device is openzweb01 
> which is an IIS web server which may handle multiple requests at the same 
> time. The rows are unique in their own right but the timestamp is the same in 
> the logging.
>
>
>
> 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7
>
>
> 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6
>
>
> 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6
>
> 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5
>
>
> 2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6
>
> 2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6
>
>
>
> This particular log file only records in seconds. So what I am doing with 
> these rows at the moment is to add an artitifical millisecond to enforce 
> uniqueness.
>
>
>
>
> 2024-05-20 00:00:14.000 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7
>
> 2024-05-20 00:00:14.001 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6
>
> 2024-05-20 00:00:14.002 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6
>
> 2024-05-20 00:00:14.003 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5
>
> 2024-05-20 00:00:15.000 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6
>
> 2024-05-20 00:00:15.001 W3SVC1 openzweb01 192.168.3.69 POST 
> /portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6
>
>
>
> For some other log files that I am processing they are in milliseconds 
> already but there is a (small) chance of dataloss if multiple requests happen 
> to be processed at the exact same time.
>
>
>
> I have been thinking about this some more and I think that rather than break 
> the IoTDB CRUD model I should handle this on the client side. In my use case 
> the log data is actually staged in an H2 database before it is sent to IoTDB 
> so I can enforce PK validation there. That way it is less expensive that 
> checking the timestamp in IoTDB for each record.
>
>
>
> Thanks
>
> Trevor Hart
>
>
>
>
>
>
>
>
>  On Fri, 17 May 2024 19:11:13 +1200 Jialin Qiao  
> wrote ---
>
>
>
> Hi Trevor,
>
> Will different values of the same timestamp be the same？
>
> 1. Same
> Time, Value
> 1, 1
> 1, 1
> 1, 1
>
> 2. Different
> Time, Value
> 1, 1
> 1, 2
> 1, 1
>
>
> Jialin Qiao
>
> Trevor Hart  于2024年5月14日周二 11:20写道：
> >
> > Thank you! I will implment some work around for now.
> >
> >
> > I would appreciate some consideration for this option in the future.
> >
> >
> > Thanks
> >
> > Trevor Hart
> >
> > Ope Limited
> >
> > w: http://www.ope.nz/
> >
> > m: +64212728039
> >
> >
> >
> >
> >
> >
> >
> >
> >  On Tue, 14 May 2024 15:17:47 +1200 Xiangdong Huang 
> >  wrote ---
> >
> >
> >
> > > 1. Checking before insert if the timestamp already exists and remedy on 
> > > the client before resend
> > > 2. Moving to Nanosecond and introducing some insignificant time value to 
> > > keep timestamp values unique.
> > Yes these maybe the best solutions for a specific application.
> >
> >
> > Analysis for IoTDB:
> > - Rejecting the write when receiving an existing timestamp in IoTDB is
> > time-costly (IoTDB needs to check historical data). I think we will do
> > not check it until we find a low-latency method.
> > - Allowing multiple value versions for a timestamp may introduce a
> > chain reaction and there may be a lot of codes that should be
> > modified, which is a huge work.
> >
> > There is a new idea (but I have no time to implement it...)
> > - Add a parameter in IoTDB: replace_strategy: first, last, avg etc...
> > - when an existing timestamp arrives, IoTDB accepts it
> > - when IoTDB runs LSM to merge data and meets multiple values for a
> > timestamp, then handles it according to the replace_startegy.
> >
> > The solution may also introduce some work to do... and we need to
> > think carefully the impact to the query process.
> > Need to survey whether this is a common requirement.
> >
> > Best,
> > ---
> > Xiangdong Huang
> >
> > Trevor Hart  

答复: Windows Installer

[Wang Critas]

Hi Trevor,

So cool.

Perhaps you can integrate it into iotdb/distribution. that would be more great

Xuan Wang

发件人: Trevor Hart 
日期: 星期一, 2024年5月20日 09:48
收件人: Dev 
主题: Windows Installer
Hello



I dont know if there is some interest in this but I have built a Windows 
installer for installing IoTDB on Windows servers.



The installer includes Open JDK 11 and also does the following;



1. Creates a Windows service for the Config Node

2. Creates a Windows service for the Data Node

3. Open port 6667 on Windows firewall



You can see the installer in action here; 
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6BzCd-vAiGc%26t%3D1s%26ab_channel%3DOpeLtd=05%7C02%7C%7Cc4c099493f3e49494ae008dc786ef772%7C84df9e7fe9f640afb435%7C1%7C0%7C638517665208767354%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=seKLvYsiEkBQsjdw6KFaIIUvL1%2FFtv7G36VQC2HVyb8%3D=0



If there is any interest in this I can make it available.


Thanks

Trevor Hart

Windows Installer

[Trevor Hart]

Hello



I dont know if there is some interest in this but I have built a Windows 
installer for installing IoTDB on Windows servers.



The installer includes Open JDK 11 and also does the following;



1. Creates a Windows service for the Config Node

2. Creates a Windows service for the Data Node

3. Open port 6667 on Windows firewall



You can see the installer in action here; 
https://www.youtube.com/watch?v=6BzCd-vAiGc=1s_channel=OpeLtd



If there is any interest in this I can make it available.


Thanks

Trevor Hart

Re: Handling Duplicate Timestamps

[Trevor Hart]

Hi Jialin



Yes the values would be different.



As as example, these are from a web server log. The device is openzweb01 which 
is an IIS web server which may handle multiple requests at the same time. The 
rows are unique in their own right but the timestamp is the same in the 
logging. 



2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7


2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6


2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6

2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5


2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6

2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6



This particular log file only records in seconds. So what I am doing with these 
rows at the moment is to add an artitifical millisecond to enforce uniqueness.




2024-05-20 00:00:14.000 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7 

2024-05-20 00:00:14.001 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6 

2024-05-20 00:00:14.002 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6

2024-05-20 00:00:14.003 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5 

2024-05-20 00:00:15.000 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6

2024-05-20 00:00:15.001 W3SVC1 openzweb01 192.168.3.69 POST 
/portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6



For some other log files that I am processing they are in milliseconds already 
but there is a (small) chance of dataloss if multiple requests happen to be 
processed at the exact same time.



I have been thinking about this some more and I think that rather than break 
the IoTDB CRUD model I should handle this on the client side. In my use case 
the log data is actually staged in an H2 database before it is sent to IoTDB so 
I can enforce PK validation there. That way it is less expensive that checking 
the timestamp in IoTDB for each record.



Thanks 

Trevor Hart








 On Fri, 17 May 2024 19:11:13 +1200 Jialin Qiao  
wrote ---



Hi Trevor, 
 
Will different values of the same timestamp be the same？ 
 
1. Same 
Time, Value 
1, 1 
1, 1 
1, 1 
 
2. Different 
Time, Value 
1, 1 
1, 2 
1, 1 
 
 
Jialin Qiao 
 
Trevor Hart  于2024年5月14日周二 11:20写道： 
> 
> Thank you! I will implment some work around for now. 
> 
> 
> I would appreciate some consideration for this option in the future. 
> 
> 
> Thanks 
> 
> Trevor Hart 
> 
> Ope Limited 
> 
> w: http://www.ope.nz/ 
> 
> m: +64212728039 
> 
> 
> 
> 
> 
> 
> 
> 
>  On Tue, 14 May 2024 15:17:47 +1200 Xiangdong Huang 
>  wrote --- 
> 
> 
> 
> > 1. Checking before insert if the timestamp already exists and remedy on the 
> > client before resend 
> > 2. Moving to Nanosecond and introducing some insignificant time value to 
> > keep timestamp values unique. 
> Yes these maybe the best solutions for a specific application. 
> 
> 
> Analysis for IoTDB: 
> - Rejecting the write when receiving an existing timestamp in IoTDB is 
> time-costly (IoTDB needs to check historical data). I think we will do 
> not check it until we find a low-latency method. 
> - Allowing multiple value versions for a timestamp may introduce a 
> chain reaction and there may be a lot of codes that should be 
> modified, which is a huge work. 
> 
> There is a new idea (but I have no time to implement it...) 
> - Add a parameter in IoTDB: replace_strategy: first, last, avg etc... 
> - when an existing timestamp arrives, IoTDB accepts it 
> - when IoTDB runs LSM to merge data and meets multiple values for a 
> timestamp, then handles it according to the replace_startegy. 
> 
> The solution may also introduce some work to do... and we need to 
> think carefully the impact to the query process. 
> Need to survey whether this is a common requirement. 
> 
> Best, 
> --- 
> Xiangdong Huang 
> 
> Trevor Hart  于2024年5月14日周二 09:55写道： 
> > 
> > Hello Yuan 
> > 
> > 
> > 
> > Correct, the first timestamp and values should be retained. 
> > 
> > 
> > 
> > I realise this is does not align with the current design. I was just asking 
> > whether there was an existing option to operate to block duplicates. 
> > 
> > 
> > 
> > In a normal RDBMS if you try to insert with a duplicate the insert will 
> > fail with a PK violation. It would be great in some 

Re: AW: Possible fix for randomly failing Jenkins builds

[Haonan Hou]

Approved and merged. 
Thanks.

Haonan

On 2024/05/17 10:14:10 Christofer Dutz wrote:
> Done:
> https://github.com/apache/iotdb/pull/12542
> Waiting for approval.
> 
> Chris
> 
> Von: Jialin Qiao 
> Datum: Freitag, 17. Mai 2024 um 09:15
> An: dev@iotdb.apache.org 
> Betreff: Re: Possible fix for randomly failing Jenkins builds
> +1 for importing the solutions in PLC4X
> 
> 
> Jialin Qiao
> 
> Christofer Dutz  于2024年5月14日周二 23:32写道：
> >
> > Hi all,
> >
> > I know that in IoTDB as well as PLC4X we were having issues of regularly 
> > but randomly failing Jenkins builds.
> >
> > Usually, these problems were because of timeouts when deploying to Nexus.
> >
> > In PLC4X I have now tried something that seems to be doing the job … 
> > possibly this could also help us here (I mean … the biggest part of the 
> > Jenkins setup is borrowed from PLC4X anyway ;-) )
> >
> > My solution was to wrap the execution of the deploy step in a loop on bash 
> > level:
> >
> > https://github.com/apache/plc4x/blob/develop/Jenkinsfile#L167
> >
> > sh 'until ./mvnw -f jenkins.pom -X -P deploy-snapshots 
> > wagon:upload || (( count++ >= 5 )); do echo "Retrying to deploy"; done'
> >
> > So, it would try up to 5 times, if deployment fails.
> >
> > I think in case of deployment the full build is done and was successful. 
> > The only reason for failures in this step usually is problems with Nexus 
> > and a stupid retry should help.
> >
> > What do you think? Should we try the same change?
> >
> >
> > Chris
> 

Re: [DISCUSS] Drop Java 8?

[Jialin Qiao]

Hi,

Drop java 8 will bring some CVE fixed, but also reduce a common option
for users,  for this, I will -0.

Maintaining two branch will surely resolve the issue, but this will
bring much additional work... For this, I will -1

Although we put 0.13 and 1.x version in our download page, we actually
do not maintain the 0.x branch and there is on plan to release a new
0.x version.

So, as there is a lot to do with 1.x this year (the table schema, data
subscription, tsfile consensus), I prefer not to do the drop this
year.

Jialin Qiao

Christofer Dutz  于2024年5月17日周五 14:43写道：
>
> So we have real numbers?
>
> I also haven't used an oracle jre for quite some time for these reasons.
>
> Would a double branch strategy work? That the project maintains a 1.x branch 
> with Javax and Java 8 and a 2.0 branch with Jakarta and Java 11?
>
> Sort of like we had the 0.x and 1.x branches around?
>
> Chris
>
>
> Gesendet von Outlook für Android
> 
> From: Trevor Hart 
> Sent: Friday, May 17, 2024 4:01:14 AM
> To: dev 
> Subject: Re: [DISCUSS] Drop Java 8?
>
> I think a lot of organisations are sticking to Java 8 because of the change 
> to Oracle license that was introduced in Java 11.
>
>
>
> If you use Oracle 11 JRE you need to pay Oracle for a license.
>
>
>
> This was why Open JDK came about.
>
>
>
> Personally I use IotDB with Open JDK 11 (Eclipse Temurin) which does not 
> require a license.
>
>
>
> Thanks
>
> Trevor Hart
>
>
>
>
>
>
>
>
>  On Fri, 17 May 2024 13:30:15 +1200 Yuan Tian  
> wrote ---
>
>
>
> Hi Chris,
>
> It seems that a lot of people still use jdk1.8 in their product environment.
>
> Best regards,
> -
> Yuan Tian
>
> On Thu, May 16, 2024 at 8:10 PM Christofer Dutz 
> 
> wrote:
>
> > Hi all,
> >
> > starting this new thread as I am not sure if others are reading the
> > Jakarta migration thread.
> >
> > I would like to propose planning on dropping Java 8 support.
> >
> > I wouldn’t immediately do that, and I would also propose to do a major
> > version update (Switching to 2.0.0)
> >
> > We could still maintain a 1.x branch for those people not able to update.
> >
> > The main reason is that we are currently blocking ourselves from updating
> > many major plugins and dependencies.
> > I noticed that when updating to the Jakarta namespace. Here there is no
> > Netty version available that supports Jakarta and supports Java 8.
> >
> > Other libraries where we are not able to update without giving up on Java
> > 8:
> >
> >   *   Airlift-Units (Stuck at 1.7 current 1.10)
> >   *   Airlift
> >   *   Antlr (Stuck at 4.9.3 current 4.13.1)
> >   *   Caffeine (Stuck at 2.9.3 current 3.1.8)
> >   *   Logback (Stuck at 1.3.14 current 1.5.6)
> >   *   Mockito (Stuck at 2.23.4 current 5.12.0)
> >   *   Thrift (Stuck at 0.17.0 current 0.20.0)
> >
> >
> >
> >   *   Spotless Plugin (We’ve got a workaround for Java 8)
> >
> >
> > In my branch where I refactored the javax namespace to Jakarta after
> > updating dependencies I was able to remove all exclusions of the
> > BanVulnerableDependencies check.
> >
> > Also does dropping Java 8 and the Jakarta migration allow embedding IoTDB
> > in recent Spring versions.
> >
> >
> > So … what do you think?
> >
> >
> > Chris
> >

AW: Possible fix for randomly failing Jenkins builds

[Christofer Dutz]

Done:
https://github.com/apache/iotdb/pull/12542
Waiting for approval.

Chris

Von: Jialin Qiao 
Datum: Freitag, 17. Mai 2024 um 09:15
An: dev@iotdb.apache.org 
Betreff: Re: Possible fix for randomly failing Jenkins builds
+1 for importing the solutions in PLC4X


Jialin Qiao

Christofer Dutz  于2024年5月14日周二 23:32写道：
>
> Hi all,
>
> I know that in IoTDB as well as PLC4X we were having issues of regularly but 
> randomly failing Jenkins builds.
>
> Usually, these problems were because of timeouts when deploying to Nexus.
>
> In PLC4X I have now tried something that seems to be doing the job … possibly 
> this could also help us here (I mean … the biggest part of the Jenkins setup 
> is borrowed from PLC4X anyway ;-) )
>
> My solution was to wrap the execution of the deploy step in a loop on bash 
> level:
>
> https://github.com/apache/plc4x/blob/develop/Jenkinsfile#L167
>
> sh 'until ./mvnw -f jenkins.pom -X -P deploy-snapshots 
> wagon:upload || (( count++ >= 5 )); do echo "Retrying to deploy"; done'
>
> So, it would try up to 5 times, if deployment fails.
>
> I think in case of deployment the full build is done and was successful. The 
> only reason for failures in this step usually is problems with Nexus and a 
> stupid retry should help.
>
> What do you think? Should we try the same change?
>
>
> Chris

Re: Possible fix for randomly failing Jenkins builds

[Jialin Qiao]

+1 for importing the solutions in PLC4X


Jialin Qiao

Christofer Dutz  于2024年5月14日周二 23:32写道：
>
> Hi all,
>
> I know that in IoTDB as well as PLC4X we were having issues of regularly but 
> randomly failing Jenkins builds.
>
> Usually, these problems were because of timeouts when deploying to Nexus.
>
> In PLC4X I have now tried something that seems to be doing the job … possibly 
> this could also help us here (I mean … the biggest part of the Jenkins setup 
> is borrowed from PLC4X anyway ;-) )
>
> My solution was to wrap the execution of the deploy step in a loop on bash 
> level:
>
> https://github.com/apache/plc4x/blob/develop/Jenkinsfile#L167
>
> sh 'until ./mvnw -f jenkins.pom -X -P deploy-snapshots 
> wagon:upload || (( count++ >= 5 )); do echo "Retrying to deploy"; done'
>
> So, it would try up to 5 times, if deployment fails.
>
> I think in case of deployment the full build is done and was successful. The 
> only reason for failures in this step usually is problems with Nexus and a 
> stupid retry should help.
>
> What do you think? Should we try the same change?
>
>
> Chris

Re: Handling Duplicate Timestamps

[Jialin Qiao]

Hi Trevor,

Will different values of the same timestamp be the same？

1. Same
Time, Value
1, 1
1, 1
1, 1

2. Different
Time, Value
1, 1
1, 2
1, 1


Jialin Qiao

Trevor Hart  于2024年5月14日周二 11:20写道：
>
> Thank you! I will implment some work around for now.
>
>
> I would appreciate some consideration for this option in the future.
>
>
> Thanks
>
> Trevor Hart
>
> Ope Limited
>
> w: http://www.ope.nz/
>
> m: +64212728039
>
>
>
>
>
>
>
>
>  On Tue, 14 May 2024 15:17:47 +1200 Xiangdong Huang  
> wrote ---
>
>
>
> > 1. Checking before insert if the timestamp already exists and remedy on the 
> > client before resend
> > 2. Moving to Nanosecond and introducing some insignificant time value to 
> > keep timestamp values unique.
> Yes these maybe the best solutions for a specific application.
>
>
> Analysis for IoTDB:
> - Rejecting the write when receiving an existing timestamp in IoTDB is
> time-costly (IoTDB needs to check historical data). I think we will do
> not check it until we find a low-latency method.
> - Allowing multiple value versions for a timestamp may introduce a
> chain reaction and there may be a lot of codes that should be
> modified, which is a huge work.
>
> There is a new idea (but I have no time to implement it...)
> - Add a parameter in IoTDB: replace_strategy: first, last, avg etc...
> - when an existing timestamp arrives, IoTDB accepts it
> - when IoTDB runs LSM to merge data and meets multiple values for a
> timestamp, then handles it according to the replace_startegy.
>
> The solution may also introduce some work to do... and we need to
> think carefully the impact to the query process.
> Need to survey whether this is a common requirement.
>
> Best,
> ---
> Xiangdong Huang
>
> Trevor Hart  于2024年5月14日周二 09:55写道：
> >
> > Hello Yuan
> >
> >
> >
> > Correct, the first timestamp and values should be retained.
> >
> >
> >
> > I realise this is does not align with the current design. I was just asking 
> > whether there was an existing option to operate to block duplicates.
> >
> >
> >
> > In a normal RDBMS if you try to insert with a duplicate the insert will 
> > fail with a PK violation. It would be great in some circumstances if IotDB 
> > at least had the option to fail this way.
> >
> >
> >
> > I am considering some options such as;
> >
> >
> >
> > 1. Checking before insert if the timestamp already exists and remedy on the 
> > client before resend
> >
> > 2. Moving to Nanosecond and introducing some insignificant time value to 
> > keep timestamp values unique.
> >
> >
> >
> > I have already done something similar to #2 with storing IIS web log files 
> > as they are recorded in seconds and not milliseconds.
> >
> >
> >
> > Thanks
> >
> > Trevor Hart
> >
> >
> >
> >
> >  On Tue, 14 May 2024 13:29:02 +1200 Yuan Tian 
> >  wrote ---
> >
> >
> >
> > Hi Trevor,
> >
> > By "rejects duplicates", you mean you want to keep the first duplicate
> > timestamp and its corresponding values?(because the following duplicated
> > ones will be rejected)
> >
> > Best regards,
> > 
> > Yuan Tian
> >
> > On Mon, May 13, 2024 at 6:24 PM Trevor Hart  
> > wrote:
> >
> > >
> > >
> > >
> > >
> > > Correct. I’m not disputing that. What I’m asking is that it
> > > would be good to have a configuration that either allows overwrites or
> > > rejects duplicates.My scenario is request log data from a server (the
> > > device). As it may be processing multiple requests at once there is a
> > > chance that there could be colliding time stamps.As it stands now I would
> > > need to check if the timestamp exists before inserting the data. Which
> > > obviously affects throughput. Thanks Trevor Hart On Fri, 10 May
> > > 2024 00:33:40 +1200  Jialin Qiao 
> > > wrote  Hi,
> > > In IoT or IIoT scenarios, we thought each data point represent a metric of
> > > a timestamp.In which case you need to store duplicated values?  Take this
> > > for an example: Time, root.sg1.car1.speed 1, 1 1, 2  Could a car has
> > > different speed at time 1?   Jialin Qiao  Yuan Tian <
> > > mailto:mailto:jackietie...@gmail.com> 于2024年5月9日周四 18:51写道： > > Hi 
> > > Trevor, > > Now we
> > > will override the duplicate timestamp with a newer one. There is > nothing
> > > we can do about it now. > > Best regards, > --- > Yuan 
> > > Tian
> > > > > On Wed, May 8, 2024 at 5:31 PM Trevor Hart 
> > > > >  wrote: > >
> > > > Hello > > > > > > > > I’m aware that when inserting a duplicate 
> > > > timestamp
> > > the values will be > > overwritten. This will obviously result in data
> > > loss. > > > > > > > > Is there a config/setting to reject or throw an 
> > > error
> > > on duplicate > > inserts? Although highly unlikely I would prefer to be
> > > alerted to the > > situation rather than lose data. > > > > > 

Re: [DISCUSS] Drop Java 8?

[Christofer Dutz]

So we have real numbers?

I also haven't used an oracle jre for quite some time for these reasons.

Would a double branch strategy work? That the project maintains a 1.x branch 
with Javax and Java 8 and a 2.0 branch with Jakarta and Java 11?

Sort of like we had the 0.x and 1.x branches around?

Chris


Gesendet von Outlook für Android

From: Trevor Hart 
Sent: Friday, May 17, 2024 4:01:14 AM
To: dev 
Subject: Re: [DISCUSS] Drop Java 8?

I think a lot of organisations are sticking to Java 8 because of the change to 
Oracle license that was introduced in Java 11.



If you use Oracle 11 JRE you need to pay Oracle for a license.



This was why Open JDK came about.



Personally I use IotDB with Open JDK 11 (Eclipse Temurin) which does not 
require a license.



Thanks

Trevor Hart








 On Fri, 17 May 2024 13:30:15 +1200 Yuan Tian  
wrote ---



Hi Chris,

It seems that a lot of people still use jdk1.8 in their product environment.

Best regards,
-
Yuan Tian

On Thu, May 16, 2024 at 8:10 PM Christofer Dutz 

wrote:

> Hi all,
>
> starting this new thread as I am not sure if others are reading the
> Jakarta migration thread.
>
> I would like to propose planning on dropping Java 8 support.
>
> I wouldn’t immediately do that, and I would also propose to do a major
> version update (Switching to 2.0.0)
>
> We could still maintain a 1.x branch for those people not able to update.
>
> The main reason is that we are currently blocking ourselves from updating
> many major plugins and dependencies.
> I noticed that when updating to the Jakarta namespace. Here there is no
> Netty version available that supports Jakarta and supports Java 8.
>
> Other libraries where we are not able to update without giving up on Java
> 8:
>
>   *   Airlift-Units (Stuck at 1.7 current 1.10)
>   *   Airlift
>   *   Antlr (Stuck at 4.9.3 current 4.13.1)
>   *   Caffeine (Stuck at 2.9.3 current 3.1.8)
>   *   Logback (Stuck at 1.3.14 current 1.5.6)
>   *   Mockito (Stuck at 2.23.4 current 5.12.0)
>   *   Thrift (Stuck at 0.17.0 current 0.20.0)
>
>
>
>   *   Spotless Plugin (We’ve got a workaround for Java 8)
>
>
> In my branch where I refactored the javax namespace to Jakarta after
> updating dependencies I was able to remove all exclusions of the
> BanVulnerableDependencies check.
>
> Also does dropping Java 8 and the Jakarta migration allow embedding IoTDB
> in recent Spring versions.
>
>
> So … what do you think?
>
>
> Chris
>

Re: [DISCUSS] Drop Java 8?

[Trevor Hart]

I think a lot of organisations are sticking to Java 8 because of the change to 
Oracle license that was introduced in Java 11.



If you use Oracle 11 JRE you need to pay Oracle for a license.



This was why Open JDK came about.



Personally I use IotDB with Open JDK 11 (Eclipse Temurin) which does not 
require a license.



Thanks 

Trevor Hart








 On Fri, 17 May 2024 13:30:15 +1200 Yuan Tian  
wrote ---



Hi Chris, 
 
It seems that a lot of people still use jdk1.8 in their product environment. 
 
Best regards, 
- 
Yuan Tian 
 
On Thu, May 16, 2024 at 8:10 PM Christofer Dutz 
 
wrote: 
 
> Hi all, 
> 
> starting this new thread as I am not sure if others are reading the 
> Jakarta migration thread. 
> 
> I would like to propose planning on dropping Java 8 support. 
> 
> I wouldn’t immediately do that, and I would also propose to do a major 
> version update (Switching to 2.0.0) 
> 
> We could still maintain a 1.x branch for those people not able to update. 
> 
> The main reason is that we are currently blocking ourselves from updating 
> many major plugins and dependencies. 
> I noticed that when updating to the Jakarta namespace. Here there is no 
> Netty version available that supports Jakarta and supports Java 8. 
> 
> Other libraries where we are not able to update without giving up on Java 
> 8: 
> 
>   *   Airlift-Units (Stuck at 1.7 current 1.10) 
>   *   Airlift 
>   *   Antlr (Stuck at 4.9.3 current 4.13.1) 
>   *   Caffeine (Stuck at 2.9.3 current 3.1.8) 
>   *   Logback (Stuck at 1.3.14 current 1.5.6) 
>   *   Mockito (Stuck at 2.23.4 current 5.12.0) 
>   *   Thrift (Stuck at 0.17.0 current 0.20.0) 
> 
> 
> 
>   *   Spotless Plugin (We’ve got a workaround for Java 8) 
> 
> 
> In my branch where I refactored the javax namespace to Jakarta after 
> updating dependencies I was able to remove all exclusions of the 
> BanVulnerableDependencies check. 
> 
> Also does dropping Java 8 and the Jakarta migration allow embedding IoTDB 
> in recent Spring versions. 
> 
> 
> So … what do you think? 
> 
> 
> Chris 
>

Re: [DISCUSS] Drop Java 8?

[Yuan Tian]

Hi Chris,

It seems that a lot of people still use jdk1.8 in their product environment.

Best regards,
-
Yuan Tian

On Thu, May 16, 2024 at 8:10 PM Christofer Dutz 
wrote:

> Hi all,
>
> starting this new thread as I am not sure if others are reading the
> Jakarta migration thread.
>
> I would like to propose planning on dropping Java 8 support.
>
> I wouldn’t immediately do that, and I would also propose to do a major
> version update (Switching to 2.0.0)
>
> We could still maintain a 1.x branch for those people not able to update.
>
> The main reason is that we are currently blocking ourselves from updating
> many major plugins and dependencies.
> I noticed that when updating to the Jakarta namespace. Here there is no
> Netty version available that supports Jakarta and supports Java 8.
>
> Other libraries where we are not able to update without giving up on Java
> 8:
>
>   *   Airlift-Units (Stuck at 1.7 current 1.10)
>   *   Airlift
>   *   Antlr (Stuck at 4.9.3 current 4.13.1)
>   *   Caffeine (Stuck at 2.9.3 current 3.1.8)
>   *   Logback (Stuck at 1.3.14 current 1.5.6)
>   *   Mockito (Stuck at 2.23.4 current 5.12.0)
>   *   Thrift (Stuck at 0.17.0 current 0.20.0)
>
>
>
>   *   Spotless Plugin (We’ve got a workaround for Java 8)
>
>
> In my branch where I refactored the javax namespace to Jakarta after
> updating dependencies I was able to remove all exclusions of the
> BanVulnerableDependencies check.
>
> Also does dropping Java 8 and the Jakarta migration allow embedding IoTDB
> in recent Spring versions.
>
>
> So … what do you think?
>
>
> Chris
>

[DISCUSS] Drop Java 8?

[Christofer Dutz]

Hi all,

starting this new thread as I am not sure if others are reading the Jakarta 
migration thread.

I would like to propose planning on dropping Java 8 support.

I wouldn’t immediately do that, and I would also propose to do a major version 
update (Switching to 2.0.0)

We could still maintain a 1.x branch for those people not able to update.

The main reason is that we are currently blocking ourselves from updating many 
major plugins and dependencies.
I noticed that when updating to the Jakarta namespace. Here there is no Netty 
version available that supports Jakarta and supports Java 8.

Other libraries where we are not able to update without giving up on Java 8:

  *   Airlift-Units (Stuck at 1.7 current 1.10)
  *   Airlift
  *   Antlr (Stuck at 4.9.3 current 4.13.1)
  *   Caffeine (Stuck at 2.9.3 current 3.1.8)
  *   Logback (Stuck at 1.3.14 current 1.5.6)
  *   Mockito (Stuck at 2.23.4 current 5.12.0)
  *   Thrift (Stuck at 0.17.0 current 0.20.0)



  *   Spotless Plugin (We’ve got a workaround for Java 8)


In my branch where I refactored the javax namespace to Jakarta after updating 
dependencies I was able to remove all exclusions of the 
BanVulnerableDependencies check.

Also does dropping Java 8 and the Jakarta migration allow embedding IoTDB in 
recent Spring versions.


So … what do you think?


Chris

AW: Jakarta migration

[Christofer Dutz]

Hi Wang,

I know that there is no schedule for that, but I would like to bring up the 
need to discuss this.

Right now, we are stuck with updating 10 or more dependencies to more recent 
ones and I think I have 2-3 CVE exclusions in the build that manually allow the 
build to succeed, even if there are vulnerabilities known for some of our 
libraries as the issues could not be resolved because of us requiring Java 8 
compatibility.

I think it would be beneficial for the project to discuss giving up on Java 8 
support.

One possible option would be to continue a 1.3.x or even better the 1.x branch 
which stays compatible with Java 8 and then to create a 2.0.0 branch which 
baselines on Java 11.

Chris


Von: Wang Critas 
Datum: Mittwoch, 15. Mai 2024 um 10:17
An: dev@iotdb.apache.org 
Betreff: 答复: Jakarta migration
However, it seems that we have not abandoned Java 8 plan

发件人: Christofer Dutz 
日期: 星期二, 2024年5月14日 23:26
收件人: dev@iotdb.apache.org 
主题: AW: Jakarta migration
Most of the build is now working correctly however there are still two 
Integration tests that are failing.
At first I thought they were due to issues with the server, but I am actually 
able to start my Jakarta version of IoTDB and connect to it with the JDBC 
driver without any issues. Also don’t I see any issues in the logs when 
starting a distribution.

I assume the remaining issues might be on the client side of the tests and will 
continue with my work as soon as I’m done with this month’s board report 
reviews.

Chris


Von: Christofer Dutz 
Datum: Montag, 13. Mai 2024 um 17:41
An: dev@iotdb.apache.org 
Betreff: Jakarta migration
Hi all,

I am currently working on the Jakarta migration … initially I thought of this 
as an experiment in order to find out what the implications would be.
Turns out my gut-feeling was right that this would not be a simple change. Less 
for the complexity of the changes, but what they would bring to the rest of the 
build.

So, for now it seems as if the only Netty version compatible with the Jakarta 
namespace would be Netty 11 (Technically they claimed 10 would be compatible, 
but that’ just not true).
Unfortunately, Netty 9 was the last version to work with Java 8 … so it seems 
as if going to the Jakarta namespace would stop us from building on Java 8.

I know I knew why I would have preferred this migration for a bigger release, 
but I guess a 1.4.0 or 2.0.0 might be more appropriate for a change like this.

Chris

答复: Jakarta migration

[Wang Critas]

However, it seems that we have not abandoned Java 8 plan

发件人: Christofer Dutz 
日期: 星期二, 2024年5月14日 23:26
收件人: dev@iotdb.apache.org 
主题: AW: Jakarta migration
Most of the build is now working correctly however there are still two 
Integration tests that are failing.
At first I thought they were due to issues with the server, but I am actually 
able to start my Jakarta version of IoTDB and connect to it with the JDBC 
driver without any issues. Also don’t I see any issues in the logs when 
starting a distribution.

I assume the remaining issues might be on the client side of the tests and will 
continue with my work as soon as I’m done with this month’s board report 
reviews.

Chris


Von: Christofer Dutz 
Datum: Montag, 13. Mai 2024 um 17:41
An: dev@iotdb.apache.org 
Betreff: Jakarta migration
Hi all,

I am currently working on the Jakarta migration … initially I thought of this 
as an experiment in order to find out what the implications would be.
Turns out my gut-feeling was right that this would not be a simple change. Less 
for the complexity of the changes, but what they would bring to the rest of the 
build.

So, for now it seems as if the only Netty version compatible with the Jakarta 
namespace would be Netty 11 (Technically they claimed 10 would be compatible, 
but that’ just not true).
Unfortunately, Netty 9 was the last version to work with Java 8 … so it seems 
as if going to the Jakarta namespace would stop us from building on Java 8.

I know I knew why I would have preferred this migration for a bigger release, 
but I guess a 1.4.0 or 2.0.0 might be more appropriate for a change like this.

Chris

Possible fix for randomly failing Jenkins builds

[Christofer Dutz]

Hi all,

I know that in IoTDB as well as PLC4X we were having issues of regularly but 
randomly failing Jenkins builds.

Usually, these problems were because of timeouts when deploying to Nexus.

In PLC4X I have now tried something that seems to be doing the job … possibly 
this could also help us here (I mean … the biggest part of the Jenkins setup is 
borrowed from PLC4X anyway ;-) )

My solution was to wrap the execution of the deploy step in a loop on bash 
level:

https://github.com/apache/plc4x/blob/develop/Jenkinsfile#L167

sh 'until ./mvnw -f jenkins.pom -X -P deploy-snapshots 
wagon:upload || (( count++ >= 5 )); do echo "Retrying to deploy"; done'

So, it would try up to 5 times, if deployment fails.

I think in case of deployment the full build is done and was successful. The 
only reason for failures in this step usually is problems with Nexus and a 
stupid retry should help.

What do you think? Should we try the same change?


Chris