AW: Session API issues

[Christofer Dutz]

I guess the simplest solution would simply be to return an error if not all 
paths in the query are valid. Right now I added a check on the client side that 
throws an exception if the size of the paths is different from that of the 
fields.

Chris



Von: Christofer Dutz 
Datum: Sonntag, 9. Juni 2024 um 16:40
An: dev@iotdb.apache.org 
Betreff: Session API issues
Hi all,

I have been playing around with the session API and have been stumbling into 
some pits, that I think would be worth addressing.

The biggest issue I have had recently is using the executeAggregationQuery 
method, passing in pairs of queries and aggregations.
However, I noticed that if for example I pass in 3 queries with the middle one 
being invalid, that I simply get back a 2-field response and I have no way of 
seeing which field was invalid.

In general, I think the passing in multiple lists with each the same size sort 
of feels a bit odd. Instead of passing in one list of objects which again have 
multiple options would be a preferrable approach.
Also, would this allow us to add these as references to the responses. So 
having a Field returned by RowRecord.getFields() have a reference to the query 
object, would make the API better to handle.
We wouldn’t have to pass the fields when returning the result, this could be 
something the client does completely in the client to avoid unnecessary 
payload. I do think however for this to work, that we would need some sort of 
error handling, passing back something like an ErrorField, if a query was wrong 
or anything similar.

Chris

Session API issues

[Christofer Dutz]

Hi all,

I have been playing around with the session API and have been stumbling into 
some pits, that I think would be worth addressing.

The biggest issue I have had recently is using the executeAggregationQuery 
method, passing in pairs of queries and aggregations.
However, I noticed that if for example I pass in 3 queries with the middle one 
being invalid, that I simply get back a 2-field response and I have no way of 
seeing which field was invalid.

In general, I think the passing in multiple lists with each the same size sort 
of feels a bit odd. Instead of passing in one list of objects which again have 
multiple options would be a preferrable approach.
Also, would this allow us to add these as references to the responses. So 
having a Field returned by RowRecord.getFields() have a reference to the query 
object, would make the API better to handle.
We wouldn’t have to pass the fields when returning the result, this could be 
something the client does completely in the client to avoid unnecessary 
payload. I do think however for this to work, that we would need some sort of 
error handling, passing back something like an ErrorField, if a query was wrong 
or anything similar.

Chris

AW: version about odbc

[Christofer Dutz]

The ODBC is not really a “compilable” example.
It contains a description how you can use a commercial product to wrap our JDBC 
driver in order to produce an ODBC driver.
Also does it contain a compilable C# project file, that demonstates the usage 
of the ODBC API.

However even if the example would be compiled as part of the build, you would 
still need to manually prepare the ODBC wrapper.

I would consider this module more documentation than “compilable code”.

Chris


Von: Yuan Tian 
Datum: Freitag, 7. Juni 2024 um 12:14
An: dev 
Betreff: version about odbc
Hi all,

I found that in master branch, we've already upgraded the version to 
1.3.3-SNAPSHOT, but the examples about odbc module is missed?

[cid:ii_lx4j3xy50]

Best regards,
-
Yuan Tian

AW: [DISCUSS] GH Actions optimizations

[Christofer Dutz]

Also is the settings currently done that it only skips any tests in non-master 
branches … so if there’s a PR on a branch, that will be optimized, but every 
commit going back to master should have all tests run.

Chris


Von: Christofer Dutz 
Datum: Mittwoch, 5. Juni 2024 um 14:22
An: dev@iotdb.apache.org 
Betreff: AW: [DISCUSS] GH Actions optimizations
Well even with the re-run option still failures get counted and a dashboard 
like this:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
Is a lot better than having to continuously go back to your PRs and click on 
“re-run failed tests”

Chris

Von: Yuan Tian 
Datum: Mittwoch, 5. Juni 2024 um 03:06
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] GH Actions optimizations
Hi Chris,

That's really good news. About the `surefire` part, I think we need to
discuss more about it because some of the previously failed tests have
indeed helped us discover concurrency issues in IoTDB. I am concerned that
doing this might reduce the likelihood of exposing such bugs.


Best regards,

Yuan Tian

AW: [DISCUSS] GH Actions optimizations

[Christofer Dutz]

Well even with the re-run option still failures get counted and a dashboard 
like this:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY
Is a lot better than having to continuously go back to your PRs and click on 
“re-run failed tests”

Chris

Von: Yuan Tian 
Datum: Mittwoch, 5. Juni 2024 um 03:06
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] GH Actions optimizations
Hi Chris,

That's really good news. About the `surefire` part, I think we need to
discuss more about it because some of the previously failed tests have
indeed helped us discover concurrency issues in IoTDB. I am concerned that
doing this might reduce the likelihood of exposing such bugs.


Best regards,

Yuan Tian

[DISCUSS] GH Actions optimizations

[Christofer Dutz]

At today’s sessions at community over code EU in a session about “Apache build 
observability” we got a little spotlight on IoTDB.
The develocity folks pointed out Apache IoTDB as being the one Apache project 
that provides the biggest room for improvement by enabling some of their 
build-optimization tools.

I have created a PR : https://github.com/apache/iotdb/pull/12654

This enables predictive test-selection which should prioritize tests that are 
most probable to be affected by changes and to skip ones that are more 
definitely not going to be affected.

I was actively approached by the develocity folks here at the conference and 
together we created a PR that should enable this for us.
Also did I take the liberty to also enable surefire to automatically re-run 
failed tests for a maximum of 3 times before actually failing a test.

These changes should allow reducing the load we have on GH Actions by 40% 
(according to the dashboards develocity has)

Also did they recommend utilizing build cache extensions, which should even 
more greatly reduce the build time and the load on the GH runners.

If anyone wants to play around with the dashboards for IoTDB …. Please have a 
look here.

https://ge.apache.org/scans?search.rootProjectNames=*IoTDB*

There are very interesting dashboards that show us stuff like flaky tests:
https://ge.apache.org/scans/tests?search.rootProjectNames=*IoTDB*=Europe%2FBratislava=FLAKY

Chris

AW: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Ok … artifacts should be available …

Chris

Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 11:26
An: dev@iotdb.apache.org 
Betreff: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache 
Thrift 0.20.0)
So, the vote passes with 7 +1 votes at least 5 I could verify as binding votes.

In the future, it would be great if people would add their name/apacheId etc. 
to the votes to make it easier to verify and count for the RM.

Releasing the maven artifacts right away.

Thanks for all those who voted.

Chris


Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 09:37
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)
Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

[RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

So, the vote passes with 7 +1 votes at least 5 I could verify as binding votes.

In the future, it would be great if people would add their name/apacheId etc. 
to the votes to make it easier to verify and count for the RM.

Releasing the maven artifacts right away.

Thanks for all those who voted.

Chris


Von: Christofer Dutz 
Datum: Dienstag, 4. Juni 2024 um 09:37
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)
Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Yeah.. Currently at community over code... Was planning on doing this in a 
break today.

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Haonan Hou 
Sent: Monday, June 3, 2024 4:11:04 AM
To: dev@iotdb.apache.org 
Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 
0.20.0)

Hi Chris,

I think 72 hours has passed. These artifacts can be released now?

Thanks,
Haonan

On 2024/05/28 14:40:39 Christofer Dutz wrote:
> Hi all,
>
> I after fixing some issues in the cmake-maven-plugin, have I updated the 
> iotdb-bin-resources module to use these.
> Now we can stage Apache Thrift binaries for all platforms.
>
> In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), 
> I also updated the Thrift version and staged a release of Thrift binaries for 
> all of our supported platforms in Nexus.
>
> I’ve staged the artifacts at:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158
>
> Tag Name: iotdb-tools-thrift-v0.20.0.0
> https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0
>
> The source bundle generally consists of a pom.xml and an assembly xml … so it 
> should be quick to review.
> https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip
>
> I’m intentionally not staging it on the release-svn, as nobody should be 
> required to consume this module but us ourselves.
>
> The vote stays open for 72hours … assuming enough support, I’ll then finish 
> changing the main repo to use these artifacts.
>
>
> Chris
>

AW: [Proposal] Merge configuration files to iotdb-system.properties

[Christofer Dutz]

However,

I think it would be good If the code for accessing the options would not be 
merged.
Because that would move the project to being more of a mono-module type of 
project.

Chris

Von: Jialin Qiao 
Datum: Samstag, 1. Juni 2024 um 08:07
An: dev@iotdb.apache.org 
Betreff: Re: [Proposal] Merge configuration files to iotdb-system.properties
Hi,

Good feature!  This will make it easy for users to set configs.

Jialin Qiao

wenwei shu  于2024年5月31日周五 21:23写道：
>
>  Hello everyone,
>
> I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are
> working on merging `iotdb-confignode.properties`,
> `iotdb-datanode.properties` and `iotdb-common.properties` into a new
> `iotdb-system.properties` file. For old version users who upgrade to a new
> version, if they don't create the new configuration file themselves during
> the upgrade, IoTDB will generate the new configuration file based on
> several old configuration files after it is started. You can check this PR
> to get more details: https://github.com/apache/iotdb/pull/12570.
> Thank you for your reading.
>
> Best regards,
> Wenwei Shu

AW: [Proposal] Data Subscription Client on IoTDB

[Christofer Dutz]

Hi Yuchen,

do I understand it correctly, that with this, a client could subscribe to data 
and receive this in TsFile format? This plus the ability of an IoTDB instance 
subscribing to such a TsFile subscription would perfectly align with my plans 
for 2024 allowing TsFile libraries on PLCs sending data to IoTDB as well as 
simplify data-collection gateways.

Chris

Von: Yuchen Ding 
Datum: Freitag, 31. Mai 2024 um 10:14
An: dev@iotdb.apache.org 
Betreff: Re: [Proposal] Data Subscription Client on IoTDB
Hello everyone,

I am VGalaxies. Recently, I have been working on providing TsFile subscription 
support for IoTDB data subscription client. The background of this feature is 
to achieve data file export and backup for multi-replica clusters using data 
subscription. Using the existing data subscription client, the subscribed data 
is in the form of SessionDataSet. The server needs to parse TsFile, and the 
client needs to rewrite TsFile. Raw data is transmitted over the network, 
without leveraging the high compression features of TsFile. Therefore, we hope 
to support exporting TsFile using the TsFile client SDK in data subscriptions.

In terms of functionality, TsFile subscription support for the data 
subscription client includes three steps. First, create a topic with a data 
presentation format of TsFile, by specifying the topic format as TsFileHandler. 
Second, when creating a consumer, specify the directory where the subscribed 
TsFile will be saved using the fileSaveDir parameter. Third, obtain the 
corresponding handler based on the type of SubscriptionMessage, which is 
SubscriptionTsFileHandler. SubscriptionTsFileHandler encapsulates operations 
like cp, mv, and rm from the Java standard library, and can also iterate data 
through the TsFile SDK, i.e., TsFileReader. Users can achieve TsFile 
subscription with minimal configuration.

Technically, we have restructured the Message Payload of the pipeSubscribe RPC 
poll type to support the transmission of data in both SessionDataSet and TsFile 
formats. The transmission of a TsFile file is divided into multiple events, 
including tsfile init event, tsfile piece event, and tsfile seal event. The 
reliable transmission of TsFile files is achieved through the interaction of 
these events between the DN side and the client side.

Currently, TsFile subscription still has some limitations. For example, when 
the format of the topic is TsFileHandler, there are certain constraints on its 
path and time configuration. We will continue to optimize these issues in 
future iterations.

I have initially implemented support for TsFile subscription in this PR[1]. I 
hope you are interested in this feature and would like to participate in the 
development and testing. You can also leave your comments and suggestions in 
this thread. Appreciate any suggestion/feedback & contribution.

Thank you for your attention and support.

Best regards,
VGalaxies

Reference:
1. https://github.com/apache/iotdb/pull/12326

On 2024/04/08 03:06:59 VGalaxies wrote:
> Hello everyone,
>
> I am VGalaxies, a new contributor to Apache IoTDB. I am excited to
> share with you a new feature that I have been working on for the past
> few months.
>
> The data subscription client is a new way to access data within IoTDB,
> distinct from the traditional method of querying data using SQL-like
> syntax. In scenarios where real-time data, quick response to data
> changes, and building highly event-driven systems are required, data
> subscription has greater advantages over data querying. For example,
> in the following two scenarios:
>
> 1. Replace extensive polling queries for large amounts of data: Avoid
> significant impacts on the performance of existing systems when
> querying frequently or when there are many data points. Also, avoid
> problems with determining the query scope and ensure downstream
> receives accurate full data.
> 2. Facilitate downstream system integration: It's easier to integrate
> with components such as Flink, Spark, Kafka/DataX, Camel/MySQL, etc.
> There's no need to customize the logic of IoTDB's data change capture
> for each big data component separately, simplifying integration
> component design and making it easier for users.
>
> The IoTDB subscription client references some features defined by some
> message queue products like Kafka. It consists of 3 core concepts:
> Topic, Consumer, and Consumer Group.
>
> - Topic is a logical concept used by the IoTDB subscription client to
> classify data, serving as a channel for data publication. Producers
> publish data to specific topics, while consumers subscribe to these
> topics to receive related data. In the IoTDB subscription client,
> topics describe the sequence characteristics, time characteristics,
> presentation forms, and optional custom processing logic of subscribed
> data.
> - Consumer is an application or service in the IoTDB subscription
> client responsible for receiving and processing data published 

AW: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

+1 (binding)

Chris

[OK] Checked the signatures
[OK] Checked the hashes
[OK] Checked the content of the source bundle
[OK] Checked that the staged binaries work on their target platforms



Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 16:40
An: dev@iotdb.apache.org 
Betreff: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)
Hi all,

I after fixing some issues in the cmake-maven-plugin, have I updated the 
iotdb-bin-resources module to use these.
Now we can stage Apache Thrift binaries for all platforms.

In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), I 
also updated the Thrift version and staged a release of Thrift binaries for all 
of our supported platforms in Nexus.

I’ve staged the artifacts at:
https://repository.apache.org/content/repositories/orgapacheiotdb-1158

Tag Name: iotdb-tools-thrift-v0.20.0.0
https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0

The source bundle generally consists of a pom.xml and an assembly xml … so it 
should be quick to review.
https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip

I’m intentionally not staging it on the release-svn, as nobody should be 
required to consume this module but us ourselves.

The vote stays open for 72hours … assuming enough support, I’ll then finish 
changing the main repo to use these artifacts.


Chris

[DISCUSS] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Hi all,

this vote is not on switching to Thrift 0.20.0. In order to try out Thrift we 
however need to have the Thrift binaries available.
That’s why I’ve started a VOTE on publishing these.

The alternative would be to work with intermediate maven repos, which just 
makes things a lot more difficult.

Chris

[VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)

[Christofer Dutz]

Hi all,

I after fixing some issues in the cmake-maven-plugin, have I updated the 
iotdb-bin-resources module to use these.
Now we can stage Apache Thrift binaries for all platforms.

In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible again), I 
also updated the Thrift version and staged a release of Thrift binaries for all 
of our supported platforms in Nexus.

I’ve staged the artifacts at:
https://repository.apache.org/content/repositories/orgapacheiotdb-1158

Tag Name: iotdb-tools-thrift-v0.20.0.0
https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0

The source bundle generally consists of a pom.xml and an assembly xml … so it 
should be quick to review.
https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip

I’m intentionally not staging it on the release-svn, as nobody should be 
required to consume this module but us ourselves.

The vote stays open for 72hours … assuming enough support, I’ll then finish 
changing the main repo to use these artifacts.


Chris

Could anyone please approve this PR?

[Christofer Dutz]

Hi all,

in the iotdb-bin-resources I have also updated the Cmake version and thrift 
version in order to be able to deploy new versions of thrift artifacts.

https://github.com/apache/iotdb-bin-resources/pull/61

Would be cool if I could get some approval.

Chris

AW: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Ok … and done:
https://repository.apache.org/content/repositories/orgapacheiotdb-1158

Here are all 6 types of binaries.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 11:01
An: dev@iotdb.apache.org 
Betreff: AW: [DISCUSS] Investigate updating to Thrift 0.20.0?
Ok …

So, I’m currently working on staging the last of the binaries (thrift for 
windows x86_64).
However, in order to make this work I think we would also need to update the 
CMake version.
I spent a lot of time over the last weekends figuring out how to support Cmake 
on all of these platforms with the maintainer of the cmake-maven-plugin.
Now we have a version that works on all systems I have.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 08:58
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?
Happy to prepare the binaries for this.

Chris

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Haonan Hou 
Sent: Tuesday, May 28, 2024 3:37:30 AM
To: dev-iotdb 
Subject: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
>
> Chris

AW: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Ok …

So, I’m currently working on staging the last of the binaries (thrift for 
windows x86_64).
However, in order to make this work I think we would also need to update the 
CMake version.
I spent a lot of time over the last weekends figuring out how to support Cmake 
on all of these platforms with the maintainer of the cmake-maven-plugin.
Now we have a version that works on all systems I have.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 28. Mai 2024 um 08:58
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?
Happy to prepare the binaries for this.

Chris

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Haonan Hou 
Sent: Tuesday, May 28, 2024 3:37:30 AM
To: dev-iotdb 
Subject: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
>
> Chris

Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Happy to prepare the binaries for this.

Chris

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Haonan Hou 
Sent: Tuesday, May 28, 2024 3:37:30 AM
To: dev-iotdb 
Subject: Re: [DISCUSS] Investigate updating to Thrift 0.20.0?

Hi,

I tried to upgrade Thrift 0.19.0 last year[1], and I think there is no 
compatibility issue if the client side still using the current v0.14.1.

Maybe we need to do more performance tests?

[1] https://github.com/apache/iotdb/pull/11183

Haonan

> On May 28, 2024, at 3:40 AM, Christofer Dutz  
> wrote:
>
> Hi all,
>
> while doing some experiments over the weekend I came to notice that Thrift 
> 0.19.0 and above is again compatible with Java 8 … this was the major reason 
> for us not updating to versions without CVEs.
> I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.
>
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Might be worth investigating to update? I’ve already got an up-to-date 
> version of the thrift compiler … if we wanted to try it out, I could stage 
> release versions for:
>
>
>  *   Mac – aarch64
>  *   Mac – x86_64
>  *   Linux  – aarch64
>  *   Linux  – x86_64
>  *   Windows  – aarch64
>  *   Windows  – x86_64
>
> Chris

Re: Missing binaries for Windows on Aarch64 systems

[Christofer Dutz]

Yeah.

But I could also produce a new set of binaries for 0.14.1...we'd have to 
re-release them too.

Chris

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Jialin Qiao 
Sent: Tuesday, May 28, 2024 3:39:59 AM
To: dev@iotdb.apache.org 
Subject: Re: Missing binaries for Windows on Aarch64 systems

Hi,

Is this will also be fixed by upgrading thrift to 0.20?

Jialin Qiao

Christofer Dutz  于2024年5月24日周五 22:58写道：
>
> Hi all,
>
> I was just verifying the build using the most recent cmake-maven-plugin works 
> and stumbled over something.
>
> Currently we don’t have thrift binaries for combinations of systems. It seems 
> we’re missing windows-aarch64.
>
> Should we plan fixing this and doing a release 0.14.1.1?
>
> Chris

[DISCUSS] Investigate updating to Thrift 0.20.0?

[Christofer Dutz]

Hi all,

while doing some experiments over the weekend I came to notice that Thrift 
0.19.0 and above is again compatible with Java 8 … this was the major reason 
for us not updating to versions without CVEs.
I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it.

https://github.com/apache/thrift/blob/master/CHANGES.md

Might be worth investigating to update? I’ve already got an up-to-date version 
of the thrift compiler … if we wanted to try it out, I could stage release 
versions for:


  *   Mac – aarch64
  *   Mac – x86_64
  *   Linux  – aarch64
  *   Linux  – x86_64
  *   Windows  – aarch64
  *   Windows  – x86_64

Chris

Missing binaries for Windows on Aarch64 systems

[Christofer Dutz]

Hi all,

I was just verifying the build using the most recent cmake-maven-plugin works 
and stumbled over something.

Currently we don’t have thrift binaries for combinations of systems. It seems 
we’re missing windows-aarch64.

Should we plan fixing this and doing a release 0.14.1.1?

Chris

AW: Possible fix for randomly failing Jenkins builds

[Christofer Dutz]

Done:
https://github.com/apache/iotdb/pull/12542
Waiting for approval.

Chris

Von: Jialin Qiao 
Datum: Freitag, 17. Mai 2024 um 09:15
An: dev@iotdb.apache.org 
Betreff: Re: Possible fix for randomly failing Jenkins builds
+1 for importing the solutions in PLC4X


Jialin Qiao

Christofer Dutz  于2024年5月14日周二 23:32写道：
>
> Hi all,
>
> I know that in IoTDB as well as PLC4X we were having issues of regularly but 
> randomly failing Jenkins builds.
>
> Usually, these problems were because of timeouts when deploying to Nexus.
>
> In PLC4X I have now tried something that seems to be doing the job … possibly 
> this could also help us here (I mean … the biggest part of the Jenkins setup 
> is borrowed from PLC4X anyway ;-) )
>
> My solution was to wrap the execution of the deploy step in a loop on bash 
> level:
>
> https://github.com/apache/plc4x/blob/develop/Jenkinsfile#L167
>
> sh 'until ./mvnw -f jenkins.pom -X -P deploy-snapshots 
> wagon:upload || (( count++ >= 5 )); do echo "Retrying to deploy"; done'
>
> So, it would try up to 5 times, if deployment fails.
>
> I think in case of deployment the full build is done and was successful. The 
> only reason for failures in this step usually is problems with Nexus and a 
> stupid retry should help.
>
> What do you think? Should we try the same change?
>
>
> Chris

Re: [DISCUSS] Drop Java 8?

[Christofer Dutz]

So we have real numbers?

I also haven't used an oracle jre for quite some time for these reasons.

Would a double branch strategy work? That the project maintains a 1.x branch 
with Javax and Java 8 and a 2.0 branch with Jakarta and Java 11?

Sort of like we had the 0.x and 1.x branches around?

Chris


Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Trevor Hart 
Sent: Friday, May 17, 2024 4:01:14 AM
To: dev 
Subject: Re: [DISCUSS] Drop Java 8?

I think a lot of organisations are sticking to Java 8 because of the change to 
Oracle license that was introduced in Java 11.



If you use Oracle 11 JRE you need to pay Oracle for a license.



This was why Open JDK came about.



Personally I use IotDB with Open JDK 11 (Eclipse Temurin) which does not 
require a license.



Thanks

Trevor Hart








 On Fri, 17 May 2024 13:30:15 +1200 Yuan Tian  
wrote ---



Hi Chris,

It seems that a lot of people still use jdk1.8 in their product environment.

Best regards,
-
Yuan Tian

On Thu, May 16, 2024 at 8:10 PM Christofer Dutz 
<mailto:christofer.d...@c-ware.de>
wrote:

> Hi all,
>
> starting this new thread as I am not sure if others are reading the
> Jakarta migration thread.
>
> I would like to propose planning on dropping Java 8 support.
>
> I wouldn’t immediately do that, and I would also propose to do a major
> version update (Switching to 2.0.0)
>
> We could still maintain a 1.x branch for those people not able to update.
>
> The main reason is that we are currently blocking ourselves from updating
> many major plugins and dependencies.
> I noticed that when updating to the Jakarta namespace. Here there is no
> Netty version available that supports Jakarta and supports Java 8.
>
> Other libraries where we are not able to update without giving up on Java
> 8:
>
>   *   Airlift-Units (Stuck at 1.7 current 1.10)
>   *   Airlift
>   *   Antlr (Stuck at 4.9.3 current 4.13.1)
>   *   Caffeine (Stuck at 2.9.3 current 3.1.8)
>   *   Logback (Stuck at 1.3.14 current 1.5.6)
>   *   Mockito (Stuck at 2.23.4 current 5.12.0)
>   *   Thrift (Stuck at 0.17.0 current 0.20.0)
>
>
>
>   *   Spotless Plugin (We’ve got a workaround for Java 8)
>
>
> In my branch where I refactored the javax namespace to Jakarta after
> updating dependencies I was able to remove all exclusions of the
> BanVulnerableDependencies check.
>
> Also does dropping Java 8 and the Jakarta migration allow embedding IoTDB
> in recent Spring versions.
>
>
> So … what do you think?
>
>
> Chris
>

[DISCUSS] Drop Java 8?

[Christofer Dutz]

Hi all,

starting this new thread as I am not sure if others are reading the Jakarta 
migration thread.

I would like to propose planning on dropping Java 8 support.

I wouldn’t immediately do that, and I would also propose to do a major version 
update (Switching to 2.0.0)

We could still maintain a 1.x branch for those people not able to update.

The main reason is that we are currently blocking ourselves from updating many 
major plugins and dependencies.
I noticed that when updating to the Jakarta namespace. Here there is no Netty 
version available that supports Jakarta and supports Java 8.

Other libraries where we are not able to update without giving up on Java 8:

  *   Airlift-Units (Stuck at 1.7 current 1.10)
  *   Airlift
  *   Antlr (Stuck at 4.9.3 current 4.13.1)
  *   Caffeine (Stuck at 2.9.3 current 3.1.8)
  *   Logback (Stuck at 1.3.14 current 1.5.6)
  *   Mockito (Stuck at 2.23.4 current 5.12.0)
  *   Thrift (Stuck at 0.17.0 current 0.20.0)



  *   Spotless Plugin (We’ve got a workaround for Java 8)


In my branch where I refactored the javax namespace to Jakarta after updating 
dependencies I was able to remove all exclusions of the 
BanVulnerableDependencies check.

Also does dropping Java 8 and the Jakarta migration allow embedding IoTDB in 
recent Spring versions.


So … what do you think?


Chris

AW: Jakarta migration

[Christofer Dutz]

Hi Wang,

I know that there is no schedule for that, but I would like to bring up the 
need to discuss this.

Right now, we are stuck with updating 10 or more dependencies to more recent 
ones and I think I have 2-3 CVE exclusions in the build that manually allow the 
build to succeed, even if there are vulnerabilities known for some of our 
libraries as the issues could not be resolved because of us requiring Java 8 
compatibility.

I think it would be beneficial for the project to discuss giving up on Java 8 
support.

One possible option would be to continue a 1.3.x or even better the 1.x branch 
which stays compatible with Java 8 and then to create a 2.0.0 branch which 
baselines on Java 11.

Chris


Von: Wang Critas 
Datum: Mittwoch, 15. Mai 2024 um 10:17
An: dev@iotdb.apache.org 
Betreff: 答复: Jakarta migration
However, it seems that we have not abandoned Java 8 plan

发件人: Christofer Dutz 
日期: 星期二, 2024年5月14日 23:26
收件人: dev@iotdb.apache.org 
主题: AW: Jakarta migration
Most of the build is now working correctly however there are still two 
Integration tests that are failing.
At first I thought they were due to issues with the server, but I am actually 
able to start my Jakarta version of IoTDB and connect to it with the JDBC 
driver without any issues. Also don’t I see any issues in the logs when 
starting a distribution.

I assume the remaining issues might be on the client side of the tests and will 
continue with my work as soon as I’m done with this month’s board report 
reviews.

Chris


Von: Christofer Dutz 
Datum: Montag, 13. Mai 2024 um 17:41
An: dev@iotdb.apache.org 
Betreff: Jakarta migration
Hi all,

I am currently working on the Jakarta migration … initially I thought of this 
as an experiment in order to find out what the implications would be.
Turns out my gut-feeling was right that this would not be a simple change. Less 
for the complexity of the changes, but what they would bring to the rest of the 
build.

So, for now it seems as if the only Netty version compatible with the Jakarta 
namespace would be Netty 11 (Technically they claimed 10 would be compatible, 
but that’ just not true).
Unfortunately, Netty 9 was the last version to work with Java 8 … so it seems 
as if going to the Jakarta namespace would stop us from building on Java 8.

I know I knew why I would have preferred this migration for a bigger release, 
but I guess a 1.4.0 or 2.0.0 might be more appropriate for a change like this.

Chris

Possible fix for randomly failing Jenkins builds

[Christofer Dutz]

Hi all,

I know that in IoTDB as well as PLC4X we were having issues of regularly but 
randomly failing Jenkins builds.

Usually, these problems were because of timeouts when deploying to Nexus.

In PLC4X I have now tried something that seems to be doing the job … possibly 
this could also help us here (I mean … the biggest part of the Jenkins setup is 
borrowed from PLC4X anyway ;-) )

My solution was to wrap the execution of the deploy step in a loop on bash 
level:

https://github.com/apache/plc4x/blob/develop/Jenkinsfile#L167

sh 'until ./mvnw -f jenkins.pom -X -P deploy-snapshots 
wagon:upload || (( count++ >= 5 )); do echo "Retrying to deploy"; done'

So, it would try up to 5 times, if deployment fails.

I think in case of deployment the full build is done and was successful. The 
only reason for failures in this step usually is problems with Nexus and a 
stupid retry should help.

What do you think? Should we try the same change?


Chris

AW: Jakarta migration

[Christofer Dutz]

Most of the build is now working correctly however there are still two 
Integration tests that are failing.
At first I thought they were due to issues with the server, but I am actually 
able to start my Jakarta version of IoTDB and connect to it with the JDBC 
driver without any issues. Also don’t I see any issues in the logs when 
starting a distribution.

I assume the remaining issues might be on the client side of the tests and will 
continue with my work as soon as I’m done with this month’s board report 
reviews.

Chris


Von: Christofer Dutz 
Datum: Montag, 13. Mai 2024 um 17:41
An: dev@iotdb.apache.org 
Betreff: Jakarta migration
Hi all,

I am currently working on the Jakarta migration … initially I thought of this 
as an experiment in order to find out what the implications would be.
Turns out my gut-feeling was right that this would not be a simple change. Less 
for the complexity of the changes, but what they would bring to the rest of the 
build.

So, for now it seems as if the only Netty version compatible with the Jakarta 
namespace would be Netty 11 (Technically they claimed 10 would be compatible, 
but that’ just not true).
Unfortunately, Netty 9 was the last version to work with Java 8 … so it seems 
as if going to the Jakarta namespace would stop us from building on Java 8.

I know I knew why I would have preferred this migration for a bigger release, 
but I guess a 1.4.0 or 2.0.0 might be more appropriate for a change like this.

Chris

Jakarta migration

[Christofer Dutz]

Hi all,

I am currently working on the Jakarta migration … initially I thought of this 
as an experiment in order to find out what the implications would be.
Turns out my gut-feeling was right that this would not be a simple change. Less 
for the complexity of the changes, but what they would bring to the rest of the 
build.

So, for now it seems as if the only Netty version compatible with the Jakarta 
namespace would be Netty 11 (Technically they claimed 10 would be compatible, 
but that’ just not true).
Unfortunately, Netty 9 was the last version to work with Java 8 … so it seems 
as if going to the Jakarta namespace would stop us from building on Java 8.

I know I knew why I would have preferred this migration for a bigger release, 
but I guess a 1.4.0 or 2.0.0 might be more appropriate for a change like this.

Chris

Using SNAPSHOT versions in our releases

[Christofer Dutz]

Hi all,

I just noticed that we have a dependency to a SNAPSHOT ratis versions in our 
build.

3.1.0-611b80a-SNAPSHOT

I wasn’t too concerned about this as I saw it was just recently set to that, 
but having a deeper look, this was changed from yet another snapshot version of 
ratis.

The problem is, that these snapshots only exist in the Apache maven repository 
and they will be cleaned up after some time. This WILL break any older version 
of IoTDB out there in the wild.
People will not be able to build our source bundles.

We really need to ensure that we only release software that consists completely 
of released aritfacts.

Chris

[DISCUSS] How about renaming the "openapi" and "antlr" modules? ... and yeah "iotdb-server" ;-)

[Christofer Dutz]

Hi all,

the more I come across these modules, the more do I think we should rename them.
Both doesn’t describe what they are, but much more which tools they use.

We don’t call the other modules “java”.

I would propose to call them according to what they are:

“openapi”:
Contains the rest API definition (and server-stub implementation)
So how about naming it “rest-api”?

“antlr”:
Contains the grammars and generated parsers for our SQL dialect and other 
parsable things.
So how about naming it “parsers”?

While at it … the “data-node” is the only one in the core directory not 
following the pattern:
ArtifactId = “iotdb-“ + directory name.
So how about naming it “iotdb-datanode”?



Chris

[DISCUSS] Migrate from the JavaX to the Jakarta Namespace?

[Christofer Dutz]

Hi all,

So, I brought this up before but bringing it up again as it’s limiting us more 
and more.

We are still using dependency versions that force us sticking to the old Javax 
namespace. However, this is no longer supported and most projects have migrated 
to the new Jakarta namespace.

Us sticking to Javax for example is causing problems when using the most recent 
Spring versions … we are forced to stay in the 2.x branch of Spring.

Also are we forced to not update some dependencies, for which newer versions 
are available, as these projects migrated to Jakarta and we’re stuck at the 
last version they still used Javax.

I think we really need to update.

I know we said that first we should release 3.1 … that’s now done, so when can 
we start this undertaking?

Chris

[DISCUSS] Refactor the MQTT subsystem to work with external MQTT Brokers?

[Christofer Dutz]

Hi all,

currently a lot of the plans we have for 2024 (at least my plans) involve MQTT 
as one form of transport for data into IoTDB.

Unfortunately, currently all of IoTDBs MQTT support is built around the 
built-in Moquette MQTT broker.

However, I think it’s totally out of the question, that any larger company 
would agree on using this broker as their communication back-bone.

Therefore, I would propose to refactor IoTDB to be able to use any MQTT broker 
as client but to keep Moquette in there, if a user doesn’t have an MQTT broker 
available.

The main change that we would need, would be to refactor the code that 
currently integrates into Moquette for processing messages and to rewrite this 
to use a normal MQTT client library.


What do you others think about this?

Chris

Should we add Java 21 to the build matrix in GitHub Actions?

[Christofer Dutz]

Hi all,

I’ve just noticed that we currently only test with 8, 11, 17 … however 21 is 
also an LTS version, so should we probably also add that to the list.

What do you folks think?

Chris

AW: Potential problem with the git-commit-id-plugin in node-commons

[Christofer Dutz]

Thinking about it a bit more … I think we should remove this plugin and the 
code that uses it.
The reasoning behind this is that I will break build reproducibility.

In theory someone should be able to download the source bundle from Apache 
servers, verify the hashes and the signatures and then be able to build binary 
identical jar files.

This plugin or the file that it generates will break this ability.

Chris

Von: Christofer Dutz 
Datum: Donnerstag, 18. April 2024 um 14:40
An: dev@iotdb.apache.org 
Betreff: AW: Potential problem with the git-commit-id-plugin in node-commons
Hi all,

digging a bit deeper I found that it would not cause problems in case of a 
source bundle, as there is this setting:

false

I assume that the worktree support does some strange things that this plugin 
doesn’t operate with correctly. So I added this setting:

false

And that solved the issue for me.


Chris

Von: Christofer Dutz 
Datum: Donnerstag, 18. April 2024 um 14:32
An: dev@iotdb.apache.org 
Betreff: Potential problem with the git-commit-id-plugin in node-commons
Hi all,

I was playing around with the new IntelliJ Worktree support (Checking out 
multiple versions of the same project) and got a failure in node-commons.

Having a deeper look at it, it seems that the build requires iotdb to be a git 
repository.

While this is true for regularly checked-out code, it would not work when 
building IoTDB from a source bundle downloaded from Apache.

I think we shouldn’t require such a plugin in our build. What is it used for 
anyway?

Chris

AW: Potential problem with the git-commit-id-plugin in node-commons

[Christofer Dutz]

Hi all,

digging a bit deeper I found that it would not cause problems in case of a 
source bundle, as there is this setting:

false

I assume that the worktree support does some strange things that this plugin 
doesn’t operate with correctly. So I added this setting:

false

And that solved the issue for me.


Chris

Von: Christofer Dutz 
Datum: Donnerstag, 18. April 2024 um 14:32
An: dev@iotdb.apache.org 
Betreff: Potential problem with the git-commit-id-plugin in node-commons
Hi all,

I was playing around with the new IntelliJ Worktree support (Checking out 
multiple versions of the same project) and got a failure in node-commons.

Having a deeper look at it, it seems that the build requires iotdb to be a git 
repository.

While this is true for regularly checked-out code, it would not work when 
building IoTDB from a source bundle downloaded from Apache.

I think we shouldn’t require such a plugin in our build. What is it used for 
anyway?

Chris

Potential problem with the git-commit-id-plugin in node-commons

[Christofer Dutz]

Hi all,

I was playing around with the new IntelliJ Worktree support (Checking out 
multiple versions of the same project) and got a failure in node-commons.

Having a deeper look at it, it seems that the build requires iotdb to be a git 
repository.

While this is true for regularly checked-out code, it would not work when 
building IoTDB from a source bundle downloaded from Apache.

I think we shouldn’t require such a plugin in our build. What is it used for 
anyway?

Chris

AW: 答复: Splitting up the repos

[Christofer Dutz]

Hmmm … well guess I’ll re-do my changes again then and split up the examples.

Next time it would be less of a waste of time if people would have responded to 
my questions quicker. I mean … I gave everyone 5 full days and got nothing.

Will do a second round of the iotdb part … the already split repo will be easy, 
but I don’t want to copy stuff back cause then we loose the commit history 
whithuot any necessity.
So, I’ll replay my changes.

Chris

Von: Haonan Hou 
Datum: Donnerstag, 18. April 2024 um 05:29
An: dev@iotdb.apache.org 
Betreff: Re: 答复: Splitting up the repos
+1

Haonan

On 2024/04/17 08:46:16 Wang Critas wrote:
> The examples could be classified into two parts
> (1) Connector example: flink, hadoop, kafka, pulsar, rabiitmq, rocketmq
> (2) IoTDB native api example: jdbc, mqtt, pipe, rest, schema, session,
> trigger, udf
>
> For (1), we could move into extra repo.
> For (2) , they should be retained in the IoTDB repo.
>
> Jialin Qiao
>
>
> Hi
>
> if we were to vote on this (which we can) I would vote +1 on ‘ The examples 
> could be classified into two parts '
>
> Voting will be open for 72hr.
> A minimum of 3 binding +1 votes and more binding +1 than binding -1
> are required to pass.
> You can achieve the above by following [4].
> [ ] +1 accept
> [ ] -1 reject (explanation required)
>
> Xuan
>
> 发件人: Christofer Dutz 
> 日期: 星期三, 2024年4月17日 16:38
> 收件人: dev@iotdb.apache.org 
> 主题: AW: Splitting up the repos
> We however still need this to be decided.
>
> Currently we have something such as one binding vote for keeping the examples 
> together and one for splitting it up.
> I have no objections to splitting it up, but we need to do things the Apache 
> way here …
>
> Otherwise this would imply that there are votes that count more than others 
> and we need to make sure this impression is not made.
>
> Chris
>
>
> Von: Wang Critas 
> Datum: Mittwoch, 17. April 2024 um 10:20
> An: dev@iotdb.apache.org 
> Betreff: 答复: Splitting up the repos
> In here
> https://github.com/apache/iotdb-extras/pull/6<https://github.com/apache/iotdb-extras/pull/6><https://github.com/apache/iotdb-extras/pull/6%3chttps:/github.com/apache/iotdb-extras/pull/6%3e>
>
> 发件人: Christofer Dutz 
> 日期: 星期一, 2024年4月15日 21:01
> 收件人: dev@iotdb.apache.org 
> 主题: AW: Splitting up the repos
> Well,
>
> as I said … I’ve already done the work of splitting things up (I had waited 5 
> days for any comments here)
>
> If we would now do it differently, I am sure someone would be able to re-do 
> the split based on my work and then simply delete the double examples from 
> the extras repo.
>
> So, I would be voting for moving all examples to the extras, you for 
> splitting the examples, guess we need at least a third vote (with hopefully 
> not a 3rd opinion ;-) )
>
> Chris
>
>
> Von: Jialin Qiao 
> Datum: Montag, 15. April 2024 um 14:38
> An: dev@iotdb.apache.org 
> Betreff: Re: Splitting up the repos
> Hi,
>
> The examples could be classified into two parts
> (1) Connector example: flink, hadoop, kafka, pulsar, rabiitmq, rocketmq
> (2) IoTDB native api example: jdbc, mqtt, pipe, rest, schema, session,
> trigger, udf
>
> For (1), we could move into extra repo.
> For (2) , they should be retained in the IoTDB repo.
>
> Jialin Qiao
>
> Christofer Dutz  于2024年4月15日周一 17:56写道：
> >
> > Hmpf,
> >
> > a little bit sooner reply would have been good … I’m already done with the 
> > changes, also with moving all examples and the parts of the distribution 
> > bundling the connectors.
> >
> > I do think also moving the examples is a good idea. Usually, examples pull 
> > in all sorts of dependencies, which show up on vulnerability reports. Also 
> > do we have some examples that refer to stuff we now moved out of the main 
> > repo, we’d be getting a cyclic dependency from that, so we would have to 
> > split up the examples in that case.
> >
> > So, if we were to vote on this (which we can) I would vote +1 on moving all 
> > examples out of the main repo.
> >
> > Chris
> >
> >
> > Von: Jialin Qiao 
> > Datum: Montag, 15. April 2024 um 11:20
> > An: dev@iotdb.apache.org 
> > Betreff: Re: Splitting up the repos
> > Hi,
> >
> > 1. Which Parts: Only iotdb-connector need to be moved,  distribution
> > and examples will impact the release and users.
> > 2. How to split up:  I prefer【Simply ignore the history, copy the
> > files to the new repo and delete them from the old】.
> >
> > Jialin Qiao
> >
> > Christofer Dutz  于2024年4月15日周一 16:27写道：
> > >
> > > Hi all,
> >

Re: Archive apache/iotdb-web-workbench repo?

[Christofer Dutz]

+1 for archiving it.

Perfect would be, that we address the issues, then release it and announce it's 
retirement. But if we don't have the capacity to fix them, then retiring the 
module is better than leaving it.

Chris

Gesendet von Outlook für Android

From: Jialin Qiao 
Sent: Thursday, April 18, 2024 5:30:37 AM
To: dev@iotdb.apache.org 
Subject: Re: Archive apache/iotdb-web-workbench repo?

+1 for archive


Jialin Qiao

Haonan Hou  于2024年4月18日周四 11:23写道：
>
> Hi all,
>
> We received some security issues about apache/iotdb-web-workbench recently. 
> However, we don't have enough maintainers to fix them. The repo has no new 
> commit since Apr 3, 2023. There is an option that we can archive that repo.
>
> Any idea?
>
> Best,
> Haonan Hou

AW: Splitting up the repos

[Christofer Dutz]

We however still need this to be decided.

Currently we have something such as one binding vote for keeping the examples 
together and one for splitting it up.
I have no objections to splitting it up, but we need to do things the Apache 
way here …

Otherwise this would imply that there are votes that count more than others and 
we need to make sure this impression is not made.

Chris


Von: Wang Critas 
Datum: Mittwoch, 17. April 2024 um 10:20
An: dev@iotdb.apache.org 
Betreff: 答复: Splitting up the repos
In here
https://github.com/apache/iotdb-extras/pull/6

发件人: Christofer Dutz 
日期: 星期一, 2024年4月15日 21:01
收件人: dev@iotdb.apache.org 
主题: AW: Splitting up the repos
Well,

as I said … I’ve already done the work of splitting things up (I had waited 5 
days for any comments here)

If we would now do it differently, I am sure someone would be able to re-do the 
split based on my work and then simply delete the double examples from the 
extras repo.

So, I would be voting for moving all examples to the extras, you for splitting 
the examples, guess we need at least a third vote (with hopefully not a 3rd 
opinion ;-) )

Chris


Von: Jialin Qiao 
Datum: Montag, 15. April 2024 um 14:38
An: dev@iotdb.apache.org 
Betreff: Re: Splitting up the repos
Hi,

The examples could be classified into two parts
(1) Connector example: flink, hadoop, kafka, pulsar, rabiitmq, rocketmq
(2) IoTDB native api example: jdbc, mqtt, pipe, rest, schema, session,
trigger, udf

For (1), we could move into extra repo.
For (2) , they should be retained in the IoTDB repo.

Jialin Qiao

Christofer Dutz  于2024年4月15日周一 17:56写道：
>
> Hmpf,
>
> a little bit sooner reply would have been good … I’m already done with the 
> changes, also with moving all examples and the parts of the distribution 
> bundling the connectors.
>
> I do think also moving the examples is a good idea. Usually, examples pull in 
> all sorts of dependencies, which show up on vulnerability reports. Also do we 
> have some examples that refer to stuff we now moved out of the main repo, 
> we’d be getting a cyclic dependency from that, so we would have to split up 
> the examples in that case.
>
> So, if we were to vote on this (which we can) I would vote +1 on moving all 
> examples out of the main repo.
>
> Chris
>
>
> Von: Jialin Qiao 
> Datum: Montag, 15. April 2024 um 11:20
> An: dev@iotdb.apache.org 
> Betreff: Re: Splitting up the repos
> Hi,
>
> 1. Which Parts: Only iotdb-connector need to be moved,  distribution
> and examples will impact the release and users.
> 2. How to split up:  I prefer【Simply ignore the history, copy the
> files to the new repo and delete them from the old】.
>
> Jialin Qiao
>
> Christofer Dutz  于2024年4月15日周一 16:27写道：
> >
> > Hi all,
> >
> >
> > So, I’ve set a tag on the main repository “before-moving-extras” 
> > (https://github.com/apache/iotdb/releases/tag/before-moving-extras<https://github.com/apache/iotdb/releases/tag/before-moving-extras<https://github.com/apache/iotdb/releases/tag/before-moving-extras%3chttps:/github.com/apache/iotdb/releases/tag/before-moving-extras>>)
> >
> > Also have I copied the content of the examples and integration modules to 
> > the new repo, duplicated the build there and updated the versions to 
> > artifacts in the main repo to reference a variable.
> >
> > The build in the extras seems to work, now I’ll have to strip out 
> > configurations, dependency management etc. for stuff that’s not needed in 
> > the extras and do the same in the main repo.
> >
> >
> >
> > Chris
> >
> >
> >
> > Von: Christofer Dutz 
> > Datum: Montag, 15. April 2024 um 09:22
> > An: dev@iotdb.apache.org 
> > Betreff: AW: Splitting up the repos
> > Ok …
> >
> > So, no comment I will simply treat as lazy consensus, therefore I will move 
> > forward with tagging the main repo with the latest changes as last revision 
> > before the split and reference that in the commit to the new repo.
> > Then I’ll simply copy over the files and delete them from the main repo.
> >
> > As with other projects however, I really dislike this form of workting 
> > together. Defaulting back to lazy consensus costs a lot of valuable time as 
> > I have to wait a reasonable amount of time. If I had gotten any “sure … I’m 
> > fine with you doing X” I could have long finished this.
> >
> > In the future it would be a lot better, if some people would actually reply.
> >
> >
> > Chris
> >
> >
> >
> > Von: Christofer Dutz 
> > Datum: Donnerstag, 11. April 2024 um 10:36
> > An: dev@iotdb.apache.org 
> > Betreff: Splitting up the repos
> > Hi all,
> >
&

AW: Totally outdated Spottless version?

[Christofer Dutz]

Hi,

well, I guess there’s a difference between which JDK is being used or which 
language-level we’re compiling for.
Different surveys, different outcome:
https://newrelic.com/resources/report/2023-state-of-the-java-ecosystem#most-used-java-versions

In PLC4X we have some dependency checks, that check that dependencies are 
class-format compatible with our desired base-line JDK version (11).
In order to be able to not be stuck with too old build tools, we decided early 
to have a higher base-line version for building PLC4X than for running 
applications.
I think till the beginning of this year we had the build require Java 11 and 
the runtime to be 8. In order to ensure the output worked, we had some tools in 
pace to ensure that.

Chris


Von: Wang Critas 
Datum: Dienstag, 16. April 2024 um 05:16
An: dev@iotdb.apache.org 
Betreff: 答复: Totally outdated Spottless version?
According to the latest report, Java 8 still has the highest developer share, 
although the 17+11 share is not significantly different.
But what we need to ensure for the upgrade is that all code can still be 
compiled and run in Java 8, or consider updating it on Apache/tsfile first

https://www.jetbrains.com/lp/devecosystem-2023/java/


50% Java 8

45% Java 17

38% Java 11

11% Java 20

8% Java 18

8% Java 19

3% Java 16

2% Java 14

2% Java 12

2% Java 7

2% Java 15

1% Java 13



发件人: Christofer Dutz 
日期: 星期一, 2024年4月15日 15:54
收件人: dev@iotdb.apache.org 
主题: Totally outdated Spottless version?
Hi all,

while going through the plugins in the new split-up repository, I came across 
the profile, that disabled spotless on Java 21 and above.
It seems this issue was fixed quite some time ago and simply updating to an a 
lot more recent version of spotless should resolve the issue.
I think the reason we stuck to the ancient version that we did, was as this was 
the last version able to build with Java 8.

I would propose to turn the logic, to exclude the spotless check for java 
versions older than the supported java version and to enable spotless on all 
newer versions.

Chris

AW: Splitting up the repos

[Christofer Dutz]

Well,

as I said … I’ve already done the work of splitting things up (I had waited 5 
days for any comments here)

If we would now do it differently, I am sure someone would be able to re-do the 
split based on my work and then simply delete the double examples from the 
extras repo.

So, I would be voting for moving all examples to the extras, you for splitting 
the examples, guess we need at least a third vote (with hopefully not a 3rd 
opinion ;-) )

Chris


Von: Jialin Qiao 
Datum: Montag, 15. April 2024 um 14:38
An: dev@iotdb.apache.org 
Betreff: Re: Splitting up the repos
Hi,

The examples could be classified into two parts
(1) Connector example: flink, hadoop, kafka, pulsar, rabiitmq, rocketmq
(2) IoTDB native api example: jdbc, mqtt, pipe, rest, schema, session,
trigger, udf

For (1), we could move into extra repo.
For (2) , they should be retained in the IoTDB repo.

Jialin Qiao

Christofer Dutz  于2024年4月15日周一 17:56写道：
>
> Hmpf,
>
> a little bit sooner reply would have been good … I’m already done with the 
> changes, also with moving all examples and the parts of the distribution 
> bundling the connectors.
>
> I do think also moving the examples is a good idea. Usually, examples pull in 
> all sorts of dependencies, which show up on vulnerability reports. Also do we 
> have some examples that refer to stuff we now moved out of the main repo, 
> we’d be getting a cyclic dependency from that, so we would have to split up 
> the examples in that case.
>
> So, if we were to vote on this (which we can) I would vote +1 on moving all 
> examples out of the main repo.
>
> Chris
>
>
> Von: Jialin Qiao 
> Datum: Montag, 15. April 2024 um 11:20
> An: dev@iotdb.apache.org 
> Betreff: Re: Splitting up the repos
> Hi,
>
> 1. Which Parts: Only iotdb-connector need to be moved,  distribution
> and examples will impact the release and users.
> 2. How to split up:  I prefer【Simply ignore the history, copy the
> files to the new repo and delete them from the old】.
>
> Jialin Qiao
>
> Christofer Dutz  于2024年4月15日周一 16:27写道：
> >
> > Hi all,
> >
> >
> > So, I’ve set a tag on the main repository “before-moving-extras” 
> > (https://github.com/apache/iotdb/releases/tag/before-moving-extras)
> >
> > Also have I copied the content of the examples and integration modules to 
> > the new repo, duplicated the build there and updated the versions to 
> > artifacts in the main repo to reference a variable.
> >
> > The build in the extras seems to work, now I’ll have to strip out 
> > configurations, dependency management etc. for stuff that’s not needed in 
> > the extras and do the same in the main repo.
> >
> >
> >
> > Chris
> >
> >
> >
> > Von: Christofer Dutz 
> > Datum: Montag, 15. April 2024 um 09:22
> > An: dev@iotdb.apache.org 
> > Betreff: AW: Splitting up the repos
> > Ok …
> >
> > So, no comment I will simply treat as lazy consensus, therefore I will move 
> > forward with tagging the main repo with the latest changes as last revision 
> > before the split and reference that in the commit to the new repo.
> > Then I’ll simply copy over the files and delete them from the main repo.
> >
> > As with other projects however, I really dislike this form of workting 
> > together. Defaulting back to lazy consensus costs a lot of valuable time as 
> > I have to wait a reasonable amount of time. If I had gotten any “sure … I’m 
> > fine with you doing X” I could have long finished this.
> >
> > In the future it would be a lot better, if some people would actually reply.
> >
> >
> > Chris
> >
> >
> >
> > Von: Christofer Dutz 
> > Datum: Donnerstag, 11. April 2024 um 10:36
> > An: dev@iotdb.apache.org 
> > Betreff: Splitting up the repos
> > Hi all,
> >
> > so now that the new repo is created 
> > (https://gitbox.apache.org/repos/asf/iotdb-extras.git, but please don’t 
> > push anything there just yet), we would need to decide on which parts 
> > should be moved there.
> >
> >
> >   *   “distribution”: Here I think we need to split the distribution. 
> > Keeping the distributions containing only core in the main repo and adding 
> > a new distribution module in the extras repo, that contains the downstream 
> > components.
> >   *   “example” (which I would propose to rename to examples as it contains 
> > multiple)
> >   *   “iotdb-connector”
> >
> >
> > As it seems that in the integration-tests there are no tests testing the 
> > connectors, I guess we can leave that as it is.
> >
> > Now the problem is: There are mu

AW: Splitting up the repos

[Christofer Dutz]

Hmpf,

a little bit sooner reply would have been good … I’m already done with the 
changes, also with moving all examples and the parts of the distribution 
bundling the connectors.

I do think also moving the examples is a good idea. Usually, examples pull in 
all sorts of dependencies, which show up on vulnerability reports. Also do we 
have some examples that refer to stuff we now moved out of the main repo, we’d 
be getting a cyclic dependency from that, so we would have to split up the 
examples in that case.

So, if we were to vote on this (which we can) I would vote +1 on moving all 
examples out of the main repo.

Chris


Von: Jialin Qiao 
Datum: Montag, 15. April 2024 um 11:20
An: dev@iotdb.apache.org 
Betreff: Re: Splitting up the repos
Hi,

1. Which Parts: Only iotdb-connector need to be moved,  distribution
and examples will impact the release and users.
2. How to split up:  I prefer【Simply ignore the history, copy the
files to the new repo and delete them from the old】.

Jialin Qiao

Christofer Dutz  于2024年4月15日周一 16:27写道：
>
> Hi all,
>
>
> So, I’ve set a tag on the main repository “before-moving-extras” 
> (https://github.com/apache/iotdb/releases/tag/before-moving-extras)
>
> Also have I copied the content of the examples and integration modules to the 
> new repo, duplicated the build there and updated the versions to artifacts in 
> the main repo to reference a variable.
>
> The build in the extras seems to work, now I’ll have to strip out 
> configurations, dependency management etc. for stuff that’s not needed in the 
> extras and do the same in the main repo.
>
>
>
> Chris
>
>
>
> Von: Christofer Dutz 
> Datum: Montag, 15. April 2024 um 09:22
> An: dev@iotdb.apache.org 
> Betreff: AW: Splitting up the repos
> Ok …
>
> So, no comment I will simply treat as lazy consensus, therefore I will move 
> forward with tagging the main repo with the latest changes as last revision 
> before the split and reference that in the commit to the new repo.
> Then I’ll simply copy over the files and delete them from the main repo.
>
> As with other projects however, I really dislike this form of workting 
> together. Defaulting back to lazy consensus costs a lot of valuable time as I 
> have to wait a reasonable amount of time. If I had gotten any “sure … I’m 
> fine with you doing X” I could have long finished this.
>
> In the future it would be a lot better, if some people would actually reply.
>
>
> Chris
>
>
>
> Von: Christofer Dutz 
> Datum: Donnerstag, 11. April 2024 um 10:36
> An: dev@iotdb.apache.org 
> Betreff: Splitting up the repos
> Hi all,
>
> so now that the new repo is created 
> (https://gitbox.apache.org/repos/asf/iotdb-extras.git, but please don’t push 
> anything there just yet), we would need to decide on which parts should be 
> moved there.
>
>
>   *   “distribution”: Here I think we need to split the distribution. Keeping 
> the distributions containing only core in the main repo and adding a new 
> distribution module in the extras repo, that contains the downstream 
> components.
>   *   “example” (which I would propose to rename to examples as it contains 
> multiple)
>   *   “iotdb-connector”
>
>
> As it seems that in the integration-tests there are no tests testing the 
> connectors, I guess we can leave that as it is.
>
> Now the problem is: There are multiple options to split up the repo and 
> keeping the entire history.
>
>   1.  Split out one directory in a separate branch and then merge all 
> branches into an empty new one
>   2.  Use the filter plugin to strip out all commits that match a regexp
>   3.  Simply ignore the history, copy the files to the new repo and delete 
> them from the old.
>
> 3 is the simples, but the person doing the move will be marked as author. In 
> general this is not that problematic, as the integration modules and the 
> examples are usually not that complex, but I would understand, if people 
> wanted to keep the history.
>
> Option 1 is probably the most work, but the most robust option, as with 
> option 2, I had to give up when doing the PLC4X split as there were bugs and 
> issues in the tooling.
>
> So, if nobody objects and we’ve decided on what should be moved, I personally 
> would opt for option 3.
>
> Chris

AW: Splitting up the repos

[Christofer Dutz]

Hi all,


So, I’ve set a tag on the main repository “before-moving-extras” 
(https://github.com/apache/iotdb/releases/tag/before-moving-extras)

Also have I copied the content of the examples and integration modules to the 
new repo, duplicated the build there and updated the versions to artifacts in 
the main repo to reference a variable.

The build in the extras seems to work, now I’ll have to strip out 
configurations, dependency management etc. for stuff that’s not needed in the 
extras and do the same in the main repo.



Chris



Von: Christofer Dutz 
Datum: Montag, 15. April 2024 um 09:22
An: dev@iotdb.apache.org 
Betreff: AW: Splitting up the repos
Ok …

So, no comment I will simply treat as lazy consensus, therefore I will move 
forward with tagging the main repo with the latest changes as last revision 
before the split and reference that in the commit to the new repo.
Then I’ll simply copy over the files and delete them from the main repo.

As with other projects however, I really dislike this form of workting 
together. Defaulting back to lazy consensus costs a lot of valuable time as I 
have to wait a reasonable amount of time. If I had gotten any “sure … I’m fine 
with you doing X” I could have long finished this.

In the future it would be a lot better, if some people would actually reply.


Chris



Von: Christofer Dutz 
Datum: Donnerstag, 11. April 2024 um 10:36
An: dev@iotdb.apache.org 
Betreff: Splitting up the repos
Hi all,

so now that the new repo is created 
(https://gitbox.apache.org/repos/asf/iotdb-extras.git, but please don’t push 
anything there just yet), we would need to decide on which parts should be 
moved there.


  *   “distribution”: Here I think we need to split the distribution. Keeping 
the distributions containing only core in the main repo and adding a new 
distribution module in the extras repo, that contains the downstream components.
  *   “example” (which I would propose to rename to examples as it contains 
multiple)
  *   “iotdb-connector”


As it seems that in the integration-tests there are no tests testing the 
connectors, I guess we can leave that as it is.

Now the problem is: There are multiple options to split up the repo and keeping 
the entire history.

  1.  Split out one directory in a separate branch and then merge all branches 
into an empty new one
  2.  Use the filter plugin to strip out all commits that match a regexp
  3.  Simply ignore the history, copy the files to the new repo and delete them 
from the old.

3 is the simples, but the person doing the move will be marked as author. In 
general this is not that problematic, as the integration modules and the 
examples are usually not that complex, but I would understand, if people wanted 
to keep the history.

Option 1 is probably the most work, but the most robust option, as with option 
2, I had to give up when doing the PLC4X split as there were bugs and issues in 
the tooling.

So, if nobody objects and we’ve decided on what should be moved, I personally 
would opt for option 3.

Chris

Totally outdated Spottless version?

[Christofer Dutz]

Hi all,

while going through the plugins in the new split-up repository, I came across 
the profile, that disabled spotless on Java 21 and above.
It seems this issue was fixed quite some time ago and simply updating to an a 
lot more recent version of spotless should resolve the issue.
I think the reason we stuck to the ancient version that we did, was as this was 
the last version able to build with Java 8.

I would propose to turn the logic, to exclude the spotless check for java 
versions older than the supported java version and to enable spotless on all 
newer versions.

Chris

AW: Splitting up the repos

[Christofer Dutz]

Ok …

So, no comment I will simply treat as lazy consensus, therefore I will move 
forward with tagging the main repo with the latest changes as last revision 
before the split and reference that in the commit to the new repo.
Then I’ll simply copy over the files and delete them from the main repo.

As with other projects however, I really dislike this form of workting 
together. Defaulting back to lazy consensus costs a lot of valuable time as I 
have to wait a reasonable amount of time. If I had gotten any “sure … I’m fine 
with you doing X” I could have long finished this.

In the future it would be a lot better, if some people would actually reply.


Chris



Von: Christofer Dutz 
Datum: Donnerstag, 11. April 2024 um 10:36
An: dev@iotdb.apache.org 
Betreff: Splitting up the repos
Hi all,

so now that the new repo is created 
(https://gitbox.apache.org/repos/asf/iotdb-extras.git, but please don’t push 
anything there just yet), we would need to decide on which parts should be 
moved there.


  *   “distribution”: Here I think we need to split the distribution. Keeping 
the distributions containing only core in the main repo and adding a new 
distribution module in the extras repo, that contains the downstream components.
  *   “example” (which I would propose to rename to examples as it contains 
multiple)
  *   “iotdb-connector”


As it seems that in the integration-tests there are no tests testing the 
connectors, I guess we can leave that as it is.

Now the problem is: There are multiple options to split up the repo and keeping 
the entire history.

  1.  Split out one directory in a separate branch and then merge all branches 
into an empty new one
  2.  Use the filter plugin to strip out all commits that match a regexp
  3.  Simply ignore the history, copy the files to the new repo and delete them 
from the old.

3 is the simples, but the person doing the move will be marked as author. In 
general this is not that problematic, as the integration modules and the 
examples are usually not that complex, but I would understand, if people wanted 
to keep the history.

Option 1 is probably the most work, but the most robust option, as with option 
2, I had to give up when doing the PLC4X split as there were bugs and issues in 
the tooling.

So, if nobody objects and we’ve decided on what should be moved, I personally 
would opt for option 3.

Chris

Splitting up the repos

[Christofer Dutz]

Hi all,

so now that the new repo is created 
(https://gitbox.apache.org/repos/asf/iotdb-extras.git, but please don’t push 
anything there just yet), we would need to decide on which parts should be 
moved there.


  *   “distribution”: Here I think we need to split the distribution. Keeping 
the distributions containing only core in the main repo and adding a new 
distribution module in the extras repo, that contains the downstream components.
  *   “example” (which I would propose to rename to examples as it contains 
multiple)
  *   “iotdb-connector”


As it seems that in the integration-tests there are no tests testing the 
connectors, I guess we can leave that as it is.

Now the problem is: There are multiple options to split up the repo and keeping 
the entire history.

  1.  Split out one directory in a separate branch and then merge all branches 
into an empty new one
  2.  Use the filter plugin to strip out all commits that match a regexp
  3.  Simply ignore the history, copy the files to the new repo and delete them 
from the old.

3 is the simples, but the person doing the move will be marked as author. In 
general this is not that problematic, as the integration modules and the 
examples are usually not that complex, but I would understand, if people wanted 
to keep the history.

Option 1 is probably the most work, but the most robust option, as with option 
2, I had to give up when doing the PLC4X split as there were bugs and issues in 
the tooling.

So, if nobody objects and we’ve decided on what should be moved, I personally 
would opt for option 3.

Chris

Re: Splitting up the main repo?

[Christofer Dutz]

Ok im use this thread as proof of consent and request the new repo as soon as 
possible.

Chris

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Jialin Qiao 
Sent: Monday, April 8, 2024 4:44:35 AM
To: dev@iotdb.apache.org 
Subject: Re: Splitting up the main repo?

+1 for moving these into extra repos.

Jialin Qiao

Xiangdong Huang  于2024年4月7日周日 09:40写道：
>
> > We surely also pull in a lot of potentially bad dependencies.
>
> and this may be a chance to re-check our dependencies and remove unnecessary..
>
> ---
> Xiangdong Huang
>
> Christofer Dutz  于2024年4月5日周五 19:13写道：
> >
> > Hi all,
> >
> > I just wanted to bring up one idea that we decided in the PLC4X project and 
> > seed the idea, if this would also be worth discussing here.
> >
> > So, we were seeing that our build kept on having sub-ideal CVE ratings as 
> > we had dependencies for which CVEs were reported.
> > However, PLC4X itself has a very limited number of dependencies. The 
> > problem was that we had several “integration” modules, that pulled in 
> > Kafka, Calcite, Nifi and some Eclipse projects.
> > Also did a lot of our examples pull in various third party libraries, for 
> > which also vulnerabilities were reported.
> >
> > We are currently in the process of splitting up our main repository into a 
> > main and an extras repository.
> > The main contains the core of the project. The extras contains the 
> > examples, additional tools and integration modules (The ones with the many, 
> > many dependencies)
> > This way we can get a much better secutity standing for the main repo.
> >
> > Would this also be a good idea for IoTDB? I know with our dependencies to:
> >
> >   *   Flink
> >   *   Grafana
> >   *   Hadoop
> >   *   Hive
> >   *   Spark
> >   *   Zeppelin (this one is really bad when it comes to CVEs)
> >   *   Pulsar (only examples)
> >   *   RabbitMQ (only examples)
> >   *   RocketMQ (only examples)
> >
> > We surely also pull in a lot of potentially bad dependencies. If we moved 
> > this out the same way we would probably have a much better CVE ranking.
> > This might become problematic in the future as in Europe and in the US 
> > CRE/PLD and other initiatives are taking form.
> >
> > Chris

Re: Biweekly meeting about iotdb & tsfile project

[Christofer Dutz]

Please be sure to post an English summary of the meeting here.

Chris

Gesendet von Outlook für Android

From: Yuan Tian 
Sent: Thursday, April 11, 2024 4:42:08 AM
To: dev 
Subject: Biweekly meeting about iotdb & tsfile project

Hi all,

The biweekly meeting will start at 8pm tonight.

The main content of the meeting is to synchronize with everyone the main
work directions of the community in the past two weeks (completed and
ongoing). During the introduction process, everyone is also welcome to
discuss current work items. If you have anything else you want to discuss
at the biweekly meeting, you can send it to me in advance and I will list
it in the document in advance.

The meeting link is:
https://meeting.tencent.com/dm/6BkPsKOYyzt9

The meeting No. is:
706-6364-8152

Best regards,
---
Yuan Tian

Splitting up the main repo?

[Christofer Dutz]

Hi all,

I just wanted to bring up one idea that we decided in the PLC4X project and 
seed the idea, if this would also be worth discussing here.

So, we were seeing that our build kept on having sub-ideal CVE ratings as we 
had dependencies for which CVEs were reported.
However, PLC4X itself has a very limited number of dependencies. The problem 
was that we had several “integration” modules, that pulled in Kafka, Calcite, 
Nifi and some Eclipse projects.
Also did a lot of our examples pull in various third party libraries, for which 
also vulnerabilities were reported.

We are currently in the process of splitting up our main repository into a main 
and an extras repository.
The main contains the core of the project. The extras contains the examples, 
additional tools and integration modules (The ones with the many, many 
dependencies)
This way we can get a much better secutity standing for the main repo.

Would this also be a good idea for IoTDB? I know with our dependencies to:

  *   Flink
  *   Grafana
  *   Hadoop
  *   Hive
  *   Spark
  *   Zeppelin (this one is really bad when it comes to CVEs)
  *   Pulsar (only examples)
  *   RabbitMQ (only examples)
  *   RocketMQ (only examples)

We surely also pull in a lot of potentially bad dependencies. If we moved this 
out the same way we would probably have a much better CVE ranking.
This might become problematic in the future as in Europe and in the US CRE/PLD 
and other initiatives are taking form.

Chris

AW: [DISCUSS] Make enabling "fat-jars" the default for the JDBC module?

[Christofer Dutz]

Yeah …

I know that you can enable that … for some reason IntelliJ seems to keep on 
forgetting which profiles were enabled every time I pull changes.
I just regularly keep on re-enabling it. But I was questioning, if the JDBC 
driver shouldn’t have this enabled per default, as I don’t see a benefit of 
having a slim-jar version.

Chris

Von: Wang Critas 
Datum: Dienstag, 2. April 2024 um 13:52
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Make enabling "fat-jars" the default for the JDBC module?
We will have the corresponding fat-jar when the version is released. Locally, 
if developing JDBC seems to add jar-with-dependencies manually, it is not 
troublesome.
Of course, we can also consider adding fat-jar to JDBC in distribution.

Xuan

Best regards,
Xuan Wang
____
From: Christofer Dutz 
Sent: Tuesday, April 2, 2024 7:42:40 PM
To: dev@iotdb.apache.org 
Subject: [DISCUSS] Make enabling "fat-jars" the default for the JDBC module?

Hi all,

so right now, I keep on having the issue, that I keep on needing to re-build 
IoTDB, as I forgot to enable the “jar-with-dependencies” profile when building 
the JDBC driver.
I would assume, that when building a native Java client, someone would possibly 
prefer using the session API and that JDBC is more needed when using IoTDB from 
a JDBC enabled client.

So, I think it would be better to have the fat-jar enabled per default.

What do you others think?

Chris

[DISCUSS] Make enabling "fat-jars" the default for the JDBC module?

[Christofer Dutz]

Hi all,

so right now, I keep on having the issue, that I keep on needing to re-build 
IoTDB, as I forgot to enable the “jar-with-dependencies” profile when building 
the JDBC driver.
I would assume, that when building a native Java client, someone would possibly 
prefer using the session API and that JDBC is more needed when using IoTDB from 
a JDBC enabled client.

So, I think it would be better to have the fat-jar enabled per default.

What do you others think?

Chris

Build issues on GitHub Actions

[Christofer Dutz]

Hi all,

So, it seems that building the CPP parts is currently failing on GitHub 
Actions, as the pre-compiled thrift executables seems to have been compiled on 
MacOS 13 however the GitHub Actions runners are still on MacOS 12.

CHris

AW: Keeping an eye on Jenkins

[Christofer Dutz]

So the code that controls when which email is sent where is here:
https://github.com/apache/iotdb/blob/master/Jenkinsfile#L185

I am not sure Jenkins can be configured to send emails to the authors of 
breaking builds. I think that’s a goodie only GitHub Actions have.

Chris

PS: It seems you are posting from an email address that is not subscribed. I 
had to moderate all of your emails today …

Von: 谭新宇 <1025599...@qq.com.INVALID>
Datum: Freitag, 15. März 2024 um 16:16
An: dev@iotdb.apache.org 
Betreff: Re: Keeping an eye on Jenkins
Hi, Chris

Late last year, we spent several months fixing the unstable UT/IT to get Github 
CI and Jenkins CI up and running. After our efforts, both of them should be 
stable for several weeks.

However, the last few months have brought some new CI instability issues, this 
time a rat file, and so on.

To produce snapshot builds successful and take CI stability into account, 
perhaps we can configure Jenkins CI to route successful builds to a trash list 
and otherwise alert the developer of failed builds. For example, send to dev 
list or a feishu group.

Best
—
Xinyu Tan

> 2024年3月13日 22:52，Christofer Dutz  写道：
>
> Hi all,
>
> So, I just had a look at our Jenkins build as in the past we had several 
> occasions where we were having failing builds for many days.
> Turns out for the last 4 days Jenkins has been continuously failing because 
> of a trivial RAT violation.
>
> However, have we not been producing SNAPSHOT versions, and nobody noticed, 
> because we decided to redirect build failure emails of Jenkins to a 
> trash-list.
>
> We really should discuss how to deal with failing builds.
>
> I personally think redirecting failed build notifications is the worst way to 
> deal with a flaky build.
> Instead, we really should work on ironing out the issues that are randomly 
> failing.
>
> Having a stable and solid build should be of upmost importance, as only with 
> this can we truly trust the CI.
>
>
> Chris

AW: [DISCUSS] Please don't write output of tests to the root directory of a module.

[Christofer Dutz]

I think I could try and go through all tests and try find which ones produce 
anything outside “target” and compile a list.
Then we know which tests need refactoring.

Chris


Von: 谭新宇 <1025599...@qq.com.INVALID>
Datum: Freitag, 15. März 2024 um 09:58
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Please don't write output of tests to the root directory 
of a module.
Hi, Chris

+1 for the fact that generating test output anywhere outside the target 
directory is really a bad idea.

It looks like the package name for this test changed on March 10th, which may 
have contributed to its failure. Maybe we can check all the tests to see if 
there are similar cases, otherwise there may be such problems in the future.



Best
———
Xinyu Tan


> 2024年3月15日 16:44，Christofer Dutz  写道：
>
> Hi Xinyu,
>
> I would assume that there were a lot of changes and updates in the rat-plugin 
> (the one that was failing the build).
> So, it could have been that this was all ok till a new version of rat was 
> being used.
>
> However that doesn’t change the fact that generating test output anywhere 
> outside the target directory is really a bad idea.
>
> Chris
>
> Von: 谭新宇 <1025599...@qq.com.INVALID>
> Datum: Freitag, 15. März 2024 um 09:38
> An: dev@iotdb.apache.org 
> Betreff: Re: [DISCUSS] Please don't write output of tests to the root 
> directory of a module.
> Hi, Chris
>
> I checked the UT and it looks like it hasn't been changed in two or three 
> years. We have been running into unexpected errors in the Jenkins 
> environment, such as data cleaning problems, which could lead to similar 
> errors.
>
> Naturally, the license header check is also included in Github CI continuous 
> integration, but it seems that github's environment is clean enough that 
> there is no problem.
>
> Maybe we need to investigate the Jenkins environment cleanup problem.
>
> Best
> 
> Xinyu Tan
>
>
>
>> 2024年3月15日 05:22，Christofer Dutz  写道：
>>
>> So,
>>
>> for the last few days we had failing builds on Jenkins again, as it was 
>> continuously complaining about two CSV files not having the right license 
>> headers.
>> Funny thing is, these tests only appear if you build with tests as one test 
>> created them.
>>
>> It’s a super-bad idea to write the output of unit-tests or integration tests 
>> into the root of a module, as exactly stuff like this happens.
>>
>> Please be sure to write them into some disposable directory underneath 
>> “target”.
>>
>> I have changed that one test now and hopefully we’ll have fresh SNAPSHOT 
>> version when I wake up tomorrow.
>>
>> This stupid little thing has caused a cascade of problems for me today which 
>> made me waste most of my working day today :-/
>>
>> So please: Pay a bit more attention to where you write stuff.
>>
>> Chris
>>

AW: [DISCUSS] Please don't write output of tests to the root directory of a module.

[Christofer Dutz]

Hi Xinyu,

I would assume that there were a lot of changes and updates in the rat-plugin 
(the one that was failing the build).
So, it could have been that this was all ok till a new version of rat was being 
used.

However that doesn’t change the fact that generating test output anywhere 
outside the target directory is really a bad idea.

Chris

Von: 谭新宇 <1025599...@qq.com.INVALID>
Datum: Freitag, 15. März 2024 um 09:38
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Please don't write output of tests to the root directory 
of a module.
Hi, Chris

I checked the UT and it looks like it hasn't been changed in two or three 
years. We have been running into unexpected errors in the Jenkins environment, 
such as data cleaning problems, which could lead to similar errors.

Naturally, the license header check is also included in Github CI continuous 
integration, but it seems that github's environment is clean enough that there 
is no problem.

Maybe we need to investigate the Jenkins environment cleanup problem.

Best

Xinyu Tan



> 2024年3月15日 05:22，Christofer Dutz  写道：
>
> So,
>
> for the last few days we had failing builds on Jenkins again, as it was 
> continuously complaining about two CSV files not having the right license 
> headers.
> Funny thing is, these tests only appear if you build with tests as one test 
> created them.
>
> It’s a super-bad idea to write the output of unit-tests or integration tests 
> into the root of a module, as exactly stuff like this happens.
>
> Please be sure to write them into some disposable directory underneath 
> “target”.
>
> I have changed that one test now and hopefully we’ll have fresh SNAPSHOT 
> version when I wake up tomorrow.
>
> This stupid little thing has caused a cascade of problems for me today which 
> made me waste most of my working day today :-/
>
> So please: Pay a bit more attention to where you write stuff.
>
> Chris
>

[DISCUSS] Please don't write output of tests to the root directory of a module.

[Christofer Dutz]

So,

for the last few days we had failing builds on Jenkins again, as it was 
continuously complaining about two CSV files not having the right license 
headers.
Funny thing is, these tests only appear if you build with tests as one test 
created them.

It’s a super-bad idea to write the output of unit-tests or integration tests 
into the root of a module, as exactly stuff like this happens.

Please be sure to write them into some disposable directory underneath “target”.

I have changed that one test now and hopefully we’ll have fresh SNAPSHOT 
version when I wake up tomorrow.

This stupid little thing has caused a cascade of problems for me today which 
made me waste most of my working day today :-/

So please: Pay a bit more attention to where you write stuff.

Chris

Keeping an eye on Jenkins

[Christofer Dutz]

Hi all,

So, I just had a look at our Jenkins build as in the past we had several 
occasions where we were having failing builds for many days.
Turns out for the last 4 days Jenkins has been continuously failing because of 
a trivial RAT violation.

However, have we not been producing SNAPSHOT versions, and nobody noticed, 
because we decided to redirect build failure emails of Jenkins to a trash-list.

We really should discuss how to deal with failing builds.

I personally think redirecting failed build notifications is the worst way to 
deal with a flaky build.
Instead, we really should work on ironing out the issues that are randomly 
failing.

Having a stable and solid build should be of upmost importance, as only with 
this can we truly trust the CI.


Chris

[DISCUSS] Internal API?

[Christofer Dutz]

I could have sworn I mentioned this before, but I couldn’t find the thread 
here, so I think maybe I didn’t ;-)

For the past months have I been embedding IoTDB into applications. These 
applications allow configuring IoTDB at runtime and handle all sorts of special 
cases.
All of these are “standalone” instances with one config node and one data node 
… all running inside the same VM.

The one thing I was always quite unhappy about, is that even though everything 
is running inside one VM, I still needed to use the Session API, which makes 
absolutely no sense to me.
Here I have strongly typed data, that I need to serialize send via socket to 
the same VM and on the other side parse it again. In most cases this doesn’t 
matter, but as some of you might have read on LinkedIn am I currently working 
on making IoTDB work as a module in Ignition (MES System). Right now, I’m still 
using an externally run IoTDB instance, so using the session API is the way to 
go, but I am planning on also providing an embedded version, where we can run 
IoTDB inside Ignition. Given sometimes very extreme volume of requests sent to 
IoTDB, I would really love to reduce the amount of wasted resources.

So, what would be needed to have an Inner-VM API, that allows me to directly 
insert data into a data-node?

Chris

AW: [DISCUSS] Refactor the CLI to allow passing in STDIN, STDOUT and STDERR?

[Christofer Dutz]

JLine just takes the PrintStream and the InputStream … it shouldn’t matter 
where it’s coming from.

Chris

Von: Wang Critas 
Datum: Mittwoch, 10. Januar 2024 um 13:36
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Refactor the CLI to allow passing in STDIN, STDOUT and 
STDERR?
hi Chris,

only modify the in/out stream, not sure if it works well with jline

Best regards,
Xuan Wang


发件人: Christofer Dutz 
发送时间: 2024年1月10日星期三 20:29
收件人: dev@iotdb.apache.org 
主题: AW: [DISCUSS] Refactor the CLI to allow passing in STDIN, STDOUT and STDERR?

Hi Wang,

I am planning on doing this change, so it should have no affect on how it 
works, just where it gets it’s variables from ;-) But sure … testing is always 
good :-)

Chris

Von: Wang Critas 
Datum: Mittwoch, 10. Januar 2024 um 11:41
An: dev@iotdb.apache.org 
Betreff: 回复: [DISCUSS] Refactor the CLI to allow passing in STDIN, STDOUT and 
STDERR?
Hi

It looks good, but compatibility may require some testing. For example, various 
versions of Windows.

Xuan Wang


发件人: Christofer Dutz 
发送时间: 2024年1月10日 17:59
收件人: dev@iotdb.apache.org 
主题: [DISCUSS] Refactor the CLI to allow passing in STDIN, STDOUT and STDERR?

Hi all,

I would love to be able to provide a Web-Based CLI. Usually, this sort of thing 
works by forwarding stdin, stdout and stderr to a remote client. I’m working on 
building a Web CLI for IoTDB, which I could integrate into a browser 
application.

I could do this by globally redirecting System.stdout … but in that case I’d be 
forwarding also Log output (if that’s logged to STDOUT) … so I would love to 
refactor the CLI to allow passing in explicit streams for all of these and to 
simply have the “main” method initialize these:

public static void main(String[] args) throws ClassNotFoundException, 
IOException {
  InputStream in = System.in;
  PrintStream out = System.out;
  PrintStream err = System.err;
  mainInternal(args, in, out, err);
}

public static void mainInternal(String[] args, InputStream in, PrintStream out, 
PrintStream err) throws ClassNotFoundException, IOException {

In that case I could call mainInternal (or whatever we name it) and pass in 
dedicated streams, which I redirect to the browser.

In that case I we would need to refactor all of the usages of System.out, 
System.in etc to use these passed along streams instead.

What do you folks think?

Chris

AW: [DISCUSS] Refactor the CLI to allow passing in STDIN, STDOUT and STDERR?

[Christofer Dutz]

Hi Wang,

I am planning on doing this change, so it should have no affect on how it 
works, just where it gets it’s variables from ;-) But sure … testing is always 
good :-)

Chris

Von: Wang Critas 
Datum: Mittwoch, 10. Januar 2024 um 11:41
An: dev@iotdb.apache.org 
Betreff: 回复: [DISCUSS] Refactor the CLI to allow passing in STDIN, STDOUT and 
STDERR?
Hi

It looks good, but compatibility may require some testing. For example, various 
versions of Windows.

Xuan Wang


发件人: Christofer Dutz 
发送时间: 2024年1月10日 17:59
收件人: dev@iotdb.apache.org 
主题: [DISCUSS] Refactor the CLI to allow passing in STDIN, STDOUT and STDERR?

Hi all,

I would love to be able to provide a Web-Based CLI. Usually, this sort of thing 
works by forwarding stdin, stdout and stderr to a remote client. I’m working on 
building a Web CLI for IoTDB, which I could integrate into a browser 
application.

I could do this by globally redirecting System.stdout … but in that case I’d be 
forwarding also Log output (if that’s logged to STDOUT) … so I would love to 
refactor the CLI to allow passing in explicit streams for all of these and to 
simply have the “main” method initialize these:

public static void main(String[] args) throws ClassNotFoundException, 
IOException {
  InputStream in = System.in;
  PrintStream out = System.out;
  PrintStream err = System.err;
  mainInternal(args, in, out, err);
}

public static void mainInternal(String[] args, InputStream in, PrintStream out, 
PrintStream err) throws ClassNotFoundException, IOException {

In that case I could call mainInternal (or whatever we name it) and pass in 
dedicated streams, which I redirect to the browser.

In that case I we would need to refactor all of the usages of System.out, 
System.in etc to use these passed along streams instead.

What do you folks think?

Chris

[DISCUSS] Refactor the CLI to allow passing in STDIN, STDOUT and STDERR?

[Christofer Dutz]

Hi all,

I would love to be able to provide a Web-Based CLI. Usually, this sort of thing 
works by forwarding stdin, stdout and stderr to a remote client. I’m working on 
building a Web CLI for IoTDB, which I could integrate into a browser 
application.

I could do this by globally redirecting System.stdout … but in that case I’d be 
forwarding also Log output (if that’s logged to STDOUT) … so I would love to 
refactor the CLI to allow passing in explicit streams for all of these and to 
simply have the “main” method initialize these:

public static void main(String[] args) throws ClassNotFoundException, 
IOException {
  InputStream in = System.in;
  PrintStream out = System.out;
  PrintStream err = System.err;
  mainInternal(args, in, out, err);
}

public static void mainInternal(String[] args, InputStream in, PrintStream out, 
PrintStream err) throws ClassNotFoundException, IOException {

In that case I could call mainInternal (or whatever we name it) and pass in 
dedicated streams, which I redirect to the browser.

In that case I we would need to refactor all of the usages of System.out, 
System.in etc to use these passed along streams instead.

What do you folks think?

Chris

AW: [DISCUSS] Switch from Singletons to using a component-framework?

[Christofer Dutz]

So, here’s a list of some possible options:

  *   Spring (Obviously the most adopted option, but also possibly quite fat)
  *   Quarkus (Jakarta EE)
 *   Weld (Just the CDI dependency injection)
  *   Micronaut
  *   Guice (Every time I’ve heard it, people were migrating away from it)

https://avaje.io/inject/ sounded interesting, as it seems to use the same 
JSR-330 API as Spring and Quarkus use (I think) but doesn’t do dependency 
injection at runtime but generates code that does the weaving. With this it 
should be possible to see what’s happening. However, I’ve never tried it before.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 9. Januar 2024 um 10:18
An: dev@iotdb.apache.org 
Betreff: [DISCUSS] Switch from Singletons to using a component-framework?
Hi all,

So, I am currently fighting with something in IoTDB. It’s the usage of static 
variables all over the place.

The problem I’m having, is that I’m working on building an App that allows to 
run IoTDB in various forms of deployment. So, depending on the type of 
deployment, you can switch the location of where data is stored.

So, in my case I’ve started IoTDB, and it stores data in a directory “data-1” … 
so if I want to change this to “data-2” I need to stop IoTDB data-node and 
config-node, change the configuration and start config-node and data-node again.

However, we are using singletons everywhere … the problem now is that these 
don’t get re-initialized and they keep on pointing to “data-1”. Right now, I’m 
preparing a PR, where I can manually trigger a re-initialization of these 
singletons, by adding a static “reinitializeStatics()” method to them. But 
that’s just a super-ugly hack.

The much better solution would be to not need statics, but to either manually 
weave the components together or to use a component framework to do the weaving.

The benefit would also definitely be, that we could start running tests without 
forking processes and we could probably even re-use VMs over tests.

What do you folks think?

Chris

[DISCUSS] Switch from Singletons to using a component-framework?

[Christofer Dutz]

Hi all,

So, I am currently fighting with something in IoTDB. It’s the usage of static 
variables all over the place.

The problem I’m having, is that I’m working on building an App that allows to 
run IoTDB in various forms of deployment. So, depending on the type of 
deployment, you can switch the location of where data is stored.

So, in my case I’ve started IoTDB, and it stores data in a directory “data-1” … 
so if I want to change this to “data-2” I need to stop IoTDB data-node and 
config-node, change the configuration and start config-node and data-node again.

However, we are using singletons everywhere … the problem now is that these 
don’t get re-initialized and they keep on pointing to “data-1”. Right now, I’m 
preparing a PR, where I can manually trigger a re-initialization of these 
singletons, by adding a static “reinitializeStatics()” method to them. But 
that’s just a super-ugly hack.

The much better solution would be to not need statics, but to either manually 
weave the components together or to use a component framework to do the weaving.

The benefit would also definitely be, that we could start running tests without 
forking processes and we could probably even re-use VMs over tests.

What do you folks think?

Chris

AW: [VOTE] Migrate the default branch from "master" to "main"/"develop"

[Christofer Dutz]

Sorry folks … I have absolutely no idea why this landed in this list and in 
this thread … It’s just a thread I pinned to the top of my folder as we still 
haven’t moved from “master” to “develop” in IoTDB.

Chris


Von: Christofer Dutz 
Datum: Montag, 8. Januar 2024 um 09:24
An: dev@iotdb.apache.org 
Betreff: AW: [VOTE] Migrate the default branch from "master" to "main"/"develop"
Here an example of what Julian started doing:
https://github.com/apache/plc4x/blob/feature/plc4rs/protocols/tsfile/src/main/resources/protocols/tsfile/tsfile.mspec

Chris

Von: Christofer Dutz 
Datum: Mittwoch, 12. Juli 2023 um 14:12
An: dev@iotdb.apache.org 
Betreff: AW: [VOTE] Migrate the default branch from "master" to "main"/"develop"
So …

we voted, the results were in favor … when are we planning on renaming the 
branches?
I’d be happy to help.

I do think we should really get rid of the “master”.
Even if it’s probably way less controversial in China, we want to welcome 
people from
all over the world and we should therefore pay attention to their feelings too.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 4. Juli 2023 um 19:35
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Migrate the default branch from "master" to "main"/"develop"
S ... I guess we had almost 3 years now ;-)

On 2020/10/12 03:56:24 "Houliang Qi" wrote:
> Hi all,
> As the vote have passed more than 72 hours, the vote results are as following
>
>
> [D] Develop +7
> [M] Main +1
> [K] Keep master 0
>
>
> And  I’d like to start a new discussion about this:
>
>
> As someone mentioned  that there should be a develop branch and a release 
> branch. The develop branch is used to submit the latest development, and the 
> release branch is used for the functions that need to be released in the 
> latest release.
>
>
> At the same time, I hope that the development of cluster can also have two 
> branches: cluster_develop branch and cluster_release branch.
>
>
> The cluster_develop branch is used to merge the stand-alone version of the 
> develop branch code and the latest development of the cluster.
>
>
> The cluster_release branch is used to release some of the latest features. 
> Only the functions that need to be released are allowed to be merged into the 
> cluster_release branch, or to fix some bugs. Other newly developed functions 
> are not allowed to be merged into the cluster_release branch. After 
> cluster_release has been fully tested, it can be released.
>
>
> Regarding the latest release, I would like to check out a cluster_release 
> branch after the cluster_premerge branch merges into the master (develop), 
> and then the master branch merges into the cluster_new (cluster_develop) 
> branch.
>
>
> And I think the new functions do not have beed tested or need more than one 
> month to tested should be switch off when release the cluster version.
>
>
> Does anyone have some ideas about this?
>
>
> Thanks,
> ---
> Houliang Qi
> On 09/24/2020 02:38，Kevin A. McGrail wrote：
> I am +1 to rename it but don't have any good input on what would be a
> good name to use going forward.  I'll defer to others on that.
>
> On 9/22/2020 4:23 AM, Xiangdong Huang wrote:
> Hi,
>
> There is a movement to move the default branch from "master" to "main" or
> "develop", because of two reasons:
>
> - Many people around the world thought the word "master" has some 
> other meaning.
> - the word "master" can not clearly describe the branch's purpose. As we
> use the branch as our main working/developing branch, "main" or "develop"
> may be better.
>
> We had a discussion on private@ and I think it is time to start a vote in
> public.
>
> So, I'd like to call a formal vote for changing the default branch:
>
> - [M] main
> - [D] develop
> - [K] Keep "master"
>
> The vote will last at least 72 hours.
> The name who gets the most votes (and >= 3 votes) wins.
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
> --
> Kevin A. McGrail
> kmcgr...@apache.org
>
> Member, Apache Software Foundation
> Chair Emeritus Apache SpamAssassin Project
> https://www.linkedin.com/in/kmcgrail - 703.798.0171
>

AW: [VOTE] Migrate the default branch from "master" to "main"/"develop"

[Christofer Dutz]

Here an example of what Julian started doing:
https://github.com/apache/plc4x/blob/feature/plc4rs/protocols/tsfile/src/main/resources/protocols/tsfile/tsfile.mspec

Chris

Von: Christofer Dutz 
Datum: Mittwoch, 12. Juli 2023 um 14:12
An: dev@iotdb.apache.org 
Betreff: AW: [VOTE] Migrate the default branch from "master" to "main"/"develop"
So …

we voted, the results were in favor … when are we planning on renaming the 
branches?
I’d be happy to help.

I do think we should really get rid of the “master”.
Even if it’s probably way less controversial in China, we want to welcome 
people from
all over the world and we should therefore pay attention to their feelings too.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 4. Juli 2023 um 19:35
An: dev@iotdb.apache.org 
Betreff: Re: [VOTE] Migrate the default branch from "master" to "main"/"develop"
S ... I guess we had almost 3 years now ;-)

On 2020/10/12 03:56:24 "Houliang Qi" wrote:
> Hi all,
> As the vote have passed more than 72 hours, the vote results are as following
>
>
> [D] Develop +7
> [M] Main +1
> [K] Keep master 0
>
>
> And  I’d like to start a new discussion about this:
>
>
> As someone mentioned  that there should be a develop branch and a release 
> branch. The develop branch is used to submit the latest development, and the 
> release branch is used for the functions that need to be released in the 
> latest release.
>
>
> At the same time, I hope that the development of cluster can also have two 
> branches: cluster_develop branch and cluster_release branch.
>
>
> The cluster_develop branch is used to merge the stand-alone version of the 
> develop branch code and the latest development of the cluster.
>
>
> The cluster_release branch is used to release some of the latest features. 
> Only the functions that need to be released are allowed to be merged into the 
> cluster_release branch, or to fix some bugs. Other newly developed functions 
> are not allowed to be merged into the cluster_release branch. After 
> cluster_release has been fully tested, it can be released.
>
>
> Regarding the latest release, I would like to check out a cluster_release 
> branch after the cluster_premerge branch merges into the master (develop), 
> and then the master branch merges into the cluster_new (cluster_develop) 
> branch.
>
>
> And I think the new functions do not have beed tested or need more than one 
> month to tested should be switch off when release the cluster version.
>
>
> Does anyone have some ideas about this?
>
>
> Thanks,
> ---
> Houliang Qi
> On 09/24/2020 02:38，Kevin A. McGrail wrote：
> I am +1 to rename it but don't have any good input on what would be a
> good name to use going forward.  I'll defer to others on that.
>
> On 9/22/2020 4:23 AM, Xiangdong Huang wrote:
> Hi,
>
> There is a movement to move the default branch from "master" to "main" or
> "develop", because of two reasons:
>
> - Many people around the world thought the word "master" has some 
> other meaning.
> - the word "master" can not clearly describe the branch's purpose. As we
> use the branch as our main working/developing branch, "main" or "develop"
> may be better.
>
> We had a discussion on private@ and I think it is time to start a vote in
> public.
>
> So, I'd like to call a formal vote for changing the default branch:
>
> - [M] main
> - [D] develop
> - [K] Keep "master"
>
> The vote will last at least 72 hours.
> The name who gets the most votes (and >= 3 votes) wins.
>
> Best,
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
> 黄向东
> 清华大学 软件学院
>
> --
> Kevin A. McGrail
> kmcgr...@apache.org
>
> Member, Apache Software Foundation
> Chair Emeritus Apache SpamAssassin Project
> https://www.linkedin.com/in/kmcgrail - 703.798.0171
>

Re: PRs about tsfile module

[Christofer Dutz]

Generally that will definitely be discussed on the d...@tsfile.apache.org list 
;-)

Chris

Gesendet von Outlook für Android

From: Giorgio Zoppi 
Sent: Friday, January 5, 2024 5:18:55 PM
To: dev@iotdb.apache.org 
Subject: Re: PRs about tsfile module

Ok, i like it. I also suggest to include docusaurs for the format docs.
https://docusaurus.io/,
so we can use it. Also what is the plan for python/cpp/rust format
implementation?
BR,
Giorgio

Il giorno ven 5 gen 2024 alle ore 12:51 Yuan Tian 
ha scritto:

> Hi all,
>
> Since we've already set up our tsfile repo, so if you want to make some
> changes to common-api and tsfile submodule in IoTDB repo, you also need to
> submit another pr for tsfile repo.
>
>
> Best regards,
> 
> Yuan Tian
>


--
Life is a chess game - Anonymous.

AW: PRs about tsfile module

[Christofer Dutz]

But submodules can be hell when releasing software … so I would actually be in 
favor of not using Git submodules.

Chris

Von: 刘旭鑫 
Datum: Freitag, 5. Januar 2024 um 14:14
An: dev@iotdb.apache.org 
Betreff: Re: PRs about tsfile module
Moreover, submodules also support allowing different branches of IoTDB to track 
corresponding branches of TsFile. For example, the master branch of IoTDB can 
refer to the master branch of TsFile, and the rel/1.3 branch of IoTDB can refer 
to the rel/1.3 branch of TsFile. When switching branches in the IoTDB 
repository, the TsFile branch will also switch accordingly. We can even have 
IoTDB directly reference a specific fixed commit of TsFile to ensure the 
stability of a branch that is no longer being updated. I believe this approach 
makes management more convenient and flexible.

> 2024年1月5日 21:03，刘旭鑫  写道：
>
> Hi, I suggest setting TsFile as a submodule of IoTDB. This would allow the 
> 'tsfile' directory to directly reference the TsFile repository. For example, 
> LevelDB provides a reference to GoogleTest in this manner: 
> https://github.com/google/leveldb/tree/main/third_party.
> The specific steps are:
>
> Add the submodule: git submodule add tsfile 
> https://github.com/apache/tsfile.git
> Initialize the submodule: git submodule init
> Update the submodule: git submodule update
>
> When cloning the IoTDB repository, use the following command to clone the 
> submodule as well:
>
> git clone --recurse-submodules https://github.com/apache/iotdb.git
>
> The advantage of this approach is that whenever the TsFile code is updated, 
> we can pull the latest code through Git. If managed through a POM, it's not 
> easy to package the code of TsFile that hasn't been released yet.
>
> ――
> Xuxin Liu
>
>> 2024年1月5日 20:50，Yuan Tian  写道：
>>
>> Hi all,
>>
>> Since we've already set up our tsfile repo, so if you want to make some
>> changes to common-api and tsfile submodule in IoTDB repo, you also need to
>> submit another pr for tsfile repo.
>>
>>
>> Best regards,
>> 
>> Yuan Tian
>

Interest in an Apache IoT Track at EclipseCon EU in Mainz (Germany) later this year?

[Christofer Dutz]

Hi folks,

I was contacted by some of my friends in the IoT Space at the Eclipse 
Foundation. 

They are planning this year's EclipseCon to work closer together with the other 
foundations in this space and proposed an Apache IoT Track in paralel to the 
EclipseCon. This will be happening in Mainz in Germany I think 4 weeks before 
Community Over Code NA.

I promised to reach out to the projects I know at Apache that count to the IoT 
space.

I really liked the idea of networking with the other foundation's projects as I 
think this is something happening far too little.

So would you folks be interested in submitting talks and attending such a 
conference?

Also the CFP for CoC EU in Bratislava is only open for a few more days and so 
far the IoT related submissions have been ... let's say ... less than I was 
hoping for. 

So if you folks are planning on submitting: Do your track-chair (me) a favor 
and don't submit on the last day ... it's always a hastle to whip up a schedule 
if you can't do it in an orderly way, but get "shot by what's there" on the 
last day.

Looking forward to feedback,

Chris

Is there a reason we're not deploying "library-udf"?

[Christofer Dutz]

Hi all,

while setting up Apache TsFile, I noticed, that we’re excluding “library-udf” 
from SNAPSHOT deployment on Jenkins. Is there a reason for this?

(Always to leave a comment in the code on stuff like that)

Chris

Category X dependency in our releases

[Christofer Dutz]

Hi all,

while doing the TsFile split and going through the dependencies, I stumbled 
over one dependency in the commons-api module:


org.openjdk.jol
jol-core


Which actually is the only dependency of that module. But the “opendjk” got me 
thinking: Isn’t everything with OpenJDK GPL licensed? And indeed. This library 
is licensed by the GPL2 license + class path exception.

We are not allowed to have this type of dependencies in our project, as it’s 
category X rated:
https://www.apache.org/legal/resolved.html

There it states clearly:

  *   GNU GPL 1, 2, 3
 *   Special exceptions to the GNU GPL (e.g. GNU Classpath) unless 
otherwise permitted elsewhere on this page.
And the only mention on that page is JavaDoc.

So, I think we should remove that dependency as quickly as possible. In both 
IoTDB and TsFile.

Chris

AW: Apache TsFile

[Christofer Dutz]

So far, only Jialin Qiao and me have signed up for the tsfile private list … 
please do so as soon as possible, so we can get started.

And when subscribing: Please use an email address you have on file at apache. 
https://whimsy.apache.org/roster/committer/{yourid}<https://whimsy.apache.org/roster/committer/%7byourid%7d>
Either in the “Email forwarded to ” or the “Email addresses (alt)” area.

I did get several subscription requests, as I’m currently the only moderator, 
but the email addresses used there I had no way to see who’s actually signing 
up, so I didn’t approve them.

If you want to do me a huge favor, then please sign up with the email you have 
in “Email forwarded to”.


Chris

Von: Christofer Dutz 
Datum: Freitag, 17. November 2023 um 18:26
An: dev@iotdb.apache.org 
Betreff: AW: Apache TsFile
Hi all,

so now the lists should be active … please subscribe.

Chris


Von: Christofer Dutz 
Datum: Freitag, 17. November 2023 um 16:37
An: dev@iotdb.apache.org 
Betreff: AW: Apache TsFile
Ok ,

there seems to be a DNS issue (it’s always a DNS issue ;-) ) … infra is working 
on it … I’ll tell you when you can try signing up for the mailinglists.

But theoretically Github should already work:
https://github.com/apache/tsfile/

Chris


Von: Christofer Dutz 
Datum: Freitag, 17. November 2023 um 09:32
An: dev@iotdb.apache.org 
Betreff: Re: Apache TsFile
The lists and the repository have been created. Please sign up for dev@ and 
private@

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>
____
From: Christofer Dutz 
Sent: Thursday, November 16, 2023 12:09:49 PM
To: dev@iotdb.apache.org 
Subject: Re: Apache TsFile

So I was told that it might take 24h for creating the lists and as soon as they 
are created, I can create the repo.

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Yuan Tian 
Sent: Thursday, November 16, 2023 11:32:20 AM
To: dev@iotdb.apache.org 
Subject: Re: Apache TsFile

Wow, that's really good news! Thanks Chris!


Best,
-
Yuan Tian

On Thu, Nov 16, 2023 at 4:52 PM Christofer Dutz 
wrote:

> Hi all,
>
> so at yesterdays board meeting the Apache TsFile TLP proposal was
> accepted. I am currently requesting things to be created. The mailinglists:
> dev, commits and private will be the first things that need to be created
> and I initiated that and need to wait for them being created.
>
> As soon as that’s done I’ll request the git repo for the project.
>
> I’ll keep you posted on when it makes sense to sign up to the mailinglists.
>
>
> Chris
>

Re: Release schedule

[Christofer Dutz]

Well it's a bit more than that... Because once we change the packages, we also 
need to change some dependencies to versions using Jakarta...

I don't expect it to be super difficult, but I do expect it to be more work 
than just changing the package names.

Chris

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Jialin Qiao 
Sent: Sunday, November 19, 2023 5:34:45 AM
To: dev@iotdb.apache.org 
Subject: Re: Release schedule

Hi,

If I understand correctly, this is just a change of package namespace.
So users may need to upgrade their imports.
We could do this in 1.3.1, which will be checkout at about 2023.12.30.

Thanks,
―
Jialin Qiao
Apache IoTDB PMC

Christofer Dutz  于2023年11月16日周四 14:11写道：
>
> About the compatability.
>
> Most apache projects have finished the migration from Javax to Jakarta. I 
> think we should do this as soon as possible, as we are cutting us off from a 
> number of dependency updates.
>
> This change however is a breaking change.
>
> I was going to propose to do that for a first release of 1.3.0, but as that 
> is now officially already cut, when do we want to do that?
>
> Chris
>
> Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>
> 
> From: Jialin Qiao 
> Sent: Wednesday, November 15, 2023 7:17:40 AM
> To: dev@iotdb.apache.org 
> Subject: Re: Release schedule
>
> Hi, sorry for a typo:
>
> Then checkout rc/1.3.1 from master at 2023-12-30, and release 1.3.1
> after 2024-1-30.
>
> Thanks,
> Jialin Qiao
>
> On 2023/11/15 05:46:11 Jialin Qiao wrote:
> > Hi,
> >
> > I come to share about our release schedule.
> >
> > Currently, we are all working on master branch, and checkout a rc
> > branch from master for release.
> > So it is essential to ensure that the master branch remains compatible
> > with the previous version at all times.
> >
> > The release will be done every two months.
> >
> > As we checkout rc/1.3.0 from master at 2023-10-30, and will release
> > 1.3.0 after 2023-11-30.
> > Then checkout rc/1.3.1 from master at 2023-12-30, and release 1.3.0
> > after 2024-1-30.
> > ..
> >
> >
> > Thanks,
> > ―
> > Jialin Qiao
> > Apache IoTDB PMC
> >

AW: Apache TsFile

[Christofer Dutz]

Hi all,

so now the lists should be active … please subscribe.

Chris


Von: Christofer Dutz 
Datum: Freitag, 17. November 2023 um 16:37
An: dev@iotdb.apache.org 
Betreff: AW: Apache TsFile
Ok ,

there seems to be a DNS issue (it’s always a DNS issue ;-) ) … infra is working 
on it … I’ll tell you when you can try signing up for the mailinglists.

But theoretically Github should already work:
https://github.com/apache/tsfile/

Chris


Von: Christofer Dutz 
Datum: Freitag, 17. November 2023 um 09:32
An: dev@iotdb.apache.org 
Betreff: Re: Apache TsFile
The lists and the repository have been created. Please sign up for dev@ and 
private@

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Christofer Dutz 
Sent: Thursday, November 16, 2023 12:09:49 PM
To: dev@iotdb.apache.org 
Subject: Re: Apache TsFile

So I was told that it might take 24h for creating the lists and as soon as they 
are created, I can create the repo.

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Yuan Tian 
Sent: Thursday, November 16, 2023 11:32:20 AM
To: dev@iotdb.apache.org 
Subject: Re: Apache TsFile

Wow, that's really good news! Thanks Chris!


Best,
-
Yuan Tian

On Thu, Nov 16, 2023 at 4:52 PM Christofer Dutz 
wrote:

> Hi all,
>
> so at yesterdays board meeting the Apache TsFile TLP proposal was
> accepted. I am currently requesting things to be created. The mailinglists:
> dev, commits and private will be the first things that need to be created
> and I initiated that and need to wait for them being created.
>
> As soon as that’s done I’ll request the git repo for the project.
>
> I’ll keep you posted on when it makes sense to sign up to the mailinglists.
>
>
> Chris
>

AW: Apache TsFile

[Christofer Dutz]

Ok ,

there seems to be a DNS issue (it’s always a DNS issue ;-) ) … infra is working 
on it … I’ll tell you when you can try signing up for the mailinglists.

But theoretically Github should already work:
https://github.com/apache/tsfile/

Chris


Von: Christofer Dutz 
Datum: Freitag, 17. November 2023 um 09:32
An: dev@iotdb.apache.org 
Betreff: Re: Apache TsFile
The lists and the repository have been created. Please sign up for dev@ and 
private@

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Christofer Dutz 
Sent: Thursday, November 16, 2023 12:09:49 PM
To: dev@iotdb.apache.org 
Subject: Re: Apache TsFile

So I was told that it might take 24h for creating the lists and as soon as they 
are created, I can create the repo.

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Yuan Tian 
Sent: Thursday, November 16, 2023 11:32:20 AM
To: dev@iotdb.apache.org 
Subject: Re: Apache TsFile

Wow, that's really good news! Thanks Chris!


Best,
-
Yuan Tian

On Thu, Nov 16, 2023 at 4:52 PM Christofer Dutz 
wrote:

> Hi all,
>
> so at yesterdays board meeting the Apache TsFile TLP proposal was
> accepted. I am currently requesting things to be created. The mailinglists:
> dev, commits and private will be the first things that need to be created
> and I initiated that and need to wait for them being created.
>
> As soon as that’s done I’ll request the git repo for the project.
>
> I’ll keep you posted on when it makes sense to sign up to the mailinglists.
>
>
> Chris
>

Re: Apache TsFile

[Christofer Dutz]

The lists and the repository have been created. Please sign up for dev@ and 
private@

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Christofer Dutz 
Sent: Thursday, November 16, 2023 12:09:49 PM
To: dev@iotdb.apache.org 
Subject: Re: Apache TsFile

So I was told that it might take 24h for creating the lists and as soon as they 
are created, I can create the repo.

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Yuan Tian 
Sent: Thursday, November 16, 2023 11:32:20 AM
To: dev@iotdb.apache.org 
Subject: Re: Apache TsFile

Wow, that's really good news! Thanks Chris!


Best,
-
Yuan Tian

On Thu, Nov 16, 2023 at 4:52 PM Christofer Dutz 
wrote:

> Hi all,
>
> so at yesterdays board meeting the Apache TsFile TLP proposal was
> accepted. I am currently requesting things to be created. The mailinglists:
> dev, commits and private will be the first things that need to be created
> and I initiated that and need to wait for them being created.
>
> As soon as that’s done I’ll request the git repo for the project.
>
> I’ll keep you posted on when it makes sense to sign up to the mailinglists.
>
>
> Chris
>

Re: Apache TsFile

[Christofer Dutz]

So I was told that it might take 24h for creating the lists and as soon as they 
are created, I can create the repo.

Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>

From: Yuan Tian 
Sent: Thursday, November 16, 2023 11:32:20 AM
To: dev@iotdb.apache.org 
Subject: Re: Apache TsFile

Wow, that's really good news! Thanks Chris!


Best,
-
Yuan Tian

On Thu, Nov 16, 2023 at 4:52 PM Christofer Dutz 
wrote:

> Hi all,
>
> so at yesterdays board meeting the Apache TsFile TLP proposal was
> accepted. I am currently requesting things to be created. The mailinglists:
> dev, commits and private will be the first things that need to be created
> and I initiated that and need to wait for them being created.
>
> As soon as that’s done I’ll request the git repo for the project.
>
> I’ll keep you posted on when it makes sense to sign up to the mailinglists.
>
>
> Chris
>

Apache TsFile

[Christofer Dutz]

Hi all,

so at yesterdays board meeting the Apache TsFile TLP proposal was accepted. I 
am currently requesting things to be created. The mailinglists: dev, commits 
and private will be the first things that need to be created and I initiated 
that and need to wait for them being created.

As soon as that’s done I’ll request the git repo for the project.

I’ll keep you posted on when it makes sense to sign up to the mailinglists.


Chris

Re: Release schedule

[Christofer Dutz]

About the compatability.

Most apache projects have finished the migration from Javax to Jakarta. I think 
we should do this as soon as possible, as we are cutting us off from a number 
of dependency updates.

This change however is a breaking change.

I was going to propose to do that for a first release of 1.3.0, but as that is 
now officially already cut, when do we want to do that?

Chris

Gesendet von Outlook für Android

From: Jialin Qiao 
Sent: Wednesday, November 15, 2023 7:17:40 AM
To: dev@iotdb.apache.org 
Subject: Re: Release schedule

Hi, sorry for a typo:

Then checkout rc/1.3.1 from master at 2023-12-30, and release 1.3.1
after 2024-1-30.

Thanks,
Jialin Qiao

On 2023/11/15 05:46:11 Jialin Qiao wrote:
> Hi,
>
> I come to share about our release schedule.
>
> Currently, we are all working on master branch, and checkout a rc
> branch from master for release.
> So it is essential to ensure that the master branch remains compatible
> with the previous version at all times.
>
> The release will be done every two months.
>
> As we checkout rc/1.3.0 from master at 2023-10-30, and will release
> 1.3.0 after 2023-11-30.
> Then checkout rc/1.3.1 from master at 2023-12-30, and release 1.3.0
> after 2024-1-30.
> ..
>
>
> Thanks,
> —
> Jialin Qiao
> Apache IoTDB PMC
>

What do we have the CI/CD Servers for?

[Christofer Dutz]

Hi folks,

So, for the last weeks I have literally been working my ass off with 80+ hours 
per weeks, so we have something to show at the SPS in Nürnberg.

Now the last thing I needed to finish this task was to have one change built 
and SNAPSHOTs with that being deployed, however I see no way on earth I’ll be 
able to do this.

Jenkins (the one deploying the SNAPSHOTS) has been red for over 8 days now and 
in GitHub I’ve been going back in the history of the commits and even after a 
full month have I not seen a green build.

I think if we don’t pay attention to what the CI/CD servers are trying to tell 
us, we should probably turn them off.

I remember that someone proposed to redirect the CI/CD emails to another list, 
because they were annoying. Redirecting them is the worst thing we could have 
done, and I think I pointed that out then. The real approach would have been to 
make sure the build doesn’t fail so often. Fix flaky tests, make the build more 
robust.

Well guess I’ll be faking a working application by hiding my mac-book in the 
shelf under the demo.

Just had to get this off my chest, :-(

Chris

AW: [DISCUSS] Apache TsFile Proposal

[Christofer Dutz]

Ok … so after pinging trademarks on the state of the name-search, they told me 
they were still missing something. Hopefully after adding that, they’ll be able 
to start their work.

Chris

Von: Jialin Qiao 
Datum: Freitag, 27. Oktober 2023 um 13:44
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Apache TsFile Proposal
Hi, Looks great! +1 for name search
―
Jialin Qiao
Apache IoTDB PMC

Christofer Dutz  于2023年10月26日周四 17:19写道：
>
> FYI,
>
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-214
>
> Chris
>
> Von: Xiangdong Huang 
> Datum: Donnerstag, 26. Oktober 2023 um 04:23
> An: dev@iotdb.apache.org 
> Betreff: Re: [DISCUSS] Apache TsFile Proposal
> Hi,
>
> The content looks fine.   +1 for name-search.
> Best,
> -------
> Xiangdong Huang
>
>
>
> Christofer Dutz  于2023年10月25日周三 22:24写道：
>
> >
> > More feedback from the board was that the description is pretty long and
> > having it shorted would leave us more wiggle-room in the future.
> > Also, should we formally have passed a name-search … if you’re ok with
> > this, I would start the Name-Search.
> >
> > So that would result in the following resolution:
> >
> >   WHEREAS, the Board of Directors deems it to be in the best interests
> > of
> >the Foundation and consistent with the Foundation's purpose to
> >establish a Project Management Committee charged with the creation
> > and
> >maintenance of open-source software, for distribution at no charge
> > to
> >the public, related to a columnar storage file format designed for
> > time
> >series data.
> >
> >NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> >(PMC), to be known as the "Apache TsFile Project", be and hereby is
> >established pursuant to Bylaws of the Foundation; and be it further
> >
> >RESOLVED, that the Apache TsFile be and hereby is responsible for
> >the creation and maintenance of software related to a columnar
> > storage
> >file format designed for time series data.
> >
> >RESOLVED, that the office of "Vice President, Apache TsFile" be and
> >hereby is created, the person holding such office to serve at the
> >direction of the Board of Directors as the chair of the Apache
> >TsFile Project, and to have primary responsibility for management of
> >the projects within the scope of responsibility of the Apache TsFile
> >Project; and be it further
> >
> >RESOLVED, that the persons listed immediately below be and hereby
> > are
> >appointed to serve as the initial members of the Apache TsFile
> >Project:
> >
> >- Yuan Tian (jackietien at apache dot org)
> >- Chao Wang (wangchao316 at apache dot org)
> >- Christofer Dutz (cdutz at apache dot org)
> >- Jinrui Zhang (xingtanzjr at apache dot org)
> >- Steve Yurong Su (rong at apache dot org)
> >- Xinyu Tan (tanxinyu at apache dot org)
> >- Haohan Hou (haonan at apache dot org)
> >- Gaofei Cao (gaogaofei at apache dot org)
> >- Jialin Qiao (qiaojialin at apache dot org)
> >- Kun Liu (liukun at apache dot org)
> >- Houliang Qi (neuyilan at apache dot org)
> >- Xiangdong Huang (hxd at apache dot org)
> >- Chao Wang (chaow at apache dot org)
> >- Jianmin Wang (jimwang at apache dot org)
> >- Tian Jiang (jiangtian at apache dot org）
> >- Xinyi Zhao (zhaoxinyi at apache dot org)
> >- Shuo Zhang (shuozhagn at apache dot org)
> >
> >NOW, THEREFORE, BE IT FURTHER RESOLVED, that {name of the chair} be
> >appointed to the office of Vice President, Apache TsFile, to serve
> > in
> >accordance with and subject to the direction of the Board of
> > Directors
> >and the Bylaws of the Foundation until death, resignation,
> > retirement,
> >removal or disqualification, or until a successor is appointed; and
> > be
> >it further
> >
> >RESOLVED, that the Apache TsFile Project be and hereby is tasked
> > with
> >the migration and rationalization of the Apache IoTDB TsFile
> > subproject;
> >and be it further
> >
> >RESOLVED, that all responsibilities pertaining to the Apache IoTDB
> >TsFile subproject encumbered upon the Apache IoTDB PMC are hereafter
> >   

AW: [DISCUSS] Apache TsFile Proposal

[Christofer Dutz]

FYI,

https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-214

Chris

Von: Xiangdong Huang 
Datum: Donnerstag, 26. Oktober 2023 um 04:23
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Apache TsFile Proposal
Hi,

The content looks fine.   +1 for name-search.
Best,
---
Xiangdong Huang



Christofer Dutz  于2023年10月25日周三 22:24写道：

>
> More feedback from the board was that the description is pretty long and
> having it shorted would leave us more wiggle-room in the future.
> Also, should we formally have passed a name-search … if you’re ok with
> this, I would start the Name-Search.
>
> So that would result in the following resolution:
>
>   WHEREAS, the Board of Directors deems it to be in the best interests
> of
>the Foundation and consistent with the Foundation's purpose to
>establish a Project Management Committee charged with the creation
> and
>maintenance of open-source software, for distribution at no charge
> to
>the public, related to a columnar storage file format designed for
> time
>series data.
>
>NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
>(PMC), to be known as the "Apache TsFile Project", be and hereby is
>established pursuant to Bylaws of the Foundation; and be it further
>
>RESOLVED, that the Apache TsFile be and hereby is responsible for
>the creation and maintenance of software related to a columnar
> storage
>file format designed for time series data.
>
>RESOLVED, that the office of "Vice President, Apache TsFile" be and
>hereby is created, the person holding such office to serve at the
>direction of the Board of Directors as the chair of the Apache
>TsFile Project, and to have primary responsibility for management of
>the projects within the scope of responsibility of the Apache TsFile
>Project; and be it further
>
>RESOLVED, that the persons listed immediately below be and hereby
> are
>appointed to serve as the initial members of the Apache TsFile
>Project:
>
>- Yuan Tian (jackietien at apache dot org)
>- Chao Wang (wangchao316 at apache dot org)
>- Christofer Dutz (cdutz at apache dot org)
>- Jinrui Zhang (xingtanzjr at apache dot org)
>- Steve Yurong Su (rong at apache dot org)
>- Xinyu Tan (tanxinyu at apache dot org)
>- Haohan Hou (haonan at apache dot org)
>- Gaofei Cao (gaogaofei at apache dot org)
>- Jialin Qiao (qiaojialin at apache dot org)
>- Kun Liu (liukun at apache dot org)
>- Houliang Qi (neuyilan at apache dot org)
>- Xiangdong Huang (hxd at apache dot org)
>- Chao Wang (chaow at apache dot org)
>- Jianmin Wang (jimwang at apache dot org)
>- Tian Jiang (jiangtian at apache dot org）
>- Xinyi Zhao (zhaoxinyi at apache dot org)
>- Shuo Zhang (shuozhagn at apache dot org)
>
>NOW, THEREFORE, BE IT FURTHER RESOLVED, that {name of the chair} be
>appointed to the office of Vice President, Apache TsFile, to serve
> in
>accordance with and subject to the direction of the Board of
> Directors
>and the Bylaws of the Foundation until death, resignation,
> retirement,
>removal or disqualification, or until a successor is appointed; and
> be
>it further
>
>RESOLVED, that the Apache TsFile Project be and hereby is tasked
> with
>the migration and rationalization of the Apache IoTDB TsFile
> subproject;
>and be it further
>
>RESOLVED, that all responsibilities pertaining to the Apache IoTDB
>TsFile subproject encumbered upon the Apache IoTDB PMC are hereafter
>discharged.
>
>
>
>
>
> Von: Christofer Dutz 
> Datum: Mittwoch, 25. Oktober 2023 um 15:03
> An: dev@iotdb.apache.org 
> Betreff: AW: [DISCUSS] Apache TsFile Proposal
> So, it seems that this document is based on an incubation proposal … sorry
> … I just reviewed it on a content-level.
> As we’re going straight TLP this should look quite a bit different. So
> I’ll try to whip up a version:
>
>   WHEREAS, the Board of Directors deems it to be in the best interests
> of
>the Foundation and consistent with the Foundation's purpose to
>establish a Project Management Committee charged with the creation
> and
>maintenance of open-source software, for distribution at no charge
> to
>the public, related to a columnar storage file format designed for
> time
>series data, which supports efficient compression, high throughput
&g

AW: [DISCUSS] Apache TsFile Proposal

[Christofer Dutz]

More feedback from the board was that the description is pretty long and having 
it shorted would leave us more wiggle-room in the future.
Also, should we formally have passed a name-search … if you’re ok with this, I 
would start the Name-Search.

So that would result in the following resolution:

  WHEREAS, the Board of Directors deems it to be in the best interests of
   the Foundation and consistent with the Foundation's purpose to
   establish a Project Management Committee charged with the creation and
   maintenance of open-source software, for distribution at no charge to
   the public, related to a columnar storage file format designed for time
   series data.

   NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
   (PMC), to be known as the "Apache TsFile Project", be and hereby is
   established pursuant to Bylaws of the Foundation; and be it further

   RESOLVED, that the Apache TsFile be and hereby is responsible for
   the creation and maintenance of software related to a columnar storage
   file format designed for time series data.

   RESOLVED, that the office of "Vice President, Apache TsFile" be and
   hereby is created, the person holding such office to serve at the
   direction of the Board of Directors as the chair of the Apache
   TsFile Project, and to have primary responsibility for management of
   the projects within the scope of responsibility of the Apache TsFile
   Project; and be it further

   RESOLVED, that the persons listed immediately below be and hereby are
   appointed to serve as the initial members of the Apache TsFile
   Project:

   - Yuan Tian (jackietien at apache dot org)
   - Chao Wang (wangchao316 at apache dot org)
   - Christofer Dutz (cdutz at apache dot org)
   - Jinrui Zhang (xingtanzjr at apache dot org)
   - Steve Yurong Su (rong at apache dot org)
   - Xinyu Tan (tanxinyu at apache dot org)
   - Haohan Hou (haonan at apache dot org)
   - Gaofei Cao (gaogaofei at apache dot org)
   - Jialin Qiao (qiaojialin at apache dot org)
   - Kun Liu (liukun at apache dot org)
   - Houliang Qi (neuyilan at apache dot org)
   - Xiangdong Huang (hxd at apache dot org)
   - Chao Wang (chaow at apache dot org)
   - Jianmin Wang (jimwang at apache dot org)
   - Tian Jiang (jiangtian at apache dot org）
   - Xinyi Zhao (zhaoxinyi at apache dot org)
   - Shuo Zhang (shuozhagn at apache dot org)

   NOW, THEREFORE, BE IT FURTHER RESOLVED, that {name of the chair} be
   appointed to the office of Vice President, Apache TsFile, to serve in
   accordance with and subject to the direction of the Board of Directors
   and the Bylaws of the Foundation until death, resignation, retirement,
   removal or disqualification, or until a successor is appointed; and be
   it further

   RESOLVED, that the Apache TsFile Project be and hereby is tasked with
   the migration and rationalization of the Apache IoTDB TsFile subproject;
   and be it further

   RESOLVED, that all responsibilities pertaining to the Apache IoTDB
   TsFile subproject encumbered upon the Apache IoTDB PMC are hereafter
   discharged.





Von: Christofer Dutz 
Datum: Mittwoch, 25. Oktober 2023 um 15:03
An: dev@iotdb.apache.org 
Betreff: AW: [DISCUSS] Apache TsFile Proposal
So, it seems that this document is based on an incubation proposal … sorry … I 
just reviewed it on a content-level.
As we’re going straight TLP this should look quite a bit different. So I’ll try 
to whip up a version:

  WHEREAS, the Board of Directors deems it to be in the best interests of
   the Foundation and consistent with the Foundation's purpose to
   establish a Project Management Committee charged with the creation and
   maintenance of open-source software, for distribution at no charge to
   the public, related to a columnar storage file format designed for time
   series data, which supports efficient compression, high throughput of
   read and write, and compatibility with various frameworks, such as Spark
   and Flink. It is easy to integrate TsFile into IoT big data processing
   frameworks.

   NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
   (PMC), to be known as the "Apache TsFile Project", be and hereby is
   established pursuant to Bylaws of the Foundation; and be it further

   RESOLVED, that the Apache TsFile be and hereby is responsible for
   the creation and maintenance of software related to a columnar storage
   file format designed for time series data, which supports efficient
   compression, high throughput of read and write, and compatibility with
   various frameworks, such as Spark and Flink. It is easy to integrate 
TsFile
   into IoT big data processing frameworks.

  

AW: [DISCUSS] Apache TsFile Proposal

[Christofer Dutz]

So, it seems that this document is based on an incubation proposal … sorry … I 
just reviewed it on a content-level.
As we’re going straight TLP this should look quite a bit different. So I’ll try 
to whip up a version:

  WHEREAS, the Board of Directors deems it to be in the best interests of
   the Foundation and consistent with the Foundation's purpose to
   establish a Project Management Committee charged with the creation and
   maintenance of open-source software, for distribution at no charge to
   the public, related to a columnar storage file format designed for time
   series data, which supports efficient compression, high throughput of
   read and write, and compatibility with various frameworks, such as Spark
   and Flink. It is easy to integrate TsFile into IoT big data processing
   frameworks.

   NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
   (PMC), to be known as the "Apache TsFile Project", be and hereby is
   established pursuant to Bylaws of the Foundation; and be it further

   RESOLVED, that the Apache TsFile be and hereby is responsible for
   the creation and maintenance of software related to a columnar storage
   file format designed for time series data, which supports efficient
   compression, high throughput of read and write, and compatibility with
   various frameworks, such as Spark and Flink. It is easy to integrate 
TsFile
   into IoT big data processing frameworks.

   RESOLVED, that the office of "Vice President, Apache TsFile" be and
   hereby is created, the person holding such office to serve at the
   direction of the Board of Directors as the chair of the Apache
   TsFile Project, and to have primary responsibility for management of
   the projects within the scope of responsibility of the Apache TsFile
   Project; and be it further

   RESOLVED, that the persons listed immediately below be and hereby are
   appointed to serve as the initial members of the Apache TsFile
   Project:

   - Yuan Tian (jackietien at apache dot org)
   - Chao Wang (wangchao316 at apache dot org)
   - Christofer Dutz (cdutz at apache dot org)
   - Jinrui Zhang (xingtanzjr at apache dot org)
   - Steve Yurong Su (rong at apache dot org)
   - Xinyu Tan (tanxinyu at apache dot org)
   - Haohan Hou (haonan at apache dot org)
   - Gaofei Cao (gaogaofei at apache dot org)
   - Jialin Qiao (qiaojialin at apache dot org)
   - Kun Liu (liukun at apache dot org)
   - Houliang Qi (neuyilan at apache dot org)
   - Xiangdong Huang (hxd at apache dot org)
   - Chao Wang (chaow at apache dot org)
   - Jianmin Wang (jimwang at apache dot org)
   - Tian Jiang (jiangtian at apache dot org）
   - Xinyi Zhao (zhaoxinyi at apache dot org)
   - Shuo Zhang (shuozhagn at apache dot org)

   NOW, THEREFORE, BE IT FURTHER RESOLVED, that {name of the chair} be
   appointed to the office of Vice President, Apache TsFile, to serve in
   accordance with and subject to the direction of the Board of Directors
   and the Bylaws of the Foundation until death, resignation, retirement,
   removal or disqualification, or until a successor is appointed; and be
   it further

   RESOLVED, that the Apache TsFile Project be and hereby is tasked with
   the migration and rationalization of the Apache IoTDB TsFile subproject;
   and be it further

   RESOLVED, that all responsibilities pertaining to the Apache IoTDB
   TsFile subproject encumbered upon the Apache IoTDB PMC are hereafter
   discharged.







Von: Christofer Dutz 
Datum: Mittwoch, 25. Oktober 2023 um 12:24
An: dev@iotdb.apache.org 
Betreff: AW: [DISCUSS] Apache TsFile Proposal
So, I gave the text a little run and tweaked a few parts (I do hope I based 
that on the latest version discussed here):



# Abstract
TsFile is a columnar storage file format designed for time series data, which 
supports efficient compression, high throughput of read and write, and 
compatibility with various frameworks, such as Spark and Flink. It is easy to 
integrate TsFile into IoT big data processing frameworks.

# Proposal
TsFile is used for managing time series data. Although it is firstly used 
inside IoTDB, many users and companies use TsFile directly as a time series 
data management solution independently. Besides, there is a growing requirement 
for Multi-Language Support of TsFile implementations, such as C++, Go and Rust.

Apache IoTDB community hereby submit this proposal of TsFile as an independent 
Apache project. This proposal outlines the key features and benefits of TsFile, 
along with the integration plan and the need for multi-language support.

# Background
Time series data is becoming increasingly important in a wide range of 
applications, including IoT, intelligent control, finance, log analysis, and

AW: [DISCUSS] Apache TsFile Proposal

[Christofer Dutz]

So, I gave the text a little run and tweaked a few parts (I do hope I based 
that on the latest version discussed here):



# Abstract
TsFile is a columnar storage file format designed for time series data, which 
supports efficient compression, high throughput of read and write, and 
compatibility with various frameworks, such as Spark and Flink. It is easy to 
integrate TsFile into IoT big data processing frameworks.

# Proposal
TsFile is used for managing time series data. Although it is firstly used 
inside IoTDB, many users and companies use TsFile directly as a time series 
data management solution independently. Besides, there is a growing requirement 
for Multi-Language Support of TsFile implementations, such as C++, Go and Rust.

Apache IoTDB community hereby submit this proposal of TsFile as an independent 
Apache project. This proposal outlines the key features and benefits of TsFile, 
along with the integration plan and the need for multi-language support.

# Background
Time series data is becoming increasingly important in a wide range of 
applications, including IoT, intelligent control, finance, log analysis, and 
monitoring systems.
TsFile has been developed by the Apache IoTDB community in Java and is 
currently located inside the Apache IoTDB code repository. Users could store 
time series data using TsFile, then read and analyze it in IoTDB, Spark and 
Flink. IoTDB could also generate TsFiles and synchronize TsFiles between two 
IoTDB instances.
Furthermore, the demand for TsFile implementations in multiple programming 
languages has been growing, as it allows developers to leverage TsFile's 
capabilities in their preferred language.

# TsFile offers several distinctive features and benefits:
Efficient Storage and Compression: TsFile employs advanced compression 
techniques to minimize storage requirements, resulting in reduced disk space 
consumption and improved system efficiency.
Flexible Schema and Metadata Management: TsFile allows for directly write data 
without predefining the schema, which is flexible for data acquisition.
High Query Performance with time range: TsFile has indexed devices, sensors, 
and time dimensions to accelerate query performance, enabling fast filtering 
and retrieval of time series data.
Seamless Integration: TsFile is designed to seamlessly integrate with existing 
big data frameworks, such as Spark,  Flink and Hadoop.

# Rationale
TsFile is the first existing standard file format for time series data. In the 
industry companies usually write time series data without unification or use 
general columnar file format such as Parquet and ORC, which makes data 
collection and processing complicated without a standard. With TsFile, 
organizations could write data in TsFile inside end devices or gateway, then 
transfer TsFile to the cloud for unified management in IoTDB and other systems. 
In this way, we lower the network transmission and the computing resource 
consumption in the cloud.

# Initial Goals
The initial goals include:
Make TsFile an independent project that has its own SDK and documentation that 
is easier to use.
Multi-Language Support of TsFile implementations apart from Java, such as C++, 
Go and Rust.
Integrate more encoding and compression methods in TsFile.
More tools for TsFile: visualization tool, parsing tool, repair tool.

# Current Status

## Meritocracy
We plan to invite a subset of the Apache IoTDB committers as initial committers 
of Apache TsFile. And we would like to follow ASF meritocratic principles and 
invite additional developers to participate. We will establish the 
documentation and encourage and monitor community participation so that 
privileges can be extended to those that contribute.

## Community
The TsFile community is grown from the Apache IoTDB Community. The IoTDB 
community is introducing TsFile at many technical conferences. Next, we will 
build the mailing list for more convenience, broader communication and archived 
discussions. We are open to recruiting more developers from diverse backgrounds.

## Core Developers
TsFile initial PMCs are from IoTDB community: Christofer Dutz, Xiangdong Huang, 
Jialin Qiao, Steve Yurong Su, Jinrui Zhang, Yuan Tian, Xinyu Tan, Haonan Hou, 
Gaofei Cao, Tian Jiang, Chao Wang(wangchao316), Chao Wang(mychaow), Houliang 
Qi, Kun Liu.
These people have extensive experience in building database and data management 
system.

## Alignment
The ASF is the natural choice to host the TsFile project as its goal of 
encouraging community-driven open-source projects fits with our vision for 
TsFile. Additionally, many other projects with which we are familiar with and 
expect TsFile to integrate with, such as Apache Spark, Apache Flink and Apache 
IoTDB are hosted by the ASF and we will benefit and provide benefits in close 
proximity to them.

# Known Risks

## Project Name
TsFile project is used in IoTDB and other scenarios for over 7 years, its name 
is unique.

## Orphaned Products

[DISCUSS] Migrate from the javax namespace over to Jakarta?

[Christofer Dutz]

Hi all,

while cleaning up the dependencies, I noticed, that for some dependencies we’re 
stuck at quite old versions as all newer releases use the Jakarta namespace.
There’s even a CVE, that we can’t get rid of without either refactoring the 
jersey-common dependency away or switching to Jakarta.

I would strongly recommend moving to Jakarta, as we must do that anyway as all 
projects are doing that transition and most have already finished that.

The longer we wait, the more work will we have to invest into keeping it.

What do you folks think?

Chris

[DISCUSS] Drop Spark 2.11

[Christofer Dutz]

Hi all,

while working on the dependency cleanup, I noticed that we were stuck with some 
pretty old spark versions in the spark_2.11 module.
The reason is that the scala 2.11 versions have been end of life for quite some 
time. Spark is now generally working with scala 2.12 and since version 3.2 also 
2.13.

We have one reported CVE that we actually can’t get rid of in the scala_2.11 
module, as there will be no release with a fix.

I therefore propose we drop the 2.11 scala spark plugin and possibly add a 2.13 
version instead.

What do you folks think?

Chris

AW: Dependency management PR ready for being merged

[Christofer Dutz]

Happy you folks like it …

might be a tiny bit more work taking care of dependencies, but we’re in a much 
better place now (security-wise) and this way things should automatically stay 
pretty stable.

Currently following up on one of the managed CVEs (netty) … seems they released 
a fixed version ;-)

Chris

Von: Jialin Qiao 
Datum: Dienstag, 17. Oktober 2023 um 05:35
An: dev@iotdb.apache.org 
Betreff: Re: Dependency management PR ready for being merged
Merged, Good work!
—
Jialin Qiao
Apache IoTDB PMC

AW: Rollcall TsFile initial committers

[Christofer Dutz]

Hi Jialin,

even if I haven’t made significant contributions to the code of the TSFile 
module, I think having me on board to help with the build side might be a good 
idea. So if you want me to help, I’m happy to do so.

Chris


Von: Yuan Tian 
Datum: Montag, 16. Oktober 2023 um 06:20
An: dev@iotdb.apache.org 
Betreff: Re: Rollcall TsFile initial committers
Hi Jialin,

I would like to be the initial committer of TsFile. I've contributed to
[Aligned]ChunkWriter, [Aligned]PageWriter, [Aligned]PageReader.
I know very well about tsfile disk file structure and familiar with the
query and write process of tsfile.

Best,
-
Yuan Tian

Dependency management PR ready for being merged

[Christofer Dutz]

Hi all,

thanks to the help of some of my colleagues, that had an Intel device to stage 
my updated version of Ratis, the PR of mine is ready to be merged:
https://github.com/apache/iotdb/pull/11150

Not only does I update most of our plugins and dependencies to new version. It 
addresses most reported CVEs in third party dependencies. Also does it make 
sure we no longer have multiple conflicting versions of dependencies in our 
dependency tree.

It would be great if reviewing this PR could happen as soon as possible as due 
to its nature, it is hugely impacted by work done on develop (Almost every 
change in any pom is a conflict, that needs to be manually merged).

Chris

Re: [DISCUSS] Incubating TsFile as an independent Apache project?

[Christofer Dutz]

I'd say, just as a board resolution to the board meeting.

Chris

Gesendet von Outlook für Android

From: Justin Mclean 
Sent: Tuesday, October 10, 2023 6:34:08 PM
To: dev@iotdb.apache.org 
Subject: Re: [DISCUSS] Incubating TsFile as an independent Apache project?

Hi,

If enough people on the initial PMC know the ASF process well, there would be 
no need to go through the Incubator but go straight to TLP.

Kind Regards,
Justin

AW: [DISCUSS] Incubating TsFile as an independent Apache project?

[Christofer Dutz]

Hi all,

not even sure we’d need to go the route throught the incubator.
If there are enough seasoned and experienced Apache forlks in the PMC, I’d even 
support. A direct to TLP route.

Chris

Von: Jialin Qiao 
Datum: Samstag, 7. Oktober 2023 um 03:59
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Incubating TsFile as an independent Apache project?
Hi,

+1 for Apache TsFile.

1. Industrial time series data management is becoming more and more
important, so a file format project for time series data is necessary.
2. When TsFile becomes an independent project, it could be developed
and released without impact the upgrading of IoTDB.

Thanks,
―
Jialin Qiao
Apache IoTDB PMC

冯 庆新  于2023年10月7日周六 10:22写道：
>
> Hi Steve Su
>
> I think this is a great idea and I fully support it.
> I believe Tsfile can generate greater influence and value.
>
> Thanks
> B.R.
> Qingxin Feng
>
>
> 从 Windows 版邮件发送
>
> 发件人: Steve Yurong Su
> 发送时间: 2023年10月7日 0:53
> 收件人: dev@iotdb.apache.org
> 主题: [DISCUSS] Incubating TsFile as an independent Apache project?
>
> Hi folks,
>
> Recently, I've been thinking about something that has been on my mind:
> I believe TsFile has the potential to develop as an independent Apache
> project (note, not a subproject like apache-iotdb-tsfile), just like
> ORC, Parquet, and Arrow. How about incubating TsFile as an independent
> Apache project?
>
> The main reason I came up with this idea is because I realized that
> TsFile is not just the data file format of IoTDB, but it can also be a
> standard format in the IoT field. It provides excellent support for
> time series data, offering various encoding and compression methods.
> It can be easily self-parsed and its statistical information enables
> fast data querying by big data software. I believe that every IoT
> software could benefit directly by supporting TsFile as a storage
> format.
>
> On the other hand, I hope to leverage the power of the community to
> drive the development of the TsFile file format. Just imagine, TsFile
> could have its own independent ecosystem: parsing tools in various
> languages, connectors to various big data software, and so on. By
> separating TsFile from IoTDB, we have the opportunity to attract more
> community contributors and introduce more ideas, thus refining the
> functionality and performance of TsFile.
>
> What do you think about this idea?
>
> In addition, how can TsFile, as a part of the current IoTDB project,
> become an independent Apache project? Should it go through the Apache
> Foundation incubation process again, or can it directly become a
> top-level project? It may be too early to ask this question now, but I
> would also like to hear your opinions.
>
> Thanks,
> Steve Su
>

Things keeping us from updating ...

[Christofer Dutz]

Hi all,

So, while going through all of our dependencies, I noticed that some 
fundamental things are keeping us from freely updating dependencies to the 
latest version.


  *   Java 8
  *   Javax namespace
  *   Spark 2.11
  *   Scalatest version
  *   Powermock version

While I understand us sticking to Java 8, we might consider addressing the 
other things:

We had a wild mixture of javax and Jakarta dependencies, however mixing them is 
a bad idea. I therefore managed all to use the javax namespace, but with 
Jakarta dependencies.
I would strongly suggest we migrate to the new Jakarta namespace, as this will 
unblock us with several deadlocks. (spring-boot, spring, jersey)

Support for Scala 2.11 seems to have been faded out … for spark we needed to 
stay at Spark 2.4.8, while the rest is at 3.5.0 (It generally seems Spark 2 was 
2.11 and 2.12 and Spark 3 is 2.12 and 2.13)

In Scalatest they have deprecated some forms of testing and the testsuite would 
require some refactoring to accommodate the new form of testing.

It seems that powermock is causing problems with Mockito and is keeping us from 
updating from 2.23.4 to the latest version that supports Java 8 (4.11.8).
In general it seems that these libraries are the ones causing a lot of red 
output during the build as they use deprecated functionality in the JVM … 
possibly updating to bytebuddy would be a good idea.

Chris

Heads-up: Managed dependencies ...

[Christofer Dutz]

Hi all,

I have spent a lot of time over the last week to address most of the most 
pressing dependency problems we were having.
I noticed that we had some pretty dangerous mixtures in some of our libraries. 
Such mixtures can produce pretty hard to diagnose problems.

https://github.com/apache/iotdb/pull/11150

So, I have turned on the enforcer plugin and increased the severity of many 
plugins:

  *   The dependency plugin now fails the build:
 *   if you use a transitive dependency and not declaring it as an import
 *   if you have a compile dependency, that should be a test-dependency
 *   if you have dependencies that are not used
  *   The enforcer plugin now fails the build, if we have multiple versions of 
transitive dependencies in the dependency tree of an artifact
  *   The enforcer plugin now fails the build, if we are using dependencies for 
which CVEs have been reported
  *   The enforcer plugin now fails the build, if multiple dependencies provide 
classes with the same name, but different content (Usually shaded or no-deps 
dependencies)

So, I have:

  *   updated the dependencies of all project to import what is needed and to 
import it into the correct classpath.
  *   managed the dependencies of conflicting transitive dependencies to pull 
in a clearly defined version.
  *   gone through dependencies for which vulnerabilities were reported and 
updated almost all to safe versions.
  *   removed most of the “with-deps” dependencies and replaced them by cleanly 
defined imports.
  *   I refactored code blocks, where we had unfortunate imports (Base64 
decoder that pulled in Glassfisch, Test that pulled in Tomcat)

The PR is still not mergeable, because I also have a PR running in Ratis, which 
we need for running my version. But I was told that a new version will be made 
available soon.

I also updated the poms so all inner-project dependencies have the full 
version-string “0.13.0-SNAPHOT” instead of “${project.version}” as it doesn’t 
add more work when releasing, and this way you can instantly see the version of 
a pom without having to scroll up. This has saved my life in other projects way 
too often.

All third-party dependencies are managed and don’t provide the version in the 
pom where they are used. All versions are defined in the parent pom.

As finding dependencies some-times is a challenge because the order of 
declaration varies from module to module, I used a plugin to sort the elements 
of the poms in a unified way.
So, dependencies are sorted by “scope”, “groupId”, “artifactId” … this makes 
finding a dependency a lot easier. I know the order of dependencies matters, 
but as we’re explicitly managing all important dependencies, the order doesn’t 
matter.

Regarding the CVEs, I couldn’t eliminate all CVEs as for some new ones there 
simply are no fixed versions yet. In these cases, I explicitly excluded them 
and added a todo to keep track of them.
Some CVEs simply don’t effect us. If a CVE is reported to have an effect on a 
component that we are not using, I also excluded these versions and added a 
comment above, describing why I excluded it.


I really hope I can finish this task asap as it was extremely difficult and 
took way too much time. But I think the project is in a way better and safer 
state with these changes.

Chris

AW: AW: AW: [DISCUSS] Updating to a newer Thrift version?

[Christofer Dutz]

Hi all,

And if all is good, we could just start a vote on the generator and switch to 
using that.

Chris


Von: Haonan Hou 
Datum: Mittwoch, 20. September 2023 um 16:10
An: dev@iotdb.apache.org 
Betreff: Re: AW: AW: [DISCUSS] Updating to a newer Thrift version?
Ok, just pushed a branch using the staged thrift 0.19.0 generator.
https://github.com/apache/iotdb/pull/11183

We need to do more tests on it before the upgrading, such as using the old 
client with Thrift 0.14.1 to connect the data node with 0.19.0. Using the old 
data node to connect the config node with new Thrift...

Best,
Haonan

On 2023/09/19 08:02:57 Christofer Dutz wrote:
> The repo id of the staged thirft 0.19.0 generator is here:
> https://repository.apache.org/content/repositories/orgapacheiotdb-1132
>
> So, adding that repository and switching to:
> 0.19.0.0
>
> 0.19.0
>
> Should do the trick.
>
> Chris
>
> Von: Christofer Dutz 
> Datum: Dienstag, 19. September 2023 um 08:50
> An: dev@iotdb.apache.org 
> Betreff: AW: AW: [DISCUSS] Updating to a newer Thrift version?
> I��m currently preparing the build-tools for 0.19.0.
> As soon as that��s done, we could simply build iotdb with 0.19.0.0 and simply 
> try it out?
> I couldn��t find any info on the binary protocol being changed in the 
> release-notes.
> But I guess simply trying it out should help.
>
> Chris
>
> Von: Xiangdong Huang 
> Datum: Dienstag, 19. September 2023 um 01:50
> An: dev@iotdb.apache.org 
> Betreff: Re: AW: [DISCUSS] Updating to a newer Thrift version?
> Hi,
> one more concern, is Thrift v0.19 compatible with Thrift v0.14? (It is
> better to be compatible for all IoTDB v1.x. )
> ---
> Xiangdong Huang
> School of Software, Tsinghua University
>
>  
> �廪��ѧ ѧԺ
>
>
> Haonan Hou  ��2023��9��18һ 18:13д
>
> > It seems that Thrift 0.19.0 is released and supports Java 8 again. ;-)
> > https://github.com/apache/thrift/blob/master/CHANGES.md
> >
> > Best,
> > Haonan
> >
> > On 2023/08/04 10:01:39 Christofer Dutz wrote:
> > > Ok ��
> > >
> > > So, it turns out that the Thrift folks intentionally updated to Java 11
> > but took that back and the next release should be based on Java 8 again.
> > > So hopefully that will come soon and then we could have a look about
> > compatibility of the generated code. I guess if the wire-protocol didn��t
> > change there should generally be no reason for it to not be compatible, and
> > if they don��t release a major version, usually this compatibility should
> > stay intact (if they sort of follow SemVer)
> > >
> > > I guess the best path of action would be to do nothing right now �� as an
> > alternative, we could of course turn on the Java compilation when building
> > thrift and to bundle the jar built by that in the assembly.
> > > If we then replace the thrift-lib dependency with a ��system�� scope
> > dependency, we could already do that now. But admittedly I would not be in
> > favor of doing that. I would recommend for us to wait for the next official
> > release.
> > >
> > > Chris
> > >
> > > Von: Christofer Dutz 
> > > Datum: Dienstag, 1. August 2023 um 08:29
> > > An: dev@iotdb.apache.org 
> > > Betreff: Re: [DISCUSS] Updating to a newer Thrift version?
> > > I opened a ticket in their jira. Might even prepare a pr... Shouldn't be
> > too difficult.
> > >
> > > Chris
> > >
> > > Gesendet von Outlook f��r Android<https://aka.ms/AAb9ysg>
> > > 
> > > From: ̷ <1025599...@qq.com.INVALID>
> > > Sent: Monday, July 31, 2023 1:57:28 PM
> > > To: dev@iotdb.apache.org 
> > > Subject: Re: [DISCUSS] Updating to a newer Thrift version?
> > >
> > > Hi, Chris
> > >
> > > In the latest version of thrift, there are some improvements we'd like
> > to make, For example, https://issues.apache.org/jira/browse/THRIFT-5502
> > will down-cycled  "connection reset"  warn logs.
> > >
> > > +1 for upgrading thrift.
> > >
> > >
> > > Thanks
> > > �D�D�D�D�D�D�D�D�D�D�D�D
> > > Xinyu Tan
> > >
> > > > 2023��7��31�� 19:50��Christofer Dutz  д
> > > >
> > > > Hi all,
> > > >
> > > > While working on the cleanup of the build, I noticed we��re working
> > with Thrift in version 0.14.1 however the latest version is 0.18.1
> > > >
> > > > Is there a reason we��re sticking to a two-year older version than the
> > newest?
> > > >
> > > > If not: with the pom-cleanup refactoring it should be a thing of
> > minutes to update this.
> > > >
> > > > Chris
> > > >
> > >
> >
>

AW: AW: [DISCUSS] Updating to a newer Thrift version?

[Christofer Dutz]

The repo id of the staged thirft 0.19.0 generator is here:
https://repository.apache.org/content/repositories/orgapacheiotdb-1132

So, adding that repository and switching to:
0.19.0.0

0.19.0

Should do the trick.

Chris

Von: Christofer Dutz 
Datum: Dienstag, 19. September 2023 um 08:50
An: dev@iotdb.apache.org 
Betreff: AW: AW: [DISCUSS] Updating to a newer Thrift version?
I’m currently preparing the build-tools for 0.19.0.
As soon as that’s done, we could simply build iotdb with 0.19.0.0 and simply 
try it out?
I couldn’t find any info on the binary protocol being changed in the 
release-notes.
But I guess simply trying it out should help.

Chris

Von: Xiangdong Huang 
Datum: Dienstag, 19. September 2023 um 01:50
An: dev@iotdb.apache.org 
Betreff: Re: AW: [DISCUSS] Updating to a newer Thrift version?
Hi,
one more concern, is Thrift v0.19 compatible with Thrift v0.14? (It is
better to be compatible for all IoTDB v1.x. )
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Haonan Hou  于2023年9月18日周一 18:13写道：

> It seems that Thrift 0.19.0 is released and supports Java 8 again. ;-)
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Best,
> Haonan
>
> On 2023/08/04 10:01:39 Christofer Dutz wrote:
> > Ok …
> >
> > So, it turns out that the Thrift folks intentionally updated to Java 11
> but took that back and the next release should be based on Java 8 again.
> > So hopefully that will come soon and then we could have a look about
> compatibility of the generated code. I guess if the wire-protocol didn’t
> change there should generally be no reason for it to not be compatible, and
> if they don’t release a major version, usually this compatibility should
> stay intact (if they sort of follow SemVer)
> >
> > I guess the best path of action would be to do nothing right now … as an
> alternative, we could of course turn on the Java compilation when building
> thrift and to bundle the jar built by that in the assembly.
> > If we then replace the thrift-lib dependency with a “system” scope
> dependency, we could already do that now. But admittedly I would not be in
> favor of doing that. I would recommend for us to wait for the next official
> release.
> >
> > Chris
> >
> > Von: Christofer Dutz 
> > Datum: Dienstag, 1. August 2023 um 08:29
> > An: dev@iotdb.apache.org 
> > Betreff: Re: [DISCUSS] Updating to a newer Thrift version?
> > I opened a ticket in their jira. Might even prepare a pr... Shouldn't be
> too difficult.
> >
> > Chris
> >
> > Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>
> > 
> > From: 谭新宇 <1025599...@qq.com.INVALID>
> > Sent: Monday, July 31, 2023 1:57:28 PM
> > To: dev@iotdb.apache.org 
> > Subject: Re: [DISCUSS] Updating to a newer Thrift version?
> >
> > Hi, Chris
> >
> > In the latest version of thrift, there are some improvements we'd like
> to make, For example, https://issues.apache.org/jira/browse/THRIFT-5502
> will down-cycled  "connection reset"  warn logs.
> >
> > +1 for upgrading thrift.
> >
> >
> > Thanks
> > �D�D�D�D�D�D�D�D�D�D�D�D
> > Xinyu Tan
> >
> > > 2023年7月31日 19:50，Christofer Dutz  写道：
> > >
> > > Hi all,
> > >
> > > While working on the cleanup of the build, I noticed we’re working
> with Thrift in version 0.14.1 however the latest version is 0.18.1
> > >
> > > Is there a reason we’re sticking to a two-year older version than the
> newest?
> > >
> > > If not: with the pom-cleanup refactoring it should be a thing of
> minutes to update this.
> > >
> > > Chris
> > >
> >
>

AW: AW: [DISCUSS] Updating to a newer Thrift version?

[Christofer Dutz]

I’m currently preparing the build-tools for 0.19.0.
As soon as that’s done, we could simply build iotdb with 0.19.0.0 and simply 
try it out?
I couldn’t find any info on the binary protocol being changed in the 
release-notes.
But I guess simply trying it out should help.

Chris

Von: Xiangdong Huang 
Datum: Dienstag, 19. September 2023 um 01:50
An: dev@iotdb.apache.org 
Betreff: Re: AW: [DISCUSS] Updating to a newer Thrift version?
Hi,
one more concern, is Thrift v0.19 compatible with Thrift v0.14? (It is
better to be compatible for all IoTDB v1.x. )
---
Xiangdong Huang
School of Software, Tsinghua University

 黄向东
清华大学 软件学院


Haonan Hou  于2023年9月18日周一 18:13写道：

> It seems that Thrift 0.19.0 is released and supports Java 8 again. ;-)
> https://github.com/apache/thrift/blob/master/CHANGES.md
>
> Best,
> Haonan
>
> On 2023/08/04 10:01:39 Christofer Dutz wrote:
> > Ok …
> >
> > So, it turns out that the Thrift folks intentionally updated to Java 11
> but took that back and the next release should be based on Java 8 again.
> > So hopefully that will come soon and then we could have a look about
> compatibility of the generated code. I guess if the wire-protocol didn’t
> change there should generally be no reason for it to not be compatible, and
> if they don’t release a major version, usually this compatibility should
> stay intact (if they sort of follow SemVer)
> >
> > I guess the best path of action would be to do nothing right now … as an
> alternative, we could of course turn on the Java compilation when building
> thrift and to bundle the jar built by that in the assembly.
> > If we then replace the thrift-lib dependency with a “system” scope
> dependency, we could already do that now. But admittedly I would not be in
> favor of doing that. I would recommend for us to wait for the next official
> release.
> >
> > Chris
> >
> > Von: Christofer Dutz 
> > Datum: Dienstag, 1. August 2023 um 08:29
> > An: dev@iotdb.apache.org 
> > Betreff: Re: [DISCUSS] Updating to a newer Thrift version?
> > I opened a ticket in their jira. Might even prepare a pr... Shouldn't be
> too difficult.
> >
> > Chris
> >
> > Gesendet von Outlook für Android<https://aka.ms/AAb9ysg>
> > 
> > From: 谭新宇 <1025599...@qq.com.INVALID>
> > Sent: Monday, July 31, 2023 1:57:28 PM
> > To: dev@iotdb.apache.org 
> > Subject: Re: [DISCUSS] Updating to a newer Thrift version?
> >
> > Hi, Chris
> >
> > In the latest version of thrift, there are some improvements we'd like
> to make, For example, https://issues.apache.org/jira/browse/THRIFT-5502
> will down-cycled  "connection reset"  warn logs.
> >
> > +1 for upgrading thrift.
> >
> >
> > Thanks
> > �D�D�D�D�D�D�D�D�D�D�D�D
> > Xinyu Tan
> >
> > > 2023年7月31日 19:50，Christofer Dutz  写道：
> > >
> > > Hi all,
> > >
> > > While working on the cleanup of the build, I noticed we’re working
> with Thrift in version 0.14.1 however the latest version is 0.18.1
> > >
> > > Is there a reason we’re sticking to a two-year older version than the
> newest?
> > >
> > > If not: with the pom-cleanup refactoring it should be a thing of
> minutes to update this.
> > >
> > > Chris
> > >
> >
>

Stories from the trenches: An IT guy working in an OT world ...

[Christofer Dutz]

Hi all,

for those of you able to come … next week on Wednesday I’ll be giving a talk at 
IoT-Hessen Kassel in Kassel (Germany) on what I experienced as an IT guy 
working in an OT world.

There are many things, folks in industrial Automation can learn from us IT 
folks, but also quite a bit of things that we can learn from them.

So, if you’re interested:
https://www.meetup.com/de-DE/iot-hessen-kassel/events/295274696/


Chris

AW: AW: Ratis SNAPSHOT versions in our latest release ...

[Christofer Dutz]

Hi Xinyu,

I totally understand the reasoning behind it. There however is a non-technical 
issue that I see for which we should not release SW that relies on SNAPSHOTS.

If we do this, we are practically embedding software in our software, which has 
not formally been approved by a PMC. This is more a legal topic.

Are the Ratis folks having technical issues with releasing? I know many 
projects at Apache write awesome software, but have problems maintaiing their 
build infrastructure. Perhaps that’s something I could help them with?

And I guess we don’t need a new Ratis release for ever IoTDB release, right?

Chris

Von: Xinyu Tan 
Datum: Donnerstag, 14. September 2023 um 07:02
An: dev@iotdb.apache.org 
Betreff: Re: AW: Ratis SNAPSHOT versions in our latest release ...
Hi, Chris

As mentioned by William, initially, we relied solely on the snapshot version of 
Ratis in the master branch to quickly validate many new bug fixes and features 
(just counted, we've added around 60 patches to the Ratis community in the past 
year). In fact, when we released a new version of IoTDB earlier this year, we 
attempted to encourage the Ratis community to release a new version as well. 
However, due to issues with some other patches in the community, the release 
went through multiple release candidates (rc), ultimately lasting close to one 
month, which affected IoTDB's release schedule.

Looking back, in 2023, IoTDB released nearly 7 versions, while the Ratis 
community released only 2 versions. This has led us to the need to find a 
solution to balance the issues arising from different release speeds. Depending 
on a long-standing snapshot version is a last resort for us.

Fortunately, with further collaboration between IoTDB and Ratis, Ratis in the 
context of IoTDB has recently become relatively stable. Therefore, we 
anticipate that in future releases, we will make an effort to ensure that 
IoTDB's release version also includes an official version of Ratis to avoid 
potential risks.

Thanks for Chris's reminder!

Xinyu Tan

On 2023/09/13 14:31:13 Christofer Dutz wrote:
> Hi all,
>
> after some discussions with colleagues it turns out that it’s not quite as 
> dramatic as I first throuhgt. So first I thought the commit hash was some way 
> to address one fixed SNAPSHOT version via some mechanism I just didn’t know 
> yet, but it turns out to be a lot simpler …. It produces a SNAPSHOT for 
> version “2.5.2-a4398bf“ … so it’s an artificial version for which then again 
> 3-5 SNAPSHOTS will be keept.
>
> Seems it’s some shorthand version of inofficially releasing things without 
> actually releasing them.
>
> So it’s still not ideal, as the referenced artifacts will never go to Maven 
> Central and could cause problems with the one or the other user, I don’t see 
> it as an immediate threat.
>
> Chris
>
> Von: Christofer Dutz 
> Datum: Mittwoch, 13. September 2023 um 11:01
> An: dev@iotdb.apache.org 
> Betreff: Ratis SNAPSHOT versions in our latest release ...
> Hi,
>
> I’m currently working on resolving some of the dependency version issues we 
> are having.
> Mostly people will not have noticed, but currently we’re pulling in up to 4 
> different versions of a jar in our build. This can cause many extremely hard 
> to spot problems.
>
> While trying to fix a problem with metrics-core in version 4.2.7 but pulling 
> in on older version in Ratis I noticed us using:
>
> 2.5.2-a4398bf-SNAPSHOT
>
> This is extremely problematic. Currently the Apache Nexus server only keeps 5 
> SNAPSHOT versions and then deletes old ones. This means that we regularly 
> have to bump the SNAPSHOT version of Ratis.
>
> This got me thinking and I checked the release branch for the 1.2.x branch. 
> Here we’re using the same.
>
> The problem with using SNAPSHOTS on master is not that severe, but using them 
> in releases it very problematic. I guess we’ll only be able to build our last 
> release for a few more days/weeks and then it will no longer be buildable.
>
> Are we relying on things in Ratis, that are not yet released?
>
> We should probably encourage the Ratis folks to head for a new release 
> (Ideally with my latest Ratis-PR merged).
>
> Chris
>

AW: Ratis SNAPSHOT versions in our latest release ...

[Christofer Dutz]

Hi all,

after some discussions with colleagues it turns out that it’s not quite as 
dramatic as I first throuhgt. So first I thought the commit hash was some way 
to address one fixed SNAPSHOT version via some mechanism I just didn’t know 
yet, but it turns out to be a lot simpler …. It produces a SNAPSHOT for version 
“2.5.2-a4398bf“ … so it’s an artificial version for which then again 3-5 
SNAPSHOTS will be keept.

Seems it’s some shorthand version of inofficially releasing things without 
actually releasing them.

So it’s still not ideal, as the referenced artifacts will never go to Maven 
Central and could cause problems with the one or the other user, I don’t see it 
as an immediate threat.

Chris

Von: Christofer Dutz 
Datum: Mittwoch, 13. September 2023 um 11:01
An: dev@iotdb.apache.org 
Betreff: Ratis SNAPSHOT versions in our latest release ...
Hi,

I’m currently working on resolving some of the dependency version issues we are 
having.
Mostly people will not have noticed, but currently we’re pulling in up to 4 
different versions of a jar in our build. This can cause many extremely hard to 
spot problems.

While trying to fix a problem with metrics-core in version 4.2.7 but pulling in 
on older version in Ratis I noticed us using:

2.5.2-a4398bf-SNAPSHOT

This is extremely problematic. Currently the Apache Nexus server only keeps 5 
SNAPSHOT versions and then deletes old ones. This means that we regularly have 
to bump the SNAPSHOT version of Ratis.

This got me thinking and I checked the release branch for the 1.2.x branch. 
Here we’re using the same.

The problem with using SNAPSHOTS on master is not that severe, but using them 
in releases it very problematic. I guess we’ll only be able to build our last 
release for a few more days/weeks and then it will no longer be buildable.

Are we relying on things in Ratis, that are not yet released?

We should probably encourage the Ratis folks to head for a new release (Ideally 
with my latest Ratis-PR merged).

Chris

AW: [DISCUSS] Switch to Java 11 for building, but staying at Java 8 for compiling?

[Christofer Dutz]

Hmmm …

I think it should be possible to run the tests without compiling them … haven’t 
tried that however.

If we run mvn surefire:test it should only run the test plugin … however we 
probably need all configuration in “pluginManagement” blocks instead of “build” 
blocks.

I could probably try it out …

Chris

Von: Haonan Hou 
Datum: Mittwoch, 13. September 2023 um 03:48
An: dev@iotdb.apache.org 
Betreff: Re: [DISCUSS] Switch to Java 11 for building, but staying at Java 8 
for compiling?
Hi,

I have a question. If we do that and hope to run UT/IT with jdk 8 by mvn 
commands, how to separate the compile and test with the different jdk versions?

Best,
Haonan

On 2023/09/12 13:27:19 Christofer Dutz wrote:
> Hi all,
>
> while updating some of our plugins, I ran into the problem, that some newer 
> versions of the plugins or dependencies only work with Java 11.
>
> In PLC4X we had similar problems. So, we decided to switch to at least Java 
> 11 for building, but still ensure the output is compatible with Java 8.
>
> This way we can benefit from plugin updates and still not loose compatibility.
>
> There are even plugins available to validate API compatibility with Java 8 
> SDKs.
>
> Chris
>
>