Re: [VOTE] Whether to automatically filter out rows that are entirely null in the result set during a scan by the storage engine
b +1 Jialin Qiao Xinyu Tan 于2024年9月2日周一 10:33写道： > > Hi > > I prefre b. > > I think it's more right to be compatible with users of the tree model and to > be consistent with relational databases for the table model. Aligning the > table and tree aligned time series semantically is something that can be > sacrificed. > > Best > - > Xinyu Tan > > On 2024/09/02 01:51:39 Yuan Tian wrote: > > Hi all, > > > > As we discussed in [1], we need to decide whether to automatically filter > > out rows that are entirely null in the result set during a scan by the > > storage engine. > > > > There are three reasonable options: > > > > [a] keep table model same as tree model ( automatically filter out rows > > that are entirely null ) > > [b] table model don't automatically filter out rows that are entirely null, > > but keep the tree model same as before. > > [c] table model and aligned time series in tree model don't automatically > > filter out rows that are entirely null, but keep nonAligned time series in > > tree model same as before > > > > > > Personally, I prefer [c], because in the storage engine, table model shares > > the same things as aligned time series in tree model, [b] will break it up > > while scanning. > > > > Feel free to vote for the option you personally want to choose. > > > > > > > > > > > > [1] https://lists.apache.org/thread/40gk1k1q3styjsf4s95ydlm5f2l2lcr4 > > > > > > Best regards, > > -- > > Yuan Tian > >
Re: Upgrading version from 1.4.0-SNAPSHOT to 2.0.0-SNAOSHOT for master branch
Hi, AI Node V1 could be first released in 2.0.0, 1.3.3 should be a bug-fix version. Jialin Qiao 赵馨逸 于2024年8月30日周五 20:40写道： > > AINode V2 will be upgrade the architecture(not start yet) and enlarge the > features provided. This will also be landing in V2.x version. > > From: "Christofer Dutz" > > Date: Fri, Aug 30, 2024, 18:59 > > Subject: AW: Upgrading version from 1.4.0-SNAPSHOT to 2.0.0-SNAOSHOT for > > master branch > > To: "dev@iotdb.apache.org" > > But isn’t the AI node already going to be in 1.3.3? > > > > Chris > > > > Von: Xinyu Tan > > Datum: Freitag, 30. August 2024 um 12:50 > > An: dev@iotdb.apache.org > > Betreff: Re: Upgrading version from 1.4.0-SNAPSHOT to 2.0.0-SNAOSHOT for > > master branch > > Hi, > > > > Since both AINode and the table model are significant work for IoTDB, +1 > > for switching to 2.0.0-snapshot. > > > > I believe we don't need to be as conservative as before when changing major > > versions. Many database products change their major version numbers every > > one to two years. > > > > Best > > - > > Xinyu Tan > > > > On 2024/08/25 05:37:21 Yuan Tian wrote: > > > Hi all, > > > > > > Given the significant changes that the table model introduces to IoTDB and > > > the associated upgrade from TsFile version V3 to V4, it is reasonable to > > > consider updating the version on the master branch to 2.0.0-SNAPSHOT. This > > > approach aligns with the intention to release the table model with the > > > version number 2.0.0, signifying a major update to the system. > > > > > > For the tree model, which will continue to evolve with bug fixes and > > > performance improvements on the 1.X.X series (such as the upcoming 1.3.3 > > > version), it's important to maintain clear versioning to differentiate > > > between the stable, incremental updates and the more substantial changes > > > introduced by the table model. What do you think? Best regards, > > > > > > > > > > > > Yuan Tian > > >
Re: Upgrading version from 1.4.0-SNAPSHOT to 2.0.0-SNAOSHOT for master branch
Hi, In previous years, we mainly focus on Data Management (Insertion & Storage & Synchronization & Simple query on one series). If we summarize the ongoing works of IoTDB, we will find the Relation module (Query across multi timeseries), AINode (Predict the trend of timeseries) are all for enhancing the Data Analysis. So, a major version bump looks acceptable, indicating IoTDB enters a new phase from Data Management to Data Analysis. Jialin Qiao Stefanie Zhao 于2024年8月26日周一 12:29写道： > > Hi Chris, > > I really appreciate that there is such a discussion on the version number of > IoTDB in the community, and here are some of my thoughts. > > I believe that upgrading the first version number requires consideration of > various impacts. The compatibility of interfaces between versions is one > aspect. At the same time, it is also necessary to consider whether the > improvements of the version will have a significant impact on other aspects > of the system, such as stability and maturity. These are also important > factors that users pay attention to, so we want to make our users perceive > this through the upgrade from 1.x to 2.0. > > Involving table model to IoTDB has the following specific changes: > 1. Milestone changes in functionality: With the addition of the table model, > IoTDB will have two sets of metadata models. This is an important large-scale > update of a module, representing that IoTDB will be able to support and > connect with traditional developer habits in the future, which will have a > milestone impact on users. > 2. Upgrade of code architecture: Due to the table model, IoTDB's data file > has been upgraded from TsFile v3 to v4, which involves adjustments to the > file structure. And also, we add a query engine for the table model. These > are all major upgrades to code architecture. > 3. Extensive impact and changes to the code: Although we are still in the > development stage of the table model, just the first PR[13119], the table > model has affected 1266 files (out of a total of 5943 Java files), involving > 116973 lines+ and 13196 lines- of code. This scope of code impact far exceeds > the impact range of an ordinary feature. > 4. Impact on system stability and maturity: Although we will strive to reduce > the impact of the above extensive functional changes and code modifications > on the performance and stability of the entire system, as software > developers, we all understand that achieving the above goals still requires > some time to mature, and the impact on stability is also part of what users > perceive. > > I have also looked at some requirments of the upgrade of the first version > number in Apache projects, but it seems that there is no clear standard. The > introduction of incompatibilities in features or interfaces you mentioned is > indeed an important consideration (Spark also explicitly mentions it, Apache > Spark Release 3.0.0), but more importantly, it is whether developers want to > convey to users "this is a major version change, which may have a significant > impact on users." So, like other Apache projects (such as Apache Doris), they > will also consider architectural upgrades, changes in metadata, or underlying > data formats as factors for upgrading the first version number (Apache Doris > Release Versioning). > > So we still hope that the table model version number will make its debut to > our users as version 2.0. > > - > Best, Stefanie > > On 2024/08/25 07:40:50 Christofer Dutz wrote: > > Usually major version upgrades introduce breaking changes. As far as I > > understood it, it just adds features. That would qualify for a second digit > > increase. > > > > If we do a major version bump we should also consider doing some other > > refactoring, that we postponed till the next major release. > > > > Chris > > > > Gesendet von Outlook für Android<https://aka.ms/AAb9ysg> > > > > From: Yuan Tian > > Sent: Sunday, August 25, 2024 7:37:21 AM > > To: dev > > Subject: Upgrading version from 1.4.0-SNAPSHOT to 2.0.0-SNAOSHOT for master > > branch > > > > Hi all, > > > > Given the significant changes that the table model introduces to IoTDB and > > the associated upgrade from TsFile version V3 to V4, it is reasonable to > > consider updating the version on the master branch to 2.0.0-SNAPSHOT. This > > approach aligns with the intention to release the table model with the > > version number 2.0.0, signifying a major update to the system. > > > > For the tree model, which will continue to evolve with bug fixes and > > performance improvements on the 1.X.X series (such as the upcoming 1.3.3 > > version), it's important to maintain clear versioning to differentiate > > between the stable, incremental updates and the more substantial changes > > introduced by the table model. What do you think? Best regards, > > > > > > > > Yuan Tian > >
Re: [VOTE] Apache IoTDB 1.3.2 RC1 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] Thanks, Jialin Qiao Christofer Dutz 于2024年6月28日周五 19:15写道： > > +1 (binding) > > Chris > > [OK] Download all staged artifacts under the url specified in the release > vote email. > [OK] Verify the signature is correct. > [OK] Check if the signature references an Apache email address. > [OK] Verify the SHA512 hashes. > [OK] Unzip the archive. > [OK] Verify the existence of LICENSE, NOTICE, README, RELEASE_NOTES files in > the extracted source bundle. > [MINOR] Verify the content of LICENSE, NOTICE, README, RELEASE_NOTES files in > the extracted source bundle. > > * RELEASE_NOTES hasn’t been updated for quite some time. > * NOTICE: Could possibly need an update on the Apache Commons Collections > section, as we’re most probably using versions released after 2019. > [MINOR] Run RAT externally to ensure there are no surprises. > > * Integration-test module contains 3 archives, 1 of them I couldn’t find > the sources to (TriggerFireTimesCounter) > [MINOR] Search for SNAPSHOT references > > * It seems the following projects were not part of the release and > therefore their version was not updated: > * example/client-cpp-example > * iotdb-client/client-cpp > [OK] Search for Copyright references, and if they are in headers, make sure > these files containing them are mentioned in the LICENSE file. > > > Von: 冯 庆新 > Datum: Donnerstag, 27. Juni 2024 um 15:05 > An: dev@iotdb.apache.org > Betreff: 回复: [VOTE] Apache IoTDB 1.3.2 RC1 release > Hello all : > +1 > > The binary distribution: > version number in CLI [ok] > start in CentOS7, jdk11 [ok] > performance verification passed: [ok] > ### Server Configurations ###// > CPU=16 > Memory=32G > Disk=1.8T HDD > Ethernet=1000Mbit > /// > ### IoTDB Configurations ###/// > MAX_HEAP_SIZE="20G" in datanode-env.sh > /// > ### Client Mode ### > Insert Non-Aligned/Aligned timeseries with SESSION_BY_TABLET > CLIENT_NUMBER=10 > GROUP_NUMBER=10 > DEVICE_NUMBER=50 > SENSOR_NUMBER=500 > BATCH_SIZE_PER_WRITE=10 > LOOP=86400 > /// > > ### Test Result ### > 1C1D: Timeseries Num : 25,000；Loaded 21,600,000,000points； > Non-Aligned Timseries: > Throughput(points/s): 17,009,956.18 Cost Time(s): 1,269.84 > Latency(ms): Avg 2.61, Min 0.67, MiddleAvg 0.83, Max 6,783.48 > Aligned Timseries: > Throughput(points/s): 18,141,900.23 Cost Time(s): 1,190.61 > Latency(ms): Avg 2.44, Min 0.79, MiddleAvg 0.99, Max 6,546.48 > /// > > > Avg: Average time cost of all ingestion operations. [ms] > MiddleAvg: Average time cost of ingestion operations without 5% head and > tail. [ms] > Min: Min single operation time cost of all ingestion operations. [ms] > Max: Max single operation time cost of all ingestion operations. [ms] > These results are tested with > iot-benchmark(https://github.com/thulab/iot-benchmark) > That's all. > Please feel free to contact me if you have any questions > Thank you. > Best Regards! > Qingxin Feng > > 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送 > > > 发件人: Haonan Hou > 发送时间: Thursday, June 27, 2024 5:55:36 PM > 收件人: dev@iotdb.apache.org > 主题: [VOTE] Apache IoTDB 1.3.2 RC1 release > > Hi all, > > Apache IoTDB 1.3.2 has been staged under [2] and it’s time to vote > on accepting it for release. All Maven artifacts are available under [1]. > Voting will be open for 72hr. > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > are required to pass. > > Release tag: v1.3.2 > Hash for the release tag: aa0ff4adf2fe00368b2145dd5f561d30df06a885 > > Before voting +1, PMC members are required to download > the signed source code package, compile it as provided, and test > the resulting executable on their own platform, along with also > verifying that the package meets the requirements of the ASF policy > on releases. [3] > > You can achieve the above by following [4]. > > [ ] +1 accept (indicate what you valida
Re: Introducing Apache IoTDB use in COVESA Central Data Service Playground
Hi, Glad to see the integration! Welcome and feel free to ask :D Jialin Qiao Pengcheng Zheng 于2024年6月15日周六 04:38写道： > > Hi Stephen, > > Very exciting project! > > Thanks for sharing the details of COVESA/CDSP. I remember that you asked > some questions in Slack several months ago, and seems that you have made > great progress ;-) > > IoTDB has already proved its excellence in the scenario of connected > vehicles before, and the key questions you've listed in the > presentation happen to be exactly what IoTDB & TsFile are addressing, such > as data syncing between all touchpoints/ bidirectional sync/ permissions & > privacy/ model updating/ subscription, etc. > > Also your ideas of the deployment in Docker and edge-cloud sync (if my > memory serves me right..) shall benefit a lot from IoTDB. > > Hopefully IoTDB could support the CDSP project well and help transform the > "Playground" to POC - and finally from theory to practice. > > Looking forward to your updates! > > > Best Wishes, > Pengcheng Zheng > Timecho Europe > > Am Fr., 14. Juni 2024 um 21:12 Uhr schrieb Stephen Lawrence < > stephen.lawre...@renesas.com>: > > > Hi, > > > > As suggested in the wiki I am introducing myself and a new OSS project > > using Apache IoTDB. > > > > My name is Stephen Lawrence from Renesas Electronics and I work in the > > COVESA[1] open automotive alliance. COVESA is focused on developing common > > approaches and technologies for connected vehicles, such as the VSS data > > model [2]. I lead the COVESA Data Architecture and Infrastructure group and > > am a Board member. > > > > We recently released the first version of the OSS COVESA Central Data > > Service Playground (CDSP)[3] which uses Apache IoTDB as a data > > store/server. The Playground is a neutral, open playground for > > investigating data services in-vehicle and off-board in the context of > > data-centric architectures. Importantly in community terms it provides a > > means to publish and collaborate on such work in the open. > > > > Constructed as a "project of projects" this first release was focused on > > creating the first version of the "toolkit". Now we pivot to improving it, > > creating examples for its use and investigating industry issues. Currently > > we are focused on Docker deployment for ease of development and connections > > to other technology. Deployment on Automotive h/w is part of the future > > roadmap. The contributors include various large vehicle OEMs. > > > > At a recent COVESA Conference I demonstrated this first release. The > > online documentation is a work in progress but contains a summary of the > > What/Why/How of the project. If you are interested in more details I led a > > track of sessions about the Playground at the conference, the presentation > > for which you can find here [4]. > > > > I know I am going to have an increasing number of questions in the coming > > months, so I wanted to first introduce myself and the project. I hope to be > > able to contribute back to the IoTDB project as well. > > > > Apologies for the corporately expanded reference URLs in plain text: > > [1] http://www.covesa.global/ > > [2] https://covesa.global/project/vehicle-signal-specification/ > > [3] https://github.com/COVESA/cdsp > > [4] > > https://wiki.covesa.global/download/attachments/98271360/Central_Data_Service_Playground_track_v2.pptx?version=1&modificationDate=1714486822982&api=v2 > > > > Best Wishes, > > > > Stephen Lawrence > > COVESA Data Architecture Lead and Board Member > > (Renesas Electronics) > >
Re: ALIGN BY DEVICE: the data types of the same measurement column should be the same across devices.
Hi, We just changed the default infer type of integer and floating to DOUBLE [1], it will be released in 1.3.2. For <=1.3.1, you could explicitly set this to DOUBLE. [1] https://github.com/apache/iotdb/pull/12223 Jialin Qiao Trevor Hart 于2024年6月12日周三 06:41写道： > > Hello Team > > > > I have a customer using my application with IoTDB 1.3.0 in the backend. > > > > Auto create schema is enabled as there is no fixed template. Multiple devices > are pushing data to IotDB. > > > > I have seen some ALIGN BY DEVICE queries fail with the error in the subject ie > > > > ALIGN BY DEVICE: the data types of the same measurement column should be the > same across devices. > > > > The "show timeseries" shows that from one device the value has been inferred > as FLOAT and another as DOUBLE. > > > Per the documentation it is suggested that the default > "floating_string_infer_type" is DOUBLE but that is not what I am seeing. Why > am I getting FLOAT? Is the default "floating_string_infer_type" not being > honoured? > > > > Currently in the config file "floating_string_infer_type" is commented out. > Should I explicitly set this to DOUBLE? > > > Thanks > > Trevor Hart
Re: IoTDB quarter report (2024 Q2)
Looks good! Jialin Qiao Xiangdong Huang 于2024年6月12日周三 09:11写道： > > Hi all, > > This is the draft of IoTDB quarter report > > ## Description: > The mission of Apache IoTDB is the creation and maintenance of software > related > to an IoT native database with high performance for data management and > analysis > > ## Project Status: > Current project status: Ongoing with high activity. > Issues for the board: > - there are some trademark issues but the community has resolved them. > > ## Membership Data: > Apache IoTDB was founded 2020-09-16 (4 years ago) > There are currently 63 committers and 29 PMC members in this project. > The Committer-to-PMC ratio is roughly 2:1. > > Community changes, past quarter: > - No new PMC members. Last addition was Steve Yurong Su on 2023-09-28. > - William Song was added as committer on 2024-03-22 > - Caiyin Yang was added as committer on 2024-06-12 > > ## Project Activity: > - Recent releases: > IOTDB-1.3.2 is on the way. > IOTDB-1.3.1 was released on 2024-04-22. > IOTDB-1.3.0 was released on 2024-01-01. > IOTDB-1.2.2 was released on 2023-10-15. > > - Main work of project: > * We are working on the table model providing data to users in a table view > in the same way as relational databases which will greatly reduce the > learning curve of IoTDB. > * Collaborate with the Apache StreamPipes community to improve the IoTDB > plugin > and explore other means on integration. > * New query optimization rules are added to greatly improve the query > performance of IoTDB, like PredicatePushDown and AggregatePushDown. > * Explain Analyze tool which can be used to analyze specific query > performance better and is a good tool for troubleshooting performance problems > > > ## Community Health: > Overall community health is good. > - Joined "Digital Economy-Urban Open Source Tour" meetup in Shanghai. > - the mailing list traffic increases (226 emails compared to 76) > > Best, > --- > Xiangdong Huang
Re: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)
Thank Chris! Jialin Qiao Christofer Dutz 于2024年6月4日周二 17:28写道： > > Ok … artifacts should be available … > > Chris > > Von: Christofer Dutz > Datum: Dienstag, 4. Juni 2024 um 11:26 > An: dev@iotdb.apache.org > Betreff: [RESULT] [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache > Thrift 0.20.0) > So, the vote passes with 7 +1 votes at least 5 I could verify as binding > votes. > > In the future, it would be great if people would add their name/apacheId etc. > to the votes to make it easier to verify and count for the RM. > > Releasing the maven artifacts right away. > > Thanks for all those who voted. > > Chris > > > Von: Christofer Dutz > Datum: Dienstag, 4. Juni 2024 um 09:37 > An: dev@iotdb.apache.org > Betreff: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift > 0.20.0) > Yeah.. Currently at community over code... Was planning on doing this in a > break today. > > Gesendet von Outlook für Android<https://aka.ms/AAb9ysg> > > From: Haonan Hou > Sent: Monday, June 3, 2024 4:11:04 AM > To: dev@iotdb.apache.org > Subject: Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift > 0.20.0) > > Hi Chris, > > I think 72 hours has passed. These artifacts can be released now? > > Thanks, > Haonan > > On 2024/05/28 14:40:39 Christofer Dutz wrote: > > Hi all, > > > > I after fixing some issues in the cmake-maven-plugin, have I updated the > > iotdb-bin-resources module to use these. > > Now we can stage Apache Thrift binaries for all platforms. > > > > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible > > again), I also updated the Thrift version and staged a release of Thrift > > binaries for all of our supported platforms in Nexus. > > > > I’ve staged the artifacts at: > > https://repository.apache.org/content/repositories/orgapacheiotdb-1158 > > > > Tag Name: iotdb-tools-thrift-v0.20.0.0 > > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0 > > > > The source bundle generally consists of a pom.xml and an assembly xml … so > > it should be quick to review. > > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip > > > > I’m intentionally not staging it on the release-svn, as nobody should be > > required to consume this module but us ourselves. > > > > The vote stays open for 72hours … assuming enough support, I’ll then finish > > changing the main repo to use these artifacts. > > > > > > Chris > >
Re: [Proposal] Merge configuration files to iotdb-system.properties
Hi, > I think it would be good If the code for accessing the options would not be > merged. +1, I think the code is not merged in Wenwei's PR. Jialin Qiao Caiyin Yang 于2024年6月3日周一 10:27写道： > > Hi Xinyu, > > If that's the case, then this merge seems to be very user-friendly. > > Best > Caiyin Yang > > Xinyu Tan 于2024年6月3日周一 10:22写道： > > > Hi, Caiyin > > > > Following Wenwei's work, we plan to introduce an > > iotdb-system.properties.template file that will contain all configuration > > items, relevant comments, and optional settings, similar to the latest user > > configuration manual. In the future, the iotdb-system.properties file will > > only include explicitly configured items by the user (such as IP addresses > > or manually configured storage engine parameters). Compared to the current > > method of finding and modifying configurations in the lengthy > > iotdb-confignode.properties and iotdb-datanode.properties files, this > > approach seems more convenient. If you have any good ideas, feel free to > > discuss your preferred method further! > > > > Best > > > > Xinyu Tan > > > > On 2024/06/01 09:18:48 ycy wi wrote: > > > Hi wenwei, > > > > > > I am curious whether this will ultimately lead to an oversized or > > redundant configuration file. I might just want to modify some > > configuration items in the dataNode. In theory, I only need to pay > > attention to the configuration items in iotdb-datanode.properties. Will the > > merge cause any confusion due to similar configuration items in > > iotdb-confignode.properties? > > > > > > Best > > > CaiyinYang > > > > > > > > > > > > > > > > > > > 2024年5月31日 21:20，wenwei shu 写道： > > > > > > > > Hello everyone, > > > > > > > > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are > > > > working on merging `iotdb-confignode.properties`, > > > > `iotdb-datanode.properties` and `iotdb-common.properties` into a new > > > > `iotdb-system.properties` file. For old version users who upgrade to a > > new > > > > version, if they don't create the new configuration file themselves > > during > > > > the upgrade, IoTDB will generate the new configuration file based on > > > > several old configuration files after it is started. You can check > > this PR > > > > to get more details: https://github.com/apache/iotdb/pull/12570. > > > > Thank you for your reading. > > > > > > > > Best regards, > > > > Wenwei Shu > > > > > > > >
Re: [Proposal] Merge configuration files to iotdb-system.properties
Hi, Good feature! This will make it easy for users to set configs. Jialin Qiao wenwei shu 于2024年5月31日周五 21:23写道： > > Hello everyone, > > I am Wenwei Shu, a new contributor to Apache IoTDB. Recently, we are > working on merging `iotdb-confignode.properties`, > `iotdb-datanode.properties` and `iotdb-common.properties` into a new > `iotdb-system.properties` file. For old version users who upgrade to a new > version, if they don't create the new configuration file themselves during > the upgrade, IoTDB will generate the new configuration file based on > several old configuration files after it is started. You can check this PR > to get more details: https://github.com/apache/iotdb/pull/12570. > Thank you for your reading. > > Best regards, > Wenwei Shu
Re: Could anyone please approve this PR?
This is merged :) Jialin Qiao Christofer Dutz 于2024年5月28日周二 22:23写道： > > Hi all, > > in the iotdb-bin-resources I have also updated the Cmake version and thrift > version in order to be able to deploy new versions of thrift artifacts. > > https://github.com/apache/iotdb-bin-resources/pull/61 > > Would be cool if I could get some approval. > > Chris
Re: [VOTE] Release IotDB-Tools: Thrift artifacts (for Apache Thrift 0.20.0)
+1 (binding) Jialin Qiao Yuan Tian 于2024年5月29日周三 09:07写道： > > +1 (binding) > > On Tue, May 28, 2024 at 10:40 PM Christofer Dutz > wrote: > > > Hi all, > > > > I after fixing some issues in the cmake-maven-plugin, have I updated the > > iotdb-bin-resources module to use these. > > Now we can stage Apache Thrift binaries for all platforms. > > > > In order to test, if Apache Thrift 0.20.0 (Which is Java 8 compatible > > again), I also updated the Thrift version and staged a release of Thrift > > binaries for all of our supported platforms in Nexus. > > > > I’ve staged the artifacts at: > > https://repository.apache.org/content/repositories/orgapacheiotdb-1158 > > > > Tag Name: iotdb-tools-thrift-v0.20.0.0 > > > > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.20.0.0 > > > > The source bundle generally consists of a pom.xml and an assembly xml … so > > it should be quick to review. > > > > https://repository.apache.org/content/repositories/orgapacheiotdb-1158/org/apache/iotdb/tools/iotdb-tools-thrift/0.20.0.0/iotdb-tools-thrift-0.20.0.0-source-release.zip > > > > I’m intentionally not staging it on the release-svn, as nobody should be > > required to consume this module but us ourselves. > > > > The vote stays open for 72hours … assuming enough support, I’ll then > > finish changing the main repo to use these artifacts. > > > > > > Chris > >
Re: Missing binaries for Windows on Aarch64 systems
Hi, Is this will also be fixed by upgrading thrift to 0.20? Jialin Qiao Christofer Dutz 于2024年5月24日周五 22:58写道： > > Hi all, > > I was just verifying the build using the most recent cmake-maven-plugin works > and stumbled over something. > > Currently we don’t have thrift binaries for combinations of systems. It seems > we’re missing windows-aarch64. > > Should we plan fixing this and doing a release 0.14.1.1? > > Chris
Re: [DISCUSS] Investigate updating to Thrift 0.20.0?
Hi, Looks good :D If the upgrading does not bring any change to our users, +1 for upgrading. You could prepare a PR, then we perform some validations. Jialin Qiao Christofer Dutz 于2024年5月28日周二 03:41写道： > > Hi all, > > while doing some experiments over the weekend I came to notice that Thrift > 0.19.0 and above is again compatible with Java 8 … this was the major reason > for us not updating to versions without CVEs. > I’ve tried out 0.20.0 and my local fork of IoTDB runs perfectly with it. > > https://github.com/apache/thrift/blob/master/CHANGES.md > > Might be worth investigating to update? I’ve already got an up-to-date > version of the thrift compiler … if we wanted to try it out, I could stage > release versions for: > > > * Mac – aarch64 > * Mac – x86_64 > * Linux – aarch64 > * Linux – x86_64 > * Windows – aarch64 > * Windows – x86_64 > > Chris
Re: Upgrading 0.12 -> 1.3
Hi, The architecture of 1.x is different from 0.13. The upgrading can not be in situ. You need to deploy an 1.3 instance, then transfer the data. There are two ways to transfer data: 1. Using Session to query from the 0.13, then write into 1.3. 2. Using TsFile API to rewrite the TsFile, then load TsFiles to 1.3. 1. is easier. You could refer to this: https://github.com/apache/iotdb/blob/master/example/session/src/main/java/org/apache/iotdb/DataMigrationExample.java Jialin Qiao Trevor Hart 于2024年5月28日周二 06:27写道： > > Hello > > > > I have an application that is running against IoTDB version 0.12 which I > would like to upgrade to 1.3 > > > > However I have some timeseries paths that contain numbers (eg below) > > > root.ABC.health.url.17.code > > > > From the documentation it seems this is not supported beyond 0.13 > > > > To migrate this data I am proposing this process; > > > > 1. Use "select into" to migrate to a new path ie root.ABC.health.url.17.code > -> root.ABC.health.url.U17.code > > 2. Delete the old timeseries ie root.ABC.health.url.17.code > > 3. Upgrade IoTDB > > > Is this the correct/best approach? > > > Thanks > > Trevor Hart
Re: Windows Installer
Hi, That's Great! If adding this installer, how much will our distribution package increase? Jialin Qiao Wang Critas 于2024年5月20日周一 09:57写道： > > Hi Trevor, > > So cool. > > Perhaps you can integrate it into iotdb/distribution. that would be more great > > Xuan Wang > > 发件人: Trevor Hart > 日期: 星期一, 2024年5月20日 09:48 > 收件人: Dev > 主题: Windows Installer > Hello > > > > I dont know if there is some interest in this but I have built a Windows > installer for installing IoTDB on Windows servers. > > > > The installer includes Open JDK 11 and also does the following; > > > > 1. Creates a Windows service for the Config Node > > 2. Creates a Windows service for the Data Node > > 3. Open port 6667 on Windows firewall > > > > You can see the installer in action here; > https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D6BzCd-vAiGc%26t%3D1s%26ab_channel%3DOpeLtd&data=05%7C02%7C%7Cc4c099493f3e49494ae008dc786ef772%7C84df9e7fe9f640afb435%7C1%7C0%7C638517665208767354%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=seKLvYsiEkBQsjdw6KFaIIUvL1%2FFtv7G36VQC2HVyb8%3D&reserved=0<https://www.youtube.com/watch?v=6BzCd-vAiGc&t=1s&ab_channel=OpeLtd> > > > > If there is any interest in this I can make it available. > > > Thanks > > Trevor Hart
Re: Handling Duplicate Timestamps
Hi Trevor, Yes, IoTDB cannot handle this scenario currently because our primary key is Path + Timestamp. This year we will focus on the table model, a lot work to do :-) Jialin Qiao Trevor Hart 于2024年5月20日周一 09:45写道： > > Hi Jialin > > > > Yes the values would be different. > > > > As as example, these are from a web server log. The device is openzweb01 > which is an IIS web server which may handle multiple requests at the same > time. The rows are unique in their own right but the timestamp is the same in > the logging. > > > > 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7 > > > 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6 > > > 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6 > > 2024-05-20 00:00:14 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5 > > > 2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6 > > 2024-05-20 00:00:15 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6 > > > > This particular log file only records in seconds. So what I am doing with > these rows at the moment is to add an artitifical millisecond to enforce > uniqueness. > > > > > 2024-05-20 00:00:14.000 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/Meriadoc 200 0 0 3339 503 7 > > 2024-05-20 00:00:14.001 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/Peregrin 200 0 0 3327 503 6 > > 2024-05-20 00:00:14.002 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/Samwise 200 0 0 3325 502 6 > > 2024-05-20 00:00:14.003 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/siteadmin 200 0 0 15279 504 5 > > 2024-05-20 00:00:15.000 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/testuser 200 0 0 1794 503 6 > > 2024-05-20 00:00:15.001 W3SVC1 openzweb01 192.168.3.69 POST > /portal/sharing/rest/community/users/testuser2 200 0 0 1794 506 6 > > > > For some other log files that I am processing they are in milliseconds > already but there is a (small) chance of dataloss if multiple requests happen > to be processed at the exact same time. > > > > I have been thinking about this some more and I think that rather than break > the IoTDB CRUD model I should handle this on the client side. In my use case > the log data is actually staged in an H2 database before it is sent to IoTDB > so I can enforce PK validation there. That way it is less expensive that > checking the timestamp in IoTDB for each record. > > > > Thanks > > Trevor Hart > > > > > > > > > On Fri, 17 May 2024 19:11:13 +1200 Jialin Qiao > wrote --- > > > > Hi Trevor, > > Will different values of the same timestamp be the same？ > > 1. Same > Time, Value > 1, 1 > 1, 1 > 1, 1 > > 2. Different > Time, Value > 1, 1 > 1, 2 > 1, 1 > > > Jialin Qiao > > Trevor Hart <mailto:tre...@ope.nz> 于2024年5月14日周二 11:20写道： > > > > Thank you! I will implment some work around for now. > > > > > > I would appreciate some consideration for this option in the future. > > > > > > Thanks > > > > Trevor Hart > > > > Ope Limited > > > > w: http://www.ope.nz/ > > > > m: +64212728039 > > > > > > > > > > > > > > > > > > On Tue, 14 May 2024 15:17:47 +1200 Xiangdong Huang > > <mailto:saint...@gmail.com> wrote --- > > > > > > > > > 1. Checking before insert if the timestamp already exists and remedy on > > > the client before resend > > > 2. Moving to Nanosecond and introducing some insignificant time value to > > > keep timestamp values unique. > > Yes these maybe the best solutions for a specific application. > > > > > > Analysis for IoTDB: > > - Rejecting the write when receiving an existing timestamp in IoTDB is > > time-costly (IoTDB needs to check historical data). I think we will do > > not check it until we find a low-latency method. > > - Allowing multiple value versions for a timestamp may introduce a > > chain reaction and there may be a lot of codes that should be > > modified, which is a huge work. > > > > There is
Re: [DISCUSS] Drop Java 8?
Hi, Drop java 8 will bring some CVE fixed, but also reduce a common option for users, for this, I will -0. Maintaining two branch will surely resolve the issue, but this will bring much additional work... For this, I will -1 Although we put 0.13 and 1.x version in our download page, we actually do not maintain the 0.x branch and there is on plan to release a new 0.x version. So, as there is a lot to do with 1.x this year (the table schema, data subscription, tsfile consensus), I prefer not to do the drop this year. Jialin Qiao Christofer Dutz 于2024年5月17日周五 14:43写道： > > So we have real numbers? > > I also haven't used an oracle jre for quite some time for these reasons. > > Would a double branch strategy work? That the project maintains a 1.x branch > with Javax and Java 8 and a 2.0 branch with Jakarta and Java 11? > > Sort of like we had the 0.x and 1.x branches around? > > Chris > > > Gesendet von Outlook für Android<https://aka.ms/AAb9ysg> > > From: Trevor Hart > Sent: Friday, May 17, 2024 4:01:14 AM > To: dev > Subject: Re: [DISCUSS] Drop Java 8? > > I think a lot of organisations are sticking to Java 8 because of the change > to Oracle license that was introduced in Java 11. > > > > If you use Oracle 11 JRE you need to pay Oracle for a license. > > > > This was why Open JDK came about. > > > > Personally I use IotDB with Open JDK 11 (Eclipse Temurin) which does not > require a license. > > > > Thanks > > Trevor Hart > > > > > > > > > On Fri, 17 May 2024 13:30:15 +1200 Yuan Tian > wrote --- > > > > Hi Chris, > > It seems that a lot of people still use jdk1.8 in their product environment. > > Best regards, > - > Yuan Tian > > On Thu, May 16, 2024 at 8:10 PM Christofer Dutz > <mailto:christofer.d...@c-ware.de> > wrote: > > > Hi all, > > > > starting this new thread as I am not sure if others are reading the > > Jakarta migration thread. > > > > I would like to propose planning on dropping Java 8 support. > > > > I wouldn’t immediately do that, and I would also propose to do a major > > version update (Switching to 2.0.0) > > > > We could still maintain a 1.x branch for those people not able to update. > > > > The main reason is that we are currently blocking ourselves from updating > > many major plugins and dependencies. > > I noticed that when updating to the Jakarta namespace. Here there is no > > Netty version available that supports Jakarta and supports Java 8. > > > > Other libraries where we are not able to update without giving up on Java > > 8: > > > > * Airlift-Units (Stuck at 1.7 current 1.10) > > * Airlift > > * Antlr (Stuck at 4.9.3 current 4.13.1) > > * Caffeine (Stuck at 2.9.3 current 3.1.8) > > * Logback (Stuck at 1.3.14 current 1.5.6) > > * Mockito (Stuck at 2.23.4 current 5.12.0) > > * Thrift (Stuck at 0.17.0 current 0.20.0) > > > > > > > > * Spotless Plugin (We’ve got a workaround for Java 8) > > > > > > In my branch where I refactored the javax namespace to Jakarta after > > updating dependencies I was able to remove all exclusions of the > > BanVulnerableDependencies check. > > > > Also does dropping Java 8 and the Jakarta migration allow embedding IoTDB > > in recent Spring versions. > > > > > > So … what do you think? > > > > > > Chris > >
Re: Possible fix for randomly failing Jenkins builds
+1 for importing the solutions in PLC4X Jialin Qiao Christofer Dutz 于2024年5月14日周二 23:32写道： > > Hi all, > > I know that in IoTDB as well as PLC4X we were having issues of regularly but > randomly failing Jenkins builds. > > Usually, these problems were because of timeouts when deploying to Nexus. > > In PLC4X I have now tried something that seems to be doing the job … possibly > this could also help us here (I mean … the biggest part of the Jenkins setup > is borrowed from PLC4X anyway ;-) ) > > My solution was to wrap the execution of the deploy step in a loop on bash > level: > > https://github.com/apache/plc4x/blob/develop/Jenkinsfile#L167 > > sh 'until ./mvnw -f jenkins.pom -X -P deploy-snapshots > wagon:upload || (( count++ >= 5 )); do echo "Retrying to deploy"; done' > > So, it would try up to 5 times, if deployment fails. > > I think in case of deployment the full build is done and was successful. The > only reason for failures in this step usually is problems with Nexus and a > stupid retry should help. > > What do you think? Should we try the same change? > > > Chris
Re: Handling Duplicate Timestamps
Hi Trevor, Will different values of the same timestamp be the same？ 1. Same Time, Value 1, 1 1, 1 1, 1 2. Different Time, Value 1, 1 1, 2 1, 1 Jialin Qiao Trevor Hart 于2024年5月14日周二 11:20写道： > > Thank you! I will implment some work around for now. > > > I would appreciate some consideration for this option in the future. > > > Thanks > > Trevor Hart > > Ope Limited > > w: http://www.ope.nz/ > > m: +64212728039 > > > > > > > > > On Tue, 14 May 2024 15:17:47 +1200 Xiangdong Huang > wrote --- > > > > > 1. Checking before insert if the timestamp already exists and remedy on the > > client before resend > > 2. Moving to Nanosecond and introducing some insignificant time value to > > keep timestamp values unique. > Yes these maybe the best solutions for a specific application. > > > Analysis for IoTDB: > - Rejecting the write when receiving an existing timestamp in IoTDB is > time-costly (IoTDB needs to check historical data). I think we will do > not check it until we find a low-latency method. > - Allowing multiple value versions for a timestamp may introduce a > chain reaction and there may be a lot of codes that should be > modified, which is a huge work. > > There is a new idea (but I have no time to implement it...) > - Add a parameter in IoTDB: replace_strategy: first, last, avg etc... > - when an existing timestamp arrives, IoTDB accepts it > - when IoTDB runs LSM to merge data and meets multiple values for a > timestamp, then handles it according to the replace_startegy. > > The solution may also introduce some work to do... and we need to > think carefully the impact to the query process. > Need to survey whether this is a common requirement. > > Best, > --- > Xiangdong Huang > > Trevor Hart <mailto:tre...@ope.nz> 于2024年5月14日周二 09:55写道： > > > > Hello Yuan > > > > > > > > Correct, the first timestamp and values should be retained. > > > > > > > > I realise this is does not align with the current design. I was just asking > > whether there was an existing option to operate to block duplicates. > > > > > > > > In a normal RDBMS if you try to insert with a duplicate the insert will > > fail with a PK violation. It would be great in some circumstances if IotDB > > at least had the option to fail this way. > > > > > > > > I am considering some options such as; > > > > > > > > 1. Checking before insert if the timestamp already exists and remedy on the > > client before resend > > > > 2. Moving to Nanosecond and introducing some insignificant time value to > > keep timestamp values unique. > > > > > > > > I have already done something similar to #2 with storing IIS web log files > > as they are recorded in seconds and not milliseconds. > > > > > > > > Thanks > > > > Trevor Hart > > > > > > > > > > On Tue, 14 May 2024 13:29:02 +1200 Yuan Tian > > <mailto:jackietie...@gmail.com> wrote --- > > > > > > > > Hi Trevor, > > > > By "rejects duplicates", you mean you want to keep the first duplicate > > timestamp and its corresponding values?(because the following duplicated > > ones will be rejected) > > > > Best regards, > > > > Yuan Tian > > > > On Mon, May 13, 2024 at 6:24 PM Trevor Hart <mailto:mailto:tre...@ope.nz> > > wrote: > > > > > > > > > > > > > > > > > Correct. I’m not disputing that. What I’m asking is that it > > > would be good to have a configuration that either allows overwrites or > > > rejects duplicates.My scenario is request log data from a server (the > > > device). As it may be processing multiple requests at once there is a > > > chance that there could be colliding time stamps.As it stands now I would > > > need to check if the timestamp exists before inserting the data. Which > > > obviously affects throughput. Thanks Trevor Hart On Fri, 10 May > > > 2024 00:33:40 +1200 Jialin Qiao<mailto:mailto:qiaojia...@apache.org> > > > wrote Hi, > > > In IoT or IIoT scenarios, we thought each data point represent a metric of > > > a timestamp.In which case you need to store duplicated values? Take this > > > for an example: Time, root.sg1.car1.speed 1, 1 1, 2 Could a car has > > > different speed at time 1? Jialin Qiao Yuan Tian < > > >
Re: Handling Duplicate Timestamps
Hi, In IoT or IIoT scenarios, we thought each data point represent a metric of a timestamp.In which case you need to store duplicated values? Take this for an example: Time, root.sg1.car1.speed 1, 1 1, 2 Could a car has different speed at time 1? Jialin Qiao Yuan Tian 于2024年5月9日周四 18:51写道： > > Hi Trevor, > > Now we will override the duplicate timestamp with a newer one. There is > nothing we can do about it now. > > Best regards, > --- > Yuan Tian > > On Wed, May 8, 2024 at 5:31 PM Trevor Hart wrote: > > > Hello > > > > > > > > I’m aware that when inserting a duplicate timestamp the values will be > > overwritten. This will obviously result in data loss. > > > > > > > > Is there a config/setting to reject or throw an error on duplicate > > inserts? Although highly unlikely I would prefer to be alerted to the > > situation rather than lose data. > > > > > > > > I read through the documentation but couldn’t find anything. > > > > > > > > Thanks > > > > Trevor Hart
Re: [VOTE] Apache IoTDB 1.3.1 RC2 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] could compile from source: ./mvnw.sh clean install [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] statements executed successfully: [ok] SET STORAGE GROUP TO root.turbine; CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; insert into root.turbine.d1(timestamp,s0) values(1,1); insert into root.turbine.d1(timestamp,s0) values(2,2); insert into root.turbine.d1(timestamp,s0) values(3,3); insert into root.turbine.d2(time,s0) values(now(),1); insert into root.turbine.d2(time,s0) values(now(),2); insert into root.turbine.d2(time,s0) values(now(),3); select ** from root; select ** from root align by device; Jialin Qiao Yuan Tian 于2024年4月22日周一 08:52写道： > > Hi, > > +1 (binding) > > The source release: > apache headers [ok] > signatures and hashes [ok] > LICENSE and NOTICE [ok] > no jar files [ok] > could compile from source: ./mvnw.sh clean install [ok] > > The binary distribution: > version number in CLI [ok] > signatures and hashes [ok] > start in mac, jdk8 [ok] > statements executed successfully: [ok] > > SET STORAGE GROUP TO root.turbine; > CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; > insert into root.turbine.d1(timestamp,s0) values(1,1); > insert into root.turbine.d1(timestamp,s0) values(2,2); > insert into root.turbine.d1(timestamp,s0) values(3,3); > insert into root.turbine.d2(time,s0) values(now(),1); > insert into root.turbine.d2(time,s0) values(now(),2); > insert into root.turbine.d2(time,s0) values(now(),3); > select ** from root; > select ** from root align by device; > > > Best, > - > Yuan Tian > > On Fri, Apr 19, 2024 at 5:14 PM Haonan Hou wrote: > > > Hi all, > > > > Apache IoTDB 1.3.1 has been staged under [2] and it’s time to vote > > on accepting it for release. All Maven artifacts are available under [1]. > > Voting will be open for 72hr. > > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > > are required to pass. > > > > Release tag: v1.3.1 > > Hash for the release tag: 214695df59915884ca43767f944ed7b921763864 > > > > Before voting +1, PMC members are required to download > > the signed source code package, compile it as provided, and test > > the resulting executable on their own platform, along with also > > verifying that the package meets the requirements of the ASF policy > > on releases. [3] > > > > You can achieve the above by following [4]. > > > > [ ] +1 accept (indicate what you validated - e.g. performed the > > non-RM items in [4]) > > [ ] -1 reject (explanation required) > > > > > > [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1157 > > [2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.1/rc2 > > [3] https://www.apache.org/dev/release.html#approving-a-release > > [4] > > https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release > > [5] https://github.com/apache/iotdb/blob/v1.3.1/RELEASE_NOTES.md > > [6] https://dist.apache.org/repos/dist/dev/iotdb/KEYS > > > > Best, > > > > Haonan Hou > >
Re: Remove TsFile module in IoTDB Project
Looks good! Jialin Qiao Yuan Tian 于2024年4月19日周五 22:01写道： > > Hi all, > > I've just merged this pr(https://github.com/apache/iotdb/pull/12283), now > tsfile module source code have all been removed from IoTDB project and > using pom to depend on TsFile Project. > > > > Best regards, > -- > Yuan Tian
Re: Archive apache/iotdb-web-workbench repo?
+1 for archive Jialin Qiao Haonan Hou 于2024年4月18日周四 11:23写道： > > Hi all, > > We received some security issues about apache/iotdb-web-workbench recently. > However, we don't have enough maintainers to fix them. The repo has no new > commit since Apr 3, 2023. There is an option that we can archive that repo. > > Any idea? > > Best, > Haonan Hou
Re: Splitting up the repos
Hi, The examples could be classified into two parts (1) Connector example: flink, hadoop, kafka, pulsar, rabiitmq, rocketmq (2) IoTDB native api example: jdbc, mqtt, pipe, rest, schema, session, trigger, udf For (1), we could move into extra repo. For (2) , they should be retained in the IoTDB repo. Jialin Qiao Christofer Dutz 于2024年4月15日周一 17:56写道： > > Hmpf, > > a little bit sooner reply would have been good … I’m already done with the > changes, also with moving all examples and the parts of the distribution > bundling the connectors. > > I do think also moving the examples is a good idea. Usually, examples pull in > all sorts of dependencies, which show up on vulnerability reports. Also do we > have some examples that refer to stuff we now moved out of the main repo, > we’d be getting a cyclic dependency from that, so we would have to split up > the examples in that case. > > So, if we were to vote on this (which we can) I would vote +1 on moving all > examples out of the main repo. > > Chris > > > Von: Jialin Qiao > Datum: Montag, 15. April 2024 um 11:20 > An: dev@iotdb.apache.org > Betreff: Re: Splitting up the repos > Hi, > > 1. Which Parts: Only iotdb-connector need to be moved, distribution > and examples will impact the release and users. > 2. How to split up: I prefer【Simply ignore the history, copy the > files to the new repo and delete them from the old】. > > Jialin Qiao > > Christofer Dutz 于2024年4月15日周一 16:27写道： > > > > Hi all, > > > > > > So, I’ve set a tag on the main repository “before-moving-extras” > > (https://github.com/apache/iotdb/releases/tag/before-moving-extras) > > > > Also have I copied the content of the examples and integration modules to > > the new repo, duplicated the build there and updated the versions to > > artifacts in the main repo to reference a variable. > > > > The build in the extras seems to work, now I’ll have to strip out > > configurations, dependency management etc. for stuff that’s not needed in > > the extras and do the same in the main repo. > > > > > > > > Chris > > > > > > > > Von: Christofer Dutz > > Datum: Montag, 15. April 2024 um 09:22 > > An: dev@iotdb.apache.org > > Betreff: AW: Splitting up the repos > > Ok … > > > > So, no comment I will simply treat as lazy consensus, therefore I will move > > forward with tagging the main repo with the latest changes as last revision > > before the split and reference that in the commit to the new repo. > > Then I’ll simply copy over the files and delete them from the main repo. > > > > As with other projects however, I really dislike this form of workting > > together. Defaulting back to lazy consensus costs a lot of valuable time as > > I have to wait a reasonable amount of time. If I had gotten any “sure … I’m > > fine with you doing X” I could have long finished this. > > > > In the future it would be a lot better, if some people would actually reply. > > > > > > Chris > > > > > > > > Von: Christofer Dutz > > Datum: Donnerstag, 11. April 2024 um 10:36 > > An: dev@iotdb.apache.org > > Betreff: Splitting up the repos > > Hi all, > > > > so now that the new repo is created > > (https://gitbox.apache.org/repos/asf/iotdb-extras.git, but please don’t > > push anything there just yet), we would need to decide on which parts > > should be moved there. > > > > > > * “distribution”: Here I think we need to split the distribution. > > Keeping the distributions containing only core in the main repo and adding > > a new distribution module in the extras repo, that contains the downstream > > components. > > * “example” (which I would propose to rename to examples as it contains > > multiple) > > * “iotdb-connector” > > > > > > As it seems that in the integration-tests there are no tests testing the > > connectors, I guess we can leave that as it is. > > > > Now the problem is: There are multiple options to split up the repo and > > keeping the entire history. > > > > 1. Split out one directory in a separate branch and then merge all > > branches into an empty new one > > 2. Use the filter plugin to strip out all commits that match a regexp > > 3. Simply ignore the history, copy the files to the new repo and delete > > them from the old. > > > > 3 is the simples, but the person doing the move will be marked as author. > > In general this is not that problematic, as the integration modules and the > > examples are usually not that complex, but I would understand, if people > > wanted to keep the history. > > > > Option 1 is probably the most work, but the most robust option, as with > > option 2, I had to give up when doing the PLC4X split as there were bugs > > and issues in the tooling. > > > > So, if nobody objects and we’ve decided on what should be moved, I > > personally would opt for option 3. > > > > Chris
Re: Splitting up the repos
Hi, 1. Which Parts: Only iotdb-connector need to be moved, distribution and examples will impact the release and users. 2. How to split up: I prefer【Simply ignore the history, copy the files to the new repo and delete them from the old】. Jialin Qiao Christofer Dutz 于2024年4月15日周一 16:27写道： > > Hi all, > > > So, I’ve set a tag on the main repository “before-moving-extras” > (https://github.com/apache/iotdb/releases/tag/before-moving-extras) > > Also have I copied the content of the examples and integration modules to the > new repo, duplicated the build there and updated the versions to artifacts in > the main repo to reference a variable. > > The build in the extras seems to work, now I’ll have to strip out > configurations, dependency management etc. for stuff that’s not needed in the > extras and do the same in the main repo. > > > > Chris > > > > Von: Christofer Dutz > Datum: Montag, 15. April 2024 um 09:22 > An: dev@iotdb.apache.org > Betreff: AW: Splitting up the repos > Ok … > > So, no comment I will simply treat as lazy consensus, therefore I will move > forward with tagging the main repo with the latest changes as last revision > before the split and reference that in the commit to the new repo. > Then I’ll simply copy over the files and delete them from the main repo. > > As with other projects however, I really dislike this form of workting > together. Defaulting back to lazy consensus costs a lot of valuable time as I > have to wait a reasonable amount of time. If I had gotten any “sure … I’m > fine with you doing X” I could have long finished this. > > In the future it would be a lot better, if some people would actually reply. > > > Chris > > > > Von: Christofer Dutz > Datum: Donnerstag, 11. April 2024 um 10:36 > An: dev@iotdb.apache.org > Betreff: Splitting up the repos > Hi all, > > so now that the new repo is created > (https://gitbox.apache.org/repos/asf/iotdb-extras.git, but please don’t push > anything there just yet), we would need to decide on which parts should be > moved there. > > > * “distribution”: Here I think we need to split the distribution. Keeping > the distributions containing only core in the main repo and adding a new > distribution module in the extras repo, that contains the downstream > components. > * “example” (which I would propose to rename to examples as it contains > multiple) > * “iotdb-connector” > > > As it seems that in the integration-tests there are no tests testing the > connectors, I guess we can leave that as it is. > > Now the problem is: There are multiple options to split up the repo and > keeping the entire history. > > 1. Split out one directory in a separate branch and then merge all > branches into an empty new one > 2. Use the filter plugin to strip out all commits that match a regexp > 3. Simply ignore the history, copy the files to the new repo and delete > them from the old. > > 3 is the simples, but the person doing the move will be marked as author. In > general this is not that problematic, as the integration modules and the > examples are usually not that complex, but I would understand, if people > wanted to keep the history. > > Option 1 is probably the most work, but the most robust option, as with > option 2, I had to give up when doing the PLC4X split as there were bugs and > issues in the tooling. > > So, if nobody objects and we’ve decided on what should be moved, I personally > would opt for option 3. > > Chris
Re: New Table Model for IoTDB
Great, this will extend the usage scenario of IoTDB. Jialin Qiao Yuan Tian 于2024年4月7日周日 19:39写道： > > Hi all, > > As introduced in the official documentation( > https://iotdb.apache.org/UserGuide/latest/Basic-Concept/Data-Model-and-Terminology.html), > the previous modeling method of IoTDB was a tree model, which formed a full > path of a sequence from the root node to the leaf node, and a device path > from the root node to the second-to-last layer node. The previous query > syntax of IoTDB was very similar to the standard relational SQL, centered > on the sequence, with the sequence prefix name in `from clause` and the > sequence suffix name in `select clause`. This query syntax is not very > friendly to users who are accustomed to relational SQL, and they cannot > apply their previous query experience to IoTDB. > > Therefore, we are designing a new schema model called table model for > IoTDB. We will provide data to users in a table view in the same way as > relational databases. Each kind of device belongs to a table, and users can > use standard SQL to query this table, which greatly reduces the learning > curve of IoTDB. > > The functional specs for table model can be found in > https://timechor.feishu.cn/docx/C2eodP84VoJ0kuxgbhlc1fejnsh，and our dev > branch is ty/TableModelGrammar。Syntax Definition file for Table model can > be found > in > iotdb-core/relational-grammar/src/main/antlr4/org/apache/iotdb/db/relational/grammar/sql/RelationalSql.g4 > > To support the table model, we also need to change the current tsfile > format, so we need to upgrade tsfile version from V3 to V4, the new file > format for tsfile V4 can be seen in > https://apache-iotdb.feishu.cn/docx/QNeVd7mpVoWaFxxobopcCsw6ne5 > > Our development is currently at a very early stage, and we would like to > invite you to discuss the functionality of the table model. Your feedback > is valuable to us and will help us shape the development of this feature. > > > Best regards, > -- > Yuan Tian
Re: Splitting up the main repo?
+1 for moving these into extra repos. Jialin Qiao Xiangdong Huang 于2024年4月7日周日 09:40写道： > > > We surely also pull in a lot of potentially bad dependencies. > > and this may be a chance to re-check our dependencies and remove unnecessary.. > > --- > Xiangdong Huang > > Christofer Dutz 于2024年4月5日周五 19:13写道： > > > > Hi all, > > > > I just wanted to bring up one idea that we decided in the PLC4X project and > > seed the idea, if this would also be worth discussing here. > > > > So, we were seeing that our build kept on having sub-ideal CVE ratings as > > we had dependencies for which CVEs were reported. > > However, PLC4X itself has a very limited number of dependencies. The > > problem was that we had several “integration” modules, that pulled in > > Kafka, Calcite, Nifi and some Eclipse projects. > > Also did a lot of our examples pull in various third party libraries, for > > which also vulnerabilities were reported. > > > > We are currently in the process of splitting up our main repository into a > > main and an extras repository. > > The main contains the core of the project. The extras contains the > > examples, additional tools and integration modules (The ones with the many, > > many dependencies) > > This way we can get a much better secutity standing for the main repo. > > > > Would this also be a good idea for IoTDB? I know with our dependencies to: > > > > * Flink > > * Grafana > > * Hadoop > > * Hive > > * Spark > > * Zeppelin (this one is really bad when it comes to CVEs) > > * Pulsar (only examples) > > * RabbitMQ (only examples) > > * RocketMQ (only examples) > > > > We surely also pull in a lot of potentially bad dependencies. If we moved > > this out the same way we would probably have a much better CVE ranking. > > This might become problematic in the future as in Europe and in the US > > CRE/PLD and other initiatives are taking form. > > > > Chris
Anything needs PR to TsFile
Hi, We plan to release the first version 1.0.0 of TsFile, is there anything that needs PR? Jialin Qiao
Re: PRs about tsfile module
Hi, TsFile is an independent project and need its own release schedule and version, so no submodule is better. Jialin Qiao Giorgio Zoppi 于2024年1月6日周六 03:41写道： > > Ok, > thanks i will join that list. > > Il giorno ven 5 gen 2024 alle ore 18:17 Christofer Dutz < > christofer.d...@c-ware.de> ha scritto: > > > Generally that will definitely be discussed on the d...@tsfile.apache.org > > list ;-) > > > > Chris > > > > Gesendet von Outlook für Android<https://aka.ms/AAb9ysg> > > > > From: Giorgio Zoppi > > Sent: Friday, January 5, 2024 5:18:55 PM > > To: dev@iotdb.apache.org > > Subject: Re: PRs about tsfile module > > > > Ok, i like it. I also suggest to include docusaurs for the format docs. > > https://docusaurus.io/, > > so we can use it. Also what is the plan for python/cpp/rust format > > implementation? > > BR, > > Giorgio > > > > Il giorno ven 5 gen 2024 alle ore 12:51 Yuan Tian > > ha scritto: > > > > > Hi all, > > > > > > Since we've already set up our tsfile repo, so if you want to make some > > > changes to common-api and tsfile submodule in IoTDB repo, you also need > > to > > > submit another pr for tsfile repo. > > > > > > > > > Best regards, > > > > > > Yuan Tian > > > > > > > > > -- > > Life is a chess game - Anonymous. > > > > > -- > Life is a chess game - Anonymous.
Re: PRs about tsfile module
+1 Jialin Qiao Yuan Tian 于2024年1月5日周五 20:51写道： > > Hi all, > > Since we've already set up our tsfile repo, so if you want to make some > changes to common-api and tsfile submodule in IoTDB repo, you also need to > submit another pr for tsfile repo. > > > Best regards, > > Yuan Tian
Re: Is there a reason we're not deploying "library-udf"?
Hi, No reason, we could enable it. Jialin Qiao Christofer Dutz 于2024年1月4日周四 20:55写道： > > Hi all, > > while setting up Apache TsFile, I noticed, that we’re excluding “library-udf” > from SNAPSHOT deployment on Jenkins. Is there a reason for this? > > (Always to leave a comment in the code on stuff like that) > > Chris >
Re: [ANNOUNCE] Apache IoTDB 1.3.0 released
Congratulations! — Jialin Qiao Apache IoTDB PMC Haonan Hou 于2024年1月2日周二 09:47写道： > > The Apache IoTDB team is pleased to announce the release of Apache IoTDB > 1.3.0. > > Apache IoTDB (Database for Internet of Things) is an IoT native database > with high performance for data management and analysis, deployable on the > edge and the cloud. > > Apache IoTDB 1.3.0 is the latest version of the v1 series. Feel free to > upgrade > from earlier version. > > ## Features & Improvements > > - Optimize the permission module and support timeseries permission control > - Optimize heap and off-heap memory configuration in startup script > - Computed-type view timeseries support LAST queries > - Add pipe-related monitoring metrics > - Pipe rename 'Extractor' to 'Source' and 'Connector' to 'Sink' > - [IOTDB-6138] Support Negative timestamp > - [IOTDB-6193] Reject Node startup when loading configuration file failed > - [IOTDB-6194] Rename target_config_node_list to seed_config_node > - [IOTDB-6200] Change schema template to device template > - [IOTDB-6207] Add Write Point Metrics for load function > - [IOTDB-6208] Node error detection through broken thrift pipe > - [IOTDB-6217] When the number of time series reaches the upper limit, the > prompt message should be changed to prioritize using device templates > - [IOTDB-6218] Rename storage_query_schema_consensus_free_memory_proportion > to datanode_memory_proportion > - [IOTDB-6219] Fix the display problem of explain that the print result is > not aligned > - [IOTDB-6220] Pipe: Add check logic to avoid self-transmission > - [IOTDB-6222] Optimize the performance of Python client > - [IOTDB-6230] Add HEAPDUMP configuration in datanode-env.sh > - [IOTDB-6231]SchemaCache supports precise eviction > - [IOTDB-6232] Adding SSL function to dn_rpc_port > > ## Bugs > > - [IOTDB-6160] while using ` in target path, select into will throw error > - [IOTDB-6167] DataNode can't register to cluster when fetch system > configuration throws NPE > - [IOTDB-6168] ConfigNode register retry logic does not worked > - [IOTDB-6171] NPE will be thrown while printing FI with debug on > - [IOTDB-6184] Merge Sort finishes one iterator too long > - [IOTDB-6191] Fix group by year not considering leap years > - [IOTDB-6226] Fix the problem of inaccurate GC monitor detection at the > beginning and adjusting the alert threshold > - [IOTDB-6239] Show regions display error create time > > The full release note is available at: > https://dlcdn.apache.org/iotdb/1.3.0/RELEASE_NOTES.md > > The release is available for download at: > http://iotdb.apache.org/Download > > Maven artifacts for JDBC driver, session SDK, TsFile SDK, Spark-connector, > Hadoop-connector, Hive-connector and Flink-connector can be found at: > https://search.maven.org/search?q=3Dg:org.apache.iotdb > > > Regards, > The Apache IoTDB team
Re: Celebrating the Release of Ratis 3.0.0 on Our Community's Contributions
Hi, Thank William for the contribution! — Jialin Qiao Apache IoTDB PMC Chao Wang 于2023年12月27日周三 11:39写道： > > Hi ,Thank you for your contribution. > I have a suggetion , When upgrading the ratis version, consider the > compatibility of the iotdb upgrade from an earlier version to a later version. > > On 2023/12/26 13:15:19 William Song wrote: > > Dear Community Members, > > > > I am excited to announce the release of Ratis version 3.0.0, marking the > > first major version update since IoTDB began utilizing Ratis for consensus > > services. This release encompasses a multitude of new features, > > enhancements, and fixes, all of which are the contributions of the > > dedicated efforts of our IoTDB community. Reflecting on the past year, it's > > hard to imagine the remarkable journey that unfolded after several key > > members (Mr. Huang, Mr. Qiao, and Xinyu Tan) made the pivotal decision to > > utilize Ratis for data replication. > > > > I wish to extend my gratitude to each one of you for your contributions, > > whether direct or indirect. Our collective success is a testament to the > > strength of our community spirit, resonating perfectly with the Apache > > Software Foundation's ethos: “Community Over Code.” > > > > In the upcoming period, we plan to integrate these features and > > improvements into the master branch. Each addition will undergo backtesting > > and validation to ensure optimal performance and reliability. Your feedback > > and insights are invaluable to us, and I eagerly look forward to your > > thoughts and suggestions. > > > > Thank you once again for your support and dedication. Together, we are > > shaping a brighter future for our project. > > > > > > > > Warm regards, > > William
Re: [VOTE] Apache IoTDB 1.3.0 RC1 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] could compile from source: ./mvnw.sh clean install [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] statements executed successfully: [ok] SET STORAGE GROUP TO root.turbine; CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; insert into root.turbine.d1(timestamp,s0) values(1,1); insert into root.turbine.d1(timestamp,s0) values(2,2); insert into root.turbine.d1(timestamp,s0) values(3,3); select ** from root; — Jialin Qiao Apache IoTDB PMC Haonan Hou 于2023年12月27日周三 12:33写道： > > Hi all, > > Apache IoTDB 1.3.0 has been staged under [2] and it’s time to vote > on accepting it for release. All Maven artifacts are available under [1]. > Voting will be open for 72hr. > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > are required to pass. > > Release tag: v1.3.0 > Hash for the release tag: fe787cac2ddd874b44482a80306d9d6ae333143c > > Before voting +1, PMC members are required to download > the signed source code package, compile it as provided, and test > the resulting executable on their own platform, along with also > verifying that the package meets the requirements of the ASF policy > on releases. [3] > > You can achieve the above by following [4]. > > [ ] +1 accept (indicate what you validated - e.g. performed the > non-RM items in [4]) > [ ] -1 reject (explanation required) > > > [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1154 > [2] https://dist.apache.org/repos/dist/dev/iotdb/1.3.0/rc1 > [3] https://www.apache.org/dev/release.html#approving-a-release > [4] > https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release > [5] https://github.com/apache/iotdb/blob/v1.3.0/RELEASE_NOTES.md > [6] https://dist.apache.org/repos/dist/dev/iotdb/KEYS > > Best, > > Haonan Hou
Re: Release schedule
Hi, If I understand correctly, this is just a change of package namespace. So users may need to upgrade their imports. We could do this in 1.3.1, which will be checkout at about 2023.12.30. Thanks, — Jialin Qiao Apache IoTDB PMC Christofer Dutz 于2023年11月16日周四 14:11写道： > > About the compatability. > > Most apache projects have finished the migration from Javax to Jakarta. I > think we should do this as soon as possible, as we are cutting us off from a > number of dependency updates. > > This change however is a breaking change. > > I was going to propose to do that for a first release of 1.3.0, but as that > is now officially already cut, when do we want to do that? > > Chris > > Gesendet von Outlook für Android<https://aka.ms/AAb9ysg> > > From: Jialin Qiao > Sent: Wednesday, November 15, 2023 7:17:40 AM > To: dev@iotdb.apache.org > Subject: Re: Release schedule > > Hi, sorry for a typo: > > Then checkout rc/1.3.1 from master at 2023-12-30, and release 1.3.1 > after 2024-1-30. > > Thanks, > Jialin Qiao > > On 2023/11/15 05:46:11 Jialin Qiao wrote: > > Hi, > > > > I come to share about our release schedule. > > > > Currently, we are all working on master branch, and checkout a rc > > branch from master for release. > > So it is essential to ensure that the master branch remains compatible > > with the previous version at all times. > > > > The release will be done every two months. > > > > As we checkout rc/1.3.0 from master at 2023-10-30, and will release > > 1.3.0 after 2023-11-30. > > Then checkout rc/1.3.1 from master at 2023-12-30, and release 1.3.0 > > after 2024-1-30. > > .. > > > > > > Thanks, > > — > > Jialin Qiao > > Apache IoTDB PMC > >
Re: Apache TsFile
Good news! Thank Chris! — Jialin Qiao Apache IoTDB PMC Christofer Dutz 于2023年11月16日周四 16:52写道： > > Hi all, > > so at yesterdays board meeting the Apache TsFile TLP proposal was accepted. I > am currently requesting things to be created. The mailinglists: dev, commits > and private will be the first things that need to be created and I initiated > that and need to wait for them being created. > > As soon as that’s done I’ll request the git repo for the project. > > I’ll keep you posted on when it makes sense to sign up to the mailinglists. > > > Chris
Re: Release schedule
Hi, sorry for a typo: Then checkout rc/1.3.1 from master at 2023-12-30, and release 1.3.1 after 2024-1-30. Thanks, Jialin Qiao On 2023/11/15 05:46:11 Jialin Qiao wrote: > Hi, > > I come to share about our release schedule. > > Currently, we are all working on master branch, and checkout a rc > branch from master for release. > So it is essential to ensure that the master branch remains compatible > with the previous version at all times. > > The release will be done every two months. > > As we checkout rc/1.3.0 from master at 2023-10-30, and will release > 1.3.0 after 2023-11-30. > Then checkout rc/1.3.1 from master at 2023-12-30, and release 1.3.0 > after 2024-1-30. > ...... > > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC >
Release schedule
Hi, I come to share about our release schedule. Currently, we are all working on master branch, and checkout a rc branch from master for release. So it is essential to ensure that the master branch remains compatible with the previous version at all times. The release will be done every two months. As we checkout rc/1.3.0 from master at 2023-10-30, and will release 1.3.0 after 2023-11-30. Then checkout rc/1.3.1 from master at 2023-12-30, and release 1.3.0 after 2024-1-30. .. Thanks, — Jialin Qiao Apache IoTDB PMC
Re: Add -XX:+ExitOnOutOfMemoryError as default JVM parameter in datanode-env and confignode-env
+1 for ExitOnOutOfMemoryError. Dump could be an option, disabled by default. — Jialin Qiao Apache IoTDB PMC Yuan Tian 于2023年11月14日周二 14:10写道： > > Hi, > > +HeapDumpOnOutOfMemroyError is not provided as default jvm args, I've > comment that out, just to provide convenience for DBAs when they want to > use this parameter. > > > On Tue, Nov 14, 2023 at 2:00 PM ZhangJian He wrote: > > > +1 for ExitOnOutOfMemoryError. > > +HeapDumpOnOutOfMemroyError may produce very large files. How to clean up > > these old files? WDYT? > > > > Thanks > > ZhangJian He > > > > > > On Tue, 14 Nov 2023 at 12:04, Gaofei Cao wrote: > > > > > +1, > > > > > > `-XX:+ExitOnOutOfMemoryError` parameter can avoid the loss of some > > > key threads, it will be beneficial to the system. > > > If the IoTDB cluster is deployed on k8s, this parameter is more > > > indispensable, because k8s can dispatch another pod to replace this > > > OOM node rapidly. > > > Besides, i think we can add the usage of `-XX:+ExitOnOutOfMemoryError` > > > and `-XX:+HeapDumpOnOutOfMemoryError` in the user/DBA manual, which is > > > important to find the root cause of OOM. > > > > > > Best, > > > -- > > > Gaofei Cao > > > > > > Yuan Tian 于2023年11月13日周一 19:52写道： > > > > > > > > Hi all, > > > > > > > > Recently, we found in some real user cases that when OOM occurs in the > > > > DataNode process (although we should ensure that OOM does not happen, > > but > > > > we all know that bugs will always exist), some threads(e.g. rpc > > listening > > > > threads) may exit unexpectedly which may cause some strange things to > > > > happen. For example, if the heartbeat listening thread on the DataNode > > > > unexpectedly exits due to OOM, and then the OOM recovers on its own > > (some > > > > large queries end, or some compaction tasks end), but this thread will > > > > never exist again, causing the DataNode to remain in unknown state, > > > because > > > > the ConfigNode can no longer contact it via heartbeat. > > > > > > > > Therefore, we feel that OOM is a high-risk error, and we should let the > > > > process exit directly to avoid the loss of some key threads. > > > > > > > > And I did an experiment and found that -XX:+ExitOnOutOfMemoryError and > > > > -XX:+HeapDumpOnOutOfMemoryError do not conflict which means that we can > > > > keep both in jvm args and when OOM happens, it will firstly dump the > > heap > > > > memory and then exit. > > > > > > > > I've made this change in my pr( > > > https://github.com/apache/iotdb/pull/11531). > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > Best, > > > > -- > > > > Yuan Tian > > > > >
Re: Rollcall TsFile initial committers
Oh sorry, updated. Yuan Tian (jackiet...@apache.org) Chao Wang (wangchao...@apache.org) Christofer Dutz (cd...@apache.org) Jinrui Zhang (xingtan...@apache.org) Steve Yurong Su (r...@apache.org) Xinyu Tan (tanxi...@apache.org) Haonan Hou (hao...@apache.org) Gaofei Cao (gaogao...@apache.org) Jialin Qiao (qiaojia...@apache.org) Kun Liu (liu...@apache.org) Houliang Qi (neuyi...@apache.org) Xiangdong Huang (h...@apache.org) Chao Wang (ch...@apache.org) Jianmin Wang (jimw...@apache.org) Tian Jiang (jiangt...@apache.org） Xinyi Zhao (zhaoxi...@apache.org) Shuo Zhang (shuozh...@apache.org) Dawei Liu(li...@apache.org) Thanks, — Jialin Qiao Apache IoTDB PMC Haonan Hou 于2023年11月7日周二 09:38写道： > > Hi Jialin, > > I just find my name in list is not correct. It should be Haonan Hou, not > Haohan Hou. ;) > > Best, > Haonan > > > > On Nov 7, 2023, at 9:31 AM, Jialin Qiao wrote: > > > > Hi, > > > > Welcome Dawei! We are still on naming search stage. > > The updated list: > > > > Yuan Tian (jackiet...@apache.org) > > Chao Wang (wangchao...@apache.org) > > Christofer Dutz (cd...@apache.org) > > Jinrui Zhang (xingtan...@apache.org) > > Steve Yurong Su (r...@apache.org) > > Xinyu Tan (tanxi...@apache.org) > > Haohan Hou (hao...@apache.org) > > Gaofei Cao (gaogao...@apache.org) > > Jialin Qiao (qiaojia...@apache.org) > > Kun Liu (liu...@apache.org) > > Houliang Qi (neuyi...@apache.org) > > Xiangdong Huang (h...@apache.org) > > Chao Wang (ch...@apache.org) > > Jianmin Wang (jimw...@apache.org) > > Tian Jiang (jiangt...@apache.org） > > Xinyi Zhao (zhaoxi...@apache.org) > > Shuo Zhang (shuozh...@apache.org) > > Dawei Liu(li...@apache.org) > > > > Thanks, > > — > > Jialin Qiao > > Apache IoTDB PMC > > > > Dawei Liu 于2023年11月6日周一 21:09写道： > >> > >> Hi, > >> Recently changed computer equipment, didn't check my email, > >> and found out that something very important was missing... > >> Can i still participate? > >> > >> > >> > >> > >> > >> > >> > >> Thanks, > >> — > >> Dawei Liu > >> Apache IoTDB PMC > >> > >> > >> > >> > >> > >> > >> > >> > >> At 2023-10-23 14:04:25, "Jialin Qiao" wrote: > >>> Hi, > >>> > >>> Welcome Shuo! The updated list: > >>> > >>> Yuan Tian (jackiet...@apache.org) > >>> Chao Wang (wangchao...@apache.org) > >>> Christofer Dutz (cd...@apache.org) > >>> Jinrui Zhang (xingtan...@apache.org) > >>> Steve Yurong Su (r...@apache.org) > >>> Xinyu Tan (tanxi...@apache.org) > >>> Haohan Hou (hao...@apache.org) > >>> Gaofei Cao (gaogao...@apache.org) > >>> Jialin Qiao (qiaojia...@apache.org) > >>> Kun Liu (liu...@apache.org) > >>> Houliang Qi (neuyi...@apache.org) > >>> Xiangdong Huang (h...@apache.org) > >>> Chao Wang (ch...@apache.org) > >>> Jianmin Wang (jimw...@apache.org) > >>> Tian Jiang (jiangt...@apache.org） > >>> Xinyi Zhao (zhaoxi...@apache.org) > >>> Shuo Zhang (shuozh...@apache.org) > >>> > >>> Thanks, > >>> — > >>> Jialin Qiao > >>> Apache IoTDB PMC > >>> > >>> Shuo Zhang 于2023年10月23日周一 11:24写道： > >>> > >>>> > >>>> Hi Jialin, > >>>> > >>>> I participated in the R&D of the early version of TsFile with our early > >>>> teammates. Although my code contributions have lessened in recent years, > >>>> I have continued to champion the file layer with the entire DB system. > >>>> Going forward, I am happy to further promote TsFile and its development. > >>>> > >>>> If you find it beneficial, I would be delighted to become one of the > >>>> initial committers. Thanks a lot. > >>>> > >>>> Best Regards, > >>>> > >>>> > >>>> Shuo ZHANG > >>>> zhangshu...@qq.com > >>>> > >>>> From: Jialin Qiao > >>>> Date: 2023-10-15 23:22 > >>>> To: dev > >>>> Subject: Rollcall TsFile initial committers > >>>> Hi, > >>>> > >>>> To make TsFile a TLP, we need to affirm the initial committer first. > >>>> The criteria for becoming the initial committer of TsFile is > >>>> prelimilary defined as meeting the following two conditions > >>>> simultaneously. > >>>> > >>>> (1) Apache IoTDB PMC member > >>>> (2) Made contributions to the TsFile module > >>>> > >>>> If you have made significant contributions to the TsFile module, you > >>>> can also apply to become an initial committer. > >>>> > >>>> If you would like to be the initial committer of TsFile, please reply > >>>> to this email in 1 week :-) > >>>> > >>>> Thanks, > >>>> — > >>>> Jialin Qiao > >>>> Apache IoTDB PMC >
Re: Re: Rollcall TsFile initial committers
Hi, Welcome Dawei! We are still on naming search stage. The updated list: Yuan Tian (jackiet...@apache.org) Chao Wang (wangchao...@apache.org) Christofer Dutz (cd...@apache.org) Jinrui Zhang (xingtan...@apache.org) Steve Yurong Su (r...@apache.org) Xinyu Tan (tanxi...@apache.org) Haohan Hou (hao...@apache.org) Gaofei Cao (gaogao...@apache.org) Jialin Qiao (qiaojia...@apache.org) Kun Liu (liu...@apache.org) Houliang Qi (neuyi...@apache.org) Xiangdong Huang (h...@apache.org) Chao Wang (ch...@apache.org) Jianmin Wang (jimw...@apache.org) Tian Jiang (jiangt...@apache.org） Xinyi Zhao (zhaoxi...@apache.org) Shuo Zhang (shuozh...@apache.org) Dawei Liu(li...@apache.org) Thanks, — Jialin Qiao Apache IoTDB PMC Dawei Liu 于2023年11月6日周一 21:09写道： > > Hi, > Recently changed computer equipment, didn't check my email, > and found out that something very important was missing... > Can i still participate? > > > > > > > > Thanks, > — > Dawei Liu > Apache IoTDB PMC > > > > > > > > > At 2023-10-23 14:04:25, "Jialin Qiao" wrote: > >Hi, > > > >Welcome Shuo! The updated list: > > > >Yuan Tian (jackiet...@apache.org) > >Chao Wang (wangchao...@apache.org) > >Christofer Dutz (cd...@apache.org) > >Jinrui Zhang (xingtan...@apache.org) > >Steve Yurong Su (r...@apache.org) > >Xinyu Tan (tanxi...@apache.org) > >Haohan Hou (hao...@apache.org) > >Gaofei Cao (gaogao...@apache.org) > >Jialin Qiao (qiaojia...@apache.org) > >Kun Liu (liu...@apache.org) > >Houliang Qi (neuyi...@apache.org) > >Xiangdong Huang (h...@apache.org) > >Chao Wang (ch...@apache.org) > >Jianmin Wang (jimw...@apache.org) > >Tian Jiang (jiangt...@apache.org） > >Xinyi Zhao (zhaoxi...@apache.org) > >Shuo Zhang (shuozh...@apache.org) > > > >Thanks, > >— > >Jialin Qiao > >Apache IoTDB PMC > > > >Shuo Zhang 于2023年10月23日周一 11:24写道： > > > >> > >> Hi Jialin, > >> > >> I participated in the R&D of the early version of TsFile with our early > >> teammates. Although my code contributions have lessened in recent years, I > >> have continued to champion the file layer with the entire DB system. Going > >> forward, I am happy to further promote TsFile and its development. > >> > >> If you find it beneficial, I would be delighted to become one of the > >> initial committers. Thanks a lot. > >> > >> Best Regards, > >> > >> > >> Shuo ZHANG > >> zhangshu...@qq.com > >> > >> From: Jialin Qiao > >> Date: 2023-10-15 23:22 > >> To: dev > >> Subject: Rollcall TsFile initial committers > >> Hi, > >> > >> To make TsFile a TLP, we need to affirm the initial committer first. > >> The criteria for becoming the initial committer of TsFile is > >> prelimilary defined as meeting the following two conditions > >> simultaneously. > >> > >> (1) Apache IoTDB PMC member > >> (2) Made contributions to the TsFile module > >> > >> If you have made significant contributions to the TsFile module, you > >> can also apply to become an initial committer. > >> > >> If you would like to be the initial committer of TsFile, please reply > >> to this email in 1 week :-) > >> > >> Thanks, > >> — > >> Jialin Qiao > >> Apache IoTDB PMC
Re: Re: Suggestions for Enhancing the IoTDB Contribution Experience
Hi, Thanks for your feedback of this contribution! I support to give more attention for each PR and do more communication. Besides, if you have a PR need to be reviewed or just want to do a modification, welcome to discuss in the mail list. Thanks, — Jialin Qiao Apache IoTDB PMC 张 于2023年11月3日周五 18:25写道： > After reading PR#11429, upon comparing my PR, it does have more > advantageous aspects from which I have greatly benefited. However, I need > to point out something that I find 'unfair' in the process. > > *First of all*, my PR was committed on Oct 12, 2023, and no one followed > up on it before I asked. > > > *The second key point *is that no one gave useful advice on where I > should make modifications or the aspects I might have overlooked. Thus, I > didn't have the opportunity to make better revisions... > > At the same time, I must clarify that due to my limited familiarity with > IoTDB, the principle I followed during the modification process was 'to > avoid making significant changes to the existing logic as much as > possible.' Therefore, I attempted to make only necessary adjustments based > on the existing source code. This is evident to anyone who carefully reads > my commits. However, this might not be a good solution in open-source PRs. > > I acknowledge that my PR might be "worse code". Nevertheless, I still hope > that in future PR reviews for IoTDB, if possible, reviewers will carefully > review the code. Even if the PR is truly subpar, reviewers should provide > feedback. This would be incredibly helpful for the person because they > would know where their issues lie. > > The above is what I want to express. I hope that the "processing" will be > more warming in the future. > > Best regards > > > At 2023-11-01 16:01:16, "Xiangdong Huang" wrote: > >Hi, > > > >I read the discussion and all related issues/pull requests, and give > >some of my comments: > > > >It is appreciated for cqzhang's contribution to iotdb, especially for > >the detailed bug feedback. This is the best contribution to the open > >source community. > > > >Being a TLP open source project of Apache Software Foundation, keeping > >diversity is the community's goal. > >Therefore, attracting more and more contributors contributing to IoTDB > >is what we always want. > > > >I have to say comments like "this issue may not be suitable for a > >newer to our system due to communication costs of these complicated > >cases" is ABSOLUTLY incorrect. It is very very harmful for the > >community. > >"newer" does not mean he/she knows little than "older", and does not > >mean his/her solution is worse than "older". > >However, "meritocracy" still takes effect, because hearing a person > >whom has make some successful experience is usual correct. > > > >The only way we need to obey is: accepting better quality of codes, > >which we still need to improve. > >For example, I find cqzhang gave a solution like "using 12 months to > >replace 1 year", which is accpeted in the final PR. This is a kind of > >contribution. > >In the closed PR (#11171), the most important shortcoming is lack of UT and > >IT. > >In the merged PR(#11323), it also has drawbacks, e.g., > >"DateTimeUtils.convertDurationStrToLong()" function semantic is still > >incorrect if the input parameter value is "y". > > > >Considering making it easy to let more developers join us, I advocate that: > >1. For each public issue in Github/JIRA, claim you are working on it > >to let all others knowing that. If some other person think he/she > >want to take over, please also claims it asap, rather than submit PR > >directly. > >2. Even though, the cases that two PRs solve the same issue may still > >occur. In this case, "better code wins" and meritocracy are still the > >best way. > > > >Best, > > > >--- > >Xiangdong Huang > >School of Software, Tsinghua University > > > > > > > >Weihao Li <18110526...@163.com> 于2023年10月31日周二 15:11写道： > >> > >> Hi cqzhang7, > >> As you can see, `not supporting group by nature year` and `not supporting > >> mixed units in group by` are two separate issues. > >> 1. For the first issue, we have discussed in > >> https://github.com/apache/iotdb/pull/11309, and the final PR was appended > >> on the discuss. > >> 2. For the second issue, your general idea is righ
Re: [DISCUSS] Champion and PMC Chair of TsFile
Hi, It's my honor to be the PMC Chair of TsFile. I will do my best for TsFile project. Bests, — Jialin Qiao Apache IoTDB PMC Chao Wang 于2023年10月26日周四 16:28写道： > > +1 for Christofer Dutz to be the Champion > > > +1 for Jialin Qiao to be the PMC Chair > > > > > Thanks! > > > Chao Wang > > > On 10/26/2023 10:50，张硕 wrote： > +1 for Christofer Dutz to be the Champion > > > +1 for Jialin Qiao to be the PMC Chair > > > > > > > ---Original--- > From: "Houliang Qi" Date: Thu, Oct 26, 2023 10:41 AM > To: "dev@iotdb.apache.org" Subject: Re: [DISCUSS] Champion and PMC Chair of TsFile > > > > > +1 for Christofer Dutz to be the Champion. Jialin Qiao to be the pmc chair. > > > > > Replied Message > | From | Chao Wang | > | Date | 10/26/2023 10:33 | > | To | | > | Subject | Re: [DISCUSS] Champion and PMC Chair of TsFile | > +1 ， for Christofer Dutz to be the Champion.Jialin Qiao to be the pmc chair. > > On 2023/10/26 02:20:14 Xiangdong Huang wrote: > Hi, > > I vote for Christofer Dutz to be the Champion. I think he is the best > person for the role. > We can feel how he is professional in another email thread (discussion > about the proposal and resolution) :D > > As for the PMC Chair, I think Jialin Qiao is a good choice, he really did a > lot on the project, knows everything about the project, including the > design, the goal, and pay almost all of his time on the project. > > Best, > --- > Xiangdong Huang > > > Jialin Qiao 于2023年10月25日周三 15:54写道： > > Hi, > > To enter the Apache, we also need to define the project champion and PMC > chair. > > For Champion, I am honored to invite Chris to be the champion of the > TsFile project. > > For PMC, welcome to recommend yourself and others. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC
Re: [DISCUSS] Apache TsFile Proposal
Hi, Looks great! +1 for name search
—
Jialin Qiao
Apache IoTDB PMC
Christofer Dutz 于2023年10月26日周四 17:19写道：
>
> FYI,
>
> https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-214
>
> Chris
>
> Von: Xiangdong Huang
> Datum: Donnerstag, 26. Oktober 2023 um 04:23
> An: dev@iotdb.apache.org
> Betreff: Re: [DISCUSS] Apache TsFile Proposal
> Hi,
>
> The content looks fine. +1 for name-search.
> Best,
> ---
> Xiangdong Huang
>
>
>
> Christofer Dutz 于2023年10月25日周三 22:24写道：
>
> >
> > More feedback from the board was that the description is pretty long and
> > having it shorted would leave us more wiggle-room in the future.
> > Also, should we formally have passed a name-search … if you’re ok with
> > this, I would start the Name-Search.
> >
> > So that would result in the following resolution:
> >
> > WHEREAS, the Board of Directors deems it to be in the best interests
> > of
> >the Foundation and consistent with the Foundation's purpose to
> >establish a Project Management Committee charged with the creation
> > and
> >maintenance of open-source software, for distribution at no charge
> > to
> >the public, related to a columnar storage file format designed for
> > time
> >series data.
> >
> >NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> >(PMC), to be known as the "Apache TsFile Project", be and hereby is
> >established pursuant to Bylaws of the Foundation; and be it further
> >
> >RESOLVED, that the Apache TsFile be and hereby is responsible for
> >the creation and maintenance of software related to a columnar
> > storage
> >file format designed for time series data.
> >
> >RESOLVED, that the office of "Vice President, Apache TsFile" be and
> >hereby is created, the person holding such office to serve at the
> >direction of the Board of Directors as the chair of the Apache
> >TsFile Project, and to have primary responsibility for management of
> >the projects within the scope of responsibility of the Apache TsFile
> >Project; and be it further
> >
> >RESOLVED, that the persons listed immediately below be and hereby
> > are
> >appointed to serve as the initial members of the Apache TsFile
> >Project:
> >
> >- Yuan Tian (jackietien at apache dot org)
> >- Chao Wang (wangchao316 at apache dot org)
> >- Christofer Dutz (cdutz at apache dot org)
> >- Jinrui Zhang (xingtanzjr at apache dot org)
> >- Steve Yurong Su (rong at apache dot org)
> >- Xinyu Tan (tanxinyu at apache dot org)
> >- Haohan Hou (haonan at apache dot org)
> >- Gaofei Cao (gaogaofei at apache dot org)
> >- Jialin Qiao (qiaojialin at apache dot org)
> >- Kun Liu (liukun at apache dot org)
> >- Houliang Qi (neuyilan at apache dot org)
> >- Xiangdong Huang (hxd at apache dot org)
> >- Chao Wang (chaow at apache dot org)
> >- Jianmin Wang (jimwang at apache dot org)
> >- Tian Jiang (jiangtian at apache dot org）
> >- Xinyi Zhao (zhaoxinyi at apache dot org)
> >- Shuo Zhang (shuozhagn at apache dot org)
> >
> >NOW, THEREFORE, BE IT FURTHER RESOLVED, that {name of the chair} be
> >appointed to the office of Vice President, Apache TsFile, to serve
> > in
> >accordance with and subject to the direction of the Board of
> > Directors
> >and the Bylaws of the Foundation until death, resignation,
> > retirement,
> >removal or disqualification, or until a successor is appointed; and
> > be
> >it further
> >
> >RESOLVED, that the Apache TsFile Project be and hereby is tasked
> > with
> >the migration and rationalization of the Apache IoTDB TsFile
> > subproject;
> >and be it further
> >
> >RESOLVED, that all responsibilities pertaining to the Apache IoTDB
> >TsFile subproject encumbered upon the Apache IoTDB PMC are hereafter
> >discharged.
> >
> >
> >
> >
> >
> > Von: Christofer Dutz
> > Datum: Mittwoch, 25. Oktober 2023 um 15:03
> > An: dev@iotdb.apache.org
> > Betreff: AW: [DISCUSS] Apache TsFile Proposal
> > So, it seems that this document is based on an incubation proposal … so
Re: [DISCUSS] Apache TsFile Proposal
Hi, After TsFile is being accepted by Apache, I will create an TsFile-Rust repo. Thanks, — Jialin Qiao Apache IoTDB PMC Giorgio Zoppi 于2023年10月25日周三 15:46写道： > > Hello, > can you add in apache a initial ts-native for the rust project? Sorry for > being pedantic. > Best Regards, > Giorgio > > Il giorno mer 25 ott 2023 alle ore 05:04 Jialin Qiao > ha scritto: > > > Hi, > > > > The TsFile proposal is as follows, feel free to give advice :-) > > > > Abstract > > > > TsFile is a columnar storage file format designed for time series > > data, which supports efficient compression, high throughput of read > > and write, and compatibility with various frameworks, such as Spark > > and Flink. It is easy to integrate TsFile into IoT big data processing > > frameworks. > > > > Proposal > > > > TsFile is used for managing time series data. Although it is firstly > > used inside IoTDB, many users and companies use TsFile directly as a > > time series data management solution independently. Besides, there is > > a growing requirement for Multi-Language Support of TsFile > > implementations, such as C++, Go and Rust. > > > > Apache IoTDB community hereby submit this proposal of TsFile as an > > independent Apache project. This proposal outlines the key features > > and benefits of TsFile, along with the integration plan and the need > > for multi-language support. > > > > Background > > > > Time series data is becoming increasingly important in a wide range of > > applications, including IoT, intelligent control, finance, log > > analysis, and monitoring systems. > > TsFile has been developed by Apache IoTDB community in Java and is in > > IoTDB repository. Users could store time series data using TsFile, > > then read and analyze it in IoTDB, Spark and Flink. IoTDB could also > > generate TsFiles and syncronize TsFiles between two IoTDB instances. > > Furthermore, the demand for TsFile implementations in multiple > > programming languages has been growing, as it allows developers to > > leverage TsFile's capabilities in their preferred language. > > > > TsFile offers several distinctive features and benefits: > > Efficient Storage and Compression: TsFile employs advanced compression > > techniques to minimize storage requirements, resulting in reduced disk > > space consumption and improved system efficiency. > > Flexible Schema and Metadata Management: TsFile allows for directly > > write data without pre defining the schema, which is flexible for data > > aquisition. > > High Query Performance with time range: TsFile has indexed devices, > > sensors and time dimensions to accelerate query performance, enabling > > fast filtering and retrieval of time series data. > > Seamless Integration: TsFile is designed to seamlessly integrate with > > existing big data frameworks, such as Spark, Flink and Hadoop. > > > > > > Rationale > > > > Before using TsFile, there doesn't exist a file format for time > > series. The industry companies usually write time series data in > > various user-defined file format without unification, or use general > > columnar file format such as Parquet and ORC, which makes data > > collection and processing complicated without a standard. After > > TsFile, organizations could write data in TsFile inside end devices or > > gateway, then transfer TsFile to the cloud for unified management in > > IoTDB and other systems. In this way, we lower the network > > transmission and the computing resource consumption in the cloud. > > > > Initial Goals > > > > The initial goals include: > > > > Make TsFile an independent project that has its own SDK and > > documentation that is easier to use. > > Multi-Language Support of TsFile implementations apart from Java, such > > as C++, Go and Rust. > > Integrate more encoding and compression method in TsFile. > > More tools for TsFile: visualization tool, parsing tool, repair tool. > > > > > > Current Status > > > > Meritocracy > > > > We plan to invite the IoTDB committer to be the initial committer of > > TsFile. And we would like to follow ASF meritocratic principles and > > invite additional developers to participate. We will establish the > > documentation and encourage and monitor community participation so > > that privileges can be extended to those that contribute. > > > > Community > > > > The TsFile community is grown from the Apache IoTDB Community. The > >
[DISCUSS] Champion and PMC Chair of TsFile
Hi, To enter the Apache, we also need to define the project champion and PMC chair. For Champion, I am honored to invite Chris to be the champion of the TsFile project. For PMC, welcome to recommend yourself and others. Thanks, — Jialin Qiao Apache IoTDB PMC
[DISCUSS] Apache TsFile Proposal
Hi, The TsFile proposal is as follows, feel free to give advice :-) Abstract TsFile is a columnar storage file format designed for time series data, which supports efficient compression, high throughput of read and write, and compatibility with various frameworks, such as Spark and Flink. It is easy to integrate TsFile into IoT big data processing frameworks. Proposal TsFile is used for managing time series data. Although it is firstly used inside IoTDB, many users and companies use TsFile directly as a time series data management solution independently. Besides, there is a growing requirement for Multi-Language Support of TsFile implementations, such as C++, Go and Rust. Apache IoTDB community hereby submit this proposal of TsFile as an independent Apache project. This proposal outlines the key features and benefits of TsFile, along with the integration plan and the need for multi-language support. Background Time series data is becoming increasingly important in a wide range of applications, including IoT, intelligent control, finance, log analysis, and monitoring systems. TsFile has been developed by Apache IoTDB community in Java and is in IoTDB repository. Users could store time series data using TsFile, then read and analyze it in IoTDB, Spark and Flink. IoTDB could also generate TsFiles and syncronize TsFiles between two IoTDB instances. Furthermore, the demand for TsFile implementations in multiple programming languages has been growing, as it allows developers to leverage TsFile's capabilities in their preferred language. TsFile offers several distinctive features and benefits: Efficient Storage and Compression: TsFile employs advanced compression techniques to minimize storage requirements, resulting in reduced disk space consumption and improved system efficiency. Flexible Schema and Metadata Management: TsFile allows for directly write data without pre defining the schema, which is flexible for data aquisition. High Query Performance with time range: TsFile has indexed devices, sensors and time dimensions to accelerate query performance, enabling fast filtering and retrieval of time series data. Seamless Integration: TsFile is designed to seamlessly integrate with existing big data frameworks, such as Spark, Flink and Hadoop. Rationale Before using TsFile, there doesn't exist a file format for time series. The industry companies usually write time series data in various user-defined file format without unification, or use general columnar file format such as Parquet and ORC, which makes data collection and processing complicated without a standard. After TsFile, organizations could write data in TsFile inside end devices or gateway, then transfer TsFile to the cloud for unified management in IoTDB and other systems. In this way, we lower the network transmission and the computing resource consumption in the cloud. Initial Goals The initial goals include: Make TsFile an independent project that has its own SDK and documentation that is easier to use. Multi-Language Support of TsFile implementations apart from Java, such as C++, Go and Rust. Integrate more encoding and compression method in TsFile. More tools for TsFile: visualization tool, parsing tool, repair tool. Current Status Meritocracy We plan to invite the IoTDB committer to be the initial committer of TsFile. And we would like to follow ASF meritocratic principles and invite additional developers to participate. We will establish the documentation and encourage and monitor community participation so that privileges can be extended to those that contribute. Community The TsFile community is grown from the Apache IoTDB Community. The IoTDB community is introducing TsFile at many technical conferences. Next, we will build the mailing list for more convenience, broader communication and archived discussions. We are open to recruiting more developers from diverse backgrounds. Core Developers TsFile initial PMCs are from IoTDB community: Christofer Dutz, Xiangdong Huang, Jialin Qiao, Steve Yurong Su, Jinrui Zhang, Yuan Tian, Xinyu Tan, Haonan Hou, Gaofei Cao, Tian Jiang, Chao Wang(wangchao316), Chao Wang(mychaow), Houliang Qi, Kun Liu. These people has extensive experience in building database and data management system. Alignment The ASF is the natural choice to host the TsFile project as its goal of encouraging community-driven open-source projects fits with our vision for TsFile. Additionally, many other projects with which we are familiar with and expect TsFile to integrate with, such as Apache Spark, Apache Flink and Apache IoTDB are hosted by the ASF and we will benefit and provide benefits in close proximity to them. Known Risks Project Name TsFile project is used in IoTDB and other scenarios for over 7 years, its name is unique. Orphaned Products The core developers plan to work full time on the project. There is very little risk of TsFile being abandoned as it is part of Apache IoTDB&
Re: Rollcall TsFile initial committers
Hi, Welcome Shuo! The updated list: Yuan Tian (jackiet...@apache.org) Chao Wang (wangchao...@apache.org) Christofer Dutz (cd...@apache.org) Jinrui Zhang (xingtan...@apache.org) Steve Yurong Su (r...@apache.org) Xinyu Tan (tanxi...@apache.org) Haohan Hou (hao...@apache.org) Gaofei Cao (gaogao...@apache.org) Jialin Qiao (qiaojia...@apache.org) Kun Liu (liu...@apache.org) Houliang Qi (neuyi...@apache.org) Xiangdong Huang (h...@apache.org) Chao Wang (ch...@apache.org) Jianmin Wang (jimw...@apache.org) Tian Jiang (jiangt...@apache.org） Xinyi Zhao (zhaoxi...@apache.org) Shuo Zhang (shuozh...@apache.org) Thanks, — Jialin Qiao Apache IoTDB PMC Shuo Zhang 于2023年10月23日周一 11:24写道： > > Hi Jialin, > > I participated in the R&D of the early version of TsFile with our early > teammates. Although my code contributions have lessened in recent years, I > have continued to champion the file layer with the entire DB system. Going > forward, I am happy to further promote TsFile and its development. > > If you find it beneficial, I would be delighted to become one of the initial > committers. Thanks a lot. > > Best Regards, > > > Shuo ZHANG > zhangshu...@qq.com > > From: Jialin Qiao > Date: 2023-10-15 23:22 > To: dev > Subject: Rollcall TsFile initial committers > Hi, > > To make TsFile a TLP, we need to affirm the initial committer first. > The criteria for becoming the initial committer of TsFile is > prelimilary defined as meeting the following two conditions > simultaneously. > > (1) Apache IoTDB PMC member > (2) Made contributions to the TsFile module > > If you have made significant contributions to the TsFile module, you > can also apply to become an initial committer. > > If you would like to be the initial committer of TsFile, please reply > to this email in 1 week :-) > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC
Re: Rollcall TsFile initial committers
Hi, Welcome! The updated list: Yuan Tian (jackiet...@apache.org) Chao Wang (wangchao...@apache.org) Christofer Dutz (cd...@apache.org) Jinrui Zhang (xingtan...@apache.org) Steve Yurong Su (r...@apache.org) Xinyu Tan (tanxi...@apache.org) Haohan Hou (hao...@apache.org) Gaofei Cao (gaogao...@apache.org) Jialin Qiao (qiaojia...@apache.org) Kun Liu (liu...@apache.org) Houliang Qi (neuyi...@apache.org) Xiangdong Huang (h...@apache.org) Chao Wang (ch...@apache.org) Jianmin Wang (jimw...@apache.org) Tian Jiang (jiangt...@apache.org） Xinyi Zhao (zhaoxi...@apache.org) Thanks — Jialin Qiao Apache IoTDB PMC Stefanie Zhao 于2023年10月23日周一 09:22写道： > > Hi Jialin, > > I’d also like to be the initial committer of TsFile project. I major > participated in the user interface, for example functions design and TsFile > user manual. > > Best, > Xinyi Zhao > > On 2023/10/15 15:22:11 Jialin Qiao wrote: > > Hi, > > > > To make TsFile a TLP, we need to affirm the initial committer first. > > The criteria for becoming the initial committer of TsFile is > > prelimilary defined as meeting the following two conditions > > simultaneously. > > > > (1) Apache IoTDB PMC member > > (2) Made contributions to the TsFile module > > > > If you have made significant contributions to the TsFile module, you > > can also apply to become an initial committer. > > > > If you would like to be the initial committer of TsFile, please reply > > to this email in 1 week :-) > > > > Thanks, > > — > > Jialin Qiao > > Apache IoTDB PMC > >
Re: Rollcall TsFile initial committers
Hi, Thanks for all of your contribution! Following our criterion, the Initial Committers are: Yuan Tian (jackiet...@apache.org) Chao Wang (wangchao...@apache.org) Christofer Dutz (cd...@apache.org) Jinrui Zhang (xingtan...@apache.org) Steve Yurong Su (r...@apache.org) Xinyu Tan (tanxi...@apache.org) Haohan Hou (hao...@apache.org) Gaofei Cao (gaogao...@apache.org) Jialin Qiao (qiaojia...@apache.org) Kun Liu (liu...@apache.org) Houliang Qi (neuyi...@apache.org) Xiangdong Huang (h...@apache.org) Chao Wang (ch...@apache.org) Jianmin Wang (jimw...@apache.org) Tian Jiang (jiangt...@apache.org） Thanks, — Jialin Qiao Apache IoTDB PMC Chao Wang 于2023年10月21日周六 23:58写道： > > Hi Jialin， > > > I also would like to be the initial committer for the TsFile project。 > > > I am familiar with the TsFile format. I have participated in the discussion > on the index organization design of version 0.11. This is also a relatively > large format adjustment for TsFile, and I have also optimized some issues > during tsfile sync. > > > TsFile now has so many features, including encoding, indexing, etc., and it > seems that it does hope to become a common format in the field of industrial > big data and contribute to the industrial field. I am currently also doing > some work on some OLTP and OLAP databases. I hope that I will have the > opportunity to promote TsFile to open up the data flow with other databases > and expand the scope of use of TsFile in the future. > > > > > Thanks! > > > Chao Wang > On 10/20/2023 23:14，Houliang Qi wrote： > Hi Jialin, > > I also would like to be the initial committer for the TsFile project。 > > I am familiar with the format of TsFile and have contributed features such as > cross multi time partition load TsFile. > > Currently, I am also working on OLAP, and some OLAP databases already support > reading file formats such as parquet and orc. I hope that TsFile can become a > top-level project in Apache, so that other OLAP databases can support reading > TsFile format and expand the usage scenarios of TsFile. I am willing to make > my contribution to this. > > Thanks, > Houliang > Replied Message > | From | Xiangdong Huang | > | Date | 10/20/2023 22:29 | > | To | | > | Subject | Re: Rollcall TsFile initial committers | > Hi Jialin, > > I would also like to be an initial committer for the TsFile project. > > Best, > --- > Xiangdong Huang > > > Kun Liu 于2023年10月20日周五 16:28写道： > > Hi jialin > > I also would like to be the initial committer for the TsFile project, and > participated the initial version of the TsFile format. > > I am familiar with the `write path` and Rust language, if user and > community need the API for Tsfile with Rust I can help to make it. > > Thanks, > Kun > > > > Jialin Qiao 于2023年10月18日周三 12:48写道： > > Hi, > > Glad to receive so much support! > I also would like to be the initial committer of TsFile. > I'm familiar with the tech part(format, read and write) of TsFile, > application scenarios and solutions with TsFile and IoTDB. > Looking forward to making TsFile the infrastructure of IoT. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > Gaofei Cao 于2023年10月18日周三 00:20写道： > > Hi Jialin, > > I would like to be the initial committer of TsFile. > I participated in the design and development of the first version of > TsFile, familiar with the query process, file structure and other > system integrations. > And now I'm participating in the work to push down value filter. I > wish I could bring more features to make TsFile a better file format. > > > Thanks > > Gaofei Cao > > Haonan Hou 于2023年10月17日周二 14:39写道： > > > Hi Jialin, > > I would like to be the initial committer of TsFile. > > I participated in the development of TsFile v2 for IoTDB 0.10.x and > 0.11.x and the update tool for v1 to v2, v2 to v3. > > Thanks > Haonan Hou > > On 2023/10/17 03:40:42 Xinyu Tan wrote: > Hi Jialin, > > I would like to be the initial committer of TsFile. > > I participated in the design and development of IoTDB 0.12 cluster > version combined with TsFile for incremental snapshot. > > Now the consensus layer is designing a consensus algorithm based on > TsFile synchronization, which is expected to make TsFile more general in > more scenarios involving CDC and consensus. > > Thanks > > Xinyu Tan > > On 2023/10/15 15:22:11 Jialin Qiao wrote: > Hi, > > To make TsFile a TLP, we need to affirm the initial committer > first. > The criteria for bec
Re: [DISCUSS] Drop Spark 2.11
+1, but adding 2.13 and dropping 2.11 should be at the same time, so that we have the spark connector all the time. — Jialin Qiao Apache IoTDB PMC Christofer Dutz 于2023年10月17日周二 15:36写道： > > Hi all, > > while working on the dependency cleanup, I noticed that we were stuck with > some pretty old spark versions in the spark_2.11 module. > The reason is that the scala 2.11 versions have been end of life for quite > some time. Spark is now generally working with scala 2.12 and since version > 3.2 also 2.13. > > We have one reported CVE that we actually can’t get rid of in the scala_2.11 > module, as there will be no release with a fix. > > I therefore propose we drop the 2.11 scala spark plugin and possibly add a > 2.13 version instead. > > What do you folks think? > > Chris >
Re: Change schema temlpate to device template
+1 — Jialin Qiao Apache IoTDB PMC Yuan Tian 于2023年10月20日周五 08:34写道： > > Hi all, > > We decided to change all `schema template` in sql syntax to `device > template`, because it's easier to understand for users. We will keep > `schema template` usage for compatibility, but all the tests and docs will > be changed to `device template`. > > > Best regards, > - > Yuan Tian
Re: Rollcall TsFile initial committers
Hi, Glad to receive so much support! I also would like to be the initial committer of TsFile. I'm familiar with the tech part(format, read and write) of TsFile, application scenarios and solutions with TsFile and IoTDB. Looking forward to making TsFile the infrastructure of IoT. Thanks, — Jialin Qiao Apache IoTDB PMC Gaofei Cao 于2023年10月18日周三 00:20写道： > > Hi Jialin, > > I would like to be the initial committer of TsFile. > I participated in the design and development of the first version of > TsFile, familiar with the query process, file structure and other > system integrations. > And now I'm participating in the work to push down value filter. I > wish I could bring more features to make TsFile a better file format. > > > Thanks > > Gaofei Cao > > Haonan Hou 于2023年10月17日周二 14:39写道： > > > > > > Hi Jialin, > > > > I would like to be the initial committer of TsFile. > > > > I participated in the development of TsFile v2 for IoTDB 0.10.x and 0.11.x > > and the update tool for v1 to v2, v2 to v3. > > > > Thanks > > Haonan Hou > > > > On 2023/10/17 03:40:42 Xinyu Tan wrote: > > > Hi Jialin, > > > > > > I would like to be the initial committer of TsFile. > > > > > > I participated in the design and development of IoTDB 0.12 cluster > > > version combined with TsFile for incremental snapshot. > > > > > > Now the consensus layer is designing a consensus algorithm based on > > > TsFile synchronization, which is expected to make TsFile more general in > > > more scenarios involving CDC and consensus. > > > > > > Thanks > > > > > > Xinyu Tan > > > > > > On 2023/10/15 15:22:11 Jialin Qiao wrote: > > > > Hi, > > > > > > > > To make TsFile a TLP, we need to affirm the initial committer first. > > > > The criteria for becoming the initial committer of TsFile is > > > > prelimilary defined as meeting the following two conditions > > > > simultaneously. > > > > > > > > (1) Apache IoTDB PMC member > > > > (2) Made contributions to the TsFile module > > > > > > > > If you have made significant contributions to the TsFile module, you > > > > can also apply to become an initial committer. > > > > > > > > If you would like to be the initial committer of TsFile, please reply > > > > to this email in 1 week :-) > > > > > > > > Thanks, > > > > — > > > > Jialin Qiao > > > > Apache IoTDB PMC > > > > > > >
Re: Dependency management PR ready for being merged
Merged, Good work! — Jialin Qiao Apache IoTDB PMC
Rollcall TsFile initial committers
Hi, To make TsFile a TLP, we need to affirm the initial committer first. The criteria for becoming the initial committer of TsFile is prelimilary defined as meeting the following two conditions simultaneously. (1) Apache IoTDB PMC member (2) Made contributions to the TsFile module If you have made significant contributions to the TsFile module, you can also apply to become an initial committer. If you would like to be the initial committer of TsFile, please reply to this email in 1 week :-) Thanks, — Jialin Qiao Apache IoTDB PMC
Re: [VOTE] Apache IoTDB 1.2.2 RC1 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] could compile from source: ./mvnw.sh clean install [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] statements executed successfully: [ok] SET STORAGE GROUP TO root.turbine; CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; insert into root.turbine.d1(timestamp,s0) values(1,1); insert into root.turbine.d1(timestamp,s0) values(2,2); select ** from root; select ** from root align by device; Thanks, — Jialin Qiao Apache IoTDB PMC Yuan Tian 于2023年10月12日周四 20:52写道： > > Hi, > > +1 (binding) > > The source release: > apache headers [ok] > signatures and hashes [ok] > LICENSE and NOTICE [ok] > no jar files [ok] > could compile from source: ./mvnw.sh clean install [ok] > > The binary distribution: > version number in CLI [ok] > signatures and hashes [ok] > start in mac, jdk8 [ok] > statements executed successfully: [ok] > > SET STORAGE GROUP TO root.turbine; > CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; > insert into root.turbine.d1(timestamp,s0) values(1,1); > insert into root.turbine.d1(timestamp,s0) values(2,2); > insert into root.turbine.d1(timestamp,s0) values(3,3); > insert into root.turbine.d2(time,s0) values(now(),1); > insert into root.turbine.d2(time,s0) values(now(),2); > insert into root.turbine.d2(time,s0) values(now(),3); > select ** from root; > select ** from root align by device; > > > Best, > - > Yuan Tian
Re: [DISCUSS] Incubating TsFile as an independent Apache project?
Hi all, I'm glad that we have reached a consensus to donate TsFile as an independent Apache project. TsFile aims to be a universal file structure for managing time series data in IoT scenarios. So the next step is the actual process of driving it to be an Apache top-level project. Is the process the same as becoming an incubator project? Does anyone know? Thanks, — Jialin Qiao Apache IoTDB PMC Jialin Qiao 于2023年10月8日周日 13:34写道： > > Hi, > > Code is not the blocking thing, it could be done before release. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > Yuan Tian 于2023年10月8日周日 08:27写道： > > > > Hi, > > > > +1. > > > > However, before we do that, I think there are lots of work that we need to > > do, like extracting some read and write public interface for TsFile, > > otherwise it will be confusing for users and they don't know which > > methods to use. And with these interfaces, it will be beneficial to our > > subsequent iterations. > > > > Best, > > - > > Yuan Tian
Re: [DISCUSS] Incubating TsFile as an independent Apache project?
Hi, Code is not the blocking thing, it could be done before release. Thanks, — Jialin Qiao Apache IoTDB PMC Yuan Tian 于2023年10月8日周日 08:27写道： > > Hi, > > +1. > > However, before we do that, I think there are lots of work that we need to > do, like extracting some read and write public interface for TsFile, > otherwise it will be confusing for users and they don't know which > methods to use. And with these interfaces, it will be beneficial to our > subsequent iterations. > > Best, > - > Yuan Tian
Re: [DISCUSS] Incubating TsFile as an independent Apache project?
Hi, +1 for Apache TsFile. 1. Industrial time series data management is becoming more and more important, so a file format project for time series data is necessary. 2. When TsFile becomes an independent project, it could be developed and released without impact the upgrading of IoTDB. Thanks, — Jialin Qiao Apache IoTDB PMC 冯 庆新 于2023年10月7日周六 10:22写道： > > Hi Steve Su > > I think this is a great idea and I fully support it. > I believe Tsfile can generate greater influence and value. > > Thanks > B.R. > Qingxin Feng > > > 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送 > > 发件人: Steve Yurong Su<mailto:r...@apache.org> > 发送时间: 2023年10月7日 0:53 > 收件人: dev@iotdb.apache.org<mailto:dev@iotdb.apache.org> > 主题: [DISCUSS] Incubating TsFile as an independent Apache project? > > Hi folks, > > Recently, I've been thinking about something that has been on my mind: > I believe TsFile has the potential to develop as an independent Apache > project (note, not a subproject like apache-iotdb-tsfile), just like > ORC, Parquet, and Arrow. How about incubating TsFile as an independent > Apache project? > > The main reason I came up with this idea is because I realized that > TsFile is not just the data file format of IoTDB, but it can also be a > standard format in the IoT field. It provides excellent support for > time series data, offering various encoding and compression methods. > It can be easily self-parsed and its statistical information enables > fast data querying by big data software. I believe that every IoT > software could benefit directly by supporting TsFile as a storage > format. > > On the other hand, I hope to leverage the power of the community to > drive the development of the TsFile file format. Just imagine, TsFile > could have its own independent ecosystem: parsing tools in various > languages, connectors to various big data software, and so on. By > separating TsFile from IoTDB, we have the opportunity to attract more > community contributors and introduce more ideas, thus refining the > functionality and performance of TsFile. > > What do you think about this idea? > > In addition, how can TsFile, as a part of the current IoTDB project, > become an independent Apache project? Should it go through the Apache > Foundation incubation process again, or can it directly become a > top-level project? It may be too early to ask this question now, but I > would also like to hear your opinions. > > Thanks, > Steve Su >
Re: [VOTE] Apache IoTDB 1.2.1 RC1 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] statements executed successfully: [ok] SET STORAGE GROUP TO root.turbine; CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; insert into root.turbine.d1(timestamp,s0) values(1,1); select ** from root; Thanks, — Jialin Qiao Apache IoTDB PMC Yuan Tian 于2023年9月15日周五 18:14写道： > > Hi, > > +1 (binding) > > The source release: > apache headers [ok] > signatures and hashes [ok] > LICENSE and NOTICE [ok] > no jar files [ok] > could compile from source: ./mvnw.sh clean install [ok] > > The binary distribution: > version number in CLI [ok] > signatures and hashes [ok] > start in mac, jdk8 [ok] > statements executed successfully: [ok] > > SET STORAGE GROUP TO root.turbine; > CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; > insert into root.turbine.d1(timestamp,s0) values(1,1); > insert into root.turbine.d1(timestamp,s0) values(2,2); > insert into root.turbine.d1(timestamp,s0) values(3,3); > insert into root.turbine.d2(time,s0) values(now(),1); > insert into root.turbine.d2(time,s0) values(now(),2); > insert into root.turbine.d2(time,s0) values(now(),3); > select ** from root; > select ** from root align by device; > > > Best, > - > Yuan Tian
Re: iotdb quarter report (Q3)
Hi, Add some information of Project Activity: 1. Apache IoTDB ranking first in the benchANT Time Series: DevOps. https://benchant.com/ranking/database-ranking 2. Import a new feature pipe framework for lightweight streaming processing. This could be used for CDC (data synchronization from IoTDB to another system) and data subscription. 3. Extend the IT framwork to support multiple cluster instance in one test case, for testing data synchronization between two IoTDB cluster. 4. Optimized the load balance strategy of data insertion. Now we could spread the newly created data partition among the datanodes. Thanks, — Jialin Qiao Apache IoTDB PMC Yuan Tian 于2023年9月12日周二 09:10写道： > > Hi, > > +1. LGTM. > > > Best, > - > Yuan Tian
Re: [VOTE] Apache IoTDB 1.2.0 RC2 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] could compile from source: ./mvnw.sh clean install [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] statements executed successfully: [ok] SET STORAGE GROUP TO root.turbine; CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; insert into root.turbine.d1(timestamp,s0) values(1,1); insert into root.turbine.d1(timestamp,s0) values(2,2); insert into root.turbine.d1(timestamp,s0) values(3,3); select ** from root; Thanks, — Jialin Qiao Apache IoTDB PMC Haonan Hou 于2023年9月5日周二 10:50写道： > > Hi all, > > Apache IoTDB 1.2.0 has been staged under [2] and it’s time to vote > on accepting it for release. All Maven artifacts are available under [1]. > Voting will be open for 72hr. > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > are required to pass. > > Release tag: v1.2.0 > Hash for the release tag: 5c1b68224b9bd77c3aab6e234de532a10c1edcae > > Before voting +1, PMC members are required to download > the signed source code package, compile it as provided, and test > the resulting executable on their own platform, along with also > verifying that the package meets the requirements of the ASF policy > on releases. [3] > > You can achieve the above by following [4]. > > [ ] +1 accept (indicate what you validated - e.g. performed the > non-RM items in [4]) > [ ] -1 reject (explanation required) > > > [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1130 > [2] https://dist.apache.org/repos/dist/dev/iotdb/1.2.0/rc2 > [3] https://www.apache.org/dev/release.html#approving-a-release > [4] > https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release > [5] https://github.com/apache/iotdb/blob/rc/1.2.0/RELEASE_NOTES.md > [6] https://dist.apache.org/repos/dist/dev/iotdb/KEYS > > Best, > > Haonan Hou
Re: Re: [VOTE] Release IotDB-Tools: Thrift artifacts
+1 (binding) — Jialin Qiao Apache IoTDB PMC 张金瑞 <329920...@qq.com.invalid> 于2023年8月21日周一 20:07写道： > > +1 （binding） > > > Thanks, > Jinrui > > > > > > > > 原始邮件 > > > > 发件人："Yuan Tian"< jackietie...@gmail.com >; > > 发件时间：2023/8/21 20:01 > > 收件人："dev"< dev@iotdb.apache.org >; > > 主题：Re: [VOTE] Release IotDB-Tools: Thrift artifacts > > > +1 (binding) > > So glad to see that. > > > Best, > - > Yuan Tian > > On Mon, Aug 21, 2023 at 3:05 PM Christofer Dutz > wrote: > > > Hi all, > > > > as you all might know, I have worked on streamlining our build for all > > possible platforms. > > In order to work around us having to re-deploy these artifacts every > > Saturday, I have moved the artifacts out of the main repo into the > > iotdb-bin-resources git repo. > > > > I’ve staged the artifacts at: > > https://repository.apache.org/content/repositories/orgapacheiotdb-1129 > > > > Tag Name: iotdb-tools-thrift-v0.14.1.0 > > > > > https://github.com/apache/iotdb-bin-resources/releases/tag/iotdb-tools-thrift-v0.14.1.0 > > > > The source bundle generally consists of a pom.xml and an assembly xml … > so > > it should be quick to review. > > > > > https://repository.apache.org/service/local/repositories/orgapacheiotdb-1129/content/org/apache/iotdb/tools/iotdb-tools-thrift/0.14.1.0/iotdb-tools-thrift-0.14.1.0-source-release.zip > > > > I’m intentionally not staging it on the release-svn, as nobody should be > > required to consume this module but us ourselves. > > > > The vote stays open for 72hours … assuming enough support, I’ll then > > finish changing the main repo to use these artifacts. > > > > > > Chris > >
Re: [VOTE] Apache IoTDB 1.1.2 RC1 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] Thanks, — Jialin Qiao Apache IoTDB PMC Yuan Tian 于2023年7月26日周三 11:13写道： > > Hi, > > +1 (binding) > > The source release: > apache headers [ok] > signatures and hashes [ok] > LICENSE and NOTICE [ok] > no jar files [ok] > > The binary distribution: > version number in CLI [ok] > signatures and hashes [ok] > start in mac, jdk8 [ok] > statements executed successfully: [ok] > > SET STORAGE GROUP TO root.sg; > CREATE TIMESERIES root.sg.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; > insert into root.sg.d1(timestamp,s0) values(1,1); > insert into root.sg.d1(timestamp,s0) values(2,2); > insert into root.sg.d1(timestamp,s0) values(3,3); > select * from root.**; > select * from root.sg.** align by device; > > Best, > - > Yuan Tian
Re: Usage of Singleton pattern?
+1 for cleaning it — Jialin Qiao Apache IoTDB PMC Yuan Tian 于2023年7月14日周五 21:28写道： > > Sure! Do as you want! We would love to see these refactoring prs. > > > Best, > -- > Yuan Tian > > On Fri, Jul 14, 2023 at 9:20 PM Christofer Dutz > wrote: > > > > Aaah cool, > > > > and sorry for asking so much … much of IoTDBs history is not recorded and > > I’m just starting to get my hands dirty. > > I am afraid, that I’ll probably have more questions like this as they come > > up. > > > > But that means, if I see something that can be refactored quickly, I can > > create PRs for stuff like that? > > > > Just asking, cause I like to do simple maintenance tasks in order to think > > about other problems. > > In PLC4X I usually did cleanups regularly for such reasons. > > > > Chris > > > > > > Von: Yuan Tian > > Datum: Freitag, 14. Juli 2023 um 13:58 > > An: dev@iotdb.apache.org > > Betreff: Re: Usage of Singleton pattern? > > Hi Chris, > > > > Actually, most of the existing Singleton pattern Class is just for > > simplifying things during development. And we did run into trouble > > when writing some UTs(In such case, I will change related Singleton > > pattern Class ). > > But there are always higher priority tasks, so there is no time to > > change all these singletons. Good news is that we have strictly > > controlled the use of singletons in the new PRs, so there should be no > > more new ones. > > > > Best, > > > > Yuan Tian > > > > On Fri, Jul 14, 2023 at 7:14 PM Christofer Dutz > > wrote: > > > > > > Hi all, > > > > > > as I’m digging into IoTDB more and more, I noticed a quite excessive > > > usage of the Singleton pattern … some times even for things where it > > > doesn’t really make much sense. For example in SystemFileFactory, the > > > methods don’t really have much state and simply fire RuntimeExceptions if > > > fsType is initialized with HDFS. > > > > > > Singletons make it a bit tricky to run tests in parallel and to do > > > dynamic reloading of modules (like in OSGI). Most projects I’ve worked on > > > try to limit the use of singletons to an absolute minimum due to these > > > problems. > > > > > > I guess embedding IoTDB in an OSGI environment hasn’t been a use-case > > > yet, but I could imagine that random test failures while running tests in > > > parallel probably could have caused some issues. > > > > > > Was this a design decision that is based on some real-world necessity, or > > > for simplifying things during development? > > > I don’t want to put too much effort into proposing cleaning up things, if > > > there was a conscious decision to do things otherwise. > > > > > > Chris > > >
Re: [DISCUSS] Setup Sonarcloud for IoTDB?
Hi, We have the sonar cloud: https://sonarcloud.io/project/overview?id=apache_incubator-iotdb But it is only for committers: https://github.com/apache/iotdb/pull/10548 Thanks, — Jialin Qiao Apache IoTDB PMC Christofer Dutz 于2023年7月14日周五 21:24写道： > > Hi folks, > > I just had a look and noticed that we currently don’t have Sonarcloud > analysis setup. > At apache we have free access to using it for open-soucre and I’d like to > suggest to start using it. Some of the things it reports are definitely going > to be false positives, but it also spots a lot of things that could cause > problems before anyone has to file a bug report. > > What are your thoughts on this? > > Chris >
Re: Change LZ4 to the default compressor
+1 for option 1: Change LZ4 to the default compressor — Jialin Qiao Apache IoTDB PMC Yuan Tian 于2023年7月7日周五 11:57写道： > > Hi, > > vote for option 1: Change LZ4 to the default compressor, since it's a > compatible changes. > > > Best, > --- > Yuan Tian > > On Fri, Jul 7, 2023 at 11:10 AM yanze chen wrote: > > > > Hi all, > > > > In iot-benchmark experiment[1], we found that LZ4 outperformed SNAPPY in > > terms of (de)compression time and compression ratio in most scenarios. In > > the VLDB2022 paper[2], we also found that LZ4 performs no worse than SNAPPY > > on real-world datasets. > > > > Therefore, we consider changing the default compressor of IoTDB from SNAPPY > > to LZ4. How do you think about it? > > > > option 1: Change LZ4 to the default compressor > > option 2: Keep SNAAPY as the default compressor > > > > [1] https://apache-iotdb.feishu.cn/docx/ThmHdPKw9oJv95xvLhWco2WnnLh > > [2] https://sxsong.github.io/doc/22vldb-encoding.pdf > > > > -- > > Yanze Chen > > School of Software, Tsinghua University
New Repo: iotdb-docs
Hi, To manage our different versions of doc uniformly and clean the folder of code repo, I'd like to create a new repo: iotdb-docs. Then, we could put all versions(0.13, 1.0, 1.1, 1.2) of docs in the main branch of iotdb-docs, along with the sidebar. It will be easier to update the docs. What do you think? Thanks, — Jialin Qiao Apache IoTDB PMC
Re: Refactor the package structure of IoTDB
Hi Chris, This change originated from your advice in Slack :) Thanks, — Jialin Qiao Apache IoTDB PMC Christofer Dutz 于2023年6月30日周五 16:07写道： > > Hi Gaofei, > > I totally agree and support changing the package structure, as it was > something I stumbled over some time ago when I was trying to integrate IoTDB > into something. > > Chris > > > > Von: Gaofei Cao > Datum: Dienstag, 27. Juni 2023 um 15:19 > An: dev@iotdb.apache.org > Betreff: Refactor the package structure of IoTDB > Hi All, > > To make the the package structure of IoTDB more concise and tidy, we > have moved the package `antlr`, `confignode`, `consensus`, `metrics`, > `mlnode`, 'node-commons' and `server` to a new package `iotdb-core`. > > The package structure before changed: > Parent > ├──antrl > ├──confignode > ├──consensus > ├──iotdb-api > ├──iotdb-client > ├──iotdb-connector > ├──iotdb-protocol > ├──library-udf > ├──metrics > ├──mlnode > ├──node-commons > ├──server > ├──tsfile > > > The package structure after changed (master branch): > Parent > ├── iotdb-api > ├── iotdb-client > ├── iotdb-connector > ├── iotdb-core > └── antlr > └── confignode > └── consensus > └── datanode (sever package before changed) > └── metrics > └── mlnode > └── node-commons > └── tsfile > ├── iotdb-protocol > ├── library-udf > > > Thanks. > > Gaofei Cao > Apache IoTDB PMC
Re: [discuss] move jenkins build message to a new mailing list
+1 for option 1 Thanks, — Jialin Qiao Apache IoTDB PMC Haonan Hou 于2023年6月30日周五 17:31写道： > > +1 for option 1. > > Best, > Haonan Hou > > > On Jun 29, 2023, at 11:44 PM, Xiangdong Huang wrote: > > > > Hi all, > > > > How do you think about move jenkins's notification > > "build-failure"/"build-stable" to a new mailing list? > > > > Now, it is sent to dev@, and some people may think it is annoying.. > > > > So, how do you think about it? > > > > option 1: move to an existed mailing list: notifications@ (which is > > for jira issues now) > > option 2: create a new mailing list. > > option 3: keep current status > > > > Best, > > --- > > Xiangdong Huang >
Re: [New Feature] PB Tree for Massive Schema Management
Hi, Good feature! PB_Tree could be PBTree. Thanks, — Jialin Qiao Apache IoTDB PMC 周钰坤 于2023年6月19日周一 15:08写道： > > Hi, > > We're going to introduce PB Tree (Prefix-B+ Tree), a new SchemaEgnine > mode, in V1.2.0, which supports evict temporarily useless schema to > disk and load them back in need at runtime. With PB Tree, users will > not suffer from memory constraint for schema management any more, > especially in scenarios with massive time series. > > To enable PB Tree in IoTDB, the value of parameter named > schema_engine_mode in iotdb-commons.properties should be configured as > PB_Tree [1]. All the memory allocated for SchemaRegion will be used by > PB Tree. The allocation can be modified by changing the value of > parameters named storage_query_schema_consensus_free_memory_proportion > and schema_memory_proportion in iotdb-commons.properties. > > PB Tree is a new implementation of MTree in SchemaRegion, consisting > of a a PB Tree file and a prefix-tree-structured cache. The PB Tree > file manages the prefix tree structure on disk by storing pointer > pointing to first disk page storing children nodes in each node > record. B+ Tree is used to manage brother nodes for fast single child > node search. The cache is similar to the existing MTree implementation > in SchemaRegion. The main difference is that the children of a node > may not cached in memory, which should be read from disk by searching > specific B+ Tree located by pointer stored in node record and cached > in memory node object. > > Reference: > [1] > https://iotdb.apache.org/zh/UserGuide/V1.2.x/Reference/Common-Config-Manual.html#%E5%85%83%E6%95%B0%E6%8D%AE%E5%BC%95%E6%93%8E%E9%85%8D%E7%BD%AE > > Thanks. > > Yunkun Zhou > Apache IoTDB Committer
Release of 1.2.0
Hi, It's been about three months since we released 1.1.0. Considering there are some new features are developed these months. It may be time to release 1.2.0 in July. The new features I know contain PBTree of schema management, pipe function for data subscription of IoTDB, and some query functions. Please supplement the new features of 1.2.0. Thanks, — Jialin Qiao Apache IoTDB PMC
Re: [VOTE] Apache IoTDB 1.1.1 RC1 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] statements executed successfully: [ok] SET STORAGE GROUP TO root.turbine; CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; insert into root.turbine.d1(timestamp,s0) values(1,1); insert into root.turbine.d1(timestamp,s0) values(2,2); insert into root.turbine.d1(timestamp,s0) values(3,3); select ** from root; Thanks, — Jialin Qiao On 2023/06/10 17:44:43 Haonan Hou wrote: > Hi all, > > Apache IoTDB 1.1.1 has been staged under [2] and it’s time to vote > on accepting it for release. All Maven artifacts are available under [1]. > Voting will be open for 72hr. > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > are required to pass. > > Release tag: v1.1.1 > Hash for the release tag: d123b8e5ce8b23144107f41d1e1b115314d47ee0 > > Before voting +1, PMC members are required to download > the signed source code package, compile it as provided, and test > the resulting executable on their own platform, along with also > verifying that the package meets the requirements of the ASF policy > on releases. [3] > > You can achieve the above by following [4]. > > [ ] +1 accept (indicate what you validated - e.g. performed the > non-RM items in [4]) > [ ] -1 reject (explanation required) > > > [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1116 > [2] https://dist.apache.org/repos/dist/dev/iotdb/1.1.1/rc1 > [3] https://www.apache.org/dev/release.html#approving-a-release > [4] > https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release > [5] https://dist.apache.org/repos/dist/dev/iotdb/1.1.1/rc1/RELEASE_NOTES.md > [6] https://dist.apache.org/repos/dist/dev/iotdb/KEYS > > Best, > > Haonan Hou >
Re: Contributing Code and Ongoing Support to the Newly Created C# Client Repository
Hi, Thanks for your contribution! — Jialin Qiao Apache IoTDB PMC ZHAN LU 于2023年5月27日周六 20:06写道： > > Dear friends, > > I am Zhan Lu, a first-year postgraduate student at the School of Software, > Tsinghua University, and a contributor to the Apache-IoTDB-Client-Csharp > <https://github.com/eedalong/Apache-IoTDB-Client-CSharp> repository. > > A new iotdb-client-csharp > <https://github.com/apache/iotdb-client-csharp> repository > has been created within the apache organization. Together with many friends > from the open-source community, we have contributed the code from our > earlier Apache-IoTDB-Client-Csharp repository to this new one, intending it > to be a strong foundation for future development and enhancements. > > We are committed to the ongoing contribution and maintenance of this > repository to facilitate continuous enhancements. > > I cordially invite all community members to utilize this new repository, > and sincerely welcome any feedback or suggestions that can help further its > development. > > Thank you for your time. Should you have any questions or need assistance > related to the new repository, please feel free to reach out. > > Best regards, > > Zhan Lu
new committer: Peichen Zhou
Hi, The Project Management Committee (PMC) for Apache IoTDB has invited Peichen Zhou to become a committer and we are pleased to announce that he has accepted. Being a committer enables easier contribution to the project since there is no need to go via the patch submission process. This should enable better productivity. You could get the write-access of our repository by: (1) Update your github id in: id.apache.org , also an email (anything to your apache email will forward to this email). (2) Accept the invitation in github to be a member of the apache group. (3) Link your accounts via: git.apache.org (To lighten the third option MFA Status, you need to download Google Authenticator (Google 身份验证器) in your mobile) (4）Sign in github by Google Authenticator (5) Clone repository with ssh. Welcome, Peichen! Yours, The Apache IoTDB PMC
[Result][VOTE] New Repo: iotdb-client-csharp
Hi, This vote [1] has lasted for 72 hours, with 3 +1 PMC votes and total 5 +1 votes, this VOTE passes! +1 votes: gitdata2 Houliang Qi Qingxin Feng Chao Wang Haonan Hou [1] https://lists.apache.org/thread/zb3wm5gz7cqxqcclsyqxxhygx643bkpr Thanks, — Jialin Qiao Apache IoTDB PMC
[VOTE] New Repo: iotdb-client-csharp
Hi, Currently, there are two Repositories of IoTDB Csharp client. eedalong/Apache-IoTDB-Client-CSharp eedalong/Apache-IoTDB-Client-CSharp-UseCase I prefer to call for a vote for creating a new repo iotdb-client-csharp for the csharp client. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason The vote is open for the next 72 hours and passes if at least three +1 votes and more +1 votes than -1 votes. Best, — Jialin Qiao Apache IoTDB PMC
Re: Multiple bug&issue trackers?
Hi, GitHub is usually used by users. Developers are all using Jira now, moving to GitHub issues is not easy... We may need to discuss the benefit of moving to github. Thanks, — Jialin Qiao Apache IoTDB PMC Christofer Dutz 于2023年5月12日周五 14:57写道： > > Hi folks, > > just one question ... is there a reason we’re supporting both github issues > as well as Jira for Issue reporting? Wouldn’t it make more sense to decide on > one and go with that? (I would definitely recommend to move to GH Issues) > > Chris
CVE-2023-24831: Apache IoTDB grafana-connector Login Bypass Vulnerability
Severity: low Description: Improper Authentication vulnerability in Apache Software Foundation Apache IoTDB.This issue affects Apache IoTDB Grafana Connector: from 0.13.0 through 0.13.3. Attackers could login without authorization. This is fixed in 0.13.4. References: https://iotdb.apache.org/ https://iotdb.apache.org https://www.cve.org/CVERecord?id=CVE-2023-24831
CVE-2023-30771: Apache IoTDB Workbench: apache/iotdb-web-workbench: forge the JWTToken to access workbench
Severity: low Description: Incorrect Authorization vulnerability in Apache Software Foundation Apache IoTDB.This issue affects the iotdb-web-workbench component on 0.13.3. iotdb-web-workbench is an optional component of IoTDB, providing a web console of the database. This problem is fixed from version 0.13.4 of iotdb-web-workbench onwards. References: https://iotdb.apache.org https://www.cve.org/CVERecord?id=CVE-2023-30771
Re: [discuss] consider revert the feature of multi-tenancy
Hi, > I think we have reached a consensus from the discussion, change the name of > this feature to resource control, and continue to contribute to this feature. +1, Thanks for Houliang and Yuhua's contribution! We indeed need a resource control module to keep the system safe :) Best, ————————— Jialin Qiao Apache IoTDB PMC Houliang Qi 于2023年4月11日周二 16:02写道： > > Hi, all > > > I think we have reached a consensus from the discussion, change the name of > this feature to resource control, and continue to contribute to this feature. > Thank you for your concern. > > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | Xiangdong Huang | > | Date | 04/11/2023 15:39 | > | To | | > | Subject | Re: [discuss] consider revert the feature of multi-tenancy | > Hi Houliang, > > It makes no sense to refer Doris. Doris is not a lightweight db, and > edge side is never its goal. > > The topic of this discussion is whether to revert the feature of > multi-tenancy. > > I wonder why you fall into these words I think I have mentioned at > least twice (or maybe 3 times) that Jialin's suggestion is fine for > me. > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > > Houliang Qi 于2023年4月11日周二 15:05写道： > > Hi Jinrui, > > (Jinrui) From my perspective, Multi-tenancy is different from > resource-control and they are not the different term for same thing. > According to our implementation, current feature focus on the resource > control on users of one tenant rather than on different tenants. If we did > not reflect the wording `multi-tenancy` in the code, why do we use it on user > docs and PR's description ? > > Sorry, I am not agree with you, from my perspective, a user is a tenant, and > each tenant has different resources. This is also multi-tenancy. Even each > tenant can only have one db. In our current implementation, a user is a > tenant. > For doris, they also mention multi-tenancy, but it is limited user > resources.[1], the same as our current implementation. > For Spanner, a tenant can also have only one db. [2] > The reason why I think that both multi-tenancy and resource-control are > suitable for us is that what we are currently doing is to limit the functions > of users or db resources. > On this point, I agree with Wang Chao's point of view. > > As for whether the multi-tenant function you mentioned affects the > positioning of IoTDB, I don't think it is accurate. I personally think that > the multi-tenant function is a term for resource isolation technology and > will not affect the positioning of IoTDB. I don't know how you define the > multi-tenant function. If it refers to the connection with the billing system > of the cloud service provider, it may be another form. This discussion will > not continue to discuss multi-tenancy. > > > > (Jinrui) REVERT does not mean REJECT. It is only a quick way to keep the code > more reliable before we reach the same page. And furthermore, I don't think > it is harmful or discouraging and it is only a regular way we use to replace > hot-fix. > (Jinrui) The reviewers may be confused by the PR's description and then focus > on whether `multi-tenant` should be integrated in current development stage > of IoTDB. > > The topic of this discussion is whether to revert the feature of > multi-tenancy. I STRONGLY think that this PR does not violate the positioning > and future development of IOTDB, so I STRONGLY think that revert is not > needed, as this function is not enabled by default, and we are continuing > Iterate and refine this feature. Before the actual release, it is necessary > to consider some scenarios and do some testing. > > > > [1] https://doris.apache.org/docs/dev/admin-manual/multi-tenant/ > [2] > https://cloud.google.com/solutions/implementing-multi-tenancy-cloud-spanner > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | Chao Wang | > | Date | 04/11/2023 13:42 | > | To | dev@iotdb.apache.org | > | Subject | Re: Re：Re: [discuss] consider revert the feature of multi-tenancy > | > Everyone's contribution counts. But what we are talking about is whether > `multi-tenancy` is suitable for current IoTDB's development. > From my perspective, Multi-tenancy is different from resource-control and > they are not the different term for same thing. According to our > implementation, current feature focus on the resource control on us
Re: [discuss] consider revert the feature of multi-tenancy
Hi, The following is my perspective. (1) The function name should be resource-control on the storage group level, rather than multi-tenancy. (2) This PR does not bring many side effects as it is disabled by default, so a revert is not a must. (3) The code may need to be improved, but it's ok as long as we keep optimizing. Thanks, — Jialin Qiao Apache IoTDB PMC Chao Wang 于2023年4月10日周一 19:14写道： > > Agree with Houliang's opinion. > > > Thanks! > > > Chao Wang > BONC ltd > On 4/10/2023 19:01，Houliang Qi wrote： > -1 > > First of all, thanks Xiangdong for pointing out IoTDB's Charter. > > "RESOLVED, that the Apache IoTDB Project be and hereby is > responsible for the creation and maintenance of software > related to an IoT native database with high performance > for data management and analysis, on the edge and the cloud." > > As the charter post, IoTDB can be deployed in the cloud, this is why we > deploy the multi-tenancy feature. > > The cloud can be a public or private cloud if we can deploy only one IoTDB > cluster, and manage multi databases and users with different resources, which > will simplify the maintenance. > > -> 1) how many side-effects the feature will bring; > > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. > > -> 2) how to reduce the effect when IoTDB is deployed on the edge. > > We supply one switch about this feature, called `quota_enable`, by default > this value is false, so it has no effect when IoTDB is deployed on the edge. > This also answers Jinrui's doubt. > > -> 3) some checks failed on WinOS, are they irrelevant? > > No, I think they are not irrelevant, the false check message is about the > Compaction module, and > I see the former pr[2][3] which have been merged 4 days ago has the same > issue, so I suspect that the compaction module has occasional bugs > > -> 4) The feature SHOULD be discussed carefully in the community, rather that > submit PRs and merged after some reviews. > > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > > The IoTDB community is open and different opinions are welcome. After all, we > all have the same original intention of wanting IoTDB's features to be more > diverse. > > [1] https://apache-iotdb.feishu.cn/docx/DbqCd8t3EoxlCFx1yYicd9N4n4s > [2] https://github.com/apache/iotdb/actions/runs/4625220921/jobs/8181102446 > [3] https://github.com/apache/iotdb/actions/runs/4531046594/jobs/7980725316 > [4] https://apache-iotdb.feishu.cn/docx/doxcnKOYKDmJ40FpVnVsPMd3nTg > [5] https://lists.apache.org/thread/y6dqcm2o7qk0nbkllb61bp8cv6d3m1h7 > > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | 张金瑞<329920...@qq.com.INVALID> | > | Date | 04/10/2023 15:03 | > | To | dev | > | Subject | Re：[discuss] consider revert the feature of multi-tenancy | > +1, > > > Agree with Xiangdong's opinion. > And on the other hand, checking this PR's side effects may take lot of > time and during this period, there may be lots of users using latest > code to deploy/upgrade their systems. So the best practice is reverting this > PR until the side-effect is eliminated > > > > Thanks, > Zhang Jinrui，Apache IoTDB PMC > > > > Original > > > > From:"Xiangdong Huang"< saint...@gmail.com >; > > Date:2023/4/10 10:05 > > To:"dev"< dev@iotdb.apache.org >; > > Subject:[discuss] consider revert the feature of multi-tenancy > > > Hi all, > > I see the multi-tenancy feature is merged, and several committers made > a lot of contributions on that. > > As multi-tenancy is quite a big feature, which may change IoTDB's > position. The feature SHOULD be discussed carefully in the community, > rather that submit PRs and merged after some reviews. > > Therefore, I call to revert the PR and discuss ASAP about the feature > after that. > > At least, the proposer need to answer the following questions, > 1) how many side-effect the feature will bring; > 2) how to reduce the effect when IoTDB is deployed on the edge. > 3) some checks failed on WinOS, are they irrelevant? > > I don't mean of rejecting any big con
Re: [VOTE] Apache IoTDB 1.1.0 RC2 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk-17 [ok] statements executed successfully: [ok] insert into root.db.d1(time,s0) values(1,1); insert into root.db.d1(time,s0) values(2,2); insert into root.db.d1(time,s0) values(3,3); select * from root.db.**; Best, — Jialin Qiao Apache IoTDB PMC Haonan Hou 于2023年3月31日周五 11:44写道： > > Hi all, > > Apache IoTDB 1.1.0 has been staged under [2] and it’s time to vote > on accepting it for release. All Maven artifacts are available under [1]. > Voting will be open for 72hr. > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > are required to pass. > > Release tag: v1.1.0 > Hash for the release tag: 807ec8b9e8006113d941ca7acd2c0a2f28829428 > > Before voting +1, PMC members are required to download > the signed source code package, compile it as provided, and test > the resulting executable on their own platform, along with also > verifying that the package meets the requirements of the ASF policy > on releases. [3] > > You can achieve the above by following [4]. > > [ ] +1 accept (indicate what you validated - e.g. performed the > non-RM items in [4]) > [ ] -1 reject (explanation required) > > > [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1115 > [2] https://dist.apache.org/repos/dist/dev/iotdb/1.1.0/rc2 > [3] https://www.apache.org/dev/release.html#approving-a-release > [4] > https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release > [5] https://dist.apache.org/repos/dist/dev/iotdb/1.1.0/rc2/RELEASE_NOTES.md > [6] https://dist.apache.org/repos/dist/dev/iotdb/KEYS > > Best, > > Haonan Hou
Re: [VOTE] Apache IoTDB 1.1.0 RC1 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] could compile from source: ./mvnw.sh clean install [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] statements executed successfully: [ok] SET STORAGE GROUP TO root.turbine; CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; insert into root.turbine.d1(timestamp,s0) values(1,1); insert into root.turbine.d1(timestamp,s0) values(2,2); insert into root.turbine.d1(timestamp,s0) values(3,3); select ** from root; Thanks, Jialin Qiao On 2023/03/25 05:20:12 Haonan Hou wrote: > Hi all, > > Apache IoTDB 1.1.0 has been staged under [2] and it’s time to vote > on accepting it for release. All Maven artifacts are available under [1]. > Voting will be open for 72hr. > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > are required to pass. > > Release tag: v1.1.0 > Hash for the release tag: 12e3f998667c72586139e023fe4521cb4db7c013 > > Before voting +1, PMC members are required to download > the signed source code package, compile it as provided, and test > the resulting executable on their own platform, along with also > verifying that the package meets the requirements of the ASF policy > on releases. [3] > > You can achieve the above by following [4]. > > [ ] +1 accept (indicate what you validated - e.g. performed the > non-RM items in [4]) > [ ] -1 reject (explanation required) > > > [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1114 > [2] https://dist.apache.org/repos/dist/dev/iotdb/1.1.0/rc1 > [3] https://www.apache.org/dev/release.html#approving-a-release > [4] > https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release > [5] https://dist.apache.org/repos/dist/dev/iotdb/1.1.0/rc1/RELEASE_NOTES.md > [6] https://dist.apache.org/repos/dist/dev/iotdb/KEYS > > Best, > > Haonan Hou
Re: Request for translated names for group by clauses
Hi, +1 for 1. 差值分段 2. 条件分段 3. 会话分段 The group by series indeed confusing, we could use "group by filter" or "group by condition". Thanks, — Jialin Qiao Apache IoTDB PMC Eric Pai 于2023年2月24日周五 10:31写道： > > Hi， > > 1. 差值分段 > > 2. 条件分段 > > 3. 会话分段 > > I think the best choices are the direct translations. BTW, why we choose the > word 'series' in GROUP BY SERIES clause? It's more likely an aggregation > crossing different timeseries, like GROUP BY LEVEL and GROUP BY TAG. > > 发件人: ycy wi > 日期: 星期四, 2023年2月23日 18:04 > 收件人: dev@iotdb.apache.org > 主题: Request for translated names for group by clauses > > Hi, everyone! > > We are supporting more 'group by' clauses in aggregation query for IoTDB > which refers to some implements with new functions. So we want to ask for the > advice of translated name in Chinese for these new features according to the > functions and SQL of them. > > 1. Group By Variation > > Group-By-Variation wil set the first point in group as base point, then if > the difference between a new point and base point is small than or equal to > delta, this point will be put in the same group. (P.S. delta is the value > users set and the difference is calculated by the expression users give). It > can be described by the following picture: > [cid:ii_legxltnb3] > Some alternative translation in Chinese: > > * 事件分段 > * 差值分段 > * Any suggestions > 2. Group By Series > Group-By-Series supports grouping the data points by the condition expression > users give (the return type of condition expression needs to be boolean). The > continuous data points which satisfy the condition expression will be put in > the same group, the data points don't satisfy the condition will be skipped > and won't be calculated by aggregation operator. > Some alternative translation in Chinese: > > * 条件分段 > * Any suggestions > 3. Group By Session > Group-By-Session will compute the difference between the value of time in > continuous data points. A timeInterval is set by users to indicate the max > gap of time in one group between continuous data points. It can be described > in the following: > [cid:ii_legvj27g1] > The alternative translation in Chinese: > > * 会话分段 > * 时间差值分段 > * Any suggestions > > You can see more details and examples in > https://iotdb.apache.org/UserGuide/Master/Query-Data/Group-By.html<https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fiotdb.apache.org%2FUserGuide%2FMaster%2FQuery-Data%2FGroup-By.html&data=05%7C01%7C%7C7787a2489d0344286beb08db15855ebd%7C84df9e7fe9f640afb435%7C1%7C0%7C638127434791287241%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NtoQ7ek7p7HXonZ837OkdW%2BajBAxL%2FnflUNYCCjQF8k%3D&reserved=0>. > > Reply 1,2,3 to represent each groupByClause and add your advice, if you > suggest "事件分段" for Group-By-Variation, you can reply > 1: "事件分段" >
Re: Remove UserGuide before 0.13 from website
Hi, You could find the history UserGuide in the "docs" folder of source code[1] [1] https://archive.apache.org/dist/iotdb/ Thanks, ————— Jialin Qiao Apache IoTDB PMC Trevor Hart 于2023年2月24日周五 01:15写道： > > +1 for having an archive available. I still use these docs and I am using > 0.12. > > > > Thanks > > Trevor Hart > > Ope Limited > > w: http://www.ope.nz/ > > m: +64212728039 > > > > > > > > > On Fri, 24 Feb 2023 06:10:08 +1300 Xiangdong Huang > wrote --- > > > > I think it is ok to remove old docs out of the source repo. > But is there a possible way to archive them? > > Best， > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > 谭新宇 <mailto:1025599...@qq.com.invalid> 于2023年2月23日周四 20:34写道： > > > > Hi, > > > > +1 for removing old docs > > > > > > Thanks > > -- > > Xinyu Tan > > > > > > > 2023年2月23日 18:30，Jialin Qiao <mailto:qiaojia...@apache.org> 写道： > > > > > > Hi, > > > > > > There are 8 versions UserGuide on our website[1], the old versions are > > > hardly used anymore. > > > > > > I prefer to remove the history versions before 0.13 (not include). > > > > > > [1] https://iotdb.apache.org/ > > > > > > Thanks, > > > — > > > Jialin Qiao > > > Apache IoTDB PMC > >
Remove UserGuide before 0.13 from website
Hi, There are 8 versions UserGuide on our website[1], the old versions are hardly used anymore. I prefer to remove the history versions before 0.13 (not include). [1] https://iotdb.apache.org/ Thanks, — Jialin Qiao Apache IoTDB PMC
Re: [VOTE] Apache IoTDB 1.0.1 RC1 release
Hi, +1 (binding) The source release: LICENSE and NOTICE [ok] signatures and hashes [ok] All files have ASF header [ok] could compile from source: ./mvnw clean install [ok] The binary distribution: LICENSE and NOTICE [ok] signatures and hashes [ok] Could run with the following statements [ok] CREATE DATABASE root.turbine; CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; insert into root.turbine.d1(timestamp,s0) values(1,1); insert into root.turbine.d1(timestamp,s0) values(2,2); insert into root.turbine.d1(timestamp,s0) values(3,3); select * from root.**; Thanks, — Jialin Qiao Apache IoTDB PMC Gaofei Cao 于2023年2月17日周五 19:26写道： > > Hi all, > > Apache IoTDB 1.0.1 has been staged under [2] and it’s time to vote > on accepting it for release. All Maven artifacts are available under [1]. > Voting will be open for 72hr. > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > are required to pass. > > Release tag: v1.0.1 > Hash for the release tag: ff6de906748d9ddc1a3b40a768c3de69fecc6733 > > Before voting +1, PMC members are required to download > the signed source code package, compile it as provided, and test > the resulting executable on their own platform, along with also > verifying that the package meets the requirements of the ASF policy > on releases. [3] > > You can achieve the above by following [4]. > > [ ] +1 accept (indicate what you validated - e.g. performed the > non-RM items in [4]) > [ ] -1 reject (explanation required) > > > [1] https://repository.apache.org/content/repositories/orgapacheiotdb-1097 > [2] https://dist.apache.org/repos/dist/dev/iotdb/1.0.1/rc1 > [3] https://www.apache.org/dev/release.html#approving-a-release > [4] > https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release > [5] https://dist.apache.org/repos/dist/dev/iotdb/1.0.1/rc1/RELEASE_NOTES.md > [6] https://dist.apache.org/repos/dist/dev/iotdb/KEYS > > Best, > > Gaofei Cao
Re: [VOTE] Apache IoTDB 0.13.4 RC1 release
Hi, +1 (binding) The source release: apache headers [ok] signatures and hashes [ok] LICENSE and NOTICE [ok] no jar files [ok] could compile from source: ./mvnw.sh clean install -DskipTests [ok] The binary distribution: version number in CLI [ok] signatures and hashes [ok] start in mac, jdk8 [ok] statements executed successfully: [ok] SET STORAGE GROUP TO root.turbine; CREATE TIMESERIES root.turbine.d1.s0 WITH DATATYPE=DOUBLE, ENCODING=GORILLA; insert into root.turbine.d1(timestamp,s0) values(1,1); insert into root.turbine.d1(timestamp,s0) values(2,2); insert into root.turbine.d1(timestamp,s0) values(3,3); select ** from root; Thanks, — Jialin Qiao School of Software, Tsinghua University 刘 旭鑫 于2023年2月7日周二 18:17写道： > > Hi all, > > Apache IoTDB 0.13.4 has been staged under [1] and it’s time to vote on > accepting it for release. All Maven artifacts are available under. Voting > will be open for 72hr. A minimum of 3 binding +1 votes and more binding +1 > than binding -1 are required to pass. > > Release tag: v0.13.4 > > Hash for the release tag: 12260bc6e1eadb8e417752e15cc2813750242ba1 > > Before voting +1, PMC members are required to download the signed source code > package, compile it as provided, and test the resulting executable on their > own platform, along with also verifying that the package meets the > requirements of the ASF policy on releases. [2] > > You can achieve the above by following [3]. > > [ ] +1 accept (indicate what you validated - e.g. performed the non-RM items > in [3]) > [ ] -1 reject (explanation required) > > > [1] https://dist.apache.org/repos/dist/dev/iotdb/0.13.4 > [2] https://www.apache.org/dev/release.html#approving-a-release > [3] > https://cwiki.apache.org/confluence/display/IOTDB/Validating+a+staged+Release > [4] https://dist.apache.org/repos/dist/dev/iotdb/0.13.4/rc1/RELEASE_NOTES.md > [5] https://dist.apache.org/repos/dist/dev/iotdb/KEYS
Re: About naming of VerticallyConcatOperator
+1 for Plan A: Rename it as HorizontallyConcatOperator. — Jialin Qiao Apache IoTDB PMC Xiangwei Wei 于2023年2月6日周一 14:35写道： > > VerticallyConcatOperator is used to horizontally concatenate TsBlocks with > the same timestamp column. For example, > TsBlock A is: > | Time | ColumnA | > |1| 1.0| > TsBlock B is: > | Time | ColumnB | > |1| 2.0| > After the VerticallyConcatOperator, the result will be: > | Time | ColumnA | ColumnB | > |1| 1.0| 2.0 | > > Literally, I think it's a horizontal concatenation. So I suggest rename it > as HorizontallyConcatOperator. > > However, I asked the author about the reason for this name. He said, "From > the view of a table, adding columns means vertical changes." > > What's your opinion? > > +1 for Plan A: Rename it as HorizontallyConcatOperator. > +1 for Plan B: Keep current name VerticallyConcatOperator. > > -- > Best, > Xiangwei Wei
[Result][Vote] Create a repo for iotdb-backup-tool
Hi, The Vote for Creating a repo for iotdb-backup-tool closes. The vote results is: Binding votes: 3 +1, NonBinding votes 8 +1. No +0 or -1 vote. The votes come from: Xiangdong Huang (binding) HW-Chao Wang (binding) Qingxin Feng Guanfei Guo Jeck1son2 Haizeiwang Ximing Gao Guanchu Shen Tao Yu Meng Li Jialin Qiao (binding) The vote passes. Thanks for your votes! https://lists.apache.org/thread/lvsmtz8vk28nxnn9dqzqp4ryxszlvwn0 Bests, — Jialin Qiao Apache IoTDB PMC
Re: [VOTE] Create a repo for iotdb-backup-tool
+1 — Jialin Qiao Apache IoTDB PMC limen...@live.cn 于2023年2月1日周三 19:19写道： > > +1 > > > > limen...@live.cn > > From: Jialin Qiao > Date: 2023-01-31 21:49 > To: dev > Subject: [VOTE] Create a repo for iotdb-backup-tool > Hi, > > Here is a tool for backup IoTDB data: > https://github.com/apache/iotdb/pull/7248 > > As iotdb-backup-tool could be used independently, it could be a separate > repository for users. > > So, I'd like to start a vote for creating a new Repo for iotdb-backup-tool. > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > The vote is open for the next 72 hours and passes if at least three +1 > votes and more +1 votes than -1 votes. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC
Re: Release of 0.13.4
Thank Xuxin! — Jialin Qiao Apache IoTDB PMC 刘 旭鑫 于2023年2月4日周六 21:21写道： > > I am glad to. It would be a great honor for me to be the release manager. > > - > THSS > Liu Xuxin > > ____ > 发件人: Jialin Qiao > 发送时间: 2023年2月4日 13:17 > 收件人: dev@iotdb.apache.org > 主题: Release of 0.13.4 > > Hi all, > > It has been over 3 months since we release 0.13.3. It's time to > release 0.13.4. > > Does anyone want to be the release manager? > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC
Release of 0.13.4
Hi all, It has been over 3 months since we release 0.13.3. It's time to release 0.13.4. Does anyone want to be the release manager? Thanks, — Jialin Qiao Apache IoTDB PMC
Release of IoTDB 0.13.4
Hi all, It has been over 3 months since we release 0.13.3. It's time to release 0.13.4, does anyone want to be the release manager? Thanks, — Jialin Qiao Apache IoTDB PMC
[VOTE] Create a repo for iotdb-backup-tool
Hi, Here is a tool for backup IoTDB data: https://github.com/apache/iotdb/pull/7248 As iotdb-backup-tool could be used independently, it could be a separate repository for users. So, I'd like to start a vote for creating a new Repo for iotdb-backup-tool. Please vote accordingly: [ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove with the reason The vote is open for the next 72 hours and passes if at least three +1 votes and more +1 votes than -1 votes. Thanks, — Jialin Qiao Apache IoTDB PMC
CVE-2023-24830: Apache IoTDB: apache/iotdb-web-workbench: create a user without authorization
Severity: low Description: Improper Authentication vulnerability in Apache Software Foundation Apache IoTDB.This issue affects Apache IoTDB: from 0.13.0 before 0.13.3. References: https://iotdb.apache.org https://www.cve.org/CVERecord?id=CVE-2023-24830
CVE-2023-24829: Apache IoTDB: apache/iotdb-web-workbench: forge the JWTToken to access workbench
Description: Incorrect Authorization vulnerability in Apache Software Foundation Apache IoTDB iotdb-web-workbench.This issue affects users' access to the system without authorization. This CVE is fixed in iotdb-web-workbench tag v0.13.3. References: https://iotdb.apache.org/ https://iotdb.apache.org https://www.cve.org/CVERecord?id=CVE-2023-24829
new committer: Rongzhao Chen
Hi, The Project Management Committee (PMC) for Apache IoTDB has invited Rongzhao Chen to become a committer and we are pleased to announce that he has accepted. Being a committer enables easier contribution to the project since there is no need to go via the patch submission process. This should enable better productivity. You could get the write-access of our repository by: (1) Update your github id in: id.apache.org , also an email (anything to your apache email will forward to this email). (2) Accept the invitation in github to be a member of the apache group. (3) Link your accounts via: git.apache.org (To lighten the third option MFA Status, you need to download Google Authenticator (Google 身份验证器) in your mobile) (4）Sign in github by Google Authenticator (5) Clone repository with ssh. Welcome, Yongzao! Yours, The Apache IoTDB PMC
