Re: Handling Duplicate Timestamps
> 1. Checking before insert if the timestamp already exists and remedy on the > client before resend > 2. Moving to Nanosecond and introducing some insignificant time value to keep > timestamp values unique. Yes these maybe the best solutions for a specific application. Analysis for IoTDB: - Rejecting the write when receiving an existing timestamp in IoTDB is time-costly (IoTDB needs to check historical data). I think we will do not check it until we find a low-latency method. - Allowing multiple value versions for a timestamp may introduce a chain reaction and there may be a lot of codes that should be modified, which is a huge work. There is a new idea (but I have no time to implement it...) - Add a parameter in IoTDB: replace_strategy: first, last, avg etc... - when an existing timestamp arrives, IoTDB accepts it - when IoTDB runs LSM to merge data and meets multiple values for a timestamp, then handles it according to the replace_startegy. The solution may also introduce some work to do... and we need to think carefully the impact to the query process. Need to survey whether this is a common requirement. Best, --- Xiangdong Huang Trevor Hart 于2024年5月14日周二 09:55写道： > > Hello Yuan > > > > Correct, the first timestamp and values should be retained. > > > > I realise this is does not align with the current design. I was just asking > whether there was an existing option to operate to block duplicates. > > > > In a normal RDBMS if you try to insert with a duplicate the insert will fail > with a PK violation. It would be great in some circumstances if IotDB at > least had the option to fail this way. > > > > I am considering some options such as; > > > > 1. Checking before insert if the timestamp already exists and remedy on the > client before resend > > 2. Moving to Nanosecond and introducing some insignificant time value to keep > timestamp values unique. > > > > I have already done something similar to #2 with storing IIS web log files as > they are recorded in seconds and not milliseconds. > > > > Thanks > > Trevor Hart > > > > > On Tue, 14 May 2024 13:29:02 +1200 Yuan Tian > wrote --- > > > > Hi Trevor, > > By "rejects duplicates", you mean you want to keep the first duplicate > timestamp and its corresponding values?(because the following duplicated > ones will be rejected) > > Best regards, > > Yuan Tian > > On Mon, May 13, 2024 at 6:24 PM Trevor Hart <mailto:tre...@ope.nz> wrote: > > > > > > > > > > > Correct. I’m not disputing that. What I’m asking is that it > > would be good to have a configuration that either allows overwrites or > > rejects duplicates.My scenario is request log data from a server (the > > device). As it may be processing multiple requests at once there is a > > chance that there could be colliding time stamps.As it stands now I would > > need to check if the timestamp exists before inserting the data. Which > > obviously affects throughput. Thanks Trevor Hart On Fri, 10 May > > 2024 00:33:40 +1200 Jialin Qiao<mailto:qiaojia...@apache.org> wrote > > Hi, > > In IoT or IIoT scenarios, we thought each data point represent a metric of > > a timestamp.In which case you need to store duplicated values? Take this > > for an example: Time, root.sg1.car1.speed 1, 1 1, 2 Could a car has > > different speed at time 1? Jialin Qiao Yuan Tian < > > mailto:jackietie...@gmail.com> 于2024年5月9日周四 18:51写道： > > Hi Trevor, > > Now > > we > > will override the duplicate timestamp with a newer one. There is > nothing > > we can do about it now. > > Best regards, > --- > Yuan Tian > > > > On Wed, May 8, 2024 at 5:31 PM Trevor Hart <mailto:tre...@ope.nz> > > > > wrote: > > > > > Hello > > > > > > > > I’m aware that when inserting a duplicate timestamp > > the values will be > > overwritten. This will obviously result in data > > loss. > > > > > > > > Is there a config/setting to reject or throw an error > > on duplicate > > inserts? Although highly unlikely I would prefer to be > > alerted to the > > situation rather than lose data. > > > > > > > > I read > > through the documentation but couldn’t find anything. > > > > > > > > > > Thanks > > > > Trevor Hart > > > > > > > > > > > > > >
Re: Splitting up the main repo?
> We surely also pull in a lot of potentially bad dependencies. and this may be a chance to re-check our dependencies and remove unnecessary.. --- Xiangdong Huang Christofer Dutz 于2024年4月5日周五 19:13写道： > > Hi all, > > I just wanted to bring up one idea that we decided in the PLC4X project and > seed the idea, if this would also be worth discussing here. > > So, we were seeing that our build kept on having sub-ideal CVE ratings as we > had dependencies for which CVEs were reported. > However, PLC4X itself has a very limited number of dependencies. The problem > was that we had several “integration” modules, that pulled in Kafka, Calcite, > Nifi and some Eclipse projects. > Also did a lot of our examples pull in various third party libraries, for > which also vulnerabilities were reported. > > We are currently in the process of splitting up our main repository into a > main and an extras repository. > The main contains the core of the project. The extras contains the examples, > additional tools and integration modules (The ones with the many, many > dependencies) > This way we can get a much better secutity standing for the main repo. > > Would this also be a good idea for IoTDB? I know with our dependencies to: > > * Flink > * Grafana > * Hadoop > * Hive > * Spark > * Zeppelin (this one is really bad when it comes to CVEs) > * Pulsar (only examples) > * RabbitMQ (only examples) > * RocketMQ (only examples) > > We surely also pull in a lot of potentially bad dependencies. If we moved > this out the same way we would probably have a much better CVE ranking. > This might become problematic in the future as in Europe and in the US > CRE/PLD and other initiatives are taking form. > > Chris
quarter report of IoTDB
Hi, I finished the draft of the quarter report. Please have a review and give feedback. I will submit it 3 days later. ## Description: The mission of Apache IoTDB is the creation and maintenance of software related to an IoT native database with high performance for data management and analysis ## Project Status: Current project status: ongoing Issues for the board: no ## Membership Data: Apache IoTDB was founded 2020-09-16 (3 years ago) There are currently 61 committers and 29 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: - No new PMC members. Last addition was Steve Yurong Su on 2023-09-28. - No new committers. Last addition was Sicheng Yu on 2023-08-01. ## Project Activity: Software development activity: - IOTDB-1.3.0 was released on 2024-01-01. - We are working on time series computing feature. - We are improving the JVM GC options for supporting diverse workload. - v1.3.1 is on the way. ## Community Health: Overall community health is good. - Joined a meetup titled "Open Source Database Technology" in Shanghai. - The community has more connection with Apache Ratis, and one of IoTDB committer became Ratis's PMC member. - The number of PR, code modifications, and mails decreased, but it is reasonable because there are many developers in Asia and there is a long holiday in this quarter. Best, --- Xiangdong Huang
Re: Inquiry Regarding Key Column Creation for IoTDB
Hi Cheongu Kim, Please send email to dev@iotdb.apache.org for discussion (remember to subscribe the mailing list first). As for your question, maybe the easiest method is add an offset on the timestamp to make they different. e.g., origin data: , , to: , , Best, --- Xiangdong Huang School of Software, Tsinghua University 김천구 于2024年1月16日周二 02:33写道： > > Dear Apache IoTDB Development Team > > Greetings, I hope this email finds you well. My name is Cheongu Kim and I am > reaching out to you today to request some assistance. I currently plan to > test IoTDB with some spatio-temporal(so-called ST) dataset which consists of > time, longitude, latitude, and serial data generated from these 3 elements. > > Since the data is about trajectories of vehicles, there are hundreds of > duplicates at the same time even in one vehicle because of its measurement > method. And the dataset can be identified by the serial data(Since space and > time are unique) > > I need to make a database which can be identified by the serial data, Which > means the serial data must be a key column or at least make sure not to lose > the data because it has the same time value or timestamp. > > I read about IoTDB papers, and explored the website but was unable to find > the information about creating the key column or unique column. I know it's a > time series database so time is the key column but how can I make another key > or unique column in iotdb? > > Thank you in advance for your time and assistance. I look forward to any > insights or information you can provide. > > Sincerely, > > Cheongu Kim > ckim191...@gmail.com
IoTDB quarter report (Q4)
Hi all, the following the quarter report of IoTDB, please give suggestions in 3 days. ## Description: The mission of Apache IoTDB is the creation and maintenance of software related to an IoT native database with high performance for data management and analysis ## Project Status: Current project status: ongoing with high activity Issues for the board: no ## Membership Data: Apache IoTDB was founded 2020-09-16 (3 years ago) There are currently 61 committers and 29 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: - Steve Yurong Su was added to the PMC on 2023-09-28 - Xinyu Tan was added to the PMC on 2023-09-28 - No new committers. Last addition was Sicheng Yu on 2023-08-01. ## Project Activity: This quotar IoTDB community mainly focus on system performance and stability. Release: IOTDB-1.2.2 was released on 2023-10-15. IOTDB-1.2.1 was released on 2023-09-19. IOTDB-1.2.0 was released on 2023-09-08. And, we are releasing v1.3.0. New project: IoTDB Community desides to seperate the TsFile from IoTDB to Apache as a top-level-project. TsFile is a columnar storage file format designed for time series data, which supports efficient compression, high throughput of read and write, and compatibility with various frameworks, such as Spark and Flink. It is easy to integrate TsFile into IoT big data processing frameworks. ## Community Health: We hold an IoTDB Summit 2023 in Beijing on 2023-12-03, nearly 300 people attended on-site and there are about 70k watch online. dev@ traffic needs PMC's attention in the next season. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
Re: Re: Suggestions for Enhancing the IoTDB Contribution Experience
Hi zq, It is appreciated that you did some research on the two PRs, which is very helpful for a developer. I have to say I had no intention to offend your code quality in my last email (I just want to claim the principle to the whole community)... I apologize for that if some words gave you the uncomfortable imposition. One more thing I want to remind is: As the community is driven by diverse developers (rather than any paid employee), we always encourage contributors to put themselves in a more visible position, rather than waiting and believing someone has obligation to do something (though as PMCs, we have some obligations to guaratee the community and the project keep growing). For examp, sending emails to dev@ mailing list to get the attention. Actually you can find many emails like that and it is very useful. One more trivial thing is that the mailing list only support plain text and does not support image/figure. If you want to attach pictures, you can upload it to somewhere and give us the url link. In the end, I have to emphasize once again that I really really appreciate your bug report, pull request, and reviews of the two PRs. I can deeply feel your interestd in Open source community and IoTDB project. Welcome you keeping to contributing to IoTDB and look forward to seeing your more contributions. Best, --- Xiangdong Huang Jialin Qiao 于2023年11月5日周日 19:12写道： > > Hi, > > Thanks for your feedback of this contribution! I support to give more > attention for each PR and do more communication. > Besides, if you have a PR need to be reviewed or just want to do a > modification, welcome to discuss in the mail list. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > > 张 于2023年11月3日周五 18:25写道： > > > After reading PR#11429, upon comparing my PR, it does have more > > advantageous aspects from which I have greatly benefited. However, I need > > to point out something that I find 'unfair' in the process. > > > > *First of all*, my PR was committed on Oct 12, 2023, and no one followed > > up on it before I asked. > > > > > > *The second key point *is that no one gave useful advice on where I > > should make modifications or the aspects I might have overlooked. Thus, I > > didn't have the opportunity to make better revisions... > > > > At the same time, I must clarify that due to my limited familiarity with > > IoTDB, the principle I followed during the modification process was 'to > > avoid making significant changes to the existing logic as much as > > possible.' Therefore, I attempted to make only necessary adjustments based > > on the existing source code. This is evident to anyone who carefully reads > > my commits. However, this might not be a good solution in open-source PRs. > > > > I acknowledge that my PR might be "worse code". Nevertheless, I still hope > > that in future PR reviews for IoTDB, if possible, reviewers will carefully > > review the code. Even if the PR is truly subpar, reviewers should provide > > feedback. This would be incredibly helpful for the person because they > > would know where their issues lie. > > > > The above is what I want to express. I hope that the "processing" will be > > more warming in the future. > > > > Best regards > > > > > > At 2023-11-01 16:01:16, "Xiangdong Huang" wrote: > > >Hi, > > > > > >I read the discussion and all related issues/pull requests, and give > > >some of my comments: > > > > > >It is appreciated for cqzhang's contribution to iotdb, especially for > > >the detailed bug feedback. This is the best contribution to the open > > >source community. > > > > > >Being a TLP open source project of Apache Software Foundation, keeping > > >diversity is the community's goal. > > >Therefore, attracting more and more contributors contributing to IoTDB > > >is what we always want. > > > > > >I have to say comments like "this issue may not be suitable for a > > >newer to our system due to communication costs of these complicated > > >cases" is ABSOLUTLY incorrect. It is very very harmful for the > > >community. > > >"newer" does not mean he/she knows little than "older", and does not > > >mean his/her solution is worse than "older". > > >However, "meritocracy" still takes effect, because hearing a person > > >whom has make some successful experience is usual correct. > > > > > >The only way we need to obey is: accepting better quality of codes, > > >which
Re: Suggestions for Enhancing the IoTDB Contribution Experience
Hi, I read the discussion and all related issues/pull requests, and give some of my comments: It is appreciated for cqzhang's contribution to iotdb, especially for the detailed bug feedback. This is the best contribution to the open source community. Being a TLP open source project of Apache Software Foundation, keeping diversity is the community's goal. Therefore, attracting more and more contributors contributing to IoTDB is what we always want. I have to say comments like "this issue may not be suitable for a newer to our system due to communication costs of these complicated cases" is ABSOLUTLY incorrect. It is very very harmful for the community. "newer" does not mean he/she knows little than "older", and does not mean his/her solution is worse than "older". However, "meritocracy" still takes effect, because hearing a person whom has make some successful experience is usual correct. The only way we need to obey is: accepting better quality of codes, which we still need to improve. For example, I find cqzhang gave a solution like "using 12 months to replace 1 year", which is accpeted in the final PR. This is a kind of contribution. In the closed PR (#11171), the most important shortcoming is lack of UT and IT. In the merged PR(#11323), it also has drawbacks, e.g., "DateTimeUtils.convertDurationStrToLong()" function semantic is still incorrect if the input parameter value is "y". Considering making it easy to let more developers join us, I advocate that: 1. For each public issue in Github/JIRA, claim you are working on it to let all others knowing that. If some other person think he/she want to take over, please also claims it asap, rather than submit PR directly. 2. Even though, the cases that two PRs solve the same issue may still occur. In this case, "better code wins" and meritocracy are still the best way. Best, --- Xiangdong Huang School of Software, Tsinghua University Weihao Li <18110526...@163.com> 于2023年10月31日周二 15:11写道： > > Hi cqzhang7, > As you can see, `not supporting group by nature year` and `not supporting > mixed units in group by` are two separate issues. > 1. For the first issue, we have discussed in > https://github.com/apache/iotdb/pull/11309, and the final PR was appended on > the discuss. > 2. For the second issue, your general idea is right, but there are still too > many other cases need to be considered, like us and ns time precision. After > discuss, we think this issue may not be suitable for a newer to our system > due to communication costs of these complicated cases. You can see the final > PR about this issue https://github.com/apache/iotdb/pull/11429. > Thanks for your issues and perfect fix ideas about them, maybe we can start > from some easy issues. Welcome to continue to contribute to IoTDB. > > > > At 2023-10-31 10:50:50, "张" wrote: > > Dear all, > I am writing to share my experiences and concerns regarding my contributions > to IoTDB. I appreciate your time in reading this and hope that my feedback > will be taken constructively. > > > Here is my experience when submitting pr to apache IoTDB: > On 2023/09/13, I used iotdb at work and found a little problem with group by > year, so I submitted issues https://github.com/apache/iotdb/issues/11133 > On 2023/09/18, After this I submitted a useless PR > https://github.com/apache/iotdb/pull/11171 without reading the source code or > testing it myself, it was really the first time I submitted a PR for an open > source component, but fortunately @HTHou viewed the PR and still replied to > me, so I decided to read the source code and fix the issues. I realized that > iotdb currently does not support the "1mo1d" scenario based on group by > nature month. So I decided to try to finish this part as well. > > > > > > > > On 2023/10/11, After reading the source code, I found new bugs > https://github.com/apache/iotdb/issues/11286, I was motivated to fix them and > work on them over and over again, trying to do my best to make it better, and > then I submitted a new PR https://github.com/apache/iotdb/pull/11290, > expecting anyone to give better suggestions. However, I found that IoTDB > doesn't seem to welcome PRs from the outside, because I have been paying > attention to this matter in the past few days, and for internal PRs, It's > always handled in a timely manner, while for PRs from the outside, it seems > to be another kind of attitude. After I asked to the relevant people in the > community WeChat group, someone did review it on the same day, but after I > replied to them one by one, there was no more responses. > It's now October 31st, 2023, and another two weeks have
Re: [DISCUSS] Apache TsFile Proposal
Hi,
The content looks fine. +1 for name-search.
Best,
---
Xiangdong Huang
Christofer Dutz 于2023年10月25日周三 22:24写道：
>
> More feedback from the board was that the description is pretty long and
> having it shorted would leave us more wiggle-room in the future.
> Also, should we formally have passed a name-search … if you’re ok with
> this, I would start the Name-Search.
>
> So that would result in the following resolution:
>
> WHEREAS, the Board of Directors deems it to be in the best interests
> of
>the Foundation and consistent with the Foundation's purpose to
>establish a Project Management Committee charged with the creation
> and
>maintenance of open-source software, for distribution at no charge
> to
>the public, related to a columnar storage file format designed for
> time
>series data.
>
>NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
>(PMC), to be known as the "Apache TsFile Project", be and hereby is
>established pursuant to Bylaws of the Foundation; and be it further
>
>RESOLVED, that the Apache TsFile be and hereby is responsible for
>the creation and maintenance of software related to a columnar
> storage
>file format designed for time series data.
>
>RESOLVED, that the office of "Vice President, Apache TsFile" be and
>hereby is created, the person holding such office to serve at the
>direction of the Board of Directors as the chair of the Apache
>TsFile Project, and to have primary responsibility for management of
>the projects within the scope of responsibility of the Apache TsFile
>Project; and be it further
>
>RESOLVED, that the persons listed immediately below be and hereby
> are
>appointed to serve as the initial members of the Apache TsFile
>Project:
>
>- Yuan Tian (jackietien at apache dot org)
>- Chao Wang (wangchao316 at apache dot org)
>- Christofer Dutz (cdutz at apache dot org)
>- Jinrui Zhang (xingtanzjr at apache dot org)
>- Steve Yurong Su (rong at apache dot org)
>- Xinyu Tan (tanxinyu at apache dot org)
>- Haohan Hou (haonan at apache dot org)
>- Gaofei Cao (gaogaofei at apache dot org)
> - Jialin Qiao (qiaojialin at apache dot org)
>- Kun Liu (liukun at apache dot org)
>- Houliang Qi (neuyilan at apache dot org)
>- Xiangdong Huang (hxd at apache dot org)
>- Chao Wang (chaow at apache dot org)
>- Jianmin Wang (jimwang at apache dot org)
>- Tian Jiang (jiangtian at apache dot org）
>- Xinyi Zhao (zhaoxinyi at apache dot org)
>- Shuo Zhang (shuozhagn at apache dot org)
>
>NOW, THEREFORE, BE IT FURTHER RESOLVED, that {name of the chair} be
>appointed to the office of Vice President, Apache TsFile, to serve
> in
>accordance with and subject to the direction of the Board of
> Directors
>and the Bylaws of the Foundation until death, resignation,
> retirement,
>removal or disqualification, or until a successor is appointed; and
> be
>it further
>
>RESOLVED, that the Apache TsFile Project be and hereby is tasked
> with
>the migration and rationalization of the Apache IoTDB TsFile
> subproject;
>and be it further
>
>RESOLVED, that all responsibilities pertaining to the Apache IoTDB
>TsFile subproject encumbered upon the Apache IoTDB PMC are hereafter
>discharged.
>
>
>
>
>
> Von: Christofer Dutz
> Datum: Mittwoch, 25. Oktober 2023 um 15:03
> An: dev@iotdb.apache.org
> Betreff: AW: [DISCUSS] Apache TsFile Proposal
> So, it seems that this document is based on an incubation proposal … sorry
> … I just reviewed it on a content-level.
> As we’re going straight TLP this should look quite a bit different. So
> I’ll try to whip up a version:
>
> WHEREAS, the Board of Directors deems it to be in the best interests
> of
>the Foundation and consistent with the Foundation's purpose to
>establish a Project Management Committee charged with the creation
> and
>maintenance of open-source software, for distribution at no charge
> to
>the public, related to a columnar storage file format designed for
> time
>series data, which supports efficient compression, high throughput
> of
>read and write, and compatibility with various frameworks, such as
> Spark
>and Flink. It is easy to integrate TsFile into IoT big data
> processing
>frameworks.
>
Re: [DISCUSS] Champion and PMC Chair of TsFile
Hi, I vote for Christofer Dutz to be the Champion. I think he is the best person for the role. We can feel how he is professional in another email thread (discussion about the proposal and resolution) :D As for the PMC Chair, I think Jialin Qiao is a good choice, he really did a lot on the project, knows everything about the project, including the design, the goal, and pay almost all of his time on the project. Best, --- Xiangdong Huang Jialin Qiao 于2023年10月25日周三 15:54写道： > Hi, > > To enter the Apache, we also need to define the project champion and PMC > chair. > > For Champion, I am honored to invite Chris to be the champion of the > TsFile project. > > For PMC, welcome to recommend yourself and others. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC >
Re: [DISCUSS] Apache TsFile Proposal
> can you add in apache a initial ts-native for the rust project? I think supporting Rust has been introduced: - The initial goals include: Multi-Language Support of TsFile implementations apart from Java, such as C++, Go and Rust. --- Xiangdong Huang Giorgio Zoppi 于2023年10月25日周三 15:46写道： > Hello, > can you add in apache a initial ts-native for the rust project? Sorry for > being pedantic. > Best Regards, > Giorgio > > Il giorno mer 25 ott 2023 alle ore 05:04 Jialin Qiao < > qiaojia...@apache.org> > ha scritto: > > > Hi, > > > > The TsFile proposal is as follows, feel free to give advice :-) > > > > Abstract > > > > TsFile is a columnar storage file format designed for time series > > data, which supports efficient compression, high throughput of read > > and write, and compatibility with various frameworks, such as Spark > > and Flink. It is easy to integrate TsFile into IoT big data processing > > frameworks. > > > > Proposal > > > > TsFile is used for managing time series data. Although it is firstly > > used inside IoTDB, many users and companies use TsFile directly as a > > time series data management solution independently. Besides, there is > > a growing requirement for Multi-Language Support of TsFile > > implementations, such as C++, Go and Rust. > > > > Apache IoTDB community hereby submit this proposal of TsFile as an > > independent Apache project. This proposal outlines the key features > > and benefits of TsFile, along with the integration plan and the need > > for multi-language support. > > > > Background > > > > Time series data is becoming increasingly important in a wide range of > > applications, including IoT, intelligent control, finance, log > > analysis, and monitoring systems. > > TsFile has been developed by Apache IoTDB community in Java and is in > > IoTDB repository. Users could store time series data using TsFile, > > then read and analyze it in IoTDB, Spark and Flink. IoTDB could also > > generate TsFiles and syncronize TsFiles between two IoTDB instances. > > Furthermore, the demand for TsFile implementations in multiple > > programming languages has been growing, as it allows developers to > > leverage TsFile's capabilities in their preferred language. > > > > TsFile offers several distinctive features and benefits: > > Efficient Storage and Compression: TsFile employs advanced compression > > techniques to minimize storage requirements, resulting in reduced disk > > space consumption and improved system efficiency. > > Flexible Schema and Metadata Management: TsFile allows for directly > > write data without pre defining the schema, which is flexible for data > > aquisition. > > High Query Performance with time range: TsFile has indexed devices, > > sensors and time dimensions to accelerate query performance, enabling > > fast filtering and retrieval of time series data. > > Seamless Integration: TsFile is designed to seamlessly integrate with > > existing big data frameworks, such as Spark, Flink and Hadoop. > > > > > > Rationale > > > > Before using TsFile, there doesn't exist a file format for time > > series. The industry companies usually write time series data in > > various user-defined file format without unification, or use general > > columnar file format such as Parquet and ORC, which makes data > > collection and processing complicated without a standard. After > > TsFile, organizations could write data in TsFile inside end devices or > > gateway, then transfer TsFile to the cloud for unified management in > > IoTDB and other systems. In this way, we lower the network > > transmission and the computing resource consumption in the cloud. > > > > Initial Goals > > > > The initial goals include: > > > > Make TsFile an independent project that has its own SDK and > > documentation that is easier to use. > > Multi-Language Support of TsFile implementations apart from Java, such > > as C++, Go and Rust. > > Integrate more encoding and compression method in TsFile. > > More tools for TsFile: visualization tool, parsing tool, repair tool. > > > > > > Current Status > > > > Meritocracy > > > > We plan to invite the IoTDB committer to be the initial committer of > > TsFile. And we would like to follow ASF meritocratic principles and > > invite additional developers to participate. We will establish the > > documentation and encourage and monitor community participation so > > that privilege
Re: Rollcall TsFile initial committers
Hi Jialin, I would also like to be an initial committer for the TsFile project. Best, --- Xiangdong Huang Kun Liu 于2023年10月20日周五 16:28写道： > Hi jialin > > I also would like to be the initial committer for the TsFile project, and > participated the initial version of the TsFile format. > > I am familiar with the `write path` and Rust language, if user and > community need the API for Tsfile with Rust I can help to make it. > > Thanks, > Kun > > > > Jialin Qiao 于2023年10月18日周三 12:48写道： > > > Hi, > > > > Glad to receive so much support! > > I also would like to be the initial committer of TsFile. > > I'm familiar with the tech part(format, read and write) of TsFile, > > application scenarios and solutions with TsFile and IoTDB. > > Looking forward to making TsFile the infrastructure of IoT. > > > > Thanks, > > — > > Jialin Qiao > > Apache IoTDB PMC > > > > Gaofei Cao 于2023年10月18日周三 00:20写道： > > > > > > Hi Jialin, > > > > > > I would like to be the initial committer of TsFile. > > > I participated in the design and development of the first version of > > > TsFile, familiar with the query process, file structure and other > > > system integrations. > > > And now I'm participating in the work to push down value filter. I > > > wish I could bring more features to make TsFile a better file format. > > > > > > > > > Thanks > > > > > > Gaofei Cao > > > > > > Haonan Hou 于2023年10月17日周二 14:39写道： > > > > > > > > > > > > Hi Jialin, > > > > > > > > I would like to be the initial committer of TsFile. > > > > > > > > I participated in the development of TsFile v2 for IoTDB 0.10.x and > > 0.11.x and the update tool for v1 to v2, v2 to v3. > > > > > > > > Thanks > > > > Haonan Hou > > > > > > > > On 2023/10/17 03:40:42 Xinyu Tan wrote: > > > > > Hi Jialin, > > > > > > > > > > I would like to be the initial committer of TsFile. > > > > > > > > > > I participated in the design and development of IoTDB 0.12 cluster > > version combined with TsFile for incremental snapshot. > > > > > > > > > > Now the consensus layer is designing a consensus algorithm based on > > TsFile synchronization, which is expected to make TsFile more general in > > more scenarios involving CDC and consensus. > > > > > > > > > > Thanks > > > > > > > > > > Xinyu Tan > > > > > > > > > > On 2023/10/15 15:22:11 Jialin Qiao wrote: > > > > > > Hi, > > > > > > > > > > > > To make TsFile a TLP, we need to affirm the initial committer > > first. > > > > > > The criteria for becoming the initial committer of TsFile is > > > > > > prelimilary defined as meeting the following two conditions > > > > > > simultaneously. > > > > > > > > > > > > (1) Apache IoTDB PMC member > > > > > > (2) Made contributions to the TsFile module > > > > > > > > > > > > If you have made significant contributions to the TsFile module, > > you > > > > > > can also apply to become an initial committer. > > > > > > > > > > > > If you would like to be the initial committer of TsFile, please > > reply > > > > > > to this email in 1 week :-) > > > > > > > > > > > > Thanks, > > > > > > — > > > > > > Jialin Qiao > > > > > > Apache IoTDB PMC > > > > > > > > > > > > > >
Re: [DISCUSS] Incubating TsFile as an independent Apache project?
Hi, +1 for make TsFile independent and then it will be reasonable for having multiple TsFile language implementation. > If there are enough seasoned and experienced Apache forlks in the PMC, I’d even support. A direct to TLP route. +1, too. And, I think there are really enough forlks :D. Best, --- Xiangdong Huang Christofer Dutz 于2023年10月7日周六 19:47写道： > Hi all, > > not even sure we’d need to go the route throught the incubator. > If there are enough seasoned and experienced Apache forlks in the PMC, I’d > even support. A direct to TLP route. > > Chris > > Von: Jialin Qiao > Datum: Samstag, 7. Oktober 2023 um 03:59 > An: dev@iotdb.apache.org > Betreff: Re: [DISCUSS] Incubating TsFile as an independent Apache project? > Hi, > > +1 for Apache TsFile. > > 1. Industrial time series data management is becoming more and more > important, so a file format project for time series data is necessary. > 2. When TsFile becomes an independent project, it could be developed > and released without impact the upgrading of IoTDB. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > 冯 庆新 于2023年10月7日周六 10:22写道： > > > > Hi Steve Su > > > > I think this is a great idea and I fully support it. > > I believe Tsfile can generate greater influence and value. > > > > Thanks > > B.R. > > Qingxin Feng > > > > > > 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送 > > > > 发件人: Steve Yurong Su<mailto:r...@apache.org> > > 发送时间: 2023年10月7日 0:53 > > 收件人: dev@iotdb.apache.org<mailto:dev@iotdb.apache.org> > > 主题: [DISCUSS] Incubating TsFile as an independent Apache project? > > > > Hi folks, > > > > Recently, I've been thinking about something that has been on my mind: > > I believe TsFile has the potential to develop as an independent Apache > > project (note, not a subproject like apache-iotdb-tsfile), just like > > ORC, Parquet, and Arrow. How about incubating TsFile as an independent > > Apache project? > > > > The main reason I came up with this idea is because I realized that > > TsFile is not just the data file format of IoTDB, but it can also be a > > standard format in the IoT field. It provides excellent support for > > time series data, offering various encoding and compression methods. > > It can be easily self-parsed and its statistical information enables > > fast data querying by big data software. I believe that every IoT > > software could benefit directly by supporting TsFile as a storage > > format. > > > > On the other hand, I hope to leverage the power of the community to > > drive the development of the TsFile file format. Just imagine, TsFile > > could have its own independent ecosystem: parsing tools in various > > languages, connectors to various big data software, and so on. By > > separating TsFile from IoTDB, we have the opportunity to attract more > > community contributors and introduce more ideas, thus refining the > > functionality and performance of TsFile. > > > > What do you think about this idea? > > > > In addition, how can TsFile, as a part of the current IoTDB project, > > become an independent Apache project? Should it go through the Apache > > Foundation incubation process again, or can it directly become a > > top-level project? It may be too early to ask this question now, but I > > would also like to hear your opinions. > > > > Thanks, > > Steve Su > > >
Re: [application] apply for permission to assign issues to zdandsomsp
Hi zdandsomsp, done. --- Xiangdong Huang School of Software, Tsinghua University ZDandsomSP <1115854...@qq.com.invalid> 于2023年9月19日周二 19:41写道： >
Re: AW: [DISCUSS] Updating to a newer Thrift version?
Hi, one more concern, is Thrift v0.19 compatible with Thrift v0.14? (It is better to be compatible for all IoTDB v1.x. ) --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Haonan Hou 于2023年9月18日周一 18:13写道： > It seems that Thrift 0.19.0 is released and supports Java 8 again. ;-) > https://github.com/apache/thrift/blob/master/CHANGES.md > > Best, > Haonan > > On 2023/08/04 10:01:39 Christofer Dutz wrote: > > Ok … > > > > So, it turns out that the Thrift folks intentionally updated to Java 11 > but took that back and the next release should be based on Java 8 again. > > So hopefully that will come soon and then we could have a look about > compatibility of the generated code. I guess if the wire-protocol didn’t > change there should generally be no reason for it to not be compatible, and > if they don’t release a major version, usually this compatibility should > stay intact (if they sort of follow SemVer) > > > > I guess the best path of action would be to do nothing right now … as an > alternative, we could of course turn on the Java compilation when building > thrift and to bundle the jar built by that in the assembly. > > If we then replace the thrift-lib dependency with a “system” scope > dependency, we could already do that now. But admittedly I would not be in > favor of doing that. I would recommend for us to wait for the next official > release. > > > > Chris > > > > Von: Christofer Dutz > > Datum: Dienstag, 1. August 2023 um 08:29 > > An: dev@iotdb.apache.org > > Betreff: Re: [DISCUSS] Updating to a newer Thrift version? > > I opened a ticket in their jira. Might even prepare a pr... Shouldn't be > too difficult. > > > > Chris > > > > Gesendet von Outlook für Android<https://aka.ms/AAb9ysg> > > > > From: 谭新宇 <1025599...@qq.com.INVALID> > > Sent: Monday, July 31, 2023 1:57:28 PM > > To: dev@iotdb.apache.org > > Subject: Re: [DISCUSS] Updating to a newer Thrift version? > > > > Hi, Chris > > > > In the latest version of thrift, there are some improvements we'd like > to make, For example, https://issues.apache.org/jira/browse/THRIFT-5502 > will down-cycled "connection reset" warn logs. > > > > +1 for upgrading thrift. > > > > > > Thanks > > > > Xinyu Tan > > > > > 2023年7月31日 19:50，Christofer Dutz 写道： > > > > > > Hi all, > > > > > > While working on the cleanup of the build, I noticed we’re working > with Thrift in version 0.14.1 however the latest version is 0.18.1 > > > > > > Is there a reason we’re sticking to a two-year older version than the > newest? > > > > > > If not: with the pom-cleanup refactoring it should be a thing of > minutes to update this. > > > > > > Chris > > > > > >
iotdb quarter report (Q3)
Hi all, The following is the draft of Board report. Please have a review and I will submit it two days later. ## Description: The mission of Apache IoTDB is the creation and maintenance of software related to an IoT native database with high performance for data management and analysis ## Project Status: Current project status: Ongoing with high activity. A good news is IoTDB's performance ranks first on Benchant.com. (https://benchant.com/ranking/database-ranking) Issues for the board: no. ## Membership Data: Apache IoTDB was founded 2020-09-16 (3 years ago) There are currently 61 committers and 27 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: - No new PMC members. Last addition was Chao Wang on 2022-12-12. - Yanze Chen was added as committer on 2023-07-25 - Lanyu Liao was added as committer on 2023-07-25 - Sicheng Yu was added as committer on 2023-08-01 New PMC election is in progress. ## Project Activity: IOTDB-1.2.0 was released on 2023-09-08. IOTDB-1.1.2 was released on 2023-07-28. IOTDB-1.1.1 was released on 2023-06-14. ## Community Health: The community looks healthy and we are trying to develop its ecosystem. At the IoT Meetup Hessen on July 14, 2023, in Frankfurt am Main, insights into the technology and use cases of Apache IoTDB and Apache PLC4X in bridging IT & OTwere shared. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
Re: [DISCUSS] Adding the generation of sboms to our build?
+1 for moving to the master branch. --- Xiangdong Huang School of Software, Tsinghua University Christofer Dutz 于2023年8月1日周二 22:00写道： > > I added the config to my pr here: > https://github.com/apache/iotdb/pull/10742/commits/c4f4d2e874fd7c1ae4332062e29770925dce7024 > > Chris > > > Von: Xiangdong Huang > Datum: Samstag, 29. Juli 2023 um 08:48 > An: dev@iotdb.apache.org > Betreff: Re: [DISCUSS] Adding the generation of sboms to our build? > Cool, CycloneDX is famous. Look forward! > --- > Xiangdong Huang > > > Christofer Dutz 于2023年7月15日周六 22:59写道： > > > > Well in PLC4X the plugin generates an XML version of the SBOM. > > We’re using this plugin: > > https://github.com/CycloneDX/cyclonedx-maven-plugin > > > > Chris > > > > Von: Xiangdong Huang > > Datum: Samstag, 15. Juli 2023 um 07:58 > > An: dev@iotdb.apache.org > > Betreff: Re: [DISCUSS] Adding the generation of sboms to our build? > > Hi Chris, > > > > Look forward! SBOM has also received a lot of attention in China. > > Which kind of format/standard it will obey? > > > > Best, > > --- > > Xiangdong Huang > > > > Christofer Dutz 于2023年7月14日周五 21:28写道： > > > > > > Hi all, > > > > > > here in Europe we’re currently preparing for quite a bit of an earthquake > > > caused by the Cyber-Resiliency-Act. In some projects I’m involved in > > > (Mainly PLC4X) I’ve started initiating small changes which could make us > > > come out without too many problems. > > > > > > One thing that seems to be coming up in both the EU as well as the US > > > acts, are the requirement to publish SBOM information (Software Bill Of > > > Material). As we are also using Maven as a build tool, I’ve got a > > > configuration in our poms that ensures an Apache release also produces an > > > SBOM, that we will be able to deploy. > > > > > > Are we interested in adding that to the IoTDB build? > > > > > > Chris
Re: [DISCUSS] Adding the generation of sboms to our build?
Cool, CycloneDX is famous. Look forward! --- Xiangdong Huang Christofer Dutz 于2023年7月15日周六 22:59写道： > > Well in PLC4X the plugin generates an XML version of the SBOM. > We’re using this plugin: > https://github.com/CycloneDX/cyclonedx-maven-plugin > > Chris > > Von: Xiangdong Huang > Datum: Samstag, 15. Juli 2023 um 07:58 > An: dev@iotdb.apache.org > Betreff: Re: [DISCUSS] Adding the generation of sboms to our build? > Hi Chris, > > Look forward! SBOM has also received a lot of attention in China. > Which kind of format/standard it will obey? > > Best, > --- > Xiangdong Huang > > Christofer Dutz 于2023年7月14日周五 21:28写道： > > > > Hi all, > > > > here in Europe we’re currently preparing for quite a bit of an earthquake > > caused by the Cyber-Resiliency-Act. In some projects I’m involved in > > (Mainly PLC4X) I’ve started initiating small changes which could make us > > come out without too many problems. > > > > One thing that seems to be coming up in both the EU as well as the US acts, > > are the requirement to publish SBOM information (Software Bill Of > > Material). As we are also using Maven as a build tool, I’ve got a > > configuration in our poms that ensures an Apache release also produces an > > SBOM, that we will be able to deploy. > > > > Are we interested in adding that to the IoTDB build? > > > > Chris
Re: Great job on the restructuring!
welcome to continuously refactor and optimize the project structure :D --- Xiangdong Huang Christofer Dutz 于2023年7月27日周四 22:48写道： > > Hi Folks, > > I sort of missed most of it as git tends to keep the moved directories, if > there’s something in there … as they all contained “target” directories, > I just noticed the cleaned-up structure of the project … awesome work. > > I took the liberty of applying the same structural changes on the reactor (No > worries, in a branch) > There I introduced intermediate sub-parent poms. > Also did I clean up the pom structure, where it needed some cleaning up. > > Now in IntelliJ (and probably every other Maven-Enabled IDE) the project > should also be structured nicely, and the modules have equally formatted > names. > > The changes should not have any real impact on the project and the build. > > In general, I would prefer adjusting the artifact-ids to replicate the module > structure, but this is already a lot better, and I wasn’t sure of the > implications of artifact renaming in upstream builds. > > Chris
Re: [DISCUSS] Refactor the way we're building/using Thrift
Hi Chris, > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). I quickly check current compile-tool, now it is only used by client-cpp. So is it really necessary to move to the root folder? > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). Yes, I also agree to compile tools ourselves, rather than download from somewhere.. (and just run once for each operation system.) > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture +1. Best, ------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年7月28日周五 20:42写道： > > Hi, > > So, I've been fighting to simplify our use of Thrift, as there were several > problems with how we do it today on my M2 Mac (Mostly however to an updated > Homebrew, which uses different paths). > > While going through the project, I noticed that everything in "protocol" > downloads a pre-built executable, which is simply downloaded and run this > doesn’t seem to run on Arm64 (new M1 and M2 Macs). > > I would like to move the compile-tools directory into the root of the project > and detach it from the rest of the project (So it’s not called from the build > if we build the project). > > This only needs to be run every now and then if something fundamental changes > (Like new OSes or new CPU Architectures). > I would like to update that module, so it will not only build an executable, > but also package the includes and the libraries for that OS and Architecture, > so we can use that in the client-cpp and the client-cpp-example. > I'd like to give it a try if we could also use this approach for the libs and > completely get rid of the thrift compilation as part of the build. > > What do you think? > > Chris >
Re: [DISCUSS] Adding the generation of sboms to our build?
Hi Chris, Look forward! SBOM has also received a lot of attention in China. Which kind of format/standard it will obey? Best, --- Xiangdong Huang Christofer Dutz 于2023年7月14日周五 21:28写道： > > Hi all, > > here in Europe we’re currently preparing for quite a bit of an earthquake > caused by the Cyber-Resiliency-Act. In some projects I’m involved in (Mainly > PLC4X) I’ve started initiating small changes which could make us come out > without too many problems. > > One thing that seems to be coming up in both the EU as well as the US acts, > are the requirement to publish SBOM information (Software Bill Of Material). > As we are also using Maven as a build tool, I’ve got a configuration in our > poms that ensures an Apache release also produces an SBOM, that we will be > able to deploy. > > Are we interested in adding that to the IoTDB build? > > Chris
Re: Usage of Singleton pattern?
+1, too. --- Xiangdong Huang Jialin Qiao 于2023年7月14日周五 21:30写道： > > +1 for cleaning it > — > Jialin Qiao > Apache IoTDB PMC > > Yuan Tian 于2023年7月14日周五 21:28写道： > > > > Sure! Do as you want! We would love to see these refactoring prs. > > > > > > Best, > > -- > > Yuan Tian > > > > On Fri, Jul 14, 2023 at 9:20 PM Christofer Dutz > > wrote: > > > > > > Aaah cool, > > > > > > and sorry for asking so much … much of IoTDBs history is not recorded and > > > I’m just starting to get my hands dirty. > > > I am afraid, that I’ll probably have more questions like this as they > > > come up. > > > > > > But that means, if I see something that can be refactored quickly, I can > > > create PRs for stuff like that? > > > > > > Just asking, cause I like to do simple maintenance tasks in order to > > > think about other problems. > > > In PLC4X I usually did cleanups regularly for such reasons. > > > > > > Chris > > > > > > > > > Von: Yuan Tian > > > Datum: Freitag, 14. Juli 2023 um 13:58 > > > An: dev@iotdb.apache.org > > > Betreff: Re: Usage of Singleton pattern? > > > Hi Chris, > > > > > > Actually, most of the existing Singleton pattern Class is just for > > > simplifying things during development. And we did run into trouble > > > when writing some UTs(In such case, I will change related Singleton > > > pattern Class ). > > > But there are always higher priority tasks, so there is no time to > > > change all these singletons. Good news is that we have strictly > > > controlled the use of singletons in the new PRs, so there should be no > > > more new ones. > > > > > > Best, > > > > > > Yuan Tian > > > > > > On Fri, Jul 14, 2023 at 7:14 PM Christofer Dutz > > > wrote: > > > > > > > > Hi all, > > > > > > > > as I’m digging into IoTDB more and more, I noticed a quite excessive > > > > usage of the Singleton pattern … some times even for things where it > > > > doesn’t really make much sense. For example in SystemFileFactory, the > > > > methods don’t really have much state and simply fire RuntimeExceptions > > > > if fsType is initialized with HDFS. > > > > > > > > Singletons make it a bit tricky to run tests in parallel and to do > > > > dynamic reloading of modules (like in OSGI). Most projects I’ve worked > > > > on try to limit the use of singletons to an absolute minimum due to > > > > these problems. > > > > > > > > I guess embedding IoTDB in an OSGI environment hasn’t been a use-case > > > > yet, but I could imagine that random test failures while running tests > > > > in parallel probably could have caused some issues. > > > > > > > > Was this a design decision that is based on some real-world necessity, > > > > or for simplifying things during development? > > > > I don’t want to put too much effort into proposing cleaning up things, > > > > if there was a conscious decision to do things otherwise. > > > > > > > > Chris > > > >
Re: [DISCUSS] Setup Sonarcloud for IoTDB?
+1 for renaming. BTW, seems IoTDB's sonar setting has some errors that test code coverage can not be catched. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Yuan Tian 于2023年7月15日周六 09:50写道： > > We would like to see that for sure. > > > Best, > -- > Yuan Tian > > On Fri, Jul 14, 2023 at 10:13 PM Christofer Dutz > wrote: > > > > We have to file a Jira ticket for Infra … but wanted to know, if it would > > be ok for you folks for me to do that. > > > > Chris > > > > Von: Yuan Tian > > Datum: Freitag, 14. Juli 2023 um 15:40 > > An: dev@iotdb.apache.org > > Betreff: Re: [DISCUSS] Setup Sonarcloud for IoTDB? > > Aha, that's right. Do you know how to change that name? I didn't find > > where to change that name before. > > > > Best, > > > > Yuan Tian > > > > On Fri, Jul 14, 2023 at 9:31 PM Christofer Dutz > > wrote: > > > > > > Hehe … ok … so because of the “incubator” in there, I didn’t find it ;-) > > > > > > Chris > > > > > > > > > Von: Jialin Qiao > > > Datum: Freitag, 14. Juli 2023 um 15:29 > > > An: dev@iotdb.apache.org > > > Betreff: Re: [DISCUSS] Setup Sonarcloud for IoTDB? > > > Hi, > > > > > > We have the sonar cloud: > > > https://sonarcloud.io/project/overview?id=apache_incubator-iotdb > > > But it is only for committers: https://github.com/apache/iotdb/pull/10548 > > > > > > Thanks, > > > — > > > Jialin Qiao > > > Apache IoTDB PMC > > > > > > Christofer Dutz 于2023年7月14日周五 21:24写道： > > > > > > > > Hi folks, > > > > > > > > I just had a look and noticed that we currently don’t have Sonarcloud > > > > analysis setup. > > > > At apache we have free access to using it for open-soucre and I’d like > > > > to suggest to start using it. Some of the things it reports are > > > > definitely going to be false positives, but it also spots a lot of > > > > things that could cause problems before anyone has to file a bug report. > > > > > > > > What are your thoughts on this? > > > > > > > > Chris > > > >
Re: What are all the empty "ext" directories for?
> I can imagine what it’s used for … I was more just asking, why it’s creating > these directories locally instead of just looking if they exist. Always feels > a bit odd, if loads of empty directories are in my filesystem. agree with that feeling... this can be indeed an improvement if it does not bring too much complexity. Best, --- Xiangdong Huang Christofer Dutz 于2023年7月14日周五 14:30写道： > > Hi Xiangdong, > > I can imagine what it’s used for … I was more just asking, why it’s creating > these directories locally instead of just looking if they exist. Always feels > a bit odd, if loads of empty directories are in my filesystem. > > Chris > > Von: Xiangdong Huang > Datum: Freitag, 14. Juli 2023 um 07:51 > An: dev@iotdb.apache.org > Betreff: Re: What are all the empty "ext" directories for? > "ext" refers to "extension", and its subfolder: > - trigger, similar with trigger in RDBMS. > - udf, user define functions. > - PIPE, a new function for ETL or data subscription. > > All the above need to users claiming their data processing logic using > jar files. > So as Trevor said, you can put your jar files there. > > > Best, > --- > Xiangdong Huang > > 18110526...@163.com <18110526...@163.com> 于2023年7月14日周五 08:55写道： > > > > We have written the relevant documentation, you can see > > https://apache-iotdb.feishu.cn/docx/SG1DdQklco67oaxSLbYcghWxnkc. > > > 2023年7月12日 20:45，Christofer Dutz 写道： > > > > > > Hi, > > > > > > so playing around with IoTDB running in embedded mode, I noticed that it > > > creates a directory “data” which totally makes sense to me, as it > > > contains all the data. > > > However it also creates a whole tree under “ext” which just seems to > > > contain loads of empty directories. > > > What’s the purpose of these and could we prevent them from being created, > > > if they are not needed? > > > > > > ext > > > /pipe > > > /install > > > /tmp > > >/trigger > > > /install > > > /tmp > > >/udf > > > /install > > > /tmp > > > > > > > > > Chris > >
Re: What are all the empty "ext" directories for?
"ext" refers to "extension", and its subfolder: - trigger, similar with trigger in RDBMS. - udf, user define functions. - PIPE, a new function for ETL or data subscription. All the above need to users claiming their data processing logic using jar files. So as Trevor said, you can put your jar files there. Best, --- Xiangdong Huang 18110526...@163.com <18110526...@163.com> 于2023年7月14日周五 08:55写道： > > We have written the relevant documentation, you can see > https://apache-iotdb.feishu.cn/docx/SG1DdQklco67oaxSLbYcghWxnkc. > > 2023年7月12日 20:45，Christofer Dutz 写道： > > > > Hi, > > > > so playing around with IoTDB running in embedded mode, I noticed that it > > creates a directory “data” which totally makes sense to me, as it contains > > all the data. > > However it also creates a whole tree under “ext” which just seems to > > contain loads of empty directories. > > What’s the purpose of these and could we prevent them from being created, > > if they are not needed? > > > > ext > > /pipe > > /install > > /tmp > >/trigger > > /install > > /tmp > >/udf > > /install > > /tmp > > > > > > Chris >
Re: New Repo: iotdb-docs
If so, do not forget to modify the iotdb-website project to compile the docs. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 HW-Chao Wang <576749...@qq.com.invalid> 于2023年7月7日周五 19:38写道： > > +1 > > > > ---Original--- > From: "Yuan Tian" Date: Fri, Jul 7, 2023 19:31 PM > To: "dev" Subject: Re: New Repo: iotdb-docs > > > +1 > > On Fri, Jul 7, 2023 at 11:58 AM Jialin Qiao wrote: > > Hi, > > To manage our different versions of doc uniformly and clean the folder > of code repo, I'd like to create a new repo: iotdb-docs. > > Then, we could put all versions(0.13, 1.0, 1.1, 1.2) of docs in the > main branch of iotdb-docs, along with the sidebar. It will be easier > to update the docs. > > What do you think? > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC
Re: Building the jdbc-diver as a fat-jar?
Hi, Actually we have the profile. Using "-P get-jar-with-dependencies" can active it. Maybe we can enable it by default. Best, --- Xiangdong Huang Trevor Hart 于2023年7月5日周三 03:59写道： > > > > > > 100% agree. I have been building my own fat jar version for the > past few years. Thanks Trevor HartOpe Limitedw: www.ope.nzm: +64212728039 > On Wed, 05 Jul 2023 06:07:39 +1200 Christofer > Dutz wrote Hi all, today I wanted to play > around with the 1.3.0-SNAPSHOT version and wanted to use IntelliJ as the SQL > client …. But when adding IoTDB JDBC driver I noticed, we’re currently only > building a default jar. This is quite a pain, if you want to manually deploy > the jdbc driver anywhere … locally I added a config, that makes the build > also build a fat-jar “iotdb-jdbc-1.3.0-SNAPSHOT-jar-with-dependencies.jar” … > would it be ok for me to commit that change? I think it’s quite useful. Chris > > > > > >
Re: Refactor the package structure of IoTDB
> I totally agree and support changing the package structure, as it was > something I stumbled over some time ago when I was trying to integrate IoTDB > into something. I missed some discussion. So does the new package structure can solve previous problems? Best, --- Xiangdong Huang Christofer Dutz 于2023年7月1日周六 00:09写道： > > LOL … well …. Then I still agree ;-) > > But I am sure that starting Monday I’ll be doing a hell of a lot more work on > and with IoTDB … looking forward to it ;-) > > Chris > > Von: Jialin Qiao > Datum: Freitag, 30. Juni 2023 um 17:00 > An: dev@iotdb.apache.org > Betreff: Re: Refactor the package structure of IoTDB > Hi Chris, > > This change originated from your advice in Slack :) > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > Christofer Dutz 于2023年6月30日周五 16:07写道： > > > > Hi Gaofei, > > > > I totally agree and support changing the package structure, as it was > > something I stumbled over some time ago when I was trying to integrate > > IoTDB into something. > > > > Chris > > > > > > > > Von: Gaofei Cao > > Datum: Dienstag, 27. Juni 2023 um 15:19 > > An: dev@iotdb.apache.org > > Betreff: Refactor the package structure of IoTDB > > Hi All, > > > > To make the the package structure of IoTDB more concise and tidy, we > > have moved the package `antlr`, `confignode`, `consensus`, `metrics`, > > `mlnode`, 'node-commons' and `server` to a new package `iotdb-core`. > > > > The package structure before changed: > > Parent > > ├──antrl > > ├──confignode > > ├──consensus > > ├──iotdb-api > > ├──iotdb-client > > ├──iotdb-connector > > ├──iotdb-protocol > > ├──library-udf > > ├──metrics > > ├──mlnode > > ├──node-commons > > ├──server > > ├──tsfile > > > > > > The package structure after changed (master branch): > > Parent > > ├── iotdb-api > > ├── iotdb-client > > ├── iotdb-connector > > ├── iotdb-core > > └── antlr > > └── confignode > > └── consensus > > └── datanode (sever package before changed) > > └── metrics > > └── mlnode > > └── node-commons > > └── tsfile > > ├── iotdb-protocol > > ├── library-udf > > > > > > Thanks. > > > > Gaofei Cao > > Apache IoTDB PMC
Re: New subscriber Hongzhi Gao's self-introduction
Hi Hongzhi, This is a belated welcome. Look forward to your contribution to the community. Best, --- Xiangdong Huang Hongzhi Gao <761417...@qq.com.invalid> 于2023年5月27日周六 13:28写道： > > Dear friends, > My name is Hongzhi Gao. I have been working in a state-owned > enterprise in Haidian, Beijing for more than one year after my master's > degree. I am good at C++ golang and other programming languages. In my spare > time, I am interested in database technology. I hope to understand and learn > IoTDB and make contributions to its development in the future. > > > Hongzhi Gao
[discuss] move jenkins build message to a new mailing list
Hi all, How do you think about move jenkins's notification "build-failure"/"build-stable" to a new mailing list? Now, it is sent to dev@, and some people may think it is annoying.. So, how do you think about it? option 1: move to an existed mailing list: notifications@ (which is for jira issues now) option 2: create a new mailing list. option 3: keep current status Best, --- Xiangdong Huang
Re: Re: Re: Re: Interest in Joining IoTDB Team as a Developer
Hi Qiuyu, now done. :D --- Xiangdong Huang Luo Qiuyu 于2023年6月28日周三 15:44写道： > > Dear Huang and Apache IoTDB PMC, > > It seems that my jira account isn't valid yet. Since public signup for Apache > IoTDB project is disabled now, my request to an account requires approval by > the PMC. > > I have requested a JIRA account (username: qiuyuluo) 10 days ago but have not > received any response, so now I'm not able to log in. Could you kindly update > me on the status of my request? > > Thank you. > > Sincerely, > Qiuyu Luo > > > > > > > > > > > - Original Message - > From: "Xiangdong Huang" > To: dev@iotdb.apache.org > Sent: Tue, 27 Jun 2023 21:59:34 +0800 > Subject: Re: Re: Re: Interest in Joining IoTDB Team as a Developer > > Hi, yes I mean your public name. But, I still can not get your info > using "QIUYU LUO". > --- > Xiangdong Huang > > Luo Qiuyu 于2023年6月27日周二 21:25写道： > > > > Hi Huang, > > > > > > Thanks so much for your patience with this confusion. > > > > > > But as for your comment about nickname, I still can't clarify what it > > points to. When I signed up for a Jira account, I only provided a username > > and real (public) name, not a nickname. > > > > > > My real (public) name: QIUYU LUO > > > > > > Hope it is useful. > > > > > > Best, > > Qiuyu Luo > > > > > > > > > > > > > > > > > > > > > > > > At 2023-06-27 20:46:55, "Xiangdong Huang" wrote: > > >Hi Qiuyu, > > > > > >Confluence added. > > > > > >Regarding to your JIRA id: you need to provide your nick name, rather > > >than your username (account id). > > > > > >Best, > > > > > >--- > > >Xiangdong Huang > > > > > > > > >Luo Qiuyu 于2023年6月26日周一 18:28写道： > > >> > > >> Dear Huang, > > >> > > >> > > >> > > >> > > >> Thanks for your prompt concern. > > >> > > >> > > >> > > >> > > >> I checked my id and these are correct: > > >> > > >> > > >> > > >> > > >> > Jira ID: qiuyuluo > > >> > > >> > Confluence ID: qiuyuluo (full name: LUO QIUYU) > > >> > > >> > > >> > > >> > > >> Sorry for any trouble it may cause. > > >> > > >> > > >> > > >> > > >> Sincerely, > > >> Qiuyu Luo > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> At 2023-06-26 18:13:50, "Xiangdong Huang" wrote: > > >> >Hi Qiuyu, > > >> > > > >> >Welcome to IoTDB community. > > >> > > > >> >Can you check your jira id and confluence id? I can not find them (you > > >> >need to register on [1] and [2] yourself). > > >> > > > >> >[1] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1 > > >> >[2] https://cwiki.apache.org/confluence/display/IoTDB > > >> > > > >> >Best, > > >> >--- > > >> >Xiangdong Huang > > >> > > > >> >qiuyuluo777 <471289...@qq.com.invalid> 于2023年6月26日周一 17:32写道： > > >> >> > > >> >> Dear IoTDB Team, > > >> >> > > >> >> > > >> >> I am a CS graduate student at Southern Methodist University and used > > >> >> to be a software development intern at the Tsinghua Sichuan Energy > > >> >> Internet Research Institute. I have extensive experience in Java and > > >> >> various relational and non-relational databases. > > >> >> > > >> >> > > >> >> I am interested in joining your team as a developer for IoTDB, as I > > >> >> find the project impressive and believe my skills would be an > > >> >> addition. > > >> >> > > >> >> > > >> >> Thank you for your consideration. > > >> >> > > >> >> > > >> >> Sincerely, > > >> >> Qiuyu Luo > > >> >> > > >> >> > > >> >> Jira ID: qiuyuluo777 > > >> >> Confluence ID: qiuyuluo
Re: I want to become an IoTDB develope
Hi Min Zhang, I have grant your confluence account permission. Now you can write documents on that. I do not grant your jira account now, and let's do that when you want to assign some issues to you. BTW, please be sure that you are using plain text mode when you write emails to dev@ mailing list. (Otherwise we are hard to read your email.) Best, --- Xiangdong Huang 张先生|| <806007...@qq.com.invalid> 于2023年6月27日周二 21:28写道： > > Dear PCM， >I am currently a graduate student in Southwest > University of Science and Technology andused to bing a Joint > training studentsintern at the Tsinghua Sichuan Energy Internet > Research Institute.我I am currently proficient in languages such as > Java, C, Python, and familiar with databases such as MySQL, Postgres, and > Redis . > soI want to become a contributor to IoTDB and hope > that you can approve my application. > > > > > > Zhangmin > > > > Jira ID: 账号：Zhang Min > Confluence ID:806007526 > > > 张先生|| > 806007...@qq.com > > > >
Re: Re: Re: Interest in Joining IoTDB Team as a Developer
Hi, yes I mean your public name. But, I still can not get your info using "QIUYU LUO". --- Xiangdong Huang Luo Qiuyu 于2023年6月27日周二 21:25写道： > > Hi Huang, > > > Thanks so much for your patience with this confusion. > > > But as for your comment about nickname, I still can't clarify what it points > to. When I signed up for a Jira account, I only provided a username and real > (public) name, not a nickname. > > > My real (public) name: QIUYU LUO > > > Hope it is useful. > > > Best, > Qiuyu Luo > > > > > > > > > > > > At 2023-06-27 20:46:55, "Xiangdong Huang" wrote: > >Hi Qiuyu, > > > >Confluence added. > > > >Regarding to your JIRA id: you need to provide your nick name, rather > >than your username (account id). > > > >Best, > > > >--- > >Xiangdong Huang > > > > > >Luo Qiuyu 于2023年6月26日周一 18:28写道： > >> > >> Dear Huang, > >> > >> > >> > >> > >> Thanks for your prompt concern. > >> > >> > >> > >> > >> I checked my id and these are correct: > >> > >> > >> > >> > >> > Jira ID: qiuyuluo > >> > >> > Confluence ID: qiuyuluo (full name: LUO QIUYU) > >> > >> > >> > >> > >> Sorry for any trouble it may cause. > >> > >> > >> > >> > >> Sincerely, > >> Qiuyu Luo > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> At 2023-06-26 18:13:50, "Xiangdong Huang" wrote: > >> >Hi Qiuyu, > >> > > >> >Welcome to IoTDB community. > >> > > >> >Can you check your jira id and confluence id? I can not find them (you > >> >need to register on [1] and [2] yourself). > >> > > >> >[1] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1 > >> >[2] https://cwiki.apache.org/confluence/display/IoTDB > >> > > >> >Best, > >> >--- > >> >Xiangdong Huang > >> > > >> >qiuyuluo777 <471289...@qq.com.invalid> 于2023年6月26日周一 17:32写道： > >> >> > >> >> Dear IoTDB Team, > >> >> > >> >> > >> >> I am a CS graduate student at Southern Methodist University and used to > >> >> be a software development intern at the Tsinghua Sichuan Energy > >> >> Internet Research Institute. I have extensive experience in Java and > >> >> various relational and non-relational databases. > >> >> > >> >> > >> >> I am interested in joining your team as a developer for IoTDB, as I > >> >> find the project impressive and believe my skills would be an addition. > >> >> > >> >> > >> >> Thank you for your consideration. > >> >> > >> >> > >> >> Sincerely, > >> >> Qiuyu Luo > >> >> > >> >> > >> >> Jira ID: qiuyuluo777 > >> >> Confluence ID: qiuyuluo
Re: Re: Interest in Joining IoTDB Team as a Developer
Hi Qiuyu, Confluence added. Regarding to your JIRA id: you need to provide your nick name, rather than your username (account id). Best, --- Xiangdong Huang Luo Qiuyu 于2023年6月26日周一 18:28写道： > > Dear Huang, > > > > > Thanks for your prompt concern. > > > > > I checked my id and these are correct: > > > > > > Jira ID: qiuyuluo > > > Confluence ID: qiuyuluo (full name: LUO QIUYU) > > > > > Sorry for any trouble it may cause. > > > > > Sincerely, > Qiuyu Luo > > > > > > > > > > > > At 2023-06-26 18:13:50, "Xiangdong Huang" wrote: > >Hi Qiuyu, > > > >Welcome to IoTDB community. > > > >Can you check your jira id and confluence id? I can not find them (you > >need to register on [1] and [2] yourself). > > > >[1] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1 > >[2] https://cwiki.apache.org/confluence/display/IoTDB > > > >Best, > >--- > >Xiangdong Huang > > > >qiuyuluo777 <471289...@qq.com.invalid> 于2023年6月26日周一 17:32写道： > >> > >> Dear IoTDB Team, > >> > >> > >> I am a CS graduate student at Southern Methodist University and used to be > >> a software development intern at the Tsinghua Sichuan Energy Internet > >> Research Institute. I have extensive experience in Java and various > >> relational and non-relational databases. > >> > >> > >> I am interested in joining your team as a developer for IoTDB, as I find > >> the project impressive and believe my skills would be an addition. > >> > >> > >> Thank you for your consideration. > >> > >> > >> Sincerely, > >> Qiuyu Luo > >> > >> > >> Jira ID: qiuyuluo777 > >> Confluence ID: qiuyuluo
Re: Interest in Joining IoTDB Team as a Developer
Hi Qiuyu, Welcome to IoTDB community. Can you check your jira id and confluence id? I can not find them (you need to register on [1] and [2] yourself). [1] https://issues.apache.org/jira/projects/IOTDB/issues/IOTDB-1 [2] https://cwiki.apache.org/confluence/display/IoTDB Best, --- Xiangdong Huang qiuyuluo777 <471289...@qq.com.invalid> 于2023年6月26日周一 17:32写道： > > Dear IoTDB Team, > > > I am a CS graduate student at Southern Methodist University and used to be a > software development intern at the Tsinghua Sichuan Energy Internet Research > Institute. I have extensive experience in Java and various relational and > non-relational databases. > > > I am interested in joining your team as a developer for IoTDB, as I find the > project impressive and believe my skills would be an addition. > > > Thank you for your consideration. > > > Sincerely, > Qiuyu Luo > > > Jira ID: qiuyuluo777 > Confluence ID: qiuyuluo
Quarterly report (Q2)
Hi all, The following is Q2's report draft. Sorry for delay. Please review before 16th June. ## Description: The mission of Apache IoTDB is the creation and maintenance of software related to an IoT native database with high performance for data management and analysis ## Project Status: Current project status: Ongoing with high activity. Issues for the board: none. ## Membership Data: Apache IoTDB was founded 2020-09-16 (3 years ago) There are currently 58 committers and 27 PMC members in this project. The Committer-to-PMC ratio is roughly 2:1. Community changes, past quarter: - No new PMC members. Last addition was Chao Wang on 2022-12-12. - Peichen Chou was added as committer on 2023-05-05 - Yuhua Ren was added as committer on 2023-05-31 ## Project Activity: IoTDB has evolved into v1.1.x, while v0.13.x is still maintained. The community is working on v1.2.0, which has some new features e.g., more complex time-series-oriented queries, more stable cluster. Recent releases: IOTDB-1.1.1 was released on 2023-06-14. IOTDB-1.1.0 was released on 2023-04-03. IOTDB-0.13.4 was released on 2023-02-23. ## Community Health: The community looks healthy and we are trying to develop its ecosystem. - Two new committers, who come from different organizations/companies are elected. - We host several meetups, e.g., IoTDB x EMQ on 15th April and 31th May. - Some committers introduced IoTDB on some international conferences, e.g., buildingIoT in Germany, Open Sourse Summit in Chongqing China. Best, --- Xiangdong Huang School of Software, Tsinghua University
Re: [discuss] consider revert the feature of multi-tenancy
Hi Houliang, It makes no sense to refer Doris. Doris is not a lightweight db, and edge side is never its goal. > The topic of this discussion is whether to revert the feature of > multi-tenancy. I wonder why you fall into these words I think I have mentioned at least twice (or maybe 3 times) that Jialin's suggestion is fine for me. Best, --- Xiangdong Huang School of Software, Tsinghua University Houliang Qi 于2023年4月11日周二 15:05写道： > > Hi Jinrui, > > > (Jinrui) From my perspective, Multi-tenancy is different from > > resource-control and they are not the different term for same thing. > > According to our implementation, current feature focus on the resource > > control on users of one tenant rather than on different tenants. If we did > > not reflect the wording `multi-tenancy` in the code, why do we use it on > > user docs and PR's description ? > > Sorry, I am not agree with you, from my perspective, a user is a tenant, and > each tenant has different resources. This is also multi-tenancy. Even each > tenant can only have one db. In our current implementation, a user is a > tenant. > For doris, they also mention multi-tenancy, but it is limited user > resources.[1], the same as our current implementation. > For Spanner, a tenant can also have only one db. [2] > The reason why I think that both multi-tenancy and resource-control are > suitable for us is that what we are currently doing is to limit the functions > of users or db resources. > On this point, I agree with Wang Chao's point of view. > > > As for whether the multi-tenant function you mentioned affects the > > positioning of IoTDB, I don't think it is accurate. I personally think > > that the multi-tenant function is a term for resource isolation technology > > and will not affect the positioning of IoTDB. I don't know how you define > > the multi-tenant function. If it refers to the connection with the billing > > system of the cloud service provider, it may be another form. This > > discussion will not continue to discuss multi-tenancy. > > > > > (Jinrui) REVERT does not mean REJECT. It is only a quick way to keep the > > code more reliable before we reach the same page. And furthermore, I don't > > think it is harmful or discouraging and it is only a regular way we use to > > replace hot-fix. > > (Jinrui) The reviewers may be confused by the PR's description and then > > focus on whether `multi-tenant` should be integrated in current development > > stage of IoTDB. > > The topic of this discussion is whether to revert the feature of > multi-tenancy. I STRONGLY think that this PR does not violate the positioning > and future development of IOTDB, so I STRONGLY think that revert is not > needed, as this function is not enabled by default, and we are continuing > Iterate and refine this feature. Before the actual release, it is necessary > to consider some scenarios and do some testing. > > > > [1] https://doris.apache.org/docs/dev/admin-manual/multi-tenant/ > [2] > https://cloud.google.com/solutions/implementing-multi-tenancy-cloud-spanner > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | Chao Wang | > | Date | 04/11/2023 13:42 | > | To | dev@iotdb.apache.org | > | Subject | Re: Re：Re: [discuss] consider revert the feature of multi-tenancy > | > Everyone's contribution counts. But what we are talking about is whether > `multi-tenancy` is suitable for current IoTDB's development. > From my perspective, Multi-tenancy is different from resource-control and > they are not the different term for same thing. According to our > implementation, current feature focus on the resource control on users of one > tenant rather than on different tenants. If we did not reflect the wording > `multi-tenancy` in the code, why do we use it on user docs and PR's > description ? > > > As I said before, the description is indeed not very clear, and the > description can be modified as a resource control. So what's the point of > wondering if this pr is a multi-tenant function? Even if it is a multi-tenant > function, how will it affect the development of IoTDB? > > > REVERT does not mean REJECT. It is only a quick way to keep the code more > reliable before we reach the same page. And furthermore, I don't think it is > harmful or discouraging and it is only a regular way we use to replace > hot-fix. > > > Yes, revert is a normal process, and PR also has some problems. Let's discuss > the reason for reverting this PR. As Xiangdong said, this is a feature that > will
Re: [discuss] consider revert the feature of multi-tenancy
The topic of this discussion only focus on "multi-tenancy", rather than others. Technology term sometimes blind people's eyes and developers may lack this sensitivity. You can try to make a small survey: if a product provides multi-tenancy feature, do you think the product is a cloud native product or ? and do you think the product is lightweight? Though I do not endorse some features (e.g, timeseries limitation by other contributor you mentationed), but "I do not like" does not mean "it can not be merged". In the community, no body has the power. I only care and intervene features that may bring far-reaching influence (according to my knowledge and judegement). To put it more bluntly: resource-control is just a feature of IoTDB, but multi-tenancy is a key feature for IoTDB's product position. So, I do not intervene the former but I will keep eyes on the latter and do not give in to the term. "reverting PR" is my suggestion but not the only and final option. Jialin's suggestion is also acceptable from my side. Best --- Xiangdong Huang School of Software, Tsinghua University Chao Wang 于2023年4月11日周二 14:34写道： > > Hi, Xiangdong, > > > Thank you for your reminder, I think it's okay to correct the description, > but it's a bit rash to go directly to revert PR without seeing what PR did in > detail. I don't think any community participant wants that. > > > As for whether the multi-tenant function you mentioned affects the > positioning of IoTDB, I don't think it is accurate. I personally think that > the multi-tenant function is a term for resource isolation technology and > will not affect the positioning of IoTDB. I don't know how you define the > multi-tenant function. If it refers to the connection with the billing system > of the cloud service provider, it may be another form. This discussion will > not continue to discuss multi-tenancy. > > > Thanks! > > > Chao Wang > BONC ltd > On 4/11/2023 14:04，Xiangdong Huang wrote： > Hi Chao, > > It is true that PMC should pay attention to the direction of the project, so > what direction does this function affect? Does it affect the edge side? What > are the effects of features that can be turned off? > > I have claimed my standpoint. > I reclaim it here once again and do not want to mention it further: > when people heard of "multi-tenancy", the first impression is "this > product is for the cloud", which is conflict with IoTDB's description. > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > Chao Wang 于2023年4月11日周二 13:19写道： > > I missed this PR. I also do not endorse this PR as I think setting > the limitation strategy is not what an open source project should > consider (It is desired only if the system will be unstable if we have > no such a limitation) > > > Why can't some restriction strategies be added to the open source system to > prevent a single user from affecting the operation of the overall system. At > present, mature open source systems have similar mechanisms, such as doris, > hbase, etc.? In addition, what does this mechanism have to do with whether > the system is open source or not? It itself is a function that a more mature > multi-user system should have. Isn't IoTDB a multi-user system? > > > > What we can do is avoid the case. But if something has conflict with the > project's position, we must do some action. > > > Does this function affect the positioning of IoTDB? IoTDB is only for the > edge side? Can't be deployed and used on the cloud side? > > > Different users have different requirements. But, the PMC need to keep awake > to know or make a CONSENSUS about where the project will go. > > > It is true that PMC should pay attention to the direction of the project, so > what direction does this function affect? Does it affect the edge side? What > are the effects of features that can be turned off? > > > > > Thanks! > > > Chao Wang > BONC ltd > On 4/11/2023 12:16，Xiangdong Huang wrote： > How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? > > I missed this PR. I also do not endorse this PR as I think setting > the limitation strategy is not what an open source project should > consider (It is desired only if the system will be unstable if we have > no such a limitation) > > Why not discuss before the PR submission, but wait until the PR submission > before discussing, wouldn't it waste the energy of community participants? I > have also seen emails s
Re: [discuss] consider revert the feature of multi-tenancy
Hi Chao, > It is true that PMC should pay attention to the direction of the project, so > what direction does this function affect? Does it affect the edge side? What > are the effects of features that can be turned off? I have claimed my standpoint. I reclaim it here once again and do not want to mention it further: when people heard of "multi-tenancy", the first impression is "this product is for the cloud", which is conflict with IoTDB's description. Best, ------- Xiangdong Huang School of Software, Tsinghua University Chao Wang 于2023年4月11日周二 13:19写道： > > > I missed this PR. I also do not endorse this PR as I think setting > > the limitation strategy is not what an open source project should > > consider (It is desired only if the system will be unstable if we have > > no such a limitation) > > > Why can't some restriction strategies be added to the open source system to > prevent a single user from affecting the operation of the overall system. At > present, mature open source systems have similar mechanisms, such as doris, > hbase, etc.? In addition, what does this mechanism have to do with whether > the system is open source or not? It itself is a function that a more mature > multi-user system should have. Isn't IoTDB a multi-user system? > > > > > What we can do is avoid the case. But if something has conflict with the > > project's position, we must do some action. > > > Does this function affect the positioning of IoTDB? IoTDB is only for the > edge side? Can't be deployed and used on the cloud side? > > > > Different users have different requirements. But, the PMC need to keep > > awake to know or make a CONSENSUS about where the project will go. > > > It is true that PMC should pay attention to the direction of the project, so > what direction does this function affect? Does it affect the edge side? What > are the effects of features that can be turned off? > > > > > Thanks! > > > Chao Wang > BONC ltd > On 4/11/2023 12:16，Xiangdong Huang wrote： > How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? > > I missed this PR. I also do not endorse this PR as I think setting > the limitation strategy is not what an open source project should > consider (It is desired only if the system will be unstable if we have > no such a limitation) > > Why not discuss before the PR submission, but wait until the PR submission > before discussing, wouldn't it waste the energy of community participants? I > have also seen emails sent before, not without notifying everyone. > > Discussing and notifying on the community is absolutely right. But it > does not mean we have to accept and do not change all the fact that > has happened. > What we can do is avoid the case. But if something has conflict with > the project's position, we must do some action. > > Another point is that the multi-tenancy function may be a function required > by other companies' IOTDB releases, but will other people's contributions to > the community affect the development of the community? I think it will be > more conducive to the development of community diversity. > > Different users have different requirements. But, the PMC need to keep > awake to know or make a CONSENSUS about where the project will go. > That is why I start this discussion though I know it will cause many > complaint. > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > Chao Wang 于2023年4月11日周二 09:16写道： > > Hi, Xiangdong, > > > what is the side effect when we manually create a time series? > > > How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? > > > This discuss is not for getting "+1" or "-1" (though anyone can reply > the vote..). > I just want to discuss that do we REALLY consider and analyze the > feature and the implementation carefully? > > > Why not discuss before the PR submission, but wait until the PR submission > before discussing, wouldn't it waste the energy of community participants? I > have also seen emails sent before, not without notifying everyone. > > > > > In addition, I think Jialin's suggestion is more reasonable. The description > of this function may not be particularly clear. It can be said in another > way, such as resource control. However, reverting will undoubtedly be harmful > to the community, will discoura
Re: [discuss] consider revert the feature of multi-tenancy
Hi Houliang, Notice that I never said the feature should be reverted because of bugs.. The key point is the feature is harmful for Industry users because most of them do not like cloud. (that is why I opt for Jialin's suggestion). > I think that we should discuss some of our discussion points clearly at the > beginning of the design, instead of how to revert the PR after the PR is > merged. I think there is a problem with this process. It is of course right, but it does not mean that we can not revert a PR if it is merged. > Leaving aside this feature, has the PR of the big feature we merged in been > discussed in detail? How to define detailed discussion? Yes for each big feature we need a discussion in detail. As I have no much time to join all the features, being the PMC chair, at least I need to keep the project following its original destination or new destination if we all agree. Considering my personal time, I judge and intervene featuers which may change the product's position. That is why I spent time to discuss whether we redesign the cluster mode, whether we split an IoTDB instance into two (CN and DN), and whether we tell IoTDB is for cloud-native... And that is why I do not care about more detailed features.. Best, --- Xiangdong Huang School of Software, Tsinghua University Houliang Qi 于2023年4月11日周二 09:51写道： > > Hi, all > > > Leaving aside this feature, has the PR of the big feature we merged in been > discussed in detail? How to define detailed discussion? > > I think that we should discuss some of our discussion points clearly at the > beginning of the design, instead of how to revert the PR after the PR is > merged. I think there is a problem with this process. > > Who can guarantee that there are no bugs and no problems in the developed > functions, and we are all improving through continuous iteration. And this > feature also refers to the design of some other excellent projects, such as > doris and hbase. > > As for the name of this feature, in doris, it is called multi-tenancy[1], in > hbase it is called quota[2], we can call it resource-control, I think it is > ok. After all, we did not reflect the wording of multi-tenancy in the code > implementation. > > > > [1] https://doris.apache.org/docs/dev/admin-manual/multi-tenant > [2] https://hbase.apache.org/book.html#quota > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | Chao Wang | > | Date | 04/11/2023 09:15 | > | To | dev@iotdb.apache.org | > | Cc | dev@iotdb.apache.org | > | Subject | Re: [discuss] consider revert the feature of multi-tenancy | > Hi, Xiangdong, > > > what is the side effect when we manually create a time series? > > > How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? > > > This discuss is not for getting "+1" or "-1" (though anyone can reply > the vote..). > I just want to discuss that do we REALLY consider and analyze the > feature and the implementation carefully? > > > Why not discuss before the PR submission, but wait until the PR submission > before discussing, wouldn't it waste the energy of community participants? I > have also seen emails sent before, not without notifying everyone. > > > > > In addition, I think Jialin's suggestion is more reasonable. The description > of this function may not be particularly clear. It can be said in another > way, such as resource control. However, reverting will undoubtedly be harmful > to the community, will discourage the enthusiasm of community participants, > and is very unfriendly to community participants. If in doubt, I think it > would be better to raise it as soon as possible, instead of waiting for > others to finish their hard work before questioning. > > > Another point is that the multi-tenancy function may be a function required > by other companies' IOTDB releases, but will other people's contributions to > the community affect the development of the community? I think it will be > more conducive to the development of community diversity. > > > > > > > Thanks! > > > Chao Wang > BONC ltd > ccgow...@163.com > On 4/10/2023 23:45，Xiangdong Huang wrote： > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > I think I know this and I have shown my concern about the possible > harm of this featuer to IoTDB's edge mode... > > 1)
Re: [discuss] consider revert the feature of multi-tenancy
> How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? I missed this PR. I also do not endorse this PR as I think setting the limitation strategy is not what an open source project should consider (It is desired only if the system will be unstable if we have no such a limitation) > Why not discuss before the PR submission, but wait until the PR submission > before discussing, wouldn't it waste the energy of community participants? I > have also seen emails sent before, not without notifying everyone. Discussing and notifying on the community is absolutely right. But it does not mean we have to accept and do not change all the fact that has happened. What we can do is avoid the case. But if something has conflict with the project's position, we must do some action. > Another point is that the multi-tenancy function may be a function required > by other companies' IOTDB releases, but will other people's contributions to > the community affect the development of the community? I think it will be > more conducive to the development of community diversity. Different users have different requirements. But, the PMC need to keep awake to know or make a CONSENSUS about where the project will go. That is why I start this discussion though I know it will cause many complaint. Best, ------- Xiangdong Huang School of Software, Tsinghua University Chao Wang 于2023年4月11日周二 09:16写道： > > Hi, Xiangdong, > > > > what is the side effect when we manually create a time series? > > > How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? > > > > This discuss is not for getting "+1" or "-1" (though anyone can reply > the vote..). > I just want to discuss that do we REALLY consider and analyze the > feature and the implementation carefully? > > > Why not discuss before the PR submission, but wait until the PR submission > before discussing, wouldn't it waste the energy of community participants? I > have also seen emails sent before, not without notifying everyone. > > > > > In addition, I think Jialin's suggestion is more reasonable. The description > of this function may not be particularly clear. It can be said in another > way, such as resource control. However, reverting will undoubtedly be harmful > to the community, will discourage the enthusiasm of community participants, > and is very unfriendly to community participants. If in doubt, I think it > would be better to raise it as soon as possible, instead of waiting for > others to finish their hard work before questioning. > > > Another point is that the multi-tenancy function may be a function required > by other companies' IOTDB releases, but will other people's contributions to > the community affect the development of the community? I think it will be > more conducive to the development of community diversity. > > > > > > > Thanks! > > > Chao Wang > BONC ltd > ccgow...@163.com > On 4/10/2023 23:45，Xiangdong Huang wrote： > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > I think I know this and I have shown my concern about the possible > harm of this featuer to IoTDB's edge mode... > > 1) how many side-effects the feature will bring; > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. > > The experiment is rather simple... > When we really want to show the added codes having no side-effects, > all the exepriemnt settings should follow a rule that how to fully > expose the possible problems. > > For example, as mult-tenancy limits the available # of devices, > timeseries, and the spaces of disk, it should have side-effect on > create new device/timeseries, and writing new data. > So, > - what is the side effect when we manually create a time series? > - what is the side effect when we use automatical creating a time series? > - what is the side effect when we write new data? (as the data can be > compressed when it is flushed on disk in async mode, how to check the > disk space?). Besides, as it impaces each write operation, we need to > focus on write operstions which's batchsize
Re: [discuss] consider revert the feature of multi-tenancy
> Multi tenant functionality is a basic capability in cloud native scenarios This is the most important reason that I disapprove this feature... "multi-tenancy" is deeply binded with "Cloud".. When people heard of "multi-tenancy", the first impression is "ha, this is a cloud native product"... However, IoTDB's domain is IIoT, and Edge mode is one of its important format. And that is why I claim the Charter. Maybe "multi-tenancy" can be used by some company's IoTDB distribution if the company provides Cloud service (like BNOC), But for Apache IoTDB, we need to consider the side effect of not only the technology but also the user reaction.. > (1) The function name should be resource-control on the storage group > level, rather than multi-tenancy. Jialin's opinion maybe a better choice. Best, --- Xiangdong Huang School of Software, Tsinghua University HW-Chao Wang <576749...@qq.com.invalid> 于2023年4月10日周一 23:43写道： > > hi all, Multi tenant functionality is a basic capability in cloud native > scenarios, and currently many open source databases have this capability, > such as clickhouse, hbase, doris, and so on. Therefore, I suggest integrating > this feature into the community. As for resource consumption, it can be > turned off at the edge through configuration. The side effects (performance > loss) that come with it require some testing to identify the problem. > > > > > ---Original--- > From: "Chao Wang" Date: Mon, Apr 10, 2023 19:14 PM > To: "dev@iotdb.apache.org" Cc: "dev@iotdb.apache.org" Subject: Re: [discuss] consider revert the feature of multi-tenancy > > > Agree with Houliang's opinion. > > > Thanks! > > > Chao Wang > BONC ltd > On 4/10/2023 19:01，Houliang Qi wrote： > -1 > > First of all, thanks Xiangdong for pointing out IoTDB's Charter. > > "RESOLVED, that the Apache IoTDB Project be and hereby is > responsible for the creation and maintenance of software > related to an IoT native database with high performance > for data management and analysis, on the edge and the cloud." > > As the charter post, IoTDB can be deployed in the cloud, this is why we > deploy the multi-tenancy feature. > > The cloud can be a public or private cloud if we can deploy only one IoTDB > cluster, and manage multi databases and users with different resources, which > will simplify the maintenance. > > - 1) how many side-effects the feature will bring; > > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. > > - 2) how to reduce the effect when IoTDB is deployed on the edge. > > We supply one switch about this feature, called `quota_enable`, by default > this value is false, so it has no effect when IoTDB is deployed on the edge. > This also answers Jinrui's doubt. > > - 3) some checks failed on WinOS, are they irrelevant? > > No, I think they are not irrelevant, the false check message is about the > Compaction module, and > I see the former pr[2][3] which have been merged 4 days ago has the same > issue, so I suspect that the compaction module has occasional bugs > > - 4) The feature SHOULD be discussed carefully in the community, rather > that submit PRs and merged after some reviews. > > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > > The IoTDB community is open and different opinions are welcome. After all, we > all have the same original intention of wanting IoTDB's features to be more > diverse. > > [1] https://apache-iotdb.feishu.cn/docx/DbqCd8t3EoxlCFx1yYicd9N4n4s > [2] https://github.com/apache/iotdb/actions/runs/4625220921/jobs/8181102446 > [3] https://github.com/apache/iotdb/actions/runs/4531046594/jobs/7980725316 > [4] https://apache-iotdb.feishu.cn/docx/doxcnKOYKDmJ40FpVnVsPMd3nTg > [5] https://lists.apache.org/thread/y6dqcm2o7qk0nbkllb61bp8cv6d3m1h7 > > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | 张金瑞<329920...@qq.com.INVALID | > | Date | 04/10/2023 15:03 | > | To | dev | > | Subject | Re：[discuss] consider revert the feature of multi-tenancy | > +1, > > > Agree with Xiangdong's opinion. > And on the other hand, checking this PR's side effects may take lot of
Re: [discuss] consider revert the feature of multi-tenancy
> Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. I think I know this and I have shown my concern about the possible harm of this featuer to IoTDB's edge mode... > 1) how many side-effects the feature will bring; > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. The experiment is rather simple... When we really want to show the added codes having no side-effects, all the exepriemnt settings should follow a rule that how to fully expose the possible problems. For example, as mult-tenancy limits the available # of devices, timeseries, and the spaces of disk, it should have side-effect on create new device/timeseries, and writing new data. So, - what is the side effect when we manually create a time series? - what is the side effect when we use automatical creating a time series? - what is the side effect when we write new data? (as the data can be compressed when it is flushed on disk in async mode, how to check the disk space?). Besides, as it impaces each write operation, we need to focus on write operstions which's batchsize=1. This discuss is not for getting "+1" or "-1" (though anyone can reply the vote..). I just want to discuss that do we REALLY consider and analyze the feature and the implementation carefully? If not, then this big feature is not the time to be merged (and I will call a vote then), and then let's rethink it and make it really available together. If yes, we also need to rethink it and improve it for better performance. Best, --- Xiangdong Huang School of Software, Tsinghua University Chao Wang 于2023年4月10日周一 19:14写道： > > Agree with Houliang's opinion. > > > Thanks! > > > Chao Wang > BONC ltd > On 4/10/2023 19:01，Houliang Qi wrote： > -1 > > First of all, thanks Xiangdong for pointing out IoTDB's Charter. > > "RESOLVED, that the Apache IoTDB Project be and hereby is > responsible for the creation and maintenance of software > related to an IoT native database with high performance > for data management and analysis, on the edge and the cloud." > > As the charter post, IoTDB can be deployed in the cloud, this is why we > deploy the multi-tenancy feature. > > The cloud can be a public or private cloud if we can deploy only one IoTDB > cluster, and manage multi databases and users with different resources, which > will simplify the maintenance. > > -> 1) how many side-effects the feature will bring; > > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. > > -> 2) how to reduce the effect when IoTDB is deployed on the edge. > > We supply one switch about this feature, called `quota_enable`, by default > this value is false, so it has no effect when IoTDB is deployed on the edge. > This also answers Jinrui's doubt. > > -> 3) some checks failed on WinOS, are they irrelevant? > > No, I think they are not irrelevant, the false check message is about the > Compaction module, and > I see the former pr[2][3] which have been merged 4 days ago has the same > issue, so I suspect that the compaction module has occasional bugs > > -> 4) The feature SHOULD be discussed carefully in the community, rather that > submit PRs and merged after some reviews. > > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > > The IoTDB community is open and different opinions are welcome. After all, we > all have the same original intention of wanting IoTDB's features to be more > diverse. > > [1] https://apache-iotdb.feishu.cn/docx/DbqCd8t3EoxlCFx1yYicd9N4n4s > [2] https://github.com/apache/iotdb/actions/runs/4625220921/jobs/8181102446 > [3] https://github.com/apache/iotdb/actions/runs/4531046594/jobs/7980725316 > [4] https://apache-iotdb.feishu.cn/docx/doxcnKOYKDmJ40FpVnVsPMd3nTg > [5] https://lists.apache.org/thread/y6dqcm2o7qk0nbkllb61bp8cv6d3m1h7 > > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message
[discuss] consider revert the feature of multi-tenancy
Hi all, I see the multi-tenancy feature is merged, and several committers made a lot of contributions on that. As multi-tenancy is quite a big feature, which may change IoTDB's position. The feature SHOULD be discussed carefully in the community, rather that submit PRs and merged after some reviews. Therefore, I call to revert the PR and discuss ASAP about the feature after that. At least, the proposer need to answer the following questions, 1) how many side-effect the feature will bring; 2) how to reduce the effect when IoTDB is deployed on the edge. 3) some checks failed on WinOS, are they irrelevant? I don't mean of rejecting any big contribution to IoTDB or harming the community's diversity, but accepting this feature is really big decision and it deserves us to take time to deliberate. Attached IoTDB's Charter: "RESOLVED, that the Apache IoTDB Project be and hereby is responsible for the creation and maintenance of software related to an IoT native database with high performance for data management and analysis, on the edge and the cloud." [1] https://github.com/apache/iotdb/pull/9534/checks Best, --- Xiangdong Huang School of Software, Tsinghua University
Re: IoTDB quarterly Report (Q1)
Ah, thanks for correction. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Pengcheng Zheng 于2023年3月13日周一 15:16写道： > > Hi Xiangdong, > > A little correction to the interactions with other ASF communities: > > -An joint online meetup for European (actually for all) developers was > hosted by IoTDB on March 2nd with SeaTunnel rather than DolphinScheduler. > > > Best regards, > Pengcheng > > > > Xiangdong Huang schrieb am Mo. 13. März 2023 um 07:19: > > > Hi all, > > > > The following is the draft of IoTDB report. > > Please have a look and let me know if there is something missing. > > > > I will submit the report tomorrow (Sorry for less than 3 days). > > > > ## Description: > > The mission of Apache IoTDB is the creation and maintenance of software > > related > > to an IoT native database with high performance for data management and > > analysis > > > > ## Issues: > > no > > > > ## Membership Data: > > Apache IoTDB was founded 2020-09-16 (2 years ago) > > There are currently 56 committers and 27 PMC members in this project. > > The Committer-to-PMC ratio is roughly 7:4. > > > > Community changes, past quarter: > > - No new PMC members. Last addition was Chao Wang on 2022-12-12. > > - Qingxin Feng was added as committer on 2022-12-15 > > - Haiming Zhu was added as committer on 2022-12-29 > > - Junqing Wang was added as committer on 2022-12-15 > > - Hongyin Zhang was added as committer on 2022-12-15 > > - Yongzao Dan was added as committer on 2023-01-04 > > > > ## Project Activity: > > IOTDB-0.13.4 was released on 2023-02-23. > > IOTDB-1.0.1 was released on 2023-02-22. > > IOTDB-1.0.0 was released on 2022-12-03. > > > > After releasing v1.0.0, > > The development of IoTDB (v1.x) is in progress. > > The community devotes itself to fixing bugs, introducing new features, > > and keeping the new version more stable. > > DBA tools like one-click deployment tool are also in development. > > > > One thing that the project and community should consider is how to > > solve the incompatibility between v0.x and v1.x. > > > > Some security reports are still opened, which should be taken care by PMC. > > > > ## Community Health: > > The community keeps growing (more stargazers, contributors, etc.). > > > > And, the interaction amng ASF communities grows: > > - one IoTDB committer is nominated to Ratis committer and acts as > > release manager of Ratis. > > - An online meetup for European developers is hosted by IoTDB joint > > DolphinSchedule on March 2nd. > > > > Mailing list traffic decreases, which the PMC should take care. > >
IoTDB quarterly Report (Q1)
Hi all, The following is the draft of IoTDB report. Please have a look and let me know if there is something missing. I will submit the report tomorrow (Sorry for less than 3 days). ## Description: The mission of Apache IoTDB is the creation and maintenance of software related to an IoT native database with high performance for data management and analysis ## Issues: no ## Membership Data: Apache IoTDB was founded 2020-09-16 (2 years ago) There are currently 56 committers and 27 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Chao Wang on 2022-12-12. - Qingxin Feng was added as committer on 2022-12-15 - Haiming Zhu was added as committer on 2022-12-29 - Junqing Wang was added as committer on 2022-12-15 - Hongyin Zhang was added as committer on 2022-12-15 - Yongzao Dan was added as committer on 2023-01-04 ## Project Activity: IOTDB-0.13.4 was released on 2023-02-23. IOTDB-1.0.1 was released on 2023-02-22. IOTDB-1.0.0 was released on 2022-12-03. After releasing v1.0.0, The development of IoTDB (v1.x) is in progress. The community devotes itself to fixing bugs, introducing new features, and keeping the new version more stable. DBA tools like one-click deployment tool are also in development. One thing that the project and community should consider is how to solve the incompatibility between v0.x and v1.x. Some security reports are still opened, which should be taken care by PMC. ## Community Health: The community keeps growing (more stargazers, contributors, etc.). And, the interaction amng ASF communities grows: - one IoTDB committer is nominated to Ratis committer and acts as release manager of Ratis. - An online meetup for European developers is hosted by IoTDB joint DolphinSchedule on March 2nd. Mailing list traffic decreases, which the PMC should take care.
Re: Remove UserGuide before 0.13 from website
I think it is ok to remove old docs out of the source repo. But is there a possible way to archive them? Best， --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 谭新宇 <1025599...@qq.com.invalid> 于2023年2月23日周四 20:34写道： > > Hi, > > +1 for removing old docs > > > Thanks > -- > Xinyu Tan > > > > 2023年2月23日 18:30，Jialin Qiao 写道： > > > > Hi, > > > > There are 8 versions UserGuide on our website[1], the old versions are > > hardly used anymore. > > > > I prefer to remove the history versions before 0.13 (not include). > > > > [1] https://iotdb.apache.org/ > > > > Thanks, > > — > > Jialin Qiao > > Apache IoTDB PMC >
Re: [Result][Vote] Create a repo for iotdb-backup-tool
Hi, I have created a repo: gitbox at: https://gitbox.apache.org/repos/asf/iotdb-backup-tool.git GitHub at: https://github.com/apache/iotdb-backup-tool.git Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2023年2月6日周一 11:13写道： > > Hi, > > The Vote for Creating a repo for iotdb-backup-tool closes. > > The vote results is: > Binding votes: 3 +1, NonBinding votes 8 +1. No +0 or -1 vote. > > The votes come from: > > Xiangdong Huang (binding) > HW-Chao Wang (binding) > Qingxin Feng > Guanfei Guo > Jeck1son2 > Haizeiwang > Ximing Gao > Guanchu Shen > Tao Yu > Meng Li > Jialin Qiao (binding) > > The vote passes. Thanks for your votes! > > https://lists.apache.org/thread/lvsmtz8vk28nxnn9dqzqp4ryxszlvwn0 > > Bests, > — > Jialin Qiao > Apache IoTDB PMC
Re: [VOTE] Create a repo for iotdb-backup-tool
+1. Though creating a new repo may reduce the focus, it will be helpful to keep iotdb main repo concise and maintainable. And accepting codes like iotdb-tool will be more convenient. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2023年1月31日周二 21:49写道： > > Hi, > > Here is a tool for backup IoTDB data: > https://github.com/apache/iotdb/pull/7248 > > As iotdb-backup-tool could be used independently, it could be a separate > repository for users. > > So, I'd like to start a vote for creating a new Repo for iotdb-backup-tool. > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > The vote is open for the next 72 hours and passes if at least three +1 > votes and more +1 votes than -1 votes. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC
Re: [DISCUSS] Change the notification settings ?
Hi Chris, I like the proposal. Just one question, how about email notifications from Github Discussion? and are there any changes for JIRA Issue and Github issue? Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2023年1月23日周一 00:51写道： > > Hi all, > > I would like to make a suggestion here on how we can possibly make it better > for people to use and follow our mailing-lists and hereby our project. > > Admittedly I am interested in IoTDB unchanged. However, over time it has > become more and more problematic to casually follow the project. > > Subscribing to the dev-list, which should be the projects town-square, > actually only helps if we’re doing a vote. Besides this, I can’t really see > any discussions at all. > > PR notifications have become so frequent, that we banned them to the > “reviews@” list, to keep dev@ clean. > However, it’s so clean now, that it doesn’t contain anything except build > success/failure notifications and the other list I think only archive-bots > have subscribed (While dev@ currently has 277 email subscribers, reviews@ > only has 3 (at least one of them being the asf archiver and a second one > being another mailbox archive outside of the asf) > > How the reviews@ list looks like: > https://drive.google.com/file/d/1KEONRKhlnqL2azhhmpXBvRIjoyHapp--/view?usp=share_link > How the dev@ list looks like: > https://drive.google.com/file/d/1QxBMpdzv_LIa12CzpTBQOFE6HowIGBjq/view?usp=share_link > > The problem I saw with GitHub PR notifications, was that they are unreadable > in any normal email client also don’t I know a single email-client that can > offer any sort of threading. > I did suggest some changes to the notification-bot the ASF uses and after > quite a bit of discussion, they now added a feature, that a project can > choose the format of the email subject. > https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-CustomsubjectlinesforGitHubevents > > For PLC4X we changed it to this: > https://github.com/apache/plc4x/blob/develop/.asf.yaml#L62 > > This has the benefit of allowing people to see what a thread is about by > making it start with the title … we all know what repo it’s gonna be about, > so we moved that to the end (For some odd reason the repo is a required field > in the subject) > By having every follow-up email be prefixed with “Re: “ every email client I > tried, displayed it as a thread correctly. > > So all in all we have changed in PLC4X: > > > * All dependabot stuff goes to commit@ (because nobody is actually > interested in these emails) > * All PR and Issue emails go to dev, but with the updated format > > Possibly worth thinking about updating the way we do it in iotdb. > > > Chris
Re: [application] apply for permission to assign issues to insere
Hi, please provide your jira id. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 m18640495736 <18640495...@163.com> 于2022年12月26日周一 15:13写道： > > Hello! I'm applying for JIRA permission. Thank you! > > > | | > 胡湘鹏 > | > | > 18640495...@163.com > |
Re: Quarterly report of IoTDB
I have fixed the two issues. Thanks. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Christofer Dutz 于2022年12月15日周四 16:51写道： > > Well, if we’re correcting this, > > > > An “audience” is already multiple people. So, I would say: “an audience of > 60k people” > > > > Chris > > > > From: 谭新宇 <1025599...@qq.com.INVALID> > Date: Thursday, 15. December 2022 at 09:50 > To: dev@iotdb.apache.org > Subject: Re: Quarterly report of IoTDB > > Hi, I seem to have found a typo. > > > On 2022-12-03 and 04, the community hosted Apache IoTDB ecosystem summit, > > there are more than 60 audiences watch the summit online. > > Maybe it’s 60k audiences, not 60 audiences. > > Best > — > Xinyu Tan > > > > > 2022年12月15日 14:26，Xiangdong Huang 写道： > > > > Hi all, > > > > The following is the quarterly report, please have a review. > > (Due to tomorrow is the DDL, I have to submit tonight... can not wait > > for 3 days :(. ) > > > > ## Description: > > The mission of Apache IoTDB is the creation and maintenance of software > > related > > to an IoT native database with high performance for data management and > > analysis > > > > ## Issues: > > There isgrumbling in the community because of the Github Action > > runs too slow.. > > Maybe self-hosted runner can solve the problem > > (https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status) > > and we have submit a ticket: > > https://issues.apache.org/jira/browse/INFRA-23712 > > > > > > ## Membership Data: > > Apache IoTDB was founded 2020-09-16 (2 years ago) > > There are currently 54 committers and 27 PMC members in this project. > > The Committer-to-PMC ratio is 2:1. > > > > Community changes, past quarter: > > - Chao Wang was added to the PMC on 2022-12-12 > > - Qingxin Feng was added as committer on 2022-12-15 > > - Junqing Wang was added as committer on 2022-12-15 > > - Hongyin Zhang was added as committer on 2022-12-15 > > > > ## Project Activity: > > - IOTDB-1.0.0 was released on 2022-12-03. > > - IoTDB-0.13.3 was released on 2022-10-21. > > > > IoTDB v1.0 is a new milestone. There are 140 contributors working for > > this version and there are 860k lines of codes added (and 455k removed). > > > > On 2022-12-03 and 04, the community hosted Apache IoTDB ecosystem summit, > > there are more than 60 audiences watch the summit online. > > > > ## Community Health: > > Now, the contributors (on Github) reaches to 231. > > Our stars reaches to >= 2900, which grows faster than usual due to the > > iotdb summit's impact. > > > > > > --- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > 黄向东 > > 清华大学 软件学院
Quarterly report of IoTDB
Hi all, The following is the quarterly report, please have a review. (Due to tomorrow is the DDL, I have to submit tonight... can not wait for 3 days :(. ) ## Description: The mission of Apache IoTDB is the creation and maintenance of software related to an IoT native database with high performance for data management and analysis ## Issues: There isgrumbling in the community because of the Github Action runs too slow.. Maybe self-hosted runner can solve the problem (https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status) and we have submit a ticket: https://issues.apache.org/jira/browse/INFRA-23712 ## Membership Data: Apache IoTDB was founded 2020-09-16 (2 years ago) There are currently 54 committers and 27 PMC members in this project. The Committer-to-PMC ratio is 2:1. Community changes, past quarter: - Chao Wang was added to the PMC on 2022-12-12 - Qingxin Feng was added as committer on 2022-12-15 - Junqing Wang was added as committer on 2022-12-15 - Hongyin Zhang was added as committer on 2022-12-15 ## Project Activity: - IOTDB-1.0.0 was released on 2022-12-03. - IoTDB-0.13.3 was released on 2022-10-21. IoTDB v1.0 is a new milestone. There are 140 contributors working for this version and there are 860k lines of codes added (and 455k removed). On 2022-12-03 and 04, the community hosted Apache IoTDB ecosystem summit, there are more than 60 audiences watch the summit online. ## Community Health: Now, the contributors (on Github) reaches to 231. Our stars reaches to >= 2900, which grows faster than usual due to the iotdb summit's impact. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
Want to disable Jenkins except for master and rel/ branches
Hi all, As we are using Github Action and rarely use Jenkins, I'd like to call to disable jenkins tasks except for the master and rel/ branches. - The master branch is actually a development branch, therefore we need to use Jenkins to upload nightly build to apache maven; - the rel/* branch are corresponding with our releases, therefore it also need a nightly build if there is changes. BTW, our Jenkins tasks have failed for a long period of time. We will fix it then. Any suggestions? (will do that 3 days later if there is no objections) Best, --- Xiangdong Huang School of Software, Tsinghua University
Re: Rename MultiLeaderConsensus to IoTConsensus
+1 for renaming. > If we now rename MultiLeaderConsensus to IoTConsensus, what should we call it > in the future when we finds a better tradeoff between consistency and > performance? maybe finally we will have a IoT consensus protocol family... Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 谭新宇 <1025599...@qq.com.invalid> 于2022年11月26日周六 17:21写道： > > Hi， > > For these two reasons, I strongly agree with the name change. > > Users do not need to know the name of MultiLeader which is strongly related > to the implementation. However, as the consensus module is currently an > extensible framework, more consensus algorithms may be integrated in the > future. At present, RatisConsensus is the consensus algorithm with the > strongest consistency and the worst performance, while MultiLeaderConsensus > is the consensus algorithm with the weakest consistency and the strongest > performance. There may be some new tradeoff between consistency and > performance in the future. > > If we now rename MultiLeaderConsensus to IoTConsensus, what should we call it > in the future when we finds a better tradeoff between consistency and > performance? > > Thanks > ——— > Xinyu Tan > > > 2022年11月26日 11:20，Jialin Qiao 写道： > > > > Hi, > > > > We plan to rename MultiLeaderConsensus to IoTConsensus > > > > 1. MultiLeader does not have many degrees of identification, and IoT > > is our characteristic. > > 2. The abbreviation of MultiLeader is ML, which is conflict with > > Machine Learning, which we may introduce in IoTDB in the future. > > > > Thanks, > > — > > Jialin Qiao > > Apache IoTDB PMC >
Re: [VOTE] Usage of Apache IoTDB Logo as Stickers for A Meetup
+1 for printing IoTDB stickers. BTW, is there a schedule? Best. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Pengcheng Zheng 于2022年11月24日周四 01:21写道： > > Hi all, > > Since we have quite a few users and contributors of Apache IoTDB in > Germany now, we would like to gather up on 08 Dez 2022 as the "First > German Apache IoTDB User Group Meeting" and a party shortly before > Christmas :). Details of the meetup could be found under [1]. > > We have designed a roll-up of IoTDB for the meetup. The file could be > found under [2], and the size is 85*200cm with bleed. > > Additionally, Timecho as the sponsor for the meetup wants to print out > some of the Apache IoTDB stickers to hand out (just the IoTDB Logo), > for which the usage need to be voted. I will send a mail to > tradema...@apache.org if it is approved. > > Please vote accordingly: > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > The vote is open for the next 72 hours and passes with at least three +1 > votes and more +1 votes than -1 votes. > > > [1] https://www.meetup.com/de-DE/apache-iotdb-meetup/events/289831235/ > [2] https://timechor.feishu.cn/file/boxcnVwjP4Ugd4EKTESW96BrHxh > > > Best regards, > > Pengcheng Zheng > Timecho
Re: [RESULT][VOTE] Create a repo for TsFile-Viewer
Hi, the repo is created: Your project is available on GitHub at: https://github.com/apache/iotdb-tsfile-viewer.git Your project is available on gitbox at: https://gitbox.apache.org/repos/asf/iotdb-tsfile-viewer.git Let us know if you want to do any setting of asf.yaml[1]. [1] https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2022年11月12日周六 09:22写道： > > Hi, > > This vote [1] has lasted for 72 hours, with 3 +1 PMC votes and No +0 > or -1 votes, this VOTE passes! > > +1 Votes: > > Haonan Hou(binding) > Jinrui Zhang(binding) > Gaofei Cao(binding) > Meng Li > 海賊王ルフィ > Chao Wang > Guanfei Guo > Liwen Fu > Xin Hua > jeck1son2 > Tao Yu > Hongshan Miao > Guanchu Shen > > Thanks for your voting! > > [1] https://lists.apache.org/thread/qhwlms519wbot3bs3lowf2wqr8borz2v > > Bests, > — > Jialin Qiao > Apache IoTDB PMC
Re: Release about 1.0.0
+1. Thanks for everyone's effort. :D --- Xiangdong Huang School of Software, Tsinghua University Jialin Qiao 于2022年10月12日周三 23:47写道： > Hi, > > Thanks for reminding! The code frozen will be at 10.30. We will start > release test at 10.31. > > (1) deployment recommendations will be. > (2) Performance test will also publish, including single node > performance and scalability. > (3) As Trigger, Select into, CQ, Sync is newly implemented in the > cluster version, the SQL will be different with 0.13. > Other APIs(SQL, Session, Rest) will be compatible with 0.13 and remain > unchanged from now on. > > We need to keep compatible under a big version 1.x.x. > > The branch management needs more discuss, whether checkout a new > branch when we release a middle version x.middle.x. > > The release work is list here: > https://apache-iotdb.feishu.cn/docx/doxcnHjc12eIvVToAhYwukzO2oh > Welcome to help prepare these materials. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > Eric Pai 于2022年10月11日周二 14:58写道： > > > > Wow, it will be a big event in IoTDB history. We have waited for the > moment in almost 1 year :) > > > > BTW, > > 1. what's the time plan of the code frozen of v1.0.0? We should let all > the contributors know the close time of new features accepted window. > > 2. And are there any APIs, including SQL, Session SDK, REST etc, needed > to do break changes? > > In my view, we had often made in incompatible changes when v0.x versions > were released. > > After we release v1.0.0, we should obey the semantic version that no > break changes are made before v2.0.0 released, or we need define an API > deprecation procedure. > > We should let user migrate the API calls smoothly after upgrading from > v1.0.0 to other new versions of v1.x. > > > > 在 2022/10/11 14:36，“Jialin Qiao” 写入: > > > > Hi all, > > > > After nearly 4 years since we join the ASF in 2018 and one year > > development of the Cluster version, we released several versions, > from > > 0.8 to 0.14. > > > > Now, we prepare to release a formal cluster version in the end of > next > > month, and I propose that we call this version 1.0.0. > > > > Thank all for contributing to the cluster version, what do you think? > > > > Thanks, > > — > > Jialin Qiao > > Apache IoTDB PMC > > >
Re: Change Of Load Syntax
Hi Sicheng, It is impossible to read this email as there are too many "" --- Xiangdong Huang School of Software, Tsinghua University yusicheng <928623...@qq.com.invalid> 于2022年10月5日周三 10:07写道： > > Hi everybody! I noticed that whether metadata is created automatically > when writing is controlled by ENABLE_AUTO_CREATE_SCHEMA in > iotdb-datanode.properties, but the load statement for loading external tsfile > has its own control parameter AUTOREGISTER, and the two functions overlap, so > I want to remove the AUTOREGISTER parameter from the load statement and let > ENABLE_AUTO_CREATE_SCHEMA control whether metadata is created automatically > when loading, consistent with writing. > In addition, I noticed that some users don't want to delete the source > tsfiles after loading them, so I want to add the parameter ONSUCCESS to the > load statement, which means how to handle the successfully loaded tsfiles. > There are two ways to handle it, DELETE, which deletes the source tsfiles > after successful loading, and NONE, which keeps the source tsfile after > successful loading. > In summary, I would like to design the new load statement as follows: > load ' [sglevel=int][verify=true/false][onSuccess=delete/none] > for reference, the origin load statement is calmed as follows: > load ' [autoregister=true/false][,verify=true/false][,sglevel=int] > > best, > > > > > >Sicheng Yu
Re: How to use quotation marks in STRING_LITERAL
Hi Lanyu, PLS PAY ATTENTION that your email format is incorrect. (Notice that the apache mailing list only supports plain text format). It is impossible to understand what you want to claim. --- Xiangdong Huang School of Software, Tsinghua University 廖兰宇 <181250...@smail.nju.edu.cn> 于2022年9月15日周四 10:21写道： > > Hi everybody, > > In 0.13/0.14, to use quotation marks in STRING_LITERAL, we support preceding > the quote character by an escape character. However, recently we found that > this could cause a problem described in issue: > https://issues.apache.org/jira/browse/IOTDB-4236. We refer to the grammar of > presto/trino/sqlite and we think that it is better that we do not support > preceding the quote character by an escape character in STRING_LITERAL. If > you want to use quotation marks in STRING_LITERAL: > > > > A'inside a string quoted with'may be written > as''. > > A"inside a string quoted with"may be written > as"". > > Any comments would be much appreciated. > > Best, > > > > Lanyu Liao > > School of Software, Tsinghua University > > 廖兰宇 > > 清华大学 软件学院
Re: [VOTE]Apache IoTDB second anniversary T-shirt
Hi, +1 for the design. --- Xiangdong Huang School of Software, Tsinghua University 秦楚晴 于2022年9月21日周三 16:01写道： > > Hi all, > > We designed tiger t-shirt swag as second anniversary gifts. > And it's time to vote on handing out the t-shirts sponsored by Timecho > Limited. > > Voting will be open for 72hr. > A minimum of 3 binding +1 votes and more binding +1 than binding -1 > are required to pass. > > (T-shirt info: https://timechor.feishu.cn/file/boxcnk2w4CBBcwwDRFnbA36Qcbg) > > Best, > Mina
[Draft] quarterly Board report of IoTDB
Hi all, this is the quarterly borad report of IoTDB, pls have a review, I will submit 3 days later. ## Description: The mission of Apache IoTDB is the creation and maintenance of software related to an IoT native database with high performance for data management and analysis ## Issues: no ## Membership Data: Apache IoTDB was founded 2020-09-16 (2 years ago) There are currently 51 committers and 26 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - No new PMC members. Last addition was Yuan Tian on 2022-04-21. - Xiulong Yuan was added as committer on 2022-06-17 ## Project Activity: [Insert your own data here] We released some new versions: - v0.12.6, and the lifecycle of v0.12.x is almost over, - v0.13.1 and v0.13.2, and the community may keep to maintain v0.13.x for a period, - v0.14.0-preview1, which includes a new implementation of the cluster mode. Recently, the ecosystem of IoTDB grows well, e.g., we have integrated with Apache Ratis, Telegraph, EdgeX (from Linux foundation) etc.. ## Community Health: [Insert your own data here] - We are glad to see there are more and more contributors. According to Github, the contributors is more than 210. - A meetup is host in Hangzhou China in August, together with Open source project EMQx. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
Consider let start-server.sh run in bachground by default
Hi, I'd like to change the behavior of sbin/start-server: - run it in background by default (in this case, we cannot use ctrl+c to kill it), and use "-f" to claim it run in foreground (then we can use Ctrl+C to kill it). - if sbin/start-server runs in background, then use 2>&1 to close the pintstream. ------- Xiangdong Huang School of Software, Tsinghua University
Re: apply to join iotdb
Hi, jira account added. Welcome --- Xiangdong Huang School of Software, Tsinghua University 瞌睡涛 <785765...@qq.com.invalid> 于2022年8月26日周五 21:45写道： > > hello, im yutao from yonyou special database technology R D department > my jira id: 785765...@qq.com my confluence id: 785765...@qq.com | > |长春大学于涛 | | 785765...@qq.com |
Re: Apply to join IotDB
Hi, jira account added. Welcome. --- Xiangdong Huang School of Software, Tsinghua University Keepromise 于2022年8月26日周五 21:43写道： > > HI, I’m Zhang Jian , from UCAS > Jira id: keepromise > Confluence id: keepromise
will call pid file back
Hi, I notice we removed the pid file after the server starts, Maybe it is because that pid file is not easy to be maintained. But in many cases, pid file is useful, especially if we want to make iotdb as a service in linux-like OS. So, I'd like to revert the behavior that recording the pid into a file after iotdb starts up. Meanwhile, it is also ok to keep current check_running_process() in start-server.sh, who uses "lsof -i: port" to check whether iotdb is alive. Best, --- Xiangdong Huang School of Software, Tsinghua University
Re: Apply to join IotDB
jira and confluence permission added. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 luozh...@yonyou.com 于2022年8月23日周二 14:26写道： > > >Hi, I'm Luo zhenTao, from CQU! > >Jira id: zhtaoluo > >Confluence id: zhtaoluo > > > luozh...@yonyou.com
Re: [DISCUSSION] The name of a UDF
A UDF to REMOVE data? (with a query sql to delete data?) --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Zhou Yifu 于2022年8月2日周二 20:19写道： > > Hmm… ‘discontinuous’ is much like the antonym of the continuous rather than > ‘distinct’ in my opinion. > > Thanks, > Yifu Zhou > > 获取 Outlook for iOS<https://aka.ms/o0ukef> > > 发件人: 18110526...@163.com <18110526...@163.com> > 发送时间: Tuesday, August 2, 2022 7:36:04 PM > 收件人: dev@iotdb.apache.org > 主题: Re: [DISCUSSION] The name of a UDF > > Hi, > > Thanks to reply. > I have an another idea : ‘ discontinuous’. This name looks like distinct but > have a different meaning. > > Best, > > Weihao Li > > > 2022年8月2日 18:53，Zhou Yifu 写道： > > > > How about duplicateFilter, which is kind of same as distinct. > > > > Thanks, > > Yifu Zhou > > > > 获取 Outlook for iOS<https://aka.ms/o0ukef> > > > > 发件人: Jialin Qiao > > 发送时间: Tuesday, August 2, 2022 6:08:53 PM > > 收件人: dev@iotdb.apache.org > > 主题: Re: [DISCUSSION] The name of a UDF > > > > Hi, > > > > How about mutation > > > > Thanks, > > Jialin Qiao > > > > On 2022/08/02 07:32:19 "18110526...@163.com" wrote: > >> Hi everyone, > >> > >> We are developing a UDF to remove a sequence of consecutive identical > >> values (keeping only the first one), but have not come up with a proper > >> name for it. We can't name it ‘distinct' because it has a different > >> meaning. Here are some of the alternatives we came up with, and if you > >> have a better one, we are happy to discuss! > >> > >> 1.removeContinuous > >> 2.removeConsecutiveIdentity > >> > >> The issue related: > >>> https://github.com/apache/iotdb/issues/6751 > >>> <https://github.com/apache/iotdb/issues/6751> > >> > >> Best, > >> > >> Weihao Li > >> > >> >
Re: The structure of distribution
Hi,
if we use the structure 2, the binary release name
"apache-iotdb-{version}-bin-all.zip" should be changed
(to "apache-iotdb-{version}-bin-cluster.zip". As the structure does
not fit the singleton release version.
The start-server.sh (or start-new-server.sh) script should also be
removed. It will be confused that
why I need to use "datanode" or "confignode" if I want to just use a
start-new-server
Best,
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
SpriCoder 于2022年7月6日周三 17:08写道：
>
> Get it!
>
>
>
>
> --Original--
> From:
>"SpriCoder"
>
> Date:Tue, Jul 5, 2022 05:54 PM
> To:"dev"
> Subject:The structure of distribution
>
>
>
> Hi all,
>
> Currently, we have confignode and datanode folder in the distribution. Which
> has conf, sbin, and will store the default folder of data and system. And
> there is a need to refactor distribution structure.
>
> I think there are two choices:
>
> 1. remove the confignode and datanode folder, and combine their script and
> configuration files into the conf and sbin under the root. In this way all
> folder generated by confignode will be put into config folder, and all folder
> generated by confignode will be put into data folder.
>
> 2. use confignode and datanode folder to manage their script and
> configuration files, like: confignode/sbin, confignode/conf, datanode/sbin,
> datanode/conf, etc. In this way all folder generated by confignode will be
> put into confignode folder and all folder generated by datanode will be put
> into datanode folder.
>
>
>
>
> What's your opinion? Looking forward to your reply.
>
>
> Best,
>
> ---
>
> Hongyin Zhang
Re: Permission to apply
done. welcome. --- Xiangdong Huang School of Software, Tsinghua University 朝阳 于2022年7月6日周三 12:15写道： > > I am liruizhi. > My Jira account is liruizhi. > My Confluence account also is liruizhi. > Please give me the authorization. > Thank you! > > >
Re: [VOTE] Apache IoTDB 0.14.0-preview1 RC1 release
Hi, How our website will look like once the version is released? especially the "document" tag. Will we open a v0.14.x tag? BTW, some documents should be updated, e.g., the start script command on QuickStart page is incorrect. --- Xiangdong Huang School of Software, Tsinghua University JT 于2022年7月2日周六 09:15写道： > > Hi, > > +1 (binding) > > The source release: > apache headers [ok] > signatures and hashes [ok] > LICENSE and NOTICE [ok] > no jar files [ok] > could compile from source: ./mvnw.sh clean install [minor issue with > a target file not ignored by apache-rat] > > > The binary distribution: > version number in CLI [ok] > signatures and hashes [ok] > start in Win11, jdk8 [ok] > statements executed successfully: [ok] > > SET STORAGE GROUP TO root.sg1; > create aligned timeseries root.sg1.d1(s1 FLOAT encoding=RLE, s2 INT32 > encoding=Gorilla compression=SNAPPY, s3 INT64, s4 BOOLEAN, s5 TEXT); > create timeseries root.sg1.d2.s1 WITH DATATYPE=FLOAT, encoding=RLE; > create timeseries root.sg1.d2.s2 WITH DATATYPE=INT32, encoding=Gorilla; > create timeseries root.sg1.d2.s3 WITH DATATYPE=INT64; > create timeseries root.sg1.d2.s4 WITH DATATYPE=BOOLEAN; > create timeseries root.sg1.d2.s5 WITH DATATYPE=TEXT; > insert into root.sg1.d1(time, s1, s2, s3, s4, s5) aligned values(1, > 1.0, 1, 1, TRUE, 'aligned_test1'); > insert into root.sg1.d1(time, s1, s2, s3, s5) aligned values(2, 2.0, > 2, 2, 'aligned_test2'); > insert into root.sg1.d1(time, s1, s3, s4, s5) aligned values(3, 3.0, > 3, FALSE, 'aligned_test3'); > insert into root.sg1.d1(time, s1, s2, s4, s5) aligned values(4, 4.0, > 4, TRUE, 'aligned_test4'); > insert into root.sg1.d1(time, s1, s2, s4, s5) aligned values(5, 5.0, > 5, TRUE, 'aligned_test5'); > insert into root.sg1.d1(time, s1, s2, s3, s4) aligned values(6, 6.0, > 6, 6, TRUE); > insert into root.sg1.d1(time, s1, s2, s3, s4, s5) aligned values(7, > 7.0, 7, 7, FALSE, 'aligned_test7'); > insert into root.sg1.d1(time, s1, s2, s3, s5) aligned values(8, 8.0, > 8, 8, 'aligned_test8'); > insert into root.sg1.d1(time, s1, s2, s3, s4, s5) aligned values(9, > 9.0, 9, 9, FALSE, 'aligned_test9'); > insert into root.sg1.d1(time, s2, s3, s4, s5) aligned values(10, 10, > 10, TRUE, 'aligned_test10'); > insert into root.sg1.d2(time, s1, s2, s3, s4, s5) values(1, 1.0, 1, 1, > TRUE, 'non_aligned_test1'); > insert into root.sg1.d2(time, s1, s2, s3, s5) values(2, 2.0, 2, 2, > 'non_aligned_test2'); > insert into root.sg1.d2(time, s1, s3, s4, s5) values(3, 3.0, 3, FALSE, > 'non_aligned_test3'); > insert into root.sg1.d2(time, s1, s2, s4, s5) values(4, 4.0, 4, TRUE, > 'non_aligned_test4'); > insert into root.sg1.d2(time, s1, s2, s4, s5) values(5, 5.0, 5, TRUE, > 'non_aligned_test5'); > insert into root.sg1.d2(time, s1, s2, s3, s4) values(6, 6.0, 6, 6, TRUE); > insert into root.sg1.d2(time, s1, s2, s3, s4, s5) values(7, 7.0, 7, 7, > FALSE, 'non_aligned_test7'); > insert into root.sg1.d2(time, s1, s2, s3, s5) values(8, 8.0, 8, 8, > 'non_aligned_test8'); > insert into root.sg1.d2(time, s1, s2, s3, s4, s5) values(9, 9.0, 9, 9, > FALSE, 'non_aligned_test9'); > insert into root.sg1.d2(time, s2, s3, s4, s5) values(10, 10, 10, TRUE, > 'non_aligned_test10'); > flush > insert into root.sg1.d1(time, s1, s3, s4, s5) aligned values(3, > 3.0, 3, TRUE, 'aligned_unseq_test3'); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(11, 11.0, 11, 11); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(12, 12.0, 12, 12); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(13, 13.0, 13, 13); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(14, 14.0, 14, 14); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(15, 15.0, 15, 15); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(16, 16.0, 16, 16); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(17, 17.0, 17, 17); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(18, 18.0, 18, 18); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(19, 19.0, 19, 19); > insert into root.sg1.d1(time, s1, s2, s3) aligned values(20, 20.0, 20, 20); > insert into root.sg1.d2(time, s1, s2, s3) values(11, 11.0, 11, 11); > insert into root.sg1.d2(time, s1, s2, s3) values(12, 12.0, 12, 12); > insert into root.sg1.d2(time, s1, s2, s3) values(13, 13.0, 13, 13); > insert into root.sg1.d2(time, s1, s2, s3) values(14, 14.0, 14, 14); > insert into root.sg1.d2(time, s1, s2, s3) values(15, 15.0, 15, 15); > insert into root.sg1.d2(time, s1, s2, s3) values(16, 16.0, 16, 16); > insert into root.sg1.d2(time, s1, s2, s3) values(17, 17.0, 17, 17); > insert into root.sg1.d2(time, s1, s2, s3) values(18, 18.0, 18, 18); > insert into root.sg1.d2(t
rename some classes in source codes before releasing
Hi, When reading the source codes on the master branch, I noticed that some classes and fields names can be renamed for better understanding. Better to rename these names before releasing because I am not sure whether the generated thrift codes are compatible if we modify them later. e.g., common.thrift: - TRegionLocation to TRegionInfo (seems this is not a "location", though I do not know the defination of "location") - TConfigNodeLocation: * internalEndPoint -> rpcEndPoint - TDataNodeLocation: * externalEndPoint -> clientRpcEndPoint * internalEndPoint -> configNodeRpcEndPoint * dataBlockManagerEndPoint -> dataNodeRpcEndPoint Best, --- Xiangdong Huang School of Software, Tsinghua University
Re: Release of new cluster version
+1 for 0.14.0-preview, and propose v0.13.1 and v0.12.6 at the same time. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2022年6月22日周三 17:12写道： > > Hi, > > +1 for '0.14.0-preview' > > I even want to use -beta previously :D > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > > Eric Pai 于2022年6月22日周三 16:55写道： > > > Good news! How about '0.14.0-preview' ? As in general development we > > always use *-alpha as an internal test version with all main features > > present. As the description of the version, it mainly contains the new > > cluster. > > > > 在 2022/6/22 16:44，“Jialin Qiao” 写入: > > > > Hi all, > > > > It has been 4~5 months since we started the new cluster version. > > Thanks for > > the contribution of all contributors! > > > > It's time to start the topic of releasing :) > > > > The summary status of the master branch is as follows: > > > > (1) The new cluster version has many advantages compared to the old > > one(0.12~0.13). > > > > * MPP Query Engine, we use volcano module with operators for better > > extendibility in data processing. > > * Configurable consensus protocol (Standalone, Raft, Multi-Leader) for > > Partition, Schema and Data. > > * Controlable partition table, so that we could control which node to > > manage data. > > * No whole cluster consensus group. > > * More flexible and lightweight scale-out, we could add a node in a few > > seconds without moving data. > > * Extensible load balance strategy. > > * Built-in metric framework to monitor the status of the cluster. > > > > (2) We have implement the basic function (read/write/schema/udf) of > > the new > > Cluster/standalone version. > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fmindnotes%2FbmncnEr7XPzmPvrZgYbcSEvMTIf%23mindmapdata=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=HpaVs90uuGUsA4MhtHGyIltGwexNWcITWmcINs00rN8%3Dreserved=0 > > > > There are still some functions that need to be implemented in new > > Cluster/standalone. > > * Trigger > > * Schema Template > > * Continuous Query > > * Select into > > * Sync framework > > > > (3) The performance optimization of the new Cluster is still going. > > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapache-iotdb.feishu.cn%2Fdocx%2Fdoxcn6xEgTopkvsLsG3TXjwlx0cdata=05%7C01%7C%7C14142baedc434ef11f1d08da542b722d%7C84df9e7fe9f640afb435%7C1%7C0%7C637914842832750235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=YDO5juXfrC0LZDmYJovmgAHw6LxZJANbLFcyobt4FOM%3Dreserved=0 > > > > * Write performance of one replication with standalone/multiLeader in > > new > > cluster. [ok]. > > * Write performance of one replication with Ratis. [underway] > > * Write performance of three replication with Ratis/MultiLeader. > > [underway] > > * Query performance of one replication with > > standalone/ratis/multileader. > > [underway] > > * Query performance of three replication with Ratis/MultiLeader. > > [underway] > > > > I suppose we could get a not-bad number by this weekend for all above > > scenaios. > > > > (4) More test needs to be done to improve the stability of the new > > cluster/standalone, and the ITs are being added. > > > > As this version does not contains the whole functions of 0.13, and the > > system performance and stability need to be improved for some time > > before > > produnction, I suggest retaining the 0.14.0 version number. > > > > So, how about using 【0.14.0-alpha】, and we only release the binary > > distribution of new cluster, without new/old standalone version. > > > > Then, for the formal 0.14.0, we could release the new > > Cluster/Standalone > > version and discard the old Standalone version. > > > > Thanks, > > — > > Jialin Qiao > > Apache IoTDB PMC > > > >
Re: Support new path pattern: 0 or more layers
Hi,
- if only for 0 or more layers, I opt for ***
- if we want to define a clear range of layers, I opt for *{start, end} or
**{start, end}
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
guoguan...@qq.com.INVALID 于2022年6月21日周二 10:14写道：
> Hi guys!
> Now IoTDB supports the following path patterns:
> * (1 asterisk) - one layer
> ** (2 asterisks) - one or more layers
>
>We need to support a new path pattern: 0 or one or more layers, there
> are some optional schemes:
>Scheme 1, support it directly with a specific syntax:
> *** (3 asterisks) - 0 or one or more layers
>
>Scheme 2, use more common syntax:
> a special symbol (such as $, or someone else) - 0 or 1 layer
> $* - 0 or 1 or more layers
>
> JIRA: https://issues.apache.org/jira/browse/IOTDB-3450
>
> Any ideas?
>
>
>
> guoguan...@qq.com
>
Re: Data filtering and aggregation with tags
+1, this feature is useful.
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
Eric Pai 于2022年6月16日周四 17:45写道：
> Good idea! If we can make use of the tags not only in metadata but also in
> data query, we can enrich the data analysis ability a lot, and help the
> business layer to achieve more goals than before. However as the query
> grammar may become more complicated, we should take the easy-use into
> consideration of SQL design as well.
>
> 在 2022/6/16 17:39，“Wz” 写入:
>
> Hi guys,
>
>
>
>
> I believe using tags to do data filtering and aggregation can be a
> common need. Putting all the attributes into the path is not a good idea
> because it makes the path extremely long, and slows down the MTree
> searching, so we take some of the attributes as tags. But that doesn't mean
> tags are not important.
>
>
>
>
>
> Let's take the following ECS management scenario as an example. IoTDB
> stores the cpu_util of each ECS instance. Besides that, an ECS instance has
> static attributes like region_id, available_zone, hostname, CPU, memory,
> storage, and OS store. Since the CPU, memory, and storage are numbers and
> OS is a string with white spaces, they are stored as tags and other
> attributes are stored as levels in the path like
> root.${region_id}.${available_zone}.${hostname}.cpu_util.
>
>
>
>
> Let's say there are some ECS instances whose cpu_util is abnormally
> high in the last hour and we want to know if the problem is caused by a
> certain version of OS. The query should be like,
>
>
>
>
> SELECT OS, COUNT(cpu_util) FROM root.** WHERE cpu_util 95.0
> GRUOP BY TAG OS ALIGN BY DEVICE
>
>
>
>
> With the ability to do filter and aggregation with tags, IoTDB can be
> more powerful in analytics processing. What do you think?
>
>
>
>
> Any suggestions are welcome :D
>
>
>
>
> Zhong Wang,
>
> Alibaba group
>
>
Re: Var-Length-Numeric encoding?
Hi, I think the encoding implementation is in src/main/java/org/apache/iotdb/tsfile/utils/ReadWriteForEncodingUtils.java @Yuan Tian implemented it. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Julian Feinauer 于2022年6月13日周一 17:47写道： > Hi, > > > > I can only comment on floating points: we dont. > > Currently we also only have var-length encoding vor u32 (not for u64). > > > > Regarding ZigZag Encoding perhaps anybody else can jump in here? > > > > Julian > > > > *Julian Feinauer* > Geschäftsführer/CEO > <https://pragmaticindustries.com/> <https://pragmaticindustries.com/> > j.feina...@pragmaticminds.de <%7BE-mail%7D> > +49 (0) 7021 87868-01 <+49%20(0)%207021%2087868-01> | > Jesinger Str. 57, 73230 Kirchheim unter Teck > www.pragmaticindustries.de <https://pragmaticindustries.com/> > > <https://www.linkedin.com/company/pragmatic-industries-gmbh/> > <https://twitter.com/pragmaticindus1> > <https://www.facebook.com/Pragmatic-industries-GmbH-102791535422112> > <https://www.instagram.com/pragmaticindustries/> > Pflichtinformationen > <https://pragmaticindustries.com/datenschutzerklaerung/> gemäß Artikel > 13 DSGVO > > *Von: *Christofer Dutz > *Datum: *Montag, 13. Juni 2022 um 09:50 > *An: *dev@iotdb.apache.org > *Betreff: *Var-Length-Numeric encoding? > > Hi all, > > Just out of curiosity. Julian told me TSFiles make use of variable length > encoding of numeric types. > I would expect the encoding for unsigned integers to be the "ordinary" one > where 7 bits of a byte are being used for encoding the numeric value and > new bytes are added as long as the first bit is 1. > However, I would be interested in which encoding is being used for > unsigned integers? Julian posted a reply in the #iotdb slack channel, but > I'm unsure which official encoding type this is. > It most likely looks like ZigZag Encoding, but I'm a bit unsure if it > really is. > Could anyone here please shed a bit of lite on this? And do we have > var-length encoding for floating-point types too? > > Chris >
Re: guanchu shen from yonyou request to join iotdb community
Hi Guanchu, Jira and Confluence account added. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 沈冠初 于2022年6月13日周一 10:32写道： > Hi, I'm guanchu shen, from yonyou. > jira id : Shenguanchu > Confluence id : guanchushen
Re: JDBC vs Java API
Qingxin's chart shows the performance difference clearly.
In most case,insertTablet > insertRecords > insertRecord > JDBC.
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
冯 庆新 于2022年6月6日周一 08:57写道：
> Hello Trevor Hart
>
> Can you access this web link:
> http://111.202.73.147:13000/d/5ZvuEYE7z/atm-biao-zhun-da-qi-ya-huan-jing?orgId=1=26
> Is this one ok?
>
> Config items in benchmark:
> DB_SWITCH=IoTDB-013-JDBC / SESSION_BY_TABLET / SESSION_BY_RECORDS /
> SESSION_BY_RECORD
> GROUP_NUMBER=10
> LOOP=1000
> DEVICE_NUMBER=50
> SENSOR_NUMBER=500
> BATCH_SIZE_PER_WRITE=100
> POINT_STEP=1000
> OP_INTERVAL=0
> IS_OUT_OF_ORDER=false
>
> B.R
> qingxin.feng
>
>
> 发件人: Trevor Hart<mailto:tre...@ope.nz>
> 发送时间: 2022年6月6日 8:46
> 收件人: dev<mailto:dev@iotdb.apache.org>
> 主题: JDBC vs Java API
>
> Hello Team
>
>
>
> Does anyone have any published benchmark results of JDBC vs the Java API?
>
>
>
> Firstly Im aware of https://github.com/thulab/iotdb-benchmark but I dont
> see any published results for the various API methods.
>
>
>
> I currently use JDBC for my non-realtime ingestion of data and while Ive
> never encountered any bottle necks I am aware that the documentation says
> that JDBC is not recommended for high velocity data.
>
>
>
> Ive done some very basic ingestions benchmarking tests of inserting 1
> million rows and the Java API is around 2x faster. Is this the typical
> improvement between JDBC and Java API?
>
>
>
> For my simplistic test I am inserting 1 millions rows of timestamp &
> incrementing row id eg insert into root.sg1.d1(timestamp,s1)
> values(${DateTime.Now}, ${n})
>
>
>
> With JDBC I get around 6000 rows per second.
>
>
>
> With the Java native API I get around 12000 rows per second using
> session.executeNonQueryStatement.
>
>
>
> I assume insertTablets and insertRecord(s) would be even faster?
>
>
>
> Thanks
>
> Trevor Hart
>
>
IoTDB quarterly report draft
Hi all, Pls review the report draft, and supply whatever you think we should report. I will submit it on Jun 8th. ## Description: The mission of Apache IoTDB is the creation and maintenance of software related to an IoT native database with high performance for data management and analysis ## Issues: There are no issues requiring board attention. ## Membership Data: Apache IoTDB was founded 2020-09-16 (2 years ago) There are currently 49 committers and 26 PMC members in this project. The Committer-to-PMC ratio is roughly 7:4. Community changes, past quarter: - Yuan Tian was added to the PMC on 2022-04-21 - Minghui Liu was added as committer on 2022-05-25 - Xuxin Liu was added as committer on 2022-05-25 - Qijun Xie was added as committer on 2022-05-28 - Yukun Zhou was added as committer on 2022-05-25 ## Project Activity: Recent releases: - IOTDB-0.13.0 was released on 2022-03-20. - IoTDB-0.12.5 was released on 2022-03-08. there are some bugs are found and fixed after the two versions were release. We will keep to maintain v0.12.x and v0.13.x until v0.14 is released and v0.13.x is relatively stable. The community is developing the new cluster version, which may be finished in the later 1-2 months. ## Community Health: In summary, the community is more and more diverse and active. There are about 90 code contributors in the past quarter, and we have 189 code contributors in total according to Github's statistics. The community submitted several topics to ApacheCon Asia, but missed the Apachecon NA. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
Call reviewers for iotdb-web-workbench
Hi all, I noticed that the iotdb-web-workbench repo lacks reviews. * Welcome all contributors ( no matter whether you are the committer) who are familiar with Vue3.0 and Spring technologies. * The Committers and PMCs should pay more attentions on this repo as there are many works left to guarantee the repo is in compliance with ASF project. Best, --- Xiangdong Huang School of Software, Tsinghua University
Re: replacing log4j
Hi, I suddenly consider one thing.. I remember IoTDB is using logback.. rather than log4j... --- Xiangdong Huang School of Software, Tsinghua University Jialin Qiao 于2022年5月25日周三 21:46写道： > Hi, > > +1 for the replacing. The PR is merged. > > Maybe the security issue is so critical that the author wants to get rid of > it by renaming it... > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > > HW-Chao Wang <576749...@qq.com.invalid> 于2022年5月24日周二 21:54写道： > > > yes, i will raise jira and alter it. > > > > > > > > ---Original--- > > From: "Xiangdong Huang" > Date: Tue, May 24, 2022 21:47 PM > > To: "dev" > Subject: Re: replacing log4j > > > > > > I see, [1] introduces the reason that reload4j is born. > > As it is just a modification in pom file and the project is forked from > > log4j 1.2.17, I think it is fine. > > > > BTW, I feel very very confusing why log4j community ends the life of > log4j > > 1 > > (and in the same time the initial author of log4j 1 forks an independent > > project...) > > > > [1] https://reload4j.qos.ch/ > > --- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > 黄向东 > > 清华大学 软件学院 > > > > > > HW-Chao Wang <576749...@qq.com.invalid 于2022年5月24日周二 17:24写道： > > > > Because of the large amount of changes, the configuration file and > > import > > of each class have to change. > > > > > > > > > > ---Original--- > > From: "Xiangdong Huang" > Date: Tue, May 24, 2022 17:17 PM > > To: "dev" > Subject: Re: replacing log4j > > > > > > Hi, I wonder why not log4j2? any comparison in other communities? > > --- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > nbsp;黄向东 > > 清华大学 软件学院 > > > > > > HW-Chao Wang <576749...@qq.com.invalidgt; 于2022年5月24日周二 > 16:23写道： > > > > gt; hi all ， > > gt; We need to consider replacing log4j1, because log4j1 is EOM > > and has > > some > > gt; CVE vulnerabilities. Reload 4J is used to replace it. Other > > open > > source > > gt; communities have been replaced. Refer to hbase-26691. > > gt; Thanksamp;nbsp; >
Re: replacing log4j
I see, [1] introduces the reason that reload4j is born. As it is just a modification in pom file and the project is forked from log4j 1.2.17, I think it is fine. BTW, I feel very very confusing why log4j community ends the life of log4j 1 (and in the same time the initial author of log4j 1 forks an independent project...) [1] https://reload4j.qos.ch/ --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 HW-Chao Wang <576749...@qq.com.invalid> 于2022年5月24日周二 17:24写道： > Because of the large amount of changes, the configuration file and import > of each class have to change. > > > > > ---Original--- > From: "Xiangdong Huang" Date: Tue, May 24, 2022 17:17 PM > To: "dev" Subject: Re: replacing log4j > > > Hi, I wonder why not log4j2? any comparison in other communities? > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > HW-Chao Wang <576749...@qq.com.invalid 于2022年5月24日周二 16:23写道： > > hi all ， > We need to consider replacing log4j1, because log4j1 is EOM and has > some > CVE vulnerabilities. Reload 4J is used to replace it. Other open > source > communities have been replaced. Refer to hbase-26691. > Thanksnbsp;
Re: replacing log4j
Hi, I wonder why not log4j2? any comparison in other communities? --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 HW-Chao Wang <576749...@qq.com.invalid> 于2022年5月24日周二 16:23写道： > hi all ， > We need to consider replacing log4j1, because log4j1 is EOM and has some > CVE vulnerabilities. Reload 4J is used to replace it. Other open source > communities have been replaced. Refer to hbase-26691. > Thanks
Re: Flush function in cluster
OK... SQL should look like a complete sentence.. So, how about "FLUSH (,)* [ON LOCAL, CLUSTER]" If [ON LOCAL] is omitted, then it just flushes locally. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Eric Pai 于2022年5月23日周一 11:53写道： > As we want to define the SQL grammar, it's not a good choice to use Unix > command line style syntax. > > 在 2022/5/23 11:42，“Xiangdong Huang” 写入: > > how about: flush [, ] [--all-nodes] [-node ] > > omitting [] means flush all sgs. > -- all-nodes means flush on each nodes > -node means flush on the given node > omitting [-node ] and [--all-nodes] equals [-node 127.0.0.1] > --all-nodes and -node are mutually exclusive > > Best, > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > Eric Pai 于2022年5月23日周一 11:27写道： > > > +1. It's not necessary to give 2 different syntax but with same > meaning. > > Just define the most suitable one. > > > > 在 2022/5/23 11:22，“Haonan Hou” 写入: > > > > Hi, > > > > +1 for `FLUSH ALL` syntax. > > > > `FLUSH` and `FLUSH sg` are the existing syntax of the current > > standalone version. > > If we execute `FLUSH ALL` on standalone IoTDB, it can be equals > to > > `Flush` command. > > `flush cluster` sounds meaningless for standalone IoTDB. > > > > Best, > > Haonan Hou > > > > > On May 23, 2022, at 11:07 AM, Jialin Qiao < > qiaojia...@apache.org> > > wrote: > > > > > > Hi, > > > > > > Flush is a frequently used command in IoTDB, which flushes > memtable > > into > > > disk and closes all tsfiles. > > > > > > In the new cluster, we need to redefine this function [1]. > > > > > > * flush: flushing current datanode > > > > > > * flush all/cluster: flushing all datanodes > > > > > > * flush sg: flush all DataRegions of a storage group > > > > > > > > > What do you think? > > > > > > [1] > > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FIOTDB-3099data=05%7C01%7C%7C9bf11e7a5a2c4b8270f708da3c6e3868%7C84df9e7fe9f640afb435%7C1%7C0%7C637888741347695139%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=UY0gbvyZNox8WctT7N0yK6hz71NiWtZh%2BtW18TO4uOw%3Dreserved=0 > > > > > > — > > > Jialin Qiao > > > Apache IoTDB PMC > > > > > > > >
Re: Re: Flush function in cluster
> " flush can reduce memory and speed up the restart process" , this assumes that all copies have been flushed synchronously, so we can ensure that the data files are logically consistent at this point. Sorry that maybe I lag behind current cluster design.. Do we need "all copies have been flushed synchronously, so we can ensure that the data files are logically consistent at this point" ? why? because of the raft protocol? ------- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 李思佳 于2022年5月23日周一 11:47写道： > " flush can reduce memory and speed up the restart process" , this assumes > that all copies have been flushed synchronously, so we can ensure that the > data files are logically consistent at this point. > > The operation of datanode flushing should be the process of resource > release before the node is shutdown(but this does not guarantee that all > copies are logically consistent at this point). For example, shutdownHook > requires the default disk flushing and resource release. We need to provide > a flush command scenario, perhaps because our node shutdown operation is > not incomplete? > > BR, > ------- > Sijia Li > > > -邮件原件- > 发件人: Xiangdong Huang > 发送时间: 2022年5月23日 11:37 > 收件人: dev > 主题: Re: Flush function in cluster > > I think distinguishing flushing on one node or on the cluster has its > meaning. > > As you said, flush can reduce memory and speed up the restart process. So, > how about if the DBA just wants to restart one node.. > > However, the default behavior can be discussed: flush on one node by > default or on the whole cluster by default. > > --- > Xiangdong Huang > School of Software, Tsinghua University > > 黄向东 > 清华大学 软件学院 > > > 李思佳 于2022年5月23日周一 11:28写道： > > > Sorry, I don't understand what the purpose and use of flushing current > > datanode is. > > > > IMO, flush all should mean that all storage group could be flushed, in > > another word, flush sg is a subset of flush all. > > > > For users, distributed is a black box, while SG is an exposed structure. > > Therefore, for cli commands, there is no need to be aware of the > > relationship between the datanode and the self-created SG. > > > > In addition, the Flush operation may speed up our restart recovery > > process. For example, when we flush an SG successfully, we can label > > the associated data files to indicate that all copies are consistent > > at that point in time(here are flush and write priorities). During the > > next restart, we can use this flag to quickly skip the verification step. > > > > In summary, here are my questions and thoughts: > > 1. Is it necessary to flush a dataNode? What are the benefits of this? > > 2. Can the Flush operation affect the consensus group or WAL for a > > quick restart? > > > > BR, > > --- > > Sijia Li > > > > > > -邮件原件- > > 发件人: Jialin Qiao > > 发送时间: 2022年5月23日 11:07 > > 收件人: dev@iotdb.apache.org > > 主题: Flush function in cluster > > > > Hi, > > > > Flush is a frequently used command in IoTDB, which flushes memtable > > into disk and closes all tsfiles. > > > > In the new cluster, we need to redefine this function [1]. > > > > * flush: flushing current datanode > > > > * flush all/cluster: flushing all datanodes > > > > * flush sg: flush all DataRegions of a storage group > > > > > > What do you think? > > > > [1] https://issues.apache.org/jira/browse/IOTDB-3099 > > > > — > > Jialin Qiao > > Apache IoTDB PMC > > >
Re: Flush function in cluster
how about: flush [, ] [--all-nodes] [-node ] omitting [] means flush all sgs. -- all-nodes means flush on each nodes -node means flush on the given node omitting [-node ] and [--all-nodes] equals [-node 127.0.0.1] --all-nodes and -node are mutually exclusive Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Eric Pai 于2022年5月23日周一 11:27写道： > +1. It's not necessary to give 2 different syntax but with same meaning. > Just define the most suitable one. > > 在 2022/5/23 11:22，“Haonan Hou” 写入: > > Hi, > > +1 for `FLUSH ALL` syntax. > > `FLUSH` and `FLUSH sg` are the existing syntax of the current > standalone version. > If we execute `FLUSH ALL` on standalone IoTDB, it can be equals to > `Flush` command. > `flush cluster` sounds meaningless for standalone IoTDB. > > Best, > Haonan Hou > > > On May 23, 2022, at 11:07 AM, Jialin Qiao > wrote: > > > > Hi, > > > > Flush is a frequently used command in IoTDB, which flushes memtable > into > > disk and closes all tsfiles. > > > > In the new cluster, we need to redefine this function [1]. > > > > * flush: flushing current datanode > > > > * flush all/cluster: flushing all datanodes > > > > * flush sg: flush all DataRegions of a storage group > > > > > > What do you think? > > > > [1] > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FIOTDB-3099data=05%7C01%7C%7Cad1f61d413164120347708da3c6b86e5%7C84df9e7fe9f640afb435%7C1%7C0%7C637888729789442308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7Csdata=6X9ZDSKSOLA0HCkAr5v9uNiK1KkI71SOrzXLW%2BH4GBs%3Dreserved=0 > > > > — > > Jialin Qiao > > Apache IoTDB PMC > > >
Re: Flush function in cluster
I think distinguishing flushing on one node or on the cluster has its meaning. As you said, flush can reduce memory and speed up the restart process. So, how about if the DBA just wants to restart one node.. However, the default behavior can be discussed: flush on one node by default or on the whole cluster by default. --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 李思佳 于2022年5月23日周一 11:28写道： > Sorry, I don't understand what the purpose and use of flushing current > datanode is. > > IMO, flush all should mean that all storage group could be flushed, in > another word, flush sg is a subset of flush all. > > For users, distributed is a black box, while SG is an exposed structure. > Therefore, for cli commands, there is no need to be aware of the > relationship between the datanode and the self-created SG. > > In addition, the Flush operation may speed up our restart recovery > process. For example, when we flush an SG successfully, we can label the > associated data files to indicate that all copies are consistent at that > point in time(here are flush and write priorities). During the next > restart, we can use this flag to quickly skip the verification step. > > In summary, here are my questions and thoughts: > 1. Is it necessary to flush a dataNode? What are the benefits of this? > 2. Can the Flush operation affect the consensus group or WAL for a quick > restart? > > BR, > --- > Sijia Li > > > -邮件原件- > 发件人: Jialin Qiao > 发送时间: 2022年5月23日 11:07 > 收件人: dev@iotdb.apache.org > 主题: Flush function in cluster > > Hi, > > Flush is a frequently used command in IoTDB, which flushes memtable into > disk and closes all tsfiles. > > In the new cluster, we need to redefine this function [1]. > > * flush: flushing current datanode > > * flush all/cluster: flushing all datanodes > > * flush sg: flush all DataRegions of a storage group > > > What do you think? > > [1] https://issues.apache.org/jira/browse/IOTDB-3099 > > — > Jialin Qiao > Apache IoTDB PMC >
Re: Flush function in cluster
how about if there is a sg called "all" or "cluster" ? --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2022年5月23日周一 11:07写道： > Hi, > > Flush is a frequently used command in IoTDB, which flushes memtable into > disk and closes all tsfiles. > > In the new cluster, we need to redefine this function [1]. > > * flush: flushing current datanode > > * flush all/cluster: flushing all datanodes > > * flush sg: flush all DataRegions of a storage group > > > What do you think? > > [1] https://issues.apache.org/jira/browse/IOTDB-3099 > > — > Jialin Qiao > Apache IoTDB PMC >
Re: Zenoh supporting IoTDB as Backend
Hi Lukas, we can not simply modify https://github.com/eclipse-zenoh/zenoh-backend-influxdb using https://iotdb.apache.org/UserGuide/Master/API/InfluxDB-Protocol.html because it is Java only now. We may have to use iotdb-rust-client to implement the program. (we have no an official rust client now, but there are some contributors working on it, see [1] BTW, I quickly read the Zenoh project, it is a pub/sub system. I'd like to know why we are interested in the project (or why an IoT application should use Zenoh..) Because of its high throughput (I read an article said the throughput can reach to 45Gb/s for a 1MB payload)? Or its lightweight resource cost? (only takes 300 Bytes on an Arduino Uno) ? [1] https://www.google.com/search?q=iotdb-rust Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Lukas Ott 于2022年5月23日周一 01:06写道： > Hi IoTDB developers, > > https://zenoh.io/docs/manual/backends-list/ In the backend there is > currently listed InfluxDB and RocksDB. I was wondering what it would take > to support https://iotdb.apache.org/ or if it can just leverage the > existing InfluxDB Backend > https://github.com/eclipse-zenoh/zenoh-backend-influxdb with a few > changes: > https://iotdb.apache.org/UserGuide/Master/API/InfluxDB-Protocol.html so > that IoTdb is also supported. Can someone go into more details? (of course > supporting TsFile output would also be really interesting to see.) > > Thanks for giving your perspective and some hints on that feature idea. > > BR / otluk >
Re: maintain the IoTDB-Skywalking plugin codes
Hi Wei, Thanks for your contribution. I will ask the PMC to create a new repo. (Then you may need to maintain the github action, and jenkins if needed) Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 刘威 于2022年5月16日周一 22:32写道： > Hi, I'm the original author of IoTDB-SkyWalking plugin. > I have voted to apache/iotdb-skywalking-storage in my last mail. > > Now, let me introduce the current situation of the plugin. > > The plugin now support SkyWalking v8.9.0, v8.9.1 and v9.0.0 with passing > all e2e tests. > I wrote a blog to introduce its design in v8.9.0. You could read it at the > link > > https://skywalking.apache.org/blog/2021-11-23-design-of-iotdb-storage-option/ > . > In v9.0.0, it has been refactored and optimized according to the > DataConverter > in the new design of SkyWalking storage, which could refer to PR#8755 in > SkyWalking repo. > > What's more, I wrote another blog about how to apply IoTDB as backend > storage. You could see the link > > https://skywalking.apache.org/blog/2021-12-08-application-guide-of-iotdb-storage-option/ > . > > In this case, considering its low maintenance frequency and its small > scope of influence, > I think it's more appropriate to move it to a separate repository. > > Thanks for the support of two mentors (@Xiangdong Huang and @Sheng Wu) and > all other contributors. > IoTDB-SkyWalking plugin is a meaningful attempt. > > -- > Wei Liu > School of Computer Science, NPU > > 刘威 > 西北工业大学计算机学院
Re: About replacing byteBuffer
If we introduce Netty, data copy when scaling a bytebuf is not what we want. Can we use compositeByteBuf to replace it and meanwhile enjoy the benefit of pooledByteBuf? --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Jialin Qiao 于2022年5月16日周一 12:35写道： > Hi, > > The serialization interface needs to be refactored afterward. > > Before that, using ByteArrayOutputStream is easier. > > Thanks, > — > Jialin Qiao > Apache IoTDB PMC > > > 李思佳 于2022年5月16日周一 11:44写道： > > > Hi all, > > > > When I was developing the snapshot interface for the configNode module, I > > noticed that the parameters received by the serialization interface were > > all defined as ByteBuffer, which seemed to have some problems. For > example, > > the external main process has no way of knowing how big the buffer will > be. > > We can only estimate a large value to allocate memory. > > > > Then I looked at the serialization interfaces of other modules, and it > > seemed that most modules did the same thing. This could be a problem once > > the actual size of the buffer exceeds our estimate. So I did a quick > survey > > of Netty's byteBuf last week, and here's the Chinese version of the > results< > > https://apache-iotdb.feishu.cn/docs/doccnW1EFoyLOScys9GTOuaEUbh>. > > > > At the same time, we found that the consensus module also has some > > ByteBuf requirements. But byteBuf doesn't seem to be enough to give us > > precise control over the size of the memory pool, and we may need to wrap > > it if we decide to use it. > > > > Finally, we decided to use stream type instead of byteBuffer in > > configNode for the time being. I will start this work to see if this is > the > > better way this week. If any idea, please let me know. > > > > By the way, Netty’s ByteBuf provides powerful tool operations that we > > will not discard outright, but rather as an option. > > > > BR, > > --- > > Sijia Li > > > > >
Re: IotDB WorkBench 1.3 monitoring is almost complete
Hi, we need to maintain the iotdb-web-workbench repo ... --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 还有多少时间 于2022年5月13日周五 08:56写道： > Hello everyone: > > > The development of IotDB WorkBench 1.3 monitoring function has been > completed, it has been tested for three times and will be released soon. > Monitoring data is taken from IotDB, but it is still fake data at present. > Real data can be used after entering IotDB. Thanks. > > > IotDB WorkBench 1.3 features ： > > > 1. Survival status > 2. JVM indicators > 3. CPU indicators > 4. memory indicators > 5. write indicators > 6. query indicators > 7.Slow query statistics > > > By default, indicators are displayed in a list. You can directly switch to > the chart panel to display indicators。 > > > > Support all modes, developer mode, operator mode, different modes can be > applied to different users. > > > zhengqiang > CISDI INFO
maintain the IoTDB-Skywalking plugin codes
Hi all, Skywalking community is discussing about only keeping self-implemented storage layer. IMO, I think it is not a bad decision to remove 3rd-part implementataion from skywalking's main repo. Because we can decide which version of skywalking we can maintain according to the community developers' time. However, IoTDB-skywalking integration has its meaning and we should keep to maintain the integration. The discussion is, where to put the integration to? a new code repo like: apache/iotdb-skywalking-storage? or into IoTDB's repo like apache/iotdb/skywalking? [1] https://github.com/apache/skywalking/discussions/9059 Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
Re: 账户权限申请
Hi weihao, Welcome, I have add you to the contributor group on JIRA and give the write permission to Confluence. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 李** <18110526...@163.com> 于2022年5月12日周四 10:24写道： > self-introduction：I'm WeiHao Li, a development engineer of IoTDB team of > Timecho. > > > Jira ID：Weihao Li > > > Confluence ID：Weihao Li
Re: [Discuss] New all-in-one distribution package structure
Hi, Before discussing the new release, we need to know why there is "all-in-one" and define what is "all-in-one" and what is not. We released "all-in-one" at that time because we have a standalone version, the cluster version, and grafana-connector. And, we thought: - users may just run IoTDB on one node as a service, so the standalone version is fine. - users may need to run IoTDB on multiple nodes as a service, so the cluster version is suitable; - grafana-connector is an additional service of IoTDB. Now, we have: 1. the standalone version, which only uses datanode module. 2. the cluster version, which uses datanode and confignode modules. 3. the replaceable components: rocksdb-metadata 4. the pluggable library: library-udf 5. the client libraries: jdbc, java, c++, etc.. 6. the grafana connector (deprecated?) 7. the grafana plugin 8. the iotdb-web-workbench (I think it is not ready for an apache release) So, which modules that "all-in-one" wants to contain? Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 Haonan Hou 于2022年5月11日周三 14:46写道： > Hi, > > I submit a PR[1] about adding config node to distribution package. There > is a change of the structure of all-in-one > package structure. Please refer the PR link below for details. > > Please let me know if you have any suggestion on this. Thanks! > > [1] https://github.com/apache/iotdb/pull/5861 > > BR, > Haonan Hou > > >
discuss about Procedure module
Hi, I see there is a procedure module on master branch, and there is some design document [1] about it. But I still have some questions about the module, and want to have a discussion: 1. what is it for? (can someone use several sentences or several paragraphs to introduce it? and put the introduction into a README.md file?) . (and I can find 11 kinds of implementations ... why..) 2. why the procedure should be considered as a new "module" rather than just a class "packet"? [1] https://github.com/apache/iotdb/pull/5477 Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院
summer-ospp is coming and welcome to participate
Hi all, The summer-ospp (like GSoC) is coming, and welcome students enjoying it. Please find the guide here [1]. There are 4 projects related to Apache IoTDB (we have 4 slots but currently only 3 projects are approved, the last one is under review). You can find all of them here [2] (in Chinese). [1] https://summer-ospp.ac.cn/#/apply [2] https://summer-ospp.ac.cn/#/org/orgdetail/ecc8407d-fdc1-4425-826b-d36f57cc68fd/ Best, --- Xiangdong Huang School of Software, Tsinghua University
Re: Refactor the rule of auth check
Hi, Just want to make a confirmation: so we cannot grant privileges to 'root.sg.**'. It is illegal, right? --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 周钰坤 于2022年5月6日周五 19:14写道： > Hi > > Currently, the rule of iotdb's auth check is prefix match, which is > inconsistent with pattern match in DDL and DML. Therefore, we want to > refactor the rule to pattern match. > For example, an old sql, 'GRANT USER ln_write_user PRIVILEGES > INSERT_TIMESERIES on root.ln', won't work any more. The replacement is > 'GRANT USER ln_write_user PRIVILEGES INSERT_TIMESERIES on root.ln.**' > . > > Besides, we introduce the concept, sub pattern, which means a > pattern's result set contains all the elements of its sub pattern's > result set. For example, 'root.sg.d.*' is a sub pattern of > 'root.sg.*.*', while 'root.sg.**' is not a sub pattern of > 'root.sg.*.*'. > When a user is granted privilege on a pattern, the pattern used in his > DDL or DML must be a sub pattern of the previlige pattern, which > guarantees that the user won't access the timeseries exceed his > privilege scope. > > To guarantee the efficiency and performance of auth check, we will > implement the auth check after the generation of statement and before > the execution of statement. > > Hope for some suggestions. > > > Best > > Yukun Zhou > School of Software, Tsinghua University > > 周钰坤 > 清华大学 软件学院 >
Re: self-introduction by Guanfei Guo
Hi Guanfei, I have added your confluence id and jira id to the user group. Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 勇敢的心 于2022年4月25日周一 10:33写道： > Hi, I'm Guanfei Guo, from Yonyou.Myjira id: guoguanfei > MyConfluence id: guoguanfei > > > > Thanks.
Re: I submitted an physical plan serialization issue【 IOTDB-2936】
I add an easy-fix label on the issue.
Welcome to contribute it.
Best,
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
苏志 <273670...@qq.com.invalid> 于2022年4月24日周日 15:12写道：
> The serialization type uses the enumeration ordinal, which can easily
> cause deserialization failures if the order of the enumeration objects
> changes.
>
> Therefore, the constant value int should be used for each type
>
> eg：
> @Override
> public void serializeImpl(ByteBuffer buffer)
> { int type = PhysicalPlanType.INSERT.ordinal(); buffer.put((byte) type);
> subSerialize(buffer); }
> @Override
> public void serializeImpl(ByteBuffer buffer)
> { int type = PhysicalPlanType.BATCHINSERT.ordinal(); buffer.put((byte)
> type); subSerialize(buffer, 0, rowCount); }
> Improvement:
>
> @Override
> public void serializeImpl(ByteBuffer buffer)
> { int type = PhysicalPlanType.INSERT.type; buffer.put((byte) type);
> subSerialize(buffer); }
> @Override
> public void serializeImpl(ByteBuffer buffer)
> { int type = PhysicalPlanType.BATCHINSERT.type; buffer.put((byte) type);
> subSerialize(buffer, 0, rowCount); }
>
>
>
> Best,
>
> gongning
>
>
>
>
>
>
Re: I submitted an IOTDB subscription issue【 IOTDB-2766】
Hi, Any idea to implement? Can the trigger module support this? (but iotdb does not support too many trigger now, I think) Does Yurong have any idea? Best, --- Xiangdong Huang School of Software, Tsinghua University 黄向东 清华大学 软件学院 苏志 <273670...@qq.com.invalid> 于2022年4月24日周日 14:44写道： > In the process of industrial production, various > environmental data such as temperature, pressure and humidity of industrial > equipment are collected by sensors. Customers are eager to detect and > notify abnormal fluctuations of sensor data in a timely manner. We hope to > calculate indicators based on the CAPABILITY of IoTDB CQ, so that > upper-layer services can subscribe to the indicator results calculated by > CQ in a timely manner, and then use the rule engine to achieve self-defined > alarm capability. For example, a steel customer wants to monitor the > abnormal fluctuation of the heating furnace in the last 15 minutes and send > an alarm. > > Now IoTDB already has CQ function, but lacks data > subscription mechanism. > > For example, when data is written to the IOTDB, other > systems can subscribe to the IOTDB's data, either through storage groups, > devices, measurement points, or nodes of any level。 > > Command format: > 1) CREATE: CREATE SUBSCRIPTION "sub0" ON storageGroup DESTINATIONS ALL ' > http://example.com:8080' > 2) Query: SHOW SUBSCRIPTIONS > 3) Delete: DROP SUBSCRIPTION > > Note:http://example.com:8080Service system address. > > Protocol format of IoTDB sending business system: > > -+ > fullpathvaluetimestamp > > > -+ > root.sg.device1.t1 82 1465839830100400200 > > Influxdb Reference address: > > https://docs.influxdata.com/influxdb/v1.8/write_protocols/line_protocol_tutorial/ > > https://docs.influxdata.com/influxdb/v1.8/administration/subscription-management/ > > > > > Best, > > gongning > > > > > >
