Hi Houliang, It makes no sense to refer Doris. Doris is not a lightweight db, and edge side is never its goal.
> The topic of this discussion is whether to revert the feature of > multi-tenancy. I wonder why you fall into these words.... I think I have mentioned at least twice (or maybe 3 times) that Jialin's suggestion is fine for me. Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University Houliang Qi <neuyi...@163.com> 于2023年4月11日周二 15:05写道： > > Hi Jinrui, > > > (Jinrui) From my perspective, Multi-tenancy is different from > > resource-control and they are not the different term for same thing. > > According to our implementation, current feature focus on the resource > > control on users of one tenant rather than on different tenants. If we did > > not reflect the wording `multi-tenancy` in the code, why do we use it on > > user docs and PR's description ? > > Sorry, I am not agree with you, from my perspective, a user is a tenant, and > each tenant has different resources. This is also multi-tenancy. Even each > tenant can only have one db. In our current implementation, a user is a > tenant. > For doris, they also mention multi-tenancy, but it is limited user > resources.[1], the same as our current implementation. > For Spanner, a tenant can also have only one db. [2] > The reason why I think that both multi-tenancy and resource-control are > suitable for us is that what we are currently doing is to limit the functions > of users or db resources. > On this point, I agree with Wang Chao's point of view. > > > As for whether the multi-tenant function you mentioned affects the > > positioning of IoTDB, I don't think it is accurate. I personally think > > that the multi-tenant function is a term for resource isolation technology > > and will not affect the positioning of IoTDB. I don't know how you define > > the multi-tenant function. If it refers to the connection with the billing > > system of the cloud service provider, it may be another form. This > > discussion will not continue to discuss multi-tenancy. > > > > > (Jinrui) REVERT does not mean REJECT. It is only a quick way to keep the > > code more reliable before we reach the same page. And furthermore, I don't > > think it is harmful or discouraging and it is only a regular way we use to > > replace hot-fix. > > (Jinrui) The reviewers may be confused by the PR's description and then > > focus on whether `multi-tenant` should be integrated in current development > > stage of IoTDB. > > The topic of this discussion is whether to revert the feature of > multi-tenancy. I STRONGLY think that this PR does not violate the positioning > and future development of IOTDB, so I STRONGLY think that revert is not > needed, as this function is not enabled by default, and we are continuing > Iterate and refine this feature. Before the actual release, it is necessary > to consider some scenarios and do some testing. > > > > [1] https://doris.apache.org/docs/dev/admin-manual/multi-tenant/ > [2] > https://cloud.google.com/solutions/implementing-multi-tenancy-cloud-spanner > > > Thanks, > --------------------------------------- > Houliang Qi > BONC, Ltd > > > ---- Replied Message ---- > | From | Chao Wang<ccgow...@163.com> | > | Date | 04/11/2023 13:42 | > | To | dev@iotdb.apache.org<dev@iotdb.apache.org> | > | Subject | Re: Re：Re: [discuss] consider revert the feature of multi-tenancy > | > Everyone's contribution counts. But what we are talking about is whether > `multi-tenancy` is suitable for current IoTDB's development. > From my perspective, Multi-tenancy is different from resource-control and > they are not the different term for same thing. According to our > implementation, current feature focus on the resource control on users of one > tenant rather than on different tenants. If we did not reflect the wording > `multi-tenancy` in the code, why do we use it on user docs and PR's > description ? > > > As I said before, the description is indeed not very clear, and the > description can be modified as a resource control. So what's the point of > wondering if this pr is a multi-tenant function? Even if it is a multi-tenant > function, how will it affect the development of IoTDB? > > > REVERT does not mean REJECT. It is only a quick way to keep the code more > reliable before we reach the same page. And furthermore, I don't think it is > harmful or discouraging and it is only a regular way we use to replace > hot-fix. > > > Yes, revert is a normal process, and PR also has some problems. Let's discuss > the reason for reverting this PR. As Xiangdong said, this is a feature that > will affect the positioning of IoTDB, so how does this feature affect the > positioning of IoTDB? > > > Agree. But if we don't make it clear before PR merged, pushing forward the > discussion is better than directly merging, from my side. > > > Agree. I remember sending an email to discuss it. Does that mean that every > PR needs to send an email to discuss clearly? After all, pushing forward the > discussion is better than directly merging. > > > > > Thanks! > > > Chao Wang > BONC ltd > On 4/11/2023 13:10，张金瑞<329920...@qq.com.INVALID> wrote： > (Sorry for the format issue in previous mail) > ====== > Hi, all > > I tried this feature locally according to the User Manual, and I am blocked > at the beginning. > > > Firstly, I didn't found the parameters `quota_enable` and `rate_limiter_type` > in iotdb-common.properties to enable this functionality. > I am not sure whether it is by design but it is not aligned with the user > manual. And I have to add these two parameters into configuration file > manually. > > > Then, I tried the function to limit devices regarding a database and it seems > some basic functionality is unexpected. See the test step below: > 1. create a databse named `root.iov` and insert 5 devices into it. > 2. run the command "set space quota devices=4 on root.iov" to set the device > limit to 4. It can be executed successfully. (UNEXPECTED) > 3. try to insert a new devices. > 4. try to use the command "set space quota devices=8 on root.iov" to increase > the threshold of device count but failed. (UNEXPECTED) > 5. I created another database named `sg2` and tried to set quota limit to it > but failed (UNEXPECTED) > 6. After this test, I tried another test with more simple scenario but still > failed. > > > The detailed test steps can be found in this doc: > https://apache-iotdb.feishu.cn/docx/IerZdPFHroEbRYxKvihcBpncnie > > My confidence in the completeness of testing regarding this feature has > greatly decreased. And at least from my perspective, I cannot guarantee how > much impact this feature will have on the user experience. > > > On the other hand, I also have some thoughts to share with you regarding the > questions raised in the previous email. > > > >> Leaving aside this feature, has the PR of the big feature we merged > in been discussed in detail? How to define detailed discussion? > (Jinrui) I think currently we are focusing on the side effects of this PR, > whether the discussion is detailed depends on whether we have enough > confidence of this feature's definition and implementation. > > > >> I think that we should discuss some of our discussion points > clearly at the beginning of the design, instead of how to revert the PR after > the PR is merged. I think there is a problem with this process. > (Jinrui) Agree. But if we don't make it clear before PR merged, pushing > forward the discussion is better than directly merging, from my side. > > > >> Who can guarantee that there are no bugs and no problems in the > developed functions, and we are all improving through continuous iteration > (Jinrui) Yes. But the developer should do enough tests including different > scenarios to ensure this feature can work smoothly. > > > >> However, reverting will undoubtedly be harmful to the community, > will discourage the enthusiasm of community participants, and is very > unfriendly to community participants > (Jinrui) REVERT does not mean REJECT. It is only a quick way to keep the code > more reliable before we reach the same page. And furthermore, I don't think > it is harmful or discouraging and it is only a regular way we use to replace > hot-fix. > > > >> If in doubt, I think it would be better to raise it as soon as > possible, instead of waiting for others to finish their hard work before > questioning. > (Jinrui) My words has no doubt and offense to anyone and it is only a > discussion about this issue. Contribution is welcomed and talking is also > welcomed. If we notice some potential issue which is worth to be talked > about, any time is ok I think. > > > >> This discuss is not for getting "+1" or "-1" (though anyone can > reply the vote..). I just want to discuss that do we REALLY consider and > analyze the feature and the implementation carefully? > (Jinrui) The reviewers may be confused by the PR's description and then focus > on whether `multi-tenant` should be integrated in current development stage > of IoTDB. > > > >> As for the name of this feature, in doris, it is called > multi-tenancy[1], in hbase it is called quota[2], we can call it > resource-control, I think it is ok. > (Jinrui) From my perspective, Multi-tenancy is different from > resource-control and they are not the different term for same thing. > According to our implementation, current feature focus on the resource > control on users of one tenant rather than on different tenants. If we did > not reflect the wording `multi-tenancy` in the code, why do we use it on user > docs and PR's description ? > > > >> Another point is that the multi-tenancy function may be a function > required by other companies' IOTDB releases, but will other people's > contributions to the community affect the development of the community? > (Jinrui) Everyone's contribution counts. But what we are talking about is > whether `multi-tenancy` is suitable for current IoTDB's development. > > > > > Thanks, > Zhang Jinrui > > > > > > > > > Original > > > > From:"Xiangdong Huang"< saint...@gmail.com >; > > Date:2023/4/11 12:27 > > To:"dev"< dev@iotdb.apache.org >; > > Subject:Re: [discuss] consider revert the feature of multi-tenancy > > > Hi Houliang, > > Notice that I never said the feature should be reverted because of > bugs.. The key point is the feature is harmful for Industry users > because most of them do not like cloud. (that is why I opt for > Jialin's suggestion). > > > I think that we should discuss some of our discussion points clearly at > the beginning of the design, instead of how to revert the PR after the PR is > merged. I think there is a problem with this process. > > It is of course right, but it does not mean that we can not revert a > PR if it is merged. > > > Leaving aside this feature, has the PR of the big feature we merged in > been discussed in detail? How to define detailed discussion? > > Yes for each big feature we need a discussion in detail. As I have no > much time to join all the features, being the PMC chair, at least I > need to keep the project following its original destination or new > destination if we all agree. > > Considering my personal time, I judge and intervene featuers which may > change the product's position. That is why I spent time to discuss > whether we redesign the cluster mode, whether we split an IoTDB > instance into two (CN and DN), and whether we tell IoTDB is for > cloud-native... And that is why I do not care about more detailed > features.. > > Best, > ----------------------------------- > Xiangdong Huang > School of Software, Tsinghua University > > Houliang Qi 于2023年4月11日周二 09:51写道： > > > > Hi, all > > > > > > Leaving aside this feature, has the PR of the big feature we merged in > been discussed in detail? How to define detailed discussion? > > > > I think that we should discuss some of our discussion points clearly at > the beginning of the design, instead of how to revert the PR after the PR is > merged. I think there is a problem with this process. > > > > Who can guarantee that there are no bugs and no problems in the > developed functions, and we are all improving through continuous iteration. > And this feature also refers to the design of some other excellent projects, > such as doris and hbase. > > > > As for the name of this feature, in doris, it is called > multi-tenancy[1], in hbase it is called quota[2], we can call it > resource-control, I think it is ok. After all, we did not reflect the wording > of multi-tenancy in the code implementation. > > > > > > > > [1] https://doris.apache.org/docs/dev/admin-manual/multi-tenant > > [2] https://hbase.apache.org/book.html#quota > > > > > > > > > > Thanks, > > --------------------------------------- > > Houliang Qi > > BONC, Ltd > > > > > > ---- Replied Message ---- > > | From | Chao Wang | > > | Date | 04/11/2023 09:15 | > > | To | dev@iotdb.apache.org | > > | Cc | dev@iotdb.apache.org | > > | Subject | Re: [discuss] consider revert the feature of multi-tenancy | > > Hi, Xiangdong, > > > > > > what is the side effect when we manually create a time series? > > > > > > How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? > > > > > > This discuss is not for getting "+1" or "-1" (though anyone can reply > > the vote..). > > I just want to discuss that do we REALLY consider and analyze the > > feature and the implementation carefully? > > > > > > Why not discuss before the PR submission, but wait until the PR > submission before discussing, wouldn't it waste the energy of community > participants? I have also seen emails sent before, not without notifying > everyone. > > > > > > > > > > In addition, I think Jialin's suggestion is more reasonable. The > description of this function may not be particularly clear. It can be said in > another way, such as resource control. However, reverting will undoubtedly be > harmful to the community, will discourage the enthusiasm of community > participants, and is very unfriendly to community participants. If in doubt, > I think it would be better to raise it as soon as possible, instead of > waiting for others to finish their hard work before questioning. > > > > > > Another point is that the multi-tenancy function may be a function > required by other companies' IOTDB releases, but will other people's > contributions to the community affect the development of the community? I > think it will be more conducive to the development of community diversity. > > > > > > > > > > > > > > Thanks! > > > > > > Chao Wang > > BONC ltd > > ccgow...@163.com > > On 4/10/2023 23:45，Xiangdong Huang wrote： > > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > > > I think I know this and I have shown my concern about the possible > > harm of this featuer to IoTDB's edge mode... > > > > 1) how many side-effects the feature will bring; > > We have done some tests under[1], which says with 20 databases and 1 > user when we set `quota_enable` to true to enable the multi-tenancy feature, > the write performance is only slowed down 1.75%, the read latency has not > much difference, we will do more tests to show the side-effects in the > feature. > > > > The experiment is rather simple... > > When we really want to show the added codes having no side-effects, > > all the exepriemnt settings should follow a rule that how to fully > > expose the possible problems. > > > > For example, as mult-tenancy limits the available # of devices, > > timeseries, and the spaces of disk, it should have side-effect on > > create new device/timeseries, and writing new data. > > So, > > - what is the side effect when we manually create a time series? > > - what is the side effect when we use automatical creating a time series? > > - what is the side effect when we write new data? (as the data can be > > compressed when it is flushed on disk in async mode, how to check the > > disk space?). Besides, as it impaces each write operation, we need to > > focus on write operstions which's batchsize=1. > > > > This discuss is not for getting "+1" or "-1" (though anyone can reply > > the vote..). > > I just want to discuss that do we REALLY consider and analyze the > > feature and the implementation carefully? > > > > If not, then this big feature is not the time to be merged (and I will > > call a vote then), and then let's rethink it and make it really > > available together. > > If yes, we also need to rethink it and improve it for better > performance. > > > > > > Best, > > ----------------------------------- > > Xiangdong Huang > > School of Software, Tsinghua University > > > > Chao Wang 于2023年4月10日周一 19:14写道： > > > > Agree with Houliang's opinion. > > > > > > Thanks! > > > > > > Chao Wang > > BONC ltd > > On 4/10/2023 19:01，Houliang Qi wrote： > > -1 > > > > First of all, thanks Xiangdong for pointing out IoTDB's Charter. > > > > "RESOLVED, that the Apache IoTDB Project be and hereby is > > responsible for the creation and maintenance of software > > related to an IoT native database with high performance > > for data management and analysis, on the edge and the cloud." > > > > As the charter post, IoTDB can be deployed in the cloud, this is why we > deploy the multi-tenancy feature. > > > > The cloud can be a public or private cloud if we can deploy only one > IoTDB cluster, and manage multi databases and users with different resources, > which will simplify the maintenance. > > > > -> 1) how many side-effects the feature will bring; > > > > We have done some tests under[1], which says with 20 databases and 1 > user when we set `quota_enable` to true to enable the multi-tenancy feature, > the write performance is only slowed down 1.75%, the read latency has not > much difference, we will do more tests to show the side-effects in the > feature. > > > > -> 2) how to reduce the effect when IoTDB is deployed on the edge. > > > > We supply one switch about this feature, called `quota_enable`, by > default this value is false, so it has no effect when IoTDB is deployed on > the edge. > > This also answers Jinrui's doubt. > > > > -> 3) some checks failed on WinOS, are they irrelevant? > > > > No, I think they are not irrelevant, the false check message is about > the Compaction module, and > > I see the former pr[2][3] which have been merged 4 days ago has the same > issue, so I suspect that the compaction module has occasional bugs > > > > -> 4) The feature SHOULD be discussed carefully in the community, > rather that submit PRs and merged after some reviews. > > > > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > > > > > The IoTDB community is open and different opinions are welcome. After > all, we all have the same original intention of wanting IoTDB's features to > be more diverse. > > > > [1] https://apache-iotdb.feishu.cn/docx/DbqCd8t3EoxlCFx1yYicd9N4n4s > > [2] > https://github.com/apache/iotdb/actions/runs/4625220921/jobs/8181102446 > > [3] > https://github.com/apache/iotdb/actions/runs/4531046594/jobs/7980725316 > > [4] https://apache-iotdb.feishu.cn/docx/doxcnKOYKDmJ40FpVnVsPMd3nTg > > [5] https://lists.apache.org/thread/y6dqcm2o7qk0nbkllb61bp8cv6d3m1h7 > > > > > > > > > > > > Thanks, > > --------------------------------------- > > Houliang Qi > > BONC, Ltd > > > > > > ---- Replied Message ---- > > | From | 张金瑞<329920...@qq.com.INVALID> | > > | Date | 04/10/2023 15:03 | > > | To | dev | > > | Subject | Re：[discuss] consider revert the feature of multi-tenancy | > > +1, > > > > > > Agree with Xiangdong's opinion. > > And on the other hand, checking this PR's side effects may take > lot of time and during this period, there may be lots of users using > latest code to deploy/upgrade their systems. So the best practice is > reverting this PR until the side-effect is eliminated > > > > > > > > Thanks, > > Zhang Jinrui，Apache IoTDB PMC > > > > > > > > Original > > > > > > > > From:"Xiangdong Huang"< saint...@gmail.com >; > > > > Date:2023/4/10 10:05 > > > > To:"dev"< dev@iotdb.apache.org >; > > > > Subject:[discuss] consider revert the feature of multi-tenancy > > > > > > Hi all, > > > > I see the multi-tenancy feature is merged, and several committers made > > a lot of contributions on that. > > > > As multi-tenancy is quite a big feature, which may change IoTDB's > > position. The feature SHOULD be discussed carefully in the community, > > rather that submit PRs and merged after some reviews. > > > > Therefore, I call to revert the PR and discuss ASAP about the feature > > after that. > > > > At least, the proposer need to answer the following questions, > > 1) how many side-effect the feature will bring; > > 2) how to reduce the effect when IoTDB is deployed on the edge. > > 3) some checks failed on WinOS, are they irrelevant? > > > > I don't mean of rejecting any big contribution to IoTDB or harming the > > community's diversity, but accepting this feature is really big > > decision and it deserves us to take time to deliberate. > > > > > > Attached IoTDB's Charter: > > "RESOLVED, that the Apache IoTDB Project be and hereby is > > responsible for the creation and maintenance of software > > related to an IoT native database with high performance > > for data management and analysis, on the edge and the cloud." > > > > > > [1] https://github.com/apache/iotdb/pull/9534/checks > > > > Best, > > ----------------------------------- > > Xiangdong Huang > > School of Software, Tsinghua University
