Re: Re：Re: [discuss] consider revert the feature of multi-tenancy
> Everyone's contribution counts. But what we are talking about is whether > `multi-tenancy` is suitable for current IoTDB's development. > From my perspective, Multi-tenancy is different from resource-control and > they are not the different term for same thing. According to our > implementation, current feature focus on the resource control on users of one > tenant rather than on different tenants. If we did not reflect the wording > `multi-tenancy` in the code, why do we use it on user docs and PR's > description ? As I said before, the description is indeed not very clear, and the description can be modified as a resource control. So what's the point of wondering if this pr is a multi-tenant function? Even if it is a multi-tenant function, how will it affect the development of IoTDB? > REVERT does not mean REJECT. It is only a quick way to keep the code more > reliable before we reach the same page. And furthermore, I don't think it is > harmful or discouraging and it is only a regular way we use to replace > hot-fix. Yes, revert is a normal process, and PR also has some problems. Let's discuss the reason for reverting this PR. As Xiangdong said, this is a feature that will affect the positioning of IoTDB, so how does this feature affect the positioning of IoTDB? > Agree. But if we don't make it clear before PR merged, pushing forward the > discussion is better than directly merging, from my side. Agree. I remember sending an email to discuss it. Does that mean that every PR needs to send an email to discuss clearly? After all, pushing forward the discussion is better than directly merging. Thanks! Chao Wang BONC ltd On 4/11/2023 13:10，张金瑞<329920...@qq.com.INVALID> wrote： (Sorry for the format issue in previous mail) == Hi, all I tried this feature locally according to the User Manual, and I am blocked at the beginning. Firstly, I didn't found the parameters `quota_enable` and `rate_limiter_type` in iotdb-common.properties to enable this functionality. I am not sure whether it is by design but it is not aligned with the user manual. And I have to add these two parameters into configuration file manually. Then, I tried the function to limit devices regarding a database and it seems some basic functionality is unexpected. See the test step below: 1. create a databse named `root.iov` and insert 5 devices into it. 2. run the command "set space quota devices=4 on root.iov" to set the device limit to 4. It can be executed successfully. (UNEXPECTED) 3. try to insert a new devices. 4. try to use the command "set space quota devices=8 on root.iov" to increase the threshold of device count but failed. (UNEXPECTED) 5. I created another database named `sg2` and tried to set quota limit to it but failed (UNEXPECTED) 6. After this test, I tried another test with more simple scenario but still failed. The detailed test steps can be found in this doc: https://apache-iotdb.feishu.cn/docx/IerZdPFHroEbRYxKvihcBpncnie My confidence in the completeness of testing regarding this feature has greatly decreased. And at least from my perspective, I cannot guarantee how much impact this feature will have on the user experience. On the other hand, I also have some thoughts to share with you regarding the questions raised in the previous email. Leaving aside this feature, has the PR of the big feature we merged in been discussed in detail? How to define detailed discussion? (Jinrui) I think currently we are focusing on the side effects of this PR, whether the discussion is detailed depends on whether we have enough confidence of this feature's definition and implementation. I think that we should discuss some of our discussion points clearly at the beginning of the design, instead of how to revert the PR after the PR is merged. I think there is a problem with this process. (Jinrui) Agree. But if we don't make it clear before PR merged, pushing forward the discussion is better than directly merging, from my side. Who can guarantee that there are no bugs and no problems in the developed functions, and we are all improving through continuous iteration (Jinrui) Yes. But the developer should do enough tests including different scenarios to ensure this feature can work smoothly. However, reverting will undoubtedly be harmful to the community, will discourage the enthusiasm of community participants, and is very unfriendly to community participants (Jinrui) REVERT does not mean REJECT. It is only a quick way to keep the code more reliable before we reach the same page. And furthermore, I don't think it is harmful or discouraging and it is only a regular way we use to replace hot-fix. If in doubt, I think it would be better to raise it as soon as possible, instead of waiting for others to finish their hard work before questioning. (Jinrui) My words has no doubt and offense to anyone and it is only a discussion about this issue. Contribution
Re: [discuss] consider revert the feature of multi-tenancy
> I missed this PR. I also do not endorse this PR as I think setting > the limitation strategy is not what an open source project should > consider (It is desired only if the system will be unstable if we have > no such a limitation) Why can't some restriction strategies be added to the open source system to prevent a single user from affecting the operation of the overall system. At present, mature open source systems have similar mechanisms, such as doris, hbase, etc.? In addition, what does this mechanism have to do with whether the system is open source or not? It itself is a function that a more mature multi-user system should have. Isn't IoTDB a multi-user system? > What we can do is avoid the case. But if something has conflict with the > project's position, we must do some action. Does this function affect the positioning of IoTDB? IoTDB is only for the edge side? Can't be deployed and used on the cloud side? > Different users have different requirements. But, the PMC need to keep awake > to know or make a CONSENSUS about where the project will go. It is true that PMC should pay attention to the direction of the project, so what direction does this function affect? Does it affect the edge side? What are the effects of features that can be turned off? Thanks! Chao Wang BONC ltd On 4/11/2023 12:16，Xiangdong Huang wrote： How about the pr https://github.com/apache/iotdb/pull/9430, limit the timeseries number of cluster, anyone analyze the side effect about creating a time series? I missed this PR. I also do not endorse this PR as I think setting the limitation strategy is not what an open source project should consider (It is desired only if the system will be unstable if we have no such a limitation) Why not discuss before the PR submission, but wait until the PR submission before discussing, wouldn't it waste the energy of community participants? I have also seen emails sent before, not without notifying everyone. Discussing and notifying on the community is absolutely right. But it does not mean we have to accept and do not change all the fact that has happened. What we can do is avoid the case. But if something has conflict with the project's position, we must do some action. Another point is that the multi-tenancy function may be a function required by other companies' IOTDB releases, but will other people's contributions to the community affect the development of the community? I think it will be more conducive to the development of community diversity. Different users have different requirements. But, the PMC need to keep awake to know or make a CONSENSUS about where the project will go. That is why I start this discussion though I know it will cause many complaint. Best, --- Xiangdong Huang School of Software, Tsinghua University Chao Wang 于2023年4月11日周二 09:16写道： Hi, Xiangdong, what is the side effect when we manually create a time series? How about the pr https://github.com/apache/iotdb/pull/9430, limit the timeseries number of cluster, anyone analyze the side effect about creating a time series? This discuss is not for getting "+1" or "-1" (though anyone can reply the vote..). I just want to discuss that do we REALLY consider and analyze the feature and the implementation carefully? Why not discuss before the PR submission, but wait until the PR submission before discussing, wouldn't it waste the energy of community participants? I have also seen emails sent before, not without notifying everyone. In addition, I think Jialin's suggestion is more reasonable. The description of this function may not be particularly clear. It can be said in another way, such as resource control. However, reverting will undoubtedly be harmful to the community, will discourage the enthusiasm of community participants, and is very unfriendly to community participants. If in doubt, I think it would be better to raise it as soon as possible, instead of waiting for others to finish their hard work before questioning. Another point is that the multi-tenancy function may be a function required by other companies' IOTDB releases, but will other people's contributions to the community affect the development of the community? I think it will be more conducive to the development of community diversity. Thanks! Chao Wang BONC ltd ccgow...@163.com On 4/10/2023 23:45，Xiangdong Huang wrote： Besides the above, when we merge this pr, we posted the design in the feishu[4] and discussed it online as least two times, and emailed and discussed it with everyone[5], it has been passed 10 days. I think I know this and I have shown my concern about the possible harm of this featuer to IoTDB's edge mode... 1) how many side-effects the feature will bring; We have done some tests under[1], which says with 20 databases and 1 user when we set `quota_enable` to true to enable the multi-tenancy feature, the write performance
Re：Re: [discuss] consider revert the feature of multi-tenancy
Re: [discuss] consider revert the feature of multi-tenancy
Hi Houliang, Notice that I never said the feature should be reverted because of bugs.. The key point is the feature is harmful for Industry users because most of them do not like cloud. (that is why I opt for Jialin's suggestion). > I think that we should discuss some of our discussion points clearly at the > beginning of the design, instead of how to revert the PR after the PR is > merged. I think there is a problem with this process. It is of course right, but it does not mean that we can not revert a PR if it is merged. > Leaving aside this feature, has the PR of the big feature we merged in been > discussed in detail? How to define detailed discussion? Yes for each big feature we need a discussion in detail. As I have no much time to join all the features, being the PMC chair, at least I need to keep the project following its original destination or new destination if we all agree. Considering my personal time, I judge and intervene featuers which may change the product's position. That is why I spent time to discuss whether we redesign the cluster mode, whether we split an IoTDB instance into two (CN and DN), and whether we tell IoTDB is for cloud-native... And that is why I do not care about more detailed features.. Best, --- Xiangdong Huang School of Software, Tsinghua University Houliang Qi 于2023年4月11日周二 09:51写道： > > Hi, all > > > Leaving aside this feature, has the PR of the big feature we merged in been > discussed in detail? How to define detailed discussion? > > I think that we should discuss some of our discussion points clearly at the > beginning of the design, instead of how to revert the PR after the PR is > merged. I think there is a problem with this process. > > Who can guarantee that there are no bugs and no problems in the developed > functions, and we are all improving through continuous iteration. And this > feature also refers to the design of some other excellent projects, such as > doris and hbase. > > As for the name of this feature, in doris, it is called multi-tenancy[1], in > hbase it is called quota[2], we can call it resource-control, I think it is > ok. After all, we did not reflect the wording of multi-tenancy in the code > implementation. > > > > [1] https://doris.apache.org/docs/dev/admin-manual/multi-tenant > [2] https://hbase.apache.org/book.html#quota > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | Chao Wang | > | Date | 04/11/2023 09:15 | > | To | dev@iotdb.apache.org | > | Cc | dev@iotdb.apache.org | > | Subject | Re: [discuss] consider revert the feature of multi-tenancy | > Hi, Xiangdong, > > > what is the side effect when we manually create a time series? > > > How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? > > > This discuss is not for getting "+1" or "-1" (though anyone can reply > the vote..). > I just want to discuss that do we REALLY consider and analyze the > feature and the implementation carefully? > > > Why not discuss before the PR submission, but wait until the PR submission > before discussing, wouldn't it waste the energy of community participants? I > have also seen emails sent before, not without notifying everyone. > > > > > In addition, I think Jialin's suggestion is more reasonable. The description > of this function may not be particularly clear. It can be said in another > way, such as resource control. However, reverting will undoubtedly be harmful > to the community, will discourage the enthusiasm of community participants, > and is very unfriendly to community participants. If in doubt, I think it > would be better to raise it as soon as possible, instead of waiting for > others to finish their hard work before questioning. > > > Another point is that the multi-tenancy function may be a function required > by other companies' IOTDB releases, but will other people's contributions to > the community affect the development of the community? I think it will be > more conducive to the development of community diversity. > > > > > > > Thanks! > > > Chao Wang > BONC ltd > ccgow...@163.com > On 4/10/2023 23:45，Xiangdong Huang wrote： > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > I think I know this and I have shown my concern about the possible > harm of this featuer to IoTDB's edge mode... > > 1) how many side-effects the feature will bring; > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the
Re: [discuss] consider revert the feature of multi-tenancy
> How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? I missed this PR. I also do not endorse this PR as I think setting the limitation strategy is not what an open source project should consider (It is desired only if the system will be unstable if we have no such a limitation) > Why not discuss before the PR submission, but wait until the PR submission > before discussing, wouldn't it waste the energy of community participants? I > have also seen emails sent before, not without notifying everyone. Discussing and notifying on the community is absolutely right. But it does not mean we have to accept and do not change all the fact that has happened. What we can do is avoid the case. But if something has conflict with the project's position, we must do some action. > Another point is that the multi-tenancy function may be a function required > by other companies' IOTDB releases, but will other people's contributions to > the community affect the development of the community? I think it will be > more conducive to the development of community diversity. Different users have different requirements. But, the PMC need to keep awake to know or make a CONSENSUS about where the project will go. That is why I start this discussion though I know it will cause many complaint. Best, --- Xiangdong Huang School of Software, Tsinghua University Chao Wang 于2023年4月11日周二 09:16写道： > > Hi, Xiangdong, > > > > what is the side effect when we manually create a time series? > > > How about the pr https://github.com/apache/iotdb/pull/9430, limit the > timeseries number of cluster, anyone analyze the side effect about creating a > time series? > > > > This discuss is not for getting "+1" or "-1" (though anyone can reply > the vote..). > I just want to discuss that do we REALLY consider and analyze the > feature and the implementation carefully? > > > Why not discuss before the PR submission, but wait until the PR submission > before discussing, wouldn't it waste the energy of community participants? I > have also seen emails sent before, not without notifying everyone. > > > > > In addition, I think Jialin's suggestion is more reasonable. The description > of this function may not be particularly clear. It can be said in another > way, such as resource control. However, reverting will undoubtedly be harmful > to the community, will discourage the enthusiasm of community participants, > and is very unfriendly to community participants. If in doubt, I think it > would be better to raise it as soon as possible, instead of waiting for > others to finish their hard work before questioning. > > > Another point is that the multi-tenancy function may be a function required > by other companies' IOTDB releases, but will other people's contributions to > the community affect the development of the community? I think it will be > more conducive to the development of community diversity. > > > > > > > Thanks! > > > Chao Wang > BONC ltd > ccgow...@163.com > On 4/10/2023 23:45，Xiangdong Huang wrote： > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > I think I know this and I have shown my concern about the possible > harm of this featuer to IoTDB's edge mode... > > 1) how many side-effects the feature will bring; > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. > > The experiment is rather simple... > When we really want to show the added codes having no side-effects, > all the exepriemnt settings should follow a rule that how to fully > expose the possible problems. > > For example, as mult-tenancy limits the available # of devices, > timeseries, and the spaces of disk, it should have side-effect on > create new device/timeseries, and writing new data. > So, > - what is the side effect when we manually create a time series? > - what is the side effect when we use automatical creating a time series? > - what is the side effect when we write new data? (as the data can be > compressed when it is flushed on disk in async mode, how to check the > disk space?). Besides, as it impaces each write operation, we need to > focus on write operstions which's batchsize=1. > > This discuss is not for getting "+1" or "-1" (though anyone can reply > the vote..). > I just want to discuss that do we REALLY consider and analyze the > feature and the implementation carefully? > > If not, then this big feature is not the time to be merged (and I will > call a vote then), and
Re: [discuss] consider revert the feature of multi-tenancy
Hi, all Leaving aside this feature, has the PR of the big feature we merged in been discussed in detail? How to define detailed discussion? I think that we should discuss some of our discussion points clearly at the beginning of the design, instead of how to revert the PR after the PR is merged. I think there is a problem with this process. Who can guarantee that there are no bugs and no problems in the developed functions, and we are all improving through continuous iteration. And this feature also refers to the design of some other excellent projects, such as doris and hbase. As for the name of this feature, in doris, it is called multi-tenancy[1], in hbase it is called quota[2], we can call it resource-control, I think it is ok. After all, we did not reflect the wording of multi-tenancy in the code implementation. [1] https://doris.apache.org/docs/dev/admin-manual/multi-tenant [2] https://hbase.apache.org/book.html#quota Thanks, --- Houliang Qi BONC, Ltd Replied Message | From | Chao Wang | | Date | 04/11/2023 09:15 | | To | dev@iotdb.apache.org | | Cc | dev@iotdb.apache.org | | Subject | Re: [discuss] consider revert the feature of multi-tenancy | Hi, Xiangdong, what is the side effect when we manually create a time series? How about the pr https://github.com/apache/iotdb/pull/9430, limit the timeseries number of cluster, anyone analyze the side effect about creating a time series? This discuss is not for getting "+1" or "-1" (though anyone can reply the vote..). I just want to discuss that do we REALLY consider and analyze the feature and the implementation carefully? Why not discuss before the PR submission, but wait until the PR submission before discussing, wouldn't it waste the energy of community participants? I have also seen emails sent before, not without notifying everyone. In addition, I think Jialin's suggestion is more reasonable. The description of this function may not be particularly clear. It can be said in another way, such as resource control. However, reverting will undoubtedly be harmful to the community, will discourage the enthusiasm of community participants, and is very unfriendly to community participants. If in doubt, I think it would be better to raise it as soon as possible, instead of waiting for others to finish their hard work before questioning. Another point is that the multi-tenancy function may be a function required by other companies' IOTDB releases, but will other people's contributions to the community affect the development of the community? I think it will be more conducive to the development of community diversity. Thanks! Chao Wang BONC ltd ccgow...@163.com On 4/10/2023 23:45，Xiangdong Huang wrote： Besides the above, when we merge this pr, we posted the design in the feishu[4] and discussed it online as least two times, and emailed and discussed it with everyone[5], it has been passed 10 days. I think I know this and I have shown my concern about the possible harm of this featuer to IoTDB's edge mode... 1) how many side-effects the feature will bring; We have done some tests under[1], which says with 20 databases and 1 user when we set `quota_enable` to true to enable the multi-tenancy feature, the write performance is only slowed down 1.75%, the read latency has not much difference, we will do more tests to show the side-effects in the feature. The experiment is rather simple... When we really want to show the added codes having no side-effects, all the exepriemnt settings should follow a rule that how to fully expose the possible problems. For example, as mult-tenancy limits the available # of devices, timeseries, and the spaces of disk, it should have side-effect on create new device/timeseries, and writing new data. So, - what is the side effect when we manually create a time series? - what is the side effect when we use automatical creating a time series? - what is the side effect when we write new data? (as the data can be compressed when it is flushed on disk in async mode, how to check the disk space?). Besides, as it impaces each write operation, we need to focus on write operstions which's batchsize=1. This discuss is not for getting "+1" or "-1" (though anyone can reply the vote..). I just want to discuss that do we REALLY consider and analyze the feature and the implementation carefully? If not, then this big feature is not the time to be merged (and I will call a vote then), and then let's rethink it and make it really available together. If yes, we also need to rethink it and improve it for better performance. Best, --- Xiangdong Huang School of Software, Tsinghua University Chao Wang 于2023年4月10日周一 19:14写道： Agree with Houliang's opinion. Thanks! Chao Wang BONC ltd On 4/10/2023 19:01，Houliang Qi wrote： -1 First of all, thanks Xiangdong for pointing out IoTDB's Charter. "RESOLVED, that
Re: [discuss] consider revert the feature of multi-tenancy
Hi, Xiangdong, > what is the side effect when we manually create a time series? How about the pr https://github.com/apache/iotdb/pull/9430, limit the timeseries number of cluster, anyone analyze the side effect about creating a time series? > This discuss is not for getting "+1" or "-1" (though anyone can reply the vote..). I just want to discuss that do we REALLY consider and analyze the feature and the implementation carefully? Why not discuss before the PR submission, but wait until the PR submission before discussing, wouldn't it waste the energy of community participants? I have also seen emails sent before, not without notifying everyone. In addition, I think Jialin's suggestion is more reasonable. The description of this function may not be particularly clear. It can be said in another way, such as resource control. However, reverting will undoubtedly be harmful to the community, will discourage the enthusiasm of community participants, and is very unfriendly to community participants. If in doubt, I think it would be better to raise it as soon as possible, instead of waiting for others to finish their hard work before questioning. Another point is that the multi-tenancy function may be a function required by other companies' IOTDB releases, but will other people's contributions to the community affect the development of the community? I think it will be more conducive to the development of community diversity. Thanks! Chao Wang BONC ltd ccgow...@163.com On 4/10/2023 23:45，Xiangdong Huang wrote： Besides the above, when we merge this pr, we posted the design in the feishu[4] and discussed it online as least two times, and emailed and discussed it with everyone[5], it has been passed 10 days. I think I know this and I have shown my concern about the possible harm of this featuer to IoTDB's edge mode... 1) how many side-effects the feature will bring; We have done some tests under[1], which says with 20 databases and 1 user when we set `quota_enable` to true to enable the multi-tenancy feature, the write performance is only slowed down 1.75%, the read latency has not much difference, we will do more tests to show the side-effects in the feature. The experiment is rather simple... When we really want to show the added codes having no side-effects, all the exepriemnt settings should follow a rule that how to fully expose the possible problems. For example, as mult-tenancy limits the available # of devices, timeseries, and the spaces of disk, it should have side-effect on create new device/timeseries, and writing new data. So, - what is the side effect when we manually create a time series? - what is the side effect when we use automatical creating a time series? - what is the side effect when we write new data? (as the data can be compressed when it is flushed on disk in async mode, how to check the disk space?). Besides, as it impaces each write operation, we need to focus on write operstions which's batchsize=1. This discuss is not for getting "+1" or "-1" (though anyone can reply the vote..). I just want to discuss that do we REALLY consider and analyze the feature and the implementation carefully? If not, then this big feature is not the time to be merged (and I will call a vote then), and then let's rethink it and make it really available together. If yes, we also need to rethink it and improve it for better performance. Best, --- Xiangdong Huang School of Software, Tsinghua University Chao Wang 于2023年4月10日周一 19:14写道： Agree with Houliang's opinion. Thanks! Chao Wang BONC ltd On 4/10/2023 19:01，Houliang Qi wrote： -1 First of all, thanks Xiangdong for pointing out IoTDB's Charter. "RESOLVED, that the Apache IoTDB Project be and hereby is responsible for the creation and maintenance of software related to an IoT native database with high performance for data management and analysis, on the edge and the cloud." As the charter post, IoTDB can be deployed in the cloud, this is why we deploy the multi-tenancy feature. The cloud can be a public or private cloud if we can deploy only one IoTDB cluster, and manage multi databases and users with different resources, which will simplify the maintenance. -> 1) how many side-effects the feature will bring; We have done some tests under[1], which says with 20 databases and 1 user when we set `quota_enable` to true to enable the multi-tenancy feature, the write performance is only slowed down 1.75%, the read latency has not much difference, we will do more tests to show the side-effects in the feature. -> 2) how to reduce the effect when IoTDB is deployed on the edge. We supply one switch about this feature, called `quota_enable`, by default this value is false, so it has no effect when IoTDB is deployed on the edge. This also answers Jinrui's doubt. -> 3) some checks failed on WinOS, are they irrelevant? No, I think they are not irrelevant, the false
Re: [discuss] consider revert the feature of multi-tenancy
> Multi tenant functionality is a basic capability in cloud native scenarios This is the most important reason that I disapprove this feature... "multi-tenancy" is deeply binded with "Cloud".. When people heard of "multi-tenancy", the first impression is "ha, this is a cloud native product"... However, IoTDB's domain is IIoT, and Edge mode is one of its important format. And that is why I claim the Charter. Maybe "multi-tenancy" can be used by some company's IoTDB distribution if the company provides Cloud service (like BNOC), But for Apache IoTDB, we need to consider the side effect of not only the technology but also the user reaction.. > (1) The function name should be resource-control on the storage group > level, rather than multi-tenancy. Jialin's opinion maybe a better choice. Best, --- Xiangdong Huang School of Software, Tsinghua University HW-Chao Wang <576749...@qq.com.invalid> 于2023年4月10日周一 23:43写道： > > hi all, Multi tenant functionality is a basic capability in cloud native > scenarios, and currently many open source databases have this capability, > such as clickhouse, hbase, doris, and so on. Therefore, I suggest integrating > this feature into the community. As for resource consumption, it can be > turned off at the edge through configuration. The side effects (performance > loss) that come with it require some testing to identify the problem. > > > > > ---Original--- > From: "Chao Wang" Date: Mon, Apr 10, 2023 19:14 PM > To: "dev@iotdb.apache.org" Cc: "dev@iotdb.apache.org" Subject: Re: [discuss] consider revert the feature of multi-tenancy > > > Agree with Houliang's opinion. > > > Thanks! > > > Chao Wang > BONC ltd > On 4/10/2023 19:01，Houliang Qi wrote： > -1 > > First of all, thanks Xiangdong for pointing out IoTDB's Charter. > > "RESOLVED, that the Apache IoTDB Project be and hereby is > responsible for the creation and maintenance of software > related to an IoT native database with high performance > for data management and analysis, on the edge and the cloud." > > As the charter post, IoTDB can be deployed in the cloud, this is why we > deploy the multi-tenancy feature. > > The cloud can be a public or private cloud if we can deploy only one IoTDB > cluster, and manage multi databases and users with different resources, which > will simplify the maintenance. > > - 1) how many side-effects the feature will bring; > > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. > > - 2) how to reduce the effect when IoTDB is deployed on the edge. > > We supply one switch about this feature, called `quota_enable`, by default > this value is false, so it has no effect when IoTDB is deployed on the edge. > This also answers Jinrui's doubt. > > - 3) some checks failed on WinOS, are they irrelevant? > > No, I think they are not irrelevant, the false check message is about the > Compaction module, and > I see the former pr[2][3] which have been merged 4 days ago has the same > issue, so I suspect that the compaction module has occasional bugs > > - 4) The feature SHOULD be discussed carefully in the community, rather > that submit PRs and merged after some reviews. > > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > > The IoTDB community is open and different opinions are welcome. After all, we > all have the same original intention of wanting IoTDB's features to be more > diverse. > > [1] https://apache-iotdb.feishu.cn/docx/DbqCd8t3EoxlCFx1yYicd9N4n4s > [2] https://github.com/apache/iotdb/actions/runs/4625220921/jobs/8181102446 > [3] https://github.com/apache/iotdb/actions/runs/4531046594/jobs/7980725316 > [4] https://apache-iotdb.feishu.cn/docx/doxcnKOYKDmJ40FpVnVsPMd3nTg > [5] https://lists.apache.org/thread/y6dqcm2o7qk0nbkllb61bp8cv6d3m1h7 > > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | 张金瑞<329920...@qq.com.INVALID | > | Date | 04/10/2023 15:03 | > | To | dev | > | Subject | Re：[discuss] consider revert the feature of multi-tenancy | > +1, > > > Agree with Xiangdong's opinion. > And on the other hand, checking this PR's side effects may take lot of > time and during this period, there may be lots of users using latest > code to deploy/upgrade their systems. So the best practice is reverting this > PR until the side-effect is eliminated > > > > Thanks, > Zhang Jinrui，Apache IoTDB PMC > > > > Original > > > > From:"Xiangdong Huang"< saint...@gmail.com ; > > Date:2023/4/10 10:05 > > To:"dev"< dev@iotdb.apache.org ; > > Subject:[discuss] consider
Re: [discuss] consider revert the feature of multi-tenancy
> Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. I think I know this and I have shown my concern about the possible harm of this featuer to IoTDB's edge mode... > 1) how many side-effects the feature will bring; > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. The experiment is rather simple... When we really want to show the added codes having no side-effects, all the exepriemnt settings should follow a rule that how to fully expose the possible problems. For example, as mult-tenancy limits the available # of devices, timeseries, and the spaces of disk, it should have side-effect on create new device/timeseries, and writing new data. So, - what is the side effect when we manually create a time series? - what is the side effect when we use automatical creating a time series? - what is the side effect when we write new data? (as the data can be compressed when it is flushed on disk in async mode, how to check the disk space?). Besides, as it impaces each write operation, we need to focus on write operstions which's batchsize=1. This discuss is not for getting "+1" or "-1" (though anyone can reply the vote..). I just want to discuss that do we REALLY consider and analyze the feature and the implementation carefully? If not, then this big feature is not the time to be merged (and I will call a vote then), and then let's rethink it and make it really available together. If yes, we also need to rethink it and improve it for better performance. Best, --- Xiangdong Huang School of Software, Tsinghua University Chao Wang 于2023年4月10日周一 19:14写道： > > Agree with Houliang's opinion. > > > Thanks! > > > Chao Wang > BONC ltd > On 4/10/2023 19:01，Houliang Qi wrote： > -1 > > First of all, thanks Xiangdong for pointing out IoTDB's Charter. > > "RESOLVED, that the Apache IoTDB Project be and hereby is > responsible for the creation and maintenance of software > related to an IoT native database with high performance > for data management and analysis, on the edge and the cloud." > > As the charter post, IoTDB can be deployed in the cloud, this is why we > deploy the multi-tenancy feature. > > The cloud can be a public or private cloud if we can deploy only one IoTDB > cluster, and manage multi databases and users with different resources, which > will simplify the maintenance. > > -> 1) how many side-effects the feature will bring; > > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. > > -> 2) how to reduce the effect when IoTDB is deployed on the edge. > > We supply one switch about this feature, called `quota_enable`, by default > this value is false, so it has no effect when IoTDB is deployed on the edge. > This also answers Jinrui's doubt. > > -> 3) some checks failed on WinOS, are they irrelevant? > > No, I think they are not irrelevant, the false check message is about the > Compaction module, and > I see the former pr[2][3] which have been merged 4 days ago has the same > issue, so I suspect that the compaction module has occasional bugs > > -> 4) The feature SHOULD be discussed carefully in the community, rather that > submit PRs and merged after some reviews. > > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > > The IoTDB community is open and different opinions are welcome. After all, we > all have the same original intention of wanting IoTDB's features to be more > diverse. > > [1] https://apache-iotdb.feishu.cn/docx/DbqCd8t3EoxlCFx1yYicd9N4n4s > [2] https://github.com/apache/iotdb/actions/runs/4625220921/jobs/8181102446 > [3] https://github.com/apache/iotdb/actions/runs/4531046594/jobs/7980725316 > [4] https://apache-iotdb.feishu.cn/docx/doxcnKOYKDmJ40FpVnVsPMd3nTg > [5] https://lists.apache.org/thread/y6dqcm2o7qk0nbkllb61bp8cv6d3m1h7 > > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | 张金瑞<329920...@qq.com.INVALID> | > | Date | 04/10/2023 15:03 | > | To | dev | > | Subject | Re：[discuss] consider revert the feature of multi-tenancy | > +1, > > > Agree with Xiangdong's opinion. > And on the other hand, checking this PR's side effects may take lot of > time
Re: [discuss] consider revert the feature of multi-tenancy
Hi, The following is my perspective. (1) The function name should be resource-control on the storage group level, rather than multi-tenancy. (2) This PR does not bring many side effects as it is disabled by default, so a revert is not a must. (3) The code may need to be improved, but it's ok as long as we keep optimizing. Thanks, — Jialin Qiao Apache IoTDB PMC Chao Wang 于2023年4月10日周一 19:14写道： > > Agree with Houliang's opinion. > > > Thanks! > > > Chao Wang > BONC ltd > On 4/10/2023 19:01，Houliang Qi wrote： > -1 > > First of all, thanks Xiangdong for pointing out IoTDB's Charter. > > "RESOLVED, that the Apache IoTDB Project be and hereby is > responsible for the creation and maintenance of software > related to an IoT native database with high performance > for data management and analysis, on the edge and the cloud." > > As the charter post, IoTDB can be deployed in the cloud, this is why we > deploy the multi-tenancy feature. > > The cloud can be a public or private cloud if we can deploy only one IoTDB > cluster, and manage multi databases and users with different resources, which > will simplify the maintenance. > > -> 1) how many side-effects the feature will bring; > > We have done some tests under[1], which says with 20 databases and 1 user > when we set `quota_enable` to true to enable the multi-tenancy feature, the > write performance is only slowed down 1.75%, the read latency has not much > difference, we will do more tests to show the side-effects in the feature. > > -> 2) how to reduce the effect when IoTDB is deployed on the edge. > > We supply one switch about this feature, called `quota_enable`, by default > this value is false, so it has no effect when IoTDB is deployed on the edge. > This also answers Jinrui's doubt. > > -> 3) some checks failed on WinOS, are they irrelevant? > > No, I think they are not irrelevant, the false check message is about the > Compaction module, and > I see the former pr[2][3] which have been merged 4 days ago has the same > issue, so I suspect that the compaction module has occasional bugs > > -> 4) The feature SHOULD be discussed carefully in the community, rather that > submit PRs and merged after some reviews. > > Besides the above, when we merge this pr, we posted the design in the > feishu[4] and discussed it online as least two times, and emailed and > discussed it with everyone[5], it has been passed 10 days. > > > The IoTDB community is open and different opinions are welcome. After all, we > all have the same original intention of wanting IoTDB's features to be more > diverse. > > [1] https://apache-iotdb.feishu.cn/docx/DbqCd8t3EoxlCFx1yYicd9N4n4s > [2] https://github.com/apache/iotdb/actions/runs/4625220921/jobs/8181102446 > [3] https://github.com/apache/iotdb/actions/runs/4531046594/jobs/7980725316 > [4] https://apache-iotdb.feishu.cn/docx/doxcnKOYKDmJ40FpVnVsPMd3nTg > [5] https://lists.apache.org/thread/y6dqcm2o7qk0nbkllb61bp8cv6d3m1h7 > > > > > > Thanks, > --- > Houliang Qi > BONC, Ltd > > > Replied Message > | From | 张金瑞<329920...@qq.com.INVALID> | > | Date | 04/10/2023 15:03 | > | To | dev | > | Subject | Re：[discuss] consider revert the feature of multi-tenancy | > +1, > > > Agree with Xiangdong's opinion. > And on the other hand, checking this PR's side effects may take lot of > time and during this period, there may be lots of users using latest > code to deploy/upgrade their systems. So the best practice is reverting this > PR until the side-effect is eliminated > > > > Thanks, > Zhang Jinrui，Apache IoTDB PMC > > > > Original > > > > From:"Xiangdong Huang"< saint...@gmail.com ; > > Date:2023/4/10 10:05 > > To:"dev"< dev@iotdb.apache.org ; > > Subject:[discuss] consider revert the feature of multi-tenancy > > > Hi all, > > I see the multi-tenancy feature is merged, and several committers made > a lot of contributions on that. > > As multi-tenancy is quite a big feature, which may change IoTDB's > position. The feature SHOULD be discussed carefully in the community, > rather that submit PRs and merged after some reviews. > > Therefore, I call to revert the PR and discuss ASAP about the feature > after that. > > At least, the proposer need to answer the following questions, > 1) how many side-effect the feature will bring; > 2) how to reduce the effect when IoTDB is deployed on the edge. > 3) some checks failed on WinOS, are they irrelevant? > > I don't mean of rejecting any big contribution to IoTDB or harming the > community's diversity, but accepting this feature is really big > decision and it deserves us to take time to deliberate. > > > Attached IoTDB's Charter: > "RESOLVED, that the Apache IoTDB Project be and hereby is > responsible for the creation and maintenance of software > related to an IoT native database with high performance > for data management and analysis, on the edge and the cloud." > > > [1]
Re: [discuss] consider revert the feature of multi-tenancy
Agree with Houliang's opinion. Thanks! Chao Wang BONC ltd On 4/10/2023 19:01，Houliang Qi wrote： -1 First of all, thanks Xiangdong for pointing out IoTDB's Charter. "RESOLVED, that the Apache IoTDB Project be and hereby is responsible for the creation and maintenance of software related to an IoT native database with high performance for data management and analysis, on the edge and the cloud." As the charter post, IoTDB can be deployed in the cloud, this is why we deploy the multi-tenancy feature. The cloud can be a public or private cloud if we can deploy only one IoTDB cluster, and manage multi databases and users with different resources, which will simplify the maintenance. -> 1) how many side-effects the feature will bring; We have done some tests under[1], which says with 20 databases and 1 user when we set `quota_enable` to true to enable the multi-tenancy feature, the write performance is only slowed down 1.75%, the read latency has not much difference, we will do more tests to show the side-effects in the feature. -> 2) how to reduce the effect when IoTDB is deployed on the edge. We supply one switch about this feature, called `quota_enable`, by default this value is false, so it has no effect when IoTDB is deployed on the edge. This also answers Jinrui's doubt. -> 3) some checks failed on WinOS, are they irrelevant? No, I think they are not irrelevant, the false check message is about the Compaction module, and I see the former pr[2][3] which have been merged 4 days ago has the same issue, so I suspect that the compaction module has occasional bugs -> 4) The feature SHOULD be discussed carefully in the community, rather that submit PRs and merged after some reviews. Besides the above, when we merge this pr, we posted the design in the feishu[4] and discussed it online as least two times, and emailed and discussed it with everyone[5], it has been passed 10 days. The IoTDB community is open and different opinions are welcome. After all, we all have the same original intention of wanting IoTDB's features to be more diverse. [1] https://apache-iotdb.feishu.cn/docx/DbqCd8t3EoxlCFx1yYicd9N4n4s [2] https://github.com/apache/iotdb/actions/runs/4625220921/jobs/8181102446 [3] https://github.com/apache/iotdb/actions/runs/4531046594/jobs/7980725316 [4] https://apache-iotdb.feishu.cn/docx/doxcnKOYKDmJ40FpVnVsPMd3nTg [5] https://lists.apache.org/thread/y6dqcm2o7qk0nbkllb61bp8cv6d3m1h7 Thanks, --- Houliang Qi BONC, Ltd Replied Message | From | 张金瑞<329920...@qq.com.INVALID> | | Date | 04/10/2023 15:03 | | To | dev | | Subject | Re：[discuss] consider revert the feature of multi-tenancy | +1, Agree with Xiangdong's opinion. And on the other hand, checking this PR's side effects may take lot of time and during this period, there may be lots of users using latest code to deploy/upgrade their systems. So the best practice is reverting this PR until the side-effect is eliminated Thanks, Zhang Jinrui，Apache IoTDB PMC Original From:"Xiangdong Huang"< saint...@gmail.com ; Date:2023/4/10 10:05 To:"dev"< dev@iotdb.apache.org ; Subject:[discuss] consider revert the feature of multi-tenancy Hi all, I see the multi-tenancy feature is merged, and several committers made a lot of contributions on that. As multi-tenancy is quite a big feature, which may change IoTDB's position. The feature SHOULD be discussed carefully in the community, rather that submit PRs and merged after some reviews. Therefore, I call to revert the PR and discuss ASAP about the feature after that. At least, the proposer need to answer the following questions, 1) how many side-effect the feature will bring; 2) how to reduce the effect when IoTDB is deployed on the edge. 3) some checks failed on WinOS, are they irrelevant? I don't mean of rejecting any big contribution to IoTDB or harming the community's diversity, but accepting this feature is really big decision and it deserves us to take time to deliberate. Attached IoTDB's Charter: "RESOLVED, that the Apache IoTDB Project be and hereby is responsible for the creation and maintenance of software related to an IoT native database with high performance for data management and analysis, on the edge and the cloud." [1] https://github.com/apache/iotdb/pull/9534/checks Best, --- Xiangdong Huang School of Software, Tsinghua University
Re:[discuss] consider revert the feature of multi-tenancy
-1 First of all, thanks Xiangdong for pointing out IoTDB's Charter. "RESOLVED, that the Apache IoTDB Project be and hereby is responsible for the creation and maintenance of software related to an IoT native database with high performance for data management and analysis, on the edge and the cloud." As the charter post, IoTDB can be deployed in the cloud, this is why we deploy the multi-tenancy feature. The cloud can be a public or private cloud if we can deploy only one IoTDB cluster, and manage multi databases and users with different resources, which will simplify the maintenance. -> 1) how many side-effects the feature will bring; We have done some tests under[1], which says with 20 databases and 1 user when we set `quota_enable` to true to enable the multi-tenancy feature, the write performance is only slowed down 1.75%, the read latency has not much difference, we will do more tests to show the side-effects in the feature. -> 2) how to reduce the effect when IoTDB is deployed on the edge. We supply one switch about this feature, called `quota_enable`, by default this value is false, so it has no effect when IoTDB is deployed on the edge. This also answers Jinrui's doubt. -> 3) some checks failed on WinOS, are they irrelevant? No, I think they are not irrelevant, the false check message is about the Compaction module, and I see the former pr[2][3] which have been merged 4 days ago has the same issue, so I suspect that the compaction module has occasional bugs -> 4) The feature SHOULD be discussed carefully in the community, rather that submit PRs and merged after some reviews. Besides the above, when we merge this pr, we posted the design in the feishu[4] and discussed it online as least two times, and emailed and discussed it with everyone[5], it has been passed 10 days. The IoTDB community is open and different opinions are welcome. After all, we all have the same original intention of wanting IoTDB's features to be more diverse. [1] https://apache-iotdb.feishu.cn/docx/DbqCd8t3EoxlCFx1yYicd9N4n4s [2] https://github.com/apache/iotdb/actions/runs/4625220921/jobs/8181102446 [3] https://github.com/apache/iotdb/actions/runs/4531046594/jobs/7980725316 [4] https://apache-iotdb.feishu.cn/docx/doxcnKOYKDmJ40FpVnVsPMd3nTg [5] https://lists.apache.org/thread/y6dqcm2o7qk0nbkllb61bp8cv6d3m1h7 Thanks, --- Houliang Qi BONC, Ltd Replied Message | From | 张金瑞<329920...@qq.com.INVALID> | | Date | 04/10/2023 15:03 | | To | dev | | Subject | Re：[discuss] consider revert the feature of multi-tenancy | +1, Agree with Xiangdong's opinion. And on the other hand, checking this PR's side effects may take lot of time and during this period, there may be lots of users using latest code to deploy/upgrade their systems. So the best practice is reverting this PR until the side-effect is eliminated Thanks, Zhang Jinrui，Apache IoTDB PMC Original From:"Xiangdong Huang"< saint...@gmail.com ; Date:2023/4/10 10:05 To:"dev"< dev@iotdb.apache.org ; Subject:[discuss] consider revert the feature of multi-tenancy Hi all, I see the multi-tenancy feature is merged, and several committers made a lot of contributions on that. As multi-tenancy is quite a big feature, which may change IoTDB's position. The feature SHOULD be discussed carefully in the community, rather that submit PRs and merged after some reviews. Therefore, I call to revert the PR and discuss ASAP about the feature after that. At least, the proposer need to answer the following questions, 1) how many side-effect the feature will bring; 2) how to reduce the effect when IoTDB is deployed on the edge. 3) some checks failed on WinOS, are they irrelevant? I don't mean of rejecting any big contribution to IoTDB or harming the community's diversity, but accepting this feature is really big decision and it deserves us to take time to deliberate. Attached IoTDB's Charter: "RESOLVED, that the Apache IoTDB Project be and hereby is responsible for the creation and maintenance of software related to an IoT native database with high performance for data management and analysis, on the edge and the cloud." [1] https://github.com/apache/iotdb/pull/9534/checks Best, --- Xiangdong Huang School of Software, Tsinghua University
