Vyacheslav, Val, can you please clarify - What is the issue if third-party plugins will create «ignite-sys-cache» from the code?
Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. > 2 дек. 2021 г., в 16:13, Вячеслав Коптилин <slava.kopti...@gmail.com> > написал(а): > > Hello Maxim, > >> I don't understand why you are using *backwards compatibility* for > completely internal things. >> Why you are thinking in terms of users usage when are talking about > ignite-sys-cache but not thinking when refactoring > First of all, we are talking about all plugin developers. It does not > matter it is open-source or proprietary plugins. > Honestly, I don't have a list of all possible plugins that have already > been developed and available under the ASF license, for instance. > Do you have such a list? Can you be sure that this change will not affect > anyone? > >> I don't understand why you are using *backwards compatibility* for > completely internal things. >> Why you are thinking in terms of users usage when are talking about > ignite-sys-cache but not thinking when refactoring > The system cache was the crucial thing in the architecture of Apache Ignite > 1.x and 2.x (at least 2.1 - 2.11?) > >> All the internals have been reworked and nobody even mentioned that it > may affect some of the plugin developers. > Yep, perhaps, some internal parts of Apache Ignite were reworked in order > to avoid using the system cache. > However, it is still a part of Ignite and it works and can be used in > plugins. > Honestly, the mentioned alternative, I mean the distributed metastorage, is > the INTERNAL thing as well. > It means that plugin developer depends on INTERNAL entities. (it does not > matter system-cache/metastorage) > IMHO, such questions should be CAREFULLY discussed with no rush. > >> I do not propose to rush with the ignite-sys-cache removal. I'd like to > collect all the technical stuff that we depend on, fix all of them and > after everything will be ready do a final removal. > Good! We are on the same page! > >> 1. Add deprecation for the 2.12 release. I hope it is still possible. > If I am not mistaken, the code freeze was October 29. I think it is too > late to introduce this change. We can add deprecation for the 2.13 release. > >> Apache Ignite core still have some dependencies on ignite-sys-cache that > should fix for 2.13 > Ok, I agree. We need to try to find out all edge cases and add new > abilities to the metastorage in order to cover all known > issues/requirements. > >> Remove the system cache in 2.14. > It depends on our progress with improving the distributed metastorage. In > general, I hope it will be possible. > > Thanks, > S. > > чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov <mmu...@apache.org>: > >> Pavel, >> >> >> I don't understand why you are using *backwards compatibility* for >> completely internal things. Why you are thinking in terms of users >> usage when are talking about ignite-sys-cache but not thinking when >> refactoring, for instance, all the checkpoint classes? Take a look at >> the [1] issue. All the internals have been reworked and nobody even >> mentioned that it may affect some of the plugin developers. This is >> really strange, so I can't agree with you. >> >> >> I do not propose to rush with the ignite-sys-cache removal. I'd like >> to collect all the technical stuff that we depend on, fix all of them >> and after everything will be ready do a final removal. Slava already >> mentioned some crucial cases, so I hope you and Val also help with the >> rest of them. Let's be more precise in terms *what we need to fix*. >> >> >> I've made some investigations related to the ignite-sys-cache and here >> is my proposal: >> >> 1. Add deprecation for the 2.12 release. I hope it is still possible. >> >> 2. Apache Ignite core still have some dependencies on ignite-sys-cache >> that should fix for 2.13: >> >> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of >> system cache is used. Let's move it to metastorage. >> 2.2. When the Security is enabled each compute task name (hash to be >> more precise) is written to the ignite-sys-cache [2]. From my point of >> view, we can remove it in 2.13. Can anyone check? >> 2.3. Slava mentioned some issues with the distributed metastorage that >> might be fixed. I think this is doable. >> >> 3. Remove the system cache in 2.14. >> >> >> >> [1] https://issues.apache.org/jira/browse/IGNITE-13143 >> [2] >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/task/GridTaskProcessor.java#L201 >> >> On Thu, 2 Dec 2021 at 12:56, Pavel Tupitsyn <ptupit...@apache.org> wrote: >>> >>> Maxim, >>> >>>> I don't think that we should wait for 3-rd party plugins to be updated >>>> this is an awful practice when the open-source product releases depend >>>> on some of the proprietary plugins >>> >>> This makes no sense, sorry. >>> >>> It is not that we depend on 3rd party plugins. >>> It is that *our users depend on us to preserve backwards compatibility*. >>> No one likes when their app breaks suddenly because of a library update. >>> >>> We know about one use case for sys-cache in GridGain, but there may be >>> more, no one knows. >>> Every breaking change should be carried out carefully and gradually. >>> >>> >>> On Thu, Dec 2, 2021 at 12:34 PM Maxim Muzafarov <mmu...@apache.org> >> wrote: >>> >>>> Slava, >>>> >>>> Thank you for the comments. >>>> >>>>> Maxim, the community must provide a reasonable time interval in >> order to >>>> notify all contributors and wait for updating all 3-rd party plugins. >>>> >>>> This is not actually true. We must notify about changes as earlier as >>>> possible and not only users but all the developers also. However, I >>>> don't think that we should wait for 3-rd party plugins to be updated >>>> this is an awful practice when the open-source product releases depend >>>> on some of the proprietary plugins. There is no dependency between >>>> Apache Ignite releases and proprietary plugin releases. If someone >>>> desires to upgrade to a new version of Apache Ignite it can update its >>>> plugins any time he likes. >>>> >>>>> We just need to provide a window that is enough to cover all related >>>> issues. >>>> >>>> Can you share all the issues you know, please? >>>> >>>>> distributed meta storage does not provide the ability to atomically >>>> change several properties at the same time (there are no transactions >> on >>>> this API). >>>> >>>> Can you share an example of what kind of properties we intend to >>>> change at the same? Will it be enough to change them through a single >>>> thread (e.g. the discovery thread or the exchange thread)? I agree >>>> here that distributed metastorage should provide such an ability, >>>> however, I can't find any real examples for Apache Ignite internal >>>> needs. Please, share the details and let's fix it. >>>> >>>> On Wed, 1 Dec 2021 at 23:19, Valentin Kulichenko >>>> <valentin.kuliche...@gmail.com> wrote: >>>>> >>>>> Agree with Slava. Two months is not enough time, especially >> considering >>>>> that the system cache is not functionally equivalent to the >> metastorage. >>>> I >>>>> suggest we do the following: >>>>> >>>>> 1. Write down the differences between the system cache and the >>>> metastorage, >>>>> and provide a transition guide for plugin developers. >>>>> 2. Deprecate the system cache in 2.13. >>>>> 3. Remove the system cache in one of the further releases. I don't >> think >>>>> it's reasonable to do this earlier than mid next year (even that is >>>>> potentially too early). >>>>> >>>>> Removing the system cache is the right move, but let's be more >>>> considerate >>>>> to the users. There is absolutely no need to rush. >>>>> >>>>> -Val >>>>> >>>>> On Wed, Dec 1, 2021 at 7:28 AM Вячеслав Коптилин < >>>> slava.kopti...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Maxim, >>>>>> >>>>>>> On the other hand, I don't see any valuable reason why the change >>>> can't >>>>>> be done and we should wait for the future that never comes. >>>>>> Maxim, the community must provide a reasonable time interval in >> order >>>> to >>>>>> notify all contributors and wait for updating all 3-rd party >> plugins. >>>>>> Honestly, it does not seem to me that two, three months (the >> current >>>>>> timeline - end of March is the release date, so the end of Feb is >> code >>>>>> freeze) are quite enough. >>>>>> >>>>>>> I don't see any reasons why we should keep something in the CORE >>>> module >>>>>> that is being used at PLUGINS only. This is a lack of architecture >>>> design >>>>>> that must be fixed for sure; >>>>>> I agree with you. We just need to provide a window that is enough >> to >>>> cover >>>>>> all related issues. >>>>>> For example, distributed meta storage does not provide the ability >> to >>>>>> atomically change several properties at the same time (there are no >>>>>> transactions on this API). >>>>>> >>>>>> Perhaps, it would be nice to plan to remove the system cache on >> 2.14 or >>>>>> even 2.15. What do you think? >>>>>> >>>>>> Thanks, >>>>>> S. >>>>>> >>>>>> >>>>>> вт, 30 нояб. 2021 г. в 13:58, Maxim Muzafarov <mmu...@apache.org>: >>>>>> >>>>>>> Hello Val, >>>>>>> >>>>>>> Thank you for sharing the details. On the one hand, I agree with >> you >>>>>>> that there is no need to haste with this change and it must be >>>>>>> prepared carefully. On the other hand, I don't see any valuable >>>> reason >>>>>>> why the change can't be done and we should wait for the future >> that >>>>>>> never comes. >>>>>>> >>>>>>> Please, consider the following my considerations: >>>>>>> >>>>>>> - The release 2.13 is not started yet. It will take months to be >>>>>>> released (e.g. the end of March as a possible release date). The >> time >>>>>>> is more than enough; >>>>>>> - The 2.13 release has breaking changes, so it is a good >> opportunity >>>>>>> to fix some gaps; >>>>>>> - I don't see any reasons why we should keep something in the >> CORE >>>>>>> module that is being used at PLUGINS only. This is a lack of >>>>>>> architecture design that must be fixed for sure; >>>>>>> - The cache affects cluster stability and require additional >> human >>>>>>> resources for the code maintenance; >>>>>>> - As a straightforward solution plugins can create their own >> internal >>>>>>> caches and use them ever they like (or use the metastorage as I >>>>>>> mentioned earlier). Moving system cache to plugin doesn't look so >>>>>>> complicated and harmful; >>>>>>> >>>>>>> On Mon, 29 Nov 2021 at 23:07, Valentin Kulichenko >>>>>>> <valentin.kuliche...@gmail.com> wrote: >>>>>>>> >>>>>>>> Maxim, >>>>>>>> >>>>>>>> You're right that the system cache is still likely to be used >> by >>>>>> plugins. >>>>>>>> We at GridGain use it for security features, for example. As >> far >>>> as I >>>>>>> know, >>>>>>>> that's not the only case. >>>>>>>> >>>>>>>> I also agree that the metastorage should be the preferred way >> for >>>> this >>>>>>> kind >>>>>>>> of purposes, but is there any harm in keeping the system cache >> for >>>> now? >>>>>>>> Unlike the legacy Service Grid, this is not a public feature >> that >>>> can >>>>>> be >>>>>>>> used directly by a regular user, so I'm not sure why the rush. >>>>>>>> >>>>>>>> How about we instead deprecate the system cache, clearly >> document >>>> how >>>>>> to >>>>>>>> use the metastorage as an alternative, and then complete the >>>> removal >>>>>>>> sometime in the future? >>>>>>>> >>>>>>>> -Val >>>>>>>> >>>>>>>> On Sun, Nov 28, 2021 at 6:49 AM Maxim Muzafarov < >> mmu...@apache.org >>>>> >>>>>>> wrote: >>>>>>>> >>>>>>>>> Igniters, >>>>>>>>> >>>>>>>>> >>>>>>>>> Since the legacy service grid implementation was removed [2] >> I'd >>>> like >>>>>>>>> to remove the ignite-sys-cache also. It is still possible >> that >>>> some >>>>>> of >>>>>>>>> Ignite plugins (e.g. security plugin) are still using this >> cache, >>>>>>>>> however, AFAIK this is not the reason to keep the outdated >> system >>>>>>>>> cache and these plugins must be migrated to the metastorage >>>> instead. >>>>>>>>> >>>>>>>>> I've filed the issue [1] for removal. Any objections? >>>>>>>>> >>>>>>>>> >>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-16008 >>>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-15758 >>>>>>>>> >>>>>>> >>>>>> >>>> >>
