Hello Nikolay, >> plugin framework allows to implement internal components and use the internal API. > Can you please, point out to documentation or place in Ignite code that describe this behaviour?
Looks like it states here https://ignite.apache.org/docs/latest/plugins > The Ignite plugin system allows you to extend the core functionality of > Ignite. Plugins have access to different internal Ignite components, such > as security processor and others, and can extend the programmatic API of > Ignite. Thanks, S. сб, 4 дек. 2021 г. в 14:54, Nikolay Izhikov <nizhi...@apache.org>: > Valentin > > > plugin framework allows to implement internal components and use the > internal API. > > Can you please, point out to documentation or place in Ignite code that > describe this behaviour? > AFAIK plugin can only use public API, internal API can be changed any time > we want. > > > Folks, can anyone explain the rush? > > No rush from my side. > > Just want to understand your vision of integration points between Ignite > and plugins. > > > Is there any specific reason for it? > > Intention of IEP-80 is to improve Ignite codebase by removing abandoned > features. > > > But the fact is that there are users that can depend on the system cache > via the plugin framework. > > Why this dependency can’t be changed to any regular cache? > Just replace `Ignite#cache` to `Ignite#getOrCreateCache` and that’s it. > > Do we have some special guarantees or logic besides system cache? > > > > 4 дек. 2021 г., в 02:14, Valentin Kulichenko < > valentin.kuliche...@gmail.com> написал(а): > > > > Nikolay, that's right, plugin framework allows to implement internal > > components and use the internal API. There is a difference between a > plugin > > and an extension that uses only public API > > > > Folks, can anyone explain the rush? Is there any specific reason for it? > > > > I think we all agree that this is a good change - system cache has to > go. I > > guess technically it's not even a breaking change, because system cache > is > > not a public API feature. But the fact is that there are users that can > > depend on the system cache via the plugin framework. Let's be respectful > to > > those users, provide a reasonable documented alternative and give time. > > > > -Val > > > > On Fri, Dec 3, 2021 at 8:18 AM Nikolay Izhikov <nizhi...@apache.org> > wrote: > > > >> I don’t understand how it possible. > >> > >> Are you talking about plugin that uses some kind of internal API? > >> > >>> 3 дек. 2021 г., в 18:50, Вячеслав Коптилин <slava.kopti...@gmail.com> > >> написал(а): > >>> > >>> Hello Nikolay, > >>> > >>> If I am not mistaken, the method you mentioned, allows to create a > "user" > >>> cache that is available through public api for the user. This does not > >>> cover the case when the plugin developer wants to hide such cache and > >>> protect it form the end user (at least, via public api). > >>> > >>> Thanks, > >>> S. > >>> > >>> пт, 3 дек. 2021 г., 14:15 Nikolay Izhikov <nizhi...@apache.org>: > >>> > >>>> Vyacheslav, Val, can you please clarify - What is the issue if > >> third-party > >>>> plugins will create «ignite-sys-cache» from the code? > >>>> > >>>> Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. > >>>> > >>>>> 2 дек. 2021 г., в 16:13, Вячеслав Коптилин <slava.kopti...@gmail.com > > > >>>> написал(а): > >>>>> > >>>>> Hello Maxim, > >>>>> > >>>>>> I don't understand why you are using *backwards compatibility* for > >>>>> completely internal things. > >>>>>> Why you are thinking in terms of users usage when are talking about > >>>>> ignite-sys-cache but not thinking when refactoring > >>>>> First of all, we are talking about all plugin developers. It does not > >>>>> matter it is open-source or proprietary plugins. > >>>>> Honestly, I don't have a list of all possible plugins that have > already > >>>>> been developed and available under the ASF license, for instance. > >>>>> Do you have such a list? Can you be sure that this change will not > >> affect > >>>>> anyone? > >>>>> > >>>>>> I don't understand why you are using *backwards compatibility* for > >>>>> completely internal things. > >>>>>> Why you are thinking in terms of users usage when are talking about > >>>>> ignite-sys-cache but not thinking when refactoring > >>>>> The system cache was the crucial thing in the architecture of Apache > >>>> Ignite > >>>>> 1.x and 2.x (at least 2.1 - 2.11?) > >>>>> > >>>>>> All the internals have been reworked and nobody even mentioned that > it > >>>>> may affect some of the plugin developers. > >>>>> Yep, perhaps, some internal parts of Apache Ignite were reworked in > >> order > >>>>> to avoid using the system cache. > >>>>> However, it is still a part of Ignite and it works and can be used in > >>>>> plugins. > >>>>> Honestly, the mentioned alternative, I mean the distributed > >> metastorage, > >>>> is > >>>>> the INTERNAL thing as well. > >>>>> It means that plugin developer depends on INTERNAL entities. (it does > >> not > >>>>> matter system-cache/metastorage) > >>>>> IMHO, such questions should be CAREFULLY discussed with no rush. > >>>>> > >>>>>> I do not propose to rush with the ignite-sys-cache removal. I'd like > >> to > >>>>> collect all the technical stuff that we depend on, fix all of them > and > >>>>> after everything will be ready do a final removal. > >>>>> Good! We are on the same page! > >>>>> > >>>>>> 1. Add deprecation for the 2.12 release. I hope it is still > possible. > >>>>> If I am not mistaken, the code freeze was October 29. I think it is > too > >>>>> late to introduce this change. We can add deprecation for the 2.13 > >>>> release. > >>>>> > >>>>>> Apache Ignite core still have some dependencies on ignite-sys-cache > >> that > >>>>> should fix for 2.13 > >>>>> Ok, I agree. We need to try to find out all edge cases and add new > >>>>> abilities to the metastorage in order to cover all known > >>>>> issues/requirements. > >>>>> > >>>>>> Remove the system cache in 2.14. > >>>>> It depends on our progress with improving the distributed > metastorage. > >> In > >>>>> general, I hope it will be possible. > >>>>> > >>>>> Thanks, > >>>>> S. > >>>>> > >>>>> чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov <mmu...@apache.org>: > >>>>> > >>>>>> Pavel, > >>>>>> > >>>>>> > >>>>>> I don't understand why you are using *backwards compatibility* for > >>>>>> completely internal things. Why you are thinking in terms of users > >>>>>> usage when are talking about ignite-sys-cache but not thinking when > >>>>>> refactoring, for instance, all the checkpoint classes? Take a look > at > >>>>>> the [1] issue. All the internals have been reworked and nobody even > >>>>>> mentioned that it may affect some of the plugin developers. This is > >>>>>> really strange, so I can't agree with you. > >>>>>> > >>>>>> > >>>>>> I do not propose to rush with the ignite-sys-cache removal. I'd like > >>>>>> to collect all the technical stuff that we depend on, fix all of > them > >>>>>> and after everything will be ready do a final removal. Slava already > >>>>>> mentioned some crucial cases, so I hope you and Val also help with > the > >>>>>> rest of them. Let's be more precise in terms *what we need to fix*. > >>>>>> > >>>>>> > >>>>>> I've made some investigations related to the ignite-sys-cache and > here > >>>>>> is my proposal: > >>>>>> > >>>>>> 1. Add deprecation for the 2.12 release. I hope it is still > possible. > >>>>>> > >>>>>> 2. Apache Ignite core still have some dependencies on > ignite-sys-cache > >>>>>> that should fix for 2.13: > >>>>>> > >>>>>> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of > >>>>>> system cache is used. Let's move it to metastorage. > >>>>>> 2.2. When the Security is enabled each compute task name (hash to be > >>>>>> more precise) is written to the ignite-sys-cache [2]. From my point > of > >>>>>> view, we can remove it in 2.13. Can anyone check? > >>>>>> 2.3. Slava mentioned some issues with the distributed metastorage > that > >>>>>> might be fixed. I think this is doable. > >>>>>> > >>>>>> 3. Remove the system cache in 2.14. > >>>>>> > >>>>>> > >>>>>> > >>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13143 > >>>>>> [2] > >>>>>> > >>>> > >> > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/task/GridTaskProcessor.java#L201 > >>>>>> > >>>>>> On Thu, 2 Dec 2021 at 12:56, Pavel Tupitsyn <ptupit...@apache.org> > >>>> wrote: > >>>>>>> > >>>>>>> Maxim, > >>>>>>> > >>>>>>>> I don't think that we should wait for 3-rd party plugins to be > >> updated > >>>>>>>> this is an awful practice when the open-source product releases > >> depend > >>>>>>>> on some of the proprietary plugins > >>>>>>> > >>>>>>> This makes no sense, sorry. > >>>>>>> > >>>>>>> It is not that we depend on 3rd party plugins. > >>>>>>> It is that *our users depend on us to preserve backwards > >>>> compatibility*. > >>>>>>> No one likes when their app breaks suddenly because of a library > >>>> update. > >>>>>>> > >>>>>>> We know about one use case for sys-cache in GridGain, but there may > >> be > >>>>>>> more, no one knows. > >>>>>>> Every breaking change should be carried out carefully and > gradually. > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Dec 2, 2021 at 12:34 PM Maxim Muzafarov <mmu...@apache.org > > > >>>>>> wrote: > >>>>>>> > >>>>>>>> Slava, > >>>>>>>> > >>>>>>>> Thank you for the comments. > >>>>>>>> > >>>>>>>>> Maxim, the community must provide a reasonable time interval in > >>>>>> order to > >>>>>>>> notify all contributors and wait for updating all 3-rd party > >> plugins. > >>>>>>>> > >>>>>>>> This is not actually true. We must notify about changes as earlier > >> as > >>>>>>>> possible and not only users but all the developers also. However, > I > >>>>>>>> don't think that we should wait for 3-rd party plugins to be > updated > >>>>>>>> this is an awful practice when the open-source product releases > >> depend > >>>>>>>> on some of the proprietary plugins. There is no dependency between > >>>>>>>> Apache Ignite releases and proprietary plugin releases. If someone > >>>>>>>> desires to upgrade to a new version of Apache Ignite it can update > >> its > >>>>>>>> plugins any time he likes. > >>>>>>>> > >>>>>>>>> We just need to provide a window that is enough to cover all > >> related > >>>>>>>> issues. > >>>>>>>> > >>>>>>>> Can you share all the issues you know, please? > >>>>>>>> > >>>>>>>>> distributed meta storage does not provide the ability to > atomically > >>>>>>>> change several properties at the same time (there are no > >> transactions > >>>>>> on > >>>>>>>> this API). > >>>>>>>> > >>>>>>>> Can you share an example of what kind of properties we intend to > >>>>>>>> change at the same? Will it be enough to change them through a > >> single > >>>>>>>> thread (e.g. the discovery thread or the exchange thread)? I agree > >>>>>>>> here that distributed metastorage should provide such an ability, > >>>>>>>> however, I can't find any real examples for Apache Ignite internal > >>>>>>>> needs. Please, share the details and let's fix it. > >>>>>>>> > >>>>>>>> On Wed, 1 Dec 2021 at 23:19, Valentin Kulichenko > >>>>>>>> <valentin.kuliche...@gmail.com> wrote: > >>>>>>>>> > >>>>>>>>> Agree with Slava. Two months is not enough time, especially > >>>>>> considering > >>>>>>>>> that the system cache is not functionally equivalent to the > >>>>>> metastorage. > >>>>>>>> I > >>>>>>>>> suggest we do the following: > >>>>>>>>> > >>>>>>>>> 1. Write down the differences between the system cache and the > >>>>>>>> metastorage, > >>>>>>>>> and provide a transition guide for plugin developers. > >>>>>>>>> 2. Deprecate the system cache in 2.13. > >>>>>>>>> 3. Remove the system cache in one of the further releases. I > don't > >>>>>> think > >>>>>>>>> it's reasonable to do this earlier than mid next year (even that > is > >>>>>>>>> potentially too early). > >>>>>>>>> > >>>>>>>>> Removing the system cache is the right move, but let's be more > >>>>>>>> considerate > >>>>>>>>> to the users. There is absolutely no need to rush. > >>>>>>>>> > >>>>>>>>> -Val > >>>>>>>>> > >>>>>>>>> On Wed, Dec 1, 2021 at 7:28 AM Вячеслав Коптилин < > >>>>>>>> slava.kopti...@gmail.com> > >>>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi Maxim, > >>>>>>>>>> > >>>>>>>>>>> On the other hand, I don't see any valuable reason why the > change > >>>>>>>> can't > >>>>>>>>>> be done and we should wait for the future that never comes. > >>>>>>>>>> Maxim, the community must provide a reasonable time interval in > >>>>>> order > >>>>>>>> to > >>>>>>>>>> notify all contributors and wait for updating all 3-rd party > >>>>>> plugins. > >>>>>>>>>> Honestly, it does not seem to me that two, three months (the > >>>>>> current > >>>>>>>>>> timeline - end of March is the release date, so the end of Feb > is > >>>>>> code > >>>>>>>>>> freeze) are quite enough. > >>>>>>>>>> > >>>>>>>>>>> I don't see any reasons why we should keep something in the > CORE > >>>>>>>> module > >>>>>>>>>> that is being used at PLUGINS only. This is a lack of > architecture > >>>>>>>> design > >>>>>>>>>> that must be fixed for sure; > >>>>>>>>>> I agree with you. We just need to provide a window that is > enough > >>>>>> to > >>>>>>>> cover > >>>>>>>>>> all related issues. > >>>>>>>>>> For example, distributed meta storage does not provide the > ability > >>>>>> to > >>>>>>>>>> atomically change several properties at the same time (there are > >> no > >>>>>>>>>> transactions on this API). > >>>>>>>>>> > >>>>>>>>>> Perhaps, it would be nice to plan to remove the system cache on > >>>>>> 2.14 or > >>>>>>>>>> even 2.15. What do you think? > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> S. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> вт, 30 нояб. 2021 г. в 13:58, Maxim Muzafarov < > mmu...@apache.org > >>> : > >>>>>>>>>> > >>>>>>>>>>> Hello Val, > >>>>>>>>>>> > >>>>>>>>>>> Thank you for sharing the details. On the one hand, I agree > with > >>>>>> you > >>>>>>>>>>> that there is no need to haste with this change and it must be > >>>>>>>>>>> prepared carefully. On the other hand, I don't see any valuable > >>>>>>>> reason > >>>>>>>>>>> why the change can't be done and we should wait for the future > >>>>>> that > >>>>>>>>>>> never comes. > >>>>>>>>>>> > >>>>>>>>>>> Please, consider the following my considerations: > >>>>>>>>>>> > >>>>>>>>>>> - The release 2.13 is not started yet. It will take months to > be > >>>>>>>>>>> released (e.g. the end of March as a possible release date). > The > >>>>>> time > >>>>>>>>>>> is more than enough; > >>>>>>>>>>> - The 2.13 release has breaking changes, so it is a good > >>>>>> opportunity > >>>>>>>>>>> to fix some gaps; > >>>>>>>>>>> - I don't see any reasons why we should keep something in the > >>>>>> CORE > >>>>>>>>>>> module that is being used at PLUGINS only. This is a lack of > >>>>>>>>>>> architecture design that must be fixed for sure; > >>>>>>>>>>> - The cache affects cluster stability and require additional > >>>>>> human > >>>>>>>>>>> resources for the code maintenance; > >>>>>>>>>>> - As a straightforward solution plugins can create their own > >>>>>> internal > >>>>>>>>>>> caches and use them ever they like (or use the metastorage as I > >>>>>>>>>>> mentioned earlier). Moving system cache to plugin doesn't look > so > >>>>>>>>>>> complicated and harmful; > >>>>>>>>>>> > >>>>>>>>>>> On Mon, 29 Nov 2021 at 23:07, Valentin Kulichenko > >>>>>>>>>>> <valentin.kuliche...@gmail.com> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> Maxim, > >>>>>>>>>>>> > >>>>>>>>>>>> You're right that the system cache is still likely to be used > >>>>>> by > >>>>>>>>>> plugins. > >>>>>>>>>>>> We at GridGain use it for security features, for example. As > >>>>>> far > >>>>>>>> as I > >>>>>>>>>>> know, > >>>>>>>>>>>> that's not the only case. > >>>>>>>>>>>> > >>>>>>>>>>>> I also agree that the metastorage should be the preferred way > >>>>>> for > >>>>>>>> this > >>>>>>>>>>> kind > >>>>>>>>>>>> of purposes, but is there any harm in keeping the system cache > >>>>>> for > >>>>>>>> now? > >>>>>>>>>>>> Unlike the legacy Service Grid, this is not a public feature > >>>>>> that > >>>>>>>> can > >>>>>>>>>> be > >>>>>>>>>>>> used directly by a regular user, so I'm not sure why the rush. > >>>>>>>>>>>> > >>>>>>>>>>>> How about we instead deprecate the system cache, clearly > >>>>>> document > >>>>>>>> how > >>>>>>>>>> to > >>>>>>>>>>>> use the metastorage as an alternative, and then complete the > >>>>>>>> removal > >>>>>>>>>>>> sometime in the future? > >>>>>>>>>>>> > >>>>>>>>>>>> -Val > >>>>>>>>>>>> > >>>>>>>>>>>> On Sun, Nov 28, 2021 at 6:49 AM Maxim Muzafarov < > >>>>>> mmu...@apache.org > >>>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Igniters, > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Since the legacy service grid implementation was removed [2] > >>>>>> I'd > >>>>>>>> like > >>>>>>>>>>>>> to remove the ignite-sys-cache also. It is still possible > >>>>>> that > >>>>>>>> some > >>>>>>>>>> of > >>>>>>>>>>>>> Ignite plugins (e.g. security plugin) are still using this > >>>>>> cache, > >>>>>>>>>>>>> however, AFAIK this is not the reason to keep the outdated > >>>>>> system > >>>>>>>>>>>>> cache and these plugins must be migrated to the metastorage > >>>>>>>> instead. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I've filed the issue [1] for removal. Any objections? > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-16008 > >>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-15758 > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>> > >>>> > >>>> > >> > >> > >
