Re: [DISCUSSION] Remove outdated ignite-sys-cache
Nikolay, that's right, plugin framework allows to implement internal components and use the internal API. There is a difference between a plugin and an extension that uses only public API Folks, can anyone explain the rush? Is there any specific reason for it? I think we all agree that this is a good change - system cache has to go. I guess technically it's not even a breaking change, because system cache is not a public API feature. But the fact is that there are users that can depend on the system cache via the plugin framework. Let's be respectful to those users, provide a reasonable documented alternative and give time. -Val On Fri, Dec 3, 2021 at 8:18 AM Nikolay Izhikov wrote: > I don’t understand how it possible. > > Are you talking about plugin that uses some kind of internal API? > > > 3 дек. 2021 г., в 18:50, Вячеслав Коптилин > написал(а): > > > > Hello Nikolay, > > > > If I am not mistaken, the method you mentioned, allows to create a "user" > > cache that is available through public api for the user. This does not > > cover the case when the plugin developer wants to hide such cache and > > protect it form the end user (at least, via public api). > > > > Thanks, > > S. > > > > пт, 3 дек. 2021 г., 14:15 Nikolay Izhikov : > > > >> Vyacheslav, Val, can you please clarify - What is the issue if > third-party > >> plugins will create «ignite-sys-cache» from the code? > >> > >> Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. > >> > >>> 2 дек. 2021 г., в 16:13, Вячеслав Коптилин > >> написал(а): > >>> > >>> Hello Maxim, > >>> > I don't understand why you are using *backwards compatibility* for > >>> completely internal things. > Why you are thinking in terms of users usage when are talking about > >>> ignite-sys-cache but not thinking when refactoring > >>> First of all, we are talking about all plugin developers. It does not > >>> matter it is open-source or proprietary plugins. > >>> Honestly, I don't have a list of all possible plugins that have already > >>> been developed and available under the ASF license, for instance. > >>> Do you have such a list? Can you be sure that this change will not > affect > >>> anyone? > >>> > I don't understand why you are using *backwards compatibility* for > >>> completely internal things. > Why you are thinking in terms of users usage when are talking about > >>> ignite-sys-cache but not thinking when refactoring > >>> The system cache was the crucial thing in the architecture of Apache > >> Ignite > >>> 1.x and 2.x (at least 2.1 - 2.11?) > >>> > All the internals have been reworked and nobody even mentioned that it > >>> may affect some of the plugin developers. > >>> Yep, perhaps, some internal parts of Apache Ignite were reworked in > order > >>> to avoid using the system cache. > >>> However, it is still a part of Ignite and it works and can be used in > >>> plugins. > >>> Honestly, the mentioned alternative, I mean the distributed > metastorage, > >> is > >>> the INTERNAL thing as well. > >>> It means that plugin developer depends on INTERNAL entities. (it does > not > >>> matter system-cache/metastorage) > >>> IMHO, such questions should be CAREFULLY discussed with no rush. > >>> > I do not propose to rush with the ignite-sys-cache removal. I'd like > to > >>> collect all the technical stuff that we depend on, fix all of them and > >>> after everything will be ready do a final removal. > >>> Good! We are on the same page! > >>> > 1. Add deprecation for the 2.12 release. I hope it is still possible. > >>> If I am not mistaken, the code freeze was October 29. I think it is too > >>> late to introduce this change. We can add deprecation for the 2.13 > >> release. > >>> > Apache Ignite core still have some dependencies on ignite-sys-cache > that > >>> should fix for 2.13 > >>> Ok, I agree. We need to try to find out all edge cases and add new > >>> abilities to the metastorage in order to cover all known > >>> issues/requirements. > >>> > Remove the system cache in 2.14. > >>> It depends on our progress with improving the distributed metastorage. > In > >>> general, I hope it will be possible. > >>> > >>> Thanks, > >>> S. > >>> > >>> чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov : > >>> > Pavel, > > > I don't understand why you are using *backwards compatibility* for > completely internal things. Why you are thinking in terms of users > usage when are talking about ignite-sys-cache but not thinking when > refactoring, for instance, all the checkpoint classes? Take a look at > the [1] issue. All the internals have been reworked and nobody even > mentioned that it may affect some of the plugin developers. This is > really strange, so I can't agree with you. > > > I do not propose to rush with the ignite-sys-cache removal. I'd like > to collect all the technical stuff that we depend on, fix all of them > and after
Re: [DISCUSSION] Shmem removal.
I think we can safely remove it. -Val On Thu, Dec 2, 2021 at 11:52 PM Ivan Daschinsky wrote: > Hi, igniters. > > Recently I've discovered one fact > 1. GridShmemCommunicationClient and all shmem functionality are broken > since 2.10. In master it is broken since August 2020. And nobody have > noticed it, only one thread in user list. > 2. We have source code for native JNI library (that is shipped in > ignite-shmem.jar), but we never built it since 2015. > 3. This code is of questionable quality, contains outdated internal gcc api > (__sync builtins, now deprecated in favour of __atomic builtins in gcc and > is not advisable to use since C++ 11). It contains a lot of autotool mess, > while just one CMakeFile.txt is enough to build the same > 4. This code doesn't even compile on modern gcc (gcc 9.3.0 namely) > > We have 2 options > 1. If this concept is useful, we (for example I can do it) should rewrite > native part, > a. Use C++ 11 and header-only boost.interprocess [1] > b. Build it regularly with CMake and incorporate build in regular TC runs > (via maven-cmake-plugin, > see for example my numa-allocator [2]). > 2. If this concept and functionality is not useful, we should remove it, > may be even in 2.12 > > > [1] -- https://www.boost.org/doc/libs/1_77_0/doc/html/interprocess.html > [2] -- > > https://github.com/apache/ignite/pull/9569/files#diff-77baf2378aa83911a8c3091814db3ff60b7bf328c4ab4850f707717ed96f3d92 > -- > Sincerely yours, Ivan Daschinskiy >
Re: [DISCUSSION] Remove outdated ignite-sys-cache
I don’t understand how it possible. Are you talking about plugin that uses some kind of internal API? > 3 дек. 2021 г., в 18:50, Вячеслав Коптилин > написал(а): > > Hello Nikolay, > > If I am not mistaken, the method you mentioned, allows to create a "user" > cache that is available through public api for the user. This does not > cover the case when the plugin developer wants to hide such cache and > protect it form the end user (at least, via public api). > > Thanks, > S. > > пт, 3 дек. 2021 г., 14:15 Nikolay Izhikov : > >> Vyacheslav, Val, can you please clarify - What is the issue if third-party >> plugins will create «ignite-sys-cache» from the code? >> >> Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. >> >>> 2 дек. 2021 г., в 16:13, Вячеслав Коптилин >> написал(а): >>> >>> Hello Maxim, >>> I don't understand why you are using *backwards compatibility* for >>> completely internal things. Why you are thinking in terms of users usage when are talking about >>> ignite-sys-cache but not thinking when refactoring >>> First of all, we are talking about all plugin developers. It does not >>> matter it is open-source or proprietary plugins. >>> Honestly, I don't have a list of all possible plugins that have already >>> been developed and available under the ASF license, for instance. >>> Do you have such a list? Can you be sure that this change will not affect >>> anyone? >>> I don't understand why you are using *backwards compatibility* for >>> completely internal things. Why you are thinking in terms of users usage when are talking about >>> ignite-sys-cache but not thinking when refactoring >>> The system cache was the crucial thing in the architecture of Apache >> Ignite >>> 1.x and 2.x (at least 2.1 - 2.11?) >>> All the internals have been reworked and nobody even mentioned that it >>> may affect some of the plugin developers. >>> Yep, perhaps, some internal parts of Apache Ignite were reworked in order >>> to avoid using the system cache. >>> However, it is still a part of Ignite and it works and can be used in >>> plugins. >>> Honestly, the mentioned alternative, I mean the distributed metastorage, >> is >>> the INTERNAL thing as well. >>> It means that plugin developer depends on INTERNAL entities. (it does not >>> matter system-cache/metastorage) >>> IMHO, such questions should be CAREFULLY discussed with no rush. >>> I do not propose to rush with the ignite-sys-cache removal. I'd like to >>> collect all the technical stuff that we depend on, fix all of them and >>> after everything will be ready do a final removal. >>> Good! We are on the same page! >>> 1. Add deprecation for the 2.12 release. I hope it is still possible. >>> If I am not mistaken, the code freeze was October 29. I think it is too >>> late to introduce this change. We can add deprecation for the 2.13 >> release. >>> Apache Ignite core still have some dependencies on ignite-sys-cache that >>> should fix for 2.13 >>> Ok, I agree. We need to try to find out all edge cases and add new >>> abilities to the metastorage in order to cover all known >>> issues/requirements. >>> Remove the system cache in 2.14. >>> It depends on our progress with improving the distributed metastorage. In >>> general, I hope it will be possible. >>> >>> Thanks, >>> S. >>> >>> чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov : >>> Pavel, I don't understand why you are using *backwards compatibility* for completely internal things. Why you are thinking in terms of users usage when are talking about ignite-sys-cache but not thinking when refactoring, for instance, all the checkpoint classes? Take a look at the [1] issue. All the internals have been reworked and nobody even mentioned that it may affect some of the plugin developers. This is really strange, so I can't agree with you. I do not propose to rush with the ignite-sys-cache removal. I'd like to collect all the technical stuff that we depend on, fix all of them and after everything will be ready do a final removal. Slava already mentioned some crucial cases, so I hope you and Val also help with the rest of them. Let's be more precise in terms *what we need to fix*. I've made some investigations related to the ignite-sys-cache and here is my proposal: 1. Add deprecation for the 2.12 release. I hope it is still possible. 2. Apache Ignite core still have some dependencies on ignite-sys-cache that should fix for 2.13: 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of system cache is used. Let's move it to metastorage. 2.2. When the Security is enabled each compute task name (hash to be more precise) is written to the ignite-sys-cache [2]. From my point of view, we can remove it in 2.13. Can anyone check? 2.3. Slava mentioned some issues
Re: [DISCUSSION] Remove outdated ignite-sys-cache
Hello Nikolay, If I am not mistaken, the method you mentioned, allows to create a "user" cache that is available through public api for the user. This does not cover the case when the plugin developer wants to hide such cache and protect it form the end user (at least, via public api). Thanks, S. пт, 3 дек. 2021 г., 14:15 Nikolay Izhikov : > Vyacheslav, Val, can you please clarify - What is the issue if third-party > plugins will create «ignite-sys-cache» from the code? > > Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. > > > 2 дек. 2021 г., в 16:13, Вячеслав Коптилин > написал(а): > > > > Hello Maxim, > > > >> I don't understand why you are using *backwards compatibility* for > > completely internal things. > >> Why you are thinking in terms of users usage when are talking about > > ignite-sys-cache but not thinking when refactoring > > First of all, we are talking about all plugin developers. It does not > > matter it is open-source or proprietary plugins. > > Honestly, I don't have a list of all possible plugins that have already > > been developed and available under the ASF license, for instance. > > Do you have such a list? Can you be sure that this change will not affect > > anyone? > > > >> I don't understand why you are using *backwards compatibility* for > > completely internal things. > >> Why you are thinking in terms of users usage when are talking about > > ignite-sys-cache but not thinking when refactoring > > The system cache was the crucial thing in the architecture of Apache > Ignite > > 1.x and 2.x (at least 2.1 - 2.11?) > > > >> All the internals have been reworked and nobody even mentioned that it > > may affect some of the plugin developers. > > Yep, perhaps, some internal parts of Apache Ignite were reworked in order > > to avoid using the system cache. > > However, it is still a part of Ignite and it works and can be used in > > plugins. > > Honestly, the mentioned alternative, I mean the distributed metastorage, > is > > the INTERNAL thing as well. > > It means that plugin developer depends on INTERNAL entities. (it does not > > matter system-cache/metastorage) > > IMHO, such questions should be CAREFULLY discussed with no rush. > > > >> I do not propose to rush with the ignite-sys-cache removal. I'd like to > > collect all the technical stuff that we depend on, fix all of them and > > after everything will be ready do a final removal. > > Good! We are on the same page! > > > >> 1. Add deprecation for the 2.12 release. I hope it is still possible. > > If I am not mistaken, the code freeze was October 29. I think it is too > > late to introduce this change. We can add deprecation for the 2.13 > release. > > > >> Apache Ignite core still have some dependencies on ignite-sys-cache that > > should fix for 2.13 > > Ok, I agree. We need to try to find out all edge cases and add new > > abilities to the metastorage in order to cover all known > > issues/requirements. > > > >> Remove the system cache in 2.14. > > It depends on our progress with improving the distributed metastorage. In > > general, I hope it will be possible. > > > > Thanks, > > S. > > > > чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov : > > > >> Pavel, > >> > >> > >> I don't understand why you are using *backwards compatibility* for > >> completely internal things. Why you are thinking in terms of users > >> usage when are talking about ignite-sys-cache but not thinking when > >> refactoring, for instance, all the checkpoint classes? Take a look at > >> the [1] issue. All the internals have been reworked and nobody even > >> mentioned that it may affect some of the plugin developers. This is > >> really strange, so I can't agree with you. > >> > >> > >> I do not propose to rush with the ignite-sys-cache removal. I'd like > >> to collect all the technical stuff that we depend on, fix all of them > >> and after everything will be ready do a final removal. Slava already > >> mentioned some crucial cases, so I hope you and Val also help with the > >> rest of them. Let's be more precise in terms *what we need to fix*. > >> > >> > >> I've made some investigations related to the ignite-sys-cache and here > >> is my proposal: > >> > >> 1. Add deprecation for the 2.12 release. I hope it is still possible. > >> > >> 2. Apache Ignite core still have some dependencies on ignite-sys-cache > >> that should fix for 2.13: > >> > >> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of > >> system cache is used. Let's move it to metastorage. > >> 2.2. When the Security is enabled each compute task name (hash to be > >> more precise) is written to the ignite-sys-cache [2]. From my point of > >> view, we can remove it in 2.13. Can anyone check? > >> 2.3. Slava mentioned some issues with the distributed metastorage that > >> might be fixed. I think this is doable. > >> > >> 3. Remove the system cache in 2.14. > >> > >> > >> > >> [1] https://issues.apache.org/jira/browse/IGNITE-13143 > >> [2] > >> >
Add ability to specify INLINE_SIZE for PK sorted index and key's schema validation [IGNITE-11402]
Hi Igniters,
There is an issue to add ability to specify INLINE_SIZE for sorted index on
the primary key and index on affinite fields [1].
Please take a look at the last comment (proposal).
I duplicate the proposal here.
*Proposal:*
add to the public QueryEntity properties:
- Integer primaryKeyInlineSize - use wrapper object to compatibility (null
value - default behavior);
- Integer affinityFieldInlineSize - use wrapper object to compatibility
(null value - default behavior);
The approach has a contradiction with cache API behavior.
Now there is no ability to specify PK sorted index with unwrapped PK fields
by cache API + QueryEntity. This functionality is available only from SQL
command 'CREATE TABLE' (see H2TableDescriptor#extractKeyColumns)
So,
- by SQL command we cannot create 'wrapped' PK sorted index for composite
PK;
- by cache API we cannot create 'unwrapped' PK sorted index for composite
PK.
I propose to add the public property:
Boolean QueryEntity#unwrapPrimaryKeyFieldsForSortedIndex
to make SQL & cache API symmetrical.
*There is a pitfall here*
User may specify the key class like below:
class MyKey{
@QuerySqlField
int id0;
@QuerySqlField
int id1;
int hiddenId;
}
In this case two objects MyKey(0, 0, 0) & MyKey(0,0,1) are different and
may be put into cache as a two different keys.
But they are similar for SQL PK.
But this scenario now may be produced by SQL command CREATE TABLE + cache
API.
I propose to add key schema validation to the
QueryTypeDescriptorImpl#validateKeyAndValue.
[1]. https://issues.apache.org/jira/browse/IGNITE-11402
Re: [DISCUSSION] Remove outdated ignite-sys-cache
> What is the issue if third-party plugins will create «ignite-sys-cache» from the code? Great idea! On Fri, Dec 3, 2021 at 2:15 PM Nikolay Izhikov wrote: > Vyacheslav, Val, can you please clarify - What is the issue if third-party > plugins will create «ignite-sys-cache» from the code? > > Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. > > > 2 дек. 2021 г., в 16:13, Вячеслав Коптилин > написал(а): > > > > Hello Maxim, > > > >> I don't understand why you are using *backwards compatibility* for > > completely internal things. > >> Why you are thinking in terms of users usage when are talking about > > ignite-sys-cache but not thinking when refactoring > > First of all, we are talking about all plugin developers. It does not > > matter it is open-source or proprietary plugins. > > Honestly, I don't have a list of all possible plugins that have already > > been developed and available under the ASF license, for instance. > > Do you have such a list? Can you be sure that this change will not affect > > anyone? > > > >> I don't understand why you are using *backwards compatibility* for > > completely internal things. > >> Why you are thinking in terms of users usage when are talking about > > ignite-sys-cache but not thinking when refactoring > > The system cache was the crucial thing in the architecture of Apache > Ignite > > 1.x and 2.x (at least 2.1 - 2.11?) > > > >> All the internals have been reworked and nobody even mentioned that it > > may affect some of the plugin developers. > > Yep, perhaps, some internal parts of Apache Ignite were reworked in order > > to avoid using the system cache. > > However, it is still a part of Ignite and it works and can be used in > > plugins. > > Honestly, the mentioned alternative, I mean the distributed metastorage, > is > > the INTERNAL thing as well. > > It means that plugin developer depends on INTERNAL entities. (it does not > > matter system-cache/metastorage) > > IMHO, such questions should be CAREFULLY discussed with no rush. > > > >> I do not propose to rush with the ignite-sys-cache removal. I'd like to > > collect all the technical stuff that we depend on, fix all of them and > > after everything will be ready do a final removal. > > Good! We are on the same page! > > > >> 1. Add deprecation for the 2.12 release. I hope it is still possible. > > If I am not mistaken, the code freeze was October 29. I think it is too > > late to introduce this change. We can add deprecation for the 2.13 > release. > > > >> Apache Ignite core still have some dependencies on ignite-sys-cache that > > should fix for 2.13 > > Ok, I agree. We need to try to find out all edge cases and add new > > abilities to the metastorage in order to cover all known > > issues/requirements. > > > >> Remove the system cache in 2.14. > > It depends on our progress with improving the distributed metastorage. In > > general, I hope it will be possible. > > > > Thanks, > > S. > > > > чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov : > > > >> Pavel, > >> > >> > >> I don't understand why you are using *backwards compatibility* for > >> completely internal things. Why you are thinking in terms of users > >> usage when are talking about ignite-sys-cache but not thinking when > >> refactoring, for instance, all the checkpoint classes? Take a look at > >> the [1] issue. All the internals have been reworked and nobody even > >> mentioned that it may affect some of the plugin developers. This is > >> really strange, so I can't agree with you. > >> > >> > >> I do not propose to rush with the ignite-sys-cache removal. I'd like > >> to collect all the technical stuff that we depend on, fix all of them > >> and after everything will be ready do a final removal. Slava already > >> mentioned some crucial cases, so I hope you and Val also help with the > >> rest of them. Let's be more precise in terms *what we need to fix*. > >> > >> > >> I've made some investigations related to the ignite-sys-cache and here > >> is my proposal: > >> > >> 1. Add deprecation for the 2.12 release. I hope it is still possible. > >> > >> 2. Apache Ignite core still have some dependencies on ignite-sys-cache > >> that should fix for 2.13: > >> > >> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of > >> system cache is used. Let's move it to metastorage. > >> 2.2. When the Security is enabled each compute task name (hash to be > >> more precise) is written to the ignite-sys-cache [2]. From my point of > >> view, we can remove it in 2.13. Can anyone check? > >> 2.3. Slava mentioned some issues with the distributed metastorage that > >> might be fixed. I think this is doable. > >> > >> 3. Remove the system cache in 2.14. > >> > >> > >> > >> [1] https://issues.apache.org/jira/browse/IGNITE-13143 > >> [2] > >> > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/task/GridTaskProcessor.java#L201 > >> > >> On Thu, 2 Dec 2021 at 12:56, Pavel
Re: [DISCUSSION] Remove outdated ignite-sys-cache
Vyacheslav, Val, can you please clarify - What is the issue if third-party plugins will create «ignite-sys-cache» from the code? Like just replacing `Ignite#cache` with the `Ignite#getOrCreateCache`. > 2 дек. 2021 г., в 16:13, Вячеслав Коптилин > написал(а): > > Hello Maxim, > >> I don't understand why you are using *backwards compatibility* for > completely internal things. >> Why you are thinking in terms of users usage when are talking about > ignite-sys-cache but not thinking when refactoring > First of all, we are talking about all plugin developers. It does not > matter it is open-source or proprietary plugins. > Honestly, I don't have a list of all possible plugins that have already > been developed and available under the ASF license, for instance. > Do you have such a list? Can you be sure that this change will not affect > anyone? > >> I don't understand why you are using *backwards compatibility* for > completely internal things. >> Why you are thinking in terms of users usage when are talking about > ignite-sys-cache but not thinking when refactoring > The system cache was the crucial thing in the architecture of Apache Ignite > 1.x and 2.x (at least 2.1 - 2.11?) > >> All the internals have been reworked and nobody even mentioned that it > may affect some of the plugin developers. > Yep, perhaps, some internal parts of Apache Ignite were reworked in order > to avoid using the system cache. > However, it is still a part of Ignite and it works and can be used in > plugins. > Honestly, the mentioned alternative, I mean the distributed metastorage, is > the INTERNAL thing as well. > It means that plugin developer depends on INTERNAL entities. (it does not > matter system-cache/metastorage) > IMHO, such questions should be CAREFULLY discussed with no rush. > >> I do not propose to rush with the ignite-sys-cache removal. I'd like to > collect all the technical stuff that we depend on, fix all of them and > after everything will be ready do a final removal. > Good! We are on the same page! > >> 1. Add deprecation for the 2.12 release. I hope it is still possible. > If I am not mistaken, the code freeze was October 29. I think it is too > late to introduce this change. We can add deprecation for the 2.13 release. > >> Apache Ignite core still have some dependencies on ignite-sys-cache that > should fix for 2.13 > Ok, I agree. We need to try to find out all edge cases and add new > abilities to the metastorage in order to cover all known > issues/requirements. > >> Remove the system cache in 2.14. > It depends on our progress with improving the distributed metastorage. In > general, I hope it will be possible. > > Thanks, > S. > > чт, 2 дек. 2021 г. в 13:36, Maxim Muzafarov : > >> Pavel, >> >> >> I don't understand why you are using *backwards compatibility* for >> completely internal things. Why you are thinking in terms of users >> usage when are talking about ignite-sys-cache but not thinking when >> refactoring, for instance, all the checkpoint classes? Take a look at >> the [1] issue. All the internals have been reworked and nobody even >> mentioned that it may affect some of the plugin developers. This is >> really strange, so I can't agree with you. >> >> >> I do not propose to rush with the ignite-sys-cache removal. I'd like >> to collect all the technical stuff that we depend on, fix all of them >> and after everything will be ready do a final removal. Slava already >> mentioned some crucial cases, so I hope you and Val also help with the >> rest of them. Let's be more precise in terms *what we need to fix*. >> >> >> I've made some investigations related to the ignite-sys-cache and here >> is my proposal: >> >> 1. Add deprecation for the 2.12 release. I hope it is still possible. >> >> 2. Apache Ignite core still have some dependencies on ignite-sys-cache >> that should fix for 2.13: >> >> 2.1. CLUSTER_NAME property: when it's not set the `deploymentId` of >> system cache is used. Let's move it to metastorage. >> 2.2. When the Security is enabled each compute task name (hash to be >> more precise) is written to the ignite-sys-cache [2]. From my point of >> view, we can remove it in 2.13. Can anyone check? >> 2.3. Slava mentioned some issues with the distributed metastorage that >> might be fixed. I think this is doable. >> >> 3. Remove the system cache in 2.14. >> >> >> >> [1] https://issues.apache.org/jira/browse/IGNITE-13143 >> [2] >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/task/GridTaskProcessor.java#L201 >> >> On Thu, 2 Dec 2021 at 12:56, Pavel Tupitsyn wrote: >>> >>> Maxim, >>> I don't think that we should wait for 3-rd party plugins to be updated this is an awful practice when the open-source product releases depend on some of the proprietary plugins >>> >>> This makes no sense, sorry. >>> >>> It is not that we depend on 3rd party plugins. >>> It is that *our users
Re: NUMA aware allocator, PR review request
Ok, TC suite is ready [1]. If there is no objections, I will merge it soon. Possible concerns -- now it is required to install build-essentials and libnuma-dev in order to build ignite on 64 bit linux. I suppose that this is not a big deal, but maybe someone will contradict? [1] -- https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_NumaAllocator/?mode=builds чт, 2 дек. 2021 г. в 12:03, Ivan Daschinsky : > >> Our runs show about 7-10 speedup, > Sorry, typo 7-10% speedup > > чт, 2 дек. 2021 г. в 12:01, Ivan Daschinsky : > >> Andrey, thanks! >> >> This allocator can be tested on every NUMA system. >> Our runs show about 7-10 speedup, if we use allocattor with interleaved >> strategy + -XX:+UseNUMA. >> But unfortunately our yardstick benches doesn't use offheap a lot, >> usually above one Gb. >> We trying to do more benches with real data and share them, possibly in >> meetup. >> >> AFAIK, GG lab servers are two-sockets machines, aren't they? So it is >> worth to run benches with a lot data on them, using >> allocator with interleaved strategy (you can skip specifying numa nodes, >> by default it will use all available) and use -XX:+UseNUMA jvm >> flag. >> >> >> >> чт, 2 дек. 2021 г. в 11:48, Andrey Mashenkov > >: >> >>> Ivan, >>> >>> Great job. PR looks good. >>> >>> This allocator in interleaved mode and passing `-XX:+UseNUMA` flag to jvm >>> > show promising results on yardstick benches. Technically, G1 is not a >>> numa >>> > aware collector for java versions less than 14, but allocation of heap >>> in >>> > interleaved mode shows good results even on java 11. >>> >>> Can you share benchmark results? >>> I'm not sure I'll have an Optane on my notebook in a reasonable time ;) >>> >>> >>> On Thu, Dec 2, 2021 at 10:41 AM Ivan Daschinsky >>> wrote: >>> >>> > Semyon D. and Maks T. -- thanks a lot for review. >>> > >>> > BTW, Igniters, I will appreciate all opinions and feedback. >>> > >>> > пн, 29 нояб. 2021 г. в 10:13, Ivan Daschinsky : >>> > >>> > > Hi, igniters! >>> > > >>> > > There is not a big secret that nowadays NUMA is quite common in >>> > > multiprocessor systems. >>> > > And this memory architecture should be treated in specific ways. >>> > > >>> > > Support for NUMA is present in many commercial and open-source >>> products. >>> > > >>> > > I've implemented a NUMA aware allocator for Apache Ignite [1] >>> > > It is a JNI wrapper around `libnuma` and supports different >>> allocation >>> > > options. >>> > > I.e. interleaved, local, interleved_mask and so on. For more >>> information, >>> > > see >>> > > [2], [3]. >>> > > This allocator in interleaved mode and passing `-XX:+UseNUMA` flag >>> to jvm >>> > > show promising results on yardstick benches. Technically, G1 is not a >>> > numa >>> > > aware collector for java versions less than 14, but allocation of >>> heap in >>> > > interleaved mode shows good results even on java 11. >>> > > >>> > > Currently, all needed libraries and tools for building this module >>> are >>> > > available on TC agents >>> > > setup of specific test suite is in progress [4] >>> > > >>> > > So I am asking for a review of my patch. >>> > > >>> > > [1] -- https://issues.apache.org/jira/browse/IGNITE-15922 >>> > > [2] -- https://man7.org/linux/man-pages/man3/numa.3.html >>> > > [3] -- https://man7.org/linux/man-pages/man2/mbind.2.html >>> > > [4] -- https://issues.apache.org/jira/browse/IGNITE-15994 >>> > > >>> > >>> > >>> > -- >>> > Sincerely yours, Ivan Daschinskiy >>> > >>> >>> >>> -- >>> Best regards, >>> Andrey V. Mashenkov >>> >> >> >> -- >> Sincerely yours, Ivan Daschinskiy >> > > > -- > Sincerely yours, Ivan Daschinskiy > -- Sincerely yours, Ivan Daschinskiy
Re: [DISCUSSION][IEP-80] Breaking changes in Ignite-2.x
It is the last point in the wishlist that probably nobody cares about. I see no discussion at all, no @Deprecation annotations and a lot of abandoned and broken code only. пт, 3 дек. 2021 г. в 11:01, Maxim Muzafarov : > Ivan, > > It seems shmem is already in the remove list [1]. > > [1] > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist > > On Fri, 3 Dec 2021 at 10:42, Ivan Daschinsky wrote: > > > > So if nobody wants it to resurrect, maybe it is worth removing it? > > > > > > пт, 3 дек. 2021 г. в 09:25, Ivan Pavlukhin : > > > > > Latest that I heard that it literally was never in use. > > > > > > 2021-11-29 19:44 GMT+03:00, Ivan Daschinsky : > > > > There is JNI library in ipc/shmem directory. It even compiles with > > > minimal > > > > modification on modern gcc (9.3.0) > > > > But there is no script to build jar with native library. > > > > > > > > May be it is possible to create separate module, refactor it a bit, > > > change > > > > build process (CMake)? > > > > Is there a technical reason why it is abandoned? > > > > > > > > пн, 29 нояб. 2021 г. в 19:24, Ivan Daschinsky : > > > > > > > >> Guys, what is the status of ignite-shmem and TcpCommunication > through > > > it? > > > >> > > > >> 1. Native part has not been built at all since 2015 > > > >> 2. It is currently broken in master -- I've fixed some NPE and > metrics > > > >> conversion locally and it works for me. It is broken for almost a > year. > > > >> 3. GridShmemCommunicationClient is not covered by tests. > > > >> > > > >> What do you think we should do with this? > > > >> > > > >> сб, 27 нояб. 2021 г. в 23:32, Vyacheslav Daradur < > daradu...@gmail.com>: > > > >> > > > >>> LGTM. Merged to master. > > > >>> > > > >>> Thank you for your contribution! > > > >>> > > > >>> On Fri, Nov 26, 2021 at 12:20 PM Maxim Muzafarov < > mmu...@apache.org> > > > >>> wrote: > > > >>> > > > >>> > Folks, > > > >>> > > > > >>> > This is a friendly reminder :-) > > > >>> > PR [1] is ready for review. Will anyone take a look? > > > >>> > > > > >>> > [1] https://github.com/apache/ignite/pull/9577/files > > > >>> > > > > >>> > On Sat, 20 Nov 2021 at 03:17, Maxim Muzafarov > > > > >>> wrote: > > > >>> > > > > > >>> > > Folks, > > > >>> > > > > > >>> > > > > > >>> > > I've prepared the changes related to the legacy Service Grid > > > removal > > > >>> > > for the 2.13 release. Here is the issue [1] and the PR [2] of > these > > > >>> > > changes. > > > >>> > > > > > >>> > > Some important notes: > > > >>> > > > > > >>> > > - Removed GridServiceProcessor legacy implementation (internal) > > > >>> > > - The IGNITE_EVENT_DRIVEN_SERVICE_PROCESSOR_ENABLED system > property > > > >>> > > was removed (this is a breaking change) > > > >>> > > - The ATTR_EVENT_DRIVEN_SERVICE_PROCESSOR_ENABLED node > attribute > > > was > > > >>> > > removed (this is a breaking change) > > > >>> > > - The node with 2.13 won't be able to connect to 2.12 (assume > > > Ignite > > > >>> > > services are used) > > > >>> > > > > > >>> > > The legacy test suite [3] will be removed after the [1] issue > will > > > >>> > > be > > > >>> > merged. > > > >>> > > > > > >>> > > > > > >>> > > Who can take a look at this PR? > > > >>> > > > > > >>> > > > > > >>> > > [1] https://issues.apache.org/jira/browse/IGNITE-15758 > > > >>> > > [2] https://github.com/apache/ignite/pull/9577/files > > > >>> > > [3] > > > >>> > > > > >>> > > > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ServiceGridLegacyMode > > > >>> > > > > > >>> > > On Fri, 22 Oct 2021 at 12:43, Nikolay Izhikov < > nizhi...@apache.org > > > > > > > >>> > wrote: > > > >>> > > > > > > >>> > > > Hello. > > > >>> > > > > > > >>> > > > In the scope of IEP-80 Breaking changes in Ignite-2.x PR for > the > > > >>> > following API are prepared: > > > >>> > > > > > > >>> > > > 1. MVCC > > > >>> > > > - https://issues.apache.org/jira/browse/IGNITE-15757 > > > >>> > > > - https://github.com/apache/ignite/pull/9516 > > > >>> > > > > > > >>> > > > 2. LOCAL caches > > > >>> > > > - https://issues.apache.org/jira/browse/IGNITE-15756 > > > >>> > > > - https://github.com/apache/ignite/pull/9515 > > > >>> > > > > > > >>> > > > 3. CacheConfiguration#rebalanceDelay > > > >>> > > > - https://issues.apache.org/jira/browse/IGNITE-15764 > > > >>> > > > - https://github.com/apache/ignite/pull/9515 > > > >>> > > > > > > >>> > > > Please, review. > > > >>> > > > > > > >>> > > > > 15 окт. 2021 г., в 16:23, Nikolay Izhikov > > > >>> > > > > > > >>> > > > > >>> > написал(а): > > > >>> > > > > > > > >>> > > > > THanks, Maksim. > > > >>> > > > > > > > >>> > > > > Tickets included in IEP scope and marked for d in > 2.12-2.13 > > > >>> > > > > > > > >>> > > > >> 15 окт. 2021 г., в 16:03, Maxim Muzafarov < > mmu...@apache.org> > > > >>> > написал(а): > > > >>> > > > >> > > > >>> > > > >> Let's deprecate RebalanceDelay and RebalanceMode=NONE > also. > > > >>> > > > >> > > > >>> > > > >> [1]
Re: [DISCUSSION][IEP-80] Breaking changes in Ignite-2.x
Ivan, It seems shmem is already in the remove list [1]. [1] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0+Wishlist On Fri, 3 Dec 2021 at 10:42, Ivan Daschinsky wrote: > > So if nobody wants it to resurrect, maybe it is worth removing it? > > > пт, 3 дек. 2021 г. в 09:25, Ivan Pavlukhin : > > > Latest that I heard that it literally was never in use. > > > > 2021-11-29 19:44 GMT+03:00, Ivan Daschinsky : > > > There is JNI library in ipc/shmem directory. It even compiles with > > minimal > > > modification on modern gcc (9.3.0) > > > But there is no script to build jar with native library. > > > > > > May be it is possible to create separate module, refactor it a bit, > > change > > > build process (CMake)? > > > Is there a technical reason why it is abandoned? > > > > > > пн, 29 нояб. 2021 г. в 19:24, Ivan Daschinsky : > > > > > >> Guys, what is the status of ignite-shmem and TcpCommunication through > > it? > > >> > > >> 1. Native part has not been built at all since 2015 > > >> 2. It is currently broken in master -- I've fixed some NPE and metrics > > >> conversion locally and it works for me. It is broken for almost a year. > > >> 3. GridShmemCommunicationClient is not covered by tests. > > >> > > >> What do you think we should do with this? > > >> > > >> сб, 27 нояб. 2021 г. в 23:32, Vyacheslav Daradur : > > >> > > >>> LGTM. Merged to master. > > >>> > > >>> Thank you for your contribution! > > >>> > > >>> On Fri, Nov 26, 2021 at 12:20 PM Maxim Muzafarov > > >>> wrote: > > >>> > > >>> > Folks, > > >>> > > > >>> > This is a friendly reminder :-) > > >>> > PR [1] is ready for review. Will anyone take a look? > > >>> > > > >>> > [1] https://github.com/apache/ignite/pull/9577/files > > >>> > > > >>> > On Sat, 20 Nov 2021 at 03:17, Maxim Muzafarov > > >>> wrote: > > >>> > > > > >>> > > Folks, > > >>> > > > > >>> > > > > >>> > > I've prepared the changes related to the legacy Service Grid > > removal > > >>> > > for the 2.13 release. Here is the issue [1] and the PR [2] of these > > >>> > > changes. > > >>> > > > > >>> > > Some important notes: > > >>> > > > > >>> > > - Removed GridServiceProcessor legacy implementation (internal) > > >>> > > - The IGNITE_EVENT_DRIVEN_SERVICE_PROCESSOR_ENABLED system property > > >>> > > was removed (this is a breaking change) > > >>> > > - The ATTR_EVENT_DRIVEN_SERVICE_PROCESSOR_ENABLED node attribute > > was > > >>> > > removed (this is a breaking change) > > >>> > > - The node with 2.13 won't be able to connect to 2.12 (assume > > Ignite > > >>> > > services are used) > > >>> > > > > >>> > > The legacy test suite [3] will be removed after the [1] issue will > > >>> > > be > > >>> > merged. > > >>> > > > > >>> > > > > >>> > > Who can take a look at this PR? > > >>> > > > > >>> > > > > >>> > > [1] https://issues.apache.org/jira/browse/IGNITE-15758 > > >>> > > [2] https://github.com/apache/ignite/pull/9577/files > > >>> > > [3] > > >>> > > > >>> > > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_ServiceGridLegacyMode > > >>> > > > > >>> > > On Fri, 22 Oct 2021 at 12:43, Nikolay Izhikov > > > > >>> > wrote: > > >>> > > > > > >>> > > > Hello. > > >>> > > > > > >>> > > > In the scope of IEP-80 Breaking changes in Ignite-2.x PR for the > > >>> > following API are prepared: > > >>> > > > > > >>> > > > 1. MVCC > > >>> > > > - https://issues.apache.org/jira/browse/IGNITE-15757 > > >>> > > > - https://github.com/apache/ignite/pull/9516 > > >>> > > > > > >>> > > > 2. LOCAL caches > > >>> > > > - https://issues.apache.org/jira/browse/IGNITE-15756 > > >>> > > > - https://github.com/apache/ignite/pull/9515 > > >>> > > > > > >>> > > > 3. CacheConfiguration#rebalanceDelay > > >>> > > > - https://issues.apache.org/jira/browse/IGNITE-15764 > > >>> > > > - https://github.com/apache/ignite/pull/9515 > > >>> > > > > > >>> > > > Please, review. > > >>> > > > > > >>> > > > > 15 окт. 2021 г., в 16:23, Nikolay Izhikov > > >>> > > > > > >>> > > > >>> > написал(а): > > >>> > > > > > > >>> > > > > THanks, Maksim. > > >>> > > > > > > >>> > > > > Tickets included in IEP scope and marked for d in 2.12-2.13 > > >>> > > > > > > >>> > > > >> 15 окт. 2021 г., в 16:03, Maxim Muzafarov > > >>> > написал(а): > > >>> > > > >> > > >>> > > > >> Let's deprecate RebalanceDelay and RebalanceMode=NONE also. > > >>> > > > >> > > >>> > > > >> [1] https://issues.apache.org/jira/browse/IGNITE-12662 > > >>> > > > >> [2] https://issues.apache.org/jira/browse/IGNITE-14613 > > >>> > > > >> > > >>> > > > >> On Fri, 15 Oct 2021 at 15:46, Anton Vinogradov > > > > >>> > wrote: > > >>> > > > >>> > > >>> > > > >>> +1 > > >>> > > > >>> > > >>> > > > >>> On Fri, Oct 15, 2021 at 3:41 PM Nikita Amelchev < > > >>> > namelc...@apache.org> > > >>> > > > >>> wrote: > > >>> > > > >>> > > >>> > > > +1 for deprecation in the 2.12 release > > >>> > > > > > >>> > > > пт, 15 окт. 2021 г. в 15:35, Nikolay Izhikov < > > >>> nizhi...@apache.org > >
