Hello, Ignite. Finally, I've implemented this ticket! :)
Jira - https://issues.apache.org/jira/browse/IGNITE-9113 PR - https://github.com/apache/ignite/pull/6388 Tests seems to be OK. Anyone wants to take a look? В Чт, 09/08/2018 в 16:41 +0300, Alexey Goncharuk пишет: > Once the OS gave us the pointer, no exceptions can be raised in the user > code. If the OS overcommitted the memory, then the process will be halted > when OOME occurs, not much we can do here. > > My point was that dynamic data region allocation can lead to another > dynamic exception that should be properly handled during cache start. When > the data region is allocated statically, this exception is handled on node > start, which is much easier. > > ср, 8 авг. 2018 г. в 18:18, Dmitriy Pavlov <[email protected]>: > > > I used to think that OS allocates pages not immediately after call to > > allocate(), but only during first touch of each page. > > > > I'm not sure every OS provides guaranee that 'allocated' memory will never > > cause OOME. Please correct me if I'm wrong. > > > > ср, 8 авг. 2018 г. в 17:38, Dmitriy Setrakyan <[email protected]>: > > > > > Alexey, > > > > > > I am not sure I understand. If cache creation proceeds, but memory region > > > was not allocated, how can the cache operate? > > > > > > D. > > > > > > On Wed, Aug 8, 2018 at 8:05 AM, Alexey Goncharuk < > > > [email protected] > > > > wrote: > > > > I do not mind making this change, but note that the reason for non-lazy > > > > region allocation is the need to gracefully handle OOME errors during > > > > > > cache > > > > start. When a region is pre-allocated, no OOME can happen. > > > > > > > > If a region is allocated dynamically, then all errors that happened > > > > > > during > > > > the node start before should be properly handled (a client node must be > > > > stopped, but cache creation should proceed). > > > > > > > > пт, 27 июл. 2018 г. в 20:04, Valentin Kulichenko < > > > > [email protected]>: > > > > > > > > > Ticket created: https://issues.apache.org/jira/browse/IGNITE-9113 > > > > > > > > > > -Val > > > > > > > > > > On Fri, Jul 27, 2018 at 5:59 AM Dmitry Pavlov <[email protected] > > > > > wrote: > > > > > > > > > > > Maxim, thank you. > > > > > > > > > > > > If it seems it is technically possible, we can file ticket for this > > > > > > > > > > change. > > > > > > > > > > > > I find this proposal reasonable, change makes perfectly sense to > > > > me. > > > > > > > > > > > > We can wait Alex G. feedback on this change before starting actual > > > > > > implementation. It can take for a while, because he is travelling > > > > > > now. > > > > > > > > > > > > пт, 27 июл. 2018 г. в 14:35, Maxim Muzafarov <[email protected]>: > > > > > > > > > > > > > Guys, > > > > > > > > > > > > > > I can miss some details, but at the first glance we have > > > > everething > > > > we > > > > > > need > > > > > > > to defer > > > > > > > region memory allocation if it has no cache groups assignments. > > > > And > > > > it > > > > > > > doesn't matter > > > > > > > where it happens on client or server nodes. > > > > > > > > > > > > > > Currently region memory allocation happens at exchange future > > > > init > > > > > > method. > > > > > > > At the > > > > > > > node startup method initCachesOnLocalJoin executes. This method > > > > > > > > > > > > resposnible > > > > > > > for > > > > > > > memory allocation (through initiating cache managers) and it also > > > > > > > > > > starts > > > > > > > caches. > > > > > > > So, at this point we have all existing caches descriptors and can > > > > > > > > find > > > > > > out > > > > > > > which > > > > > > > cache matches which region to defer some regions initialization > > > > to > > > > the > > > > > > > moment when > > > > > > > newly cache assings to this region (happend at > > > > > > onCacheChangeRequest). > > > > > > > > > > > > > > Please, сorrect me if I'm wrong and missing something. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 25 Jul 2018 at 19:32 Dmitry Pavlov < > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > Hi Maxim, > > > > > > > > > > > > > > > > thank you for stepping in. How do you think, is it possible to > > > > > > > > check > > > > > > > cache > > > > > > > > assignment to region at stage of memory allocation? > > > > > > > > > > > > > > > > Sincerely, > > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > > > ср, 25 июл. 2018 г. в 18:22, Maxim Muzafarov < > > > > [email protected] > > > > : > > > > > > > > > > > > > > > > > Folks, > > > > > > > > > > > > > > > > > > I've checked memory allocation. It looks like we are > > > > allocating > > > > > > memory > > > > > > > > only > > > > > > > > > on the first exchange future init on local join occurs on > > > > node. > > > > > Also, > > > > > > > > seems > > > > > > > > > like we are allocating only the first chunk of memory (not > > > > the > > > > > whole > > > > > > > > bunch) > > > > > > > > > and it calculates as: > > > > > > > > > > > > > > > > > > Math.max((maxSize - startSize) / (SEG_CNT - 1), 256L * 1024 * > > > > > > > > 1024) > > > > > > > > > > > > > > > > > > But, I'm agree with Val. It's better to allocate memory only > > > > > > when > > > > > > when > > > > > > > > > the first cache assigned to this region. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also, It seems like we have some problem with user > > > > notification > > > > > about > > > > > > > > > available > > > > > > > > > physical resources. For client nodes method requiredOffheap() > > > > > > > > > > returns > > > > > > > > > always > > > > > > > > > zero [1]. That's why WARN message shown here [2] would be not > > > > > > not > > > > > > quite > > > > > > > > > right > > > > > > > > > if we have a lot of client nodes in cluster. > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/ignite/blob/master/modules/ > > > > > > > > core/src/main/java/org/apache/ignite/internal/managers/discovery/ > > > > GridDiscoveryManager.java#L1501 > > > > > > > > > [2] > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/ignite/blob/master/modules/ > > > > > > > > core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L1489 > > > > > > > > > > > > > > > > > > сб, 21 июл. 2018 г. в 14:15, Dmitriy Setrakyan < > > > > > > > > > > > > [email protected] > > > > > > > > : > > > > > > > > > > > > > > > > > > > On Sat, Jul 21, 2018 at 5:22 AM, Valentin Kulichenko < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > Actually, I would go even further: only allocate a data > > > > > > > > region > > > > > > on a > > > > > > > > > node > > > > > > > > > > > when the first cache assigned to this region is deployed > > > > on > > > > > that > > > > > > > > node. > > > > > > > > > > > Because issue is broader than client nodes and local > > > > > > caches. > > > > > One > > > > > > > can > > > > > > > > > have > > > > > > > > > > > server nodes without any caches as well - running only > > > > > > > > > > services, > > > > > > > for > > > > > > > > > > > example. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > It would be great if this was possible, but to my > > > > knowledge, > > > > > > regions > > > > > > > > need > > > > > > > > > > to be allocated on startup. > > > > > > > > > > > > > > > > > > > > Alexey Goncharuk, do you have any suggestions on this? > > > > > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > -- > > > > > > > > > Maxim Muzafarov > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -- > > > > > > > Maxim Muzafarov > > > > > > >
signature.asc
Description: This is a digitally signed message part
