Re: [IGNITE-5717] improvements of MemoryPolicy default size

I'm ok with using 1GB only on a 32-bit system. Note, though, that there is no reliable way to detect this, so this will be a best-effort change. 2017-10-31 14:07 GMT+03:00 Dmitry Pavlov : > Hi Igniters, > > Up this discussion: there is still some thing to do in context of

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Hi Igniters, Up this discussion: there is still some thing to do in context of 32 bit VMs, see issue https://issues.apache.org/jira/browse/IGNITE-5618 Denis M. suggested solution for this: let's make the default size calculation more sophisticated. If Ignite is running in a 32 process and 20%

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Folks, I filed a ticket [1] to address the concern of starting too many nodes. Please review it. [1] https://issues.apache.org/jira/browse/IGNITE-6003 Thanks, Sergey. On Mon, Aug 7, 2017 at 12:53 PM, Sergey Chugunov wrote: > Dmitriy, > > When Ignite node "allocates

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Dmitriy, When Ignite node "allocates memory" it actually just reserves a chunk in its address space, almost no physical RAM is used. I can easily start half a dozen of ignite nodes with current defaults on my laptop with only 16 Gigs of RAM; and each node will "allocate" around 12 Gigs; 72

Re: [IGNITE-5717] improvements of MemoryPolicy default size

But why? We allocate the memory, so we should know when it runs out. What am i missing? ⁣D.​ On Aug 4, 2017, 11:55 AM, at 11:55 AM, Sergey Chugunov wrote: >I used GC and java only as an example, they are not applicable to >Ignite >case where we manage offheap

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Hang on. I thought we were talking about offheap size, GC should not be relevant. Am I wrong? ⁣D.​ On Aug 4, 2017, 11:38 AM, at 11:38 AM, Sergey Chugunov wrote: >Do you see an obvious way of implementing it? > >In java there is a heap and GC working on it. And for

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Do you see an obvious way of implementing it? In java there is a heap and GC working on it. And for instance, it is possible to make a decision to throw an OOM based on some gc metrics. I may be wrong but I don't see a mechanism in Ignite to use it right away for such purposes. And implementing

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Without #3, the #1 and #2 make little sense. Why is #3 so difficult? ⁣D.​ On Aug 4, 2017, 10:46 AM, at 10:46 AM, Sergey Chugunov wrote: >Dmitriy, > >Last item makes perfect sense to me, one may think of it as an >"OutOfMemoryException" in java. >However, it looks

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Dmitriy, Last item makes perfect sense to me, one may think of it as an "OutOfMemoryException" in java. However, it looks like such feature requires considerable efforts to properly design and implement it, so I would propose to create a separate ticket and agree upon target version for it.

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Here is what we should do: 1. Pick an acceptable number. Does not matter if it is 10% or 50%. 2. Print the allocated memory in *BOLD* letters into the log. 3. Make sure that Ignite server never hangs due to the low memory issue. We should sense it and kick the node out automatically,

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Vladimir, Dmitriy P., Please see inline > On Aug 2, 2017, at 7:20 AM, Vladimir Ozerov wrote: > > Denis, > > The reason is that product should not hang user's computer. How else this > could be explained? I am developer. I start Ignite, 1 node, 2 nodes, X > nodes, observe

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Denis, The reason is that product should not hang user's computer. How else this could be explained? I am developer. I start Ignite, 1 node, 2 nodes, X nodes, observe how they join topology. Add one key, 10 keys, 1M keys. Then I do a bug in example and load 100M keys accidentally - restart the

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Hi Igniters, When I was Ignite user before installing product on production server I’ve always used this page https://apacheignite.readme.io/docs/jvm-and-system-tuning for selecting appropriate parameters. But before go live, I’ve used Ignition.start() 20 times per day to check if my changes in

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Sergey, That’s expectable because as we revealed from this discussion the allocation works different depending on whether the persistence is used or not: 1) In-memory mode (the persistence is disabled) - the space will be allocated incrementally until the max threshold is reached. Good! 2)

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Denis, Just a simple example from our own codebase: I tried to execute PersistentStoreExample with default settings and two server nodes and client node got frozen even on initial load of data into the grid. Although with one server node the example finishes pretty quickly. And my laptop isn't

Re: [IGNITE-5717] improvements of MemoryPolicy default size

> As far as allocating 80% of available RAM - I was against this even for > In-memory mode and still think that this is a wrong default. Looking at > free RAM is even worse because it gives you undefined behavior. Guys, I can not understand how this dynamic memory allocation's high-level

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Free RAM is variable. For example, I can start Ignite before IDE (Eclipse, or so), or after IDE. In the first case it will hang, in the send it won't. Unstable. To be clear - I propose to have "maxMemory" set to 10%, not "initialMemory". It doesn't matter how exactly we reach it - in one hop, or

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Dmitriy, The reason behind this is the need to to be able to evict and load pages to disk, thus we need to preserve a PageId->Pointer mapping in memory. In order to do this in the most efficient way, we need to know in advance all the address ranges we work with. We can add dynamic memory

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Dima, Probably folks who worked closely with storage know why. However, even increments will not help us. Most developers work on laptops. Common laptop has 8-16 Gb RAM today. OS and working programs typically consume about a half. So, for example, I have 8Gb total, 4Gb consumed, 4Gb free. And

Re: [IGNITE-5717] improvements of MemoryPolicy default size

On Wed, Aug 2, 2017 at 7:27 AM, Vladimir Ozerov wrote: > Please see original Sergey's message - when persistence is enabled, memory > is not allocated incrementally, maxSize is used. > Why? > Default settings must allow for normal work on developer's environment. >

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Please see original Sergey's message - when persistence is enabled, memory is not allocated incrementally, maxSize is used. But this is a problem even for in-memory case. Consider that I wanted to load several gigs of data as an experiment on my laptop - frequent scenario. First several seconds

Re: [IGNITE-5717] improvements of MemoryPolicy default size

> Why not allocate in increments automatically? This is exactly how the allocation works right now. The memory will grow incrementally until the max size is reached (80% of RAM by default). — Denis > On Aug 1, 2017, at 3:03 PM, dsetrak...@apache.org wrote: > > Vova, 1GB seems a bit too small

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Vova, 1GB seems a bit too small for me, and frankly i do not want t o guess. Why not allocate in increments automatically? ⁣D.​ On Aug 1, 2017, 11:03 PM, at 11:03 PM, Vladimir Ozerov wrote: >Denis, >No doubts you haven't heard about it - AI 2.1 with persistence, when

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Vladimir, 80% allocation approach was implemented in AI 2.0. I personally explained users on the forums its behavior. Actually, dynamic vs static memory allocation scenarios were analyzed and discussed long time ago and eventually we decided to go for with the former:

Re: [IGNITE-5717] improvements of MemoryPolicy default size

Denis, No doubts you haven't heard about it - AI 2.1 with persistence, when 80% of RAM is allocated right away, was released several days ago. How do you think, how many users tried it already? Guys, Do you really think allocating 80% of available RAM is a normal thing? Take your laptop and check

Re: [IGNITE-5717] improvements of MemoryPolicy default size

My vote goes for option #1 too. I don’t think that 80% is too aggressive to bring it down. IGNITE-5717 was created to fix the issue of the 80% RAM allocation on 64 bit systems when Ignite works on top of 32 bit JVM. I’ve not heard of any other complaints in regards the default allocation size.

Re: [IGNITE-5717] improvements of MemoryPolicy default size

+1 to Sergey, I think OOME may confuse user and and may spoil the user's impression at first check of Apache Ignite. вт, 1 авг. 2017 г. в 12:36, Vladimir Ozerov : > +1 > > 80% of RAM is way too aggressive. With this value virtually every user will > have problems with OOME

Re: [IGNITE-5717] improvements of MemoryPolicy default size

+1 80% of RAM is way too aggressive. With this value virtually every user will have problems with OOME or excessive swapping during development. I would set maxMemory to some relatively small value, may be even less that 1Gb to let Ignite run smoothly on developer's laptops, and print a