Igor, can you please describe the algorithm with all the thresholds? On Mon, Aug 28, 2017 at 4:56 AM, Seliverstov Igor <gvvinbl...@gmail.com> wrote:
> The suggestion here is based on initial settings, and it's so because there > is no other nodes on the host in the example. > > The algorithm tries to preserve the original ratio of memory policies > keeping numbers reasonable (for example after some thresshold it will > suggest not to use several memory policies if there is not enough of memory > for all of them) and taking into consideration nodes count on the host, > each jvm heap, needed memory for OS, etc > > 2017-08-28 14:38 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>: > > > Looks good, but why in the example provided are we suggesting 8GB? 2 > nodes > > with 8GB will completely exhaust the available memory. I would suggest 6 > or > > 7GB. > > > > Also, why 100MB for default policy. Anything under 1GB seems too small. > > > > Can you please comment? > > > > D. > > > > On Mon, Aug 28, 2017 at 3:31 AM, Seliverstov Igor <gvvinbl...@gmail.com> > > wrote: > > > > > One more example of possible warning: > > > > > > ----------------------------------------------------- > > > Excessive memory usage by Ignite node process (performance may drop) > > > [requested=44613MB, available=15942MB]. > > > > > > Please tune the folowing settings as suggested: > > > MemoryPolicyConfiguration.initialSize for bigPlc: 8102MB > > > MemoryPolicyConfiguration.maxSize for bigPlc: 8102MB > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 100MB > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 100MB > > > > > > Current settings: > > > Java Heap maxSize: 3543MB > > > Java Heap initSize: 250MB > > > MemoryPolicyConfiguration.initialSize for bigPlc: 256MB > > > MemoryPolicyConfiguration.maxSize for bigPlc: 40960MB > > > MemoryPolicyConfiguration.initialSize for dfltPlc: 10MB > > > MemoryPolicyConfiguration.maxSize for dfltPlc: 10MB > > > The overall expected memory usage by all Ignite nodes on the host: > > > 44613MB > > > ----------------------------------------------------- > > > > > > Your thoughts? > > > > > > 2017-08-28 5:06 GMT+03:00 Denis Magda <dma...@apache.org>: > > > > > > > Guys, > > > > > > > > ML lib profile is missing in 2.1 release! That must be fixed and > rolled > > > > out in this emergency release: > > > > https://issues.apache.org/jira/browse/IGNITE-6193 < > > > > https://issues.apache.org/jira/browse/IGNITE-6193> > > > > > > > > Oleg, Yuri, please step in and handle the issue. > > > > > > > > BTW, who is considered to be the release manager of this release? > > > > > > > > — > > > > Denis > > > > > > > > > On Aug 25, 2017, at 2:29 PM, Dmitriy Setrakyan < > > dsetrak...@apache.org> > > > > wrote: > > > > > > > > > > I like the format proposed by Denis, very clear. > > > > > > > > > > However, I also do not understand why a user should change the size > > of > > > > some > > > > > system cache. How would a user ever know what value to put there? > > This > > > > > value should be configured by Ignite automatically. > > > > > > > > > > D. > > > > > > > > > > On Fri, Aug 25, 2017 at 2:24 PM, Denis Magda <dma...@apache.org> > > > wrote: > > > > > > > > > >> Igor, > > > > >> > > > > >> Let me suggest this format. > > > > >> > > > > >> --------------------------------------------- > > > > >> Excessive memory usage by Ignite node process (performance may > drop) > > > > >> [requested=29251MB, available=15942MB] > > > > >> > > > > >> Please tune the following settings: > > > > >> [MemoryConfiguration.defaultMemoryPolicySize = suggested value] > > > > >> MemoryConfiguration.systemCacheMaxSize = suggested value > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_1} = > suggested > > > > >> value] > > > > >> [MemoryPolicyConfiguration.maxSize for {policy_name_2} = > suggested > > > > >> value] > > > > >> > > > > >> Current settings: > > > > >> [DefaultMemoryPolicySize = value] > > > > >> [{policy_name_1} size = value] > > > > >> [{policy_name_1} size = value] > > > > >> SystemCacheInitialSize = value > > > > >> SystemCacheMaxSize = value > > > > >> Java Heap Init Size = value > > > > >> Java Heap Max Size = value > > > > >> > > > > >> The overall memory usage by all Ignite nodes on the host: value > > > > >> ------------------------------------------- > > > > >> > > > > >> Records in […] are optional. If custom memory policy is not set or > > the > > > > >> default memory policy is overridden the output will miss some of > the > > > > rows. > > > > >> > > > > >> As for systemCacheMaxSize, it should be show ONLY if the parameter > > was > > > > set > > > > >> explicitly by user code. Otherwise, the platform should be wise > > enough > > > > to > > > > >> instantiate it properly depending on the host memory usage. > > > > >> > > > > >> — > > > > >> Denis > > > > >> > > > > >>> On Aug 25, 2017, at 1:49 PM, Seliverstov Igor < > > gvvinbl...@gmail.com> > > > > >> wrote: > > > > >>> > > > > >>> The message without logging layout: > > > > >>> > > > > >>> Not enough memory for current process [required=29251MB, > > > > >> available=15942MB]. > > > > >>> Please change MemoryConfiguration.systemCacheMaxSize and > > > > >>> MemoryConfiguration.defaultMemoryPolicySize to decrease memory > > > > allocated > > > > >>> for each node. > > > > >>> > > > > >>> Current settings: > > > > >>> HeapInit=250MB > > > > >>> HeapMax=3543MB > > > > >>> DefaultMemoryPolicySize=12753MB > > > > >>> SystemCacheInitialSize=40MB > > > > >>> SystemCacheMaxSize=100MB > > > > >>> > > > > >>> Other ignite instanses on the server require: 12853MB > > > > >>> > > > > >>> I think it's make sense to describe what these numbers are > consist > > > of. > > > > >>> > > > > >>> We simple say which parameters have an impact on how much memory > > the > > > > >>> instance needs and their (parameters) actual values. > > > > >>> > > > > >>> Also we notice that more than one Ignite instance are ran on the > > > server > > > > >> or > > > > >>> workstation and it also consumes memory. > > > > >>> > > > > >>> 25 авг. 2017 г. 21:30 пользователь "Dmitriy Setrakyan" < > > > > >>> dsetrak...@apache.org> написал: > > > > >>> > > > > >>>> Igor, what is this flood of WARN messaging coming after the > text? > > > Are > > > > we > > > > >>>> really going to print this whole thing out? > > > > >>>> > > > > >>>> On Fri, Aug 25, 2017 at 9:49 AM, Seliverstov Igor < > > > > gvvinbl...@gmail.com > > > > >>> > > > > >>>> wrote: > > > > >>>> > > > > >>>>> This message appears on topology change in case the available > > > memory > > > > is > > > > >>>>> exceeded > > > > >>>>> > > > > >>>>> 2017-08-25 19:47 GMT+03:00 Seliverstov Igor < > > gvvinbl...@gmail.com > > > >: > > > > >>>>> > > > > >>>>>> An example of current impl: > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Not enough > memory > > > for > > > > >>>>>> current process [required=29251MB, available=15942MB]. > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Please change > > > > >>>>>> MemoryConfiguration.systemCacheMaxSize and > MemoryConfiguration. > > > > >>>>> defaultMemoryPolicySize > > > > >>>>>> to decrease memory allocated for each node. > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Current > settings: > > > > >>>>>> [2017-08-25 19:44:37,740][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > HeapInit=250MB > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > HeapMax=3543MB > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > >>>> DefaultMemoryPolicySize= > > > > >>>>>> 12753MB > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > >>>>>> SystemCacheInitialSize=40MB > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > >>>> SystemCacheMaxSize=100MB > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] Other ignite > > > > instanses > > > > >>>> on > > > > >>>>>> the server require: 12853MB > > > > >>>>>> [2017-08-25 19:44:37,741][WARN ][disco-event-worker-#29% > > internal. > > > > >>>>>> GridHomePathSelfTest0%][GridDiscoveryManager] > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> 2017-08-25 17:40 GMT+03:00 Sergey Kozlov < > skoz...@gridgain.com > > >: > > > > >>>>>> > > > > >>>>>>> I suppose we should not forget JVM heap size and suggest to > > > reduce > > > > >>>> both > > > > >>>>>>> options > > > > >>>>>>> > > > > >>>>>>> On Fri, Aug 25, 2017 at 5:24 PM, Dmitriy Setrakyan < > > > > >>>>> dsetrak...@apache.org > > > > >>>>>>>> > > > > >>>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> Igor, I would change the message. How about this: > > > > >>>>>>>> > > > > >>>>>>>> Required RAM size is larger than total physical memory > > available > > > > for > > > > >>>>> OS. > > > > >>>>>>>>> Please change MemoryConfiguration.WhichProperty and > > > > >>>>>>>>> MemoryPolicyConfiguration.WhichProperty to decrease memory > > > > >>>>> allocated > > > > >>>>>>> for > > > > >>>>>>>>> each node. > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> Also, can we calculate what the memory size allocated for > each > > > > node > > > > >>>>>>> should > > > > >>>>>>>> be? In that case we should suggest it. > > > > >>>>>>>> > > > > >>>>>>>> D. > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> On Fri, Aug 25, 2017 at 7:20 AM, Seliverstov Igor < > > > > >>>>> gvvinbl...@gmail.com > > > > >>>>>>>> > > > > >>>>>>>> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> What do you, guys think about next warning? > > > > >>>>>>>>> > > > > >>>>>>>>> [2017-08-25 17:17:04,718][INFO > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > GridHomePat > > > > >>>>>>> hSelfTest0] > > > > >>>>>>>>> System cache's MemoryPolicy size is configured to 40 MB. > Use > > > > >>>>>>>>> MemoryConfiguration.systemCacheMemorySize property to > change > > > the > > > > >>>>>>>> setting. > > > > >>>>>>>>> [2017-08-25 17:17:04,718][WARN > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > GridHomePat > > > > >>>>>>> hSelfTest0] > > > > >>>>>>>>> > > > > >>>>>>>>>>>> Required RAM size is larger than total physical memory > > > > >>>> available > > > > >>>>>>> for > > > > >>>>>>>>> OS. > > > > >>>>>>>>>>>> Check your configuration to avoid swap partition usage. > > > > >>>>>>>>>>>> Use MemoryConfiguration and MemoryPolicyConfiguration to > > > > >>>> change > > > > >>>>>>> the > > > > >>>>>>>>> settings. > > > > >>>>>>>>>>>> Physical memory [required=16397MB, available=15942MB] > > > > >>>>>>>>> > > > > >>>>>>>>> [2017-08-25 17:17:04,726][WARN > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > GridHomePat > > > > >>>>>>> hSelfTest0] > > > > >>>>>>>>> Peer class loading is enabled (disable it in production for > > > > >>>>>>> performance > > > > >>>>>>>> and > > > > >>>>>>>>> deployment consistency reasons) > > > > >>>>>>>>> [2017-08-25 17:17:04,726][INFO > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > GridHomePat > > > > >>>>>>> hSelfTest0] > > > > >>>>>>>>> Configured caches [in 'sysMemPlc' memoryPolicy: > > > > >>>>> ['ignite-sys-cache']] > > > > >>>>>>>>> [2017-08-25 17:17:04,731][INFO > > > > >>>>>>>>> ][test-runner-#1%internal.GridHomePathSelfTest%][ > GridHomePat > > > > >>>>>>> hSelfTest0] > > > > >>>>>>>>> 3-rd party licenses can be found at: > > > > >>>>>>>>> /home/gvvinblade/projects/ignite/incubator-ignite/libs/ > > > licenses > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>>> 2017-08-25 13:26 GMT+03:00 Yakov Zhdanov < > > yzhda...@apache.org > > > >: > > > > >>>>>>>>> > > > > >>>>>>>>>> Agree, let's release new version including tickets > mentioned > > > by > > > > >>>>>>> Denis > > > > >>>>>>>> and > > > > >>>>>>>>>> Ivan. > > > > >>>>>>>>>> > > > > >>>>>>>>>> --Yakov > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> -- > > > > >>>>>>> Sergey Kozlov > > > > >>>>>>> GridGain Systems > > > > >>>>>>> www.gridgain.com > > > > >>>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>> > > > > >> > > > > >> > > > > > > > > > > > > > >
