> If this flag is true will Ignite proactively allocate and use all pages in a data region, rather than incrementally?
LazyMemoryAllocation means whether memory for DataRegion will be allocated only when the first cache is created in that region (when true), or immediately (when false) https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/DataRegionConfiguration.html#isLazyMemoryAllocation-- On Wed, Jul 19, 2023 at 12:50 PM Raymond Wilson <raymond_wil...@trimble.com> wrote: > Just FYI, we have held off any memory pressure changes in the meantime > while we continue to investigate the memory issues we have. > > On Tue, 18 Jul 2023 at 9:07 AM, Raymond Wilson <raymond_wil...@trimble.com> > wrote: > >> Hi Pavel, >> >> This area is confusing. There is no indication that the memory pressure >> applies to any individual object or allocation, so there is clearly no >> association between memory pressure and any particular resource. >> >> I get your argument that .Net can 'see' allocated memory. What is unclear >> is whether it cares about actually allocated and used pages, or committed >> pages. >> >> I see there is a LazyMemoryAllocation (default: true) for data regions. >> Some data regions set this to false, eg: >> >> ^-- sysMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> >> The documentation is not clear on the effect of this flag other than to >> say it is for 'Lazy memory allocation'. If this flag is true will Ignite >> proactively allocate and use all pages in a data region, rather than >> incrementally? >> >> Thanks, >> Raymond. >> >> >> On Tue, Jul 11, 2023 at 10:55 PM Pavel Tupitsyn <ptupit...@apache.org> >> wrote: >> >>> > I can’t see another way of letting . Net know that it can’t have >>> access to all the ‘free’ memory in the process >>> >>> You don't need to tell .NET how much memory is currently available. It >>> is the job of the OS. .NET can "see" the size of the unmanaged heap. >>> >>> To quote another explanation [1]: >>> >>> > The point of AddMemoryPressure is to tell the garbage collector that >>> there's a large amount of memory allocated with that object. >>> > If it's unmanaged, the garbage collector doesn't know about it; only >>> the managed portion. >>> > Since the managed portion is relatively small, the GC may let it pass >>> for garbage collection several times, essentially wasting memory that might >>> need to be freed. >>> >>> I really don't think AddMemoryPressure is the right thing to do in your >>> case. >>> If you run into OOM issues, then look into Ignite memory region settings >>> [2] and/or adjust application memory usage on the .NET side, so that the >>> sum of those is not bigger than available RAM. >>> >>> [1] >>> https://stackoverflow.com/questions/1149181/what-is-the-point-of-using-gc-addmemorypressure-with-an-unmanaged-resource >>> [2] >>> https://ignite.apache.org/docs/latest/memory-configuration/data-regions#configuring-default-data-region >>> >>> On Tue, Jul 11, 2023 at 11:48 AM Raymond Wilson < >>> raymond_wil...@trimble.com> wrote: >>> >>>> How do Ignite .Net server nodes manage this memory issue in other >>>> projects? >>>> >>>> On Tue, Jul 11, 2023 at 5:32 PM Raymond Wilson < >>>> raymond_wil...@trimble.com> wrote: >>>> >>>>> Oops, commutes => committed >>>>> >>>>> On Tue, 11 Jul 2023 at 4:34 PM, Raymond Wilson < >>>>> raymond_wil...@trimble.com> wrote: >>>>> >>>>>> I can’t see another way of letting . Net know that it can’t have >>>>>> access to all the ‘free’ memory in the process when a large slab of that >>>>>> is >>>>>> spoken for in terms of memory commutes to Ignite data regions. >>>>>> >>>>>> In the current setup, as time goes on and Ignite progressively fills >>>>>> the allocated cache ram then system behaviour changes and can result in >>>>>> out >>>>>> of memory issues. I think I would prefer consistent system behaviour wrt >>>>>> to >>>>>> allocated resources from the start. >>>>>> >>>>>> Raymond. >>>>>> >>>>>> On Tue, 11 Jul 2023 at 3:57 PM, Pavel Tupitsyn <ptupit...@apache.org> >>>>>> wrote: >>>>>> >>>>>>> Are you sure this is necessary? >>>>>>> >>>>>>> GC.AddMemoryPressure documentation [1] states that this will >>>>>>> "improve performance only for types that exclusively depend on >>>>>>> finalizers". >>>>>>> >>>>>>> [1] >>>>>>> https://learn.microsoft.com/en-us/dotnet/api/system.gc.addmemorypressure?view=net-7.0 >>>>>>> >>>>>>> On Tue, Jul 11, 2023 at 1:02 AM Raymond Wilson < >>>>>>> raymond_wil...@trimble.com> wrote: >>>>>>> >>>>>>>> I'm making changes to add memory pressure to the GC to take into >>>>>>>> account memory committed to the Ignite data regions as this will be >>>>>>>> unmanaged memory allocations from the perspective of the GC. >>>>>>>> >>>>>>>> I don't call seeing anything related to this for .Net clients in >>>>>>>> the documentation. Are you aware of any? >>>>>>>> >>>>>>>> Raymond. >>>>>>>> >>>>>>>> On Mon, Jul 10, 2023 at 9:41 PM Raymond Wilson < >>>>>>>> raymond_wil...@trimble.com> wrote: >>>>>>>> >>>>>>>>> Thanks Pavel, this makes sense. >>>>>>>>> >>>>>>>>> Querying the .Net Process instance shows this as the difference >>>>>>>>> between PagesMemorySize (includes committed) versus WorkingSet >>>>>>>>> (includes >>>>>>>>> uses/written to) size. >>>>>>>>> Raymond. >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> <http://www.trimble.com/> >>>>>>>> Raymond Wilson >>>>>>>> Trimble Distinguished Engineer, Civil Construction Software (CCS) >>>>>>>> 11 Birmingham Drive | >>>>>>>> <https://www.google.com/maps/search/11+Birmingham+Drive%C2%A0%7C%C2%A0+Christchurch,+New+Zealand?entry=gmail&source=g>Christchurch, >>>>>>>> New Zealand >>>>>>>> <https://www.google.com/maps/search/11+Birmingham+Drive%C2%A0%7C%C2%A0+Christchurch,+New+Zealand?entry=gmail&source=g> >>>>>>>> raymond_wil...@trimble.com >>>>>>>> >>>>>>>> >>>>>>>> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >>>>>>>> >>>>>>> -- >>>>>> <http://www.trimble.com/> >>>>>> Raymond Wilson >>>>>> Trimble Distinguished Engineer, Civil Construction Software (CCS) >>>>>> 11 Birmingham Drive | Christchurch, New Zealand >>>>>> raymond_wil...@trimble.com >>>>>> >>>>>> >>>>>> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >>>>>> >>>>> -- >>>>> <http://www.trimble.com/> >>>>> Raymond Wilson >>>>> Trimble Distinguished Engineer, Civil Construction Software (CCS) >>>>> 11 Birmingham Drive | Christchurch, New Zealand >>>>> raymond_wil...@trimble.com >>>>> >>>>> >>>>> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >>>>> >>>> >>>> >>>> -- >>>> <http://www.trimble.com/> >>>> Raymond Wilson >>>> Trimble Distinguished Engineer, Civil Construction Software (CCS) >>>> 11 Birmingham Drive | Christchurch, New Zealand >>>> raymond_wil...@trimble.com >>>> >>>> >>>> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >>>> >>> >> >> -- >> <http://www.trimble.com/> >> Raymond Wilson >> Trimble Distinguished Engineer, Civil Construction Software (CCS) >> 11 Birmingham Drive | Christchurch, New Zealand >> raymond_wil...@trimble.com >> >> >> <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >> > -- > <http://www.trimble.com/> > Raymond Wilson > Trimble Distinguished Engineer, Civil Construction Software (CCS) > 11 Birmingham Drive | Christchurch, New Zealand > raymond_wil...@trimble.com > > > <https://worksos.trimble.com/?utm_source=Trimble&utm_medium=emailsign&utm_campaign=Launch> >
