TL;DR: It depends on the kernels configuration and the amount of swap space.
Longer explanation: When a malloc() is granted by the kernel, it allocates the space virtually. It doesn't yet assign physical DRAM space to the allocation because it speculates that you are not going to use all the space up front, and thus allocating it a bit at a time is more efficient. Whenever you access a new portion of the virtual space, a kernel trap is triggered and the kernel allocates physical space for the given portion and resumes the program. Linux kernels can overcommit memory. This means the kernel can hand out memory it doesn't have guaranteed available. It does so in the hope that when you actually need the space, some other program has completed and then it can use that space. Another way to get more memory is by using swap space as additional memory, albeit at a slower pace. What likely happens on peterGo's machine is the following: The extra 2.2gb memory is allocated virtually, but as you walk through that memory space getting kernel trapped all the time, the garbage collection completes and advices the kernel there is a large space of memory which isn't needed anymore. So when it finally comes along and needs space because it is about to run out, a newly freed up portion is available. The alternative hypothesis is that peterGo has ample amounts of swap space available and uses this partially. Ubuntu doesn't overcommit by default, as already stated in this thread. Solaris/Illumos can't overcommit memory at all for instance. Note that the scenario is likely to be brittle: depending on timing, your program may either run much slower than envisioned, or not even run at all when you are close to the boundary of available memory in a machine. The kernel will also have to find the space somewhere, usually by erasing the disk cache to make up for the demand of memory pressure. On Wed, May 3, 2017 at 5:23 PM Serhat Sevki Dincer <jfcga...@gmail.com> wrote: > Ok if GOGC is the trigger point, then out-of-memory error should be > consistent on all Ubuntu 64-bit 4gb systems, right? > If so, why cant peterGo reproduce the crash? > > 2 May 2017 16:08 tarihinde "Юрий Соколов" <funny.fal...@gmail.com> yazdı: > > There is nowhere told that GOGC defines border that could not be reached. >> GOGC defines proportion that triggers GC, ie GC is triggered AFTER GOGC >> proportion crossed, not before. >> >> In fact, if you allocates by small portions, then GC tries to predict >> when you will reach this border, and tries to start earlier. But it doesn't >> stop your program if you run faster and cross this border, cause current >> Golang GC us concurrent and tries to minimize GC pause. How could it >> minimize GC pause if it ought to stop whole your program to not cross GOGC >> proportion? >> >> And you partially right: GC of old Go's versions may stop program before >> GOGC proportion crossed. So, if you try Go 1.0 most likely your programm >> will run. >> >> But new behaviour is much better. >> >> The point: you should know your instrument and environment. No one GC >> enabled runtime will be happy if you allocate huge arrays (perl/python/php >> are exceptions, cause they use reference counting). If you ought to >> allocate huge arrays in GC runtime, use off-heap allocation (Java men learn >> that hard way). >> >> For example, you may use mmap to allocate huge array. >> >> Or, I'll repeat my self, use datastructure without huge continuous >> allocation (ie slice of slices for your example). Then GC will have a >> chance to trigger ealier and free some memory before it exhausted. >> >> 2 мая 2017 г. 1:59 PM пользователь "Serhat Sevki Dincer" < >> jfcga...@gmail.com> написал: >> >> The allocation request (make) is made to the runtime which covers GC, >> right? GC percent is also set to 10%. >> After 1st call returns, my app has about 2gb ram allocated. When 2nd call >> requests allocation, runtime cannot: >> - first allocate another 2gb >> - free the 1st buffer later >> due to the definition of the GOGC (GC percent). >> I think this is a GC bug. >> >> 2 May 2017 07:09 tarihinde "Sokolov Yura" <funny.fal...@gmail.com> yazdı: >> >> GC is triggered *after* allocation than crosses boundary. So your second >>> allocation is actually tries to complete before first allocation freed. And >>> Ubuntu with 4GB memory doesn't allow to allocate 4GB memory cause >>> overcommit is not enabled by default. >>> >>> Use C/C++, or buy more memory, or change your datastructure from slice >>> to slice of slice and allocate second dimension lazely, or call >>> runtime.GC() explicitely between calls to f() (it will block until GC >>> finishes). >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "golang-nuts" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/golang-nuts/HrZpsyfb3i0/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> golang-nuts+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
