We used go1.16.5 before go1.19 released. Occassionally we found ballast takes up RSS :)
About go1.19 GOMEMLIMIT, I tested and have some thinking: - GOMEMLIMIT is a soft limit, if we deploy by per container per service, GOGC=off+GOMEMLIMIT=70%*totalMemory, it may works as expected. And it may replace the ballast directly. - If we deploy multiple services in the same host (not per container per service), and if we specify GOGC=off+GOMEMLIMIT=<?>, we may calculate how much memory (and max memory) a service should use. I think per container per service will be better and easier to deploy to use GOMEMLIMIT. On Friday, November 11, 2022 at 5:58:54 AM UTC+8 mkny...@google.com wrote: > That's correct. The runtime has a simple heuristic for avoiding zeroing > but it's far from perfect. As a result, a ballast is inherently always > going to be a little risky. This is especially true on some platforms, such > as Windows, since there's no way to avoid marking the memory as committed > (Windows is free to use demand paging for memory in the range, so overall > system memory pressure may increase, but you can't avoid it being > *counted* as committed for a particular process). > > Taking a step back: why a ballast? What about your application makes a > ballast a better idea than, for example, setting GOMEMLIMIT=<something> and > GOGC=off? > > (For additional context, back when the memory limit was proposed, so was a > memory target ( > https://go.googlesource.com/proposal/+/master/design/44309-user-configurable-memory-target.md) > > which more directly replaces a ballast. I found very little interest in > this feature.) > > On Monday, November 7, 2022 at 9:08:23 AM UTC-5 hit.zh...@gmail.com wrote: > >> Hi, guys, I know what happened. >> >> When we write `ballast := make([]byte, 1<<30)`, it will call makeslice to >> create a new slice. It's a large object. The memory will be allocated >> directly via allocLarge() function from heap. >> >> Actually, after allocating a mspan for it, it will check the address >> range whether it should be zeroed. Please check function `func >> allocNeedsZero(base, npage uintptr) bool`. When this function returns true, >> it means the slice underlying memory will be written to zero, otherwise it >> won't write it to zero. >> >> When we make a ballast as mentioned before, maybe it succeed (no RSS >> taken up) or fail (RSS taken up), it's relevant with the return value of >> function allocNeedsZero(...). And this function return true or false is >> relevant with the arena's state which is affected by previous object >> allocation and recycle. >> >> On Monday, November 7, 2022 at 1:26:41 PM UTC+8 tapi...@gmail.com wrote: >> >>> I ever also found this problem: global ballast doesn't work. >>> So I always use local ballast instead now. >>> >>> On Sunday, November 6, 2022 at 8:37:55 PM UTC+8 hit.zh...@gmail.com >>> wrote: >>> >>>> Before, I think the memory is allocated by mmap anon, which Linux OS >>>> guarantees reading will return zero value and no physical pages allocated. >>>> When writing happens, the physical pages will be allocated. Then I think >>>> the create a byte slice maybe the same. >>>> >>>> Your idea is clear. I agree with it. >>>> >>>> Just now, I use gdb to dump the ballast anon memory and use hexdump to >>>> check its dirty content, all data is zero (Maybe zeroing happens). >>>> But after turning GC off, it works as expected (no RSS is taken, no >>>> DIRTY). >>>> I think there must be something I didn't get it. >>>> >>>> On Sunday, November 6, 2022 at 8:11:51 PM UTC+8 Jan Mercl wrote: >>>> >>>>> On Sun, Nov 6, 2022 at 12:54 PM Kn (Kn) <hit.zh...@gmail.com> wrote: >>>>> >>>>> > Now the problem begins. I expect the ballast like `ballast := >>>>> make([]byte, 1<<30)` shouldn't take up any physical memory because >>>>> there's >>>>> no any writing to it. >>>>> >>>>> The backing array is specified to be zeroed, so we cannot say there's >>>>> no writing to it. Depending on the size of the backing array and the >>>>> operating system it may not get written as an optimization if backed >>>>> by memory the OS can guarantee to be zero filled. Only then it may >>>>> remain not yet bound to physical memory. >>>>> >>>>> A simple implementation will just zero it, meaning the opposite >>>>> happens - every byte of the backing array gets written and backing >>>>> pages for it get allocated. >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/79914750-fafd-4904-881c-db0c1b9fe81bn%40googlegroups.com.
