Hi Micheal, Sorry about delayed response! Please see my replies inline below.
On Wednesday, December 6, 2023 at 8:22:34 PM UTC-8 Michael Knyszek wrote: On Tuesday, December 5, 2023 at 1:06:19 AM UTC-5 Zhihui Jiang wrote: Hi there, We are running a large scale recommendation system using Golang and we are working on some GC related improvement recently. One of the changes we are trying to apply is to use soft memory limit with GOGC=off as suggested here: https://github.com/golang/go/issues/48409. But during our testing, the memory usage never reaches the memory limit we set. For example, we have 100GB memory available and we set the memory limit to 90GB, but the actual memory usage is very low like <50GB. We also observed the GC is actually very frequent which cost a lot of CPU time. That's odd. Are you positive GOGC=off is set correctly? GOGC=0 is not the same, and will in fact trigger the garbage collector constantly. Can you collect the stderr of your program running with GODEBUG=gctrace=1 set? That'll help identify what's going on. This behavior isn't what I would expect, so if you suspect a bug, please file an issue on GitHub. Yes, I'm pretty sure GOGC is set to OFF when the issue happened. Here is a sample GC trace log when we set GOMEMLIMIT=165GB: - *gc 42787 @354529.311s 4%: 0.43+680+0.11 ms clock, 24+6.4/9387/23913+6.2 ms cpu, 54417->55384->32404 MB, 66903 MB goal, 0 MB stacks, 1 MB globals, 56 P* All other gc trace logs are quite similar with the different heap goal numbers ranging from 65GB to 70GB which is much smaller than 165GB. We also had another test by setting GOGC=100, it turned out the GC behavior and memory usage is quite similar to GOGC=OFF. Also need to note that we have a *big in memory cache of around 30GB* which will always exist in memory, not sure that will affect how GC heap goal is decided in this case. We had another tweak to set GOGC to a high value like 200, and the memory usage is quite close to the memory limit. I have two questions here: 1. Is it expected behavior that the memory usage is much lower than memory limit when GOGC is set to off? From the official doc ( https://github.com/golang/go/issues/48409), it claims " by setting GOGC=off, the Go runtime will always grow the heap to the full memory limit"? No, I would expect the runtime's total memory use to always be roughly at the soft limit in that case. 1. How is the GC frequency decided when using soft memory limit + GOGC=off? Is there some internal defalut value for GOGC in this case to decide when to GC? The frequency isn't decided directly. The heap goal (the total target heap size) is determined entirely from the memory limit. Approximately: heap goal = memory limit - all other memory the runtime is using. The frequency then depends on how big your live heap is and how fast your application allocates from the live heap size to the total target heap size. Thanks! -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c387bdc3-67e1-40d1-97aa-90083febfdafn%40googlegroups.com.
