It is perhaps easiest if I go through the conditions which must be met for
data to be reclaimed by the Operating System:
First, the data must be unreachable by goroutines through a path of
pointers. When you nil the object, you break such a path, but you must make
sure there are no other such paths to the object. Once there is no
reachability of the object, it is eligible for collection.
Second, garbage collection must run. It runs periodically, but it can take
some time before it does so, and if you check memory usage right after
you've freed up data, it might not have run yet. You can run Go with GC
debug output (env GODEBUG=gctrace=1 ...) to see when it actually runs. This
is a property of using a mark'n'sweep garbage collection strategy as the
memory is scavenged for unreachable data immediately. It is somewhat in
contrast to reference counting methods which will usually free up the
memory quicker.
Third, the GC must have some excess data it wants to give back to the OS.
Usually it likes to reuse a memory space if it is needed shortly and will
only give memory back eventually if the program consistently runs in less
memory.
There are some subtle things you might want to look out for however. When
you unmarshal from the string into the concrete report type, you should
probably avoid keeping a slice to the underlying source string. Otherwise,
it will keep said string alive for the duration of the reports existence.
Explicitly copying data and making sure you have no reference to the source
string is one thing that is important to get right, especially if the
source string is much larger than the report in memory size. Another thing
is that I would just do `delete(reports, timestamp)` since it is a no-op if
timestamp doesn't exist, and deletion effectively breaks the pointer anyway.
I hope this helps somewhat in narrowing down what could be wrong.
On Fri, Sep 11, 2020 at 7:56 PM Christopher Crotty <
crotty.christop...@gmail.com> wrote:
> Apologies in advance if this is not the right group to post this...
>
> I am having a memory leak issue in an app that reads data reports from a
> sensor and stores them for a length of time. Running pprof on the app, It
> appears that the memory is not getting released when the old data is
> purged.
>
> There can be several report types, so I created an interface definition
> and am storing these reports under a slice of interfaces in a map with a
> timestamp key
>
> When it comes time to purge, I'm setting the slice at that timestamp to
> nil, but the memory does not seem to be reclaimed.
>
> Dirty Details:
> The data comes in as a string message which I unmarshal into one of the
> concrete report types. The reports are stored in the following structure...
> map[string]map[int64][]ReportIface where the first map has a key of
> sensorID and the inner map has a key of timestamp. The ReportIface is an
> interface definition that all of the reports implement
>
> During purging, I roll through the inner map and if the timestamp key is
> less than the purge time I run the following
> if reports[timestamp] != nil {
> reports[timestamp] = nil
> }
> delete(reports, timestamp).
>
> Is there anything else I need to do for this? Pprof is showing the
> memory from the unmarshal as still in use even after the purge.
>
> Thanks in advance,
> Chris
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/d0a1eee7-7968-4560-9cd4-98d69e48ea91n%40googlegroups.com
> <https://groups.google.com/d/msgid/golang-nuts/d0a1eee7-7968-4560-9cd4-98d69e48ea91n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
J.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAGrdgiXze%3DOmCk96Ro7RG-J6hjGRHZ7vC-faTiNW%3DX7vSaZgXg%40mail.gmail.com.
