I would also use pprof and ensure your Go code is not leaking memory. There are
multiple tutorials on using memory profilers to detect memory leaks.
> On Mar 16, 2020, at 9:12 AM, Robert Engels <reng...@ix.netcom.com> wrote:
>
>
> Looks like pattern_db is a global. Are you sure you don’t have multiple Go
> routines calling the load? The global changes may not be visible- so the free
> is doing nothing.
>
> You probably need synchronization or pinning the access routine to a thread.
>
>>> On Mar 16, 2020, at 9:03 AM, Nitish Saboo <nitish.sabo...@gmail.com> wrote:
>>>
>>
>> Hi,
>>
>> From reading the code, I assume that the `pattern_db_new()` does some sort
>> of allocation of a new pattern DB. I don't see any code releasing the
>> pattern DB. Is that just missing from your post or something that some
>> automatic mechanism does?
>>
>> >>Apologies, that part got missed in the code snippet. I am making the free
>> >>call in the C code.
>>
>> node.c
>> ---------
>> int load_pattern_db(const gchar* file, key_value_cb cb)
>> {
>> if(patterndb != NULL){
>> pattern_db_free(patterndb);
>> }
>> patterndb = pattern_db_new();
>> pattern_db_reload_ruleset(patterndb, configuration, file);
>> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>> return 0;
>> }
>>
>> You can try a similar C program that instantiates and frees the structure to
>> check for similar behavior.
>>
>> >> To verify if the C code had some issue, I called the C wrapper code
>> >> method 'load_pattern_db' from my main.c to completely eliminate Go code
>> >> here. What I found is there is no increase in memory consumption after
>> >> every call ('load_pattern_db' was called 5 times). Hence there is no
>> >> issue from C code.
>>
>> Thanks,
>> Nitish
>>
>>> On Mon, Mar 16, 2020 at 7:04 PM Gregor Best <b...@pferdewetten.de> wrote:
>>> This might be a dumb question but...
>>>
>>> From reading the code, I assume that the `pattern_db_new()` does some sort
>>> of allocation of a new pattern DB. I don't see any code releasing the
>>> pattern DB. Is that just missing from your post or something that some
>>> automatic mechanism does?
>>>
>>> If not, that might be your leak.
>>>
>>>> On 09.03.20 12:33, Nitish Saboo wrote:
>>>> Hi
>>>>
>>>> Following are my Go code and C header file and C wrapper code
>>>>
>>>> parser.go
>>>> ==========
>>>> var f *os.File
>>>>
>>>> func LoadPatternDB(patterndb string) {
>>>> path := C.CString(patterndb)
>>>> defer C.free(unsafe.Pointer(path))
>>>> C.load_pattern_db(path, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))
>>>> }
>>>>
>>>> //export ParsedData
>>>> func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
>>>> f.WriteString(C.GoString(k))
>>>> f.WriteString("\n")
>>>> }
>>>>
>>>> cfunc.go
>>>> ========
>>>> /*
>>>> #include <stdio.h>
>>>> // The gateway function
>>>> void callOnMeGo_cgo(char *k, char *val, size_t val_len)
>>>> {
>>>> void ParsedData(const char *k, const char *val, size_t val_len);
>>>> ParsedData(k, val, val_len);
>>>> }
>>>> */
>>>> import "C"
>>>>
>>>> node.h
>>>> =======
>>>>
>>>> #ifndef TEST_H_INCLUDED
>>>> #define TEST_H_INCLUDED
>>>>
>>>> #include <stdlib.h>
>>>>
>>>> typedef void (*key_value_cb)(const char* k, const char* val, size_t
>>>> val_len);
>>>> int load_pattern_db(const char* file, key_value_cb cb);
>>>>
>>>> #endif
>>>>
>>>> node.c
>>>> ---------
>>>> int load_pattern_db(const gchar* file, key_value_cb cb)
>>>> {
>>>> patterndb = pattern_db_new();
>>>> pattern_db_reload_ruleset(patterndb, configuration, file);
>>>> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>>>> return 0;
>>>> }
>>>>
>>>>
>>>> I am calling 'LoadPatternDB' method in my parser.go file that makes a cgo
>>>> call 'C.load_pattern_db' where I am passing a callback function to the C
>>>> code.
>>>> The C code is a wrapper code that internally calls some syslog-ng library
>>>> apis'
>>>>
>>>> What I observed is:
>>>>
>>>> 1)As soon as I call LoadPatternDB() method in parser.go there is some
>>>> increase in memory consumption(some memory leak).Ideally that should not
>>>> have happened.
>>>>
>>>> 2)To verify if the C code had some issue, I called the C wrapper code
>>>> method 'load_pattern_db' from my main.c in the following manner to
>>>> completely eliminate Go code here.What I found is there is no increase in
>>>> memory consumption after every call ('load_pattern_db' was called 5
>>>> times).Hence there is no memory leak from C code.So the issue lies in the
>>>> Go code in 'LoadPatternDB' method in parser.go
>>>>
>>>> main.c
>>>> =======
>>>>
>>>> void check(char *key, char *value, size_t value_len)
>>>> {
>>>> printf("I am in function check\n");
>>>> }
>>>>
>>>> int main(void){
>>>> char* filename = "/home/nitish/default.xml";
>>>> key_value_cb s = check;
>>>> int i;
>>>> for (i=1; i<=5; i++)
>>>> {
>>>> load_pattern_db(filename, s);
>>>> printf("Sleeping for 5 second.\n");
>>>> sleep(5);
>>>> }
>>>> printf("Loading done 5 times.\n");
>>>> return 0;
>>>> }
>>>>
>>>> 3)Can someone please guide me and help me figure out the mem-leak in
>>>> 'LoadPatternDB' method in parser.go at very first glance? Is the callback
>>>> function pointer an issue here ?
>>>>
>>>> 4)What tool can I use to check this mem-leak ?
>>>>
>>>> Thanks,
>>>> Nitish
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq5cAJ19CQ8OmMPSJmVB1P3t4hE0CKZ4HsEsH-mp6zm3Ng%40mail.gmail.com.
>>> --
>>> --
>>> Gregor Best
>>> b...@pferdewetten.de
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/fee07a13-e3d7-4a2e-6a6c-fdb11140ce21%40pferdewetten.de.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq4-%3DcsuKRP4Xa-MVgrm5MJdWLNVe0v5_i3jswjesHQTqA%40mail.gmail.com.
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/4A1AFEDD-2D70-4B33-A183-83887A79A9A0%40ix.netcom.com.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/B4B6F587-D1DB-401C-8840-A12C77D52857%40ix.netcom.com.
