Because the same Go routine doesn’t mean it will be called on the same thread.
It is essentially a race condition. I could be wrong - I’m sure someone will
correct me. :)
> On Mar 16, 2020, at 9:32 AM, Nitish Saboo <nitish.sabo...@gmail.com> wrote:
>
>
> Hi Robert,
>
> Looks like pattern_db is a global. Are you sure you don’t have multiple Go
> routines calling the load? The global changes may not be visible- so the free
> is doing nothing.
>
> patterndb is a global variable on C side.load method is getting called from a
> single go routine every 3 mins.I used gdb to put a break point on
> pattern_db_free and that is getting called.
>
> The global changes may not be visible- so the free is doing nothing.
>
> >>Since it is a global variable it should be visible..right?
>
> You probably need synchronization or pinning the access routine to a thread.
>
> >>Why we need to pin routine to a thread ?
>
> Thanks,
> Nitish
>
>> On Mon, Mar 16, 2020 at 7:41 PM Robert Engels <reng...@ix.netcom.com> wrote:
>> Looks like pattern_db is a global. Are you sure you don’t have multiple Go
>> routines calling the load? The global changes may not be visible- so the
>> free is doing nothing.
>>
>> You probably need synchronization or pinning the access routine to a thread.
>>
>>>> On Mar 16, 2020, at 9:03 AM, Nitish Saboo <nitish.sabo...@gmail.com> wrote:
>>>>
>>>
>>> Hi,
>>>
>>> From reading the code, I assume that the `pattern_db_new()` does some sort
>>> of allocation of a new pattern DB. I don't see any code releasing the
>>> pattern DB. Is that just missing from your post or something that some
>>> automatic mechanism does?
>>>
>>> >>Apologies, that part got missed in the code snippet. I am making the free
>>> >>call in the C code.
>>>
>>> node.c
>>> ---------
>>> int load_pattern_db(const gchar* file, key_value_cb cb)
>>> {
>>> if(patterndb != NULL){
>>> pattern_db_free(patterndb);
>>> }
>>> patterndb = pattern_db_new();
>>> pattern_db_reload_ruleset(patterndb, configuration, file);
>>> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>>> return 0;
>>> }
>>>
>>> You can try a similar C program that instantiates and frees the structure
>>> to check for similar behavior.
>>>
>>> >> To verify if the C code had some issue, I called the C wrapper code
>>> >> method 'load_pattern_db' from my main.c to completely eliminate Go code
>>> >> here. What I found is there is no increase in memory consumption after
>>> >> every call ('load_pattern_db' was called 5 times). Hence there is no
>>> >> issue from C code.
>>>
>>> Thanks,
>>> Nitish
>>>
>>>> On Mon, Mar 16, 2020 at 7:04 PM Gregor Best <b...@pferdewetten.de> wrote:
>>>> This might be a dumb question but...
>>>>
>>>> From reading the code, I assume that the `pattern_db_new()` does some sort
>>>> of allocation of a new pattern DB. I don't see any code releasing the
>>>> pattern DB. Is that just missing from your post or something that some
>>>> automatic mechanism does?
>>>>
>>>> If not, that might be your leak.
>>>>
>>>>> On 09.03.20 12:33, Nitish Saboo wrote:
>>>>> Hi
>>>>>
>>>>> Following are my Go code and C header file and C wrapper code
>>>>>
>>>>> parser.go
>>>>> ==========
>>>>> var f *os.File
>>>>>
>>>>> func LoadPatternDB(patterndb string) {
>>>>> path := C.CString(patterndb)
>>>>> defer C.free(unsafe.Pointer(path))
>>>>> C.load_pattern_db(path,
>>>>> (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))
>>>>> }
>>>>>
>>>>> //export ParsedData
>>>>> func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
>>>>> f.WriteString(C.GoString(k))
>>>>> f.WriteString("\n")
>>>>> }
>>>>>
>>>>> cfunc.go
>>>>> ========
>>>>> /*
>>>>> #include <stdio.h>
>>>>> // The gateway function
>>>>> void callOnMeGo_cgo(char *k, char *val, size_t val_len)
>>>>> {
>>>>> void ParsedData(const char *k, const char *val, size_t val_len);
>>>>> ParsedData(k, val, val_len);
>>>>> }
>>>>> */
>>>>> import "C"
>>>>>
>>>>> node.h
>>>>> =======
>>>>>
>>>>> #ifndef TEST_H_INCLUDED
>>>>> #define TEST_H_INCLUDED
>>>>>
>>>>> #include <stdlib.h>
>>>>>
>>>>> typedef void (*key_value_cb)(const char* k, const char* val, size_t
>>>>> val_len);
>>>>> int load_pattern_db(const char* file, key_value_cb cb);
>>>>>
>>>>> #endif
>>>>>
>>>>> node.c
>>>>> ---------
>>>>> int load_pattern_db(const gchar* file, key_value_cb cb)
>>>>> {
>>>>> patterndb = pattern_db_new();
>>>>> pattern_db_reload_ruleset(patterndb, configuration, file);
>>>>> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>>>>> return 0;
>>>>> }
>>>>>
>>>>>
>>>>> I am calling 'LoadPatternDB' method in my parser.go file that makes a cgo
>>>>> call 'C.load_pattern_db' where I am passing a callback function to the C
>>>>> code.
>>>>> The C code is a wrapper code that internally calls some syslog-ng library
>>>>> apis'
>>>>>
>>>>> What I observed is:
>>>>>
>>>>> 1)As soon as I call LoadPatternDB() method in parser.go there is some
>>>>> increase in memory consumption(some memory leak).Ideally that should not
>>>>> have happened.
>>>>>
>>>>> 2)To verify if the C code had some issue, I called the C wrapper code
>>>>> method 'load_pattern_db' from my main.c in the following manner to
>>>>> completely eliminate Go code here.What I found is there is no increase in
>>>>> memory consumption after every call ('load_pattern_db' was called 5
>>>>> times).Hence there is no memory leak from C code.So the issue lies in the
>>>>> Go code in 'LoadPatternDB' method in parser.go
>>>>>
>>>>> main.c
>>>>> =======
>>>>>
>>>>> void check(char *key, char *value, size_t value_len)
>>>>> {
>>>>> printf("I am in function check\n");
>>>>> }
>>>>>
>>>>> int main(void){
>>>>> char* filename = "/home/nitish/default.xml";
>>>>> key_value_cb s = check;
>>>>> int i;
>>>>> for (i=1; i<=5; i++)
>>>>> {
>>>>> load_pattern_db(filename, s);
>>>>> printf("Sleeping for 5 second.\n");
>>>>> sleep(5);
>>>>> }
>>>>> printf("Loading done 5 times.\n");
>>>>> return 0;
>>>>> }
>>>>>
>>>>> 3)Can someone please guide me and help me figure out the mem-leak in
>>>>> 'LoadPatternDB' method in parser.go at very first glance? Is the callback
>>>>> function pointer an issue here ?
>>>>>
>>>>> 4)What tool can I use to check this mem-leak ?
>>>>>
>>>>> Thanks,
>>>>> Nitish
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "golang-nuts" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq5cAJ19CQ8OmMPSJmVB1P3t4hE0CKZ4HsEsH-mp6zm3Ng%40mail.gmail.com.
>>>> --
>>>> --
>>>> Gregor Best
>>>> b...@pferdewetten.de
>>>> --
>>>> You received this message because you are subscribed to the Google Groups
>>>> "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/golang-nuts/fee07a13-e3d7-4a2e-6a6c-fdb11140ce21%40pferdewetten.de.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq4-%3DcsuKRP4Xa-MVgrm5MJdWLNVe0v5_i3jswjesHQTqA%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/02BB1DD2-0168-452A-A978-A43FAEE38107%40ix.netcom.com.
