Re: [go-nuts] libgo is more fast that grouting.

[wagner riffel]

On Mon, 16 Mar 2020 20:13:10 +0800
"'Benjamin' via golang-nuts"  wrote:

> How do you all think about it?
> 
IMO there's no such thing as "Go-style concurrency" if you don't have
`select`, which doesn't looks like this library provides (as many
others).

—wagner

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20200317012317.0485b4a6%40pampas.

[go-nuts] Re: [ANN] Gio: portable immediate mode GUI programs in Go for iOS/tvOS, Android, macOS, Linux, Windows

[Jason E. Aten]

As an addition to Elias' examples, I wrote a little hello world++ here.

https://github.com/glycerine/hello_gio

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2608ddeb-86b7-4b19-81cc-e861afef213b%40googlegroups.com.

Re: [go-nuts] Re: Mem-Leak in Go method

[Robert Engels]

In the single Go routine, use LockOSThread(). Then it was always be accessed on 
the same thread removing the memory synchronization problems. 

> On Mar 16, 2020, at 11:28 AM, Nitish Saboo  wrote:
> 
> 
> Hi,
> 
> So finally I got a little hint of the problem from what Robert described 
> earlier in the mail. Thank you so much Robert.
> Looks like patterndb instance is not getting freed.
> 
> node.c
> -
> 
> PatternDB *patterndb;
> 
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
>  if(patterndb != NULL){
>   printf("Patterndb Free called\n"); <<< Not getting printed
> pattern_db_free(patterndb);
>   }
>   patterndb = pattern_db_new();
>   pattern_db_reload_ruleset(patterndb, configuration, file);
>   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>   return 0;
> }
> 
> 
> patterndb is a global variable in C wrapper code that internally calls some 
> syslog-ng library api's.Since load_pattern_db() method is getting called from 
> a single goroutine every 3 mins, patterndb instance is not getting free 
> because the statement inside if clause ('if(patterndb != NULL)') is not 
> getting printed when I call 'load_pattern_db()' method.Looks like that is the 
> leak here.
> 
> 
> 1)Can someone please help me understand the problem in detail as in why am I 
> facing this issue?
> 
> 2)Though patterndb instance is a global variable in the C wrapper code, why 
> is it not getting freed?
> 
> 3)How can I fix this issue?
> 
> Thanks,
> Nitish
> 
>> On Mon, Mar 16, 2020 at 8:17 PM Nitish Saboo  
>> wrote:
>> Hi Robert,
>> 
>> Sorry I did not understand your point completely.
>> I have a global variable patterndb on C side and It is getting called from a 
>> single goroutine every 3 mins. Why do I need to synchronize it?
>> Even though the goroutine gets pinned to different threads, it can access 
>> the same global variable every time and free it ...right ?
>> 
>> Thanks,
>> Nitish
>> 
>> 
>>> On Mon, Mar 16, 2020 at 8:10 PM Robert Engels  wrote:
>>> Yes, you have a shared global variable you need to synchronize. 
>>> 
> On Mar 16, 2020, at 9:35 AM, Nitish Saboo  
> wrote:
> 
 
 Hi,
 
 Are you saying it is working as expected?
 
 Thanks,
 Nitish
 
> On Mon, Mar 16, 2020 at 7:42 PM Volker Dobler 
>  wrote:
>> On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:
>> Hi,
>> 
>> I upgraded the go version and compiled the binary against go version 'go 
>> version go1.12.4 linux/amd64'.
>> I ran the program for some time. I made almost 30-40 calls to the method 
>> Load_Pattern_Db().
>> The program starts with 6% Mem Usage. The memory usage increases only 
>> when I call 'LoadPatternDb()' method and LoadPatternDb() method is 
>> called by a goroutine at regular intervals of 3 minutes(making use of 
>> ticker here ).
>> 
>> What I observed is:
>> 
>> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory 
>> usage got almost constant at 29%. But I did not expect the program to 
>> take this much memory.
>>When I restart the service the Mem Usage again starts with 6%.
>> 
>> a) Is this the sign of memory leaking?
> 
> No, as explained above.
>  
>> 
>> b) Till this moment I did not see memory getting reclaimed or going down 
>> but it did become constant.
>> As mentioned by experts above, the same sort of behavior is seen here. 
>> But I did not expect the memory usage to grow this much. Is this 
>> expected? 
> Yes. (Well, no. But your gut feeling of how much memory
> should grow is not a suitable benchmark to compare
> actual growth to.)
>  
>> 
>> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc) as 
>> mentioned in the earlier email.
>> 
>> a) Which all mem-stats variables should I look into for debugging this 
>> kind of behavior?
> Alloc/HeapAlloc 
> But probably this is plain useless as nothing here indicates
> that you do have any memory issues.
> 
> V.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com.
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to golang-nuts+unsubscr...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com.

-- 

Re: [go-nuts] Re: Mem-Leak in Go method

[Nitish Saboo]

Hi,

So finally I got a little hint of the problem from what Robert described
earlier in the mail. Thank you so much Robert.
Looks like patterndb instance is not getting freed.

node.c
-

PatternDB *patterndb;

int load_pattern_db(const gchar* file, key_value_cb cb)
{
 if(patterndb != NULL){
  printf("Patterndb Free called\n"); <<< Not getting printed
pattern_db_free(patterndb);
  }
  patterndb = pattern_db_new();
  pattern_db_reload_ruleset(patterndb, configuration, file);
  pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
  return 0;
}


patterndb is a global variable in C wrapper code that internally calls some
syslog-ng library api's.Since load_pattern_db() method is getting called
from a single goroutine every 3 mins, patterndb instance is not getting
free because the statement inside if clause ('if(patterndb != NULL)') is
not getting printed when I call 'load_pattern_db()' method.Looks like that
is the leak here.


1)Can someone please help me understand the problem in detail as in why am
I facing this issue?

2)Though patterndb instance is a global variable in the C wrapper code, why
is it not getting freed?

3)How can I fix this issue?

Thanks,
Nitish

On Mon, Mar 16, 2020 at 8:17 PM Nitish Saboo 
wrote:

> Hi Robert,
>
> Sorry I did not understand your point completely.
> I have a global variable patterndb on C side and It is getting called from
> a single goroutine every 3 mins. Why do I need to synchronize it?
> Even though the goroutine gets pinned to different threads, it can access
> the same global variable every time and free it ...right ?
>
> Thanks,
> Nitish
>
>
> On Mon, Mar 16, 2020 at 8:10 PM Robert Engels 
> wrote:
>
>> Yes, you have a shared global variable you need to synchronize.
>>
>> On Mar 16, 2020, at 9:35 AM, Nitish Saboo 
>> wrote:
>>
>> 
>> Hi,
>>
>> Are you saying it is working as expected?
>>
>> Thanks,
>> Nitish
>>
>> On Mon, Mar 16, 2020 at 7:42 PM Volker Dobler 
>> wrote:
>>
>>> On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:

 Hi,

 I upgraded the go version and compiled the binary against go version
 'go version go1.12.4 linux/amd64'.
 I ran the program for some time. I made almost 30-40 calls to the
 method Load_Pattern_Db().
 The program starts with 6% Mem Usage. The memory usage increases only
 when I call 'LoadPatternDb()' method and LoadPatternDb() method is called
 by a goroutine at regular intervals of 3 minutes(making use of ticker here
 ).

 What I observed is:

 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory
 usage got almost constant at 29%. But I did not expect the program to take
 this much memory.
When I restart the service the Mem Usage again starts with 6%.

 a) Is this the sign of memory leaking?

>>>
>>> No, as explained above.
>>>
>>>

 b) Till this moment I did not see memory getting reclaimed or going
 down but it did become constant.
 As mentioned by experts above, the same sort of behavior is seen here.
 But I did not expect the memory usage to grow this much. Is this expected?

>>> Yes. (Well, no. But your gut feeling of how much memory
>>> should grow is not a suitable benchmark to compare
>>> actual growth to.)
>>>
>>>

 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc)
 as mentioned in the earlier email.

 a) Which all mem-stats variables should I look into for debugging this
 kind of behavior?

>>> Alloc/HeapAlloc
>>> But probably this is plain useless as nothing here indicates
>>> that you do have any memory issues.
>>>
>>> V.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com
>> 
>> .
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

[go-nuts] libgo is more fast that grouting.

['Benjamin' via golang-nuts]

I find a project that is very interesting, https://github.com/yyzybb537/libgo 


How do you all think about it?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/92B08B84-B9F9-4EB3-9065-DEABFCDC4210%40icloud.com.

Re: [go-nuts] Re: Mem-Leak in Go method

[Nitish Saboo]

Hi Robert,

Sorry I did not understand your point completely.
I have a global variable patterndb on C side and It is getting called from
a single goroutine every 3 mins. Why do I need to synchronize it?
Even though the goroutine gets pinned to different threads, it can access
the same global variable every time and free it ...right ?

Thanks,
Nitish


On Mon, Mar 16, 2020 at 8:10 PM Robert Engels  wrote:

> Yes, you have a shared global variable you need to synchronize.
>
> On Mar 16, 2020, at 9:35 AM, Nitish Saboo 
> wrote:
>
> 
> Hi,
>
> Are you saying it is working as expected?
>
> Thanks,
> Nitish
>
> On Mon, Mar 16, 2020 at 7:42 PM Volker Dobler 
> wrote:
>
>> On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:
>>>
>>> Hi,
>>>
>>> I upgraded the go version and compiled the binary against go version 'go
>>> version go1.12.4 linux/amd64'.
>>> I ran the program for some time. I made almost 30-40 calls to the method
>>> Load_Pattern_Db().
>>> The program starts with 6% Mem Usage. The memory usage increases only
>>> when I call 'LoadPatternDb()' method and LoadPatternDb() method is called
>>> by a goroutine at regular intervals of 3 minutes(making use of ticker here
>>> ).
>>>
>>> What I observed is:
>>>
>>> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory
>>> usage got almost constant at 29%. But I did not expect the program to take
>>> this much memory.
>>>When I restart the service the Mem Usage again starts with 6%.
>>>
>>> a) Is this the sign of memory leaking?
>>>
>>
>> No, as explained above.
>>
>>
>>>
>>> b) Till this moment I did not see memory getting reclaimed or going down
>>> but it did become constant.
>>> As mentioned by experts above, the same sort of behavior is seen here.
>>> But I did not expect the memory usage to grow this much. Is this expected?
>>>
>> Yes. (Well, no. But your gut feeling of how much memory
>> should grow is not a suitable benchmark to compare
>> actual growth to.)
>>
>>
>>>
>>> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc) as
>>> mentioned in the earlier email.
>>>
>>> a) Which all mem-stats variables should I look into for debugging this
>>> kind of behavior?
>>>
>> Alloc/HeapAlloc
>> But probably this is plain useless as nothing here indicates
>> that you do have any memory issues.
>>
>> V.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com
> 
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CALjMrq4qOC1zBDSdwnFRMSQZB_R%2Bf%2B6fcwwFPOMy9ArFO-4uoQ%40mail.gmail.com.

Re: [go-nuts] Re: Mem-Leak in Go method

[Robert Engels]

Yes, you have a shared global variable you need to synchronize. 

> On Mar 16, 2020, at 9:35 AM, Nitish Saboo  wrote:
> 
> 
> Hi,
> 
> Are you saying it is working as expected?
> 
> Thanks,
> Nitish
> 
>> On Mon, Mar 16, 2020 at 7:42 PM Volker Dobler  
>> wrote:
>>> On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:
>>> Hi,
>>> 
>>> I upgraded the go version and compiled the binary against go version 'go 
>>> version go1.12.4 linux/amd64'.
>>> I ran the program for some time. I made almost 30-40 calls to the method 
>>> Load_Pattern_Db().
>>> The program starts with 6% Mem Usage. The memory usage increases only when 
>>> I call 'LoadPatternDb()' method and LoadPatternDb() method is called by a 
>>> goroutine at regular intervals of 3 minutes(making use of ticker here ).
>>> 
>>> What I observed is:
>>> 
>>> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory usage 
>>> got almost constant at 29%. But I did not expect the program to take this 
>>> much memory.
>>>When I restart the service the Mem Usage again starts with 6%.
>>> 
>>> a) Is this the sign of memory leaking?
>> 
>> No, as explained above.
>>  
>>> 
>>> b) Till this moment I did not see memory getting reclaimed or going down 
>>> but it did become constant.
>>> As mentioned by experts above, the same sort of behavior is seen here. But 
>>> I did not expect the memory usage to grow this much. Is this expected? 
>> Yes. (Well, no. But your gut feeling of how much memory
>> should grow is not a suitable benchmark to compare
>> actual growth to.)
>>  
>>> 
>>> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc) as 
>>> mentioned in the earlier email.
>>> 
>>> a) Which all mem-stats variables should I look into for debugging this kind 
>>> of behavior?
>> Alloc/HeapAlloc 
>> But probably this is plain useless as nothing here indicates
>> that you do have any memory issues.
>> 
>> V.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0F39BE6B-52E6-4665-93C4-B18490BC7C23%40ix.netcom.com.

Re: [go-nuts] Mem-Leak in Go method

[Robert Engels]

Because the same Go routine doesn’t mean it will be called on the same thread. 
It is essentially a race condition. I could be wrong - I’m sure someone will 
correct me. :)

> On Mar 16, 2020, at 9:32 AM, Nitish Saboo  wrote:
> 
> 
> Hi Robert,
> 
> Looks like pattern_db is a global. Are you sure you don’t have multiple Go 
> routines calling the load? The global changes may not be visible- so the free 
> is doing nothing. 
> 
> patterndb is a global variable on C side.load method is getting called from a 
> single go routine every 3 mins.I used gdb to put a break point on 
> pattern_db_free and that is getting called.
> 
> The global changes may not be visible- so the free is doing nothing. 
> 
> >>Since it is a global variable it should be visible..right?
> 
> You probably need synchronization or pinning the access routine to a thread. 
> 
> >>Why we need to pin routine to a thread ?
> 
> Thanks,
> Nitish
> 
>> On Mon, Mar 16, 2020 at 7:41 PM Robert Engels  wrote:
>> Looks like pattern_db is a global. Are you sure you don’t have multiple Go 
>> routines calling the load? The global changes may not be visible- so the 
>> free is doing nothing. 
>> 
>> You probably need synchronization or pinning the access routine to a thread. 
>> 
 On Mar 16, 2020, at 9:03 AM, Nitish Saboo  wrote:
 
>>> 
>>> Hi,
>>> 
>>> From reading the code, I assume that the `pattern_db_new()` does some sort 
>>> of allocation of a new pattern DB. I don't see any code releasing the 
>>> pattern DB. Is that just missing from your post or something that some 
>>> automatic mechanism does?
>>> 
>>> >>Apologies, that part got missed in the code snippet. I am making the free 
>>> >>call in the C code.
>>> 
>>> node.c
>>> -
>>> int load_pattern_db(const gchar* file, key_value_cb cb)
>>> {
>>>  if(patterndb != NULL){
>>> pattern_db_free(patterndb);
>>>   }
>>>   patterndb = pattern_db_new();
>>>   pattern_db_reload_ruleset(patterndb, configuration, file);
>>>   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>>>   return 0;
>>> }
>>> 
>>> You can try a similar C program that instantiates and frees the structure 
>>> to check for similar behavior. 
>>> 
>>> >> To verify if the C code had some issue, I called the C wrapper code 
>>> >> method 'load_pattern_db' from my main.c to completely eliminate Go code 
>>> >> here. What I found is there is no increase in memory consumption after 
>>> >> every call ('load_pattern_db' was called 5 times). Hence there is no 
>>> >> issue from C code.
>>> 
>>> Thanks,
>>> Nitish
>>> 
 On Mon, Mar 16, 2020 at 7:04 PM Gregor Best  wrote:
 This might be a dumb question but...
 
 From reading the code, I assume that the `pattern_db_new()` does some sort 
 of allocation of a new pattern DB. I don't see any code releasing the 
 pattern DB. Is that just missing from your post or something that some 
 automatic mechanism does?
 
 If not, that might be your leak.
 
> On 09.03.20 12:33, Nitish Saboo wrote:
> Hi
> 
> Following are my Go code and C header file and C wrapper code
> 
> parser.go
> ==
> var f *os.File
> 
> func LoadPatternDB(patterndb string) {
> path := C.CString(patterndb)
> defer C.free(unsafe.Pointer(path))
> C.load_pattern_db(path, 
> (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))
> }
> 
> //export ParsedData
> func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
> f.WriteString(C.GoString(k))
> f.WriteString("\n")
> }
> 
> cfunc.go
> 
> /*
> #include 
> // The gateway function
> void callOnMeGo_cgo(char *k, char *val, size_t val_len)
> {
> void ParsedData(const char *k, const char *val, size_t val_len);
> ParsedData(k, val, val_len);
> }
> */
> import "C"
> 
> node.h
> ===
> 
> #ifndef TEST_H_INCLUDED
> #define TEST_H_INCLUDED
> 
> #include 
> 
> typedef void (*key_value_cb)(const char* k, const char* val, size_t 
> val_len);
> int load_pattern_db(const char* file, key_value_cb cb);
> 
> #endif
> 
> node.c
> -
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
>   patterndb = pattern_db_new();
>   pattern_db_reload_ruleset(patterndb, configuration, file);
>   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>   return 0;
> }
> 
> 
> I am calling 'LoadPatternDB' method in my parser.go file that makes a cgo 
> call 'C.load_pattern_db' where I am passing a callback function to the C 
> code.
> The C code is a wrapper code that internally calls some syslog-ng library 
> apis'
> 
> What I observed is:
> 
> 1)As soon as I call LoadPatternDB() method in parser.go there is some 
> increase in memory consumption(some memory leak).Ideally that should not 
> have happened.
> 

Re: [go-nuts] Re: Mem-Leak in Go method

[Nitish Saboo]

Hi,

Are you saying it is working as expected?

Thanks,
Nitish

On Mon, Mar 16, 2020 at 7:42 PM Volker Dobler 
wrote:

> On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:
>>
>> Hi,
>>
>> I upgraded the go version and compiled the binary against go version 'go
>> version go1.12.4 linux/amd64'.
>> I ran the program for some time. I made almost 30-40 calls to the method
>> Load_Pattern_Db().
>> The program starts with 6% Mem Usage. The memory usage increases only
>> when I call 'LoadPatternDb()' method and LoadPatternDb() method is called
>> by a goroutine at regular intervals of 3 minutes(making use of ticker here
>> ).
>>
>> What I observed is:
>>
>> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory
>> usage got almost constant at 29%. But I did not expect the program to take
>> this much memory.
>>When I restart the service the Mem Usage again starts with 6%.
>>
>> a) Is this the sign of memory leaking?
>>
>
> No, as explained above.
>
>
>>
>> b) Till this moment I did not see memory getting reclaimed or going down
>> but it did become constant.
>> As mentioned by experts above, the same sort of behavior is seen here.
>> But I did not expect the memory usage to grow this much. Is this expected?
>>
> Yes. (Well, no. But your gut feeling of how much memory
> should grow is not a suitable benchmark to compare
> actual growth to.)
>
>
>>
>> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc) as
>> mentioned in the earlier email.
>>
>> a) Which all mem-stats variables should I look into for debugging this
>> kind of behavior?
>>
> Alloc/HeapAlloc
> But probably this is plain useless as nothing here indicates
> that you do have any memory issues.
>
> V.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com.

Re: [go-nuts] Mem-Leak in Go method

[Nitish Saboo]

Hi Robert,

Looks like pattern_db is a global. Are you sure you don’t have multiple Go
routines calling the load? The global changes may not be visible- so the
free is doing nothing.

patterndb is a global variable on C side.load method is getting called from
a single go routine every 3 mins.I used gdb to put a break point on
pattern_db_free and that is getting called.

The global changes may not be visible- so the free is doing nothing.

>>Since it is a global variable it should be visible..right?

You probably need synchronization or pinning the access routine to a
thread.

>>Why we need to pin routine to a thread ?

Thanks,
Nitish

On Mon, Mar 16, 2020 at 7:41 PM Robert Engels  wrote:

> Looks like pattern_db is a global. Are you sure you don’t have multiple Go
> routines calling the load? The global changes may not be visible- so the
> free is doing nothing.
>
> You probably need synchronization or pinning the access routine to a
> thread.
>
> On Mar 16, 2020, at 9:03 AM, Nitish Saboo 
> wrote:
>
> 
> Hi,
>
> From reading the code, I assume that the `pattern_db_new()` does some sort
> of allocation of a new pattern DB. I don't see any code releasing the
> pattern DB. Is that just missing from your post or something that some
> automatic mechanism does?
>
> >>Apologies, that part got missed in the code snippet. I am making the
> free call in the C code.
>
> node.c
> -
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
>  if(patterndb != NULL){
> pattern_db_free(patterndb);
>   }
>   patterndb = pattern_db_new();
>   pattern_db_reload_ruleset(patterndb, configuration, file);
>   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>   return 0;
> }
>
> You can try a similar C program that instantiates and frees the structure
> to check for similar behavior.
>
> >> To verify if the C code had some issue, I called the C wrapper code
> method '*load_pattern_db*' from my main.c to completely eliminate Go code
> here. What I found is there is no increase in memory consumption after
> every call ('load_pattern_db' was called 5 times). Hence there is no issue
> from C code.
>
> Thanks,
> Nitish
>
> On Mon, Mar 16, 2020 at 7:04 PM Gregor Best  wrote:
>
>> This might be a dumb question but...
>>
>> From reading the code, I assume that the `pattern_db_new()` does some
>> sort of allocation of a new pattern DB. I don't see any code releasing the
>> pattern DB. Is that just missing from your post or something that some
>> automatic mechanism does?
>>
>> If not, that might be your leak.
>> On 09.03.20 12:33, Nitish Saboo wrote:
>>
>> Hi
>>
>> Following are my Go code and C header file and C wrapper code
>>
>> parser.go
>> ==
>> var f *os.File
>>
>> func LoadPatternDB(patterndb string) {
>> path := C.CString(patterndb)
>> defer C.free(unsafe.Pointer(path))
>> C.load_pattern_db(path,
>> (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))
>> }
>>
>> //export ParsedData
>> func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
>> f.WriteString(C.GoString(k))
>> f.WriteString("\n")
>> }
>>
>> cfunc.go
>> 
>> /*
>> #include 
>> // The gateway function
>> void callOnMeGo_cgo(char *k, char *val, size_t val_len)
>> {
>> void ParsedData(const char *k, const char *val, size_t val_len);
>> ParsedData(k, val, val_len);
>> }
>> */
>> import "C"
>>
>> node.h
>> ===
>>
>> #ifndef TEST_H_INCLUDED
>> #define TEST_H_INCLUDED
>>
>> #include 
>>
>> typedef void (*key_value_cb)(const char* k, const char* val, size_t
>> val_len);
>> int load_pattern_db(const char* file, key_value_cb cb);
>>
>> #endif
>>
>> node.c
>> -
>> int load_pattern_db(const gchar* file, key_value_cb cb)
>> {
>>   patterndb = pattern_db_new();
>>   pattern_db_reload_ruleset(patterndb, configuration, file);
>>   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>>   return 0;
>> }
>>
>>
>> I am calling '*LoadPatternDB*' method in my parser.go file that makes a
>> cgo call '*C.load_pattern_db'* where I am passing a callback function to
>> the C code.
>> The C code is a wrapper code that internally calls some syslog-ng library
>> apis'
>>
>> What I observed is:
>>
>> 1)As soon as I call LoadPatternDB() method in parser.go there is some
>> increase in memory consumption(some memory leak).Ideally that should not
>> have happened.
>>
>> 2)To verify if the C code had some issue, I called the C wrapper code
>> method '*load_pattern_db*' from my main.c in the following manner to
>> completely eliminate Go code here.What I found is there is no increase in
>> memory consumption after every call ('load_pattern_db' was called 5
>> times).Hence there is no memory leak from C code.So the issue lies in the
>> Go code in '*LoadPatternDB*' method in parser.go
>>
>> main.c
>> ===
>>
>> void check(char *key, char *value, size_t value_len)
>> {
>>   printf("I am in function check\n");
>> }
>>
>> int main(void){
>> char* filename = "/home/nitish/default.xml";
>> key_value_cb s 

Re: [go-nuts] Mem-Leak in Go method

[Robert Engels]

I would also use pprof and ensure your Go code is not leaking memory. There are 
multiple tutorials on using memory profilers to detect memory leaks. 

> On Mar 16, 2020, at 9:12 AM, Robert Engels  wrote:
> 
> 
> Looks like pattern_db is a global. Are you sure you don’t have multiple Go 
> routines calling the load? The global changes may not be visible- so the free 
> is doing nothing. 
> 
> You probably need synchronization or pinning the access routine to a thread. 
> 
>>> On Mar 16, 2020, at 9:03 AM, Nitish Saboo  wrote:
>>> 
>> 
>> Hi,
>> 
>> From reading the code, I assume that the `pattern_db_new()` does some sort 
>> of allocation of a new pattern DB. I don't see any code releasing the 
>> pattern DB. Is that just missing from your post or something that some 
>> automatic mechanism does?
>> 
>> >>Apologies, that part got missed in the code snippet. I am making the free 
>> >>call in the C code.
>> 
>> node.c
>> -
>> int load_pattern_db(const gchar* file, key_value_cb cb)
>> {
>>  if(patterndb != NULL){
>> pattern_db_free(patterndb);
>>   }
>>   patterndb = pattern_db_new();
>>   pattern_db_reload_ruleset(patterndb, configuration, file);
>>   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>>   return 0;
>> }
>> 
>> You can try a similar C program that instantiates and frees the structure to 
>> check for similar behavior. 
>> 
>> >> To verify if the C code had some issue, I called the C wrapper code 
>> >> method 'load_pattern_db' from my main.c to completely eliminate Go code 
>> >> here. What I found is there is no increase in memory consumption after 
>> >> every call ('load_pattern_db' was called 5 times). Hence there is no 
>> >> issue from C code.
>> 
>> Thanks,
>> Nitish
>> 
>>> On Mon, Mar 16, 2020 at 7:04 PM Gregor Best  wrote:
>>> This might be a dumb question but...
>>> 
>>> From reading the code, I assume that the `pattern_db_new()` does some sort 
>>> of allocation of a new pattern DB. I don't see any code releasing the 
>>> pattern DB. Is that just missing from your post or something that some 
>>> automatic mechanism does?
>>> 
>>> If not, that might be your leak.
>>> 
 On 09.03.20 12:33, Nitish Saboo wrote:
 Hi
 
 Following are my Go code and C header file and C wrapper code
 
 parser.go
 ==
 var f *os.File
 
 func LoadPatternDB(patterndb string) {
 path := C.CString(patterndb)
 defer C.free(unsafe.Pointer(path))
 C.load_pattern_db(path, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))
 }
 
 //export ParsedData
 func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
 f.WriteString(C.GoString(k))
 f.WriteString("\n")
 }
 
 cfunc.go
 
 /*
 #include 
 // The gateway function
 void callOnMeGo_cgo(char *k, char *val, size_t val_len)
 {
 void ParsedData(const char *k, const char *val, size_t val_len);
 ParsedData(k, val, val_len);
 }
 */
 import "C"
 
 node.h
 ===
 
 #ifndef TEST_H_INCLUDED
 #define TEST_H_INCLUDED
 
 #include 
 
 typedef void (*key_value_cb)(const char* k, const char* val, size_t 
 val_len);
 int load_pattern_db(const char* file, key_value_cb cb);
 
 #endif
 
 node.c
 -
 int load_pattern_db(const gchar* file, key_value_cb cb)
 {
   patterndb = pattern_db_new();
   pattern_db_reload_ruleset(patterndb, configuration, file);
   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
   return 0;
 }
 
 
 I am calling 'LoadPatternDB' method in my parser.go file that makes a cgo 
 call 'C.load_pattern_db' where I am passing a callback function to the C 
 code.
 The C code is a wrapper code that internally calls some syslog-ng library 
 apis'
 
 What I observed is:
 
 1)As soon as I call LoadPatternDB() method in parser.go there is some 
 increase in memory consumption(some memory leak).Ideally that should not 
 have happened.
 
 2)To verify if the C code had some issue, I called the C wrapper code 
 method 'load_pattern_db' from my main.c in the following manner to 
 completely eliminate Go code here.What I found is there is no increase in 
 memory consumption after every call ('load_pattern_db' was called 5 
 times).Hence there is no memory leak from C code.So the issue lies in the 
 Go code in 'LoadPatternDB' method in parser.go
 
 main.c
 ===
 
 void check(char *key, char *value, size_t value_len)
 {
   printf("I am in function check\n");
 }
 
 int main(void){
 char* filename = "/home/nitish/default.xml";
 key_value_cb s = check;
 int i;
 for (i=1; i<=5; i++)
 {
 load_pattern_db(filename, s);
 printf("Sleeping for 5 second.\n");
 sleep(5);
 }
  

Re: [go-nuts] Mem-Leak in Go method

[Robert Engels]

Looks like pattern_db is a global. Are you sure you don’t have multiple Go 
routines calling the load? The global changes may not be visible- so the free 
is doing nothing. 

You probably need synchronization or pinning the access routine to a thread. 

> On Mar 16, 2020, at 9:03 AM, Nitish Saboo  wrote:
> 
> 
> Hi,
> 
> From reading the code, I assume that the `pattern_db_new()` does some sort of 
> allocation of a new pattern DB. I don't see any code releasing the pattern 
> DB. Is that just missing from your post or something that some automatic 
> mechanism does?
> 
> >>Apologies, that part got missed in the code snippet. I am making the free 
> >>call in the C code.
> 
> node.c
> -
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
>  if(patterndb != NULL){
> pattern_db_free(patterndb);
>   }
>   patterndb = pattern_db_new();
>   pattern_db_reload_ruleset(patterndb, configuration, file);
>   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>   return 0;
> }
> 
> You can try a similar C program that instantiates and frees the structure to 
> check for similar behavior. 
> 
> >> To verify if the C code had some issue, I called the C wrapper code method 
> >> 'load_pattern_db' from my main.c to completely eliminate Go code here. 
> >> What I found is there is no increase in memory consumption after every 
> >> call ('load_pattern_db' was called 5 times). Hence there is no issue from 
> >> C code.
> 
> Thanks,
> Nitish
> 
>> On Mon, Mar 16, 2020 at 7:04 PM Gregor Best  wrote:
>> This might be a dumb question but...
>> 
>> From reading the code, I assume that the `pattern_db_new()` does some sort 
>> of allocation of a new pattern DB. I don't see any code releasing the 
>> pattern DB. Is that just missing from your post or something that some 
>> automatic mechanism does?
>> 
>> If not, that might be your leak.
>> 
>>> On 09.03.20 12:33, Nitish Saboo wrote:
>>> Hi
>>> 
>>> Following are my Go code and C header file and C wrapper code
>>> 
>>> parser.go
>>> ==
>>> var f *os.File
>>> 
>>> func LoadPatternDB(patterndb string) {
>>> path := C.CString(patterndb)
>>> defer C.free(unsafe.Pointer(path))
>>> C.load_pattern_db(path, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))
>>> }
>>> 
>>> //export ParsedData
>>> func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
>>> f.WriteString(C.GoString(k))
>>> f.WriteString("\n")
>>> }
>>> 
>>> cfunc.go
>>> 
>>> /*
>>> #include 
>>> // The gateway function
>>> void callOnMeGo_cgo(char *k, char *val, size_t val_len)
>>> {
>>> void ParsedData(const char *k, const char *val, size_t val_len);
>>> ParsedData(k, val, val_len);
>>> }
>>> */
>>> import "C"
>>> 
>>> node.h
>>> ===
>>> 
>>> #ifndef TEST_H_INCLUDED
>>> #define TEST_H_INCLUDED
>>> 
>>> #include 
>>> 
>>> typedef void (*key_value_cb)(const char* k, const char* val, size_t 
>>> val_len);
>>> int load_pattern_db(const char* file, key_value_cb cb);
>>> 
>>> #endif
>>> 
>>> node.c
>>> -
>>> int load_pattern_db(const gchar* file, key_value_cb cb)
>>> {
>>>   patterndb = pattern_db_new();
>>>   pattern_db_reload_ruleset(patterndb, configuration, file);
>>>   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>>>   return 0;
>>> }
>>> 
>>> 
>>> I am calling 'LoadPatternDB' method in my parser.go file that makes a cgo 
>>> call 'C.load_pattern_db' where I am passing a callback function to the C 
>>> code.
>>> The C code is a wrapper code that internally calls some syslog-ng library 
>>> apis'
>>> 
>>> What I observed is:
>>> 
>>> 1)As soon as I call LoadPatternDB() method in parser.go there is some 
>>> increase in memory consumption(some memory leak).Ideally that should not 
>>> have happened.
>>> 
>>> 2)To verify if the C code had some issue, I called the C wrapper code 
>>> method 'load_pattern_db' from my main.c in the following manner to 
>>> completely eliminate Go code here.What I found is there is no increase in 
>>> memory consumption after every call ('load_pattern_db' was called 5 
>>> times).Hence there is no memory leak from C code.So the issue lies in the 
>>> Go code in 'LoadPatternDB' method in parser.go
>>> 
>>> main.c
>>> ===
>>> 
>>> void check(char *key, char *value, size_t value_len)
>>> {
>>>   printf("I am in function check\n");
>>> }
>>> 
>>> int main(void){
>>> char* filename = "/home/nitish/default.xml";
>>> key_value_cb s = check;
>>> int i;
>>> for (i=1; i<=5; i++)
>>> {
>>> load_pattern_db(filename, s);
>>> printf("Sleeping for 5 second.\n");
>>> sleep(5);
>>> }
>>> printf("Loading done 5 times.\n");
>>> return 0;
>>> }
>>> 
>>> 3)Can someone please guide me and help me figure out the mem-leak in 
>>> 'LoadPatternDB' method in parser.go at very first glance? Is the callback 
>>> function pointer an issue here ?
>>> 
>>> 4)What tool can I use to check this mem-leak ?
>>> 
>>> Thanks,
>>> Nitish
>>> -- 
>>> You received 

Re: [go-nuts] Re: Mem-Leak in Go method

[Volker Dobler]

On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:
>
> Hi,
>
> I upgraded the go version and compiled the binary against go version 'go 
> version go1.12.4 linux/amd64'.
> I ran the program for some time. I made almost 30-40 calls to the method 
> Load_Pattern_Db().
> The program starts with 6% Mem Usage. The memory usage increases only when 
> I call 'LoadPatternDb()' method and LoadPatternDb() method is called by a 
> goroutine at regular intervals of 3 minutes(making use of ticker here ).
>
> What I observed is:
>
> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory 
> usage got almost constant at 29%. But I did not expect the program to take 
> this much memory.
>When I restart the service the Mem Usage again starts with 6%.
>
> a) Is this the sign of memory leaking?
>

No, as explained above.
 

>
> b) Till this moment I did not see memory getting reclaimed or going down 
> but it did become constant.
> As mentioned by experts above, the same sort of behavior is seen here. But 
> I did not expect the memory usage to grow this much. Is this expected? 
>
Yes. (Well, no. But your gut feeling of how much memory
should grow is not a suitable benchmark to compare
actual growth to.)
 

>
> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc) as 
> mentioned in the earlier email.
>
> a) Which all mem-stats variables should I look into for debugging this 
> kind of behavior?
>
Alloc/HeapAlloc 
But probably this is plain useless as nothing here indicates
that you do have any memory issues.

V.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com.

Re: [go-nuts] Mem-Leak in Go method

[Nitish Saboo]

Hi,

>From reading the code, I assume that the `pattern_db_new()` does some sort
of allocation of a new pattern DB. I don't see any code releasing the
pattern DB. Is that just missing from your post or something that some
automatic mechanism does?

>>Apologies, that part got missed in the code snippet. I am making the free
call in the C code.

node.c
-
int load_pattern_db(const gchar* file, key_value_cb cb)
{
 if(patterndb != NULL){
pattern_db_free(patterndb);
  }
  patterndb = pattern_db_new();
  pattern_db_reload_ruleset(patterndb, configuration, file);
  pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
  return 0;
}

You can try a similar C program that instantiates and frees the structure
to check for similar behavior.

>> To verify if the C code had some issue, I called the C wrapper code
method '*load_pattern_db*' from my main.c to completely eliminate Go code
here. What I found is there is no increase in memory consumption after
every call ('load_pattern_db' was called 5 times). Hence there is no issue
from C code.

Thanks,
Nitish

On Mon, Mar 16, 2020 at 7:04 PM Gregor Best  wrote:

> This might be a dumb question but...
>
> From reading the code, I assume that the `pattern_db_new()` does some sort
> of allocation of a new pattern DB. I don't see any code releasing the
> pattern DB. Is that just missing from your post or something that some
> automatic mechanism does?
>
> If not, that might be your leak.
> On 09.03.20 12:33, Nitish Saboo wrote:
>
> Hi
>
> Following are my Go code and C header file and C wrapper code
>
> parser.go
> ==
> var f *os.File
>
> func LoadPatternDB(patterndb string) {
> path := C.CString(patterndb)
> defer C.free(unsafe.Pointer(path))
> C.load_pattern_db(path, (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))
> }
>
> //export ParsedData
> func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
> f.WriteString(C.GoString(k))
> f.WriteString("\n")
> }
>
> cfunc.go
> 
> /*
> #include 
> // The gateway function
> void callOnMeGo_cgo(char *k, char *val, size_t val_len)
> {
> void ParsedData(const char *k, const char *val, size_t val_len);
> ParsedData(k, val, val_len);
> }
> */
> import "C"
>
> node.h
> ===
>
> #ifndef TEST_H_INCLUDED
> #define TEST_H_INCLUDED
>
> #include 
>
> typedef void (*key_value_cb)(const char* k, const char* val, size_t
> val_len);
> int load_pattern_db(const char* file, key_value_cb cb);
>
> #endif
>
> node.c
> -
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
>   patterndb = pattern_db_new();
>   pattern_db_reload_ruleset(patterndb, configuration, file);
>   pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>   return 0;
> }
>
>
> I am calling '*LoadPatternDB*' method in my parser.go file that makes a
> cgo call '*C.load_pattern_db'* where I am passing a callback function to
> the C code.
> The C code is a wrapper code that internally calls some syslog-ng library
> apis'
>
> What I observed is:
>
> 1)As soon as I call LoadPatternDB() method in parser.go there is some
> increase in memory consumption(some memory leak).Ideally that should not
> have happened.
>
> 2)To verify if the C code had some issue, I called the C wrapper code
> method '*load_pattern_db*' from my main.c in the following manner to
> completely eliminate Go code here.What I found is there is no increase in
> memory consumption after every call ('load_pattern_db' was called 5
> times).Hence there is no memory leak from C code.So the issue lies in the
> Go code in '*LoadPatternDB*' method in parser.go
>
> main.c
> ===
>
> void check(char *key, char *value, size_t value_len)
> {
>   printf("I am in function check\n");
> }
>
> int main(void){
> char* filename = "/home/nitish/default.xml";
> key_value_cb s = check;
> int i;
> for (i=1; i<=5; i++)
> {
> load_pattern_db(filename, s);
> printf("Sleeping for 5 second.\n");
> sleep(5);
> }
> printf("Loading done 5 times.\n");
> return 0;
> }
>
> 3)Can someone please guide me and help me figure out the mem-leak in
> 'LoadPatternDB' method in parser.go at very first glance? Is the callback
> function pointer an issue here ?
>
> 4)What tool can I use to check this mem-leak ?
>
> Thanks,
> Nitish
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CALjMrq5cAJ19CQ8OmMPSJmVB1P3t4hE0CKZ4HsEsH-mp6zm3Ng%40mail.gmail.com
> 
> .
>
> --
> --
>   Gregor Best
>   b...@pferdewetten.de
>
> --
> You received this message because you are subscribed to the Google Groups
> 

Re: [go-nuts] Mem-Leak in Go method

[Gregor Best]

This might be a dumb question but...

From reading the code, I assume that the `pattern_db_new()` does some 
sort of allocation of a new pattern DB. I don't see any code releasing 
the pattern DB. Is that just missing from your post or something that 
some automatic mechanism does?


If not, that might be your leak.

On 09.03.20 12:33, Nitish Saboo wrote:

Hi

Following are my Go code and C header file and C wrapper code

parser.go
==
var f *os.File

func LoadPatternDB(patterndb string) {
path := C.CString(patterndb)
defer C.free(unsafe.Pointer(path))
C.load_pattern_db(path, 
(C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))

}

//export ParsedData
func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
f.WriteString(C.GoString(k))
f.WriteString("\n")
}

cfunc.go

/*
#include 
// The gateway function
void callOnMeGo_cgo(char *k, char *val, size_t val_len)
{
void ParsedData(const char *k, const char *val, size_t val_len);
ParsedData(k, val, val_len);
}
*/
import "C"

node.h
===

#ifndef TEST_H_INCLUDED
#define TEST_H_INCLUDED

#include 

typedef void (*key_value_cb)(const char* k, const char* val, size_t 
val_len);

int load_pattern_db(const char* file, key_value_cb cb);

#endif

node.c
-
int load_pattern_db(const gchar* file, key_value_cb cb)
{
  patterndb = pattern_db_new();
  pattern_db_reload_ruleset(patterndb, configuration, file);
  pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
  return 0;
}


I am calling '*LoadPatternDB*' method in my parser.go file that makes 
a cgo call '*C.load_pattern_db'* where I am passing a callback 
function to the C code.
The C code is a wrapper code that internally calls some syslog-ng 
library apis'


What I observed is:

1)As soon as I call LoadPatternDB() method in parser.go there is some 
increase in memory consumption(some memory leak).Ideally that should 
not have happened.


2)To verify if the C code had some issue, I called the C wrapper code 
method '*load_pattern_db*' from my main.c in the following manner to 
completely eliminate Go code here.What I found is there is no increase 
in memory consumption after every call ('load_pattern_db' was called 5 
times).Hence there is no memory leak from C code.So the issue lies in 
the Go code in '*LoadPatternDB*' method in parser.go


main.c
===

void check(char *key, char *value, size_t value_len)
{
  printf("I am in function check\n");
}

int main(void){
        char* filename = "/home/nitish/default.xml";
        key_value_cb s = check;
        int i;
    for (i=1; i<=5; i++)
    {
        load_pattern_db(filename, s);
        printf("Sleeping for 5 second.\n");
        sleep(5);
    }
    printf("Loading done 5 times.\n");
        return 0;
}

3)Can someone please guide me and help me figure out the mem-leak in 
'LoadPatternDB' method in parser.go at very first glance? Is the 
callback function pointer an issue here ?


4)What tool can I use to check this mem-leak ?

Thanks,
Nitish
--
You received this message because you are subscribed to the Google 
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to golang-nuts+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CALjMrq5cAJ19CQ8OmMPSJmVB1P3t4hE0CKZ4HsEsH-mp6zm3Ng%40mail.gmail.com 
.


--
--
  Gregor Best
  b...@pferdewetten.de

--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/fee07a13-e3d7-4a2e-6a6c-fdb11140ce21%40pferdewetten.de.

Re: [go-nuts] Re: Mem-Leak in Go method

[Robert Engels]

Sounds like it. Probably in the C code. You need to check that your 
release/free code is correct. 

You can try a similar C program that instantiates and frees the structure to 
check for similar behavior. 

> On Mar 16, 2020, at 8:25 AM, Nitish Saboo  wrote:
> 
> 
> Hi,
> 
> I upgraded the go version and compiled the binary against go version 'go 
> version go1.12.4 linux/amd64'.
> I ran the program for some time. I made almost 30-40 calls to the method 
> Load_Pattern_Db().
> The program starts with 6% Mem Usage. The memory usage increases only when I 
> call 'LoadPatternDb()' method and LoadPatternDb() method is called by a 
> goroutine at regular intervals of 3 minutes(making use of ticker here ).
> 
> What I observed is:
> 
> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory usage 
> got almost constant at 29%. But I did not expect the program to take this 
> much memory.
>When I restart the service the Mem Usage again starts with 6%.
> 
> a) Is this the sign of memory leaking?
> 
> b) Till this moment I did not see memory getting reclaimed or going down but 
> it did become constant.
> As mentioned by experts above, the same sort of behavior is seen here. But I 
> did not expect the memory usage to grow this much. Is this expected?
> 
> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc) as 
> mentioned in the earlier email.
> 
> a) Which all mem-stats variables should I look into for debugging this kind 
> of behavior?
> 
> Thanks,
> Nitish
> 
>> On Fri, Mar 13, 2020 at 6:22 PM Michael Jones  
>> wrote:
>> hi. get the time at the start, check the elapsed time in your infinite loop, 
>> and trigger the write/exit after a minute, 10 minutes, 100 minutes, ...
>> 
>>> On Fri, Mar 13, 2020 at 5:45 AM Nitish Saboo  
>>> wrote:
>>> Hi Michael,
>>> 
>>> Thanks for your response.
>>> 
>>> That code looks wrong. I see the end but not the start. Look here and copy 
>>> carefully:
>>> 
>>> >>Since I did not want cpu profiling I omitted the start of the code and 
>>> >>just added memory profiling part.
>>> 
>>> Call at end, on way out.
>>> 
>>> >>Oh yes, I missed that.I have to call memory profiling code at the end on 
>>> >>the way out.But the thing is that it runs as a service in infinite for 
>>> >>loop.
>>> 
>>> func main() {
>>> flag.Parse()
>>> if *cpuprofile != "" {
>>> f, err := os.Create(*cpuprofile)
>>> if err != nil {
>>> fmt.Println("could not create CPU profile: ", err)
>>> }
>>> defer f.Close() // error handling omitted for example
>>> if err := pprof.StartCPUProfile(f); err != nil {
>>> fmt.Print("could not start CPU profile: ", err)
>>> }
>>> defer pprof.StopCPUProfile()
>>> }
>>> 
>>> A_chan := make(chan bool)
>>> B_chan := make(chan bool)
>>> go util.A(A_chan)
>>> go util.B(B_chan)
>>> (..Rest of the code..)
>>> 
>>> for {
>>> select {
>>> case <-A_chan: 
>>> continue
>>> case <-B_chan: 
>>> continue
>>> 
>>> }
>>> }
>>> 
>>> }
>>> 
>>> What would be the correct way to add the memprofile code changes, since it 
>>> is running in an infinite for loop ?
>>> 
>>> Also, as shared by others above, there are no promises about how soon the 
>>> dead allocations go away, The speed gets faster and faster version to 
>>> version, and is impressive indeed now, so old versions are not the best to 
>>> use, ubt even so, if the allocation feels small to th GC the urgency to 
>>> free it will be low. You need to loop in allocating and see if the memory 
>>> grows and grows.
>>> 
>>> >> Yes, got it.I will try using the latest version of Go and check the 
>>> >> behavior.
>>> 
>>> Thanks,
>>> Nitish
>>> 
 On Fri, Mar 13, 2020 at 6:20 AM Michael Jones  
 wrote:
 That code looks wrong. I see the end but not the start. Look here and copy 
 carefully:
 https://golang.org/pkg/runtime/pprof/
 
 Call at end, on way out.
 
 Also, as shared by others above, there are no promises about how soon the 
 dead allocations go away, The speed gets faster and faster version to 
 version, and is impressive indeed now, so old versions are not the best to 
 use, ubt even so, if the allocation feels small to th GC the urgency to 
 free it will be low. You need to loop in allocating and see if the memory 
 grows and grows.
 
> On Thu, Mar 12, 2020 at 9:22 AM Nitish Saboo  
> wrote:
> Hi,
> 
> I have compiled my Go binary against go version 'go1.7 linux/amd64'.
> I added the following code change in the main function to get the memory 
> profiling of my service 
> 
> var memprofile = flag.String("memprofile", "", "write memory profile to 
> `file`")
> 
> func main() {
> flag.Parse()
> if *memprofile != "" {
> f, err := os.Create(*memprofile)
> if err != nil {
> fmt.Println("could not create memory profile: ", err)
> }
> defer f.Close() // error handling omitted for example
> runtime.GC() // get up-to-date statistics
> if err := 

Re: [go-nuts] Re: Mem-Leak in Go method

[Nitish Saboo]

Hi,

I upgraded the go version and compiled the binary against go version 'go
version go1.12.4 linux/amd64'.
I ran the program for some time. I made almost 30-40 calls to the method
Load_Pattern_Db().
The program starts with 6% Mem Usage. The memory usage increases only when
I call 'LoadPatternDb()' method and LoadPatternDb() method is called by a
goroutine at regular intervals of 3 minutes(making use of ticker here ).

What I observed is:

1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory usage
got almost constant at 29%. But I did not expect the program to take this
much memory.
   When I restart the service the Mem Usage again starts with 6%.

a) Is this the sign of memory leaking?

b) Till this moment I did not see memory getting reclaimed or going down
but it did become constant.
As mentioned by experts above, the same sort of behavior is seen here. But
I did not expect the memory usage to grow this much. Is this expected?

2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc) as
mentioned in the earlier email.

a) Which all mem-stats variables should I look into for debugging this kind
of behavior?

Thanks,
Nitish

On Fri, Mar 13, 2020 at 6:22 PM Michael Jones 
wrote:

> hi. get the time at the start, check the elapsed time in your infinite
> loop, and trigger the write/exit after a minute, 10 minutes, 100 minutes,
> ...
>
> On Fri, Mar 13, 2020 at 5:45 AM Nitish Saboo 
> wrote:
>
>> Hi Michael,
>>
>> Thanks for your response.
>>
>> That code looks wrong. I see the end but not the start. Look here and
>> copy carefully:
>>
>> >>Since I did not want cpu profiling I omitted the start of the code and
>> just added memory profiling part.
>>
>> Call at end, on way out.
>>
>> >>Oh yes, I missed that.I have to call memory profiling code at the end
>> on the way out.But the thing is that it runs as a service in infinite for
>> loop.
>>
>> func main() {
>> flag.Parse()
>> if *cpuprofile != "" {
>> f, err := os.Create(*cpuprofile)
>> if err != nil {
>> fmt.Println("could not create CPU profile: ", err)
>> }
>> defer f.Close() // error handling omitted for example
>> if err := pprof.StartCPUProfile(f); err != nil {
>> fmt.Print("could not start CPU profile: ", err)
>> }
>> defer pprof.StopCPUProfile()
>> }
>>
>> A_chan := make(chan bool)
>> B_chan := make(chan bool)
>> go util.A(A_chan)
>> go util.B(B_chan)
>> (..Rest of the code..)
>>
>> for {
>> select {
>> case <-A_chan:
>> continue
>> case <-B_chan:
>> continue
>>
>> }
>> }
>>
>> }
>>
>> What would be the correct way to add the memprofile code changes, since
>> it is running in an infinite for loop ?
>>
>> Also, as shared by others above, there are no promises about how soon the
>> dead allocations go away, The speed gets faster and faster version to
>> version, and is impressive indeed now, so old versions are not the best to
>> use, ubt even so, if the allocation feels small to th GC the urgency to
>> free it will be low. You need to loop in allocating and see if the memory
>> grows and grows.
>>
>> >> Yes, got it.I will try using the latest version of Go and check the
>> behavior.
>>
>> Thanks,
>> Nitish
>>
>> On Fri, Mar 13, 2020 at 6:20 AM Michael Jones 
>> wrote:
>>
>>> That code looks wrong. I see the end but not the start. Look here and
>>> copy carefully:
>>> https://golang.org/pkg/runtime/pprof/
>>>
>>> Call at end, on way out.
>>>
>>> Also, as shared by others above, there are no promises about how soon
>>> the dead allocations go away, The speed gets faster and faster version to
>>> version, and is impressive indeed now, so old versions are not the best to
>>> use, ubt even so, if the allocation feels small to th GC the urgency to
>>> free it will be low. You need to loop in allocating and see if the memory
>>> grows and grows.
>>>
>>> On Thu, Mar 12, 2020 at 9:22 AM Nitish Saboo 
>>> wrote:
>>>
 Hi,

 I have compiled my Go binary against go version 'go1.7 linux/amd64'.
 I added the following code change in the main function to get the
 memory profiling of my service

 var memprofile = flag.String("memprofile", "", "write memory profile to
 `file`")

 func main() {
 flag.Parse()
 if *memprofile != "" {
 f, err := os.Create(*memprofile)
 if err != nil {
 fmt.Println("could not create memory profile: ", err)
 }
 defer f.Close() // error handling omitted for example
 runtime.GC() // get up-to-date statistics
 if err := pprof.WriteHeapProfile(f); err != nil {
 fmt.Println("could not write memory profile: ", err)
 }
 }
 ..
 ..
 (Rest code to follow)

 I ran the binary with the following command:

 nsaboo@ubuntu:./main -memprofile=mem.prof

 After running the service for couple of minutes, I stopped it and got
 the file 'mem.prof'

 1)mem.prof contains the following:

 nsaboo@ubuntu:~/Desktop/memprof$ vim mem.prof

 heap profile: 0: 0 [0: 0] @ heap/1048576