Hi,
Yeah, even I did not expect the program to have 29% memory usage.Not sure
if this is how the Go GC works.
Thanks,
Nitish
On Wed, Mar 18, 2020 at 12:48 AM Robert Engels <reng...@ix.netcom.com>
wrote:
> My only thought was that maybe you had more Go routines accessing it than
> you thought.
>
> It is remains constant after a while it is most likely not a memory leak.
> It is done what surprising that the memory consumption in a steady state
> would be 4x the equivalent C program.
>
> On Mar 17, 2020, at 9:21 AM, Nitish Saboo <nitish.sabo...@gmail.com>
> wrote:
>
>
> Hi Robert,
>
> Thanks for your response.
> Since patterndb is a global variable(not a thread-local variable) and I
> have a single goroutine that calls load_pattern_db() method, therefore it
> was not looking correct to me to pin a goroutine to a thread.
> I once again tested the code flow. Apologies for making confusion in my
> last mail. When I called load_pattern_db() for about 6-7 times, every time
> the following lines were getting printed. It looks like patterndb instance
> is getting freed and the memory is becoming constant at around 29%.
>
> Patterndb Free Entered
> Patterndb Free called
> Patterndb New called
>
> node.c
> ---------
>
> PatternDB *patterndb;
>
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
> printf("Patterndb Free Entered\n")
> if(patterndb != NULL){
> printf("Patterndb Free called\n"); <<< It is getting printed
> pattern_db_free(patterndb);
> }
> printf("Patterndb New called\n")
> patterndb = pattern_db_new();
> pattern_db_reload_ruleset(patterndb, configuration, file);
> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
> return 0;
> }
>
> But, what made you feel that Go global variable would report a race
> condition? Since it is a single goroutine what would cause a race condition
> here ?
>
> Thanks,
> Nitish
>
> On Tue, Mar 17, 2020 at 6:31 PM Robert Engels <reng...@ix.netcom.com>
> wrote:
>
>> I’ve been thinking about this some more, and I think that LockOSThread()
>> should not be needed - that the Go thread multiplexing must perform memory
>> fences otherwise the simplest of Go apps would have concurrency issues.
>>
>> So, that leads me to believe that your “single routine” is not correct. I
>> would add code on the Go side that does similar Go global variable handling
>> at the call site for the C call. Then run under the race detector - I’m
>> guessing that it will report a race on the Go global.
>>
>> On Mar 16, 2020, at 2:46 PM, Robert Engels <reng...@ix.netcom.com> wrote:
>>
>> In the single Go routine, use LockOSThread(). Then it was always be
>> accessed on the same thread removing the memory synchronization problems.
>>
>> On Mar 16, 2020, at 11:28 AM, Nitish Saboo <nitish.sabo...@gmail.com>
>> wrote:
>>
>>
>> Hi,
>>
>> So finally I got a little hint of the problem from what Robert described
>> earlier in the mail. Thank you so much Robert.
>> Looks like patterndb instance is not getting freed.
>>
>> node.c
>> ---------
>>
>> PatternDB *patterndb;
>>
>> int load_pattern_db(const gchar* file, key_value_cb cb)
>> {
>> if(patterndb != NULL){
>> printf("Patterndb Free called\n"); <<< Not getting printed
>> pattern_db_free(patterndb);
>> }
>> patterndb = pattern_db_new();
>> pattern_db_reload_ruleset(patterndb, configuration, file);
>> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>> return 0;
>> }
>>
>>
>> patterndb is a global variable in C wrapper code that internally calls
>> some syslog-ng library api's.Since load_pattern_db() method is getting
>> called from a single goroutine every 3 mins, patterndb instance is not
>> getting free because the statement inside if clause ('if(patterndb !=
>> NULL)') is not getting printed when I call 'load_pattern_db()' method.Looks
>> like that is the leak here.
>>
>>
>> 1)Can someone please help me understand the problem in detail as in why
>> am I facing this issue?
>>
>> 2)Though patterndb instance is a global variable in the C wrapper code,
>> why is it not getting freed?
>>
>> 3)How can I fix this issue?
>>
>> Thanks,
>> Nitish
>>
>> On Mon, Mar 16, 2020 at 8:17 PM Nitish Saboo <nitish.sabo...@gmail.com>
>> wrote:
>>
>>> Hi Robert,
>>>
>>> Sorry I did not understand your point completely.
>>> I have a global variable patterndb on C side and It is getting called
>>> from a single goroutine every 3 mins. Why do I need to synchronize it?
>>> Even though the goroutine gets pinned to different threads, it can
>>> access the same global variable every time and free it ...right ?
>>>
>>> Thanks,
>>> Nitish
>>>
>>>
>>> On Mon, Mar 16, 2020 at 8:10 PM Robert Engels <reng...@ix.netcom.com>
>>> wrote:
>>>
>>>> Yes, you have a shared global variable you need to synchronize.
>>>>
>>>> On Mar 16, 2020, at 9:35 AM, Nitish Saboo <nitish.sabo...@gmail.com>
>>>> wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> Are you saying it is working as expected?
>>>>
>>>> Thanks,
>>>> Nitish
>>>>
>>>> On Mon, Mar 16, 2020 at 7:42 PM Volker Dobler <
>>>> dr.volker.dob...@gmail.com> wrote:
>>>>
>>>>> On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I upgraded the go version and compiled the binary against go version
>>>>>> 'go version go1.12.4 linux/amd64'.
>>>>>> I ran the program for some time. I made almost 30-40 calls to the
>>>>>> method Load_Pattern_Db().
>>>>>> The program starts with 6% Mem Usage. The memory usage increases only
>>>>>> when I call 'LoadPatternDb()' method and LoadPatternDb() method is called
>>>>>> by a goroutine at regular intervals of 3 minutes(making use of ticker
>>>>>> here
>>>>>> ).
>>>>>>
>>>>>> What I observed is:
>>>>>>
>>>>>> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory
>>>>>> usage got almost constant at 29%. But I did not expect the program to
>>>>>> take
>>>>>> this much memory.
>>>>>> When I restart the service the Mem Usage again starts with 6%.
>>>>>>
>>>>>> a) Is this the sign of memory leaking?
>>>>>>
>>>>>
>>>>> No, as explained above.
>>>>>
>>>>>
>>>>>>
>>>>>> b) Till this moment I did not see memory getting reclaimed or going
>>>>>> down but it did become constant.
>>>>>> As mentioned by experts above, the same sort of behavior is seen
>>>>>> here. But I did not expect the memory usage to grow this much. Is this
>>>>>> expected?
>>>>>>
>>>>> Yes. (Well, no. But your gut feeling of how much memory
>>>>> should grow is not a suitable benchmark to compare
>>>>> actual growth to.)
>>>>>
>>>>>
>>>>>>
>>>>>> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc)
>>>>>> as mentioned in the earlier email.
>>>>>>
>>>>>> a) Which all mem-stats variables should I look into for debugging
>>>>>> this kind of behavior?
>>>>>>
>>>>> Alloc/HeapAlloc
>>>>> But probably this is plain useless as nothing here indicates
>>>>> that you do have any memory issues.
>>>>>
>>>>> V.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "golang-nuts" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com
>>>>> <https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com
>>>> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/2BD0A731-0F46-44DF-AEDE-8CC2F182D1B3%40ix.netcom.com
>> <https://groups.google.com/d/msgid/golang-nuts/2BD0A731-0F46-44DF-AEDE-8CC2F182D1B3%40ix.netcom.com?utm_medium=email&utm_source=footer>
>> .
>>
>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CALjMrq5DwL%3DMkXSubzhrs1RU32ps272wOdowxKG4LsW20kGHug%40mail.gmail.com.
