Hi Robert,
Thanks for your response.
Since patterndb is a global variable(not a thread-local variable) and I
have a single goroutine that calls load_pattern_db() method, therefore it
was not looking correct to me to pin a goroutine to a thread.
I once again tested the code flow. Apologies for making confusion in my
last mail. When I called load_pattern_db() for about 6-7 times, every time
the following lines were getting printed. It looks like patterndb instance
is getting freed and the memory is becoming constant at around 29%.
Patterndb Free Entered
Patterndb Free called
Patterndb New called
node.c
---------
PatternDB *patterndb;
int load_pattern_db(const gchar* file, key_value_cb cb)
{
printf("Patterndb Free Entered\n")
if(patterndb != NULL){
printf("Patterndb Free called\n"); <<< It is getting printed
pattern_db_free(patterndb);
}
printf("Patterndb New called\n")
patterndb = pattern_db_new();
pattern_db_reload_ruleset(patterndb, configuration, file);
pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
return 0;
}
But, what made you feel that Go global variable would report a race
condition? Since it is a single goroutine what would cause a race condition
here ?
Thanks,
Nitish
On Tue, Mar 17, 2020 at 6:31 PM Robert Engels <reng...@ix.netcom.com> wrote:
> I’ve been thinking about this some more, and I think that LockOSThread()
> should not be needed - that the Go thread multiplexing must perform memory
> fences otherwise the simplest of Go apps would have concurrency issues.
>
> So, that leads me to believe that your “single routine” is not correct. I
> would add code on the Go side that does similar Go global variable handling
> at the call site for the C call. Then run under the race detector - I’m
> guessing that it will report a race on the Go global.
>
> On Mar 16, 2020, at 2:46 PM, Robert Engels <reng...@ix.netcom.com> wrote:
>
> In the single Go routine, use LockOSThread(). Then it was always be
> accessed on the same thread removing the memory synchronization problems.
>
> On Mar 16, 2020, at 11:28 AM, Nitish Saboo <nitish.sabo...@gmail.com>
> wrote:
>
>
> Hi,
>
> So finally I got a little hint of the problem from what Robert described
> earlier in the mail. Thank you so much Robert.
> Looks like patterndb instance is not getting freed.
>
> node.c
> ---------
>
> PatternDB *patterndb;
>
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
> if(patterndb != NULL){
> printf("Patterndb Free called\n"); <<< Not getting printed
> pattern_db_free(patterndb);
> }
> patterndb = pattern_db_new();
> pattern_db_reload_ruleset(patterndb, configuration, file);
> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
> return 0;
> }
>
>
> patterndb is a global variable in C wrapper code that internally calls
> some syslog-ng library api's.Since load_pattern_db() method is getting
> called from a single goroutine every 3 mins, patterndb instance is not
> getting free because the statement inside if clause ('if(patterndb !=
> NULL)') is not getting printed when I call 'load_pattern_db()' method.Looks
> like that is the leak here.
>
>
> 1)Can someone please help me understand the problem in detail as in why am
> I facing this issue?
>
> 2)Though patterndb instance is a global variable in the C wrapper code,
> why is it not getting freed?
>
> 3)How can I fix this issue?
>
> Thanks,
> Nitish
>
> On Mon, Mar 16, 2020 at 8:17 PM Nitish Saboo <nitish.sabo...@gmail.com>
> wrote:
>
>> Hi Robert,
>>
>> Sorry I did not understand your point completely.
>> I have a global variable patterndb on C side and It is getting called
>> from a single goroutine every 3 mins. Why do I need to synchronize it?
>> Even though the goroutine gets pinned to different threads, it can access
>> the same global variable every time and free it ...right ?
>>
>> Thanks,
>> Nitish
>>
>>
>> On Mon, Mar 16, 2020 at 8:10 PM Robert Engels <reng...@ix.netcom.com>
>> wrote:
>>
>>> Yes, you have a shared global variable you need to synchronize.
>>>
>>> On Mar 16, 2020, at 9:35 AM, Nitish Saboo <nitish.sabo...@gmail.com>
>>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> Are you saying it is working as expected?
>>>
>>> Thanks,
>>> Nitish
>>>
>>> On Mon, Mar 16, 2020 at 7:42 PM Volker Dobler <
>>> dr.volker.dob...@gmail.com> wrote:
>>>
>>>> On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I upgraded the go version and compiled the binary against go version
>>>>> 'go version go1.12.4 linux/amd64'.
>>>>> I ran the program for some time. I made almost 30-40 calls to the
>>>>> method Load_Pattern_Db().
>>>>> The program starts with 6% Mem Usage. The memory usage increases only
>>>>> when I call 'LoadPatternDb()' method and LoadPatternDb() method is called
>>>>> by a goroutine at regular intervals of 3 minutes(making use of ticker here
>>>>> ).
>>>>>
>>>>> What I observed is:
>>>>>
>>>>> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory
>>>>> usage got almost constant at 29%. But I did not expect the program to take
>>>>> this much memory.
>>>>> When I restart the service the Mem Usage again starts with 6%.
>>>>>
>>>>> a) Is this the sign of memory leaking?
>>>>>
>>>>
>>>> No, as explained above.
>>>>
>>>>
>>>>>
>>>>> b) Till this moment I did not see memory getting reclaimed or going
>>>>> down but it did become constant.
>>>>> As mentioned by experts above, the same sort of behavior is seen here.
>>>>> But I did not expect the memory usage to grow this much. Is this expected?
>>>>>
>>>> Yes. (Well, no. But your gut feeling of how much memory
>>>> should grow is not a suitable benchmark to compare
>>>> actual growth to.)
>>>>
>>>>
>>>>>
>>>>> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc)
>>>>> as mentioned in the earlier email.
>>>>>
>>>>> a) Which all mem-stats variables should I look into for debugging this
>>>>> kind of behavior?
>>>>>
>>>> Alloc/HeapAlloc
>>>> But probably this is plain useless as nothing here indicates
>>>> that you do have any memory issues.
>>>>
>>>> V.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "golang-nuts" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/2BD0A731-0F46-44DF-AEDE-8CC2F182D1B3%40ix.netcom.com
> <https://groups.google.com/d/msgid/golang-nuts/2BD0A731-0F46-44DF-AEDE-8CC2F182D1B3%40ix.netcom.com?utm_medium=email&utm_source=footer>
> .
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CALjMrq7z5m2uyLc%3DO%3DeFzcgg3__687YCfeKhtR4JgHETigDJRA%40mail.gmail.com.
