Hi,
Are there any other commands that provide an exact allocation of memory for
each of the functions or help to analyze the memory allocation much better?
Thanks,
Nitish
On Thu, Mar 19, 2020 at 7:08 PM Robert Engels <reng...@ix.netcom.com> wrote:
> You are only using 1.5 mb on the Go side... so if your process is
> consuming lots of memory it’s on the C side.
>
> On Mar 19, 2020, at 7:55 AM, Nitish Saboo <nitish.sabo...@gmail.com>
> wrote:
>
>
> Hi Michael,
>
> I used something like this to generate a mem-prof for 60 minutes
>
> func main() {
> flag.Parse()
> if *cpuprofile != "" {
> f, err := os.Create(*cpuprofile)
> if err != nil {
> fmt.Println("could not create CPU profile: ", err)
> }
> defer f.Close() // error handling omitted for example
> if err := pprof.StartCPUProfile(f); err != nil {
> fmt.Print("could not start CPU profile: ", err)
> }
> defer pprof.StopCPUProfile()
> }
> timeout := time.After(60 * time.Minute)
> A_chan := make(chan bool)
> B_chan := make(chan bool)
> go util.A(A_chan)
> go util.B(B_chan)
> (..Rest of the code..)
>
> for {
> select {
> case <-A_chan:
> continue
> case <-B_chan:
> continue
> case <-timeout:
> break
> }
> break
> }
>
> if *memprofile != "" {
> count = count + 1
> fmt.Println("Generating Mem Profile:")
> fmt.Print(count)
> f, err := os.Create(*memprofile)
> if err != nil {
> fmt.Println("could not create memory profile: ", err)
> }
> defer f.Close() // error handling omitted for example
> runtime.GC() // get up-to-date statistics
> if err := pprof.WriteHeapProfile(f); err != nil {
> fmt.Println("could not write memory profile: ", err)
> }
>
> }
> }
>
> I got the following output from the mem.prof:
>
> ~/Desktop/memprof:go tool pprof mem.prof
> File: main
> Build ID: 331d79200cabd2a81713918e51b8c9a63e3f7d29
> Type: inuse_space
> Time: Mar 19, 2020 at 3:57pm (IST)
> Entering interactive mode (type "help" for commands, "o" for options)
> (pprof) top 14
> Showing nodes accounting for 1581.40kB, 100% of 1581.40kB total
> flat flat% sum% cum cum%
> 1024.14kB 64.76% 64.76% 1024.14kB 64.76%
> github.com/aws/aws-sdk-go/aws/endpoints.init.ializers
> 557.26kB 35.24% 100% 557.26kB 35.24% crypto/elliptic.initTable
> 0 0% 100% 557.26kB 35.24%
> crypto/elliptic.(*p256Point).p256BaseMult
> 0 0% 100% 557.26kB 35.24% crypto/elliptic.GenerateKey
> 0 0% 100% 557.26kB 35.24%
> crypto/elliptic.p256Curve.ScalarBaseMult
> 0 0% 100% 557.26kB 35.24% crypto/tls.(*Conn).Handshake
> 0 0% 100% 557.26kB 35.24%
> crypto/tls.(*Conn).clientHandshake
> 0 0% 100% 557.26kB 35.24%
> crypto/tls.(*clientHandshakeState).doFullHandshake
> 0 0% 100% 557.26kB 35.24%
> crypto/tls.(*clientHandshakeState).handshake
> 0 0% 100% 557.26kB 35.24%
> crypto/tls.(*ecdheKeyAgreement).processServerKeyExchange
> 0 0% 100% 557.26kB 35.24%
> crypto/tls.generateECDHEParameters
> 0 0% 100% 557.26kB 35.24%
> net/http.(*persistConn).addTLS.func2
> 0 0% 100% 1024.14kB 64.76% runtime.main
> 0 0% 100% 557.26kB 35.24% sync.(*Once).Do
>
> (pprof)
>
> Can you please share some commands or any link which I can refer to to
> analyze the data?
>
> Thanks,
> Nitish
>
>
>
> On Fri, Mar 13, 2020 at 6:22 PM Michael Jones <michael.jo...@gmail.com>
> wrote:
>
>> hi. get the time at the start, check the elapsed time in your infinite
>> loop, and trigger the write/exit after a minute, 10 minutes, 100 minutes,
>> ...
>>
>> On Fri, Mar 13, 2020 at 5:45 AM Nitish Saboo <nitish.sabo...@gmail.com>
>> wrote:
>>
>>> Hi Michael,
>>>
>>> Thanks for your response.
>>>
>>> That code looks wrong. I see the end but not the start. Look here and
>>> copy carefully:
>>>
>>> >>Since I did not want cpu profiling I omitted the start of the code and
>>> just added memory profiling part.
>>>
>>> Call at end, on way out.
>>>
>>> >>Oh yes, I missed that.I have to call memory profiling code at the end
>>> on the way out.But the thing is that it runs as a service in infinite for
>>> loop.
>>>
>>> func main() {
>>> flag.Parse()
>>> if *cpuprofile != "" {
>>> f, err := os.Create(*cpuprofile)
>>> if err != nil {
>>> fmt.Println("could not create CPU profile: ", err)
>>> }
>>> defer f.Close() // error handling omitted for example
>>> if err := pprof.StartCPUProfile(f); err != nil {
>>> fmt.Print("could not start CPU profile: ", err)
>>> }
>>> defer pprof.StopCPUProfile()
>>> }
>>>
>>> A_chan := make(chan bool)
>>> B_chan := make(chan bool)
>>> go util.A(A_chan)
>>> go util.B(B_chan)
>>> (..Rest of the code..)
>>>
>>> for {
>>> select {
>>> case <-A_chan:
>>> continue
>>> case <-B_chan:
>>> continue
>>>
>>> }
>>> }
>>>
>>> }
>>>
>>> What would be the correct way to add the memprofile code changes, since
>>> it is running in an infinite for loop ?
>>>
>>> Also, as shared by others above, there are no promises about how soon
>>> the dead allocations go away, The speed gets faster and faster version to
>>> version, and is impressive indeed now, so old versions are not the best to
>>> use, ubt even so, if the allocation feels small to th GC the urgency to
>>> free it will be low. You need to loop in allocating and see if the memory
>>> grows and grows.
>>>
>>> >> Yes, got it.I will try using the latest version of Go and check the
>>> behavior.
>>>
>>> Thanks,
>>> Nitish
>>>
>>> On Fri, Mar 13, 2020 at 6:20 AM Michael Jones <michael.jo...@gmail.com>
>>> wrote:
>>>
>>>> That code looks wrong. I see the end but not the start. Look here and
>>>> copy carefully:
>>>> https://golang.org/pkg/runtime/pprof/
>>>>
>>>> Call at end, on way out.
>>>>
>>>> Also, as shared by others above, there are no promises about how soon
>>>> the dead allocations go away, The speed gets faster and faster version to
>>>> version, and is impressive indeed now, so old versions are not the best to
>>>> use, ubt even so, if the allocation feels small to th GC the urgency to
>>>> free it will be low. You need to loop in allocating and see if the memory
>>>> grows and grows.
>>>>
>>>> On Thu, Mar 12, 2020 at 9:22 AM Nitish Saboo <nitish.sabo...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have compiled my Go binary against go version 'go1.7 linux/amd64'.
>>>>> I added the following code change in the main function to get the
>>>>> memory profiling of my service
>>>>>
>>>>> var memprofile = flag.String("memprofile", "", "write memory profile
>>>>> to `file`")
>>>>>
>>>>> func main() {
>>>>> flag.Parse()
>>>>> if *memprofile != "" {
>>>>> f, err := os.Create(*memprofile)
>>>>> if err != nil {
>>>>> fmt.Println("could not create memory profile: ", err)
>>>>> }
>>>>> defer f.Close() // error handling omitted for example
>>>>> runtime.GC() // get up-to-date statistics
>>>>> if err := pprof.WriteHeapProfile(f); err != nil {
>>>>> fmt.Println("could not write memory profile: ", err)
>>>>> }
>>>>> }
>>>>> ..
>>>>> ..
>>>>> (Rest code to follow)
>>>>>
>>>>> I ran the binary with the following command:
>>>>>
>>>>> nsaboo@ubuntu:./main -memprofile=mem.prof
>>>>>
>>>>> After running the service for couple of minutes, I stopped it and got
>>>>> the file 'mem.prof'
>>>>>
>>>>> 1)mem.prof contains the following:
>>>>>
>>>>> nsaboo@ubuntu:~/Desktop/memprof$ vim mem.prof
>>>>>
>>>>> heap profile: 0: 0 [0: 0] @ heap/1048576
>>>>>
>>>>> # runtime.MemStats
>>>>> # Alloc = 761184
>>>>> # TotalAlloc = 1160960
>>>>> # Sys = 3149824
>>>>> # Lookups = 10
>>>>> # Mallocs = 8358
>>>>> # Frees = 1981
>>>>> # HeapAlloc = 761184
>>>>> # HeapSys = 1802240
>>>>> # HeapIdle = 499712
>>>>> # HeapInuse = 1302528
>>>>> # HeapReleased = 0
>>>>> # HeapObjects = 6377
>>>>> # Stack = 294912 / 294912
>>>>> # MSpan = 22560 / 32768
>>>>> # MCache = 2400 / 16384
>>>>> # BuckHashSys = 2727
>>>>> # NextGC = 4194304
>>>>> # PauseNs = [752083 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>>>> 0
>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>>>> 0
>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>>>> 0
>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>>>> 0
>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>>>> 0
>>>>> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >>>>> 0
>>>>> 0 0]
>>>>> # NumGC = 1
>>>>> # DebugGC = false
>>>>>
>>>>> 2)When I tried to open the file using the following command, it just
>>>>> goes into interactive mode and shows nothing
>>>>>
>>>>> a)Output from go version go1.7 linux/amd64 for mem.prof
>>>>>
>>>>> nsaboo@ubuntu:~/Desktop/memprof$ go tool pprof mem.prof
>>>>> Entering interactive mode (type "help" for commands)
>>>>> (pprof) top
>>>>> profile is empty
>>>>> (pprof)
>>>>>
>>>>> b)Output from go version go1.12.4 linux/amd64 for mem.prof
>>>>>
>>>>> nsaboo@ubuntu:~/Desktop/memprof$ go tool pprof mem.prof
>>>>> Type: space
>>>>> No samples were found with the default sample value type.
>>>>> Try "sample_index" command to analyze different sample values.
>>>>> Entering interactive mode (type "help" for commands, "o" for options)
>>>>> (pprof) o
>>>>> call_tree = false
>>>>> compact_labels = true
>>>>> cumulative = flat //: [cum | flat]
>>>>> divide_by = 1
>>>>> drop_negative = false
>>>>> edgefraction = 0.001
>>>>> focus = ""
>>>>> granularity = functions //: [addresses |
>>>>> filefunctions | files | functions | lines]
>>>>> hide = ""
>>>>> ignore = ""
>>>>> mean = false
>>>>> nodecount = -1 //: default
>>>>> nodefraction = 0.005
>>>>> noinlines = false
>>>>> normalize = false
>>>>> output = ""
>>>>> prune_from = ""
>>>>> relative_percentages = false
>>>>> sample_index = space //: [objects |
>>>>> space]
>>>>> show = ""
>>>>> show_from = ""
>>>>> tagfocus = ""
>>>>> taghide = ""
>>>>> tagignore = ""
>>>>> tagshow = ""
>>>>> trim = true
>>>>> trim_path = ""
>>>>> unit = minimum
>>>>> (pprof) space
>>>>> (pprof) sample_index
>>>>> (pprof) top
>>>>> Showing nodes accounting for 0, 0% of 0 total
>>>>> flat flat% sum% cum cum%
>>>>>
>>>>>
>>>>> 3)Please let me know if it is this the correct way of getting the
>>>>> memory profiling ?
>>>>>
>>>>> 4)Can we deduce something from this memory stats that points us to
>>>>> increase in memory usage?
>>>>>
>>>>> 5)I am just thinking out loud, since I am using go1.7, can that be the
>>>>> reason for the issue of increase in memory usage that might get fixed with
>>>>> latest go versions ?
>>>>>
>>>>> Thanks,
>>>>> Nitish
>>>>>
>>>>> On Tue, Mar 10, 2020 at 6:56 AM Jake Montgomery <jake6...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> On Monday, March 9, 2020 at 1:37:00 PM UTC-4, Nitish Saboo wrote:
>>>>>>>
>>>>>>> Hi Jake,
>>>>>>>
>>>>>>> The memory usage remains constant when the rest of the service is
>>>>>>> running.Only when LoadPatternDB() method is called within the service,
>>>>>>> Memory Consumption increases which actually should not happen.
>>>>>>> I am assuming if there is a memory leak while calling this method
>>>>>>> because the memory usage then becomes constant after getting increased
>>>>>>> and
>>>>>>> then further increases on next call.
>>>>>>>
>>>>>>
>>>>>> Its possible that I am not fully understanding, perhaps a language
>>>>>> problem. But from what you have written above I still don't see that this
>>>>>> means you definitely have a memory leak. To test for that you would need
>>>>>> to *continuously
>>>>>> *call LoadPatternDB() and monitor memory for a *considerable time*.
>>>>>> If it eventually stabilizes to a constant range then there is no leak,
>>>>>> just
>>>>>> normal Go-GC variation. If it never stops climbing, and eventually
>>>>>> consumes
>>>>>> all the memory, then it would probably be a leak. Just because it goes up
>>>>>> after one call, or a few calls doe not mean there is a leak.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "golang-nuts" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/golang-nuts/f897fdb1-8968-4435-9fe9-02e167e09a36%40googlegroups.com
>>>>>> <https://groups.google.com/d/msgid/golang-nuts/f897fdb1-8968-4435-9fe9-02e167e09a36%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "golang-nuts" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq6DC98p4M4V2QCbQFTcsL1PtOWELvg8MEcMYj9EM9ui_A%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq6DC98p4M4V2QCbQFTcsL1PtOWELvg8MEcMYj9EM9ui_A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *Michael T. jonesmichael.jo...@gmail.com <michael.jo...@gmail.com>*
>>>>
>>>
>>
>> --
>>
>> *Michael T. jonesmichael.jo...@gmail.com <michael.jo...@gmail.com>*
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CALjMrq5-gjzMUKAgWjUjJFYRFVyrctT5p-Sesd_f%3DRR-h9-39A%40mail.gmail.com
> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq5-gjzMUKAgWjUjJFYRFVyrctT5p-Sesd_f%3DRR-h9-39A%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CALjMrq4zzYiO7aj6Cq3%2BaTvK9%2B4jxoJTddTbgH1iVuNu%3D0jtOQ%40mail.gmail.com.
