[go-nuts] Tracking down RSS growth in cgo library
I've been working to track this down for 2 days now and I'm just taking a
long shot to see if anyone might have a new idea for me.
My cgo-based bindings library seems to have unbounded RSS memory growth,
which I have been able to reduce to the smallest benchmark test and even
pinpoint the exact call, but the reason behind it still eludes me.
The context is that I have a struct in C++ that will store a const char*
for the last exception that was thrown, as a strdup() copy which gets
cleaned up properly each time.
typedef struct _HandleContext {
HandleId handle;
const char* last_error;
} _HandleContext;
const char* getLastError(_HandleContext* ctx);
On the Go side, I have a function for lastError() to return the last error
value
func (c *Config) lastError() error {
err := C.getLastError(c.ptr)
if err == nil {
return nil
}
e := C.GoString(err)
if e == "" {
return nil
}
runtime.KeepAlive(c)
// return nil // <- would result in no memory growth
return errors.New(e) // <- results in memory growth
}
What I am seeing in my benchmark test is that the RSS grows something like
20MB a second, yet the GODEBUG=gctrace=1 and the pprof memory profile don't
reflect this memory usage at all, aside from showing a hotspot (in pprof)
being the GoString() call:
gc 4 @4.039s 0%: 0.006+0.14+0.003 ms clock,
0.024+0.10/0.039/0.070+0.014 ms cpu, 4->4->0 MB, 5 MB goal, 4 P
gc 5 @6.857s 0%: 0.003+0.20+0.004 ms clock,
0.015+0.069/0.025/0.15+0.016 ms cpu, 4->4->0 MB, 5 MB goal, 4 P
...
gc 26 @69.498s 0%: 0.005+0.12+0.003 ms clock,
0.021+0.10/0.044/0.093+0.014 ms cpu, 4->4->0 MB, 5 MB goal, 4 P
// 800MB RSS usage
gc 27 @71.824s 0%: 0.006+2.2+0.003 ms clock,
0.025+0.063/0.058/0.11+0.014 ms cpu, 4->4->0 MB, 5 MB goal, 4 P
(pprof) top10
Showing nodes accounting for 46083.69kB, 100% of 46083.69kB total
Showing top 10 nodes out of 19
flat flat% sum%cum cum%
30722.34kB 66.67% 66.67% 30722.34kB 66.67% <...>._Cfunc_GoStringN (inline)
7168.11kB 15.55% 82.22% 7168.11kB 15.55% errors.New (inline)
3073.16kB 6.67% 88.89% 46083.69kB 100% <...>.testLeak
1536.02kB 3.33% 92.22% 1536.02kB 3.33%
<...>.(*DisplayTransform).SetInputColorSpace.func1
1024.02kB 2.22% 94.44% 1024.02kB 2.22% <...>.(*Config).NumViews.func1
1024.02kB 2.22% 96.67% 1024.02kB 2.22% <...>.(*Config).View.func1
512.01kB 1.11% 97.78% 512.01kB 1.11%
<...>.(*DisplayTransform).SetView.func1
512.01kB 1.11% 98.89% 512.01kB 1.11% <...>._Cfunc_GoString (inline)
512.01kB 1.11% 100% 512.01kB 1.11% <...>.newProcessor (inline)
0 0% 100% 512.01kB 1.11%
<...>.(*Config).ColorSpaceNameByIndex
Regardless of whether I ignore the error return value in my test, it grows.
If I return nil instead of errors.New(e), it will stay around 20MB RSS.
I MUST be doing something stupid, but I can't see any reason for the memory
growth based on returning this string wrapped in an error. At first I
thought I was leaking in C/C++ but it a led to this one factor on the Go
side. Any tips would help greatly, since I have tried debug GC output,
pprof reports, valgrind, address sanitizer, and refactoring the entire
memory management of my C bindings layer.
Justin
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAPGFgA20ai9u9Gy8Zk_XTq5ZK9bz_oo0ZrZs5ruvEwfwK3ukqA%40mail.gmail.com.
[go-nuts] pprof with TLS client authentication
Does go tool pprof support TLS with client authentication? I see it has TLS options, but when I set them it doesn't work. But, it works if I use the standalone pprof tool. $ go version go version go1.14.1 linux/amd64 $ go tool pprof -tls_ca=./ca.crt -tls_cert=./client.user.crt -tls_key=./client.user.key https://localhost:8080/debug/pprof/heap Fetching profile over HTTP from https://localhost:8080/debug/pprof/heap https://localhost:8080/debug/pprof/heap: Get "https://localhost:8080/debug/pprof/heap": x509: certificate signed by unknown authority failed to fetch any source profiles $ go get -u github.com/google/pprof $ pprof -tls_ca=./ca.crt -tls_cert=./client.user.crt -tls_key=./client.user.key https://localhost:8080/debug/pprof/heap Fetching profile over HTTP from https://localhost:8080/debug/pprof/heap Saved profile in /home/dmai/pprof/pprof.dgraph.alloc_objects.alloc_space.inuse_objects.inuse_space.5629.pb.gz -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/1cb5039b-393a-48f4-b0cd-45672efb177b%40googlegroups.com.
[go-nuts] Delve v1.4.1 released!
Delve v1.4.1 is released! Thanks to everyone who contributed to this release! This release includes some new features, optimizations and bug fixes. Notable changes: * Support for linux/386 * DAP server initial release * New commands `display` and `examinemem` * Tracing improvements * Better support for debugging CLI apps ... & more. For more information and more complete list of changes, check out the tag on GitHub and the change list: https://github.com/go-delve/delve/releases/tag/v1.4.1 -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/300552eb-6889-41f9-8395-a84f3856855f%40googlegroups.com.
[go-nuts] Re: MediaStreamError
This forum is about the Go language. Perhaps I am missing something, but this looks like Java? Does this relate to go in some way? On Friday, May 22, 2020 at 4:21:10 AM UTC-4, Ali Hassan wrote: > > I plug video streaming with my website but I got MediaStreamError . > > > How to resolve this error > https://github.com/ali2210/WizDwarf/blob/master/js/webrtc-video.js > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/47a993d2-d857-4044-8ae5-73e5a2f92fbc%40googlegroups.com.
Re: [go-nuts] Maddy - composable all-in-one mail server
Thanks for plugging this. Out of curiosity, what does "composable" mean in this case? On Thu, 21 May 2020 at 06:14, wrote: > > Hello, > > I would like to introduce a project that, in my opinion, needs more attention > as it has a great potential. It is a secure-by-default MTA and MDA > implementation in a single server called Maddy, which works similarly to > Caddy web server. > From what I witness, it is already working pretty well, despite being quite > alpha-quality software. > > You can read more about it on a GitHub page: > https://github.com/foxcpp/maddy > > Cheers. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/002d5e0d-2469-4f15-8eb0-e8bdc67419f8%40googlegroups.com. -- Adrian Ratnapala -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAN%2BHj7j81-8JjZpxsBbef9SApv%2BDG%2BiCqrtEK9Rquz73UyvU7A%40mail.gmail.com.
[go-nuts] Re: Need help with go modules
It looks like I am not only one to struggle with (new?) go modules. I still consider me as novice to GO but the major problem is as usual, the focus. 1. Modules are complicated to understand 2. Lack of examples (of real use, not just POC) 3. Focus on extra (scale up) details (that are surely important) and none of focus on basic things (which are essentially more important as they make things working and not working) I have spent several days in attempt to make a working module and I failed. I learned a lot of things on the way but still did not manage to make a working example. The working example in my understanding shall be given as: Variant 1: Making executable 1. how the main (executable) uses the module (import and function call, how to avoid function name conflicts?) 2. how to build a module that 1.(main program uses) 3. how and where to compile and what and what compilation output to expect Variant 2: Making library In a way a module is a library but, one may want to make his own library that will be used in his project and within that library use his custom modules, so we go the same 1. how library modules uses the custom module (import and function call and how to avoid function name conflicts from different modules? and 2. which shall be the same as for Variant 1. How really to make a working module. 3. which shall be the same as for Variant 1. how and where to compile and what and what compilation output to expect this is more a feedback on what conceptually I am struggling with go and modules --- On the other note the relation between gopath and modules. Currently (I am trying to use go 1.14) I understand that there is a method of 'gopath' where go will seek for libraries. This goes to compilation part. Say have i have a custom module site/name/module (literally) what i put in package part? what would be relation between main and others. In others - is there package site/name ? where to put sources so go finds them? (while compilation gopath method suggests where to put sources - it is a help but is this correct way of doing things? how to compile these sources so it is explained which modules are used (and from where)? if I just compile a module - what do I get ? If I get some sort of an object - where to look for it? How it is called? How do I reference it from the source. For example in my experimental trials I get (a hello world executable trying to call function defined in a module) I am getting error: 'site/name/module' imported and not used of course since module is not found I also get error: undefined: Hello which is a function I try to call. Hope all of these make sense, and possibly someone wrote somewhere something of these but so far I did not find it. And I LIKE reading documentation, but documentation is just partially helpful. Thanks On Tuesday, May 5, 2020 at 1:28:55 PM UTC+2, Amnon Baron Cohen wrote: > > Interesting. At first sight this should work. > > You definitely don't need a go.mod file in ~HOME/src/myrepo/cmd/cmd1 > > Which go version are you running? > > What is your $GOPATH set to? > > What output does cd ~HOME/src/myrepo; go build give? > > The usual convention is to push the code in a VCS such as github and use > the vcs path as an argument to go mod init > > e.g. > go mod init github.com/myuser/myrepo > > - Amnon > > On Monday, 4 May 2020 14:59:22 UTC+1, web user wrote: >> >> I have a personal project using GOPATH >> >> But recently, I wanted to use go-redis for a project and v7 forces you to >> use go mod. So I figured, I'd migrate the account over to go mod. But I'm >> stuck. >> >> >> My directory structure is: >> >> ~HOME/src/myrepo >> ~HOME/src/myrepo/cmd/cmd1 >> ~HOME/src/myrepo/commong >> ~HOME/src/github >> >> So I did the following command: cd ~HOME/src/myrepo; go mod init >> ~HOME/src/myrepo >> >> when I run the command go build ~HOME/src/myrepo/cmd/cmd1/... >> >> It does not update the go.mod in the directory ~HOME/src/myrepo? Do I >> need a go.mod in ~HOME/src/myrepo/cmd/cmd1 >> >> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/55afd9c3-d410-4347-a7c5-a4ed3e5d64c3%40googlegroups.com.
[go-nuts] Re: opencv with my golang project
Thank you for support but i change my plan because this error resolve through npm which I had not using. I check stackflow On Thursday, May 21, 2020 at 1:27:27 PM UTC+5, Ali Hassan wrote: > > Opencv install [github.com/shrkwd/opencv-js] > Os : Linux 18.04 > > // opencv > file >// > video features > // > utils file to start > > When I compile my project , everything fine but opencv can't start. > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ef0ae65d-021e-4d0c-a420-83762ad713e2%40googlegroups.com.
[go-nuts] Re: opencv with my golang project
Thank you I have change my alternative because opencv not ready to integrate with golang.This is javascript error which I had checkout on stackoverflow On Thursday, May 21, 2020 at 1:27:27 PM UTC+5, Ali Hassan wrote: > > Opencv install [github.com/shrkwd/opencv-js] > Os : Linux 18.04 > > // opencv > file >// > video features > // > utils file to start > > When I compile my project , everything fine but opencv can't start. > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/267898fc-b528-40d1-bd89-5a7cdd5022e1%40googlegroups.com.
[go-nuts] MediaStreamError
I plug video streaming with my website but I got MediaStreamError . How to resolve this error https://github.com/ali2210/WizDwarf/blob/master/js/webrtc-video.js -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c136db13-f11e-4786-9bbb-c00ed7917459%40googlegroups.com.
Re: [go-nuts] Weird problem that CPU is nearly 100%
On Fri, May 22, 2020 at 9:05 AM tokers wrote: > > Thanks for you reply. > > Yeah, we have the plan to upgrade our go version to 1.13.10. Note that 1.13 does not have goroutine preemption Ian was talking about wrt 1.14. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAA40n-V89dbZMuvkQ4ye13fRfvtYm-0_hJB%2B8a9fGS0JZwVT%3Dw%40mail.gmail.com.
Re: [go-nuts] Weird problem that CPU is nearly 100%
Thanks for you reply. Yeah, we have the plan to upgrade our go version to 1.13.10. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5a4c4918-82f7-4979-9a86-8c289c25e1b8%40googlegroups.com.
