[go-nuts] Re: Investigating explosive memory growth with race detector
ideas to get a profile under duress:
a) rent a bigger box with tons of cpu and ram for 15 minutes to get your
profile
b) terminate the pprof profile programatically after a short amount of
time; e.g.
f, err := os.Create("my_cpu_profile")
panicOn(err)
pprof.StartCPUProfile(f)
go func() {
time.Sleep(time.Second * 5)
pprof.StopCPUProfile()
}
On Tuesday, March 21, 2023 at 3:26:59 PM UTC-5 Marco P. wrote:
> I have a program, normal memory usage is <50MB and CPU ~5%. This doesn't
> change over time.
>
> Rebuilding with `-race` shows memory <100MB and CPU ~25%.
> (Consistent with overhead described here:
> https://go.dev/doc/articles/race_detector#Runtime_Overheads)
>
> However, with `-race` enabled after a couple of minutes, the CPU suddenly
> jumps to 100%, and skyrockets to multiple GBs within seconds.
>
> e.g.:
>
> T0: CPU:25%, MEM:95MB
> T1: CPU:25%, MEM:95MB
> (...)
> T100: CPU:25%, MEM:95MB
> T101: CPU:99%, MEM:500MB
> T102: CPU:99%, MEM:2GB
> T103: CPU:99%, MEM:4GB
> T104: CPU:99%, MEM:6GB
> T105: CPU:99%, MEM:8GB
> => OOM
>
> The CPU jump is drastic and instantaneous, and the memory seems to grow as
> fast as it can be allocated.
>
> The race detector docs says:
>
> > The race detector currently allocates an extra 8 bytes per defer and
> recover statement. Those extra allocations are not recovered until the
> goroutine exits. This means that if you have a long-running goroutine that
> is periodically issuing defer and recover calls, the program memory usage
> may grow without bound. These memory allocations will not show up in the
> output of runtime.ReadMemStats or runtime/pprof.
>
> Here's my question.
>
> I would like to:
> 1. Confirm the extra memory is due to the race detector overhead related
> to defer/recover (as opposed to some other bug in the program that only
> surfaces when building with `-race`
> 2. Find the coroutine(s?) responsible for that defer/recover
>
> Any idea on how to investigate?
>
> I have tried capturing with pprof. Even if the data race allocation are
> not visible ("These memory allocations will not show up in the output of
> runtime.ReadMemStats or runtime/pprof."), I could at least confirm it's not
> the program code allocating.
>
> However pprof does not work for a different reason: once the program is in
> "100%CPU" mode, pprof times out.
> So I can't ever capture a trace/heap/profile while the system is showing
> the behavior (because CPU and memory are already too pegged to handle a
> pprof dump)
>
> Anything else I could try to get to the bottom of this?
>
> For example, is there a way to trace all defer/recover calls?
> Or is there a way to attach a debugger and pause when memory usage exceeds
> a certain amount?
>
> I searched for these and more, but couldn't find much. Maybe some wizard
> on this list has some ideas or pointers.
>
> [go1.19.7.linux-amd64]
>
> Thank you,
> M.
>
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/3e680fee-d047-45ff-bedf-d473a24f866bn%40googlegroups.com.
[go-nuts] Re: Mocking out calls in standard library
I hasten to add, my library sshego makes it pretty easy to spin up and down an sshd. e.g. https://github.com/glycerine/sshego/blob/master/server_test.go On Friday, March 31, 2023 at 11:20:41 PM UTC-5 Jason E. Aten wrote: > I would just start a server on 127.0.0.1 and have it provide the expected > responses. This is simple and straightforward, and gives a much better > test; end-to-end. > > The less mocking the better from my point of view. > > If you have to, you would simply wrap the existing structs, creating a > higher layer that abstracts. Instead of going below, go above. > On Friday, March 31, 2023 at 8:24:14 AM UTC-5 Inian Vasanth wrote: > >> Hello Community, >> >> I'm testing out some code for increasing code coverage of our repository, >> i.e. to maximum level possible. One of the methods I'm dealing with are few >> methods in standard library (ssh, sftp) that need to be modified for custom >> behavior. >> >> See Goplay snippet here - https://go.dev/play/p/DGYUqgEKKXh >> >> Basically, I want to modify the type definition of `sshDialer` and >> `sftpNewClient` to return an interface that is implemented by the native >> type. If I could do this, I would generate a mock struct satisfying the >> same interface and then perform our tests. >> >> I know mocking standard libraries are not the best of choice, but this is >> just to get familiar with the language and an experiment with it. >> >> Also would appreciate other ways to solve the same problem. Thanks >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5235d4ef-7992-4a4e-93ec-c71bccf02865n%40googlegroups.com.
[go-nuts] Re: Mocking out calls in standard library
I would just start a server on 127.0.0.1 and have it provide the expected responses. This is simple and straightforward, and gives a much better test; end-to-end. The less mocking the better from my point of view. If you have to, you would simply wrap the existing structs, creating a higher layer that abstracts. Instead of going below, go above. On Friday, March 31, 2023 at 8:24:14 AM UTC-5 Inian Vasanth wrote: > Hello Community, > > I'm testing out some code for increasing code coverage of our repository, > i.e. to maximum level possible. One of the methods I'm dealing with are few > methods in standard library (ssh, sftp) that need to be modified for custom > behavior. > > See Goplay snippet here - https://go.dev/play/p/DGYUqgEKKXh > > Basically, I want to modify the type definition of `sshDialer` and > `sftpNewClient` to return an interface that is implemented by the native > type. If I could do this, I would generate a mock struct satisfying the > same interface and then perform our tests. > > I know mocking standard libraries are not the best of choice, but this is > just to get familiar with the language and an experiment with it. > > Also would appreciate other ways to solve the same problem. Thanks > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/aa3b7ac7-0724-4303-a5bc-4c1708e85cfcn%40googlegroups.com.
[go-nuts] Re: Calling panic() is corrupting system stack
Hello Go experts,
Could someone please help to resolve this issue?
Best Regards
Mariappan
On Thu, Mar 30, 2023 at 2:52 PM mariappan balraj
wrote:
> Hello Go Experts,
>
> When panic() is called from Go function, in the below call sequence,
> runtime.unwindm() [which is defer function of runtime.cgocallbackg1] is
> called. This function is unwinding the system stack. Later, as part of the
> function addOneOpenDeferFrame(), systemstack() is called to run a function
> in the system stack. This will use the stack which is allocated for C
> function calls. This makes stack unwinding impossible in case of panic() is
> called. Can someone please help me to fix this issue?
>
> (dlv) bt
> 0 0x004054e6 in runtime.unwindm
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:326
> 1 0x00405446 in runtime.cgocallbackg1.func3
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:304
> 2 0x004340c8 in runtime.deferCallSave
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:796
> 3 0x00433fa5 in runtime.runOpenDeferFrame
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:769
> 4 0x00434332 in runtime.gopanic
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:884
> 5 0x004642a7 in main.Test4
> at ./export.go:7
> at ./export.go:7
> 6 0x0046431c in _cgoexp_78b81bbf688e_Test4
> at _cgo_gotypes.go:61
> 7 0x0040535b in runtime.cgocallbackg1
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:315
> 8 0x00405079 in runtime.cgocallbackg
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:234
> 9 0x00461b0f in runtime.cgocallbackg
> at :1
> 10 0x0045f3f4 in runtime.cgocallback
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/asm_amd64.s:998
> 11 0x004641bd in crosscall2
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgo/asm_amd64.s:30
> 12 0x00464386 in C.Test4
> at /tmp/go-build/_cgo_export.c:33
> 13 0x00464432 in C.test1
> at ./hello.go:9
> 14 0x0046444d in C.test2
> at ./hello.go:14
> 15 0x00464468 in C.test3
> 15 0x00464468 in C.test3
> at ./hello.go:19
> 16 0x004644a7 in C._cgo_78b81bbf688e_Cfunc_test3
> at /tmp/go-build/cgo-gcc-prolog:49
> 17 0x0045f2e4 in runtime.asmcgocall
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/asm_amd64.s:848
> 18 0x0046448a in C._cgo_78b81bbf688e_Cfunc_test3
> at /tmp/go-build/cgo-gcc-prolog:44
> 19 0x00404f0a in runtime.cgocall
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/cgocall.go:167
> 20 0x00464245 in main._Cfunc_test3
> at _cgo_gotypes.go:39
> 21 0x004642d7 in main.main
> at ./hello.go:33
> 22 0x00437073 in runtime.main
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/proc.go:250
> 23 0x0045f5e1 in runtime.goexit
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/asm_amd64.s:1598
>
> 254 func cgocallbackg1(fn, frame unsafe.Pointer, ctxt uintptr) {
> 304 defer unwindm()
>
> 326 func unwindm(restore *bool) {
> 327 if *restore {
> 328 // Restore sp saved by cgocallback during
> 329 // unwind of g's stack (see comment at top of file).
> 330 mp := acquirem()
> 331 sched :=
> 332 sched.sp = *(*uintptr)(unsafe.Pointer(sched.sp +
> alignUp(sys.MinFrameSize, sys.StackAlign)))
>
> (dlv)
> > runtime.addOneOpenDeferFrame()
> /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:642 (PC:
> 0x433a45)
> Warning: debugging optimized function
>637: func addOneOpenDeferFrame(gp *g, pc uintptr, sp unsafe.Pointer) {
>638: var prevDefer *_defer
>639: if sp == nil {
>640: prevDefer = gp._defer
>641: pc = prevDefer.framepc
> => 642: sp = unsafe.Pointer(prevDefer.sp)
>643: }
>644: systemstack(func() {
>645: gentraceback(pc, uintptr(sp), 0, gp, 0, nil,
> 0x7fff,
>646: func(frame *stkframe, unused
> unsafe.Pointer) bool {
>647: if prevDefer != nil &&
> prevDefer.sp == frame.sp {
> (dlv) bt
> 0 0x00433a45 in runtime.addOneOpenDeferFrame
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:642
> 1 0x00434357 in runtime.gopanic
> at /home/soomohan/mbalraj/GO/go1.20.2/go/src/runtime/panic.go:886
> 2 0x004642a7 in main.Test4
> at ./export.go:7
>
> On Tue, Mar 28, 2023 at 4:14 PM mariappan balraj <
> mariappan.bal...@gmail.com> wrote:
>
>> Hello Go Experts,
>>
>> I am observing system stack corruption when panic() is called from go
>> function. When panic is called from Test4(), as part of
>>
Re: [go-nuts] getting the size of an io.ReaderAt?
Not a direct answer to your question (your code looks like a reasonable
implementation modulo any bugs I did not notice) but: Aren't you over
engineering things? If the source is a PDF file, why not also pass the size
to the function instead of doing all this work to get it? At some point you
have to open the file, right? And checking the size of a file is trivial
(it is just a stat call) and then you do not need to do that.
On Fri, Mar 31, 2023 at 4:30 PM Jochen Voss wrote:
> Dear all,
>
> I am trying to get the size of an io.ReaderAt, i.e. the offset after
> which no more data can be read. I have some working (?) code (
> https://go.dev/play/p/wTouYbaJ7RG , also reproduced below), but I am not
> sure whether what I do is correct and the best way to do this. Some
> questions:
>
>- Does the standard library provide a way to get the size of an
>io,ReaderAt?
>- Is my code correct?
>- Is there a better way to do this?
>- If a type provides not only a ReadAt() method, but also a Size()
>method, would it be save to assume that Size() returns how many bytes are
>accessible via ReadAt()? This seems to work for bytes.Reader and
>strings.Reader, but there may be types out there where Size() does
>something different?
>- If ReadAt(p, x) returns io.EOF for an offset x, is it then
>guaranteed that then ReadAt(p, y) also returns io.EOF for all y > x?
>Or could there be different error messages, or "files with holes", or
>whatnot?
>- Which types in the standard library provide ReadAt methods? I know
>of os.File, strings.Reader, and bytes.Reader. Any others?
>
> For context: this is for reading the cross reference table of PDF files,
> which have to be located by following some convoluted route starting *from
> the end* of the PDF file.
>
> Many thanks,
> Jochen
>
> func getSize(r io.ReaderAt) (int64, error) {
> if f, ok := r.(*os.File); ok {
> fi, err := f.Stat()
> if err != nil {
> return 0, err
> }
> return fi.Size(), nil
> }
> if b, ok := r.(*bytes.Reader); ok {
> return int64(b.Size()), nil
> }
> if s, ok := r.(*strings.Reader); ok {
> return int64(s.Size()), nil
> }
>
> buf := make([]byte, 1024)
> n, err := r.ReadAt(buf, 0)
> if err == io.EOF {
> return int64(n), nil
> } else if err != nil {
> return 0, err
> }
>
> lowerBound := int64(n) // all bytes before lowerBound are known to be
> present
> var upperBound int64 // at least one byte before upperBound is known to
> be missing
> for {
> test := 2 * lowerBound
> _, err := r.ReadAt(buf[:1], test-1)
> if err == io.EOF {
> upperBound = test
> break
> } else if err != nil {
> return 0, err
> }
> lowerBound = test
> }
>
> for lowerBound+1 < upperBound {
> test := (lowerBound + upperBound + 1) / 2
> _, err := r.ReadAt(buf[:1], test-1)
> if err == io.EOF {
> upperBound = test
> } else if err != nil {
> return 0, err
> } else {
> lowerBound = test
> }
> }
> return lowerBound, nil
> }
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/aa604ac0-3542-4a9e-adef-a9eaecfd8170n%40googlegroups.com
>
> .
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAEd86Tw3FQxjr5PpBfKT3eJKS1RTa-CJZyieKxj%2BSVz7Qh_hbg%40mail.gmail.com.
[go-nuts] getting the size of an io.ReaderAt?
Dear all,
I am trying to get the size of an io.ReaderAt, i.e. the offset after which
no more data can be read. I have some working (?) code
(https://go.dev/play/p/wTouYbaJ7RG , also reproduced below), but I am not
sure whether what I do is correct and the best way to do this. Some
questions:
- Does the standard library provide a way to get the size of an
io,ReaderAt?
- Is my code correct?
- Is there a better way to do this?
- If a type provides not only a ReadAt() method, but also a Size()
method, would it be save to assume that Size() returns how many bytes are
accessible via ReadAt()? This seems to work for bytes.Reader and
strings.Reader, but there may be types out there where Size() does
something different?
- If ReadAt(p, x) returns io.EOF for an offset x, is it then guaranteed
that then ReadAt(p, y) also returns io.EOF for all y > x? Or could
there be different error messages, or "files with holes", or whatnot?
- Which types in the standard library provide ReadAt methods? I know of
os.File, strings.Reader, and bytes.Reader. Any others?
For context: this is for reading the cross reference table of PDF files,
which have to be located by following some convoluted route starting *from
the end* of the PDF file.
Many thanks,
Jochen
func getSize(r io.ReaderAt) (int64, error) {
if f, ok := r.(*os.File); ok {
fi, err := f.Stat()
if err != nil {
return 0, err
}
return fi.Size(), nil
}
if b, ok := r.(*bytes.Reader); ok {
return int64(b.Size()), nil
}
if s, ok := r.(*strings.Reader); ok {
return int64(s.Size()), nil
}
buf := make([]byte, 1024)
n, err := r.ReadAt(buf, 0)
if err == io.EOF {
return int64(n), nil
} else if err != nil {
return 0, err
}
lowerBound := int64(n) // all bytes before lowerBound are known to be
present
var upperBound int64 // at least one byte before upperBound is known to
be missing
for {
test := 2 * lowerBound
_, err := r.ReadAt(buf[:1], test-1)
if err == io.EOF {
upperBound = test
break
} else if err != nil {
return 0, err
}
lowerBound = test
}
for lowerBound+1 < upperBound {
test := (lowerBound + upperBound + 1) / 2
_, err := r.ReadAt(buf[:1], test-1)
if err == io.EOF {
upperBound = test
} else if err != nil {
return 0, err
} else {
lowerBound = test
}
}
return lowerBound, nil
}
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/aa604ac0-3542-4a9e-adef-a9eaecfd8170n%40googlegroups.com.
[go-nuts] Mocking out calls in standard library
Hello Community, I'm testing out some code for increasing code coverage of our repository, i.e. to maximum level possible. One of the methods I'm dealing with are few methods in standard library (ssh, sftp) that need to be modified for custom behavior. See Goplay snippet here - https://go.dev/play/p/DGYUqgEKKXh Basically, I want to modify the type definition of `sshDialer` and `sftpNewClient` to return an interface that is implemented by the native type. If I could do this, I would generate a mock struct satisfying the same interface and then perform our tests. I know mocking standard libraries are not the best of choice, but this is just to get familiar with the language and an experiment with it. Also would appreciate other ways to solve the same problem. Thanks -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/34727788-73b0-4928-9945-2f76a83fecddn%40googlegroups.com.
