Re: [go-nuts] Go Crashes on a long running process (with stack trace)

[Konstantin Khomoutov]

On Tue, Nov 01, 2022 at 10:14:28AM -0700, Vineet Jain wrote:

> We have a long running process that listens to and processes a large number 
> of network packets. It died today with the following stack trace. Anyone 
> seen this before, or have any idea what is causing it?
> 
> Nov 01 12:40:30  sv_svc[100448]: I1101 12:40:30.865268  100448 
> stratconn.go:83] Message Sequence(109845000): Q Length(1)
> Nov 01 12:40:31  sv_svc[100448]: n on lineno rlimitpanicwrap: unexpected 
> string after ty
> Nov 01 12:40:31  sv_svc[100448]: 
> erfreecr1742248535156257105427357601001nTpUpd0x1NFOTyp0x0 pc=0x0]
> Nov 01 12:40:31  sv_svc[100448]: goroutine 4 [wner di]:
> Nov 01 12:40:31  sv_svc[100448]: runtime.throw({0x75b1b0?, 0x0?})
> Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/panic.go:1047 +0x5d 
> fp=0xcc8080 sp=0xcc8050 pc=0x436a1d
> Nov 01 12:40:31  sv_svc[100448]: runtime: g 4invalid use of 
> scannermalforuntime.sigpanic_ERRORGC scav0x0
[...]

The stack trace looks pretty suspicious because it itself appears to be
corrupted as certain prhases are missing letters in the words or have
different letters instead of "expected" ones.

I can conjure that may be that's just some person typed the text while looking
at the screen - instead of copying and pasting. Or maybe you have run the
"search and replace" operation on the stack trace's text, and it replaced more
stuff than was intended.
But I'm asking because if it really was copied and pasted, I'd propose you
either had a memory corruption in your process - for instance, because of
misbehaving C library code linked to the program or a bug in the code using
the `unsafe` package - directly or indirectly.

Still, these tree lines

| I1101 12:40:30.865268  100448 stratconn.go:83] Message Sequence(109845000): Q 
Length(1)
| n on lineno rlimitpanicwrap: unexpected string after ty
| erfreecr1742248535156257105427357601001nTpUpd0x1NFOTyp0x0 pc=0x0]

(I have stripped the logger prefixes from them for brewity)

look like a manifestation of a "normal" panic from the application's code
or the code of one of the libraries it uses. The clues are:

 - stratconn.go does not appear to be a file from the stdlib "net" package.

 - The "rlimitpanicwrap" cannot be found in the Go runtime's source code.
   Judging from the name, I'd say it's some sort of a handler to catch
   panics and do something; I guess, attach the stack trace to them and
   maybe bubble them up or whatever. It's not really relevant to the issue
   at hand, though.

 - The guru meditation string "Message Sequence(109845000): Q Length(1)"
   appears to be related to parsing of the protocol your application
   may be using.

Based on this evidence, I'd say you have merely found a regular bug in your
code.

Still, I need to confess that

| runtime: g 4invalid use of scannermalforuntime.sigpanic_ERRORGC scav0x0

looks scary and related to the runtime.
The problem is that 1) the message text appearch to be damaged, as I've
already stated, and 2) I fail to find even the piecs of this string in the Go
runtime source code.


That said, asking such question w/o stating the Go flavor used to build the
program, and its exact version is useless. Including at least some details
about the runtime environment (GOOS and GOARCH) would not harm, either.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221104110033.yjfmnsh6xvl72rbx%40carbon.

[go-nuts] Go Crashes on a long running process (with stack trace)

[Vineet Jain]

We have a long running process that listens to and processes a large number 
of network packets. It died today with the following stack trace. Anyone 
seen this before, or have any idea what is causing it?

Nov 01 12:40:30  sv_svc[100448]: I1101 12:40:30.865268  100448 
stratconn.go:83] Message Sequence(109845000): Q Length(1)
Nov 01 12:40:31  sv_svc[100448]: n on lineno rlimitpanicwrap: unexpected 
string after ty
Nov 01 12:40:31  sv_svc[100448]: 
erfreecr1742248535156257105427357601001nTpUpd0x1NFOTyp0x0 pc=0x0]
Nov 01 12:40:31  sv_svc[100448]: goroutine 4 [wner di]:
Nov 01 12:40:31  sv_svc[100448]: runtime.throw({0x75b1b0?, 0x0?})
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/panic.go:1047 +0x5d 
fp=0xcc8080 sp=0xcc8050 pc=0x436a1d
Nov 01 12:40:31  sv_svc[100448]: runtime: g 4invalid use of 
scannermalforuntime.sigpanic_ERRORGC scav0x0
Nov 01 12:40:31  sv_svc[100448]: olding lockssync.0xcc8080, 
fp:0xcc80d0} stack=[0xcc7800,0xcc8000)
Nov 01 12:40:31  sv_svc[100448]: 0x00cc7f80:  0x000b08be 
 0x412f0ff6
Nov 01 12:40:31  sv_svc[100448]: 0x00cc7f90:  0x00cc7fb8 
 0x0042449f 
Nov 01 12:40:31  sv_svc[100448]: 0x00cc7fa0:  0x00a071c0 
 0x412f0ff6
Nov 01 12:40:31  sv_svc[100448]: 0x00cc7fb0:  0x00cf 
 0x00cc7fd0
Nov 01 12:40:31  sv_svc[100448]: 0x00cc7fc0:  0x0041ac46 
  0x
Nov 01 12:40:31  sv_svc[100448]: 0x00cc7fd0:  0x 
 0x00467501 
Nov 01 12:40:31  sv_svc[100448]: 0x00cc7fe0:  0x 
 0x
Nov 01 12:40:31  sv_svc[100448]: 0x00cc7ff0:  0x 
 0x
Nov 01 12:40:31  sv_svc[100448]: runtime.sigpanic()
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/signal_unix.go:819 +0x369 
fp=0xcc80d0 sp=0xcc8080 pc=0x44cee9
Nov 01 12:40:31  sv_svc[100448]: created by runtime.gcenable
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/mgc.go:179 +0xaa
Nov 01 12:40:31  sv_svc[100448]: goroutine 1 [semacquire, 1562.locktra]:
Nov 01 12:40:31  sv_svc[100448]: runtime.gopark(0xc0027ee400?, 
0x7f8110208f50?, 0x80?, 0x76?, 0x7f81103445b8?)
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/proc.go:363 +0xd6 
fp=0xc000810d20 sp=0xc000810d00 pc=0x439636
Nov 01 12:40:31  sv_svc[100448]: runtime.goparkunlockcopyst   
 /go/src/runtime/proc.go:369
Nov 01 12:40:31  sv_svc[100448]: runtime.semacquire1(0xa37928, 0xd0?, 0x1, 
0x0)
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/sema.go:150 +0x1fe 
fp=0xc000810d88 sp=0xc000810d20 pc=0x44a31e
Nov 01 12:40:31  sv_svc[100448]: sync.runtime_Semacquire(0xc0005222c0?)
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/sema.go:62 +0x25 
fp=0xc000810db8 sp=0xc000810d88 pc=0x4635e5
Nov 01 12:40:31  sv_svc[100448]: sync.(*WaitGroup).Wait(0x77359400?)
Nov 01 12:40:31  sv_svc[100448]: /go/src/sync/waitgroup.go:139 +0x52 
fp=0xc000810de0 sp=0xc000810db8 pc=0x46f4f2
Nov 01 12:40:31  sv_svc[100448]: main.main()
Nov 01 12:40:31  sv_svc[100448]: /sv/svgo/feed/cmd/mcast/main.go:72 +0x753 
fp=0xc000810f80 sp=0xc000810de0 pc=0x6b5293
Nov 01 12:40:31  sv_svc[100448]: runtime.main()
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/proc.go:250 +0x212 
fp=0xc000810fe0 sp=0xc000810f80 pc=0x439272
Nov 01 12:40:31  sv_svc[100448]: runtime.goexit()
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/asm_amd64.s:1594 +0x1 
fp=0xc000810fe8 sp=0xc000810fe0 pc=0x467501
Nov 01 12:40:31  sv_svc[100448]: goroutine 2 [force gc (idle), 2.locktra]:
Nov 01 12:40:31  sv_svc[100448]: runtime.gopark(0x31b5261ac10cf2?, 0x0?, 
0x0?, 0x0?, 0x0?)
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/proc.go:363 +0xd6 
fp=0xcc6fb0 sp=0xcc6f90 pc=0x439636
Nov 01 12:40:31  sv_svc[100448]: runtime.goparkunlockcopyst   
 /go/src/runtime/proc.go:369
Nov 01 12:40:31  sv_svc[100448]: runtime.forcegchelper()
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/proc.go:302 +0xad 
fp=0xcc6fe0 sp=0xcc6fb0 pc=0x4394cd
Nov 01 12:40:31  sv_svc[100448]: runtime.goexit()
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/asm_amd64.s:1594 +0x1 
fp=0xcc6fe8 sp=0xcc6fe0 pc=0x467501
Nov 01 12:40:31  sv_svc[100448]: created by runtime.init.6
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/proc.go:290 +0x25
Nov 01 12:40:31  sv_svc[100448]: goroutine 3 [ionbad specia]:
Nov 01 12:40:31  sv_svc[100448]: runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 
0x0?)
Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/proc.go:363 +0xd6 
fp=0xcc7790 sp=0xcc7770 pc=0x439636
Nov 01 12:40:31  sv_svc[100448]: n on lineno r font file formatb
Nov 01 12:40:31  sv_svc[100448]: pflag %q begins wit
Nov 01 12:40:31  sv_svc[100448]: runtime stack:

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit