@Brian, We did not try that as the issue is happening with go 1.13.7 as well
On Thu, 9 Sep, 2021, 10:48 pm Brian Candler, <b.cand...@pobox.com> wrote: > Just a random thought, but have you tried 1.16 > with GODEBUG=asyncpreemptoff=1 ? The preemptive scheduling stuff was > introduced in 1.14 I believe. > > On Thursday, 9 September 2021 at 16:28:38 UTC+1 varun...@gmail.com wrote: > >> @Kurtis, Thanks for the reply. >> >> Yes. We upgraded to 1.16.5 due to this issue and had to revert back to >> 1.13.7 (over earlier working version) as we are frequently seeing these >> errors in 1.16.5. >> >> I looked into all the related issues on golang forums. We are not on >> NetBSD (We are on CentOS7.8) >> >> We also suspect it is a bug in CGO code (specifically some memory >> violation) but finding the bug is like finding needle in a haystack. The >> SIGSEGV CGO signal handler did not catch anything at the time of this >> error. The error message by itself does not mention any thing debuggable. >> There is no stack or core dump. We tried working with GOTRACEBACK but that >> did not generate any core dump as well. >> >> Thanks, >> Varun >> >> >> On Thursday, September 9, 2021 at 8:07:00 PM UTC+5:30 Kurtis Rader wrote: >> >>> Googling "go morestack on gsignal" turns up quite a few reports. Some of >>> which involved kernel bugs (e.g., >>> https://github.com/golang/go/issues/19652) but most seem to involve >>> SIGSEGV errors in non-kernel code; sometimes the Go runtime or stdlib >>> (e.g., https://github.com/golang/go/issues/35235) and sometimes user >>> code. Given that you're using CGO a likely explanation is a bug in your C >>> code. However, you say you're using Go 1.13.7 which is getting long in the >>> tooth. Are you sure you're seeing the same failure with Go 1.16.5? If yes >>> then I would bet the bug is in your C code. If no then it could be a Go bug >>> that has likely already been fixed. >>> >>> On Thu, Sep 9, 2021 at 4:15 AM varun...@gmail.com <varun...@gmail.com> >>> wrote: >>> >>>> Minor update: >>>> >>>> a. The crash with "morestack on gsignal" happens to be independent of >>>> the kernel versions mentioned earlier >>>> b. We implemented a signal handler to catch SIGSEGV from CGO. That is >>>> not helping either >>>> c. From system audit logs, we can see the following message: >>>> >>>> ./audit/audit.log.2:22944:type=ANOM_ABEND >>>> msg=audit(1630376735.155:5486532): auid=4294967295 uid=996 gid=994 >>>> ses=4294967295 subj=system_u:system_r:unconfined_service_t:s0 pid=26422 >>>> comm="indexer" reason="memory violation" sig=5 >>>> >>>> Sig=5 is SIGTRAP. At this point, we suspect there is a SIGSEGV in CGO >>>> layer and that is being returned as SIGTRAP by runtime. >>>> >>>> I am still clueless as to why a process can crash with "morestack on >>>> gsignal". Any pointers to debug this further would be of great help. >>>> >>>> Thanks, >>>> Varun >>>> On Wednesday, September 1, 2021 at 2:44:52 PM UTC+5:30 >>>> varun...@gmail.com wrote: >>>> >>>>> Hello, >>>>> >>>>> Recently we have been seeing multiple occurrences of "fatal: morestack >>>>> on gsignal" error in our system tests. The issue happens with golang >>>>> 1.13.7 >>>>> and golang 1.16.5 runtimes. The process restarts after this error and no >>>>> stack trace is dumped. Our process uses CGO calls, I/O ops etc. >>>>> >>>>> So far, the issue seem to happen only on CentOS7 with linux kernel >>>>> version: 3.10.0-1127.19.1.el7.x86_64. The issue is not seen on: >>>>> 3.10.0-693.5.2.el7.x86_64 >>>>> >>>>> At this point, we have no clue as to why this is happening and no >>>>> stack trace is making it difficult to debug. Any pointers on debugging the >>>>> issue would be of great help. >>>>> >>>>> Thanks, >>>>> Varun >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "golang-nuts" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to golang-nuts...@googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/golang-nuts/9f36ab30-3222-4eed-8388-93ecd259bf55n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/golang-nuts/9f36ab30-3222-4eed-8388-93ecd259bf55n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Kurtis Rader >>> Caretaker of the exceptional canines Junior and Hank >>> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "golang-nuts" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/golang-nuts/msVmzxWjIOI/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/4e6c8f7d-86be-4ecf-b4f5-19554f3d1379n%40googlegroups.com > <https://groups.google.com/d/msgid/golang-nuts/4e6c8f7d-86be-4ecf-b4f5-19554f3d1379n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMzZVzBrwcw%2BMJOjvXgg_YhFA0ZgYD%3DiLhYJyYHCxzXFKYD%2BeQ%40mail.gmail.com.
