> > Ok, so your data collectors never complete. A simple change to make this > easier to diagnose is to not spin up another collector controller at the 2 > min mark if the previous has not completed. > > I would determine if the stuck collector is BQ or Elastic and check the > server side logs. > We start a data collector (DC) every 2 mins irrespective of whether the previous DC has completed or not, so we would have a few DCs stuck but not all. (2 from backtrace)
I would determine if the stuck collector is BQ or Elastic and check the > server side logs. > In server side logs we don't see any activity at all. That is the confusing part why regular user requests are not reaching the go server from the users. Will check BQ and elastic logs around that time. But after restart everything was working fine. Could be a bug in the http/2 implementation. I would disable http/2 and see > if you encounter the problem. > If we hit it again we will collect github.com/robaho/goanalyzer & see what is happening. Between is there a compile time flag to disable http2 while building ? Thanks, On Thu, Aug 27, 2020 at 4:05 AM Robert Engels <reng...@ix.netcom.com> wrote: > Could be a bug in the http/2 implementation. I would disable http/2 and > see if you encounter the problem. > > On Aug 27, 2020, at 6:02 AM, Robert Engels <reng...@ix.netcom.com> wrote: > > > Ok, so your data collectors never complete. A simple change to make this > easier to diagnose is to not spin up another collector controller at the 2 > min mark if the previous has not completed. > > I would determine if the stuck collector is BQ or Elastic and check the > server side logs. > > On Aug 26, 2020, at 11:29 PM, Kurtis Rader <kra...@skepticism.us> wrote: > > > On Wed, Aug 26, 2020 at 8:51 PM Siddhesh Divekar < > siddhesh.dive...@gmail.com> wrote: > >> Right, then it looks less likely that we are blocked on a mutex. >> >> Every 2 minutes we spin up a go routine which then in turn spins up a >> bunch of go routines to collect data from big query & elastic (data >> collector routines). >> The 2 minutes go routine then in turn waits on waitgroup for data >> collector routines to come back. So its not single go routine from our side >> at least. >> From backtrace we have from sigabort we see only one such data collector >> go routine blocked and other 2 2 mins thread waiting on waitgroup. >> > > Are you spinning up a Go routine every 2 minutes regardless of whether the > previous instance had completed? Your comment is not clear whether you have > one, or more than one, goroutine issuing a BigQuery query at the time you > saw the problem and captured the backtraces. If you are, in fact, starting > those Go routines without regard to whether the prior instance had > completed that seems like a "yellow flag", "here be dragons", situation. > > -- > Kurtis Rader > Caretaker of the exceptional canines Junior and Hank > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-y%3DQg_dx5W0qDWiMV9sB0eFToTEoiD83_mbn7RHC%3DQ%3Dg%40mail.gmail.com > <https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-y%3DQg_dx5W0qDWiMV9sB0eFToTEoiD83_mbn7RHC%3DQ%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- -Siddhesh. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAMjfk%2Bhc%3DYyBByn0txHK5MqR6wA%3DUfWsj49TAov2ticeSbWSQg%40mail.gmail.com.
