Thanks Jeremy! replies below On Friday, September 6, 2019 at 7:53:00 AM UTC-7, Jeremy Evans wrote: > > On Thursday, September 5, 2019 at 11:06:25 PM UTC-7, Ryan W wrote: >> >> Recently, we began experiencing more of these during rolling restarts and >> busy periods. >> >> Sequel::DatabaseDisconnectError: Mysql2::Error::ConnectionError: Lost >> connection to MySQL server during query >> > > Does this error only occur when doing a rolling restart during a busy > period, or does it occur when doing a rolling restart (when not busy) and > during busy periods (even when not doing a rolling restart)? >
It happens during all rolling restarts from what I've observed, which is where we first started seeing it. As load has increased it has started happening more throughout other times well after restarts, perhaps when passenger is both spawning more processes or shutting down processes. I'll try to get more specific details about that behavior. I've also been able to reproduce occasionally on a test environment under load when stopping nginx while the Passenger queue is full, or when triggering a rolling Passenger restart when the queue is full. > In either case, it's challenging to handle this error because you probably > can't use the connection_validator extension to handle it. If the > connection is lost during a query, you can probably only use the :retry_on > feature with Database#transaction, and then you need to be sure the rest of > the transaction block is idempotent. > > >> Setup >> * Sequel multiple databases (Primary/Replicas) are configured >> * sequel gem version 5.20.0 >> * mysql2 gem version 0.5.2 >> * GCP Cloud SQL MySQL 5.7 >> * Using Passenger smart spawning >> * Passenger concurrency method = process (single threaded) >> >> In reviewing the sequel/passenger docs, the initial attempt to eliminate >> or reduce the errors was to add `DB.disconnect` in config.ru (pre fork) >> to avoid sharing connections between processes. The errors came back >> though. We will test further with the after fork hooks with disconnect. >> > > You definitely want to disconnect before forking. You could turn off > smart spawning as a test and see if the errors go away. That will at least > confirm the problem is related to connection sharing. > I did test this, and still saw errors (outside of a rolling restart) when smart spawning was turned off. I'm going to try it again though. > > >> However, I noticed the errors are alwas on the replica connections. The >> sequel source[1] mentions that all connections will be closed, but do the >> replica connections need to be cleaned up explicitly? >> > > DB.disconnect will disconnect all connections for all shards unless you > pass a :server option to specify specific shards. > Ok, thanks for confirming. > Thanks, > Jeremy > Thanks! Ryan -- You received this message because you are subscribed to the Google Groups "sequel-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/sequel-talk/8fb8dca5-ba13-49db-8596-e8bab93d6c88%40googlegroups.com.
