Re: Help troubleshooting SubtransControlLock problems

2018-03-08 Thread Scott Frazer
On Wed, Mar 7, 2018 at 3:50 PM, Alvaro Herrera wrote: > > Scott Frazer wrote: > > > It's only happening on the read replicas, though. I've just set my master > > to handle all the traffic, but that's not really sustainable > > I failed to notice at start of thread that these were replicas. I > su

Re: Help troubleshooting SubtransControlLock problems

2018-03-07 Thread Alvaro Herrera
Scott Frazer wrote: > Currently this seems to be happening when the server just starts taking > connections. It doesn't even need time to scale up anymore (previously it > took about 8 hours for the problem to re-appear) > > It's only happening on the read replicas, though. I've just set my maste

Re: Help troubleshooting SubtransControlLock problems

2018-03-07 Thread Scott Frazer
On Wed, Mar 7, 2018 at 10:39 AM, Alvaro Herrera wrote: > > Laurenz Albe wrote: > I think you could get in this situation if the range of open > transactions exceeds what fits in the buffers for subtrans.c pages, and > the subtransaction cache overflows (64 entries apiece; > PGPROC_MAX_CACHED_SUBXI

Re: Help troubleshooting SubtransControlLock problems

2018-03-07 Thread Alvaro Herrera
Laurenz Albe wrote: > Scott Frazer wrote: > > Hi, we have a Postgres 9.6 setup using replication that has recently > > started seeing a lot of processes stuck in > > "SubtransControlLock" as a wait_event on the read-replicas. Like this, only > > usually about 300-800 of them: > > > > > > 17970

Re: Help troubleshooting SubtransControlLock problems

2018-03-07 Thread Laurenz Albe
Scott Frazer wrote: > > On Wed, Mar 7, 2018 at 9:52 AM, Laurenz Albe wrote: > > Scott Frazer wrote: > > > Hi, we have a Postgres 9.6 setup using replication that has recently > > > started seeing a lot of processes stuck in > > > "SubtransControlLock" as a wait_event on the read-replicas. Like t

Re: Help troubleshooting SubtransControlLock problems

2018-03-07 Thread Scott Frazer
Server version is 9.6.5 Is there a decent guide to getting a stack trace on Centos7 when using the official Postgres repo? trying to follow the Fedora guide put the debug info for 9.2.23 on the box instead of the 9.6.5 version. On Wed, Mar 7, 2018 at 9:52 AM, Laurenz Albe wrote: > Scott Frazer

Re: Help troubleshooting SubtransControlLock problems

2018-03-07 Thread Laurenz Albe
Scott Frazer wrote: > Hi, we have a Postgres 9.6 setup using replication that has recently started > seeing a lot of processes stuck in > "SubtransControlLock" as a wait_event on the read-replicas. Like this, only > usually about 300-800 of them: > > > 179706 | LWLockNamed | SubtransContro

Re: Help troubleshooting SubtransControlLock problems

2018-03-07 Thread Scott Frazer
These don't seem like normal locks. Nothing shows up in a "SELECT relation::regclass, * FROM pg_locks WHERE NOT GRANTED;" These processes are all active but the wait_event and wait_event_type fields indicate they are waiting on (I believe) shared memory locks. pid | usesysid | usename|

Re: Help troubleshooting SubtransControlLock problems

2018-03-06 Thread Rene Romero Benavides
For such issues, I find this view very useful (the first one): https://wiki.postgresql.org/wiki/Lock_Monitoring Examine blocking_pid's , and tell us what kind of operation is blocking the other processes . Also, are there many long running transactions in your server? 2018-03-06 21:24 GMT-06:00

Help troubleshooting SubtransControlLock problems

2018-03-06 Thread Scott Frazer
Hi, we have a Postgres 9.6 setup using replication that has recently started seeing a lot of processes stuck in "SubtransControlLock" as a wait_event on the read-replicas. Like this, only usually about 300-800 of them: 179706 | LWLockNamed | SubtransControlLock 186602 | LWLockNamed | S