Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Thomas Munro
On Fri, Aug 21, 2020 at 9:58 AM Jim Jarvie wrote: > However, as I scale up the number of concurrent connections, I see a spike in > CPU (to 100% across 80 cores) when the SELECT FOR UPDATE SKIP LOCKED executes > and the select processes wait for multiple minutes (10-20 minutes) before > complet

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread David Rowley
On Fri, 21 Aug 2020 at 11:01, Michael Lewis wrote: > > On Thu, Aug 20, 2020 at 4:40 PM Jim Jarvie wrote: >> >> On 20-Aug.-2020 17:42, Michael Lewis wrote: >> >> Can you share an explain analyze for the query that does the select for >> update? I wouldn't assume that partition pruning is possible

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Michael Lewis
On Thu, Aug 20, 2020 at 4:40 PM Jim Jarvie wrote: > On 20-Aug.-2020 17:42, Michael Lewis wrote: > > Can you share an explain analyze for the query that does the select for > update? I wouldn't assume that partition pruning is possible at all with > hash, and it would be interesting to see how it

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Jim Jarvie
On 20-Aug.-2020 17:42, Michael Lewis wrote: Can you share an explain analyze for the query that does the select for update? I wouldn't assume that partition pruning is possible at all with hash, and it would be interesting to see how it is finding those rows. Well this got interesting  - the a

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Michael Lewis
Can you share an explain analyze for the query that does the select for update? I wouldn't assume that partition pruning is possible at all with hash, and it would be interesting to see how it is finding those rows. >

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Jim Jarvie
On 20-Aug.-2020 13:30, Michael Lewis wrote: Great to hear that some of the issues are now mitigated. Though, perhaps you actually require that ORDER BY if items are expected to be sitting in the queue quite some time because you have incoming queue items in a burst pattern and have to play catch

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Michael Lewis
Great to hear that some of the issues are now mitigated. Though, perhaps you actually require that ORDER BY if items are expected to be sitting in the queue quite some time because you have incoming queue items in a burst pattern and have to play catch up sometimes. If so, I highly suspect the inde

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Jim Jarvie
Updates added, mixture of good and bad news: On 18-Aug.-2020 20:39, Michael Lewis wrote: How long does each process take in total? How strict does that FIFO really need to be when you are already doing SKIP LOCKED anyway? The processes all bottleneck[ed] for several minutes, approximately exp