Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-09-11 Thread Jim Jarvie
Hi Gunther On 07-Sep.-2020 14:04, r...@gusw.net wrote: Hi all, especially Jim Jarvie, I saw your email to me only now on my related issue. My issue remains this one: ---8<--- I remain with my original question from 30th of June, to me it feels like a bug of some sort: "tuple to b

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-21 Thread Jim Jarvie
Merging a couple of emails: On 20-Aug.-2020 19:37, David Rowley wrote: When updates occur in a non-partitioned table we can follow item pointer chains to find the live row and check if the WHERE clause still matches to determine if the row should be updated, or in this case just locked since it

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Jim Jarvie
On 20-Aug.-2020 17:42, Michael Lewis wrote: Can you share an explain analyze for the query that does the select for update? I wouldn't assume that partition pruning is possible at all with hash, and it would be interesting to see how it is finding those rows. Well this got interesting  - the a

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Jim Jarvie
On 20-Aug.-2020 13:30, Michael Lewis wrote: Great to hear that some of the issues are now mitigated. Though, perhaps you actually require that ORDER BY if items are expected to be sitting in the queue quite some time because you have incoming queue items in a burst pattern and have to play catch

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-20 Thread Jim Jarvie
Updates added, mixture of good and bad news: On 18-Aug.-2020 20:39, Michael Lewis wrote: How long does each process take in total? How strict does that FIFO really need to be when you are already doing SKIP LOCKED anyway? The processes all bottleneck[ed] for several minutes, approximately exp

Re: CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-18 Thread Jim Jarvie
Thank you for the quick response. No adjustments of fill factors.  Hadn't though of that - I'll investigate and try some options to see if I can measure an effect. There is some ordering on the select [ ORDER BY q_id] so each block of 250 is sequential-ish queue items; I just need them more o

CPU hogged by concurrent SELECT..FOR UPDATE SKIP LOCKED

2020-08-18 Thread Jim Jarvie
Using V12, Linux [Ubuntu 16.04LTS] I have a system which implements a message queue with the basic pattern that a process selects a group of, for example 250, rows for processing via SELECT .. LIMIT 250 FOR UPDATE SKIP LOCKED. When there are a small number of concurrent connections to process

Re: Is there a known bug with SKIP LOCKED and "tuple to be locked was already moved to another partition due to concurrent update"?

2020-08-11 Thread Jim Jarvie
Hi Gunther & List, I think I have an extremely similar issue and things point in the same direction of a potential issue for skip locked on partitioned tables. Background is I had a queue table on v9.6 with fairly high volume (>50M items, growth in the 1+M/daily). Processing the queue with