Re: [GENERAL] Possible multiprocess lock/unlock-loop problem in Postgresql 9.2

2014-03-15 Thread Yngve N. Pettersen
Hi again, A further update, and it looks like I have finally been able to fix the problem. I used gdb to discover where the process is hanging. As far as I can tell, the processes are looping inside ExecScan calling ExecQual calling ExecEvalScalarArrayOp ExecScan was

Re: [GENERAL] Possible multiprocess lock/unlock-loop problem in Postgresql 9.2

2014-03-15 Thread Yngve N. Pettersen
Hmmm, just tested with the extra conditional indexes in the production system, and it still took 19 minutes for the first group of queries (23 million entries, 6000 record updates, 8 processes). Afterwards, there were no such delays. Munin reports that during those 19 minutes there were,

Re: [GENERAL] Possible multiprocess lock/unlock-loop problem in Postgresql 9.2

2014-03-09 Thread Yngve N. Pettersen
Hi again, I have now had time to do further research about this issue. I have been able to produce a script (available on request) that reproduces the problem, even in tables as small as 100 items and using a single thread, and as a result may have located an area that may cause the problem: A

[GENERAL] Possible multiprocess lock/unlock-loop problem in Postgresql 9.2

2014-01-04 Thread Yngve N. Pettersen
Hello all, I am running a Postgresql 9.2 system (IIRC v9.2.4; I am about to upgrade to 9.2.6) on a system with 32-cores, 256GB RAM, 64GB shared RAM for postgresql. The applications I am running are written using Django (currently v1.5) For a while I have been observing what may be

Re: [GENERAL] Possible multiprocess lock/unlock-loop problem in Postgresql 9.2

2014-01-04 Thread Andrew Sullivan
On Sat, Jan 04, 2014 at 12:14:42PM +0100, Yngve N. Pettersen wrote: The update query looks like this: UPDATE queue SET state = E'S' WHERE state = E'I' AND id IN (list of integers) RETURNING id; There is a BEGIN/COMMIT wrap around the operation, including the SELECT query. Am I right

Re: [GENERAL] Possible multiprocess lock/unlock-loop problem in Postgresql 9.2

2014-01-04 Thread Yngve N. Pettersen
Hi, On Sat, 04 Jan 2014 16:23:42 +0100, Andrew Sullivan a...@crankycanuck.ca wrote: On Sat, Jan 04, 2014 at 12:14:42PM +0100, Yngve N. Pettersen wrote: The update query looks like this: UPDATE queue SET state = E'S' WHERE state = E'I' AND id IN (list of integers) RETURNING id; There is a

Re: [GENERAL] Possible multiprocess lock/unlock-loop problem in Postgresql 9.2

2014-01-04 Thread Andrew Sullivan
On Sat, Jan 04, 2014 at 07:07:08PM +0100, Yngve N. Pettersen wrote: I tried that before, but ran into some issues, IIRC a similar looping problem as this where queries never ended. I split it up in an attempt to solve that problem. Pulling the data out into the application and sending it back

Re: [GENERAL] Possible multiprocess lock/unlock-loop problem in Postgresql 9.2

2014-01-04 Thread Yngve N. Pettersen
On Sat, 04 Jan 2014 19:40:31 +0100, Andrew Sullivan a...@crankycanuck.ca wrote: On Sat, Jan 04, 2014 at 07:07:08PM +0100, Yngve N. Pettersen wrote: I tried that before, but ran into some issues, IIRC a similar looping problem as this where queries never ended. I split it up in an attempt