Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-08-02 Thread Heikki Linnakangas
On 07/30/2015 04:23 PM, Spiros Ioannou wrote: I'm very sorry but we don't have a synthetic load generator for our testing setup, only production and that is on SLA. I would be happy to test the next release though. Ok, no worries. I've pushed this changes, it will appear in the next release.

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-30 Thread Tatsuo Ishii
Heikki, A-ha, I succeeded to reproduce this now on my laptop, with pgbench! It seems to be important to have a very large number of connections: pgbench -n -c400 -j4 -T600 -P5 That got stuck after a few minutes. I'm using commit_delay=100. Because the original reporter seems to have

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-30 Thread Scott Marlowe
You might want to try pg replay: http://laurenz.github.io/pgreplay/ On Thu, Jul 30, 2015 at 7:23 AM, Spiros Ioannou siv...@inaccess.com wrote: I'm very sorry but we don't have a synthetic load generator for our testing setup, only production and that is on SLA. I would be happy to test the

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-30 Thread Spiros Ioannou
That would need a replica of the data probably which is not possible (tablespace is 4TB). *Spiros Ioannou IT Manager, inAccesswww.inaccess.com http://www.inaccess.comM: +30 6973-903808T: +30 210-6802-358* On 30 July 2015 at 21:47, Scott Marlowe scott.marl...@gmail.com wrote: You might

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-29 Thread Heikki Linnakangas
On 07/28/2015 11:36 PM, Heikki Linnakangas wrote: A-ha, I succeeded to reproduce this now on my laptop, with pgbench! It seems to be important to have a very large number of connections: pgbench -n -c400 -j4 -T600 -P5 That got stuck after a few minutes. I'm using commit_delay=100. Now that I

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-28 Thread Spiros Ioannou
Hmm. With that CTE query, were there other queries running at the same time? yes, a multitude of INSERT and one COMMIT as I remember. I've noticed that in every stuck situation there was always 1 stuck COMMIT running. We do mainly INSERT and SELECT, very rarely UPDATE or DELETE. but the stuck

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-28 Thread Heikki Linnakangas
A-ha, I succeeded to reproduce this now on my laptop, with pgbench! It seems to be important to have a very large number of connections: pgbench -n -c400 -j4 -T600 -P5 That got stuck after a few minutes. I'm using commit_delay=100. Now that I have something to work with, I'll investigate this

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-27 Thread Heikki Linnakangas
On 07/23/2015 02:36 PM, Heikki Linnakangas wrote: On 07/23/2015 11:31 AM, Spiros Ioannou wrote: Well, so far with commit_delay=0 no problems. I will report back of couse if something happens, but I believe that the problem may indeed be solved/masked with that setting. Rough description of our

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-23 Thread Heikki Linnakangas
On 07/23/2015 11:31 AM, Spiros Ioannou wrote: Well, so far with commit_delay=0 no problems. I will report back of couse if something happens, but I believe that the problem may indeed be solved/masked with that setting. Rough description of our setup, or how to reproduce: * Timeseries data in

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-23 Thread Spiros Ioannou
Well, so far with commit_delay=0 no problems. I will report back of couse if something happens, but I believe that the problem may indeed be solved/masked with that setting. Rough description of our setup, or how to reproduce: * Timeseries data in table , say, measurements, size: 3-4TB, about

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Spiros Ioannou
Hi Tom, thank you for your input. The DB was stuck again, I attach all logs and stack traces. A stack trace from a COMMIT, an INSERT, an UPDATE, the wal writer, the writer, and a sequence. Stracing the commit was stuck at: semop(3145761, {{12, -1, 0}}, 1 Please tell me what else could I do to

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Alvaro Herrera
Spiros Ioannou wrote: Hi Tom, thank you for your input. The DB was stuck again, I attach all logs and stack traces. A stack trace from a COMMIT, an INSERT, an UPDATE, the wal writer, the writer, and a sequence. Stracing the commit was stuck at: semop(3145761, {{12, -1, 0}}, 1 Hmm, isn't

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Andres Freund
On 2015-07-20 13:06:51 +0200, Alvaro Herrera wrote: Spiros Ioannou wrote: Hi Tom, thank you for your input. The DB was stuck again, I attach all logs and stack traces. A stack trace from a COMMIT, an INSERT, an UPDATE, the wal writer, the writer, and a sequence. Stracing the

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Andres Freund
On 2015-07-20 13:22:42 +0200, Andres Freund wrote: Hm. The problem seems to be the WaitXLogInsertionsToFinish() call in XLogFlush(). These are the relevant stack traces: db9lock/debuglog-commit.txt #2 0x7f7405bd44f4 in LWLockWaitForVar (l=0x7f70f2ab6680, valptr=0x7f70f2ab66a0,

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Andres Freund
Heikki, On 2015-07-20 13:27:12 +0200, Andres Freund wrote: On 2015-07-20 13:22:42 +0200, Andres Freund wrote: Hm. The problem seems to be the WaitXLogInsertionsToFinish() call in XLogFlush(). These are the relevant stack traces: db9lock/debuglog-commit.txt #2 0x7f7405bd44f4 in

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Andres Freund
Hi, On 2015-07-20 15:58:33 +0300, Spiros Ioannou wrote: Happened again, another backtrace from a COMMIT process. I changed the commit_delay to 0 (it was 4000 to help with our storage) and will report back. What hardware OS is this happening on? Regards, Andres -- Sent via pgsql-general

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Spiros Ioannou
Debian 6, x64, HP Proliant, dbsize: 3TB on EMC AX-4i (iscsi), 16 cores, 24GB RAM. kernel: Linux vserver11 2.6.32-5-xen-amd64 #1 SMP Sun Dec 7 22:52:42 UTC 2014 x86_64 GNU/Linux I attach postgresql.conf *Spiros Ioannou IT Manager, inAccesswww.inaccess.com http://www.inaccess.comM: +30

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Heikki Linnakangas
On 07/20/2015 03:01 PM, Andres Freund wrote: Heikki, On 2015-07-20 13:27:12 +0200, Andres Freund wrote: On 2015-07-20 13:22:42 +0200, Andres Freund wrote: Hm. The problem seems to be the WaitXLogInsertionsToFinish() call in XLogFlush(). These are the relevant stack traces:

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Spiros Ioannou
Happened again, another backtrace from a COMMIT process. I changed the commit_delay to 0 (it was 4000 to help with our storage) and will report back. Already logging to debuglog.txt. #0 0x7f47b0789ec7 in semop () from /lib/libc.so.6 #1 0x7f47b2513d91 in PGSemaphoreLock

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Spiros Ioannou
FYI we have an 9.3.5 with commit_delay = 4000 and commit_siblings = 5 with a 8TB dataset which seems fine. (Runs on different - faster hardware though). *Spiros Ioannou IT Manager, inAccesswww.inaccess.com http://www.inaccess.comM: +30 6973-903808T: +30 210-6802-358* On 20 July 2015 at

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-20 Thread Andres Freund
On 2015-07-20 17:00:52 +0300, Spiros Ioannou wrote: FYI we have an 9.3.5 with commit_delay = 4000 and commit_siblings = 5 with a 8TB dataset which seems fine. (Runs on different - faster hardware though). 9.4 has a different xlog insertion algorithm (scaling much better), so that unfortunately

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-19 Thread Spiros Ioannou
Os is debian 6

[GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-19 Thread Spiros Ioannou
We upgraded to 9.4.4 from 8.4 We have tried 9.3.x in the past with success. Unfortunately in 9.4 gets stuck 2-3 times a week. Lots of queries seem stuck, that is they have waiting:false. Queries can be COMMIT or INSERT. No blocking locks seem to exist. strace reveals that all stuck processes they

Re: [GENERAL] Lots of stuck queries after upgrade to 9.4

2015-07-19 Thread Tom Lane
Spiros Ioannou siv...@inaccess.com writes: We upgraded to 9.4.4 from 8.4 We have tried 9.3.x in the past with success. Unfortunately in 9.4 gets stuck 2-3 times a week. Lots of queries seem stuck, that is they have waiting:false. Queries can be COMMIT or INSERT. No blocking locks seem to