Re: [PERFORM] Queries intermittently slow

2016-01-07 Thread Jim Nasby
On 1/7/16 1:47 PM, Andres Freund wrote: On 2016-01-07 13:34:51 -0500, Tom Lane wrote: It's fairly well established that the implementation of transparent huge pages in Linux kernels from the 2.6-or-so era sucks, and you're best off turning it off if you care about consistency of performance. I

Re: [PERFORM] Queries intermittently slow

2016-01-07 Thread Andres Freund
On 2016-01-07 13:34:51 -0500, Tom Lane wrote: > It's fairly well established that the implementation of transparent > huge pages in Linux kernels from the 2.6-or-so era sucks, and you're > best off turning it off if you care about consistency of performance. I think the feature wasn't introduced i

Re: [PERFORM] Queries intermittently slow

2016-01-07 Thread Tom Lane
Rick Otten writes: > On Thu, Jan 7, 2016 at 12:51 PM, Scott Rankin wrote: >> Winner! Both of those settings were set to always, and as soon as I >> turned them off, the query times smoothed right out. > Is this generally true for all PostgreSQL systems on Linux, or only for > specific use cases

Re: [PERFORM] Queries intermittently slow

2016-01-07 Thread Scott Rankin
On 1/7/16, 10:19 AM, "Tom Lane" wrote: >Scott Rankin writes: >> Any other ideas? > >If there's no lock waits, and the tables are static, it's difficult to >arrive at any other conclusion but that the process is simply losing the >CPU for long intervals. What else is going on on this box? Have

Re: [PERFORM] Queries intermittently slow

2016-01-07 Thread Tom Lane
Scott Rankin writes: > Any other ideas? If there's no lock waits, and the tables are static, it's difficult to arrive at any other conclusion but that the process is simply losing the CPU for long intervals. What else is going on on this box? Have you made any attempt to correlate the slow quer

Re: [PERFORM] Queries intermittently slow

2016-01-07 Thread Scott Rankin
On 1/6/16, 11:14 PM, "Jim Nasby" wrote: >On 1/6/16 12:01 PM, Scott Rankin wrote: >> I guess we’re back to lock contention? > >Is there by chance an anti-wraparound vacuum happening on that table? > >Actually, for that matter... if autovacuum is hitting that table it's >locking could be causing p

Re: [PERFORM] Queries intermittently slow

2016-01-06 Thread Jim Nasby
On 1/6/16 12:01 PM, Scott Rankin wrote: I guess we’re back to lock contention? Is there by chance an anti-wraparound vacuum happening on that table? Actually, for that matter... if autovacuum is hitting that table it's locking could be causing problems, and it won't release it's locks until

Re: [PERFORM] Queries intermittently slow

2016-01-06 Thread Scott Rankin
> >So what seems more likely after more thought is that the pages are staying >in our shared buffer arena just fine, but the kernel is randomly choosing >to swap out parts of the arena, and the delays correspond to swap-in >waits. (There would still have to be a mighty crummy disk subsystem >under

Re: [PERFORM] Queries intermittently slow

2016-01-06 Thread Tom Lane
I wrote: > A possible theory is that the slow cases represent times when the desired > page is not in cache, but you'd have to have a seriously overloaded disk > subsystem for a disk fetch to take hundreds of ms. Unless maybe this is > running on some cloud service with totally unspecified I/O ban

Re: [PERFORM] Queries intermittently slow

2016-01-06 Thread Scott Rankin
> >Hm, well, given that you are able to capture instances of the behavior >in EXPLAIN ANALYZE, I'd suggest trying EXPLAIN (ANALYZE,BUFFERS). >That will tell you the number of pages it found in shared buffers vs. >having to read them. Now, a "read" just means we had to ask the kernel, >not necessar

Re: [PERFORM] Queries intermittently slow

2016-01-06 Thread Tom Lane
Scott Rankin writes: > On 1/6/16, 10:38 AM, "Tom Lane" wrote: >> A possible theory is that the slow cases represent times when the desired >> page is not in cache, but you'd have to have a seriously overloaded disk >> subsystem for a disk fetch to take hundreds of ms. Unless maybe this is >> run

Re: [PERFORM] Queries intermittently slow

2016-01-06 Thread Scott Rankin
On 1/6/16, 10:38 AM, "Tom Lane" wrote: > >A possible theory is that the slow cases represent times when the desired >page is not in cache, but you'd have to have a seriously overloaded disk >subsystem for a disk fetch to take hundreds of ms. Unless maybe this is >running on some cloud service wi

Re: [PERFORM] Queries intermittently slow

2016-01-06 Thread Tom Lane
Scott Rankin writes: > Thanks for the update. The query in question is a pretty simple one - it > joins 3 tables, all of which are static - they don’t have any writes being > done against them. They have very few rows, and the query plan for them > indicates that they are all sequential scan

Re: [PERFORM] Queries intermittently slow

2016-01-06 Thread Scott Rankin
On 1/6/16, 10:19 AM, "Tom Lane" wrote: >Scott Rankin writes: >> I’m trying to track down why some queries on my database system are >> intermittently much slower than usual. I have some queries that run, on >> average, 2-3ms, and they run at a rate of about 10-20 queries/second. >> However

Re: [PERFORM] Queries intermittently slow

2016-01-06 Thread Tom Lane
Scott Rankin writes: > I’m trying to track down why some queries on my database system are > intermittently much slower than usual. I have some queries that run, on > average, 2-3ms, and they run at a rate of about 10-20 queries/second. > However, every 3-5 seconds, one of the queries will

[PERFORM] Queries intermittently slow

2016-01-06 Thread Scott Rankin
Hi all, I’m trying to track down why some queries on my database system are intermittently much slower than usual. I have some queries that run, on average, 2-3ms, and they run at a rate of about 10-20 queries/second. However, every 3-5 seconds, one of the queries will be 500-100ms. This is