Re: Perplexing, regular decline in performance

2019-07-18 Thread Andres Freund
Hi, On 2019-07-18 16:01:46 -0400, Hugh Ranalli wrote: > I've been going by a couple of articles I found about interpreting > pg_buffercache ( > https://www.keithf4.com/a-large-database-does-not-mean-large-shared_buffers), > and so far shared buffers look okay. Our database is 486 GB, with shared >

Re: Perplexing, regular decline in performance

2019-07-18 Thread Hugh Ranalli
On Tue, 25 Jun 2019 at 12:23, Justin Pryzby wrote: > It's possible that the "administrative" queries are using up lots of your > shared_buffers, which are (also/more) needed by the customer-facing > queries. I > would install pg_buffercache to investigate. Or, just pause the admin > queries > a

Re: Perplexing, regular decline in performance

2019-07-17 Thread Andres Freund
Hi On 2019-07-17 13:55:51 -0400, Alvaro Herrera wrote: > Be careful with pg_buffercache though, as it can cause a hiccup in > operation. I think that's been fixed a few years back: commit 6e654546fb61f62cc982d0c8f62241b3b30e7ef8 Author: Heikki Linnakangas Date: 2016-09-29 13:16:30 +0300

Re: Perplexing, regular decline in performance

2019-07-17 Thread Alvaro Herrera
On 2019-Jun-26, Justin Pryzby wrote: > > Also, Should pg_buffercache perhaps be run at the beginning and end of the > > week, to see if there is a significant difference? > > Yes; buffercache can be pretty volatile, so I'd save it numerous times each at > beginning and end of week. Be careful wi

Re: Perplexing, regular decline in performance

2019-07-17 Thread Hugh Ranalli
On Wed, 26 Jun 2019 at 15:18, Tom Lane wrote: > Alvaro Herrera writes: > > On 2019-Jun-26, Hugh Ranalli wrote: > >> From my research in preparing for the upgrade, I understood transparent > >> huge pages were a good thing, and should be enabled. Is this not > correct? > > > It is not. > > Yeah .

Re: Perplexing, regular decline in performance

2019-06-26 Thread Justin Pryzby
On Wed, Jun 26, 2019 at 03:00:43PM -0400, Hugh Ranalli wrote: > Pausing the admin queries isn't an option in our environment, especially as > the issue reveals itself over the course of days, not minutes or hours. Perhaps you can pause it for a short while at EOW and see if there's a dramatic impr

Re: Perplexing, regular decline in performance

2019-06-26 Thread Tom Lane
Alvaro Herrera writes: > On 2019-Jun-26, Hugh Ranalli wrote: >> From my research in preparing for the upgrade, I understood transparent >> huge pages were a good thing, and should be enabled. Is this not correct? > It is not. Yeah ... they would be a good thing perhaps if the quality of the kern

Re: Perplexing, regular decline in performance

2019-06-26 Thread Alvaro Herrera
On 2019-Jun-26, Hugh Ranalli wrote: > From my research in preparing for the upgrade, I understood transparent > huge pages were a good thing, and should be enabled. Is this not correct? It is not. > Wouldn't the plan be the same at both > the start of the week (when the problematic table is esse

Re: Perplexing, regular decline in performance

2019-06-26 Thread Peter Geoghegan
On Wed, Jun 26, 2019 at 12:02 PM Hugh Ranalli wrote: > I'm sorry, but I'm not sure what you mean by the "distribution of values > within the columns." Can you clarify or provide an link to an example? I would mostly just like to see the schema of the table in question, including indexes, and a h

Re: Perplexing, regular decline in performance

2019-06-26 Thread Hugh Ranalli
On Wed, 26 Jun 2019 at 14:52, Peter Geoghegan wrote: > Can you show us the definition of the table, including its indexes? > Can you describe the data and distribution of values within the > columns, particularly where they're indexed? > I'm sorry, but I'm not sure what you mean by the "distribu

Re: Perplexing, regular decline in performance

2019-06-26 Thread Hugh Ranalli
On Tue, 25 Jun 2019 at 12:23, Justin Pryzby wrote: > What kernel? Version? OS? > Ubuntu 18.04; current kernel is 4.15.0-51-generic4 If Linux, I wonder if transparent hugepages or KSM are enabled ? It seems > possible that truncating the table is clearing enough RAM to mitigate the > issue, si

Re: Perplexing, regular decline in performance

2019-06-26 Thread Peter Geoghegan
On Tue, Jun 25, 2019 at 8:49 AM Hugh Ranalli wrote: > What we continued to notice was a milder but still definite trend of > increased query times, during the course of each week, from the mid to high > 200 ms, to the high 300 ms to low 400 ms. Some years ago, someone had noticed > that as the

Re: Perplexing, regular decline in performance

2019-06-26 Thread Hugh Ranalli
On Tue, 25 Jun 2019 at 11:55, Benjamin Scherrey wrote: > Have you done a VACUUM ANALYZE FULL on your database? This needs to be > done periodically to inform the server of the statistics of how the data > and relations are distributed across the database. Without this bad > assumptions by the pla

Re: Perplexing, regular decline in performance

2019-06-25 Thread Justin Pryzby
On Tue, Jun 25, 2019 at 11:49:03AM -0400, Hugh Ranalli wrote: > I'm hoping people can help me figure out where to look to solve an odd > PostgreSQL performance problem. What kernel? Version? OS? If Linux, I wonder if transparent hugepages or KSM are enabled ? It seems possible that truncating

Re: Perplexing, regular decline in performance

2019-06-25 Thread Benjamin Scherrey
I didn't say do it all the time, I said if he hasn't done it already he should try that as a way of ensuring the database server's understanding of the data as it stands is correct. Otherwise there isn't enough information to suggest other solutions as there is no description of the operating syste

Re: Perplexing, regular decline in performance

2019-06-25 Thread Justin Pryzby
On Tue, Jun 25, 2019 at 10:55:22PM +0700, Benjamin Scherrey wrote: > Have you done a VACUUM ANALYZE FULL on your database? This needs to be done > periodically to inform the server of the statistics of how the data and > relations are distributed across the database. I think this is wrong. VACUUM

Re: Perplexing, regular decline in performance

2019-06-25 Thread Benjamin Scherrey
Have you done a VACUUM ANALYZE FULL on your database? This needs to be done periodically to inform the server of the statistics of how the data and relations are distributed across the database. Without this bad assumptions by the planner can cause degradation of performance. Also, if you are using