Re: [PERFORM] Tuning one Recurcive CTE

2016-11-09 Thread Andreas Joseph Krogh
På onsdag 09. november 2016 kl. 15:30:20, skrev Henrik Ekenberg <
hen...@ekenberg.pw >:
Hi,

 I will need to anonymized before sending it.
 Do you know if there is any tuning documents related to CTE scans

You might want to read this:
http://blog.2ndquadrant.com/postgresql-ctes-are-optimization-fences/

https://robots.thoughtbot.com/advanced-postgres-performance-tips#common-table-expressions-and-subqueries

https://www.postgresql.org/message-id/CAPo4y_XUJR1sijvTySy9W%2BShpORwzbhSdEzE9pgtc1%3DcTkvpkw%40mail.gmail.com
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andr...@visena.com 
www.visena.com 
 


 


Re: [PERFORM] Tuning one Recurcive CTE

2016-11-09 Thread Henrik Ekenberg

Hi,

I will need to anonymized before sending it.
Do you know if there is any tuning documents related to CTE scans

//H


På onsdag 09. november 2016 kl. 14:05:55, skrev Henrik Ekenberg
:


Hi,

I try to tune one Recursive CTE.

Explain Plan can be found here
https://explain.depesz.com/s/yLVd

Anyone can give me direction to check?

//H.


   
  Rule number one; Always provide the query in question when asking for
help tuning it.
   
 --ANDREAS JOSEPH KROGH
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andr...@visena.com
www.visena.com[1]
[1]

   




Links:
--
[1] https://www.visena.com


Re: [PERFORM] Tuning one Recurcive CTE

2016-11-09 Thread Andreas Joseph Krogh
På onsdag 09. november 2016 kl. 14:05:55, skrev Henrik Ekenberg <
hen...@ekenberg.pw >:
Hi,

 I try to tune one Recursive CTE.

 Explain Plan can be found here
 https://explain.depesz.com/s/yLVd

 Anyone can give me direction to check?

 //H.

 
Rule number one; Always provide the query in question when asking for help 
tuning it.
 
-- Andreas Joseph Krogh
CTO / Partner - Visena AS
Mobile: +47 909 56 963
andr...@visena.com 
www.visena.com 
 


 


[PERFORM] Tuning one Recurcive CTE

2016-11-09 Thread Henrik Ekenberg

Hi,

I try to tune one Recursive CTE.

Explain Plan can be found here
https://explain.depesz.com/s/yLVd

Anyone can give me direction to check?

//H.


Re: [PERFORM] Tuning Checkpoints

2016-10-31 Thread Tomas Vondra

On 10/31/2016 08:19 PM, Andre Henry wrote:

My PG 9.4.5 server runs on Amazon RDS some times of the day we have a
lot of checkpoints really close (less than 1 minute apart, see logs
below) and we are trying to tune the DB to minimize the impact of the
checkpoint or reduce the number of checkpoints.

Server Stats

· Instance Type db.r3.4xl

• 16 vCPUs 122GB of RAM

• PostgreSQL 9.4.5 on x86_64-unknown-linux-gnu, compiled by gcc
(GCC) 4.8.2 20140120 (Red Hat 4.8.2-16), 64-bit



Some PG Stats

• Shared Buffers = 31427608kB

• Checkpoint Segments = 64

• Checkpoint completion target = .9

• Rest of the configuration is below



Things we are doing

• We have a huge table where each row is over 1kB and its very
busy. We are splitting that into multiple tables especially the one json
field that making it large.



Questions

• Each checkpoint log writes out the following checkpoint
complete: wrote 166481 buffers (4.2%); 0 transaction log file(s) added,
0 removed, 64 recycled; write=32.441 s, sync=0.050 s, total=32.550 s;
sync files=274, longest=0.049 s, average=0.000 s



OK, each checkpoint has to write all dirty data from checkpoints. You 
have ~170k buffers worth of dirty data, i.e. ~1.3GB.



• What does buffers mean? How do I find out how much RAM that is
equivalent to?



Buffer holds 8kB of data, which is the "chunk" of data files.


• Based on my RDS stats I don't think IOPs will help, because I
don't see any flat lines on my write operations / second graph. Is this
a good assumption?



Not sure what you mean by this. Also, maybe you should talk to AWS if 
you're on RDS.



• What else can we tune to spread out checkpoints?



Based on the logs, your checkpoints are triggered by filling WAL. I see 
your checkpoints happen every 30 - 40 seconds, and you only have 64 
segments.


So to get checkpoints checkpoints triggered by timeout (which I assume 
is 5 minutes, because you have not mentioned checkpoint_timeout), you 
need to increase checkpoint_segments enough to hold 5 minutes worth of WAL.


That means 300/30 * 64, i.e. roughly 640 segments (it's likely an 
overestimate, due to full page writes, but well).


regards

--
Tomas Vondra  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-07 Thread Mark Kirkwood

On 08/07/16 02:09, Wes Vaske (wvaske) wrote:

?The Crucial drive does not have power loss protection. The Samsung drive does.


(The Crucial M550 has capacitors to protect data that's already been written to the 
device but not the entire cache. For instance, if data is read from the device during a 
garbage collection operation, the M550 will protect that data instead of introducing 
corruption of old data. This is listed as "power loss protection" on the spec 
sheet but it's not the level of protection that people on this list would expect from a 
drive)



Yes - the MX200 board (see):

http://www.anandtech.com/show/9258/crucial-mx200-250gb-500gb-1tb-ssd-review

looks to have the same sort of capacitors that the M550 uses, so not 
ideal for db or transaction logs!


Cheers

Mark


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-07 Thread Kaixi Luo
> Regarding write back cache:
> Disabling the write back cache won't have a real large impact on the
> endurance of the drive unless it reduces the total number of bytes written
> (which it won't). I've seen drives that perform better with it disabled and
> drives that perform better with it enabled. I would test in your
> environment and make the decision based on performance.
>
>
Thanks. I assume you are referring to the write back cache on the RAID
controller here and not the disk cache itself.

Kaixi


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-07 Thread Scott Marlowe
On Thu, Jul 7, 2016 at 10:27 AM, Merlin Moncure  wrote:
> On Wed, Jul 6, 2016 at 4:48 PM, Scott Marlowe  wrote:
>> On Wed, Jul 6, 2016 at 12:13 PM, Merlin Moncure  wrote:
>>> Disabling write back cache for write heavy database loads will will
>>> destroy it in short order due to write amplication and will generally
>>> cause it to underperform hard drives in my experience.
>>
>> Interesting. We found our best performance with a RAID-5 of 10 800GB
>> SSDs (Intel 3500/3700 series) that we got MUCH faster performance with
>> all write caching turned off on our LSI MEgaRAID controllers. We went
>> from 3 to 4ktps to 15 to 18ktps. And after a year of hard use we still
>> show ~90% life left (these machines handle thousands of writes per
>> second in real use) It could be that the caching was getting in the
>> way of RAID calcs or some other issue. With RAID-1 I have no clue what
>> the performance will be with write cache on or off.
>
> Right -- by that I meant disabling the write back cache on the drive
> itself, so that all writes are immediately flushed.  Disabling write
> back on the raid controller should be the right choice; each of these
> drives essentially is a 'caching raid controller' for all intents and
> purposes.  Hardware raid controllers are engineered around performance
> and reliability assumptions that are no longer correct in an SSD
> world.  Personally I would have plugged the drives directly to the
> motherboard (assuming it's a got enough lanes) and mounted the raid
> against mdadm and compared.

Oh yeah definitely. And yea we've found that mdadm and raw HBAs work
better than most RAID controllers for SSDs.


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-07 Thread Merlin Moncure
On Wed, Jul 6, 2016 at 4:48 PM, Scott Marlowe  wrote:
> On Wed, Jul 6, 2016 at 12:13 PM, Merlin Moncure  wrote:
>> Disabling write back cache for write heavy database loads will will
>> destroy it in short order due to write amplication and will generally
>> cause it to underperform hard drives in my experience.
>
> Interesting. We found our best performance with a RAID-5 of 10 800GB
> SSDs (Intel 3500/3700 series) that we got MUCH faster performance with
> all write caching turned off on our LSI MEgaRAID controllers. We went
> from 3 to 4ktps to 15 to 18ktps. And after a year of hard use we still
> show ~90% life left (these machines handle thousands of writes per
> second in real use) It could be that the caching was getting in the
> way of RAID calcs or some other issue. With RAID-1 I have no clue what
> the performance will be with write cache on or off.

Right -- by that I meant disabling the write back cache on the drive
itself, so that all writes are immediately flushed.  Disabling write
back on the raid controller should be the right choice; each of these
drives essentially is a 'caching raid controller' for all intents and
purposes.  Hardware raid controllers are engineered around performance
and reliability assumptions that are no longer correct in an SSD
world.  Personally I would have plugged the drives directly to the
motherboard (assuming it's a got enough lanes) and mounted the raid
against mdadm and compared.

merlin


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-07 Thread Wes Vaske (wvaske)
?The Crucial drive does not have power loss protection. The Samsung drive does.


(The Crucial M550 has capacitors to protect data that's already been written to 
the device but not the entire cache. For instance, if data is read from the 
device during a garbage collection operation, the M550 will protect that data 
instead of introducing corruption of old data. This is listed as "power loss 
protection" on the spec sheet but it's not the level of protection that people 
on this list would expect from a drive)



From: pgsql-performance-ow...@postgresql.org 
<pgsql-performance-ow...@postgresql.org> on behalf of Kaixi Luo 
<kaixi...@gmail.com>
Sent: Thursday, July 7, 2016 2:49 AM
To: Mark Kirkwood
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

It's a Crucial CT250MX200SSD1 and a Samsung MZ7LM480HCHP-3.

Regards,

Kaixi


On Thu, Jul 7, 2016 at 6:59 AM, Mark Kirkwood 
<mark.kirkw...@catalyst.net.nz<mailto:mark.kirkw...@catalyst.net.nz>> wrote:
On 06/07/16 07:17, Mkrtchyan, Tigran wrote:
Hi,

We had a similar situation and the best performance was with 64MB
background_bytes and 512 MB dirty_bytes.

Tigran.

On Jul 5, 2016 16:51, Kaixi Luo <kaixi...@gmail.com<mailto:kaixi...@gmail.com>> 
wrote:


 Here are my server specs:

 RAID1 - 2x480GB Samsung SSD with power loss protection (will be used to
 store the PostgreSQL database)
 RAID1 - 2x240GB Crucial SSD with power loss protection. (will be used to
 store PostgreSQL transactions logs)


Can you tell the exact model numbers for the Samsung and Crucial SSD's? It 
typically matters! E.g I have some Crucial M550 that have capacitors and 
(originally) claimed to be power off safe, but with testing have been shown to 
be not really power off safe at all. I'd be dubious about Samsungs too.

The Intel Datacenter range (S3700 and similar) are known to have power off 
safety that does work.

regards

Mark



--
Sent via pgsql-performance mailing list 
(pgsql-performance@postgresql.org<mailto:pgsql-performance@postgresql.org>)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance



-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-07 Thread Kaixi Luo
It's a Crucial CT250MX200SSD1 and a Samsung MZ7LM480HCHP-3.

Regards,

Kaixi


On Thu, Jul 7, 2016 at 6:59 AM, Mark Kirkwood  wrote:

> On 06/07/16 07:17, Mkrtchyan, Tigran wrote:
>
>> Hi,
>>
>> We had a similar situation and the best performance was with 64MB
>> background_bytes and 512 MB dirty_bytes.
>>
>> Tigran.
>>
>> On Jul 5, 2016 16:51, Kaixi Luo  wrote:
>>
>>
>>  Here are my server specs:
>>
>>  RAID1 - 2x480GB Samsung SSD with power loss protection (will be used
>> to
>>  store the PostgreSQL database)
>>  RAID1 - 2x240GB Crucial SSD with power loss protection. (will be
>> used to
>>  store PostgreSQL transactions logs)
>>
>>
> Can you tell the exact model numbers for the Samsung and Crucial SSD's? It
> typically matters! E.g I have some Crucial M550 that have capacitors and
> (originally) claimed to be power off safe, but with testing have been shown
> to be not really power off safe at all. I'd be dubious about Samsungs too.
>
> The Intel Datacenter range (S3700 and similar) are known to have power off
> safety that does work.
>
> regards
>
> Mark
>
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-06 Thread Mark Kirkwood

On 06/07/16 07:17, Mkrtchyan, Tigran wrote:

Hi,

We had a similar situation and the best performance was with 64MB
background_bytes and 512 MB dirty_bytes.

Tigran.

On Jul 5, 2016 16:51, Kaixi Luo  wrote:


 Here are my server specs:

 RAID1 - 2x480GB Samsung SSD with power loss protection (will be used to
 store the PostgreSQL database)
 RAID1 - 2x240GB Crucial SSD with power loss protection. (will be used to
 store PostgreSQL transactions logs)



Can you tell the exact model numbers for the Samsung and Crucial SSD's? 
It typically matters! E.g I have some Crucial M550 that have capacitors 
and (originally) claimed to be power off safe, but with testing have 
been shown to be not really power off safe at all. I'd be dubious about 
Samsungs too.


The Intel Datacenter range (S3700 and similar) are known to have power 
off safety that does work.


regards

Mark


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-06 Thread Wes Vaske (wvaske)
Regarding the Nordeus blog Merlin linked.

They say:
"This doesn't mean the data was really written to disk, it can still remain in 
the disk cache, but enterprise drives usually make sure the data was really 
written to disk on fsync calls."

This isn't actually true for enterprise drives (when I say enterprise in the 
context of an SSD, I'm assuming full power loss protection via capacitors on 
the drive like the Intel DC S3x00 series). Most enterprise SSDs will ignore 
calls to disable disk cache or to flush the disk cache as doing so is entirely 
unnecessary.


Regarding write back cache:
Disabling the write back cache won't have a real large impact on the endurance 
of the drive unless it reduces the total number of bytes written (which it 
won't). I've seen drives that perform better with it disabled and drives that 
perform better with it enabled. I would test in your environment and make the 
decision based on performance. 


Regarding the Crucial drive for logs:
As far as I'm aware, none of the Crucial drives have power loss protection. To 
use these drives you would want to disable disk cache which would drop your 
performance a fair bit.


Write amplification:
I wouldn't expect write amplification to be a serious issue unless you hit 
every LBA on the device early in its life and never execute TRIM. This is one 
of the reasons software RAID can be a better solution for something like this. 
MDADM supports TRIM in RAID devices.  So unless you run the drives above 90% 
full, the write amplification would be minimal so long as you have a daily 
fstrim cron job.

Wes Vaske | Senior Storage Solutions Engineer
Micron Technology


From: pgsql-performance-ow...@postgresql.org 
<pgsql-performance-ow...@postgresql.org> on behalf of Merlin Moncure 
<mmonc...@gmail.com>
Sent: Wednesday, July 6, 2016 1:13 PM
To: Kaixi Luo
Cc: postgres performance list
Subject: Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

On Tue, Jul 5, 2016 at 9:50 AM, Kaixi Luo <kaixi...@gmail.com> wrote:
> Hello,
>
> I've been reading Mr. Greg Smith's "Postgres 9.0 - High Performance" book
> and I have some questions regarding the guidelines I found in the book,
> because I suspect some of them can't be followed blindly to the letter on a
> server with lots of RAM and SSDs.
>
> Here are my server specs:
>
> Intel Xeon E5-1650 v3 Hexa-Core Haswell
> 256GB DDR4 ECC RAM
> Battery backed hardware RAID with 512MB of WriteBack cache (LSI MegaRAID SAS
> 9260-4i)
> RAID1 - 2x480GB Samsung SSD with power loss protection (will be used to
> store the PostgreSQL database)
> RAID1 - 2x240GB Crucial SSD with power loss protection. (will be used to
> store PostgreSQL transactions logs)
>
> First of all, the book suggests that I should enable the WriteBack cache of
> the HWRAID and disable the disk cache to increase performance and ensure
> data safety. Is it still advisable to do this on SSDs, specifically the step
> of disabling the disk cache? Wouldn't that increase the wear rate of the
> SSD?

At the time that book was written, the majority of SSDs were known not
to be completely honest and/or reliable about data integrity in the
face of a power event.  Now it's a hit or miss situation (for example,
see here: http://blog.nordeus.com/dev-ops/power-failure-testing-with-ssds.htm).
The intel drives S3500/S3700 and their descendants are the standard
against which other drives should be judged IMO. The S3500 family in
particular offers tremendous value for database usage.  Do your
research; the warning is still relevant but the blanket statement no
longer applies.  Spinning drives are completely obsolete for database
applications in my experience.

Disabling write back cache for write heavy database loads will will
destroy it in short order due to write amplication and will generally
cause it to underperform hard drives in my experience.

With good SSDs and a good motherboard, I do not recommend a caching
raid controller; software raid is a better choice for many reasons.

One parameter that needs to be analyzed with SSD is
effective_io_concurrency.  see
https://www.postgresql.org/message-id/CAHyXU0yiVvfQAnR9cyH%3DHWh1WbLRsioe%3DmzRJTHwtr%3D2azsTdQ%40mail.gmail.com

merlin


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-06 Thread Scott Marlowe
On Wed, Jul 6, 2016 at 12:13 PM, Merlin Moncure  wrote:
> Disabling write back cache for write heavy database loads will will
> destroy it in short order due to write amplication and will generally
> cause it to underperform hard drives in my experience.

Interesting. We found our best performance with a RAID-5 of 10 800GB
SSDs (Intel 3500/3700 series) that we got MUCH faster performance with
all write caching turned off on our LSI MEgaRAID controllers. We went
from 3 to 4ktps to 15 to 18ktps. And after a year of hard use we still
show ~90% life left (these machines handle thousands of writes per
second in real use) It could be that the caching was getting in the
way of RAID calcs or some other issue. With RAID-1 I have no clue what
the performance will be with write cache on or off.

-- 
To understand recursion, one must first understand recursion.


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-06 Thread Merlin Moncure
On Tue, Jul 5, 2016 at 9:50 AM, Kaixi Luo  wrote:
> Hello,
>
> I've been reading Mr. Greg Smith's "Postgres 9.0 - High Performance" book
> and I have some questions regarding the guidelines I found in the book,
> because I suspect some of them can't be followed blindly to the letter on a
> server with lots of RAM and SSDs.
>
> Here are my server specs:
>
> Intel Xeon E5-1650 v3 Hexa-Core Haswell
> 256GB DDR4 ECC RAM
> Battery backed hardware RAID with 512MB of WriteBack cache (LSI MegaRAID SAS
> 9260-4i)
> RAID1 - 2x480GB Samsung SSD with power loss protection (will be used to
> store the PostgreSQL database)
> RAID1 - 2x240GB Crucial SSD with power loss protection. (will be used to
> store PostgreSQL transactions logs)
>
> First of all, the book suggests that I should enable the WriteBack cache of
> the HWRAID and disable the disk cache to increase performance and ensure
> data safety. Is it still advisable to do this on SSDs, specifically the step
> of disabling the disk cache? Wouldn't that increase the wear rate of the
> SSD?

At the time that book was written, the majority of SSDs were known not
to be completely honest and/or reliable about data integrity in the
face of a power event.  Now it's a hit or miss situation (for example,
see here: http://blog.nordeus.com/dev-ops/power-failure-testing-with-ssds.htm).
The intel drives S3500/S3700 and their descendants are the standard
against which other drives should be judged IMO. The S3500 family in
particular offers tremendous value for database usage.  Do your
research; the warning is still relevant but the blanket statement no
longer applies.  Spinning drives are completely obsolete for database
applications in my experience.

Disabling write back cache for write heavy database loads will will
destroy it in short order due to write amplication and will generally
cause it to underperform hard drives in my experience.

With good SSDs and a good motherboard, I do not recommend a caching
raid controller; software raid is a better choice for many reasons.

One parameter that needs to be analyzed with SSD is
effective_io_concurrency.  see
https://www.postgresql.org/message-id/CAHyXU0yiVvfQAnR9cyH%3DHWh1WbLRsioe%3DmzRJTHwtr%3D2azsTdQ%40mail.gmail.com

merlin


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-05 Thread Mkrtchyan, Tigran
Hi, 
We had a similar situation and the best performance was with 64MB background_bytes and 512 MB dirty_bytes.
Tigran.

On Jul 5, 2016 16:51, Kaixi Luo  wrote:Hello,I've been reading Mr. Greg Smith's "Postgres 9.0 - High Performance" book and I have some questions regarding the guidelines I found in the book, because I suspect some of them can't be followed blindly to the letter on a server with lots of RAM and SSDs.Here are my server specs:Intel Xeon E5-1650 v3 Hexa-Core Haswell 256GB DDR4 ECC RAMBattery backed hardware RAID with 512MB of WriteBack cache (LSI MegaRAID SAS 9260-4i)RAID1 - 2x480GB Samsung SSD with power loss protection (will be used to store the PostgreSQL database)RAID1 - 2x240GB Crucial SSD with power loss protection. (will be used to store PostgreSQL transactions logs)First of all, the book suggests that I should enable the WriteBack cache of the HWRAID and disable the disk cache to increase performance and ensure data safety. Is it still advisable to do this on SSDs, specifically the step of disabling the disk cache? Wouldn't that increase the wear rate of the SSD?Secondly, the book suggests that we increase the device readahead from 256 to 4096. As far as I understand, this was done in order to reduce the number of seeks on a rotating hard drive, so again, is this still applicable to SSDs?The other tunable I've been looking into is vm.dirty_ratio and vm.dirty_background_ratio. I reckon that the book's recommendation to lower vm.dirty_background_ratio to 5 and vm.dirty_ratio to 10 is not enough for a server with such big amount of RAM. How much lower should I set these values, given that my RAID's WriteBack cache size is 512MB?Thank you very much.Kaixi Luo


[PERFORM] Tuning guidelines for server with 256GB of RAM and SSDs?

2016-07-05 Thread Kaixi Luo
Hello,

I've been reading Mr. Greg Smith's "Postgres 9.0 - High Performance" book
and I have some questions regarding the guidelines I found in the book,
because I suspect some of them can't be followed blindly to the letter on a
server with lots of RAM and SSDs.

Here are my server specs:

Intel Xeon E5-1650 v3 Hexa-Core Haswell
256GB DDR4 ECC RAM
Battery backed hardware RAID with 512MB of WriteBack cache (LSI MegaRAID
SAS 9260-4i)
RAID1 - 2x480GB Samsung SSD with power loss protection (will be used to
store the PostgreSQL database)
RAID1 - 2x240GB Crucial SSD with power loss protection. (will be used to
store PostgreSQL transactions logs)

First of all, the book suggests that I should enable the WriteBack cache of
the HWRAID and disable the disk cache to increase performance and ensure
data safety. Is it still advisable to do this on SSDs, specifically the
step of disabling the disk cache? Wouldn't that increase the wear rate of
the SSD?

Secondly, the book suggests that we increase the device readahead from 256
to 4096. As far as I understand, this was done in order to reduce the
number of seeks on a rotating hard drive, so again, is this still
applicable to SSDs?

The other tunable I've been looking into is vm.dirty_ratio and
vm.dirty_background_ratio. I reckon that the book's recommendation to lower
vm.dirty_background_ratio to 5 and vm.dirty_ratio to 10 is not enough for a
server with such big amount of RAM. How much lower should I set these
values, given that my RAID's WriteBack cache size is 512MB?

Thank you very much.

Kaixi Luo


Re: [PERFORM] Tuning the configuration

2014-12-16 Thread Graeme B. Bell
 
 I don't understand the logic behind using drives, 
 which are best for random io, for sequent io workloads.

Because they are also best for sequential IO. I get 1.3-1.4GB/second from 4 
SSDs in RAID or 500MB/s for single disk systems, even with cheap models. 
Are you getting more than that from high-end spinning rust?

Graeme.

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-16 Thread Evgeniy Shishkin

 On 16 Dec 2014, at 14:51, Graeme B. Bell g...@skogoglandskap.no wrote:
 
 
 I don't understand the logic behind using drives, 
 which are best for random io, for sequent io workloads.
 
 Because they are also best for sequential IO. I get 1.3-1.4GB/second from 4 
 SSDs in RAID or 500MB/s for single disk systems, even with cheap models. 
 Are you getting more than that from high-end spinning rust?


I better use ssd for random iops when database doesn't fit in ram.
For wal logs i use raid with bbu cache and couple of sas drives.



-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-15 Thread Graeme B. Bell
 Very much agree with this.  Because SSD is fast doesn't make it suited for 
 certain things, and a streaming sequential 100% write workload is one of 
 them.   I've worked with everything from local disk to high-end SAN and even 
 at the high end we've always put any DB logs on spinning disk.  RAID1 is 
 generally sufficient.  SSD is king for read heavy random I/O workload.


1. Here we found SSD sustained serial writes were faster on SSD than to disk, 
by a factor of 3, both in RAID and single disk configurations. 

2. Also, something to watch out for is extended stalls due to synchronous write 
activity / clearing out of cache, when a lot of data has been building up in 
write caches.  By placing the WAL on the same disk as the ordinary database, 
you avoid having too much dirty cache building up because the WAL forces the 
disk to flush more often. So you can trade off some DB filesystem performance 
here to avoid blocking / IO lag spikes.

3. There's also the question of disk bays. When you have extra disks for OS, 
for logs, etc. , in some situations you're using up disks that could be used to 
extend your main database filesystem, particularly when those disks also need 
to be protected by the appropriate RAID mirrors and RAID hotspares. It can be 
cheaper to put the logs to SSD than to have 1 extra hdd + its RAID1 mirror + 
its hotspare + possible shelfspare, plus pay for a bigger chassis to have 3 
more disk bays.

4. Finally there's the issue of simplicity. If you get a fast SSD and run 
OS/logs/DB off a single RAID volume, there's less chance for error when some 
unlucky person has to do an emergency fix/rebuild later, than if they have to 
check disk caching policy etc across a range of devices and ensure different 
parts of the filesystem are mounted in all the right places. Makes 
documentation easier. 

Graeme Bell



-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-11 Thread Maila Fatticcioni
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 12/10/2014 06:47 PM, Patrick Krecker wrote:
 On Wed, Dec 10, 2014 at 2:44 AM, Maila Fatticcioni 
 mfatticci...@mbigroup.it wrote: Hello. I need to tune a postgres
 installation I've just made to get a better performance. I use two
 identical servers with a hot replication configuration. The two
 servers have the following hardware:
 
 Dual Processor Intel Xeon E5-2640V2 20Mb cache 2.00Ghz, Ram Mem.
 32Gb DDR-3 Ecc Registered, Controller MegaRaid 8-ports 1Gb cache, 4
 Enterprise Hdd NL Sas 600 4Tb Sata, 2 Samsung SSD 840 Pro Series
 512Gb, 2 Hdd 500 Gb
 
 I made a software raid with the last two hard disks with ext4 and
 I installed Ubuntu 14.04.1 LTS (I have to use this SO) on it. I
 made a hardware raid with the four SAS hard disks and I mount the
 partition on it with ext4 without journaling and I put the database
 on it.
 
 Now I have two more steps to do.
 
 1- could you please help tuning the configuration? What are the
 best value I should use for wal_buffers and shared_buffers? 2- I
 would like to use the two SDD to store the wal file. Do you think 
 it is useful or how should I use them?
 
 Thank you for your answers.
 
 Best Regards, Maila Fatticcioni
 
 
 -- Sent via pgsql-performance mailing list
 (pgsql-performance@postgresql.org) To make changes to your
 subscription: 
 http://www.postgresql.org/mailpref/pgsql-performance
 
 We used [1] to great effect when setting our server up. We have
 not had to diverge much from the recommendations in that document.
 
 Generally, the specifics of tuning depend on the workload of your 
 specific instance.
 
 [1] https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
 

Hello.
Indeed I followed this document to set up my configuration. I am glad
that you recommend this as well.

Eventually I use this setup:

max_connections = 150
shared_buffers = 8GB
work_mem = 32MB
checkpoint_segments = 128
checkpoint_completion_target = 0.9

Best Regards,
Maila Fatticcioni
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlSJUaEACgkQi2q3wPb3FcPsuQCeLR5P49d60anErETNiX0iHNLe
Eu4An0QN3nzr/kvlPUTm9Q1A0GkjB/gw
=kdGU
-END PGP SIGNATURE-


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-11 Thread Andrea Suisani

On 12/10/2014 11:44 AM, Maila Fatticcioni wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello.
I need to tune a postgres installation I've just made to get a better
performance. I use two identical servers with a hot replication
configuration. The two servers have the following hardware:

Dual Processor Intel Xeon E5-2640V2 20Mb cache 2.00Ghz,
Ram Mem. 32Gb DDR-3 Ecc Registered,
Controller MegaRaid 8-ports 1Gb cache,
4 Enterprise Hdd NL Sas 600 4Tb Sata,
2 Samsung SSD 840 Pro Series 512Gb,
2 Hdd 500 Gb

I made a software raid with the last two hard disks with ext4 and I
installed Ubuntu 14.04.1 LTS (I have to use this SO) on it. I made a
hardware raid with the four SAS hard disks and I mount the partition
on it with ext4 without journaling and I put the database on it.


Leaving aside all the valid points Patrick already made, as of late I've found
xfs a better choice for Postgres, performance wise.


Now I have two more steps to do.

1- could you please help tuning the configuration? What are the best
value I should use for wal_buffers and shared_buffers?


it's probably outdated but you could try to read Greg Smith's
PostgreSQL 9.0 High Performance, because at least you
could have an idea of almost all the attack-points you
could use to increase you overall performance.

Even in the archive of this very mailinglist you'll surely find
a lot of good advice, e.g. one that I've read here recently is
avoid using any kernels between ver 3.0 and 3.8
(http://www.databasesoup.com/2014/09/why-you-need-to-avoid-linux-kernel-32.html)


2- I would like to use the two SDD to store the wal file. Do you think
it is useful or how should I use them?


I definitely would give it a try.

Andrea



--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-11 Thread Evgeniy Shishkin

 On 11 Dec 2014, at 15:02, Andrea Suisani sick...@opinioni.net wrote:
 
 On 12/10/2014 11:44 AM, Maila Fatticcioni wrote:
 2- I would like to use the two SDD to store the wal file. Do you think
 it is useful or how should I use them?
 
 I definitely would give it a try.
 


I don't understand the logic behind using drives, 
which are best for random io, for sequent io workloads.

Better use 10k sas with BBU raid for wal, money wise.



-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-11 Thread Maila Fatticcioni
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 12/11/2014 01:11 PM, Evgeniy Shishkin wrote:
 
 On 11 Dec 2014, at 15:02, Andrea Suisani sick...@opinioni.net
 wrote:
 
 On 12/10/2014 11:44 AM, Maila Fatticcioni wrote:
 2- I would like to use the two SDD to store the wal file. Do
 you think it is useful or how should I use them?
 
 I definitely would give it a try.
 
 
 
 I don't understand the logic behind using drives, which are best
 for random io, for sequent io workloads.
 
 Better use 10k sas with BBU raid for wal, money wise.
 
 
 

Would you mind to explain me better why you do suggest me to use the
sas raid for wal please?

Thanks,
M.


-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlSJkokACgkQi2q3wPb3FcOOZQCgrhy3sOP3Jds1eGlPqjSW+GhM
xFIAn3YbZgEFAlwTC+SX7GG2My0pElys
=Bsn7
-END PGP SIGNATURE-


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-11 Thread Andrea Suisani



Would you mind to explain me better why you do suggest me to use the
sas raid for wal please?


SSDs are known to shine when they have to deal with random access pattern
rather than sequential, on the other hand 10/15K rpm SAS disk is known to be
better for sequential io workloads (in general rotating disk use to be
better at sequential rather than random access)

Having said that it seems that SSDs are catching up, see:

http://www.anandtech.com/show/6935/seagate-600-ssd-review/5

Andrea


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-11 Thread Andrea Suisani

On 12/11/2014 01:11 PM, Evgeniy Shishkin wrote:



On 11 Dec 2014, at 15:02, Andrea Suisani sick...@opinioni.net wrote:

On 12/10/2014 11:44 AM, Maila Fatticcioni wrote:

2- I would like to use the two SDD to store the wal file. Do you think
it is useful or how should I use them?


I definitely would give it a try.




I don't understand the logic behind using drives,
which are best for random io, for sequent io workloads.

Better use 10k sas with BBU raid for wal, money wise.


Well since Malia had already used the 4 sas hd for the DB,
I thought that it'd be quite quick to setup a raid1 array
(even at software level, e.g. using md), placing pg_xlog
in such array and measure the performance.

As a following step, depending on the time constraints involved,
Malia could rearrange the disk setup enterly and use the SAS
disks as location for pg_xlog.


Andrea




--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-11 Thread Eric Pierce


From: pgsql-performance-ow...@postgresql.org 
pgsql-performance-ow...@postgresql.org on behalf of Evgeniy Shishkin 
itparan...@gmail.com
Sent: Thursday, December 11, 2014 7:11 AM
To: Andrea Suisani
Cc: mfatticci...@mbigroup.it; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning the configuration

 On 11 Dec 2014, at 15:02, Andrea Suisani sick...@opinioni.net wrote:

 On 12/10/2014 11:44 AM, Maila Fatticcioni wrote:
 2- I would like to use the two SDD to store the wal file. Do you think
 it is useful or how should I use them?

 I definitely would give it a try.



 I don't understand the logic behind using drives,
 which are best for random io, for sequent io workloads.

 Better use 10k sas with BBU raid for wal, money wise.

Very much agree with this.  Because SSD is fast doesn't make it suited for 
certain things, and a streaming sequential 100% write workload is one of them.  
 I've worked with everything from local disk to high-end SAN and even at the 
high end we've always put any DB logs on spinning disk.  RAID1 is generally 
sufficient.  SSD is king for read heavy random I/O workload.



--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-11 Thread Mark Kirkwood

On 12/12/14 11:36, Eric Pierce wrote:



From: pgsql-performance-ow...@postgresql.org pgsql-performance-ow...@postgresql.org 
on behalf of Evgeniy Shishkin itparan...@gmail.com
Sent: Thursday, December 11, 2014 7:11 AM
To: Andrea Suisani
Cc: mfatticci...@mbigroup.it; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning the configuration


On 11 Dec 2014, at 15:02, Andrea Suisani sick...@opinioni.net wrote:

On 12/10/2014 11:44 AM, Maila Fatticcioni wrote:

2- I would like to use the two SDD to store the wal file. Do you think
it is useful or how should I use them?


I definitely would give it a try.





I don't understand the logic behind using drives,
which are best for random io, for sequent io workloads.



Better use 10k sas with BBU raid for wal, money wise.


Very much agree with this.  Because SSD is fast doesn't make it suited for 
certain things, and a streaming sequential 100% write workload is one of them.  
 I've worked with everything from local disk to high-end SAN and even at the 
high end we've always put any DB logs on spinning disk.  RAID1 is generally 
sufficient.  SSD is king for read heavy random I/O workload.




Mind you wal is a little different - the limiting factor is (usually) 
not raw sequential speed but fsync latency. These days a modern SSD has 
fsync response pretty much equal to that of a card with BBU + spinners - 
and has more high speed storage available (cards usually have only a 
1G or so of RAM on them).



regards

Mark


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


[PERFORM] Tuning the configuration

2014-12-10 Thread Maila Fatticcioni
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello.
I need to tune a postgres installation I've just made to get a better
performance. I use two identical servers with a hot replication
configuration. The two servers have the following hardware:

Dual Processor Intel Xeon E5-2640V2 20Mb cache 2.00Ghz,
Ram Mem. 32Gb DDR-3 Ecc Registered,
Controller MegaRaid 8-ports 1Gb cache,
4 Enterprise Hdd NL Sas 600 4Tb Sata,
2 Samsung SSD 840 Pro Series 512Gb,
2 Hdd 500 Gb

I made a software raid with the last two hard disks with ext4 and I
installed Ubuntu 14.04.1 LTS (I have to use this SO) on it. I made a
hardware raid with the four SAS hard disks and I mount the partition
on it with ext4 without journaling and I put the database on it.

Now I have two more steps to do.

1- could you please help tuning the configuration? What are the best
value I should use for wal_buffers and shared_buffers?
2- I would like to use the two SDD to store the wal file. Do you think
it is useful or how should I use them?

Thank you for your answers.

Best Regards,
Maila Fatticcioni
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlSII/gACgkQi2q3wPb3FcPUuACgg2m2o9dQWavLrN2EmmmCpGEv
YnMAoN0R/gejcKwnxf0qFPKXtaGaIG1A
=oLxU
-END PGP SIGNATURE-


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning the configuration

2014-12-10 Thread Patrick Krecker
On Wed, Dec 10, 2014 at 2:44 AM, Maila Fatticcioni
mfatticci...@mbigroup.it wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hello.
 I need to tune a postgres installation I've just made to get a better
 performance. I use two identical servers with a hot replication
 configuration. The two servers have the following hardware:

 Dual Processor Intel Xeon E5-2640V2 20Mb cache 2.00Ghz,
 Ram Mem. 32Gb DDR-3 Ecc Registered,
 Controller MegaRaid 8-ports 1Gb cache,
 4 Enterprise Hdd NL Sas 600 4Tb Sata,
 2 Samsung SSD 840 Pro Series 512Gb,
 2 Hdd 500 Gb

 I made a software raid with the last two hard disks with ext4 and I
 installed Ubuntu 14.04.1 LTS (I have to use this SO) on it. I made a
 hardware raid with the four SAS hard disks and I mount the partition
 on it with ext4 without journaling and I put the database on it.

 Now I have two more steps to do.

 1- could you please help tuning the configuration? What are the best
 value I should use for wal_buffers and shared_buffers?
 2- I would like to use the two SDD to store the wal file. Do you think
 it is useful or how should I use them?

 Thank you for your answers.

 Best Regards,
 Maila Fatticcioni
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1

 iEYEARECAAYFAlSII/gACgkQi2q3wPb3FcPUuACgg2m2o9dQWavLrN2EmmmCpGEv
 YnMAoN0R/gejcKwnxf0qFPKXtaGaIG1A
 =oLxU
 -END PGP SIGNATURE-


 --
 Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-performance

We used [1] to great effect when setting our server up. We have not
had to diverge much from the recommendations in that document.

Generally, the specifics of tuning depend on the workload of your
specific instance.

[1] https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] tuning postgresql 9.3.5 and multiple cores

2014-08-26 Thread Soni M
Changing to a higher rate CPU would be more helpful if you run less than 32
queries at a time.


On Tue, Aug 26, 2014 at 8:51 AM, Jeff Janes jeff.ja...@gmail.com wrote:

 On Monday, August 25, 2014, Jeison Bedoya Delgado 
 jeis...@audifarma.com.co wrote:

 hi, recently i change the hardware of my database 32 cores up to 64 cores
 and 128GB Ram, but the performance is the same.  Perhaps i have to change
 any parameter in the postgresql.conf?.



 PostgreSQL does not (yet) automatically parallelize queries.

 Unless you have more than 32 queries trying to run at the same time,
 increasing the number of cores from 32 to 64 is unlikely to be useful.

 Cheers,

 Jeff




-- 
Regards,

Soni Maula Harriz


Re: [PERFORM] tuning postgresql 9.3.5 and multiple cores

2014-08-26 Thread Mark Kirkwood

On 26/08/14 06:47, Jeison Bedoya Delgado wrote:

hi, recently i change the hardware of my database 32 cores up to 64
cores and 128GB Ram, but the performance is the same.  Perhaps i have to
change any parameter in the postgresql.conf?.



In addition to the points that others have made, even if you do have  
32 active sessions it it not clear that 64 cores will automagically get 
you twice (or in fact any) better performance than 32. We are seeing 
exactly this effect with a (60 core) machine that gets pretty much the 
same performance as an older generation 32 core one.


Interestingly while this is *likely* a software issue - it is not 
immediately obvious where it lies - we tested Postgres (9.3/9.4/9.5) and 
Mysql (5.5/5.6/5.7) *all* of which exhibited the the lack of improvement 
with more cores.


Profiling suggested numa effects - but trying to eliminate these seemed 
to simply throw up new factors to inhibit performance. My *guess* (and 
it is a guess) is that we are seeing 2 (perhaps more) performance 
bottlenecks very close to each other: numa and spinlock contention at least.


Regards

Mark




--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


[PERFORM] tuning postgresql 9.3.5 and multiple cores

2014-08-25 Thread Jeison Bedoya Delgado
hi, recently i change the hardware of my database 32 cores up to 64 
cores and 128GB Ram, but the performance is the same.  Perhaps i have to 
change any parameter in the postgresql.conf?.


Thanks by your help

--
Atentamente,


JEISON BEDOYA DELGADO
.


--
NOTA VERDE:
No imprima este correo a menos que sea absolutamente necesario.
Ahorre papel, ayude a salvar un arbol.


Este mensaje ha sido analizado por MailScanner
en busca de virus y otros contenidos peligrosos,
y se considera que esta limpio.


Este texto fue anadido por el servidor de correo de Audifarma S.A.:

Las opiniones contenidas en este mensaje no necesariamente coinciden
con las institucionales de Audifarma. La informacion y todos sus
archivos Anexos, son confidenciales, privilegiados y solo pueden ser
utilizados por sus destinatarios. Si por error usted recibe este
mensaje, le ofrecemos disculpas, solicitamos eliminarlo de inmediato,
notificarle de su error a la persona que lo envio y abstenerse de
utilizar su contenido.




--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] tuning postgresql 9.3.5 and multiple cores

2014-08-25 Thread Jeff Janes
On Monday, August 25, 2014, Jeison Bedoya Delgado jeis...@audifarma.com.co
wrote:

 hi, recently i change the hardware of my database 32 cores up to 64 cores
 and 128GB Ram, but the performance is the same.  Perhaps i have to change
 any parameter in the postgresql.conf?.



PostgreSQL does not (yet) automatically parallelize queries.

Unless you have more than 32 queries trying to run at the same time,
increasing the number of cores from 32 to 64 is unlikely to be useful.

Cheers,

Jeff


Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-15 Thread Nick Eubank
On Mon, Apr 14, 2014 at 5:19 PM, Jeff Janes jeff.ja...@gmail.com wrote:

 On Mon, Apr 14, 2014 at 2:46 PM, Nick Eubank nickeub...@gmail.com wrote:

 Any rules of thumb for work_mem, maintenance_work_mem, shared_buffer,
 etc. for a database that DOESN'T anticipate concurrent connections and that
 is doing lots of aggregate functions on large tables? All the advice I
 can find online on tuning 
 (thishttp://wiki.postgresql.org/wiki/Performance_Optimization
 , 
 thishttp://media.revsys.com/talks/djangocon/2011/secrets-of-postgresql-performance.pdf
 , this http://www.revsys.com/writings/postgresql-performance.html etc.)
 is written for people anticipating lots of concurrent connections.

 I'm a social scientist looking to use Postgres not as a database to be
 shared by multiple users, but rather as my own tool for manipulating a
 massive data set (I have 5 billion transaction records (600gb in csv) and
 want to pull out unique user pairs, estimate aggregates for individual
 users, etc.). This also means almost no writing, except to creation of new
 tables based on selections from the main table.

 I'm on a Windows 8 VM with 16gb ram, SCSI VMware HD, and 3 cores if
 that's important.


 I'd go with a small shared_buffers, like 128MB, and let the OS cache as
 much as possible.  This minimizes the amount of double buffering.

 And set work_mem to about 6GB, then bump it up if that doesn't seem to
 cause problems.

 In the scenario you describe, it is probably no big deal if you guess too
 high.  Monitor the process, if it it starts to go nuts just kill it and
 start again with a lower work_mem.  If it is a single user system, you can
 afford to be adventurous.

 If you need to build indexes, you should bump up maintenance_work_mem, but
 I just would do that in the local session not system wide.

 Cheers,

 Jeff



Quick followup Jeff: it seems that I can't set work_mem above about 1gb
(can't get to 2gb. When I update config, the values just don't change in
SHOW ALL -- integer constraint?). Is there a work around, or should I
tweak something else accordingly?

Thanks!

Nick

(Properly bottom posted this time?)


Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-15 Thread Jeff Janes
On Tue, Apr 15, 2014 at 9:12 AM, Nick Eubank nickeub...@gmail.com wrote:

 On Mon, Apr 14, 2014 at 5:19 PM, Jeff Janes jeff.ja...@gmail.com wrote:


 I'd go with a small shared_buffers, like 128MB, and let the OS cache as
 much as possible.  This minimizes the amount of double buffering.

 And set work_mem to about 6GB, then bump it up if that doesn't seem to
 cause problems.

 In the scenario you describe, it is probably no big deal if you guess too
 high.  Monitor the process, if it it starts to go nuts just kill it and
 start again with a lower work_mem.  If it is a single user system, you can
 afford to be adventurous.

 If you need to build indexes, you should bump up maintenance_work_mem,
 but I just would do that in the local session not system wide.

 Cheers,

 Jeff



 Quick followup Jeff: it seems that I can't set work_mem above about 1gb
 (can't get to 2gb. When I update config, the values just don't change in
 SHOW ALL -- integer constraint?). Is there a work around, or should I
 tweak something else accordingly?



What version are you using?  What is the exact line you put in your config
file?  Did you get any errors when using that config file?  Are you sure
you actually reloaded the server, so that it reread the config file, rather
than just changing the file and then not applying the change?

I usually set work_mem within a psql connection, in which case you need to
quote the setting if you use units:

set work_mem=3GB;

But if you set it system wide in the config file the quotes should not be
needed.



 Thanks!

 Nick

 (Properly bottom posted this time?)


Looked good to me.

Cheers,

Jeff


Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-15 Thread Schnabel, Robert D.


From: pgsql-performance-ow...@postgresql.org 
[mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Nick Eubank
Sent: Tuesday, April 15, 2014 11:12 AM
To: Jeff Janes
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning Postgres for Single connection use



On Mon, Apr 14, 2014 at 5:19 PM, Jeff Janes jeff.ja...@gmail.com wrote:
On Mon, Apr 14, 2014 at 2:46 PM, Nick Eubank nickeub...@gmail.com wrote:
Any rules of thumb for work_mem, maintenance_work_mem, shared_buffer, etc. for 
a database that DOESN'T anticipate concurrent connections and that is doing 
lots of aggregate functions on large tables? All the advice I can find online 
on tuning (this, this, this etc.) is written for people anticipating lots of 
concurrent connections.
I'm a social scientist looking to use Postgres not as a database to be shared 
by multiple users, but rather as my own tool for manipulating a massive data 
set (I have 5 billion transaction records (600gb in csv) and want to pull out 
unique user pairs, estimate aggregates for individual users, etc.). This also 
means almost no writing, except to creation of new tables based on selections 
from the main table. 
I'm on a Windows 8 VM with 16gb ram, SCSI VMware HD, and 3 cores if that's 
important.

I'd go with a small shared_buffers, like 128MB, and let the OS cache as much as 
possible.  This minimizes the amount of double buffering.

And set work_mem to about 6GB, then bump it up if that doesn't seem to cause 
problems.

In the scenario you describe, it is probably no big deal if you guess too high. 
 Monitor the process, if it it starts to go nuts just kill it and start again 
with a lower work_mem.  If it is a single user system, you can afford to be 
adventurous.

If you need to build indexes, you should bump up maintenance_work_mem, but I 
just would do that in the local session not system wide.

Cheers,

Jeff
 


Quick followup Jeff: it seems that I can't set work_mem above about 1gb (can't 
get to 2gb. When I update config, the values just don't change in SHOW ALL -- 
integer constraint?). Is there a work around, or should I tweak something else 
accordingly? 

Thanks!

Nick

(Properly bottom posted this time?) 

[Schnabel, Robert D.] 

Nick,

I asked the same question a while ago about work_mem on Windows.  See this 
thread:
http://www.postgresql.org/message-id/17895.1315869...@sss.pgh.pa.us

Bob


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-15 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes:
 On Tue, Apr 15, 2014 at 9:12 AM, Nick Eubank nickeub...@gmail.com wrote:
 Quick followup Jeff: it seems that I can't set work_mem above about 1gb
 (can't get to 2gb. When I update config, the values just don't change in
 SHOW ALL -- integer constraint?). Is there a work around, or should I
 tweak something else accordingly?

 What version are you using?  What is the exact line you put in your config
 file?  Did you get any errors when using that config file?  Are you sure
 you actually reloaded the server, so that it reread the config file, rather
 than just changing the file and then not applying the change?

 I usually set work_mem within a psql connection, in which case you need to
 quote the setting if you use units:
 set work_mem=3GB;

FWIW, it's generally considered a seriously *bad* idea to set work_mem as
high as 1GB in postgresql.conf: you're promising that each query running
on the server can use 1GB per sort or hash step.  You probably don't have
the machine resources to honor that promise.  (If you do, I'd like to have
your IT budget ;-))  Recommended practice is to keep the global setting
conservatively small, and bump it up locally in your session (with SET)
for individual queries that need the very large value.

But having said that, Postgres doesn't try to enforce any such practice.
My money is on what Jeff is evidently thinking: you forgot to do pg_ctl
reload, or else the setting is too large for your platform, in which case
there should have been a complaint in the postmaster log.  As noted
elsewhere, the limit for Windows is a hair under 2GB even if it's 64-bit
Windows.

regards, tom lane


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance



[PERFORM] Tuning Postgres for Single connection use

2014-04-15 Thread Nick Eubank


 On Tuesday, April 15, 2014, Tom Lane t...@sss.pgh.pa.us wrote:

 Jeff Janes jeff.ja...@gmail.com writes:
  On Tue, Apr 15, 2014 at 9:12 AM, Nick Eubank nickeub...@gmail.com
 wrote:
  Quick followup Jeff: it seems that I can't set work_mem above about 1gb
  (can't get to 2gb. When I update config, the values just don't change in
  SHOW ALL -- integer constraint?). Is there a work around, or should I
  tweak something else accordingly?

  What version are you using?  What is the exact line you put in your
 config
  file?  Did you get any errors when using that config file?  Are you sure
  you actually reloaded the server, so that it reread the config file,
 rather
  than just changing the file and then not applying the change?

  I usually set work_mem within a psql connection, in which case you need
 to
  quote the setting if you use units:
  set work_mem=3GB;

 FWIW, it's generally considered a seriously *bad* idea to set work_mem as
 high as 1GB in postgresql.conf: you're promising that each query running
 on the server can use 1GB per sort or hash step.  You probably don't have
 the machine resources to honor that promise.  (If you do, I'd like to have
 your IT budget ;-))  Recommended practice is to keep the global setting
 conservatively small, and bump it up locally in your session (with SET)
 for individual queries that need the very large value.

 But having said that, Postgres doesn't try to enforce any such practice.
 My money is on what Jeff is evidently thinking: you forgot to do pg_ctl
 reload, or else the setting is too large for your platform, in which case
 there should have been a complaint in the postmaster log.  As noted
 elsewhere, the limit for Windows is a hair under 2GB even if it's 64-bit
 Windows.

 regards, tom lane


Thanks Tom --  quick follow up: I know that 1gb work_mem is a terrible idea
for normal postgres users with lots of concurrent users, but for my
situations where there will only ever be one connection running one query,
why is that a problem on a machine with 16gb of ram.

Re:Robert -- thanks for that clarification!


[PERFORM] Tuning Postgres for Single connection use

2014-04-14 Thread Nick Eubank
Any rules of thumb for work_mem, maintenance_work_mem, shared_buffer, etc.
for a database that DOESN'T anticipate concurrent connections and that is
doing lots of aggregate functions on large tables? All the advice I can
find online on tuning
(thishttp://wiki.postgresql.org/wiki/Performance_Optimization
, 
thishttp://media.revsys.com/talks/djangocon/2011/secrets-of-postgresql-performance.pdf
, this http://www.revsys.com/writings/postgresql-performance.html etc.)
is written for people anticipating lots of concurrent connections.

I'm a social scientist looking to use Postgres not as a database to be
shared by multiple users, but rather as my own tool for manipulating a
massive data set (I have 5 billion transaction records (600gb in csv) and
want to pull out unique user pairs, estimate aggregates for individual
users, etc.). This also means almost no writing, except to creation of new
tables based on selections from the main table.

I'm on a Windows 8 VM with 16gb ram, SCSI VMware HD, and 3 cores if that's
important.

Thanks!


Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-14 Thread Gavin Flower

On 15/04/14 09:46, Nick Eubank wrote:


Any rules of thumb for |work_mem|, |maintenance_work_mem|, 
|shared_buffer|, etc. for a database that DOESN'T anticipate 
concurrent connections and that is doing lots of aggregate functions 
on large tables? All the advice I can find online on tuning (this 
http://wiki.postgresql.org/wiki/Performance_Optimization, this 
http://media.revsys.com/talks/djangocon/2011/secrets-of-postgresql-performance.pdf, 
this 
http://www.revsys.com/writings/postgresql-performance.html etc.) is 
written for people anticipating lots of concurrent connections.


I'm a social scientist looking to use Postgres not as a database to be 
shared by multiple users, but rather as my own tool for manipulating a 
massive data set (I have 5 billion transaction records (600gb in csv) 
and want to pull out unique user pairs, estimate aggregates for 
individual users, etc.). This also means almost no writing, except to 
creation of new tables based on selections from the main table.


I'm on a Windows 8 VM with 16gb ram, SCSI VMware HD, and 3 cores if 
that's important.


Thanks!

Well for serious database work, I suggest upgrading to Linux - you will 
get better performance out of the same hardware and probably (a year or 
so ago, I noticed some tuning options did not apply to Microsoft O/S's, 
but I don't recall the details - these options may, or may not, apply to 
your situation) more scope for tuning.  Apart from anything else, your 
processing will not be slowed down by having to run anti-virus software!


Note that in Linux you have a wide choice of distributions and desktop 
environments: I chose Mate (http://mate-desktop.org), some people prefer 
xfce (http://www.xfce.org), I used to use GNOME 2.



Cheers,
Gavin


Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-14 Thread Nick Eubank
Thanks Gavin -- would LOVE to. Sadly I'm in a weird situation
where my hardware is not under my control, so I'm stuck making the best of
what I have. Next time though! :)

On Monday, April 14, 2014, Gavin Flower gavinflo...@archidevsys.co.nz
wrote:

  On 15/04/14 09:46, Nick Eubank wrote:

  Any rules of thumb for work_mem, maintenance_work_mem, shared_buffer,
 etc. for a database that DOESN'T anticipate concurrent connections and that
 is doing lots of aggregate functions on large tables? All the advice I
 can find online on tuning 
 (thishttp://wiki.postgresql.org/wiki/Performance_Optimization
 , 
 thishttp://media.revsys.com/talks/djangocon/2011/secrets-of-postgresql-performance.pdf
 , this http://www.revsys.com/writings/postgresql-performance.html etc.)
 is written for people anticipating lots of concurrent connections.

 I'm a social scientist looking to use Postgres not as a database to be
 shared by multiple users, but rather as my own tool for manipulating a
 massive data set (I have 5 billion transaction records (600gb in csv) and
 want to pull out unique user pairs, estimate aggregates for individual
 users, etc.). This also means almost no writing, except to creation of new
 tables based on selections from the main table.

 I'm on a Windows 8 VM with 16gb ram, SCSI VMware HD, and 3 cores if that's
 important.

 Thanks!

 Well for serious database work, I suggest upgrading to Linux - you will
 get better performance out of the same hardware and probably (a year or so
 ago, I noticed some tuning options did not apply to Microsoft O/S's, but I
 don't recall the details - these options may, or may not, apply to your
 situation) more scope for tuning.  Apart from anything else, your
 processing will not be slowed down by having to run anti-virus software!

 Note that in Linux you have a wide choice of distributions and desktop
 environments: I chose Mate (http://mate-desktop.org), some people prefer
 xfce (http://www.xfce.org), I used to use GNOME 2.


 Cheers,
 Gavin



Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-14 Thread Andrew Dunstan


On 04/14/2014 05:46 PM, Nick Eubank wrote:


Any rules of thumb for |work_mem|, |maintenance_work_mem|, 
|shared_buffer|, etc. for a database that DOESN'T anticipate 
concurrent connections and that is doing lots of aggregate functions 
on large tables? All the advice I can find online on tuning (this 
http://wiki.postgresql.org/wiki/Performance_Optimization, this 
http://media.revsys.com/talks/djangocon/2011/secrets-of-postgresql-performance.pdf, 
this 
http://www.revsys.com/writings/postgresql-performance.html etc.) is 
written for people anticipating lots of concurrent connections.


I'm a social scientist looking to use Postgres not as a database to be 
shared by multiple users, but rather as my own tool for manipulating a 
massive data set (I have 5 billion transaction records (600gb in csv) 
and want to pull out unique user pairs, estimate aggregates for 
individual users, etc.). This also means almost no writing, except to 
creation of new tables based on selections from the main table.


I'm on a Windows 8 VM with 16gb ram, SCSI VMware HD, and 3 cores if 
that's important.






First up would probably be don't run on Windows. shared_buffers above 
512Mb causes performance to degrade on Windows, while that threshold is 
much higher on *nix.


cheers

andrew



--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-14 Thread Jeff Janes
On Mon, Apr 14, 2014 at 2:46 PM, Nick Eubank nickeub...@gmail.com wrote:

 Any rules of thumb for work_mem, maintenance_work_mem, shared_buffer,
 etc. for a database that DOESN'T anticipate concurrent connections and that
 is doing lots of aggregate functions on large tables? All the advice I
 can find online on tuning 
 (thishttp://wiki.postgresql.org/wiki/Performance_Optimization
 , 
 thishttp://media.revsys.com/talks/djangocon/2011/secrets-of-postgresql-performance.pdf
 , this http://www.revsys.com/writings/postgresql-performance.html etc.)
 is written for people anticipating lots of concurrent connections.

 I'm a social scientist looking to use Postgres not as a database to be
 shared by multiple users, but rather as my own tool for manipulating a
 massive data set (I have 5 billion transaction records (600gb in csv) and
 want to pull out unique user pairs, estimate aggregates for individual
 users, etc.). This also means almost no writing, except to creation of new
 tables based on selections from the main table.

 I'm on a Windows 8 VM with 16gb ram, SCSI VMware HD, and 3 cores if that's
 important.


I'd go with a small shared_buffers, like 128MB, and let the OS cache as
much as possible.  This minimizes the amount of double buffering.

And set work_mem to about 6GB, then bump it up if that doesn't seem to
cause problems.

In the scenario you describe, it is probably no big deal if you guess too
high.  Monitor the process, if it it starts to go nuts just kill it and
start again with a lower work_mem.  If it is a single user system, you can
afford to be adventurous.

If you need to build indexes, you should bump up maintenance_work_mem, but
I just would do that in the local session not system wide.

Cheers,

Jeff


Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-14 Thread Gavin Flower

In this list, please bottom post!

I've added potentially useful advice below.

On 15/04/14 11:39, Nick Eubank wrote:
Thanks Gavin -- would LOVE to. Sadly I'm in a weird situation 
where my hardware is not under my control, so I'm stuck making the 
best of what I have. Next time though! :)


On Monday, April 14, 2014, Gavin Flower gavinflo...@archidevsys.co.nz 
mailto:gavinflo...@archidevsys.co.nz wrote:


On 15/04/14 09:46, Nick Eubank wrote:


Any rules of thumb for |work_mem|, |maintenance_work_mem|,
|shared_buffer|, etc. for a database that DOESN'T anticipate
concurrent connections and that is doing lots of aggregate
functions on large tables? All the advice I can find online on
tuning (this
http://wiki.postgresql.org/wiki/Performance_Optimization, this

http://media.revsys.com/talks/djangocon/2011/secrets-of-postgresql-performance.pdf,
this
http://www.revsys.com/writings/postgresql-performance.html etc.) is
written for people anticipating lots of concurrent connections.

I'm a social scientist looking to use Postgres not as a database
to be shared by multiple users, but rather as my own tool for
manipulating a massive data set (I have 5 billion transaction
records (600gb in csv) and want to pull out unique user pairs,
estimate aggregates for individual users, etc.). This also means
almost no writing, except to creation of new tables based on
selections from the main table.

I'm on a Windows 8 VM with 16gb ram, SCSI VMware HD, and 3 cores
if that's important.

Thanks!


Well for serious database work, I suggest upgrading to Linux - you
will get better performance out of the same hardware and probably
(a year or so ago, I noticed some tuning options did not apply to
Microsoft O/S's, but I don't recall the details - these options
may, or may not, apply to your situation) more scope for tuning. 
Apart from anything else, your processing will not be slowed down

by having to run anti-virus software!

Note that in Linux you have a wide choice of distributions and
desktop environments: I chose Mate (http://mate-desktop.org), some
people prefer xfce (http://www.xfce.org), I used to use GNOME 2.


Cheers,
Gavin


Yeah, I know the feeling!

I have a client that uses MySQL (ugh!), but I won't even bother 
mentioning PostgreSQL!


Hopefully, someone more knowledgeable will give you some good advice 
specific to your O/S.


For tables that don't change, consider a packing density of 100%.

Take care in how you design your tables, and the column types.

Consider carefully the queries you are likely to use, so you can design 
appropriate indexes.


Some advice will depend on the schema you plan to use, and the type of 
queries.



Cheers,
Gavin




Re: [PERFORM] Tuning Postgres for Single connection use

2014-04-14 Thread Nick Eubank
Terrific -- thanks Gavin and Jeff! That's incredibly helpful for a n00b
like me!


On Mon, Apr 14, 2014 at 5:29 PM, Gavin Flower gavinflo...@archidevsys.co.nz
 wrote:

  In this list, please bottom post!

 I've added potentially useful advice below.


 On 15/04/14 11:39, Nick Eubank wrote:

 Thanks Gavin -- would LOVE to. Sadly I'm in a weird situation
 where my hardware is not under my control, so I'm stuck making the best of
 what I have. Next time though! :)

 On Monday, April 14, 2014, Gavin Flower gavinflo...@archidevsys.co.nz
 wrote:

  On 15/04/14 09:46, Nick Eubank wrote:

  Any rules of thumb for work_mem, maintenance_work_mem, shared_buffer,
 etc. for a database that DOESN'T anticipate concurrent connections and that
 is doing lots of aggregate functions on large tables? All the advice I
 can find online on tuning 
 (thishttp://wiki.postgresql.org/wiki/Performance_Optimization
 , 
 thishttp://media.revsys.com/talks/djangocon/2011/secrets-of-postgresql-performance.pdf
 , this http://www.revsys.com/writings/postgresql-performance.html etc.)
 is written for people anticipating lots of concurrent connections.

 I'm a social scientist looking to use Postgres not as a database to be
 shared by multiple users, but rather as my own tool for manipulating a
 massive data set (I have 5 billion transaction records (600gb in csv) and
 want to pull out unique user pairs, estimate aggregates for individual
 users, etc.). This also means almost no writing, except to creation of new
 tables based on selections from the main table.

 I'm on a Windows 8 VM with 16gb ram, SCSI VMware HD, and 3 cores if
 that's important.

 Thanks!

 Well for serious database work, I suggest upgrading to Linux - you will
 get better performance out of the same hardware and probably (a year or so
 ago, I noticed some tuning options did not apply to Microsoft O/S's, but I
 don't recall the details - these options may, or may not, apply to your
 situation) more scope for tuning.  Apart from anything else, your
 processing will not be slowed down by having to run anti-virus software!

 Note that in Linux you have a wide choice of distributions and desktop
 environments: I chose Mate (http://mate-desktop.org), some people prefer
 xfce (http://www.xfce.org), I used to use GNOME 2.


 Cheers,
 Gavin

  Yeah, I know the feeling!

 I have a client that uses MySQL (ugh!), but I won't even bother mentioning
 PostgreSQL!

 Hopefully, someone more knowledgeable will give you some good advice
 specific to your O/S.

 For tables that don't change, consider a packing density of 100%.

 Take care in how you design your tables, and the column types.

 Consider carefully the queries you are likely to use, so you can design
 appropriate indexes.

 Some advice will depend on the schema you plan to use, and the type of
 queries.


 Cheers,
 Gavin





Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Walker, James Les
Exactly, if turning off fsync gives me 100 commits/sec then I know where my 
bottleneck is and I can attack it. Keep in mind though that I already turned 
off synchronous commit -- *really* dangerous -- and it didn't have any effect.

-- Les

-Original Message-
From: pgsql-performance-ow...@postgresql.org 
[mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Merlin Moncure
Sent: Monday, April 30, 2012 6:04 PM
To: Thomas Kellerer
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows

On Mon, Apr 30, 2012 at 5:00 PM, Thomas Kellerer spam_ea...@gmx.net wrote:
 Merlin Moncure wrote on 30.04.2012 23:43:

 Trying turning off fsync in postgrsql.conf to be sure.


 This is a dangerous advise.
 Turning off fsync can potentially corrupt the database in case of a 
 system failure (e.g. power outage).


sure. that said, we're just trying to figure out why he's getting
around 40tps.   since he's only benchmarking test data it's perfectly
ok to do that.

merlin

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are 
confidential. If you are not the named recipient please notify the sender and 
immediately delete it. You may not disseminate, distribute, or forward this 
e-mail message or disclose its contents to anybody else. Copyright and any 
other intellectual property rights in its contents are the sole property of 
Cantor Fitzgerald.
 E-mail transmission cannot be guaranteed to be secure or error-free. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.
 Although we routinely screen for viruses, addressees should check this 
e-mail and any attachments for viruses. We make no representation or warranty 
as to the absence of viruses in this e-mail or any attachments. Please note 
that to ensure regulatory compliance and for the protection of our customers 
and business, we may monitor and read e-mails sent to and from our server(s). 

For further important information, please see  
http://www.cantor.com/legal/statement


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Merlin Moncure
On Tue, May 1, 2012 at 7:51 AM, Walker, James Les jawal...@cantor.com wrote:
 Exactly, if turning off fsync gives me 100 commits/sec then I know where my 
 bottleneck is and I can attack it. Keep in mind though that I already turned 
 off synchronous commit -- *really* dangerous -- and it didn't have any effect.

well synchronous commit is not as dangerous:
fsync off + power failure = corrupt database
synchronous commit off + power failure = some lost transactions

still waiting on the ssd model #.  worst case scenario is that you tps
rate is in fact sync bound and you have a ssd without capacitor backed
buffers (for example, the intel 320 has them); the probable workaround
would be to set the drive cache from write through to write back but
it would unsafe in that case.  in other words, tps rates in the triple
digits would be physically impossible.

another less likely scenario is you are having network issues
(assuming you are connecting to the database through tcp/ip).  20
years in, microsoft is still figuring out how to properly configure a
network socket.

merlin

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Walker, James Les
SSD is OCZ-VERTEX3 MI. Controller is LSI SAS2 2008 Falcon. I'm working on 
installing EDB. Then I can give you some I/O numbers.

-- Les

-Original Message-
From: Merlin Moncure [mailto:mmonc...@gmail.com] 
Sent: Tuesday, May 01, 2012 9:07 AM
To: Walker, James Les
Cc: Thomas Kellerer; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows

On Tue, May 1, 2012 at 7:51 AM, Walker, James Les jawal...@cantor.com wrote:
 Exactly, if turning off fsync gives me 100 commits/sec then I know where my 
 bottleneck is and I can attack it. Keep in mind though that I already turned 
 off synchronous commit -- *really* dangerous -- and it didn't have any effect.

well synchronous commit is not as dangerous:
fsync off + power failure = corrupt database synchronous commit off + power 
failure = some lost transactions

still waiting on the ssd model #.  worst case scenario is that you tps rate is 
in fact sync bound and you have a ssd without capacitor backed buffers (for 
example, the intel 320 has them); the probable workaround would be to set the 
drive cache from write through to write back but it would unsafe in that case.  
in other words, tps rates in the triple digits would be physically impossible.

another less likely scenario is you are having network issues (assuming you are 
connecting to the database through tcp/ip).  20 years in, microsoft is still 
figuring out how to properly configure a network socket.

merlin
CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are 
confidential. If you are not the named recipient please notify the sender and 
immediately delete it. You may not disseminate, distribute, or forward this 
e-mail message or disclose its contents to anybody else. Copyright and any 
other intellectual property rights in its contents are the sole property of 
Cantor Fitzgerald.
 E-mail transmission cannot be guaranteed to be secure or error-free. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.
 Although we routinely screen for viruses, addressees should check this 
e-mail and any attachments for viruses. We make no representation or warranty 
as to the absence of viruses in this e-mail or any attachments. Please note 
that to ensure regulatory compliance and for the protection of our customers 
and business, we may monitor and read e-mails sent to and from our server(s). 

For further important information, please see  
http://www.cantor.com/legal/statement


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Andy Colson

On 5/1/2012 8:06 AM, Merlin Moncure wrote:

On Tue, May 1, 2012 at 7:51 AM, Walker, James Lesjawal...@cantor.com  wrote:

Exactly, if turning off fsync gives me 100 commits/sec then I know where my 
bottleneck is and I can attack it. Keep in mind though that I already turned 
off synchronous commit -- *really* dangerous -- and it didn't have any effect.


well synchronous commit is not as dangerous:
fsync off + power failure = corrupt database
synchronous commit off + power failure = some lost transactions

still waiting on the ssd model #.  worst case scenario is that you tps
rate is in fact sync bound and you have a ssd without capacitor backed
buffers (for example, the intel 320 has them); the probable workaround
would be to set the drive cache from write through to write back but
it would unsafe in that case.  in other words, tps rates in the triple
digits would be physically impossible.

another less likely scenario is you are having network issues
(assuming you are connecting to the database through tcp/ip).  20
years in, microsoft is still figuring out how to properly configure a
network socket.

merlin



Even if its all local, windows doesnt have domain sockets (correct?), so 
all that traffic still has to go thru some bit of network stack, yes?


-Andy

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Merlin Moncure
On Tue, May 1, 2012 at 8:14 AM, Walker, James Les jawal...@cantor.com wrote:
 SSD is OCZ-VERTEX3 MI. Controller is LSI SAS2 2008 Falcon. I'm working on 
 installing EDB. Then I can give you some I/O numbers.

It looks like the ssd doesn't have a nv cache and the raid card is a
simple sas hba (which likely isn't doing much for the ssd besides
masking TRIM).  The OCZ 'pro' versions are the ones with power loss
protection (see:
http://hothardware.com/Reviews/OCZ-Vertex-3-Pro-SandForce-SF2000-Based-SSD-Preview/).
 Note the bullet: Implements SandForce 2582 Controller with power
loss data protection.  It doesn't look like the Vertex 3 Pro is out
yet.

If my hunch is correct, the issue here is that the drive is being
asked to sync data physically and SSD really don't perform well when
the controller isn't in control of when and how to sync data.  However
full physical sync is the only way to guarantee data is truly safe in
the context of a unexpected power loss (an nv cache is basically a
compromise on this point).

merlin

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Walker, James Les
I installed the enterprisedb distribution and immediately saw a 400% 
performance increase. Turning off fsck made it an order of magnitude better. 
I'm now peaking at over 400 commits per second. Does that sound right?

If I understand what you're saying, then to sustain this high rate I'm going to 
need a controller that can defer fsync requests from the host because it has 
some sort of battery backup that guarantees the full write.

-- Les

-Original Message-
From: Merlin Moncure [mailto:mmonc...@gmail.com] 
Sent: Tuesday, May 01, 2012 9:43 AM
To: Walker, James Les
Cc: Thomas Kellerer; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows

On Tue, May 1, 2012 at 8:14 AM, Walker, James Les jawal...@cantor.com wrote:
 SSD is OCZ-VERTEX3 MI. Controller is LSI SAS2 2008 Falcon. I'm working on 
 installing EDB. Then I can give you some I/O numbers.

It looks like the ssd doesn't have a nv cache and the raid card is a simple sas 
hba (which likely isn't doing much for the ssd besides masking TRIM).  The OCZ 
'pro' versions are the ones with power loss protection (see:
http://hothardware.com/Reviews/OCZ-Vertex-3-Pro-SandForce-SF2000-Based-SSD-Preview/).
 Note the bullet: Implements SandForce 2582 Controller with power loss data 
protection.  It doesn't look like the Vertex 3 Pro is out yet.

If my hunch is correct, the issue here is that the drive is being asked to sync 
data physically and SSD really don't perform well when the controller isn't in 
control of when and how to sync data.  However full physical sync is the only 
way to guarantee data is truly safe in the context of a unexpected power loss 
(an nv cache is basically a compromise on this point).

merlin
CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are 
confidential. If you are not the named recipient please notify the sender and 
immediately delete it. You may not disseminate, distribute, or forward this 
e-mail message or disclose its contents to anybody else. Copyright and any 
other intellectual property rights in its contents are the sole property of 
Cantor Fitzgerald.
 E-mail transmission cannot be guaranteed to be secure or error-free. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.
 Although we routinely screen for viruses, addressees should check this 
e-mail and any attachments for viruses. We make no representation or warranty 
as to the absence of viruses in this e-mail or any attachments. Please note 
that to ensure regulatory compliance and for the protection of our customers 
and business, we may monitor and read e-mails sent to and from our server(s). 

For further important information, please see  
http://www.cantor.com/legal/statement


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Merlin Moncure
On Tue, May 1, 2012 at 9:44 AM, Walker, James Les jawal...@cantor.com wrote:
 I installed the enterprisedb distribution and immediately saw a 400% 
 performance increase. Turning off fsck made it an order of magnitude better. 
 I'm now peaking at over 400 commits per second. Does that sound right?

yeah -- well it's hard to say but that sounds plausible based on what
i know.  it would be helpful to see the queries you're running to get
apples to apples idea of what's going on.

 If I understand what you're saying, then to sustain this high rate I'm going 
 to need a controller that can defer fsync requests from the host because it 
 has some sort of battery backup that guarantees the full write.

yes --  historically, they way to get your tps rate up was to get a
battery backed cache.  this can give you burst (although not
necessarily sustained) tps rates well above what the drive can handle.
 lately, a few of the better built ssd also have on board capacitors
which provide a similar function and allow the drives to safely hit
high tps rates as well.  take a good look at the intel 320 and 710
drives.

merlin

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Thomas Kellerer

Walker, James Les wrote on 01.05.2012 16:44:

I installed the enterprisedb distribution and immediately saw a 400% 
performance increase.


What exactly is the enterprisedb distribution?
Are you talking about the the Advanced Server?

I would be very surprised if the code base would differ so much to allow such a 
performance gain.
Could it be that the default settings for the Advanced Server are different than those of 
the community edition?

And what did you have installed before that? (as the Windows binary are always 
distributed by EnterpriseDB)

Thomas


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Walker, James Les
Yes. I didn't know the proper vernacular :-)

It is very likely that the default settings are different. I'm looking at that 
right now.

-- Les Walker

-Original Message-
From: pgsql-performance-ow...@postgresql.org 
[mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Thomas Kellerer
Sent: Tuesday, May 01, 2012 1:00 PM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows

Walker, James Les wrote on 01.05.2012 16:44:
 I installed the enterprisedb distribution and immediately saw a 400% 
 performance increase.

What exactly is the enterprisedb distribution?
Are you talking about the the Advanced Server?

I would be very surprised if the code base would differ so much to allow such a 
performance gain.
Could it be that the default settings for the Advanced Server are different 
than those of the community edition?

And what did you have installed before that? (as the Windows binary are always 
distributed by EnterpriseDB)

Thomas


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are 
confidential. If you are not the named recipient please notify the sender and 
immediately delete it. You may not disseminate, distribute, or forward this 
e-mail message or disclose its contents to anybody else. Copyright and any 
other intellectual property rights in its contents are the sole property of 
Cantor Fitzgerald.
 E-mail transmission cannot be guaranteed to be secure or error-free. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.
 Although we routinely screen for viruses, addressees should check this 
e-mail and any attachments for viruses. We make no representation or warranty 
as to the absence of viruses in this e-mail or any attachments. Please note 
that to ensure regulatory compliance and for the protection of our customers 
and business, we may monitor and read e-mails sent to and from our server(s). 

For further important information, please see  
http://www.cantor.com/legal/statement


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-05-01 Thread Walker, James Les
Turns out the 40% was due to a configuration problem with my application. I'm 
now getting the same performance with community edition.

It appears that I'm now CPU bound. My CPU's are all pegged.

-- Les Walker

-Original Message-
From: pgsql-performance-ow...@postgresql.org 
[mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Walker, James Les
Sent: Tuesday, May 01, 2012 3:14 PM
To: 'Thomas Kellerer'; pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows

Yes. I didn't know the proper vernacular :-)

It is very likely that the default settings are different. I'm looking at that 
right now.

-- Les Walker

-Original Message-
From: pgsql-performance-ow...@postgresql.org 
[mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Thomas Kellerer
Sent: Tuesday, May 01, 2012 1:00 PM
To: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] Tuning Postgres 9.1 on Windows

Walker, James Les wrote on 01.05.2012 16:44:
 I installed the enterprisedb distribution and immediately saw a 400% 
 performance increase.

What exactly is the enterprisedb distribution?
Are you talking about the the Advanced Server?

I would be very surprised if the code base would differ so much to allow such a 
performance gain.
Could it be that the default settings for the Advanced Server are different 
than those of the community edition?

And what did you have installed before that? (as the Windows binary are always 
distributed by EnterpriseDB)

Thomas


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are 
confidential. If you are not the named recipient please notify the sender and 
immediately delete it. You may not disseminate, distribute, or forward this 
e-mail message or disclose its contents to anybody else. Copyright and any 
other intellectual property rights in its contents are the sole property of 
Cantor Fitzgerald.
 E-mail transmission cannot be guaranteed to be secure or error-free. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.
 Although we routinely screen for viruses, addressees should check this 
e-mail and any attachments for viruses. We make no representation or warranty 
as to the absence of viruses in this e-mail or any attachments. Please note 
that to ensure regulatory compliance and for the protection of our customers 
and business, we may monitor and read e-mails sent to and from our server(s). 

For further important information, please see  
http://www.cantor.com/legal/statement


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


[PERFORM] Tuning Postgres 9.1 on Windows

2012-04-30 Thread Walker, James Les
I'm trying to benchmark Postgres vs. several other databases on my workstation. 
My workstation is running 64 bit Windows 7. It has 12 gb of RAM and a W3550 @ 3 
Ghz. I installed Postgres 9.1 using the windows installer. The data directory 
is on a 6Gb/s SATA SSD.

My application is multithreaded and uses pooled connections via JDBC. It's got 
around 20 threads doing asynchronous transactions against the database. It's 
about 70% read/30% write. Transactions are very small. There are no 
long-running transactions. I start with an empty database and I only run about 
5,000 business transactions in my benchmark. That results in 10,000 - 15,000 
commits.

When I first installed Postgres I did no tuning at all and was able to get 
around 40 commits per-second which is quite slow. I wanted to establish a 
top-end so I turned off synchronous commit and ran the same test and got the 
same performance of 40 commits per second. I turned on the large system cache 
option on Windows 7 and got the same results. There seems to be some resource 
issues that's limiting me to 40 commits per second but I can't imagine what it 
could be or how to detect it.

I'm not necessarily looking for advice on how to increase performance, but I at 
least need to know how to find the bottleneck.

-- Les Walker

CONFIDENTIAL: This e-mail, including its contents and attachments, if any, are 
confidential. If you are not the named recipient please notify the sender and 
immediately delete it. You may not disseminate, distribute, or forward this 
e-mail message or disclose its contents to anybody else. Copyright and any 
other intellectual property rights in its contents are the sole property of 
Cantor Fitzgerald.
 E-mail transmission cannot be guaranteed to be secure or error-free. The 
sender therefore does not accept liability for any errors or omissions in the 
contents of this message which arise as a result of e-mail transmission.  If 
verification is required please request a hard-copy version.
 Although we routinely screen for viruses, addressees should check this 
e-mail and any attachments for viruses. We make no representation or warranty 
as to the absence of viruses in this e-mail or any attachments. Please note 
that to ensure regulatory compliance and for the protection of our customers 
and business, we may monitor and read e-mails sent to and from our server(s). 

For further important information, please see  
http://www.cantor.com/legal/statement


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-04-30 Thread Andy Colson

On 4/30/2012 8:49 AM, Walker, James Les wrote:

I’m trying to benchmark Postgres vs. several other databases on my
workstation. My workstation is running 64 bit Windows 7. It has 12 gb of
RAM and a W3550 @ 3 Ghz. I installed Postgres 9.1 using the windows
installer. The data directory is on a 6Gb/s SATA SSD.

My application is multithreaded and uses pooled connections via JDBC.
It’s got around 20 threads doing asynchronous transactions against the
database. It’s about 70% read/30% write. Transactions are very small.
There are no long-running transactions. I start with an empty database
and I only run about 5,000 business transactions in my benchmark. That
results in 10,000 – 15,000 commits.

When I first installed Postgres I did no tuning at all and was able to
get around 40 commits per-second which is quite slow. I wanted to
establish a top-end so I turned off synchronous commit and ran the same
test and got the same performance of 40 commits per second. I turned on
the “large system cache” option on Windows 7 and got the same results.
There seems to be some resource issues that’s limiting me to 40 commits
per second but I can’t imagine what it could be or how to detect it.

I’m not necessarily looking for advice on how to increase performance,
but I at least need to know how to find the bottleneck.

-- Les Walker



One thing I'd look at is your hardware and determine if you are CPU 
bound or IO bound.  I use Linux so don't know how you'd do that on windows.


Have you checked your sql statements with explain analyze?

I don't know anything about config file settings on windows, but on 
Linux its really important.  google could probably help you there.


Knowing if you are CPU bound or IO bound, and if you have any bad plans, 
will tell you what config file changes to make.


-Andy


--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-04-30 Thread Merlin Moncure
On Mon, Apr 30, 2012 at 8:49 AM, Walker, James Les jawal...@cantor.com wrote:
 I’m trying to benchmark Postgres vs. several other databases on my
 workstation. My workstation is running 64 bit Windows 7. It has 12 gb of RAM
 and a W3550 @ 3 Ghz. I installed Postgres 9.1 using the windows installer.
 The data directory is on a 6Gb/s SATA SSD.



 My application is multithreaded and uses pooled connections via JDBC. It’s
 got around 20 threads doing asynchronous transactions against the database.
 It’s about 70% read/30% write. Transactions are very small. There are no
 long-running transactions. I start with an empty database and I only run
 about 5,000 business transactions in my benchmark. That results in 10,000 –
 15,000 commits.



 When I first installed Postgres I did no tuning at all and was able to get
 around 40 commits per-second which is quite slow. I wanted to establish a
 top-end so I turned off synchronous commit and ran the same test and got the
 same performance of 40 commits per second. I turned on the “large system
 cache” option on Windows 7 and got the same results. There seems to be some
 resource issues that’s limiting me to 40 commits per second but I can’t
 imagine what it could be or how to detect it.



 I’m not necessarily looking for advice on how to increase performance, but I
 at least need to know how to find the bottleneck.

It's almost certainly coming from postgres being anal about making
sure the data is syncing all the way back to the ssd through all the
buffers.  Although ssd are quite fast, if you run them this way they
are no better than hard drives.  Trying turning off fsync in
postgrsql.conf to be sure.  If you're still seeing poor performance,
try posting and explain analyze of the queries you think might be
slowing you down.

Also, which ssd?

merlin

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-04-30 Thread Thomas Kellerer

Merlin Moncure wrote on 30.04.2012 23:43:

Trying turning off fsync in postgrsql.conf to be sure.


This is a dangerous advise.
Turning off fsync can potentially corrupt the database in case of a system 
failure (e.g. power outage).






--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Postgres 9.1 on Windows

2012-04-30 Thread Merlin Moncure
On Mon, Apr 30, 2012 at 5:00 PM, Thomas Kellerer spam_ea...@gmx.net wrote:
 Merlin Moncure wrote on 30.04.2012 23:43:

 Trying turning off fsync in postgrsql.conf to be sure.


 This is a dangerous advise.
 Turning off fsync can potentially corrupt the database in case of a system
 failure (e.g. power outage).


sure. that said, we're just trying to figure out why he's getting
around 40tps.   since he's only benchmarking test data it's perfectly
ok to do that.

merlin

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


[PERFORM] Tuning wizard

2012-03-12 Thread Michael Kopljan
I just upgraded to 9.1 and downloaded EnterpriseDB Tuning wizard, but the
list of servers in ComboBox support only 8.2, 8.3 i 8.4 version and x86
build. How can I  tune 9.1, 64 bit version?

Is there any workaround, other version for download... ?

 

Any help?

 

Michael.

 



Re: [PERFORM] Tuning wizard

2012-03-12 Thread Tomas Vondra
On 9 Březen 2012, 16:07, Michael Kopljan wrote:
 I just upgraded to 9.1 and downloaded EnterpriseDB Tuning wizard, but the
 list of servers in ComboBox support only 8.2, 8.3 i 8.4 version and x86
 build. How can I  tune 9.1, 64 bit version?

 Is there any workaround, other version for download... ?

Given that Tuning wizard is an EntepriseDB tool, a more appropriate place
for this question is probably
http://forums.enterprisedb.com/forums/list.page


Tomas


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning wizard

2012-03-12 Thread Scott Marlowe
On Mon, Mar 12, 2012 at 10:03 AM, Tomas Vondra t...@fuzzy.cz wrote:
 On 9 Březen 2012, 16:07, Michael Kopljan wrote:
 I just upgraded to 9.1 and downloaded EnterpriseDB Tuning wizard, but the
 list of servers in ComboBox support only 8.2, 8.3 i 8.4 version and x86
 build. How can I  tune 9.1, 64 bit version?

 Is there any workaround, other version for download... ?

 Given that Tuning wizard is an EntepriseDB tool, a more appropriate place
 for this question is probably
 http://forums.enterprisedb.com/forums/list.page

This thread from 30 November 2011 seems to acknowledge there's a problem:

http://forums.enterprisedb.com/posts/list/2973.page

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning wizard

2012-03-12 Thread Raghavendra
On Mon, Mar 12, 2012 at 10:07 PM, Scott Marlowe scott.marl...@gmail.comwrote:

 On Mon, Mar 12, 2012 at 10:03 AM, Tomas Vondra t...@fuzzy.cz wrote:
  On 9 Březen 2012, 16:07, Michael Kopljan wrote:
  I just upgraded to 9.1 and downloaded EnterpriseDB Tuning wizard, but
 the
  list of servers in ComboBox support only 8.2, 8.3 i 8.4 version and x86
  build. How can I  tune 9.1, 64 bit version?
 
  Is there any workaround, other version for download... ?
 
  Given that Tuning wizard is an EntepriseDB tool, a more appropriate place
  for this question is probably
  http://forums.enterprisedb.com/forums/list.page

 This thread from 30 November 2011 seems to acknowledge there's a problem:

 http://forums.enterprisedb.com/posts/list/2973.page


True, its in feature enhancement, soon we expect a new release with PG 9.1
compatible.

---
Regards,
Raghavendra
EnterpriseDB Corporation
Blog: http://raghavt.blogspot.com/


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Andy Colson

On 8/16/2011 8:35 PM, Ogden wrote:

Hope all is well. I have received tremendous help from this list prior and 
therefore wanted some more advice.

I bought some new servers and instead of RAID 5 (which I think greatly hindered 
our writing performance), I configured 6 SCSI 15K drives with RAID 10. This is 
dedicated to /var/lib/pgsql. The main OS has 2 SCSI 15K drives on a different 
virtual disk and also Raid 10, a total of 146Gb. I was thinking of putting 
Postgres' xlog directory on the OS virtual drive. Does this even make sense to 
do?

The system memory is 64GB and the CPUs are dual Intel E5645 chips (they are 
6-core each).

It is a dedicated PostgreSQL box and needs to support heavy read and moderately 
heavy writes.

Currently, I have this for the current system which as 16Gb Ram:

  max_connections = 350

work_mem = 32MB
maintenance_work_mem = 512MB
wal_buffers = 640kB

# This is what I was helped with before and made reporting queries blaze by
seq_page_cost = 1.0
random_page_cost = 3.0
cpu_tuple_cost = 0.5
effective_cache_size = 8192MB

Any help and input is greatly appreciated.

Thank you

Ogden


What seems to be the problem?  I mean, if nothing is broke, then don't 
fix it :-)


You say reporting query's are fast, and the disk's should take care of 
your slow write problem from before.  (Did you test the write 
performance?)  So, whats wrong?



-Andy

--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Ogden

On Aug 17, 2011, at 8:41 AM, Andy Colson wrote:

 On 8/16/2011 8:35 PM, Ogden wrote:
 Hope all is well. I have received tremendous help from this list prior and 
 therefore wanted some more advice.
 
 I bought some new servers and instead of RAID 5 (which I think greatly 
 hindered our writing performance), I configured 6 SCSI 15K drives with RAID 
 10. This is dedicated to /var/lib/pgsql. The main OS has 2 SCSI 15K drives 
 on a different virtual disk and also Raid 10, a total of 146Gb. I was 
 thinking of putting Postgres' xlog directory on the OS virtual drive. Does 
 this even make sense to do?
 
 The system memory is 64GB and the CPUs are dual Intel E5645 chips (they are 
 6-core each).
 
 It is a dedicated PostgreSQL box and needs to support heavy read and 
 moderately heavy writes.
 
 Currently, I have this for the current system which as 16Gb Ram:
 
  max_connections = 350
 
 work_mem = 32MB
 maintenance_work_mem = 512MB
 wal_buffers = 640kB
 
 # This is what I was helped with before and made reporting queries blaze by
 seq_page_cost = 1.0
 random_page_cost = 3.0
 cpu_tuple_cost = 0.5
 effective_cache_size = 8192MB
 
 Any help and input is greatly appreciated.
 
 Thank you
 
 Ogden
 
 What seems to be the problem?  I mean, if nothing is broke, then don't fix it 
 :-)
 
 You say reporting query's are fast, and the disk's should take care of your 
 slow write problem from before.  (Did you test the write performance?)  So, 
 whats wrong?


 I was wondering what the best parameters would be with my new setup. The 
work_mem obviously will increase as will everything else as it's a 64Gb machine 
as opposed to a 16Gb machine. The configuration I posted was for a 16Gb machine 
but this new one is 64Gb. I needed help in how to jump these numbers up. 

Thank you

Ogden
-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Tomas Vondra
On 17 Srpen 2011, 3:35, Ogden wrote:
 Hope all is well. I have received tremendous help from this list prior and
 therefore wanted some more advice.

 I bought some new servers and instead of RAID 5 (which I think greatly
 hindered our writing performance), I configured 6 SCSI 15K drives with
 RAID 10. This is dedicated to /var/lib/pgsql. The main OS has 2 SCSI 15K
 drives on a different virtual disk and also Raid 10, a total of 146Gb. I
 was thinking of putting Postgres' xlog directory on the OS virtual drive.
 Does this even make sense to do?

Yes, but it greatly depends on the amount of WAL and your workload. If you
need to write a lot of WAL data (e.g. during bulk loading), this may
significantly improve performance. It may also help when you have a
write-heavy workload (a lot of clients updating records, background writer
etc.) as that usually means a lot of seeking (while WAL is written
sequentially).

 The system memory is 64GB and the CPUs are dual Intel E5645 chips (they
 are 6-core each).

 It is a dedicated PostgreSQL box and needs to support heavy read and
 moderately heavy writes.

What is the size of the database? So those are the new servers? What's the
difference compared to the old ones? What is the RAID controller, how much
write cache is there?

 Currently, I have this for the current system which as 16Gb Ram:

  max_connections = 350

 work_mem = 32MB
 maintenance_work_mem = 512MB
 wal_buffers = 640kB

Are you really using 350 connections? Something like #cpus + #drives is
usually recommended as a sane number, unless the connections are idle most
of the time. And even in that case a pooling is recommended usually.

Anyway if this worked fine for your workload, I don't think you need to
change those settings. I'd probably bump up the wal_buffers to 16MB - it
might help a bit, definitely won't hurt and it's so little memory it's not
worth the effort I guess.


 # This is what I was helped with before and made reporting queries blaze
 by
 seq_page_cost = 1.0
 random_page_cost = 3.0
 cpu_tuple_cost = 0.5
 effective_cache_size = 8192MB

Are you sure the cpu_tuple_cost = 0.5 is correct? That seems a bit crazy
to me, as it says reading a page sequentially is just twice as expensive
as processing it. This value should be abou 100x lower or something like
that.

What are the checkpoint settings (segments, completion target). What about
shared buffers?

Tomas


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Tomas Vondra
On 17 Srpen 2011, 16:28, Ogden wrote:
  I was wondering what the best parameters would be with my new setup. The
 work_mem obviously will increase as will everything else as it's a 64Gb
 machine as opposed to a 16Gb machine. The configuration I posted was for
 a 16Gb machine but this new one is 64Gb. I needed help in how to jump
 these numbers up.

Well, that really depends on how you come to the current work_mem settings.

If you've decided that with this amount of work_mem the queries run fine
and higher values don't give you better performance (because the amount of
data that needs to be sorted / hashed) fits into the work_mem, then don't
increase it.

But if you've just set it so that the memory is not exhausted, increasing
it may actually help you.

What I think you should review is the amount of shared buffers,
checkpoints and page cache settings (see this for example
http://notemagnet.blogspot.com/2008/08/linux-write-cache-mystery.html).

Tomas


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Ogden

On Aug 17, 2011, at 9:44 AM, Tomas Vondra wrote:

 On 17 Srpen 2011, 3:35, Ogden wrote:
 Hope all is well. I have received tremendous help from this list prior and
 therefore wanted some more advice.
 
 I bought some new servers and instead of RAID 5 (which I think greatly
 hindered our writing performance), I configured 6 SCSI 15K drives with
 RAID 10. This is dedicated to /var/lib/pgsql. The main OS has 2 SCSI 15K
 drives on a different virtual disk and also Raid 10, a total of 146Gb. I
 was thinking of putting Postgres' xlog directory on the OS virtual drive.
 Does this even make sense to do?
 
 Yes, but it greatly depends on the amount of WAL and your workload. If you
 need to write a lot of WAL data (e.g. during bulk loading), this may
 significantly improve performance. It may also help when you have a
 write-heavy workload (a lot of clients updating records, background writer
 etc.) as that usually means a lot of seeking (while WAL is written
 sequentially).

The database is about 200Gb so using /usr/local/pgsql/pg_xlog on a virtual disk 
with 100Gb should not be a problem with the disk space should it?

 The system memory is 64GB and the CPUs are dual Intel E5645 chips (they
 are 6-core each).
 
 It is a dedicated PostgreSQL box and needs to support heavy read and
 moderately heavy writes.
 
 What is the size of the database? So those are the new servers? What's the
 difference compared to the old ones? What is the RAID controller, how much
 write cache is there?
 

I am sorry I overlooked specifying this. The database is about 200Gb and yes 
these are new servers which bring more power (RAM, CPU) over the last one. The 
RAID Controller is a Perc H700 and there is 512Mb write cache. The servers are 
Dells. 

 Currently, I have this for the current system which as 16Gb Ram:
 
 max_connections = 350
 
 work_mem = 32MB
 maintenance_work_mem = 512MB
 wal_buffers = 640kB
 
 Are you really using 350 connections? Something like #cpus + #drives is
 usually recommended as a sane number, unless the connections are idle most
 of the time. And even in that case a pooling is recommended usually.
 
 Anyway if this worked fine for your workload, I don't think you need to
 change those settings. I'd probably bump up the wal_buffers to 16MB - it
 might help a bit, definitely won't hurt and it's so little memory it's not
 worth the effort I guess.

So just increasing the wal_buffers is okay? I thought there would be more as 
the memory in the system is now 4 times as much. Perhaps shared_buffers too 
(down below). 

 
 # This is what I was helped with before and made reporting queries blaze
 by
 seq_page_cost = 1.0
 random_page_cost = 3.0
 cpu_tuple_cost = 0.5
 effective_cache_size = 8192MB
 
 Are you sure the cpu_tuple_cost = 0.5 is correct? That seems a bit crazy
 to me, as it says reading a page sequentially is just twice as expensive
 as processing it. This value should be abou 100x lower or something like
 that.

These settings are for the old server, keep in mind. It's a 16GB machine (the 
new one is 64Gb). The value for cpu_tuple_cost should be 0.005? How are the 
other ones?


 What are the checkpoint settings (segments, completion target). What about
 shared buffers?


#checkpoint_segments = 3# in logfile segments, min 1, 16MB each
#checkpoint_timeout = 5min  # range 30s-1h
checkpoint_completion_target = 0.9  # checkpoint target duration, 0.0 - 1.0 
- was 0.5
#checkpoint_warning = 30s   # 0 disables

And

shared_buffers = 4096MB


Thank you very much

Ogden



-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Tomas Vondra
On 17 Srpen 2011, 18:39, Ogden wrote:
 Yes, but it greatly depends on the amount of WAL and your workload. If
 you
 need to write a lot of WAL data (e.g. during bulk loading), this may
 significantly improve performance. It may also help when you have a
 write-heavy workload (a lot of clients updating records, background
 writer
 etc.) as that usually means a lot of seeking (while WAL is written
 sequentially).

 The database is about 200Gb so using /usr/local/pgsql/pg_xlog on a virtual
 disk with 100Gb should not be a problem with the disk space should it?

I think you've mentioned the database is on 6 drives, while the other
volume is on 2 drives, right? That makes the OS drive about 3x slower
(just a rough estimate). But if the database drive is used heavily, it
might help to move the xlog directory to the OS disk. See how is the db
volume utilized and if it's fully utilized, try to move the xlog
directory.

The only way to find out is to actualy try it with your workload.

 What is the size of the database? So those are the new servers? What's
 the difference compared to the old ones? What is the RAID controller, how
 much write cache is there?

 I am sorry I overlooked specifying this. The database is about 200Gb and
 yes these are new servers which bring more power (RAM, CPU) over the last
 one. The RAID Controller is a Perc H700 and there is 512Mb write cache.
 The servers are Dells.

OK, sounds good although I don't have much experience with this controller.

 Currently, I have this for the current system which as 16Gb Ram:

 max_connections = 350

 work_mem = 32MB
 maintenance_work_mem = 512MB
 wal_buffers = 640kB

 Anyway if this worked fine for your workload, I don't think you need to
 change those settings. I'd probably bump up the wal_buffers to 16MB - it
 might help a bit, definitely won't hurt and it's so little memory it's
 not
 worth the effort I guess.

 So just increasing the wal_buffers is okay? I thought there would be more
 as the memory in the system is now 4 times as much. Perhaps shared_buffers
 too (down below).

Yes, I was just commenting that particular piece of config. Shared buffers
should be increased too.

 # This is what I was helped with before and made reporting queries
 blaze
 by
 seq_page_cost = 1.0
 random_page_cost = 3.0
 cpu_tuple_cost = 0.5
 effective_cache_size = 8192MB

 Are you sure the cpu_tuple_cost = 0.5 is correct? That seems a bit crazy
 to me, as it says reading a page sequentially is just twice as expensive
 as processing it. This value should be abou 100x lower or something like
 that.

 These settings are for the old server, keep in mind. It's a 16GB machine
 (the new one is 64Gb). The value for cpu_tuple_cost should be 0.005? How
 are the other ones?

The default values are like this:

seq_page_cost = 1.0
random_page_cost = 4.0
cpu_tuple_cost = 0.01
cpu_index_tuple_cost = 0.005
cpu_operator_cost = 0.0025

Increasing the cpu_tuple_cost to 0.5 makes it way too expensive I guess,
so the database believes processing two 8kB pages is just as expensive as
reading one from the disk. I guess this change penalizes plans that read a
lot of pages, e.g. sequential scans (and favor index scans etc.). Maybe it
makes sense in your case, I'm just wondering why you set it like that.

 What are the checkpoint settings (segments, completion target). What
 about
 shared buffers?


 #checkpoint_segments = 3# in logfile segments, min 1, 16MB
 each
 #checkpoint_timeout = 5min  # range 30s-1h
 checkpoint_completion_target = 0.9  # checkpoint target duration, 0.0
 - 1.0 - was 0.5
 #checkpoint_warning = 30s   # 0 disables

You need to bump checkpoint segments up, e.g. 64 or maybe even more. This
means how many WAL segments will be available until a checkpoint has to
happen. Checkpoint is a process when dirty buffers from shared buffers are
written to the disk, so it may be very I/O intensive. Each segment is
16MB, so 3 segments is just 48MB of data, while 64 is 1GB.

More checkpoint segments result in longer recovery in case of database
crash (because all the segments since last checkpoint need to be applied).
But it's essential for good write performance.

Completion target seems fine, but I'd consider increasing the timeout too.

 shared_buffers = 4096MB

The usual recommendation is about 25% of RAM for shared buffers, with 64GB
of RAM that is 16GB. And you should increase effective_cache_size too.

See this: http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server

Tomas


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Ogden

On Aug 17, 2011, at 1:56 PM, Tomas Vondra wrote:

 On 17 Srpen 2011, 18:39, Ogden wrote:
 Yes, but it greatly depends on the amount of WAL and your workload. If
 you
 need to write a lot of WAL data (e.g. during bulk loading), this may
 significantly improve performance. It may also help when you have a
 write-heavy workload (a lot of clients updating records, background
 writer
 etc.) as that usually means a lot of seeking (while WAL is written
 sequentially).
 
 The database is about 200Gb so using /usr/local/pgsql/pg_xlog on a virtual
 disk with 100Gb should not be a problem with the disk space should it?
 
 I think you've mentioned the database is on 6 drives, while the other
 volume is on 2 drives, right? That makes the OS drive about 3x slower
 (just a rough estimate). But if the database drive is used heavily, it
 might help to move the xlog directory to the OS disk. See how is the db
 volume utilized and if it's fully utilized, try to move the xlog
 directory.
 
 The only way to find out is to actualy try it with your workload.

Thank you for your help. I just wanted to ask then, for now I should also put 
the xlog directory in the /var/lib/pgsql directory which is on the RAID 
container that is over 6 drives. You see, I wanted to put it on the container 
with the 2 drives because just the OS is installed on it and has the space 
(about 100Gb free). 

But you don't think it will be a problem to put the xlog directory along with 
everything else on /var/lib/pgsql/data? I had seen someone suggesting 
separating it for their setup and it sounded like a good idea so I thought why 
not, but in retrospect and what you are saying with the OS drives being 3x 
slower, it may be okay just to put them on the 6 drives. 

Thoughts?

Thank you once again for your tremendous help

Ogden
-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Scott Marlowe
On Wed, Aug 17, 2011 at 12:56 PM, Tomas Vondra t...@fuzzy.cz wrote:

 I think you've mentioned the database is on 6 drives, while the other
 volume is on 2 drives, right? That makes the OS drive about 3x slower
 (just a rough estimate). But if the database drive is used heavily, it
 might help to move the xlog directory to the OS disk. See how is the db
 volume utilized and if it's fully utilized, try to move the xlog
 directory.

 The only way to find out is to actualy try it with your workload.

This is a very important point.  I've found on most machines with
hardware caching RAID and  8 or fewer 15k SCSI drives it's just as
fast to put it all on one big RAID-10 and if necessary partition it to
put the pg_xlog on its own file system.  After that depending on the
workload you might need a LOT of drives in the pg_xlog dir or just a
pair.Under normal ops many dbs will use only a tiny % of a
dedicated pg_xlog.  Then something like a site indexer starts to run,
and writing heavily to the db, and the usage shoots to 100% and it's
the bottleneck.

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Ogden

On Aug 17, 2011, at 2:14 PM, Scott Marlowe wrote:

 On Wed, Aug 17, 2011 at 12:56 PM, Tomas Vondra t...@fuzzy.cz wrote:
 
 I think you've mentioned the database is on 6 drives, while the other
 volume is on 2 drives, right? That makes the OS drive about 3x slower
 (just a rough estimate). But if the database drive is used heavily, it
 might help to move the xlog directory to the OS disk. See how is the db
 volume utilized and if it's fully utilized, try to move the xlog
 directory.
 
 The only way to find out is to actualy try it with your workload.
 
 This is a very important point.  I've found on most machines with
 hardware caching RAID and  8 or fewer 15k SCSI drives it's just as
 fast to put it all on one big RAID-10 and if necessary partition it to
 put the pg_xlog on its own file system.  After that depending on the
 workload you might need a LOT of drives in the pg_xlog dir or just a
 pair.Under normal ops many dbs will use only a tiny % of a
 dedicated pg_xlog.  Then something like a site indexer starts to run,
 and writing heavily to the db, and the usage shoots to 100% and it's
 the bottleneck.

I suppose this is my confusion. Or rather I am curious about this. On my 
current production database the pg_xlog directory is 8Gb (our total database is 
200Gb). Does this warrant a totally separate setup (and hardware) than PGDATA?
-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning Tips for a new Server

2011-08-17 Thread Tomas Vondra
On 17 Srpen 2011, 21:22, Ogden wrote:
 This is a very important point.  I've found on most machines with
 hardware caching RAID and  8 or fewer 15k SCSI drives it's just as
 fast to put it all on one big RAID-10 and if necessary partition it to
 put the pg_xlog on its own file system.  After that depending on the
 workload you might need a LOT of drives in the pg_xlog dir or just a
 pair.Under normal ops many dbs will use only a tiny % of a
 dedicated pg_xlog.  Then something like a site indexer starts to run,
 and writing heavily to the db, and the usage shoots to 100% and it's
 the bottleneck.

 I suppose this is my confusion. Or rather I am curious about this. On my
 current production database the pg_xlog directory is 8Gb (our total
 database is 200Gb). Does this warrant a totally separate setup (and
 hardware) than PGDATA?

This is not about database size, it's about the workload - the way you're
using your database. Even a small database may produce a lot of WAL
segments, if the workload is write-heavy. So it's impossible to recommend
something except to try that on your own.

Tomas


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


[PERFORM] Tuning Tips for a new Server

2011-08-16 Thread Ogden
Hope all is well. I have received tremendous help from this list prior and 
therefore wanted some more advice. 

I bought some new servers and instead of RAID 5 (which I think greatly hindered 
our writing performance), I configured 6 SCSI 15K drives with RAID 10. This is 
dedicated to /var/lib/pgsql. The main OS has 2 SCSI 15K drives on a different 
virtual disk and also Raid 10, a total of 146Gb. I was thinking of putting 
Postgres' xlog directory on the OS virtual drive. Does this even make sense to 
do?

The system memory is 64GB and the CPUs are dual Intel E5645 chips (they are 
6-core each). 

It is a dedicated PostgreSQL box and needs to support heavy read and moderately 
heavy writes. 

Currently, I have this for the current system which as 16Gb Ram:

 max_connections = 350

work_mem = 32MB
maintenance_work_mem = 512MB
wal_buffers = 640kB

# This is what I was helped with before and made reporting queries blaze by
seq_page_cost = 1.0
random_page_cost = 3.0
cpu_tuple_cost = 0.5
effective_cache_size = 8192MB

Any help and input is greatly appreciated. 

Thank you

Ogden
-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] tuning on ec2

2011-05-12 Thread Josh Berkus

 Sounds like a reasonable starting point.  You could certainly fiddle
 around a bit - especially with shared_buffers - to see if some other
 setting works better, but that should be in the ballpark.

I tend to set it a bit higher on EC2 to discourage the VM from
overcommitting memory I need.  So, I'd do 2.5GB for that one.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


[PERFORM] tuning on ec2

2011-04-26 Thread Joel Reymont
I'm running pgsql on an m1.large EC2 instance with 7.5gb available memory. 

The free command shows 7gb of free+cached. My understand from the docs is that 
I should dedicate 1.75gb to shared_buffers (25%) and set effective_cache_size 
to 7gb. 

Is this correct? I'm running 64-bit Ubuntu 10.10, e.g. 

Linux ... 2.6.35-28-virtual #50-Ubuntu SMP Fri Mar 18 19:16:26 UTC 2011 x86_64 
GNU/Linux

Thanks, Joel

--
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
-++---
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
-++---




-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning massive UPDATES and GROUP BY's?

2011-03-14 Thread Marti Raudsepp
On Sun, Mar 13, 2011 at 18:36, runner run...@winning.com wrote:
 Tried removing the indexes and other constraints just for
 the import but for a noob like me, this was too much to ask.  Maybe
 when I get more experience.

pgAdmin should make it pretty easy. Choose each index and constraint,
save the code from the SQL pane for when you need to restore it, and
do a right click - Drop

 Other than being very inefficient, and consuming
 more time than necessary, is there any other down side to importing
 into an indexed table?

Doing so will result in somewhat larger (more bloated) indexes, but
generally the performance impact of this is minimal.

Regards,
Marti

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning massive UPDATES and GROUP BY's?

2011-03-14 Thread Greg Spiegelberg
On Mon, Mar 14, 2011 at 4:17 AM, Marti Raudsepp ma...@juffo.org wrote:

 On Sun, Mar 13, 2011 at 18:36, runner run...@winning.com wrote:
  Other than being very inefficient, and consuming
  more time than necessary, is there any other down side to importing
  into an indexed table?

 Doing so will result in somewhat larger (more bloated) indexes, but
 generally the performance impact of this is minimal.


Bulk data imports of this size I've done with minimal pain by simply
breaking the raw data into chunks (10M records becomes 10 files of 1M
records), on a separate spindle from the database, and performing multiple
COPY commands but no more than 1 COPY per server core.  I tested this a
while back on a 4 core server and when I attempted 5 COPY's at a time the
time to complete went up almost 30%.  I don't recall any benefit having
fewer than 4 in this case but the server was only processing my data at the
time.  Indexes were on the target table however I dropped all constraints.
 The UNIX split command is handy for breaking the data up into individual
files.

Greg


Re: [PERFORM] Tuning massive UPDATES and GROUP BY's?

2011-03-13 Thread runner
Don't insert data into an indexed table. A very important point with


bulk-loading is that you should load all the data first, then create
the indexes. Running multiple (different) CREATE INDEX queries in
parallel can additionally save a lot of time. Also don't move data
back and forth between the tables, just drop the original when you're
done.

I just saw your post and it looks similar to what I'm doing.
We're going to be loading 12G of data from a MySQL dump into our 
pg 9.0.3 database next weekend.  I've been testing this for the last
two weeks.  Tried removing the indexes and other constraints just for
the import but for a noob like me, this was too much to ask.  Maybe
when I get more experience.  So I *WILL* be importing all of my data
into indexed tables.  I timed it and it will take eight hours.  

I'm sure I could get it down to two or three hours for the import
if I really knew more about postgres but that's the price you pay when
you slam dunk a project and your staff isn't familiar with the 
database back-end.  Other than being very inefficient, and consuming 
more time than necessary, is there any other down side to importing 
into an indexed table?  In the four test imports I've done,
everything seems to work fine, just takes a long time.

Sorry for hijacking your thread here!

 


Re: [PERFORM] Tuning massive UPDATES and GROUP BY's?

2011-03-12 Thread Marti Raudsepp
On Fri, Mar 11, 2011 at 21:06, fork forkandw...@gmail.com wrote:
 Like the following?  Will it rebuild the indexes in a sensical way?

Don't insert data into an indexed table. A very important point with
bulk-loading is that you should load all the data first, then create
the indexes. Running multiple (different) CREATE INDEX queries in
parallel can additionally save a lot of time. Also don't move data
back and forth between the tables, just drop the original when you're
done.

Doing this should give a significant performance win. Partitioning
them to fit in cache should improve it further, but I'm not sure
anymore that it's worthwhile considering the costs and extra
maintenance.

 Is there a rule of thumb on tradeoffs in a partitioned table?

The only certain thing is that you'll lose group aggregate and
merge join query plans. If you only see HashAggregate plans when
you EXPLAIN your GROUP BY queries then it probably won't make much of
a difference.

 I would use the partition column whatever I am most likely
 to cluster by in a single big table, right?

Yes.

Regards,
Marti

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning massive UPDATES and GROUP BY's?

2011-03-11 Thread fork
Marti Raudsepp marti at juffo.org writes:

 If you don't mind long recovery times in case of a crash, set
 checkpoint_segments to ~100 and checkpoint_completion_target=0.9; this
 will improve write throughput significantly.

Sounds good.

 Also, if you don't mind CORRUPTing your database after a crash,
 setting fsync=off and full_page_writes=off gives another significant
 boost.

I probably won't do this... ;)

 UPDATE on a table with many indexes will probably be slower. If you
 want to speed up this part, use INSERT INTO x SELECT and take this
 chance to partition your table, 

Like the following?  Will it rebuild the indexes in a sensical way?

BEGIN;
CREATE TABLE tempfoo as SELECT *, foo + bar AS newcol FROM bar;
TRUNCATE foo;
ALTER TABLE foo ADD COLUMN newcol;
INSERT INTO foo SELECT * FROM tempfoo;
DROP TABLE tempfoo;
END;

 such that each individual partition
 and most indexes will fit in your cache. 

Is there a rule of thumb on tradeoffs in a partitioned table?  About half the
time, I will want to do GROUP BY's that use the partition column, but about half
the time I won't.  (I would use the partition column whatever I am most likely
to cluster by in a single big table, right?)

For example, I might intuitively partition by age5 (into 20 tables like tab00,
tab05, tab10, etc). Often a query would be SELECT ... FROM PARENTTABLE GROUP BY
age5, race, etc, but often it would be GROUP BY state or whatever with no
age5 component.

I know I can experiment ;), but it takes a while to load anything, and i would
rather stand on the shoulders.

Thanks so much for all your helps!




-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


[PERFORM] Tuning massive UPDATES and GROUP BY's?

2011-03-10 Thread fork
Given that doing a massive UPDATE SET foo = bar || ' ' || baz; on a 12 million
row table (with about 100 columns -- the US Census PUMS for the 2005-2009 ACS)
is never going to be that fast, what should one do to make it faster?

I set work_mem to 2048MB, but it currently is only using a little bit of memory
and CPU. (3% and 15% according to top; on a SELECT DISTINCT ... LIMIT earlier,
it was using 70% of the memory).

The data is not particularly sensitive; if something happened and it rolled
back, that wouldnt be the end of the world.  So I don't know if I can use
dangerous setting for WAL checkpoints etc.   There are also aren't a lot of
concurrent hits on the DB, though a few.

I am loathe to create a new table from a select, since the indexes themselves
take a really long time to build.

As the title alludes, I will also be doing GROUP BY's on the data, and would
love to speed these up, mostly just for my own impatience...  




-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning massive UPDATES and GROUP BY's?

2011-03-10 Thread Merlin Moncure
On Thu, Mar 10, 2011 at 9:40 AM, fork forkandw...@gmail.com wrote:
 Given that doing a massive UPDATE SET foo = bar || ' ' || baz; on a 12 million
 row table (with about 100 columns -- the US Census PUMS for the 2005-2009 ACS)
 is never going to be that fast, what should one do to make it faster?

 I set work_mem to 2048MB, but it currently is only using a little bit of 
 memory
 and CPU. (3% and 15% according to top; on a SELECT DISTINCT ... LIMIT earlier,
 it was using 70% of the memory).

 The data is not particularly sensitive; if something happened and it rolled
 back, that wouldnt be the end of the world.  So I don't know if I can use
 dangerous setting for WAL checkpoints etc.   There are also aren't a lot of
 concurrent hits on the DB, though a few.

 I am loathe to create a new table from a select, since the indexes themselves
 take a really long time to build.

you are aware that updating the field for the entire table, especially
if there is an index on it (or any field being updated), will cause
all your indexes to be rebuilt anyways?  when you update a record, it
gets a new position in the table, and a new index entry with that
position.  insert/select to temp, + truncate + insert/select back is
usually going to be faster and will save you the reindex/cluster.
otoh, if you have foreign keys it can be a headache.

 As the title alludes, I will also be doing GROUP BY's on the data, and would
 love to speed these up, mostly just for my own impatience...

need to see the query here to see if you can make them go faster.

merlin

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning massive UPDATES and GROUP BY's?

2011-03-10 Thread fork
Merlin Moncure mmoncure at gmail.com writes:

  I am loathe to create a new table from a select, since the indexes 
  themselves
  take a really long time to build.
 
 you are aware that updating the field for the entire table, especially
 if there is an index on it (or any field being updated), will cause
 all your indexes to be rebuilt anyways?  when you update a record, it
 gets a new position in the table, and a new index entry with that
 position.  
 insert/select to temp, + truncate + insert/select back is
 usually going to be faster and will save you the reindex/cluster.
 otoh, if you have foreign keys it can be a headache.

Hmph.  I guess I will have to find a way to automate it, since there will be a
lot of times I want to do this. 

  As the title alludes, I will also be doing GROUP BY's on the data, and would
  love to speed these up, mostly just for my own impatience...
 
 need to see the query here to see if you can make them go faster.

I guess I was hoping for a blog entry on general guidelines given a DB that is
really only for batch analysis versus transaction processing.  Like put all
your temp tables on a different disk or whatever.  I will post specifics later.


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Tuning massive UPDATES and GROUP BY's?

2011-03-10 Thread Marti Raudsepp
On Thu, Mar 10, 2011 at 17:40, fork forkandw...@gmail.com wrote:
 The data is not particularly sensitive; if something happened and it rolled
 back, that wouldnt be the end of the world.  So I don't know if I can use
 dangerous setting for WAL checkpoints etc.   There are also aren't a lot of
 concurrent hits on the DB, though a few.

If you don't mind long recovery times in case of a crash, set
checkpoint_segments to ~100 and checkpoint_completion_target=0.9; this
will improve write throughput significantly.

Also, if you don't mind CORRUPTing your database after a crash,
setting fsync=off and full_page_writes=off gives another significant
boost.

 I am loathe to create a new table from a select, since the indexes themselves
 take a really long time to build.

UPDATE on a table with many indexes will probably be slower. If you
want to speed up this part, use INSERT INTO x SELECT and take this
chance to partition your table, such that each individual partition
and most indexes will fit in your cache. Index builds from a warm
cache are very fast in PostgreSQL. You can create several indexes at
once in separate sessions, and the table will only be scanned once.

Don't forget to bump up maintenance_work_mem for index builds, 256MB
might be a reasonable arbitrary value.

The downside is that partitioning can interfere with your read queries
if they expect the data in a sorted order. But then, HashAggregate
tends to be faster than GroupAggregate in many cases, so this might
not matter for your queries. Alternatively you can experiment with
PostgreSQL 9.1 alpha, which has mostly fixed this shortcoming with the
merge append plan node.

 As the title alludes, I will also be doing GROUP BY's on the data, and would
 love to speed these up, mostly just for my own impatience...

I think regular tuning is the best you can do here.

Regards,
Marti

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


[PERFORM] tuning auto vacuum for highly active tables

2010-03-25 Thread Bhella Paramjeet-PFCW67
Hi All,

 

We have a postgres database in which couple of tables get bloated due to
heavy inserts and deletes. Auto vacuum is running. My question is  how
can I make auto vacuum more aggressive? I am thinking of enabling
autovacuum_vacuum_cost_delay and autovacuum_vacuum_cost_limit
parameters. Can anyone suggest how to calculate the appropriate values
for these parameters and if there are any side effects of enabling these
parameters.Any help will be highly appreciated.

 

Thanks

Paramjeet Kaur



Re: [PERFORM] Tuning Postgresql on Windows XP Pro 32 bit [was on HACKERS list]

2008-01-18 Thread Kevin Grittner
 On Tue, Jan 15, 2008 at 11:05 AM, in message
[EMAIL PROTECTED], Doug Knight
[EMAIL PROTECTED] wrote: 
 
 We tried reducing the memory footprint of the postgres processes, via
 shared_buffers (from 3 on Linux to 3000 on Windows),
 
I would never go below 1.  2 to 3 is a good start.
 
 max_fsm_pages (from 2000250 on Linux to 10 on Windows)
 max_fsm_relations (from 2 on Linux to 5000 on Windows)
 
Figure out what you need and use that.  Low values can cause bloat.
Check the output of VACUUM ANALYZE VERBOSE (for the database)
for a line like this:
 
INFO:  free space map contains 717364 pages in 596 relations
 
 and max_connections (from 222 on Linux to 100 on Windows).
 
If you don't need more than 100, good.
 
 Another variable we played with was effective_cache_size
 (174000 on Linux, 43700 on Windows).
 
Figure out how much space is available to cache data,
counting both the shared buffers and the Windows cache.
This setting has no affect on memory usage, just the planner.
 
 On Mon, 2008-01-07 at 19:49 +0500, Usama Dar wrote: 
 
 Doug Knight wrote:
  We are running the binary distribution, version 8.2.5-1, installed on 
  Windows XP Pro 32 bit with SP2. We typically run postgres on linux, 
  but have a need to run it under windows as well. Our typical admin 
  tuning for postgresql.conf doesn't seem to be as applicable for windows.
 
 So what have you tuned so far? what are your current postgresql settings 
 that you have modified? What are your system specs for Hardware, RAM , 
 CPU etc?
 
I would add that you should post what problems you're seeing.
 
 Is there a place where I can find information about tuning
 postgresql running on a Windows XP Pro 32 bit system? I
 installed using the binary installer. I am seeing a high page
 fault delta and total page faults for one of the postgresql
 processes.
 
Are those hard or soft page faults.  There's a big difference.
Review Windows docs for descriptions and how to check.
You may not be able to do much about soft page faults,
and they may not have much impact on your performance.
 
We abandoned Windows for database servers after testing
identical processing on identical hardware -- Linux was twice
as fast.  If you really need to run under Windows, you may need
to adjust your performance expectations.
 
-Kevin
 


---(end of broadcast)---
TIP 7: You can help support the PostgreSQL project by donating at

http://www.postgresql.org/about/donate


[PERFORM] Tuning Postgresql on Windows XP Pro 32 bit [was on HACKERS list]

2008-01-15 Thread Doug Knight
All,
I have been asked to move this thread to the performance list. Below is
the full discussion to this point.

Doug Knight
WSI Corp
Andover, MA

 Forwarded Message 
From: Doug Knight [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]
Subject: Re: [HACKERS] Tuning Postgresql on Windows XP Pro 32 bit
Date: Tue, 15 Jan 2008 09:09:16 -0500

We tried reducing the memory footprint of the postgres processes, via
shared_buffers (from 3 on Linux to 3000 on Windows), max_fsm_pages
(from 2000250 on Linux to 10 on Windows), max_fsm_relations (from
2 on Linux to 5000 on Windows), and max_connections (from 222 on
Linux to 100 on Windows). Another variable we played with was
effective_cache_size  (174000 on Linux, 43700 on Windows). None of these
reduced memory usage, or improved performance,  significantly. We still
see the high page fault rate too. Other things we tried were reducing
the number of WAL buffers, and changing the wal_sync_method to
opendata_sync, all with minimal effect. I've attached the latest version
of our Windows postgresql.conf file.

Doug



On Mon, 2008-01-07 at 19:49 +0500, Usama Dar wrote: 

 Doug Knight wrote:
  We are running the binary distribution, version 8.2.5-1, installed on 
  Windows XP Pro 32 bit with SP2. We typically run postgres on linux, 
  but have a need to run it under windows as well. Our typical admin 
  tuning for postgresql.conf doesn't seem to be as applicable for windows.
 
 
 So what have you tuned so far? what are your current postgresql settings 
 that you have modified? What are your system specs for Hardware, RAM , 
 CPU etc?
 
 
 On Sun, 2008-01-06 at 18:23 +0500, Usama Dar wrote:
 
 
 
 On Jan 3, 2008 8:57 PM, Doug Knight [EMAIL PROTECTED] wrote:
 
 All,
 Is there a place where I can find information about tuning
 postgresql running on a Windows XP Pro 32 bit system? I
 installed using the binary installer. I am seeing a high page
 fault delta and total page faults for one of the postgresql
 processes. Any help would be great. 
 
 
 
 
 Which version of postgres? the process you are seeing this for is a
 user process? 
 
 
 
 -- 
 Usama Munir Dar http://www.linkedin.com/in/usamadar
 Consultant Architect
 Cell:+92 321 5020666
 Skype: usamadar 


[PERFORM] tuning for TPC-C benchmark

2007-11-22 Thread [EMAIL PROTECTED]
Hello all,
I'm doing tests on various Database and in particular I'm running a 
comparison between Oracle 10g and Postgres 8.1 on a dedicated server 
with 2 processors Dual-Core AMD Opteron 2218  2.6 GHz, 4GB of memory 
and Debian GNU / Linux version 2.6.18-5. Performance is very similar up 
to 30 users, but from 40 onwards with Postgres fall quickly. That is 
not what happens with Oracle that comes to 600 users. Can you help me 
with the tuning ?
Thanks a lot
My postgresql.conf configuration is:
#---
# FILE LOCATIONS
#---

# The default values of these variables are driven from the -D command 
line
# switch or PGDATA environment variable, represented here as 
ConfigDir.

#data_directory = 'ConfigDir'   # use data in another 
directory
hba_file = '/etc/postgresql/8.1/main/pg_hba.conf'   # host-based 
authentication file
ident_file = '/etc/postgresql/8.1/main/pg_ident.conf'   # IDENT 
configuration file

# If external_pid_file is not explicitly set, no extra pid file is 
written.
external_pid_file = '/var/run/postgresql/8.1-main.pid'  # 
write an extra pid file


#---
# CONNECTIONS AND AUTHENTICATION
#---

# - Connection Settings -

#listen_addresses = 'localhost' # what IP address(es) to 
listen on;
  # comma-
separated list of addresses;
 # defaults to 
'localhost', '*' = all
listen_addresses = '*'
port = 5432
max_connections = 220
# note: increasing max_connections costs ~400 bytes of shared memory 
per
# connection slot, plus lock space (see max_locks_per_transaction).  
You
# might also need to raise shared_buffers to support more connections.
#superuser_reserved_connections = 2
unix_socket_directory = '/var/run/postgresql'
#unix_socket_group = ''
#unix_socket_permissions = 0777 # octal
#bonjour_name = ''  # defaults to the computer 
name

# - Security  Authentication -

#authentication_timeout = 60# 1-600, in seconds
ssl = true
#password_encryption = on
#db_user_namespace = off

# Kerberos
#krb_server_keyfile = ''
#krb_srvname = 'postgres'
#krb_server_hostname = ''   # empty string matches any 
keytab entry
#krb_caseins_users = off

# - TCP Keepalives -
# see 'man 7 tcp' for details

#tcp_keepalives_idle = 0# TCP_KEEPIDLE, in seconds;
   # 0 selects the 
system default
#tcp_keepalives_interval = 0# TCP_KEEPINTVL, in seconds;
 # 0 selects 
the system default
#tcp_keepalives_count = 0  # TCP_KEEPCNT;
 # 0 selects 
the system default


#---
# RESOURCE USAGE (except WAL)
#---

# - Memory -

shared_buffers = 49152# min 16 or max_connections*2, 
8KB each, 384MB
temp_buffers = 1000 # min 100, 8KB each
max_prepared_transactions = 350 # can be 0 or more
# note: increasing max_prepared_transactions costs ~600 bytes of 
shared memory
# per transaction slot, plus lock space (see 
max_locks_per_transaction).
work_mem = 1024 # min 64, size in 
KB
maintenance_work_mem = 524288   # min 1024, size in KB, -512
MB-
max_stack_depth = 6144  # min 100, size in KB

# - Free Space Map -
max_fsm_pages = 58000   # min max_fsm_relations*16, 6 
bytes each
max_fsm_relations = 3000# min 100, ~70 bytes each

# - Kernel Resource Usage -

#max_files_per_process = 1000   # min 25
#preload_libraries = ''

# - Cost-Based Vacuum Delay -

#vacuum_cost_delay = 0  # 0-1000 milliseconds
#vacuum_cost_page_hit = 1   # 0-1 credits
#vacuum_cost_page_miss = 10 # 0-1 credits
#vacuum_cost_page_dirty = 20# 0-1 credits
#vacuum_cost_limit = 200# 0-1 credits

# - Background writer -

#bgwriter_delay = 5000  # 10-1 milliseconds 
between rounds
bgwriter_lru_percent = 0# 0-100% of LRU buffers 
scanned/round
bgwriter_lru_maxpages = 0   # 0-1000 buffers max 
written/round
bgwriter_all_percent = 0# 0-100% of all buffers 
scanned/round
bgwriter_all_maxpages = 0   # 0-1000 buffers max 
written/round


#---
# WRITE AHEAD LOG

Re: [PERFORM] tuning for TPC-C benchmark

2007-11-22 Thread Bill Moran
[EMAIL PROTECTED] [EMAIL PROTECTED] wrote:

 Hello all,
 I'm doing tests on various Database and in particular I'm running a 
 comparison between Oracle 10g and Postgres 8.1 on a dedicated server 
 with 2 processors Dual-Core AMD Opteron 2218  2.6 GHz, 4GB of memory 
 and Debian GNU / Linux version 2.6.18-5. Performance is very similar up 
 to 30 users, but from 40 onwards with Postgres fall quickly. That is 
 not what happens with Oracle that comes to 600 users. Can you help me 
 with the tuning ?

If you're doing perf comparisons, you should start out with the latest
PostgreSQL: 8.2.5

Also, beware that you may violate license agreements if you publish
benchmarks of Oracle ... and posting partial results to a mailing list
could potentially be considered publishing benchmarks to Oracle's
lawyers.

I've added a few more comments inline, but overall it looks like you've
done a good job tuning.  In order to tweak it any further, we're probably
going to need more details, such as iostat output during the run, details
of the test you're running, etc.

 Thanks a lot
 My postgresql.conf configuration is:
 #---
 # FILE LOCATIONS
 #---
 
 # The default values of these variables are driven from the -D command 
 line
 # switch or PGDATA environment variable, represented here as 
 ConfigDir.
 
 #data_directory = 'ConfigDir'   # use data in another 
 directory
 hba_file = '/etc/postgresql/8.1/main/pg_hba.conf'   # host-based 
 authentication file
 ident_file = '/etc/postgresql/8.1/main/pg_ident.conf'   # IDENT 
 configuration file
 
 # If external_pid_file is not explicitly set, no extra pid file is 
 written.
 external_pid_file = '/var/run/postgresql/8.1-main.pid'  # 
 write an extra pid file
 
 
 #---
 # CONNECTIONS AND AUTHENTICATION
 #---
 
 # - Connection Settings -
 
 #listen_addresses = 'localhost' # what IP address(es) to 
 listen on;
   # comma-
 separated list of addresses;
  # defaults to 
 'localhost', '*' = all
 listen_addresses = '*'
 port = 5432
 max_connections = 220
 # note: increasing max_connections costs ~400 bytes of shared memory 
 per
 # connection slot, plus lock space (see max_locks_per_transaction).  
 You
 # might also need to raise shared_buffers to support more connections.
 #superuser_reserved_connections = 2
 unix_socket_directory = '/var/run/postgresql'
 #unix_socket_group = ''
 #unix_socket_permissions = 0777 # octal
 #bonjour_name = ''  # defaults to the computer 
 name
 
 # - Security  Authentication -
 
 #authentication_timeout = 60# 1-600, in seconds
 ssl = true
 #password_encryption = on
 #db_user_namespace = off
 
 # Kerberos
 #krb_server_keyfile = ''
 #krb_srvname = 'postgres'
 #krb_server_hostname = ''   # empty string matches any 
 keytab entry
 #krb_caseins_users = off
 
 # - TCP Keepalives -
 # see 'man 7 tcp' for details
 
 #tcp_keepalives_idle = 0# TCP_KEEPIDLE, in seconds;
# 0 selects the 
 system default
 #tcp_keepalives_interval = 0# TCP_KEEPINTVL, in seconds;
  # 0 selects 
 the system default
 #tcp_keepalives_count = 0  # TCP_KEEPCNT;
  # 0 selects 
 the system default
 
 
 #---
 # RESOURCE USAGE (except WAL)
 #---
 
 # - Memory -
 
 shared_buffers = 49152# min 16 or max_connections*2, 
 8KB each, 384MB

With 4G of ram, you might want to try this closer to 1G and see if it
helps.  You may want to install the pg_buffercache module to monitor
shared_buffer usage.  I doubt you want to use it during actual timing
of the test, but it should help you get a feel for what the best
setting is for shared_buffers.

 temp_buffers = 1000 # min 100, 8KB each
 max_prepared_transactions = 350 # can be 0 or more
 # note: increasing max_prepared_transactions costs ~600 bytes of 
 shared memory
 # per transaction slot, plus lock space (see 
 max_locks_per_transaction).
 work_mem = 1024 # min 64, size in 
 KB
 maintenance_work_mem = 524288   # min 1024, size in KB, -512
 MB-
 max_stack_depth = 6144  # min 100, size in KB
 
 # - Free Space Map -
 max_fsm_pages = 58000   # min max_fsm_relations*16, 6 
 bytes each
 max_fsm_relations = 3000  

Re: [PERFORM] tuning for TPC-C benchmark

2007-11-22 Thread Jonah H. Harris
On Nov 22, 2007 10:45 AM, Kevin Grittner [EMAIL PROTECTED] wrote:
 I suggest testing with some form of connection pooling.

Yeah, that's one of the reasons I suggested DBT-2.  It pools
connections and is the most mature TPC-C-like test for Postgres.

-- 
Jonah H. Harris, Sr. Software Architect | phone: 732.331.1324
EnterpriseDB Corporation| fax: 732.331.1301
499 Thornall Street, 2nd Floor  | [EMAIL PROTECTED]
Edison, NJ 08837| http://www.enterprisedb.com/

---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster


Re: [PERFORM] tuning for TPC-C benchmark

2007-11-22 Thread Heikki Linnakangas

[EMAIL PROTECTED] wrote:

Hello all,
I'm doing tests on various Database and in particular I'm running a 
comparison between Oracle 10g and Postgres 8.1 on a dedicated server 
with 2 processors Dual-Core AMD Opteron 2218  2.6 GHz, 4GB of memory 
and Debian GNU / Linux version 2.6.18-5. Performance is very similar up 
to 30 users, but from 40 onwards with Postgres fall quickly. That is 
not what happens with Oracle that comes to 600 users. Can you help me 
with the tuning ?


The fact that you didn't give any details on your I/O configuration 
tells me that you don't have much experience with TPC-C. TPC-C is 
basically limited by random I/O. That means that a good RAID controller 
and a lot of disks is a must. Looking at some of the results at 
www.tpc.org, systems with 4 cores have multiple RAID controllers and 
about a hundred hard drives.


You can of course run smaller tests, but those 4 cores are going spend 
all their time waiting for I/O. See for example these old DBT-2 results 
I ran to test the Load Distributed Checkpoints feature in 8.3.


Now that we got that out of the way, what kind of a test configuration 
are you using? How many warehouses? Are you using the think-times, per 
the spec, or are you running something like BenchmarkSQL which just 
pushes as many queries it can to the server?


I'm not sure what you mean by # of users, but you shouldn't use more 
than 10-30 connections on a test like that. More won't help, because 
they'll all have to queue for the same resources, whether it's I/O or CPU.


How long tests are you running? After some time, you'll need to run 
vacuums, which make a big difference.


8.3 will perform better, thanks to HOT which reduces the need to vacuum, 
varvarlen which reduces storage size, leading to better use of the cache 
and less I/O, and Load Distributed Checkpoints, which reduce the 
checkpoint spikes which otherwise throw you over the response time 
requirements.


And last but not least, why are you running the benchmark? It's going to 
be practically irrelevant for any real application. You should benchmark 
with your application, and your data, to get a comparison that matters 
for you.


--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
  subscribe-nomail command to [EMAIL PROTECTED] so that your
  message can get through to the mailing list cleanly


[PERFORM] Tuning Help - What did I do wrong?

2007-10-04 Thread Josh Trutwin
We have a pretty busy linux server running postgres 8.1.4, waiting to
upgrade until 8.3 to avoid dump/restoring twice.

# cat /proc/meminfo
total:used:free:  shared: buffers:  cached:
Mem:  3704217600 3592069120 1121484800 39460864 2316271616
Swap: 2516918272   270336 2516647936

# cat /proc/cpuinfo
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 15
model   : 4
model name  : Intel(R) Xeon(TM) CPU 3.00GHz
stepping: 3
cpu MHz : 2992.795

The postgresql.conf was basically the default so I decided to
increase the cache size and a couple paramaters to make more use of
that memory - here's what I did:

shared_buffers = 16384 (was 1000)
work_mem = 16384 (was 1024)
wal_buffers = 24 (was 8)
checkpoint_segments = 5 (was 3)
effective_cache_size = 1 (was 1000)
stats_command_string = on (was off)
stats_block_level = on (was off)
stats_row_level = on (was off)

In order to do this I had to change /proc/sys/kernel/shmmax to
536870912 (don't have /etc/sysctl)

Also, the entire cluster gets vacuumed analyzed nightly. 

After making these changes, the performance on the server actually
worsened.   I slowly backed off on some of the paramaters but didn't
seem to help.

Wondering if those changes are silly?  For a server this size I
didn't think this would be problematic.

Thank you,

Josh


---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings


Re: [PERFORM] Tuning Help - What did I do wrong?

2007-10-04 Thread Scott Marlowe
On 10/4/07, Josh Trutwin [EMAIL PROTECTED] wrote:
 We have a pretty busy linux server running postgres 8.1.4, waiting to
 upgrade until 8.3 to avoid dump/restoring twice.

You should immediate update your version to 8.1.whateverislatest.
That requires no dump / restore and it is a bug fix update.  I doubt
this problem is because you're out of date on patches, but who
knows...

 # cat /proc/meminfo
 total:used:free:  shared: buffers:  cached:
 Mem:  3704217600 3592069120 1121484800 39460864 2316271616
 Swap: 2516918272   270336 2516647936

Well, you've got plenty of memory, and a large chunk is being used as cache.

 The postgresql.conf was basically the default so I decided to
 increase the cache size and a couple paramaters to make more use of
 that memory - here's what I did:

 shared_buffers = 16384 (was 1000)
 work_mem = 16384 (was 1024)
 wal_buffers = 24 (was 8)
 checkpoint_segments = 5 (was 3)
 effective_cache_size = 1 (was 1000)
 stats_command_string = on (was off)
 stats_block_level = on (was off)
 stats_row_level = on (was off)

Your changes seem reasonable.

 Also, the entire cluster gets vacuumed analyzed nightly.

You should look into running the autovacuum daemon.  for heavily used
databases nightly vacuuming may not be enough.

 After making these changes, the performance on the server actually
 worsened.   I slowly backed off on some of the paramaters but didn't
 seem to help.

Most likely turning on stats collection slowed you down a bit.

We need to see examples of what's slow, including explain analyze
output for slow queries.  Also a brief explanation of the type of load
your database server is seeing.  I.e. is it a lot of little
transactions, mostly read, batch processing, lots of users, one user,
etc...   Right now we don't have enough info to really help you.

---(end of broadcast)---
TIP 4: Have you searched our list archives?

   http://archives.postgresql.org


Re: [PERFORM] Tuning Help - What did I do wrong?

2007-10-04 Thread Scott Marlowe
Oh, and in addition to  my previous message, you should use tools like
vmstat, iostat and top to get an idea of what your server is doing.

What kind of drive subsystem do you have?  What kind of raid controller?  etc...

---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings


Re: [PERFORM] Tuning Help - What did I do wrong?

2007-10-04 Thread Josh Trutwin
On Thu, 4 Oct 2007 11:19:22 -0500
Scott Marlowe [EMAIL PROTECTED] wrote:

 We need to see examples of what's slow, including explain analyze
 output for slow queries.  Also a brief explanation of the type of
 load your database server is seeing.  I.e. is it a lot of little
 transactions, mostly read, batch processing, lots of users, one
 user, etc...   Right now we don't have enough info to really help
 you.

Sorry, this server is for a few (100+?) websites so it's running
along site apache, php.  All connections to postgresql (except for
the occaional psql console login) are done from php requests, using
the same user (basically there are two users, the one php uses and
postgres).  The bulk of the activity would be reads, but
certainly inesrts/updates/deletes would be interspersed in there.
Most of the activity is done via auto-commits, not many long
transactions.

From your followup email:

 ... you should use tools like vmstat, iostat and top to get an idea
 of what your server is doing.

# vmstat
   procs  memoryswap  io
system cpu
 r  b  w   swpd   free   buff  cache  si  sobibo   incs
us  sy  id
 3  1  0268  68332  39016 2201436   0   0 3 34
2   3   4   2

sorry about the wrapping...

iostat is not found - will see if I can download it.  top typically
shows postmaster as the top process with 10-15% of the CPU, followed
by apache threads.

 12:01pm  up 104 days, 12:05,  2 users,  load average: 9.75, 9.30,
7.70
215 processes: 214 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  0.1% user,  0.0% system,  0.0% nice,  0.4% idle
Mem:  3617400K av, 3552784K used,  64616K free,  0K shrd,  37456K
buff
Swap: 2457928K av,264K used, 2457664K free
2273664K cached

  PID USER PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME
COMMAND
31797 postgres  17   0 28836  28M  1784 S   0  8.5  0.7  10:15
postmaster

 What kind of drive subsystem do you have?  What kind of raid
 controller?  etc...

Gathering more information on this - Raid is a software
RAID-1.  Some information: 
I believe itI believe it
# df -h
FilesystemSize  Used Avail Use% Mounted on
/dev/md0   66G   50G   16G  76% /
/dev/sda1  15M  6.6M  8.5M  44% /boot

# cat /proc/mdstat
Personalities : [raid0] [raid1]
read_ahead 1024 sectors
md0 : active raid1 sdb3[0] sdc3[1]
  70573440 blocks [2/2] [UU]

unused devices: none

Thanks for your help, I'm more of a developer guy so let me know what
else is useful.

Josh

---(end of broadcast)---
TIP 6: explain analyze is your friend


Re: [PERFORM] Tuning Help - What did I do wrong?

2007-10-04 Thread Scott Marlowe
On 10/4/07, Josh Trutwin [EMAIL PROTECTED] wrote:
 On Thu, 4 Oct 2007 11:19:22 -0500
 Scott Marlowe [EMAIL PROTECTED] wrote:

  We need to see examples of what's slow, including explain analyze
  output for slow queries.  Also a brief explanation of the type of
  load your database server is seeing.  I.e. is it a lot of little
  transactions, mostly read, batch processing, lots of users, one
  user, etc...   Right now we don't have enough info to really help
  you.

 Sorry, this server is for a few (100+?) websites so it's running
 along site apache, php.  All connections to postgresql (except for
 the occaional psql console login) are done from php requests, using
 the same user (basically there are two users, the one php uses and
 postgres).  The bulk of the activity would be reads, but
 certainly inesrts/updates/deletes would be interspersed in there.
 Most of the activity is done via auto-commits, not many long
 transactions.

So, are there certain queries that are much slower than the others?
Run them from psql with explain analyze in front of them and post the
query and the output here.

 From your followup email:

  ... you should use tools like vmstat, iostat and top to get an idea
  of what your server is doing.

 # vmstat
procs  memoryswap  io
 system cpu
  r  b  w   swpd   free   buff  cache  si  sobibo   incs
 us  sy  id
  3  1  0268  68332  39016 2201436   0   0 3 34
 2   3   4   2

vmstat needs to be run for a while to give you useful numbers.   try:

vmstat 5

and let it run for a few minutes.  The first line won't count so much,
but after that you'll get more reasonable numbers.

 iostat is not found - will see if I can download it.  top typically
 shows postmaster as the top process with 10-15% of the CPU, followed
 by apache threads.

What OS are you on?

  12:01pm  up 104 days, 12:05,  2 users,  load average: 9.75, 9.30,
 7.70

That's pretty heavy load.  I notice there's no wait % listed for CPU,
so I assume it's not a late model Linux kernel or anything.

 215 processes: 214 sleeping, 1 running, 0 zombie, 0 stopped
 CPU states:  0.1% user,  0.0% system,  0.0% nice,  0.4% idle
 Mem:  3617400K av, 3552784K used,  64616K free,  0K shrd,  37456K
 buff
 Swap: 2457928K av,264K used, 2457664K free
 2273664K cached

   PID USER PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME
 COMMAND
 31797 postgres  17   0 28836  28M  1784 S   0  8.5  0.7  10:15
 postmaster

Are the postmasters using most of the CPU?  OR the other processes?

  What kind of drive subsystem do you have?  What kind of raid
  controller?  etc...

 Gathering more information on this - Raid is a software
 RAID-1.  Some information:

OK, given that it's read mostly, it's likely not a problem that a
faster RAID controller would help.  Possibly more drives in a RAID 10
would help a little, but let's look at optimizing your query and
postmaster first.

Do you have the postmaster configured to log long running queries?
That's a good starting point.  also google pg_fouine (I think I spelt
it right) for analyzing your logs.

It's quite likely the issue here is one long running query that
chewing all your I/O or CPU and making everything else slow.  Once we
find that query things should get better and we can worry about
performance tuning in a more leisurely manner.

---(end of broadcast)---
TIP 4: Have you searched our list archives?

   http://archives.postgresql.org


Re: [PERFORM] Tuning Help - What did I do wrong?

2007-10-04 Thread Kevin Grittner
 On Thu, Oct 4, 2007 at 10:28 AM, in message
[EMAIL PROTECTED], Josh Trutwin
[EMAIL PROTECTED] wrote: 
 running postgres 8.1.4
 
 # cat /proc/meminfo
 total:used:free:  shared: buffers:  cached:
 Mem:  3704217600 3592069120 1121484800 39460864 2316271616
 
 shared_buffers = 16384 (was 1000)
 effective_cache_size = 1 (was 1000)
 
It's kind of silly to tell PostgreSQL that its total cache space is 1
pages when you've got more than that in shared buffers plus all that OS
cache space.  Try something around 285000 pages for effective_cache_size.
 
 stats_command_string = on (was off)
 stats_block_level = on (was off)
 stats_row_level = on (was off)
 
 After making these changes, the performance on the server actually
 worsened.   I slowly backed off on some of the paramaters but didn't
 seem to help.
 
Did you try turning off the collection of those additional statistics?
That isn't free.
 
You didn't get specific about what you saw in performance problems.  If
you are seeing occasional freezes of all queries, you are likely looking
at a known issue with spikiness of disk output.  For some this can be
corrected by using very aggressive background writer settings.  Some have
solved it by disabling OS write delays.  Some haven't found a solution and
are waiting for 8.3; there have been some serious changes made to attempt
to resolve this issue.
 
-Kevin
 



---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
   choose an index scan if your joining column's datatypes do not
   match


  1   2   3   >