Re: [GENERAL] Reducing pg_dump pg_restore times
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/27/06 02:26, Coder At Heart wrote: Hi! Thanks so much for the responses. The system configuration is something like this- That's a start... Linux, Presumably a 2.6 kernel? Which, and which distro version? PG 8.1, OK ext3, OK RAID 1 setup with two controllers Not enough info. Soft RAID, controller RAID, IDE, SATA, SCSI U160, U320, SAS, 10K RPM, 15K RPM, etc, etc? And tape drive. What kind of tape drive? Are the disks on one controller and the tape drive on another? The machine has 4 processors with 32 Gig of RAM. Presumably modern. AMD, Intel, GHz? And then there's SK's suggestions: *) increasing maintainance_work_mem and work_mem massively *) increasing the number of checkpoint segments *) disabling fsync during the load Thanks! Shivkumar On 10/27/06, Stefan Kaltenbrunner [EMAIL PROTECTED] wrote: Ron Johnson wrote: On 10/26/06 01:20, Chris wrote: Coder At Heart wrote: Hi! [snip] So, - What OS? - What version of PG? - what kind of disk system does the DB live on? - How many controllers? - What kind of tape drive? - Is it on it's own SCSI controller? in addition to that basic information we need there are: *) increasing maintainance_work_mem and work_mem massively *) increasing the number of checkpoint segments *) disabling fsync during the load - -- Ron Johnson, Jr. Jefferson LA USA Is common sense really valid? For example, it is common sense to white-power racists that whites are superior to blacks, and that those with brown skins are mud people. However, that common sense is obviously wrong. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQFFQdEoS9HxQb37XmcRAt2GAKCVARbdY5wwcoOIINP22T9u/1VA1wCgo9F+ Qi5ou/YfdLCjzcpmRHONlwE= =ONU4 -END PGP SIGNATURE- ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster
Re: [GENERAL] Reducing pg_dump pg_restore times
Coder At Heart wrote: Hi! Restore backup take time in the tune of 6 hrs on a Linux, 4 proc, 32 G RAM machine for a 60GB database. Please always cc the mailing list. I've never played with databases that large so I don't know if that's good or bad.. others on the list will be able to offer advice/suggestions. -- Postgresql php tutorials http://www.designmagick.com/ ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
Re: [GENERAL] Reducing pg_dump pg_restore times
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/26/06 01:20, Chris wrote: Coder At Heart wrote: Hi! Restore backup take time in the tune of 6 hrs on a Linux, 4 proc, 32 G RAM machine for a 60GB database. Please always cc the mailing list. I've never played with databases that large so I don't know if that's good or bad.. others on the list will be able to offer advice/suggestions. RAM certainly helps, but pg_dump is single-threaded, and, by definition, backing up and restoring 60GB of data is *extremely* IO constrained. So, - - What OS? - - What version of PG? - - what kind of disk system does the DB live on? - - How many controllers? - - What kind of tape drive? - - Is it on it's own SCSI controller? - -- Ron Johnson, Jr. Jefferson LA USA Is common sense really valid? For example, it is common sense to white-power racists that whites are superior to blacks, and that those with brown skins are mud people. However, that common sense is obviously wrong. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) iD8DBQFFQH10S9HxQb37XmcRAhFpAKCeBZbTPQW8mhY8EgG26R7OgpgrlQCeMAek KmPbaWmLx+N+kYQvHNOgiL0= =YsM3 -END PGP SIGNATURE- ---(end of broadcast)--- TIP 9: In versions below 8.0, the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [GENERAL] Reducing pg_dump pg_restore times
On Thursday 26 October 2006 02:20, Chris wrote:
Coder At Heart wrote:
Hi!
Restore backup take time in the tune of 6 hrs on a Linux, 4 proc, 32 G
RAM machine for a 60GB database.
Please always cc the mailing list.
I've never played with databases that large so I don't know if that's
good or bad.. others on the list will be able to offer advice/suggestions.
A while back I wrote something up on increasing restore times, might be worth
checking out.
http://people.planetpostgresql.org/xzilla/index.php?/archives/133-Getting-faster-database-restores-on-postgresql-8.1.html
--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org/
Re: [GENERAL] Reducing pg_dump pg_restore times
Ron Johnson wrote: On 10/26/06 01:20, Chris wrote: Coder At Heart wrote: Hi! Restore backup take time in the tune of 6 hrs on a Linux, 4 proc, 32 G RAM machine for a 60GB database. Please always cc the mailing list. I've never played with databases that large so I don't know if that's good or bad.. others on the list will be able to offer advice/suggestions. RAM certainly helps, but pg_dump is single-threaded, and, by definition, backing up and restoring 60GB of data is *extremely* IO constrained. well from what I have seen is that on large databases and a sufficiently fast disk-io subsystem actually CPU(or rather the speed of a single core) starts to be the bottleneck with current postgresql versions. That is true for both COPY and to a much greater effect index creation (I have seen restores that took days and most of that was purely index creation). 8.2 has improved considerably on that due to the massive improvments in the external sorting code but it will still be bottleneck by the single-process nature of psql. So, - What OS? - What version of PG? - what kind of disk system does the DB live on? - How many controllers? - What kind of tape drive? - Is it on it's own SCSI controller? in addition to that basic information we need there are: *) increasing maintainance_work_mem and work_mem massively *) increasing the number of checkpoint segments *) disabling fsync during the load Stefan ---(end of broadcast)--- TIP 6: explain analyze is your friend
Re: [GENERAL] Reducing pg_dump pg_restore times
C.M. Shivkumar wrote: Hello all! I have databases of sizes ranging from 30-60GB that have to be backed-up and restored. I want to reduce the pg_dump and pg_restore times for the same. Can someone throw some light on how to go about it. I have tried tweaking the postgresql.conf properties. Though, I am not sure what each property means. If someone could point to a document on the same it would be really helpful. How long does a backup take (roughly) ? This page links off to each section you need to look at: http://www.postgresql.org/docs/8.1/static/runtime-config.html -- Postgresql php tutorials http://www.designmagick.com/ ---(end of broadcast)--- TIP 1: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly
