Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Scott Marlowe
On Tue, Feb 24, 2009 at 12:40 AM, Jordan Tomkinson jor...@moodle.com wrote: On Tue, Feb 24, 2009 at 11:42 AM, Greg Smith gsm...@gregsmith.com wrote: Right, the useful thing to do in this case is to take a look at how big all the relations (tables, indexes) involved are at each of the steps

Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Aidan Van Dyk
* Greg Smith gsm...@gregsmith.com [090201 00:00]: Shouldn't someone have ranted about RAID-5 by this point in the thread? What? Sorry, I wasn't paying attention... You mean someone's actually still using RAID-5? ;-) -- Aidan Van Dyk Create like a

Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Jordan Tomkinson
On Wed, Feb 25, 2009 at 12:05 AM, Aidan Van Dyk ai...@highrise.ca wrote: * Greg Smith gsm...@gregsmith.com [090201 00:00]: Shouldn't someone have ranted about RAID-5 by this point in the thread? What? Sorry, I wasn't paying attention... You mean someone's actually still using RAID-5?

Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Joshua D. Drake
On Wed, 2009-02-25 at 09:21 +0900, Jordan Tomkinson wrote: On Wed, Feb 25, 2009 at 12:05 AM, Aidan Van Dyk ai...@highrise.ca wrote: * Greg Smith gsm...@gregsmith.com [090201 00:00]: Shouldn't someone have ranted about RAID-5 by this point in the thread?

Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Greg Williamson
--- On Wed, 2/25/09, Jordan Tomkinson jor...@moodle.com wrote: ... What exactly is wrong with RAID5 and what should we have gone with? RAID10 is often used. As others have pointed out, it is very slow for random writes. It also has issues that expose your data to total loss, see for

Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Jordan Tomkinson
On Wed, Feb 25, 2009 at 9:23 AM, Joshua D. Drake j...@commandprompt.comwrote: RAID5 outside of RAID 0 is the worst possible RAID level to run with a database. (of the commonly used raid level's that is). It is very, very slow on random writes which is what databases do. Switch to RAID 10.

Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Scott Marlowe
On Tue, Feb 24, 2009 at 5:21 PM, Jordan Tomkinson jor...@moodle.com wrote: On Wed, Feb 25, 2009 at 12:05 AM, Aidan Van Dyk ai...@highrise.ca wrote: * Greg Smith gsm...@gregsmith.com [090201 00:00]: Shouldn't someone have ranted about RAID-5 by this point in the thread? What?  Sorry, I

Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Joshua D. Drake
On Wed, 2009-02-25 at 09:44 +0900, Jordan Tomkinson wrote: On Wed, Feb 25, 2009 at 9:23 AM, Joshua D. Drake j...@commandprompt.com wrote: RAID5 outside of RAID 0 is the worst possible RAID level to run with a database. (of the commonly used raid

Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Jordan Tomkinson
On Tue, Feb 24, 2009 at 4:40 PM, Jordan Tomkinson jor...@moodle.com wrote: taken before the ~7000 rows were entered. relation| size ---+ public.mdl_log| 595 MB public.mdl_forum_posts| 375 MB

Re: [GENERAL] High cpu usage after many inserts

2009-02-24 Thread Ron Mayer
Joshua D. Drake wrote: On Wed, 2009-02-25 at 09:21 +0900, Jordan Tomkinson wrote: On Wed, Feb 25, 2009 at 12:05 AM, Aidan Van Dyk ai...@highrise.ca wrote: * Greg Smith gsm...@gregsmith.com [090201 00:00]: Shouldn't someone have ranted about RAID-5 by this point in

Re: [GENERAL] High cpu usage after many inserts

2009-02-23 Thread Jordan Tomkinson
Scott, DB Schema: http://demo.moodle.org/db_schema.txt SQL Query log: http://demo.moodle.org/querylog.txt There are _much_ more queries than I anticipated :/ Jordan

Re: [GENERAL] High cpu usage after many inserts

2009-02-23 Thread Scott Marlowe
On Mon, Feb 23, 2009 at 12:42 AM, Jordan Tomkinson jor...@moodle.com wrote: On Mon, Feb 23, 2009 at 4:29 PM, Scott Marlowe scott.marl...@gmail.com wrote: Oh yeah, what OS is this? Version and all that. I should probably clarify that the high cpu only exists while the jmeter tests are

Re: [GENERAL] High cpu usage after many inserts

2009-02-23 Thread Jordan Tomkinson
On Mon, Feb 23, 2009 at 5:20 PM, Scott Marlowe scott.marl...@gmail.comwrote: On Mon, Feb 23, 2009 at 12:42 AM, Jordan Tomkinson jor...@moodle.com wrote: On Mon, Feb 23, 2009 at 4:29 PM, Scott Marlowe scott.marl...@gmail.com wrote: Oh yeah, what OS is this? Version and all that.

Re: [GENERAL] High cpu usage after many inserts

2009-02-23 Thread Scott Marlowe
On Mon, Feb 23, 2009 at 1:29 AM, Jordan Tomkinson jor...@moodle.com wrote: On Mon, Feb 23, 2009 at 5:20 PM, Scott Marlowe scott.marl...@gmail.com wrote: On Mon, Feb 23, 2009 at 12:42 AM, Jordan Tomkinson jor...@moodle.com wrote: On Mon, Feb 23, 2009 at 4:29 PM, Scott Marlowe

Re: [GENERAL] High cpu usage after many inserts

2009-02-23 Thread Markus Wanner
Hi, Scott Marlowe wrote: Oh, what is an LMS? A Learning Management System, not to be confused with a CMS, which might also stand for a Course Management System ;-) Regards Markus Wanner -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your

Re: [GENERAL] High cpu usage after many inserts

2009-02-23 Thread Jordan Tomkinson
On Tue, Feb 24, 2009 at 12:35 AM, Scott Marlowe scott.marl...@gmail.comwrote: On Mon, Feb 23, 2009 at 1:29 AM, Jordan Tomkinson jor...@moodle.com wrote: As per the spreadsheet (http://spreadsheets.google.com/pub?key=pu_k0R6vNvOVP26TRZdtdYw) CPU usage is around 50% and starts climbing

Re: [GENERAL] High cpu usage after many inserts

2009-02-23 Thread Greg Smith
On Mon, 23 Feb 2009, Scott Marlowe wrote: well that's pretty normal as the indexes grow large enough to not fit in cache, then not fit in memory, etc... Right, the useful thing to do in this case is to take a look at how big all the relations (tables, indexes) involved are at each of the

Re: [GENERAL] High cpu usage after many inserts

2009-02-23 Thread Jordan Tomkinson
On Tue, Feb 24, 2009 at 11:42 AM, Greg Smith gsm...@gregsmith.com wrote: On Mon, 23 Feb 2009, Scott Marlowe wrote: well that's pretty normal as the indexes grow large enough to not fit in cache, then not fit in memory, etc... Right, the useful thing to do in this case is to take a look at

Re: [GENERAL] High cpu usage after many inserts

2009-02-23 Thread Jordan Tomkinson
On Tue, Feb 24, 2009 at 11:42 AM, Greg Smith gsm...@gregsmith.com wrote: Right, the useful thing to do in this case is to take a look at how big all the relations (tables, indexes) involved are at each of the steps in the process. The script at http://wiki.postgresql.org/wiki/Disk_Usage will

[GENERAL] High cpu usage after many inserts

2009-02-22 Thread Jordan Tomkinson
Hi list, We are running postgresql 8.3.5 and are trying to stress test our LMS. The problem is when our stress tester (Jmeter) inserts around 10,000 rows (in 3 hours) over 2 tables (5000 rows each table) the CPU of the sql server hits 100% over all 4 cores for all future inserts. I have tried

Re: [GENERAL] High cpu usage after many inserts

2009-02-22 Thread Scott Marlowe
On Sun, Feb 22, 2009 at 11:55 PM, Jordan Tomkinson jor...@moodle.com wrote: Hi list, We are running postgresql 8.3.5 and are trying to stress test our LMS. The problem is when our stress tester (Jmeter) inserts around 10,000 rows (in 3 hours) over 2 tables (5000 rows each table) the CPU of

Re: [GENERAL] High cpu usage after many inserts

2009-02-22 Thread Scott Marlowe
On Sun, Feb 22, 2009 at 11:55 PM, Jordan Tomkinson jor...@moodle.com wrote: Hi list, We are running postgresql 8.3.5 and are trying to stress test our LMS. The problem is when our stress tester (Jmeter) inserts around 10,000 rows (in 3 hours) over 2 tables (5000 rows each table) the CPU of

Re: [GENERAL] High cpu usage after many inserts

2009-02-22 Thread Scott Marlowe
One last thing. You were doing vacuum fulls but NOT reindexing, right? I quote from the document at google docs: 13:50:00vacuum full analyze on all databases through pgadmin 1: Do you have evidence that regular autovacuum isn't keeping up? 2: If you have such evidence, and you have to

Re: [GENERAL] High cpu usage after many inserts

2009-02-22 Thread Jordan Tomkinson
On Mon, Feb 23, 2009 at 4:03 PM, Scott Marlowe scott.marl...@gmail.comwrote: On Sun, Feb 22, 2009 at 11:55 PM, Jordan Tomkinson jor...@moodle.com wrote: Hi list, We are running postgresql 8.3.5 and are trying to stress test our LMS. The problem is when our stress tester (Jmeter) inserts

Re: [GENERAL] High cpu usage after many inserts

2009-02-22 Thread Jordan Tomkinson
On Mon, Feb 23, 2009 at 4:08 PM, Scott Marlowe scott.marl...@gmail.comwrote: One last thing. You were doing vacuum fulls but NOT reindexing, right? I quote from the document at google docs: 13:50:00vacuum full analyze on all databases through pgadmin 1: Do you have evidence that

Re: [GENERAL] High cpu usage after many inserts

2009-02-22 Thread Scott Marlowe
On Mon, Feb 23, 2009 at 12:18 AM, Jordan Tomkinson jor...@moodle.com wrote: On Mon, Feb 23, 2009 at 4:08 PM, Scott Marlowe scott.marl...@gmail.com wrote: One last thing. You were doing vacuum fulls but NOT reindexing, right? I quote from the document at google docs: 13:50:00

Re: [GENERAL] High cpu usage after many inserts

2009-02-22 Thread Scott Marlowe
Oh yeah, what OS is this? Version and all that. -- Sent via pgsql-general mailing list (pgsql-general@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general

Re: [GENERAL] High cpu usage after many inserts

2009-02-22 Thread Jordan Tomkinson
On Mon, Feb 23, 2009 at 4:29 PM, Scott Marlowe scott.marl...@gmail.comwrote: Oh yeah, what OS is this? Version and all that. Red Hat Enterprise Linux 5.3 x64 kernel 2.6.18-128.el5 os and hardware details are in the google spreadsheet, you might have to refresh it. Im working on getting the

Re: [GENERAL] High cpu usage after many inserts

2009-02-22 Thread Jordan Tomkinson
On Mon, Feb 23, 2009 at 4:29 PM, Scott Marlowe scott.marl...@gmail.comwrote: Oh yeah, what OS is this? Version and all that. I should probably clarify that the high cpu only exists while the jmeter tests are running, once the tests are finished the cpu returns to 0% (this isnt a production