Re: [PERFORM] Query Performance and IOWait
Just as an update, We installed RHE Update4 beta kernel on a box and it seems to have solved our issues. Woody iGLASS Networks 211-A S. Salem St Apex NC 27502 (919) 387-3550 x813 www.iglass.net ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
Re: [PERFORM] Query Performance and IOWait
Check the linux-dell list for more...The PERC3/Di cards are specifically
Adaptec, not most. PERC4/DC is LSI Megaraid. Unless you buy the cheaper
version, most will come with battery.
-anjan
-Original Message-
From: Andrew Janian [mailto:[EMAIL PROTECTED]
Sent: Friday, November 19, 2004 4:22 PM
To: [EMAIL PROTECTED]
Subject: Re: [PERFORM] Query Performance and IOWait
The data that we are accessing is via QLogic cards connected to an EMC
Clarion. We have tried it on local SCSI disks with the same (bad)
results.
When the machine gets stuck in a 100% IOWAIT state it often crashes soon
after that.
The disks are fine, have been replaced and checked.
Here are my results from hdparm -Tt /dev/sda1 (which is the EMC disk
array)
/dev/sda1:
Timing buffer-cache reads: 2976 MB in 2.00 seconds = 1488.00 MB/sec
Timing buffered disk reads: 44 MB in 3.13 seconds = 14.06 MB/sec
-Original Message-
From: Dave Cramer [mailto:[EMAIL PROTECTED]
Sent: Thursday, November 18, 2004 11:14 AM
To: Andrew Janian
Cc: [EMAIL PROTECTED]
Subject: Re: [PERFORM] Query Performance and IOWait
Andrew,
Dell's aren't well known for their disk performance, apparently most of
the perc controllers sold with dell's are actually adaptec controllers.
Also apparently they do not come with the battery required to use the
battery backed up write cache ( In fact according to some Dell won't
even sell the battery to you). Also Dell's monitoring software is quite
a memory hog.
Have you looked at top ?, and also hdparm -Tt /dev/sd?
Dave
Andrew Janian wrote:
>Hello All,
>
>I have a setup with a Dell Poweredge 2650 with Red Hat and Postgres
7.4.5 with a database with about 27GB of data. The table in question
has about 35 million rows.
>
>I am running the following query:
>
>SELECT *
>FROM mb_fix_message
>WHERE msg_client_order_id IN (
> SELECT msg_client_order_id
> FROM mb_fix_message
> WHERE msg_log_time >= '2004-06-01'
> AND msg_log_time < '2004-06-01 13:30:00.000'
> AND msg_message_type IN ('D','G')
> AND mb_ord_type = '1'
> )
> AND msg_log_time > '2004-06-01'
> AND msg_log_time < '2004-06-01 23:59:59.999'
> AND msg_message_type = '8'
> AND (mb_raw_text LIKE '%39=1%' OR mb_raw_text LIKE '%39=2%');
>
>with the following plan:
>
>
QUERY PLAN
>Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
> -> Index Scan using mfi_log_time on mb_fix_message
(cost=0.00..22231.31 rows=2539 width=526)
> Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp
without time zone) AND (msg_log_time < '2004-06-01
23:59:59.999'::timestamp without time zone))
> Filter: (((msg_message_type)::text = '8'::text) AND
(((mb_raw_text)::text ~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~
'%39=2%'::text)))
> -> Index Scan using mfi_client_ordid on mb_fix_message
(cost=0.00..445.56 rows=1 width=18)
> Index Cond: (("outer".msg_client_order_id)::text =
(mb_fix_message.msg_client_order_id)::text)
> Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp
without time zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp
without time zone) AND ((msg_message_type)::text = 'D'::text) OR
((msg_message_type)::text = 'G'::text)) AND ((mb_ord_type)::text =
'1'::text))
>
>While running, this query produces 100% iowait usage on its processor
and takes a ungodly amount of time (about an hour).
>
>The postgres settings are as follows:
>
>shared_buffers = 32768 # min 16, at least max_connections*2,
8KB each
>sort_mem = 262144 # min 64, size in KB
>
>And the /etc/sysctl.conf has:
>kernel.shmall = 274235392
>kernel.shmmax = 274235392
>
>The system has 4GB of RAM.
>
>I am pretty sure of these settings, but only from my reading of the
docs and others' recommendations online.
>
>Thanks,
>
>Andrew Janian
>OMS Development
>Scottrade Financial Services
>(314) 965-1555 x 1513
>Cell: (314) 369-2083
>
>---(end of
broadcast)---
>TIP 7: don't forget to increase your free space map settings
>
>
--
Dave Cramer
http://www.postgresintl.com
519 939 0336
ICQ#14675561
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
---(end of broadcast)---
TIP 8: explain analyze is your friend
Re: [PERFORM] Query Performance and IOWait
The data that we are accessing is via QLogic cards connected to an EMC Clarion.
We have tried it on local SCSI disks with the same (bad) results.
When the machine gets stuck in a 100% IOWAIT state it often crashes soon after
that.
The disks are fine, have been replaced and checked.
Here are my results from hdparm -Tt /dev/sda1 (which is the EMC disk array)
/dev/sda1:
Timing buffer-cache reads: 2976 MB in 2.00 seconds = 1488.00 MB/sec
Timing buffered disk reads: 44 MB in 3.13 seconds = 14.06 MB/sec
-Original Message-
From: Dave Cramer [mailto:[EMAIL PROTECTED]
Sent: Thursday, November 18, 2004 11:14 AM
To: Andrew Janian
Cc: [EMAIL PROTECTED]
Subject: Re: [PERFORM] Query Performance and IOWait
Andrew,
Dell's aren't well known for their disk performance, apparently most of
the perc controllers sold with dell's are actually adaptec controllers.
Also apparently they do not come with the battery required to use the
battery backed up write cache ( In fact according to some Dell won't
even sell the battery to you). Also Dell's monitoring software is quite
a memory hog.
Have you looked at top ?, and also hdparm -Tt /dev/sd?
Dave
Andrew Janian wrote:
>Hello All,
>
>I have a setup with a Dell Poweredge 2650 with Red Hat and Postgres 7.4.5 with
>a database with about 27GB of data. The table in question has about 35
>million rows.
>
>I am running the following query:
>
>SELECT *
>FROM mb_fix_message
>WHERE msg_client_order_id IN (
> SELECT msg_client_order_id
> FROM mb_fix_message
> WHERE msg_log_time >= '2004-06-01'
> AND msg_log_time < '2004-06-01 13:30:00.000'
> AND msg_message_type IN ('D','G')
> AND mb_ord_type = '1'
> )
> AND msg_log_time > '2004-06-01'
> AND msg_log_time < '2004-06-01 23:59:59.999'
> AND msg_message_type = '8'
> AND (mb_raw_text LIKE '%39=1%' OR mb_raw_text LIKE '%39=2%');
>
>with the following plan:
>
>
>
>
> QUERY PLAN
>Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
> -> Index Scan using mfi_log_time on mb_fix_message (cost=0.00..22231.31
> rows=2539 width=526)
> Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp without
> time zone) AND (msg_log_time < '2004-06-01 23:59:59.999'::timestamp without
> time zone))
> Filter: (((msg_message_type)::text = '8'::text) AND
> (((mb_raw_text)::text ~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~
> '%39=2%'::text)))
> -> Index Scan using mfi_client_ordid on mb_fix_message (cost=0.00..445.56
> rows=1 width=18)
> Index Cond: (("outer".msg_client_order_id)::text =
> (mb_fix_message.msg_client_order_id)::text)
> Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp without time
> zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp without time zone)
> AND ((msg_message_type)::text = 'D'::text) OR ((msg_message_type)::text =
> 'G'::text)) AND ((mb_ord_type)::text = '1'::text))
>
>While running, this query produces 100% iowait usage on its processor and
>takes a ungodly amount of time (about an hour).
>
>The postgres settings are as follows:
>
>shared_buffers = 32768 # min 16, at least max_connections*2, 8KB each
>sort_mem = 262144 # min 64, size in KB
>
>And the /etc/sysctl.conf has:
>kernel.shmall = 274235392
>kernel.shmmax = 274235392
>
>The system has 4GB of RAM.
>
>I am pretty sure of these settings, but only from my reading of the docs and
>others' recommendations online.
>
>Thanks,
>
>Andrew Janian
>OMS Development
>Scottrade Financial Services
>(314) 965-1555 x 1513
>Cell: (314) 369-2083
>
>---(end of broadcast)---
>TIP 7: don't forget to increase your free space map settings
>
>
--
Dave Cramer
http://www.postgresintl.com
519 939 0336
ICQ#14675561
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
Re: [PERFORM] Query Performance and IOWait
On Thu, 18 Nov 2004 12:14:00 -0500 Dave Cramer <[EMAIL PROTECTED]> wrote: > Andrew, > > Dell's aren't well known for their disk performance, apparently most > of the perc controllers sold with dell's are actually adaptec > controllers. Also apparently they do not come with the battery > required to use the battery backed up write cache ( In fact according > to some Dell won't even sell the battery to you). Also Dell's > monitoring software is quite a memory hog. > > Have you looked at top ?, and also hdparm -Tt /dev/sd? I haven't seen any PERC controllers that were really Adaptec ones, but I for one quit buying Dell RAID controllers several years ago because of poor Linux support and performance. On one machine (not a PostgreSQL server) we saw a 20% speed improvement by switching to software raid. If you have a test machine, I would suggest moving the data to a box without a RAID controller and see if you get better results. - Frank Wiles <[EMAIL PROTECTED]> http://www.wiles.org - ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [PERFORM] Query Performance and IOWait
From our experience it is not just a postgres issue, but all IO with the Update 3 kernel. We have a box with Update 3 that queries a remote postgres database(Running RH7.3, RH3 Update2) and writes to a file on an NFS server. The update 3 box does half the work with 2-3 times the load as our update 1 and 2 boxes. Looking at top the box is always above 90% IO Wait on the CPU. When we downgrade the kernel to Update 2 it seems to fix the issue. We several Update 3 boxes that run postgres locally and they all struggle compared to the Update 2 boxes We have tried the Fedora Core 3 with not much more success and we are going to try the Update 4 beta kernel next week to see if it is any better. There are several threads on the Taroon mailing list discussing the issue. Woody -Original Message- From: Josh Berkus [mailto:[EMAIL PROTECTED] Sent: Thursday, November 18, 2004 1:34 PM To: [EMAIL PROTECTED] Cc: Woody Woodring; 'Andrew Janian' Subject: Re: [PERFORM] Query Performance and IOWait Woody, > What version of Redhat are you running? We have found running > Enterprise Update 3 kernel kills our Dell boxes with IOWait, both NFS > and local disk traffic. Update 2 kernel does not seem to have the > issue, and we are in the process of trying Update 4 beta to see if it is better. This is interesting; do you have more to say about it? I've been having some mysterious issues with RHES that I've not been able to pin down. -- Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match
Re: [PERFORM] Query Performance and IOWait
Woody, > What version of Redhat are you running? We have found running Enterprise > Update 3 kernel kills our Dell boxes with IOWait, both NFS and local disk > traffic. Update 2 kernel does not seem to have the issue, and we are in > the process of trying Update 4 beta to see if it is better. This is interesting; do you have more to say about it? I've been having some mysterious issues with RHES that I've not been able to pin down. -- Josh Berkus Aglio Database Solutions San Francisco ---(end of broadcast)--- TIP 5: Have you checked our extensive FAQ? http://www.postgresql.org/docs/faqs/FAQ.html
Re: [PERFORM] Query Performance and IOWait
Andrew,
Dell's aren't well known for their disk performance, apparently most of
the perc controllers sold with dell's are actually adaptec controllers.
Also apparently they do not come with the battery required to use the
battery backed up write cache ( In fact according to some Dell won't
even sell the battery to you). Also Dell's monitoring software is quite
a memory hog.
Have you looked at top ?, and also hdparm -Tt /dev/sd?
Dave
Andrew Janian wrote:
Hello All,
I have a setup with a Dell Poweredge 2650 with Red Hat and Postgres 7.4.5 with
a database with about 27GB of data. The table in question has about 35 million
rows.
I am running the following query:
SELECT *
FROM mb_fix_message
WHERE msg_client_order_id IN (
SELECT msg_client_order_id
FROM mb_fix_message
WHERE msg_log_time >= '2004-06-01'
AND msg_log_time < '2004-06-01 13:30:00.000'
AND msg_message_type IN ('D','G')
AND mb_ord_type = '1'
)
AND msg_log_time > '2004-06-01'
AND msg_log_time < '2004-06-01 23:59:59.999'
AND msg_message_type = '8'
AND (mb_raw_text LIKE '%39=1%' OR mb_raw_text LIKE '%39=2%');
with the following plan:
QUERY PLAN
Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
-> Index Scan using mfi_log_time on mb_fix_message (cost=0.00..22231.31
rows=2539 width=526)
Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp without time
zone) AND (msg_log_time < '2004-06-01 23:59:59.999'::timestamp without time zone))
Filter: (((msg_message_type)::text = '8'::text) AND (((mb_raw_text)::text
~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~ '%39=2%'::text)))
-> Index Scan using mfi_client_ordid on mb_fix_message (cost=0.00..445.56
rows=1 width=18)
Index Cond: (("outer".msg_client_order_id)::text =
(mb_fix_message.msg_client_order_id)::text)
Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp without time
zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp without time zone) AND
((msg_message_type)::text = 'D'::text) OR ((msg_message_type)::text = 'G'::text)) AND
((mb_ord_type)::text = '1'::text))
While running, this query produces 100% iowait usage on its processor and takes
a ungodly amount of time (about an hour).
The postgres settings are as follows:
shared_buffers = 32768 # min 16, at least max_connections*2, 8KB each
sort_mem = 262144 # min 64, size in KB
And the /etc/sysctl.conf has:
kernel.shmall = 274235392
kernel.shmmax = 274235392
The system has 4GB of RAM.
I am pretty sure of these settings, but only from my reading of the docs and
others' recommendations online.
Thanks,
Andrew Janian
OMS Development
Scottrade Financial Services
(314) 965-1555 x 1513
Cell: (314) 369-2083
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
--
Dave Cramer
http://www.postgresintl.com
519 939 0336
ICQ#14675561
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
Re: [PERFORM] Query Performance and IOWait
Hello,
What is your statistics target?
What is your effective_cache_size?
Have you tried running the query as a cursor?
Sincerely,
Joshua D. Drake
Andrew Janian wrote:
I have run ANALYZE right before running this query.
I will run EXPLAIN ANALYZE when I can. I started running the query when I sent
the first email and it is still running. Looke like it longer than an hour.
I will post the results of EXPLAIN ANALYZE in a few hours when I get them.
Thanks for all your help,
Andrew
-Original Message-
From: Tom Lane [mailto:[EMAIL PROTECTED]
Sent: Thursday, November 18, 2004 9:40 AM
To: Andrew Janian
Cc: [EMAIL PROTECTED]
Subject: Re: [PERFORM] Query Performance and IOWait
"Andrew Janian" <[EMAIL PROTECTED]> writes:
QUERY PLAN
Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
-> Index Scan using mfi_log_time on mb_fix_message (cost=0.00..22231.31
rows=2539 width=526)
Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp without time
zone) AND (msg_log_time < '2004-06-01 23:59:59.999'::timestamp without time zone))
Filter: (((msg_message_type)::text = '8'::text) AND (((mb_raw_text)::text
~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~ '%39=2%'::text)))
-> Index Scan using mfi_client_ordid on mb_fix_message (cost=0.00..445.56
rows=1 width=18)
Index Cond: (("outer".msg_client_order_id)::text =
(mb_fix_message.msg_client_order_id)::text)
Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp without time
zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp without time zone) AND
((msg_message_type)::text = 'D'::text) OR ((msg_message_type)::text = 'G'::text)) AND
((mb_ord_type)::text = '1'::text))
While running, this query produces 100% iowait usage on its processor and takes a ungodly amount of time (about an hour).
This plan looks fairly reasonable if the rowcount estimates are
accurate. Have you ANALYZEd the table lately? You might need to
bump up the statistics target for the msg_log_time column to improve
the quality of the estimates. It would be useful to see EXPLAIN
ANALYZE results too (yes I know it'll take you an hour to get them...)
regards, tom lane
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
--
Command Prompt, Inc., home of PostgreSQL Replication, and plPHP.
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - [EMAIL PROTECTED] - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL
begin:vcard
fn:Joshua D. Drake
n:Drake;Joshua D.
org:Command Prompt, Inc.
adr:;;PO Box 215;Cascade Locks;Oregon;97014;USA
email;internet:[EMAIL PROTECTED]
title:Consultant
tel;work:503-667-4564
tel;fax:503-210-0334
note:Command Prompt, Inc. is the largest and oldest US based commercial PostgreSQL support provider. We provide the only commercially viable integrated PostgreSQL replication solution, but also custom programming, and support. We authored the book Practical PostgreSQL, the procedural language plPHP, and adding trigger capability to plPerl.
x-mozilla-html:FALSE
url:http://www.commandprompt.com/
version:2.1
end:vcard
---(end of broadcast)---
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match
Re: [PERFORM] Query Performance and IOWait
I have run ANALYZE right before running this query.
I will run EXPLAIN ANALYZE when I can. I started running the query when I sent
the first email and it is still running. Looke like it longer than an hour.
I will post the results of EXPLAIN ANALYZE in a few hours when I get them.
Thanks for all your help,
Andrew
-Original Message-
From: Tom Lane [mailto:[EMAIL PROTECTED]
Sent: Thursday, November 18, 2004 9:40 AM
To: Andrew Janian
Cc: [EMAIL PROTECTED]
Subject: Re: [PERFORM] Query Performance and IOWait
"Andrew Janian" <[EMAIL PROTECTED]> writes:
>
>
>
>QUERY PLAN
> Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
> -> Index Scan using mfi_log_time on mb_fix_message (cost=0.00..22231.31
> rows=2539 width=526)
>Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp without
> time zone) AND (msg_log_time < '2004-06-01 23:59:59.999'::timestamp without
> time zone))
>Filter: (((msg_message_type)::text = '8'::text) AND
> (((mb_raw_text)::text ~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~
> '%39=2%'::text)))
> -> Index Scan using mfi_client_ordid on mb_fix_message (cost=0.00..445.56
> rows=1 width=18)
>Index Cond: (("outer".msg_client_order_id)::text =
> (mb_fix_message.msg_client_order_id)::text)
>Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp without
> time zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp without time
> zone) AND ((msg_message_type)::text = 'D'::text) OR ((msg_message_type)::text
> = 'G'::text)) AND ((mb_ord_type)::text = '1'::text))
> While running, this query produces 100% iowait usage on its processor and
> takes a ungodly amount of time (about an hour).
This plan looks fairly reasonable if the rowcount estimates are
accurate. Have you ANALYZEd the table lately? You might need to
bump up the statistics target for the msg_log_time column to improve
the quality of the estimates. It would be useful to see EXPLAIN
ANALYZE results too (yes I know it'll take you an hour to get them...)
regards, tom lane
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
Re: [PERFORM] Query Performance and IOWait
"Andrew Janian" <[EMAIL PROTECTED]> writes:
>
>
>
>QUERY PLAN
> Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
> -> Index Scan using mfi_log_time on mb_fix_message (cost=0.00..22231.31
> rows=2539 width=526)
>Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp without
> time zone) AND (msg_log_time < '2004-06-01 23:59:59.999'::timestamp without
> time zone))
>Filter: (((msg_message_type)::text = '8'::text) AND
> (((mb_raw_text)::text ~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~
> '%39=2%'::text)))
> -> Index Scan using mfi_client_ordid on mb_fix_message (cost=0.00..445.56
> rows=1 width=18)
>Index Cond: (("outer".msg_client_order_id)::text =
> (mb_fix_message.msg_client_order_id)::text)
>Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp without
> time zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp without time
> zone) AND ((msg_message_type)::text = 'D'::text) OR ((msg_message_type)::text
> = 'G'::text)) AND ((mb_ord_type)::text = '1'::text))
> While running, this query produces 100% iowait usage on its processor and
> takes a ungodly amount of time (about an hour).
This plan looks fairly reasonable if the rowcount estimates are
accurate. Have you ANALYZEd the table lately? You might need to
bump up the statistics target for the msg_log_time column to improve
the quality of the estimates. It would be useful to see EXPLAIN
ANALYZE results too (yes I know it'll take you an hour to get them...)
regards, tom lane
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faqs/FAQ.html
Re: [PERFORM] Query Performance and IOWait
Andrew,
What version of Redhat are you running? We have found running Enterprise
Update 3 kernel kills our Dell boxes with IOWait, both NFS and local disk
traffic. Update 2 kernel does not seem to have the issue, and we are in the
process of trying Update 4 beta to see if it is better.
Woody
iGLASS Networks
www.iglass.net
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Andrew Janian
Sent: Thursday, November 18, 2004 9:02 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: [PERFORM] Query Performance and IOWait
Actually, unfortunately, that won't work. The subquery gets a list of
message IDs and then the outer query gets the responses to those messages.
Also, I dumped this data and imported it all to ms sql server and then ran
it there. The query ran in 2s.
Andrew
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]
Sent: Thursday, November 18, 2004 7:57 AM
To: Andrew Janian
Cc: [EMAIL PROTECTED];
[EMAIL PROTECTED]
Subject: Re: [PERFORM] Query Performance and IOWait
Andrew,
It seems that you could combine the subquery's WHERE clause with the main
query's to produce a simpler query, i.e. one without a subquery.
Rick
"Andrew Janian"
<[EMAIL PROTECTED]>To:
<[EMAIL PROTECTED]>
Sent by: cc:
[EMAIL PROTECTED]Subject: [PERFORM]
Query Performance and IOWait
tgresql.org
11/18/2004 08:42 AM
Hello All,
I have a setup with a Dell Poweredge 2650 with Red Hat and Postgres 7.4.5
with a database with about 27GB of data. The table in question has about
35 million rows.
I am running the following query:
SELECT *
FROM mb_fix_message
WHERE msg_client_order_id IN (
SELECT msg_client_order_id
FROM mb_fix_message
WHERE msg_log_time >= '2004-06-01'
AND msg_log_time < '2004-06-01 13:30:00.000'
AND msg_message_type IN ('D','G')
AND mb_ord_type = '1'
)
AND msg_log_time > '2004-06-01'
AND msg_log_time < '2004-06-01 23:59:59.999'
AND msg_message_type = '8'
AND (mb_raw_text LIKE '%39=1%' OR mb_raw_text LIKE '%39=2%');
with the following plan:
QUERY PLAN
Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
-> Index Scan using mfi_log_time on mb_fix_message (cost=0.00..22231.31
rows=2539 width=526)
Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp without
time zone) AND (msg_log_time < '2004-06-01 23:59:59.999'::timestamp without
time zone))
Filter: (((msg_message_type)::text = '8'::text) AND
(((mb_raw_text)::text ~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~
'%39=2%'::text)))
-> Index Scan using mfi_client_ordid on mb_fix_message
(cost=0.00..445.56 rows=1 width=18)
Index Cond: (("outer".msg_client_order_id)::text =
(mb_fix_message.msg_client_order_id)::text)
Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp without
time zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp without time
zone) AND ((msg_message_type)::text = 'D'::text) OR
((msg_message_type)::text = 'G'::text)) AND ((mb_ord_type)::text =
'1'::text))
While running, this query produces 100% iowait usage on its processor and
takes a ungodly amount of time (about an hour).
The postgres settings are as follows:
shared_buffers = 32768 # min 16, at least max_connections*2, 8KB
each
sort_mem = 262144 # min 64, size in KB
And the /etc/sysctl.conf has:
kernel.shmall = 274235392
kernel.shmmax = 274235392
The system has 4GB of RAM.
I am pretty sure of these settings, but only from my reading of the docs and
others' recommendations online.
Thanks,
Andrew Janian
OMS Development
Scottrade Financial Services
(314) 965-1555 x 1513
Cell: (314) 369-2083
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that your
message can get through to the mailing list cleanly
Re: [PERFORM] Query Performance and IOWait
Actually, unfortunately, that won't work. The subquery gets a list of message
IDs and then the outer query gets the responses to those messages.
Also, I dumped this data and imported it all to ms sql server and then ran it
there. The query ran in 2s.
Andrew
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]
Sent: Thursday, November 18, 2004 7:57 AM
To: Andrew Janian
Cc: [EMAIL PROTECTED];
[EMAIL PROTECTED]
Subject: Re: [PERFORM] Query Performance and IOWait
Andrew,
It seems that you could combine the subquery's WHERE clause with the main
query's to produce a simpler query, i.e. one without a subquery.
Rick
"Andrew Janian"
<[EMAIL PROTECTED]>To:
<[EMAIL PROTECTED]>
Sent by: cc:
[EMAIL PROTECTED]Subject: [PERFORM] Query
Performance and IOWait
tgresql.org
11/18/2004 08:42 AM
Hello All,
I have a setup with a Dell Poweredge 2650 with Red Hat and Postgres 7.4.5
with a database with about 27GB of data. The table in question has about
35 million rows.
I am running the following query:
SELECT *
FROM mb_fix_message
WHERE msg_client_order_id IN (
SELECT msg_client_order_id
FROM mb_fix_message
WHERE msg_log_time >= '2004-06-01'
AND msg_log_time < '2004-06-01 13:30:00.000'
AND msg_message_type IN ('D','G')
AND mb_ord_type = '1'
)
AND msg_log_time > '2004-06-01'
AND msg_log_time < '2004-06-01 23:59:59.999'
AND msg_message_type = '8'
AND (mb_raw_text LIKE '%39=1%' OR mb_raw_text LIKE '%39=2%');
with the following plan:
QUERY PLAN
Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
-> Index Scan using mfi_log_time on mb_fix_message (cost=0.00..22231.31
rows=2539 width=526)
Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp
without time zone) AND (msg_log_time < '2004-06-01 23:59:59.999'::timestamp
without time zone))
Filter: (((msg_message_type)::text = '8'::text) AND
(((mb_raw_text)::text ~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~
'%39=2%'::text)))
-> Index Scan using mfi_client_ordid on mb_fix_message
(cost=0.00..445.56 rows=1 width=18)
Index Cond: (("outer".msg_client_order_id)::text =
(mb_fix_message.msg_client_order_id)::text)
Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp without
time zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp without
time zone) AND ((msg_message_type)::text = 'D'::text) OR
((msg_message_type)::text = 'G'::text)) AND ((mb_ord_type)::text =
'1'::text))
While running, this query produces 100% iowait usage on its processor and
takes a ungodly amount of time (about an hour).
The postgres settings are as follows:
shared_buffers = 32768 # min 16, at least max_connections*2, 8KB
each
sort_mem = 262144 # min 64, size in KB
And the /etc/sysctl.conf has:
kernel.shmall = 274235392
kernel.shmmax = 274235392
The system has 4GB of RAM.
I am pretty sure of these settings, but only from my reading of the docs
and others' recommendations online.
Thanks,
Andrew Janian
OMS Development
Scottrade Financial Services
(314) 965-1555 x 1513
Cell: (314) 369-2083
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
Re: [PERFORM] Query Performance and IOWait
Andrew,
It seems that you could combine the subquery's WHERE clause with the main
query's to produce a simpler query, i.e. one without a subquery.
Rick
"Andrew Janian"
<[EMAIL PROTECTED]>To:
<[EMAIL PROTECTED]>
Sent by: cc:
[EMAIL PROTECTED] Subject: [PERFORM] Query
Performance and IOWait
tgresql.org
11/18/2004 08:42 AM
Hello All,
I have a setup with a Dell Poweredge 2650 with Red Hat and Postgres 7.4.5
with a database with about 27GB of data. The table in question has about
35 million rows.
I am running the following query:
SELECT *
FROM mb_fix_message
WHERE msg_client_order_id IN (
SELECT msg_client_order_id
FROM mb_fix_message
WHERE msg_log_time >= '2004-06-01'
AND msg_log_time < '2004-06-01 13:30:00.000'
AND msg_message_type IN ('D','G')
AND mb_ord_type = '1'
)
AND msg_log_time > '2004-06-01'
AND msg_log_time < '2004-06-01 23:59:59.999'
AND msg_message_type = '8'
AND (mb_raw_text LIKE '%39=1%' OR mb_raw_text LIKE '%39=2%');
with the following plan:
QUERY PLAN
Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
-> Index Scan using mfi_log_time on mb_fix_message (cost=0.00..22231.31
rows=2539 width=526)
Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp
without time zone) AND (msg_log_time < '2004-06-01 23:59:59.999'::timestamp
without time zone))
Filter: (((msg_message_type)::text = '8'::text) AND
(((mb_raw_text)::text ~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~
'%39=2%'::text)))
-> Index Scan using mfi_client_ordid on mb_fix_message
(cost=0.00..445.56 rows=1 width=18)
Index Cond: (("outer".msg_client_order_id)::text =
(mb_fix_message.msg_client_order_id)::text)
Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp without
time zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp without
time zone) AND ((msg_message_type)::text = 'D'::text) OR
((msg_message_type)::text = 'G'::text)) AND ((mb_ord_type)::text =
'1'::text))
While running, this query produces 100% iowait usage on its processor and
takes a ungodly amount of time (about an hour).
The postgres settings are as follows:
shared_buffers = 32768 # min 16, at least max_connections*2, 8KB
each
sort_mem = 262144 # min 64, size in KB
And the /etc/sysctl.conf has:
kernel.shmall = 274235392
kernel.shmmax = 274235392
The system has 4GB of RAM.
I am pretty sure of these settings, but only from my reading of the docs
and others' recommendations online.
Thanks,
Andrew Janian
OMS Development
Scottrade Financial Services
(314) 965-1555 x 1513
Cell: (314) 369-2083
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
---(end of broadcast)---
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to [EMAIL PROTECTED])
[PERFORM] Query Performance and IOWait
Hello All,
I have a setup with a Dell Poweredge 2650 with Red Hat and Postgres 7.4.5 with
a database with about 27GB of data. The table in question has about 35 million
rows.
I am running the following query:
SELECT *
FROM mb_fix_message
WHERE msg_client_order_id IN (
SELECT msg_client_order_id
FROM mb_fix_message
WHERE msg_log_time >= '2004-06-01'
AND msg_log_time < '2004-06-01 13:30:00.000'
AND msg_message_type IN ('D','G')
AND mb_ord_type = '1'
)
AND msg_log_time > '2004-06-01'
AND msg_log_time < '2004-06-01 23:59:59.999'
AND msg_message_type = '8'
AND (mb_raw_text LIKE '%39=1%' OR mb_raw_text LIKE '%39=2%');
with the following plan:
QUERY PLAN
Nested Loop IN Join (cost=0.00..34047.29 rows=1 width=526)
-> Index Scan using mfi_log_time on mb_fix_message (cost=0.00..22231.31
rows=2539 width=526)
Index Cond: ((msg_log_time > '2004-06-01 00:00:00'::timestamp without
time zone) AND (msg_log_time < '2004-06-01 23:59:59.999'::timestamp without
time zone))
Filter: (((msg_message_type)::text = '8'::text) AND
(((mb_raw_text)::text ~~ '%39=1%'::text) OR ((mb_raw_text)::text ~~
'%39=2%'::text)))
-> Index Scan using mfi_client_ordid on mb_fix_message (cost=0.00..445.56
rows=1 width=18)
Index Cond: (("outer".msg_client_order_id)::text =
(mb_fix_message.msg_client_order_id)::text)
Filter: ((msg_log_time >= '2004-06-01 00:00:00'::timestamp without time
zone) AND (msg_log_time < '2004-06-01 13:30:00'::timestamp without time zone)
AND ((msg_message_type)::text = 'D'::text) OR ((msg_message_type)::text =
'G'::text)) AND ((mb_ord_type)::text = '1'::text))
While running, this query produces 100% iowait usage on its processor and takes
a ungodly amount of time (about an hour).
The postgres settings are as follows:
shared_buffers = 32768 # min 16, at least max_connections*2, 8KB each
sort_mem = 262144 # min 64, size in KB
And the /etc/sysctl.conf has:
kernel.shmall = 274235392
kernel.shmmax = 274235392
The system has 4GB of RAM.
I am pretty sure of these settings, but only from my reading of the docs and
others' recommendations online.
Thanks,
Andrew Janian
OMS Development
Scottrade Financial Services
(314) 965-1555 x 1513
Cell: (314) 369-2083
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
