second
it's "?column10?" - are you sure the query was identical in each case.
I'm guessing the unidentified column in query 2 is the reason for the
sort a couple of lines below it, which seems to take up a large chunk of
time.
--
Richard Huxton
Archonet Ltd
-
hered over long periods of time.
Plenty of others have discussed the technical reasons why you are seeing
these connection issues. If you find it difficult to change your way of
working, you might find the pgpool connection-pooling project useful:
http://pgpool.projects.postgresql.org/
HTH
--
Ri
Partitioning on a few very large Tables.
I believe these are being worked on at the moment. You might want to
search the archives of the hackers mailing list to see if the plans will
suit your needs.
--
Richard Huxton
Archonet Ltd
---(end of broadcast
can give me a Tip or Hint where in can
some usefull Information about it!
Look into having a separate server (process or actual hardware) to
handle requests for static text and images. Keep the Java server for
actually processing data.
--
Richard Huxton
Archonet Ltd
if disabling
seq-scan or the relevant join-type does anything (I'm not sure it will)
2. Try against 8.0 - there may be some improvement there.
Other people on this list have experience on larger systems than me, so
they may be able to help more.
--
Richard Huxton
Archonet Ltd
--
e talking about?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 6: Have you searched our list archives?
http://archives.postgresql.org
this index) not planner statistics. You're right - it would be foolish
to throw away planner stats.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
Yann Michel wrote:
Hi,
On Thu, Jun 09, 2005 at 02:11:22PM +0100, Richard Huxton wrote:
To my question: I found the parameter "stats_reset_on_server_start"
which is set to true by default. Why did you choose this (and not false)
and what are the impacts of changeing it to false?
ively rendering
the server unusable, because of the exclusive lock.
You can vacuum full a single table - shouldn't take an hour for just the
one table. Unless your disk I/O is *constantly* running flat-out.
--
Richard Huxton
Archonet Ltd
---(end of
ld PG update 21 indexes ?
There's only one index affected !
No - all 21 are affected. MVCC creates a new row on disk.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 3: if posting/reading through Usenet, please send an appropria
want what is known as a "materialised view" which is
basically a summary table that is kept up to date by triggers. You query
the table instead of actually recalculating every time. Perhaps google
for "postgresql materialized view" (you might want a "z" or &quo
w
from your cursor with/without the ORDER BY. Without should be quicker.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
aracter-set). Locale is something like "C" or
"en_US" or "fr_FR".
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
/www.postgresql.org/community/international
PostgreSQL is licensed under the BSD licence, which means you can freely
download or deploy it in a commercial setting if you desire.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
ROM upload JOIN main_table ON u1=f1 AND u2=f2 AND u3=f3;
SELECT * FROM upload JOIN main_table ON u1=f1 AND u2=f2 AND u3=f4;
Are you saying that deleting these rows and then inserting takes too long?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 6: explain analyze is your friend
nverted them to left-joins:
INSERT INTO ...
SELECT * FROM upload LEFT JOIN ... WHERE f3 IS NULL
UNION
SELECT * FROM upload LEFT JOIN ... WHERE f4 IS NULL
The UNION will remove duplicates for you, but this might turn out to be
slower than two separate queries.
--
Richard Huxton
Archonet Ltd
-
ave multiple users running a mix of
multi-table joins and updates then PG will have a chance to stretch its
legs.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
a query just on WHERE b.r=516081780 and see if it gets the estimate
right for that.
If it's a common query, it might be worth an index on (r,c)
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 1: if posting/reading through Use
still the
problems exists.
You don't say what the nature of the problem with dbmirror is. Are they
saturating their bandwidth? Are one or both servers unable to keep pace
with the updates?
--
Richard Huxton
Archonet Ltd
---(end of broa
ng report query
isn't that different from a vacuum.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
olumn, when you start a new
"heavy" query, insert that query's cost, then sleep
SUM(estimated_cost)/100 secs or something. When the query ends, delete
the cost-row.
Hmm - actually rather than dividing by 100, perhaps make it a tunable value.
--
Richard Huxton
Archonet Ltd
--
external storage array with a hardware
RAID controller is used, the normal advice to separate the data from the
pg_xlog seems to come unstuck, or are we missing something?
Well, I think the advice then is actually "get 2 external arrays..."
--
Richard Huxton
Archonet Ltd
ro" looks odd though,
especially since it expects 24339 matches (out of 2.8 million rows -
that should favour an index).
Of course, I've not considered the context of the rest of the query, but
I'd expect the index to be used.
Do you have any unusual config settings?
-
or some examples.
http://www.postgresql.org/docs/8.0/static/sql-createindex.html
http://www.postgresql.org/docs/8.0/static/indexes-opclass.html
HTH
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
dding another reporting user, or importing
another logfile?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL PROTECTED] so that
manuals for 7.4.x
and 8.0.x and see what it says about memory and plpgsql.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to [EMAIL P
il at this particular query.
http://www.powerpostgresql.com/PerfList
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
s has a system performance monitoring
tool that can show CPU/Memory/Disk IO, and *nix tools have vmstat or iostat.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
memory from other nodes in the same query
instead of the swap?
I don't know of any clustered version of PG that can spread queries over
multiple machines. Can I ask what you are using?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)--
ble recently. So - everything will start
to get slower.
So - for a small, rapidly updated table make sure you vacuum a lot
(perhaps as often as once a minute). Or, run autovacuum and let it cope.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)--
licts.
If the system is otherwise idle, it might be worthwhile to compare
before and after values of pg_stat* (user-tables and user-indexes).
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
the tableA in "UPDATE" is the
same as that in your "FROM" clause. If so, why are you SETting a.val1?
If not, you've probably got an unconstrained join.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 9: I
stamptz value "infinity", but only for timestamps and not
for dates. I'd probably cheat a little and use an end date of
'-12-31' or similar to simulate "infinity". Then your test is simply:
WHERE
...
AND (rol.begin <= CURRENT_DATE AND rol.end >= CURRENT_DATE)
That should estimate simply enough.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
TISTICS 998;
So, when I do something like
SELECT alias_id FROM ma_data GROUP BY alias_id
Why are you using GROUP BY without any aggregate functions?
What happens if you use something like
SELECT DISTINCT alias_id FROM ma_data;
--
Richard Huxton
Archonet Ltd
--
tgresql.com/PerfList
or
http://www.varlena.com/varlena/GeneralBits/Tidbits/perf.html
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
orrect your configuration settings so PG estimates the cost of an index
query correctly and all should be well.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subs
ith concurrency issues
yourself.
Storing the values in a table and having cached access to them during
the session is probably your best bet.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
ore without a clear example of your requirements.
Even if you choose to alter your design for performance reasons, you
should make sure you run tests with realistic workloads and hardware.
But first, trust PG to do its job and design your database according to
the problem requirements.
R, Rajesh (STSD) wrote:
Thanks.
I've already understood that
I need to post it in another list.
Sorry for wasting your precious time.
No time wasted. It was a perfectly reasonable question, just to the
wrong lists.
--
Richard Huxton
Archonet Ltd
---(e
ALÝ ÇELÝK wrote:
FreeBSD or Linux , which system has better performance for PostgreSQL
Depends on the underlying hardware and your experience. I'd recommend
going with whichever you are more familiar, so long as it will support
the hardware you need to buy.
--
Richard Huxton
Arc
me),
KEY Date (Date)
The word "KEY" isn't valid here either - are you trying to define an
index? If so, see the "CREATE INDEX" section of the SQL reference.
http://www.postgresql.org/docs/8.0/static/sql-commands.html
If you reply to this message
to do it won't place too much load on your
system.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
ely attractive.
It might be useful if there was a way to trap certain queries and
rewrite/replace them. That way more complex queries could be
transparently redirected to a summary table etc. I'm guessing that the
overhead to check every query would quickly destroy any gains thou
Nörder-Tuitje wrote:
Hello,
I have a strange effect on upcoming structure :
People will be wanting the output of EXPLAIN ANALYSE on that query.
They'll also ask whether you've VACUUMed, ANALYSEd and configured your
postgresql.conf correctly.
--
Richard Huxton
Ar
workaround. Maybe the io-costs are
configured to cheap.
Possibly - the explain analyse will show you.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 1: if posting/reading through Usenet, please send an appropriate
subscribe
ke sense. The output from EXPLAIN ANALYSE would show us
whether that estimate is correct - is it?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
gap there? Is it not 2,125,270 rows
matching which would suggest PG is getting it more right than wrong.
Try issuing "SET enable_seqscan=false" before running the explain
analyse - that will force the planner to use any indexes it can find and
should show us whether the index would help.
--
plicaton.
6. If your lead table is updated only rarely, you could try a CLUSTER
on the table by mailing_id - that should speed the scan. Read the manual
for the cluster command first though.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)-
"ORDER BY thread_id,message_id" and see if that nudges things
your way.
2. Keep #1 and try replacing the index on (thread_id) with
(thread_id,message_id)
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
you have a specific problem with
the index bloating. Are you seeing this?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining c
, nested sql-statement ? I especially think exection
planner-wise.
The planner tries to push conditions "inside" views where it can. It's
not perfect though, and if you're writing a big query by hand you might
be able to do better than it.
In short, I'd test if you
capabilites for someone like me (that is very used to
write reasonably fast and complex sql, can read c-code, but does not
really want to dig into the source code)
There is some stuff in the "Internals" section of the manuals and it
might be worth rummaging aro
Jan Wieck wrote:
On 10/27/2005 7:29 AM, Richard Huxton wrote:
Svenne Krap wrote:
What do you mean exactly but "pushing conditions inside" ?
If I have something like "SELECT * FROM complicated_view WHERE foo =
7" then the planner can look "inside" complica
il, you seem to suspect your configuration
needs some work. Once you are happy that your settings in general are
good, you can override some by issuing set statements before your query.
For example:
SET work_mem = 1;
might well improve example #2 where you had a hash.
--
Richard Hu
use the index? Or do you use some tricks in
this scenario? Thanks for your suggestions.
Try "UNION ALL", since UNION is defined as removing duplicates, which
probably accounts for the sort.
--
Richard Huxton
Archonet Ltd
---(end of
lue.
Then, we'll need to look at your other tuning settings. Have you made
any changes to your postgresql.conf settings, in particular those
mentioned here:
http://www.powerpostgresql.com/PerfList
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 6: explain analyze is your friend
RC. That should let PG do its sorting in
larger chunks.
Also, if your most common access pattern is ordered via tlid look into
clustering the table on that.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 1: if posting/reading
ed along with the table), but only for the first column.
1. Are all of test_a/b/c/d/e bigint rather than int?
2. Have you tried explicitly casting your query parameters?
...WHERE test_a=123::bigint AND test_b=456::bigint...
--
Richard Huxton
Archonet Ltd
---(end
eone's going
to ask what filesystem this is (ext2/xfs/etc). And probably to see the
strace too.
Hmm - the only things I can think to check:
Do vmstat/iostat show any unusual activity?
Are your system logs on these disks too?
Could it be the journalling on the fs clashing with the WAL?
--
me to complete.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
nteresting posts.
Well, I'm prepared to swap Luke *TWO* $1000 systems for one $80,000
system if he's got one going :-)
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will i
r a large copy in case the
statistics about how many different values are present changes.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
newer drivers than Win2k.
What do the MS performance-charts show is happening? Specifically, CPU
and disk I/O.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 6: explain analyze is your friend
our inserts so
that all the inserts for table1 go together, then all the inserts for
table2 go together and so on. This should help with the fragmentation by
making sure the files get extended in larger chunks.
Are you sure it's not possible to spend 15 mins offline to solve this?
--
Richard Hu
that is the same
row?
However, even if you removed the condition on origem, I don't think the
planner will notice that it can eliminate the join. It's just too
unusual a case for the planner to have a rule for it.
I might be wrong about the planner - I'm just another user. One of the
Edison Azzi wrote:
Richard Huxton escreveu:
However, even if you removed the condition on origem, I don't think
the planner will notice that it can eliminate the join. It's just too
unusual a case for the planner to have a rule for it.
I might be wrong about the planner - I'
ry and then
re-plan the outer query based on those results. Of course, someone might
pop up and tell me I'm wrong now...
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your des
ught they'd fixed it for XP.
If not that, could some firewall/security system be slowing network traffic?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
which processes?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
> ( select max(a.end_nlogid) from
activity_log_import_history a)
and dtCreateDate < '2006-12-18 9:10'
Can you post the EXPLAIN ANALYSE for this one please? That'll show us
exactly what it's doing.
--
Richard Huxton
Archonet Ltd
--
e returned into the query above.
Then, try the same but with a prepared query. If it's down to nlogid
estimates then the first should be fast and the second slow.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 1: if posting/readi
m max()). Then it
could plan the query as above. I'm not sure how tricky that is to do though.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 7: You can help support the PostgreSQL project by donating at
http://www.postgresql.org/about/donate
k_mem, maintenance_work_mem
Also consider temp_buffers and random_page_cost.
A lot will depend on how much of the data you handle ends up cached.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 6: explain analyze is your friend
you can say
is that random_page_cost should be 1 if all your database fits in RAM.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
both
remotely and locally, \d responds immediately. Could the issue be with
the client programs somehow?
Couldn't be some DNS problems that only affect the 8.2 client I suppose?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
he indexes up-to-date too.
I find it's easier to spot where to put indexes during testing. It's
easy to add indexes where they're never used.
HTH
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
l tables?
3. What about dependencies on user-defined types, functions, etc?
Is there not some gain from just a "standard" partitioning of pg_class
into: (system-objects, user-objects, temp-objects)? I'd expect them to
form a hierarchy of change+vacuum rates (if you see what I me
ime < '2007-01-01'::date;
CREATE INDEX event_time_state_idx ON events (event_time, state);
You'll want to replace the index/update the query once a year/month etc.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
stats tables useful too:
http://www.postgresql.org/docs/8.2/static/monitoring-stats.html
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
t might be that one of your java
libraries is clearing its cache though, causing it to issue more queries.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
to compact
the table.
If you are running a normal vacuum often enough then the size of the
database should remain about the same (unless you insert more rows of
course).
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 5: don'
s?
If there are too many, an index isn't going to help.
Can you share the EXPLAIN ANALYSE output? You might want to try
increasing work_mem for this one query to speed any sorting.
How often is the table updated? Clustering might buy you some
improvements (but not
Igor Lobanov wrote:
Richard Huxton wrote:
I have rather large table with about 5 millions of rows and a dozen
of columns. Let's suppose that columns are named 'a', 'b', 'c' etc. I
need to query distinct pairs of ('a';'b') from thi
in the sort, not in the SeqScan.
Increase your work_mem.
Well, even if the sort was instant it's only going to get him down to
20secs.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 6: explain analyze is your friend
forms better, but for
queries on a common name, the dual-subselect query performs better.
Difficult to say much without seeing the full explain analyse. Did the
row estimates look reasonable?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
25 for names should be a good choice.
You could try partial indexes for those cases where you have
particularly common values of name/port:
CREATE INDEX idx1 ON host_events (ip) WHERE port=80;
--
Richard Huxton
Archonet Ltd
---(end of broadcast)
1 rows=52
loops=1)
-> Sort (cost=9238..9288 rows=1965 width=72) (actual time=61..61
rows=63 loops=1)
OK, so what do the plans look like for port=80 or something larger like
that?
Then try adding an index to the various host/network_events tables
CREATE INDEX ... ON ... (ip) WHERE port=80;
--
alloc: *** vm_allocate(size=8421376) failed (error code=3)
Is this actually in psql - the client code rather than the backend?
Could it be that its allocating memory for its 7million result rows and
running out of space for your user account?
--
Richard Huxton
Archone
some postgresql-client or postgresql-dev rpm
files?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining column's datatyp
~ 700MB.
If it is the backend, you'll need to give some of the tuning parameters
you're using, since it works here on my much smaller dev server (1GB RAM
and plenty of other stuff using it).
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
t know this.
If this is a common query, you could try an index on zipcode - that
might cut down the other side.
However, I have to ask why you're using a left-join? Do you really have
rows in client_contact without a matching cli_id in client_company?
--
Richard Huxt
results rather than buffering them - shouldn't be too
difficult.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
r?
Does it do this with psql too?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 6: explain analyze is your friend
oops=1)"
If you look at the "actual time" it's completing very quickly indeed. So
- it must be something to do with either:
1. Fetching/formatting the data
2. Transferring the data to the client.
What happens if you only select half the rows? Does the time to run the
selec
ntire database will want larger numbers
than 100 connections running many small queries.
HTH
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
management code wasn't great with
large shared_buffers values too.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose an index scan if your joining colum
ow then it's your network setup. If it's much faster then it's
something to do with the bytea type.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
postgres"
3. What do your logfiles say?
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
Note - try to cc: the mailing list, I don't always read this inbox
Gauri Kanekar wrote:
On 2/26/07, Richard Huxton wrote:
Gauri Kanekar wrote:
> Hi List,
>
> Machine was down due to some hardware problem.
>
> After then when i issue this command /usr/local/pgsql/bin/psq
Arjen.
Thanks for the notice on this.
--
Richard Huxton
Archonet Ltd
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
1 - 100 of 550 matches
Mail list logo