into RAM then they are in-memory as
long as they're actively being used.
Hashtables and GDBM, as far as I know, are only useful for key->value
lookups. However, for this they are *fast*. If you can figure out a way
to make them work I'll bet things speed up.
--
Matthew Nuzum
new
45 min to a little over an hour but decreased
the memory usage to something like 45MB (vs dozens or hundreds of MB per
hashtable)
--
Matthew Nuzum
newz2000 on freenode
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
ing that can grow over time as our needs change. I don't want to
> buy a server only to find out later that it cannot meet our needs with
> future database projects. I have to balance a limited budget, room for
> future performance growth, and current system requirements.
)
Pricing is tight-lipped, but searching shows $1.85 /GB. That's close
to $500,000 for 250GB. One report says a person paid $219,000 for 32GB
and 1TB costs "well over $1,000,000."
But they "guarantee the performance."
Too rich for me.
--
Matthew Nuzum
www.bearfrui
varlena/GeneralBits/Tidbits/index.php
Notice there's a section on performance tips.
Also, this list works because volunteers who have knowledge and free
time choose to help when they can. If you really need answers ASAP,
there are a few organizations who provide paid support.
at the explain analyze output of the query from pg 7.3,
figure out why the plan is bad and tweak your query to get optimum
performance.
Yes, I agree with the other statements that say, "upgrade to 7.4 or
8.x if you can" but if you can't, then you can still work on it.
--
Matthew
of linux, kernel and all.
>
> No, linux vserver is equivalent to a jail - and they work superbly imho.
> developer.pgadmin.org is just one such VM that I run.
>
> http://www.linux-vserver.org/
>
> Regards, Dave.
I can confirm this. I've been using linux-vserver for years. It
On 3/6/06, Marc G. Fournier <[EMAIL PROTECTED]> wrote:
> On Mon, 6 Mar 2006, Matthew Nuzum wrote:
> > My problem with running PG inside of a VPS was that the VPS used a
> > virtual filesystem... basically, a single file that had been formatted
> > and loop mounted so th
at works on pretty
much any linux OS.
Try it out, tinker with the values and that way you won't have to
guess when making your purchase decission.
[1] http://www.colinux.org/ Coperative Linux
[2] http://linux-vserver.org/ Linux-vserver project
--
Matthew Nuzum
www.bearfruit.org
---
d probably want to start with the GDB technique unless
you have a ton of available ram.
You might interpret this as being a knock against PostgreSQL since I
pulled the data out of the db, but it's not; You'd be hard pressed to
find anything as fast as the in-memory hashtable or th
oss *twice* this year by using SMART
hard drive monitoring software.
I can't tell you how good it feels to replace a drive that is about to
die, as compared to restoring data because a drive died.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)
ideology so that a server should be replaced
after 3 years, where before I aimed for 5.
It seems to me that the least reliable components in servers these
days are the fans.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 9: In
ng like this
would definately peg your disk i/o.
Throwing more hardware at your problem will definately help, but I'm a
performance freak and I like to optimize everything to the max.
*Sometimes* you can get drastic improvements without adding any
hardware. I have seen some truly miraculu
ow.
I would suggest posting the explain analyze output for one of your
slow updates. I'll bet it is much more revealing and takes out a lot
of the guesswork.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
essage is being sent to the list to serve as a
warning to other data warehouse admins that when you reach your
capacity, the downward spiral happens rather quickly.
Crud... Outlook just froze while composing the PHB memo. I've been
working on that for an hour. What a bad day.
--
Matthew Nuzum
ww
> /tmp/warn.txt
echo >> /tmp/warn.txt
top -bn 1 >> /tmp/warn.txt
echo >> /tmp/warn.txt
fi
NOW=`date`
CPU_LOAD=`cat /proc/loadavg | cut --delimiter=" " -f 1,2,3
--output-delimiter=\|`
echo -e $NOW\|$CPU_LOAD\|$DB_LOAD >> ~/LOAD_MONITOR.LOG
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
iggest wins will come from RAM, disk and network investments.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 4: Have you searched our list archives?
http://archives.postgresql.org
to lose any data then you need to consider it imperitive to use some
type of RAID setup (not RAID 0) and to achieve great performance
you'll want more than 2 drives.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
s of the app developer,
which results in a better db schema, optimized queries and generally
*thinking* about the performance of the code. I personally feel that
to be a very rewarding aspect of my job. (As a hobby I program
microntrollers that run at 4MHz and have only 256 bytes of RAM, so
that
r to throw more hardware)
If you have particular queries that are too slow, posting the explain
analyze for each on the list should garner some help.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 3: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faq
ups from about 15 minutes to 30,
bringing the whole process to about 45 minutes.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 5: don't forget to increase your free space map settings
e +++ experience optimzing slow queries on big databases. So
queries now that run in 20 ms but slow down to 7 seconds when your
tables grow will likely benefit from optimizing.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
fair to python, I can write the
dictionary lookup code in 25% (aprox) fewer lines than similar hash
table code in C#. I could go on but I think I'm starting to get off
topic.)
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 4: Don't 'kill -9' the postmaster
s. I then used C# and in memory hash
tables to drop the time to 2 hours, but I couldn't get mono installed
on some of my older servers. Python proved the fastest and I can
process 24 hours worth of logs in about 15 minutes. Common reports run
in < 1 sec and custom reports run in < 15 se
3. Create a cron entry to run the command once a day, it might look like this:
0 6 * * * /usr/bin/psql dbname < /home/admin/create_mat_view.txt
or maybe like this:
0 6 * * * "C:\Program Files\PostgreSQL\8.0\psql.exe" dbname <
"C:\create_mat_view.txt"
I hope this helps
erely add new records to the existing table, but if
your data changes, the drop table technique can be faster than doing a
delete or update.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 1: subscribe and unsubscribe commands go to [EMAIL PROTECTED]
d to serve a request on a moderately loaded
server.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 8: explain analyze is your friend
double
the RAM. I don't know anything about your application though so use
the guidlines above.
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
On 4/26/05, Steinar H. Gunderson <[EMAIL PROTECTED]> wrote:
> On Tue, Apr 26, 2005 at 03:16:57PM -0500, Matthew Nuzum wrote:
> > Seq Scan on usage_access (cost=0.00..1183396.40 rows=12713851
> > width=116) (actual time=481796.22..481839.43 rows=3343 loops=1)
>
> T
On 4/26/05, Steinar H. Gunderson <[EMAIL PROTECTED]> wrote:
> On Tue, Apr 26, 2005 at 03:16:57PM -0500, Matthew Nuzum wrote:
> > Seq Scan on usage_access (cost=0.00..1183396.40 rows=12713851
> > width=116) (actual time=481796.22..481839.43 rows=3343 loops=1)
>
> T
-> Index Scan Backward using usage_access_atime on
usage_access (cost=0.00..22657796.18 rows=38141552 width=8) (actual
time=0.40..0.41 rows=2 loops=1)
Total runtime: 481842.47 msec
It doesn't look like this will help at all.
This table is primarily append, however I j
Would the increased data density of the higher capacity drive be of
greater benefit than the faster spindle speed of drive A?
--
Matthew Nuzum
www.bearfruit.org
---(end of broadcast)---
TIP 7: don't forget to increase your free space map settings
nsiderate group of people as these on the
performance list.
--
Matthew Nuzum <[EMAIL PROTECTED]>
www.followers.net - Makers of "Elite Content Management System"
View samples of Elite CMS in action by visiting
http://www.followers.net/portfolio/
--
the increased data density of the higher capacity drive be of
greater benefit than the faster spindle speed of drive A?
--
Matthew Nuzum <[EMAIL PROTECTED]>
www.followers.net - Makers of “Elite Content Management System”
View samples of Elite CMS in action by visiting
h
;
> Joshua D. Drake
>
Thanks, I'm looking at your product and will contact you off list for more
details soon.
Out of curiosity, does batch mode produce a lighter load? Live updating will
provide maximum data security, and I'm most interested in how it affects the
server.
--
Mat
ostgreSQL 7.3 and 7.4, but I'm currently using
7.3.
I'm eager to hear your thoughts and experiences,
--
Matthew Nuzum <[EMAIL PROTECTED]>
www.followers.net - Makers of "Elite Content Management System"
Earn a commission of $100 - $750 by recommending Elite CMS. Visi
query my server load rose to unacceptable levels.
FWIW, the explain was run from psql running on the db server, the test query
the other day was run from one of the webservers. Should I run this on the
db server to minimize load?
--
Matthew Nuzum <[EMAIL PROTECTED]>
www.followers.net - Makers
35366.31 rows=35678383 loops=1)
Sort Key: accountid, sessionid
-> Seq Scan on usage_access (cost=0.00..1018901.84
rows=35678384 width=28) (actual time=8.13..416580.35 rows=35678383 loops=1)
Total runtime: 12625498.84 msec
(7 rows)
--
Matthe
ut
10,000,000 groups. PostgreSQL version is 7.3.2 and the sort_mem is at the
default setting.
(I know that's an old version. We've been testing with 7.4 now and are
nearly ready to upgrade.)
--
Matthew Nuzum <[EMAIL PROTECTED]>
www.followers.net - Makers of "Elite Content Manage
its been running for
hours already and I don't know when it will finish.
--
Matthew Nuzum <[EMAIL PROTECTED]>
www.followers.net - Makers of "Elite Content Management System"
View samples of Elite CMS in action by visiting
http://www.elitecms.com/
---(
thought. There are people who use proxying in apache to
redirect expensive tasks to other servers that are dedicated to just one
heavy challenge. In that case you likely do have 99% dynamic content.
Matthew Nuzum | Makers of "Elite Content Management System"
www.followers.net
p to scroll off the screen.
--
Matthew Nuzum + "Man was born free, and everywhere
www.bearfruit.org : he is in chains," Rousseau
+~~+ "Then you will know the truth, and
the TRUTH will set you free," Jesus Christ (John 8:32 NIV)
-Original Message--
elt compelled to chime in.
--
Matthew Nuzum + "Man was born free, and everywhere
www.bearfruit.org : he is in chains," Rousseau
+~~+ "Then you will know the truth, and
the TRUTH will set you free," Jesus Christ (John 8:32 NIV)
-Original Message-
From: [
however, the proven benefits of NFS and
> NAS outweigh this penalty in most production environments.
Matthew Nuzum | ISPs: Make $200 - $5,000 per referral by
www.followers.net | recomending Elite CMS to your customers!
[EMAIL PROTECTED] | http://www.followers.net/
.. I'm getting this... which
is problematic because..."
The more clearly you state the abstract goal the more creative answers
you'll get with people often suggesting things you'd never considered.
I hope this helps and I hope that you achieve your goals of a well
performing applica
On Wed, 2004-06-02 at 17:39, Greg Stark wrote:
> "Matthew Nuzum" <[EMAIL PROTECTED]> writes:
>
> > I have colinux running on a Fedora Core 1 image. I have the rhdb 3 (or
> > PostgreSQL RedHat Edition 3) on it running. Here are tests with fsync on
> > and
it. I must leave my office at 4:15 EDT and will not return
until Friday, although I can do another test on my home computer Thursday.
Matthew Nuzum | Makers of "Elite Content Management System"
www.followers.net | View samples of Elite CMS in action
[EMAIL PROTECTED]
day-to-day tasks
though I think you'll like it because you can detach the terminal and let it
run in the background. When I do that I often forget it is running because
it produces such a low load on the system. If you are going to give it a
try, the one trick I used to get things going was to d
used by default out of the box to give good performance.
Maybe your computer is using all of it's I/O capacity because it's using PIO
mode or some other non-optimal method of accessing the disk.
Just a suggestion, I hope it helps,
Matthew Nuzum | ISPs: Make $200 - $5,000
pect that after the server gets loaded up the
impact will become more of a problem.
By the way, I must say that this thread has been very useful.
Matthew Nuzum | Makers of "Elite Content Management System"
www.followers.net | View samples
have what I call an "elitist" attitude;
meaning they expect you to be an expert or dedicated to their software in
order to use it. Invariably this mentality stifles the usefulness of the
product. It seems that there is a relative minority of people on this list
who feel that you
ion params in an easy to read format. Both of these refer to the
thorough reference manual that breaks each possible option down into it's
nitty gritty details so that a user can get more information if they so
desire.
Matthew Nuzum | Makers of "Elite Content Management System&quo
e are lot's of good ideas that
fly around here but never get followed up on.
Additionally, I have an increasingly large production database that I would
be willing to do some test-cases on. I don't really know how to do it
though... If someone where able to give instructions I could run te
53 matches
Mail list logo
