a very large number of rows
(probably at least 10M), the overhead of a text datatype over a
smallint or int/oid gets to be very large.
--
Decibel!, aka Jim C. Nasby, Database Architect deci...@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828
--
Sent via pgsql
.
If you want an example of that, we had Command Prompt create a full
set of hash datatypes (SHA*, and I think md5). That stuff should be
on pgFoundry; if it's not drop me a note at jna...@cashnetusa.com and
I'll get it added.
--
Decibel!, aka Jim C. Nasby, Database Architect deci
.
If you want an example of that, we had Command Prompt create a full
set of hash datatypes (SHA*, and I think md5). That stuff should be
on pgFoundry; if it's not drop me a note at jna...@cashnetusa.com and
I'll get it added.
--
Decibel!, aka Jim C. Nasby, Database Architect deci
in
work_mem...
Are you using temp tables? Those end up in pgsql_tmp as well.
--
Decibel!, aka Jim C. Nasby, Database Architect deci...@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org
.
FWIW, you can avoid that with SET LOCAL (though it would still affect
the rest of the transaction).
You could also store whatever enable_seqscan was set to in a variable
before setting it to false and then set it back when you're done.
--
Decibel!, aka Jim C. Nasby, Database Architect deci
is a serious pain in the rear.
--
Decibel!, aka Jim C. Nasby, Database Architect deci...@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http
FBSD would allow that, but if it does it wouldn't surprise me
if kernel/OS performance stunk. If Postgres then used the same
settings it would make matters even worse (IIRC there is some code
that's different in an AMD vs Intel build).
--
Decibel!, aka Jim C. Nasby, Database Architect deci
when you respond back to
help collect feedback.
Do you have a self-contained test case? I have several boxes with 16-
cores worth of Xeon with 96GB I could try it on (though you might not
care about having only 16 cores :P)
--
Decibel!, aka Jim C. Nasby, Database Architect deci
, isn't there a DTrace probe that would provide that info? It
certainly seems like something you'd want to know...
--
Decibel!, aka Jim C. Nasby, Database Architect deci...@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828
--
Sent via pgsql-performance mailing list
system (such as FreeBSD) with different levels of
-j handed to make (of course you'll need to wait until everything is
in cache, and I'm assuming you have enough memory so that everything
would fit in cache).
--
Decibel!, aka Jim C. Nasby, Database Architect deci...@decibel.org
Give your
The good news is there's now at least enough runnable procs. What I
find *extremely* odd is the CPU usage is almost dead constant...
--
Decibel!, aka Jim C. Nasby, Database Architect deci...@decibel.org
Give your computer some brain candy! www.distributed.net Team #1828
--
Sent via pgsql
). AFAIK it's considerably slower
to scan.
Actually, that's not necessarily true. If both partno and procuder_id
are ints and you're on a 64bit platform, there won't be any change in
index size, due to alignment issues.
--
Decibel!, aka Jim C. Nasby, Database Architect deci...@decibel.org
Give
On Nov 10, 2008, at 9:20 PM, Tom Lane wrote:
Jim 'Decibel!' Nasby [EMAIL PROTECTED] writes:
On Nov 10, 2008, at 1:31 PM, Tom Lane wrote:
On my machine this runs about twice as fast as the original view.
Am I missing some magic? I'm still getting the subquery scan.
Hmm, I'm getting a core
On Nov 11, 2008, at 1:15 PM, Tom Lane wrote:
Jim 'Decibel!' Nasby [EMAIL PROTECTED] writes:
On Nov 10, 2008, at 9:20 PM, Tom Lane wrote:
8.3 gets it right though.
Doesn't seem to for me... :/
Oh, I was looking at select * from v2 not select count(*) from v2.
HEAD is a bit smarter about
ON ltp.loan_task_code_id = ltc.id;
The stuff I omitted is just some fields and a few other NULLs. This
is 8.2.9.
--
Decibel! [EMAIL PROTECTED] (512) 569-9461
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org
On Nov 10, 2008, at 7:06 AM, Tom Lane wrote:
Jim 'Decibel!' Nasby [EMAIL PROTECTED] writes:
loan_tasks effectively does SELECT * FROM loan_tasks_committed UNION
ALL SELECT * FROM loan_tasks_pending;.
You seem to have neglected to mention a join or two.
Yeah, though I did show them
On Nov 10, 2008, at 1:31 PM, Tom Lane wrote:
Jim 'Decibel!' Nasby [EMAIL PROTECTED] writes:
Here's the commands to generate the test case:
create table a(a int, b text default 'test text');
create table c(c_id serial primary key, c_text text);
insert into c(c_text) values('a'),('b'),('c
for changing the definition of the composite type?
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
, that might be
an easy way to deal with this stuff. Create a composite type of
(name_id, value) and store that in an array.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME
right now, but it
will only be used to occasionally pull up single entities), but you
have to be really careful with it. I don't see it working very well
for what it sounds like you're trying to do.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some
about it the last time this was brought up
there was discussion about something that doesn't force a particular
execution method, but instead provides improved information to the
planner. It might be worth pursuing that, as I think there was less
opposition to it.
--
Decibel!, aka Jim C. Nasby
take that
approach.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
of a database, but keeps the data away from your
production data.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
that information.
BTW, it would probably be better to store data either in the main
table, or the history table, but not both places.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S
. And it's actually the 1st query that
was faster, because it returned fewer rows (15k instead of 45k).
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
. If you're doing random writes that perfectly align with the
raid stripe size, you'll see virtually no RAID5 overhead, and you'll
get the performance of N-1 drives, as opposed to RAID10 giving you N/2.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some
that's insert
timestamp; those two fields would correlate highly, and you should
even be able to correlate the two histograms; that would allow you to
infer that most of the insert times for _id's between 100 and 200
will be between 2008-01-01 00:10 and 2008-01-01 00:20, for example.
--
Decibel
.
If you do this *please* post it. I really think it would be worth
while for us to have fixed-size data types for common forms of binary
data; MD5, SHA1 and SHA256 come to mind.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy
distribution of account number touches, and might look a little better
on this type of test.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
but this is the
part that made the difference) after a little change the query took
~1 second:
select * from t1, t2 where t1.id 158507 and t2.id 158507 and
t1.id = t2.id;
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy
). As I recall, if we fall off of the
assumption, the penalty for long scans get large-ish (maybe 2X), but
since when do people full table scan when they're updates/inserts are
so scattered across TIDs? It's an obvious big win for DW work.
--
Decibel!, aka Jim C. Nasby, Database Architect
that is recommend a
backgrounded vacuum after data load.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
the actual expense in the exception
block) is fairly CPU-intensive, but it's not common for a database
server to be CPU-bound, even for OLTP. You're usually still waiting
on disk.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy
with others though: it certainly doesn't sound like there's
any reason to be using temp tables here at all. This sounds like a
case of trying to apply procedural programming techniques to a
database instead of using set theory (which generally doesn't work
well).
--
Decibel!, aka Jim C
to be NULL; as such the comments above would apply.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
pgpjPrQhlOQDS.pgp
Description: PGP signature
second, tops. Of course
not all pages will be dirty, but still...
I normally use between 10 and 20 for cost_delay (lower values for faster
drive arrays).
autovacuum_vacuum_cost_limit = 120
Why'd you reduce this? I'd put it back to 200...
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL
), and then
SELECT sum(relpages) FROM pg_class c JOIN pg_namespace n ON
(c.relnamespace=n.oid) AND n.nspname='pg_temp_blah';
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME
...
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
where a cron'd vacuum that runs once a minute is
probably a wise idea.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
On Nov 19, 2007, at 9:23 AM, Tom Lane wrote:
Decibel! [EMAIL PROTECTED] writes:
FWIW, 20k rows isn't all that big, so I'm assuming that the
descriptions make the table very wide. Unless those descriptions are
what's being updated frequently, I suggest you put those in a
separate table (vertical
expensive.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
... ;)
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
(vertical partitioning). That will make the main table
much easier to vacuum, as well as reducing the impact of the high
churn rate.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S
.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
smime.p7s
Description: S/MIME cryptographic signature
towards top on linux == dumb.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
as 'cached', then no, the
information it's providing is not correct.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
---(end of broadcast)---
TIP 9: In versions
; if you're just using the default isolation level of read
committed, you're fine with CLUSTER.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
pgpJrcbgbZXeA.pgp
Description: PGP signature
into how vacuum works based on that data.
Take a look at the stuff at http://decibel.org/~decibel/pervasive/, it'd
hopefully provide a useful starting point.
--
Decibel!, aka Jim C. Nasby, Database Architect [EMAIL PROTECTED]
Give your computer some brain candy! www.distributed.net Team #1828
going to kill you. Even if it does fit in memory, if
you're doing much writing at all you're going to be in big trouble.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpvMfqORWRv7.pgp
Description: PGP
not sure. The reason we're
changing machines is that we might be changing ISPs and we're renting /
leasing the machines from the ISP.
Get yourself the ability to benchmark your application. This is
invaluable^W a requirement for any kind of performance tuning.
--
Decibel!, aka Jim Nasby
getting anywhere near the performance
it should be, even with a simple dd test.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgprqUCHZieqB.pgp
Description: PGP signature
? This is a regular event in
Actually, he does. :) Or at least I don't think Portland gets a lot of
t-storms, just rain by the bucketful.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpkpeZ765p7x.pgp
, or getting drowned by a break in the city reservoir that's
a couple hundred yards up the hill (but at least I needn't worry about
Invest in sponges. Lots of them. :)
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell
suspect that latency will be far more critical
than overall bandwidth. I don't know if it's inherent to Gig-E, but my
limited experience has been that Gig-E has higher latency than 100mb.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com
subtract less.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpbj9aYHBI3X.pgp
Description: PGP signature
On Tue, Sep 11, 2007 at 02:49:28PM -0400, Jean-David Beyer wrote:
It is tricky for me to find a big enough file to test. I tried one of the
dd if=/dev/zero of=bigfile bs=8192 count=100
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http
sure your FSM is big enough
(periodic vacuumdb -av | tail is an easy way to check that).
Try a REINDEX. VACUUM FULL is especially hard on the indexes, and it's
easy for them to seriously bloat.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http
now-a-days, so I don't think
there's a huge difference there.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgp0JJsI29xNu.pgp
Description: PGP signature
On Tue, Sep 11, 2007 at 05:09:00PM -0400, Michael Stone wrote:
On Tue, Sep 11, 2007 at 03:55:51PM -0500, Decibel! wrote:
Also, to reply to someone else's email... there is one big reason to use
a SAN over direct storage: you can do HA that results in 0 data loss.
Good SANs are engineered
.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpCY6kMFsQLb.pgp
Description: PGP signature
On Wed, Sep 12, 2007 at 12:02:46AM +0100, Gregory Stark wrote:
Decibel! [EMAIL PROTECTED] writes:
On Tue, Sep 11, 2007 at 02:49:28PM -0400, Jean-David Beyer wrote:
It is tricky for me to find a big enough file to test. I tried one of the
dd if=/dev/zero of=bigfile bs=8192 count=100
output.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
---(end of broadcast)---
TIP 9: In versions below 8.0, the planner will ignore your desire to
choose
a restart? I really don't know.
No, it sounds to me like you just weren't vacuuming aggressively enough
to keep up with demand.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpl6Jx8Grbto.pgp
Description
will be very similar in
performance to async commit.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpoy8I6Hfg5B.pgp
Description: PGP signature
better than it is from what I'm hearing.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpIHiEv5ElNs.pgp
Description: PGP signature
On Mon, Aug 27, 2007 at 04:56:33PM -0500, Kevin Grittner wrote:
Decibel! [EMAIL PROTECTED] 08/27/07 4:00 PM
They're running version 8.1.4
As for your pg_dump idea... why not just do a CREATE TABLE AS SELECT *
FROM bloated_table? That would likely be much faster than messing around
, and it can cache writes (it has a
BBU), I'd actually lean towards putting all 12 drives into one raid
10. A good controller will be able to handle WAL fsyncs plenty fast
enough, so having a separate WAL mirror would likely hurt more than
help.
--
Decibel!, aka Jim Nasby
in the BIOS,
but is this a lie?
Unless they use the plus notation (ie: RAID 1+0 or RAID 0+1), you
never truly know what you're getting.
BTW, there's other reasons that RAID 0+1 stinks, beyond just
performance.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http
low priority on testing RAID5... it's rarely a good idea for a database.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpAVdtC8cRhR.pgp
Description: PGP signature
.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgp3mph03PXnn.pgp
Description: PGP signature
with indexed fields, it
doesn't have to touch those either.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgplqWULgqzjL.pgp
Description: PGP signature
set of
data.)
Something else that might be worth looking at is having your existing
workload modeled; that allows building a pretty accurate estimate of
what kind of hardware would be required to hit a different workload.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED
makes an extra controller much less useful.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpDQ9nJ7lGKI.pgp
Description: PGP signature
On Thu, Aug 09, 2007 at 08:58:19PM -0500, Scott Marlowe wrote:
On Thu, Aug 09, 2007 at 05:29:18PM -0500, Scott Marlowe wrote:
On 8/9/07, Decibel! [EMAIL PROTECTED] wrote:
Also, a good RAID controller can spread reads out across both drives in
each mirror on a RAID10. Though
, and would only be recommended if you didn't expect the index contents
to change?
Yes.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgp4WeZLcqCcp.pgp
Description: PGP signature
table's been updated just once before, even
with vacuuming you're now at 10G of data.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpC7cdBrUEbs.pgp
Description: PGP signature
On Tue, Aug 07, 2007 at 02:36:18PM -0500, Scott Marlowe wrote:
On 8/7/07, Decibel! [EMAIL PROTECTED] wrote:
On Tue, Aug 07, 2007 at 02:33:19PM +0100, Richard Huxton wrote:
Mark Makarowsky wrote:
I have a table with 4,889,820 records in it. The
table also has 47 fields. I'm having
, which
hopefully eliminates much of that bottleneck... but if you don't do
noatime you're probably still spewing a lot out to the drive.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpSoBDcFGFrM.pgp
of priority inversion before
going into production with this solution.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgpwzHh5wWykY.pgp
Description: PGP signature
Please reply-all so others can learn and contribute.
On Sun, Jul 29, 2007 at 09:38:12PM -0700, Craig James wrote:
Decibel! wrote:
It's unlikely that it's going to be faster to index scan 2.3M rows than
to sequential scan them. Try setting enable_seqscan=false and see if it
is or not.
Out
, and 20ms on
slower hardware.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
---(end of broadcast)---
TIP 2: Don't 'kill -9' the postmaster
above won't work correctly.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgp5eYJsdKDrX.pgp
Description: PGP signature
than
to sequential scan them. Try setting enable_seqscan=false and see if it
is or not.
--
Decibel!, aka Jim Nasby[EMAIL PROTECTED]
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)
pgp7DXcKrZWrf.pgp
Description: PGP signature
83 matches
Mail list logo