Andrew,
I will be the first to admit that there are probably some very good
possibilities for optimisation of this code. My impression though has been
that in almost all cases it's fast enough anyway. I know that on some very
modest hardware I have managed to load a 6m row TPC line-items
Andrew,
OK ... that seems fair enough. The next question is where the data being
loaded comes from? pg_dump? How does load speed compare with using COPY's
binary mode?
Oddly, our tests in the past have shown that binary is actually slower.
Luke
---(end of
Bruce,
I found that parentheses in gawk regular expressions require backslashes
so they are not treated as regex groupings:
$ echo '('|awk '$0 ~ /(/ {print $0}'
awk: cmd. line:1: fatal: Unmatched ( or \(: /(/
$ echo '('|awk '$0 ~ /\(/ {print $0}'
(
Now, it seems closing parentheses are
Bruce,
On 7/15/05 9:59 PM, Bruce Momjian pgman@candle.pha.pa.us wrote:
Actually, mine returns ')' too for the last command. I didn't copy
that into the email. How about the top tests? Notice I get an error on
the first one without the backslash. Are you OK escaping '(' but not
')'?
Good points on all, another element in the performance expectations is the
ratio of CPU speed to I/O subsystem speed, as Alon had hinted earlier.
This patch substantially (500%) improves the efficiency of parsing in the
COPY path, which, on a 3GHz P4 desktop with a commodity disk drive
represents
Cool!
At what rate does your disk setup write sequential data, e.g.:
time dd if=/dev/zero of=bigfile bs=8k count=50
(sized for 2x RAM on a system with 2GB)
BTW - the Compaq smartarray controllers are pretty broken on Linux from a
performance standpoint in our experience. We've had
Joshua,
On 7/21/05 5:08 PM, Joshua D. Drake [EMAIL PROTECTED] wrote:
O.k. this strikes me as interesting, now we know that Compaq and Dell
are borked for Linux. Is there a name brand server (read Enterprise)
that actually does provide reasonable performance?
I think late model Dell (post the
Joshua,
On 7/21/05 7:53 PM, Joshua D. Drake [EMAIL PROTECTED] wrote:
Well I know that isn't true at least not with ANY of the Dells my
customers have purchased in the last 18 months. They are still really,
really slow.
That's too bad, can you cite some model numbers? SCSI?
I have great
Tom,
Thanks for finding the bugs and reworking things.
I had some difficulty in generating test cases that weren't largely
I/O-bound, but AFAICT the patch as applied is about the same speed
as what you submitted.
You achieve the important objective of knocking the parsing stage down a
lot,
Tom,
The previous timings were for a table with 15 columns of mixed type. We
also test with 1 column to make the parsing overhead more apparent. In the
case of 1 text column with 145MB of input data:
Your patch:
Time: 6612.599 ms
Alon's patch:
Time: 6119.244 ms
Alon's patch is 7.5%
Tom,
My direct e-mails to you are apparently blocked, so I'll send this to the
list.
I've attached the case we use for load performance testing, with the data
generator modified to produce a single row version of the dataset.
I do believe that you/we will need to invert the processing loop to
Tom,
On 8/6/05 9:08 PM, Tom Lane [EMAIL PROTECTED] wrote:
Luke Lonergan [EMAIL PROTECTED] writes:
I had some difficulty in generating test cases that weren't largely
I/O-bound, but AFAICT the patch as applied is about the same speed
as what you submitted.
You achieve the important
Simon,
That part of the code was specifically written to take advantage of
processing pipelines in the hardware, not because the actual theoretical
algorithm for that approach was itself faster.
Yup, good point.
Nobody's said what compiler/hardware they have been using, so since both
Alvaro,
On 8/10/05 9:46 AM, Alvaro Herrera [EMAIL PROTECTED] wrote:
AFAIR he never claimed otherwise ... his point was that to gain that
additional speedup, the code has to be made considerable worse (in
maintenability terms.) Have you (or Alon) tried to port the rest of the
speed
Bruce,
On 12/27/05 10:20 AM, Bruce Momjian pgman@candle.pha.pa.us wrote:
OK, original patch applied to HEAD and smaller version to 8.1.X, and
regression test added, now attached.
Great, good catch.
Have you tested performance, before and after?
The only good way to test performance is using
Thanks for this Jim! We'll test it and report results.
- Luke
From: [EMAIL PROTECTED] on behalf of Jim C. Nasby
Sent: Sat 2/11/2006 9:26 PM
To: pgsql-patches@postgresql.org
Subject: [PATCHES] OS X shared memory documentation
I finally figured out what the
Title: Re: [PATCHES] WIP: further sorting speedup
Cool!
Well test this sometime soon and get back to you. Were kind of jammed this week, hopefully well get some time.
So you know, weve done some more work on the external sort to remove the tape abstraction from the code, which makes a
:
Luke Lonergan [EMAIL PROTECTED] writes:
So you know, we=B9ve done some more work on the external sort to remove the
=B3tape=B2 abstraction from the code, which makes a significant improvement.
Improvement where? That code's down in the noise so far as I can tell.
I see results like
Tom,
On 5/8/06 11:46 AM, Tom Lane [EMAIL PROTECTED] wrote:
I made a table of 16M rows with an
index over a random-data integer column. With a thoroughly disordered
index (built on-the-fly as the random data was inserted), the time to
VACUUM after deleting a small number of rows was 615
Nice one Martijn - we have immediate need for this, as one of our sizeable
queries under experimentation took 3 hours without EXPLAIN ANALYZE, then
over 20 hours with it...
- Luke
On 5/9/06 2:38 PM, Martijn van Oosterhout kleptog@svana.org wrote:
On Tue, May 09, 2006 at 05:16:57PM -0400,
Martijn,
On 5/11/06 12:17 AM, Martijn van Oosterhout kleptog@svana.org wrote:
Did you test it? There are some cases where this might still leave some
noticable overhead (high loop count). I'm just not sure if they occur
all that often in practice...
I've sent it to our team for testing,
Bruce,
We have a bitmap index AM in Bizgres (on PG 8.1.3) that is functional and
achieves very worthwhile (10x+) performance and space objectives.
It is a large patch, and introduces the access method along with modifying
the appropriate executor nodes. The latter part was necessary because of
So, if I understand this correctly, we're calling Alloc and ContextAlloc 10
times for every row being summed?
There are approx 10M rows and the profile snippet below shows 100M calls to
each of those.
- Luke
On 11/24/06 4:46 PM, Mark Kirkwood [EMAIL PROTECTED] wrote:
time seconds
Hi Heikki,
On 5/17/07 10:28 AM, Heikki Linnakangas [EMAIL PROTECTED] wrote:
is also visible on larger scans that don't fit in cache with bigger I/O
hardware, and this patch would increase the max. I/O throughput that we
can handle on such hardware. I don't have such hardware available, I
Hi All,
On 5/31/07 12:40 AM, Heikki Linnakangas [EMAIL PROTECTED] wrote:
BTW, we've been talking about the L2 cache effect but we don't really
know for sure if the effect has anything to do with the L2 cache. But
whatever it is, it's real.
The mailing list archives contain the ample evidence
We just applied this and saw a 5 percent speedup on a hash aggregation query
with four colums in a 'group by' clause run against a single TPC-H table
(lineitem).
CK - can you post the query?
- Luke
Msg is shrt cuz m on ma treo
-Original Message-
From: Simon Riggs [mailto:[EMAIL
: Simon Riggs [mailto:[EMAIL PROTECTED]
Sent: Sunday, October 28, 2007 04:48 PM Eastern Standard Time
To: CK.Tan
Cc: Luke Lonergan; Kenneth Marshall; pgsql-patches@postgresql.org; [EMAIL
PROTECTED]; [EMAIL PROTECTED]
Subject:Re: [PATCHES] updated hash functions for postgresql v1
Nice!
- Luke
On 1/30/08 9:22 AM, Gregory Stark [EMAIL PROTECTED] wrote:
Here's the WIP patch for doing prereading when doing bitmap index scans.
I was performance testing it as I was developing it here:
http://archives.postgresql.org/pgsql-hackers/2007-12/msg00395.php
Note that
Cool! It's been a while since we've done the same kind of thing :-)
- Luke
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
Heikki Linnakangas
Sent: Saturday, February 23, 2008 5:30 PM
To: pgsql-patches@postgresql.org
Subject: [PATCHES]
Hi all,
I think the intent of the syntax / parser patch from Gavin and Jeff was to get
consensus from PG on the syntax prior to proceeding with the next chunk of work.
The next chunk of work is now well underway - with support for ALTER TABLE
and partitioning, along with fast inserts into the
30 matches
Mail list logo
