Re: [HACKERS] optimizer questions

2006-02-15 Thread Jens-Wolfhard Schicke
--On Dienstag, Februar 14, 2006 10:35:12 -0600 hector Corrada Bravo [EMAIL PROTECTED] wrote: Before I start trying this (creating aggregate paths seems the reasonable thing to do) I would like your counsel. 1) Regardless of the optimization problem, is the executor able to execute aggregate

Re: [HACKERS] Patch Submission Guidelines

2006-02-15 Thread Martijn van Oosterhout
On Tue, Feb 14, 2006 at 05:28:54PM -0500, Tom Lane wrote: Simon Riggs [EMAIL PROTECTED] writes: How much time would you need? I think having every patch built before anyone even looks at the code would sort out most of the issues I mentioned. If I ran a buildfarm machine, I'd turn it

Re: [HACKERS] Patch Submission Guidelines

2006-02-15 Thread Simon Riggs
On Tue, 2006-02-14 at 21:47 -0500, Neil Conway wrote: On Tue, 2006-02-14 at 22:54 +, Simon Riggs wrote: On Tue, 2006-02-14 at 17:28 -0500, Tom Lane wrote: IMHO the thing we are really seriously short of is patch reviewers. [...] Well that was the basis of my original suggestion.

Re: [HACKERS] Patch Submission Guidelines

2006-02-15 Thread Chris Browne
[EMAIL PROTECTED] (Robert Treat) writes: On Tuesday 14 February 2006 16:00, Martijn van Oosterhout wrote: I would like to suggest that we increase substantially the FAQ entries relating to patch submission. By we, I actually mean please could the committers sit down and agree some

[HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Tom Lane
Gary Doades [EMAIL PROTECTED] writes: If I run the script again, it is not always the first case that is slow, it varies from run to run, which is why I repeated it quite a few times for the test. For some reason I hadn't immediately twigged to the fact that your test script is just N

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Gary Doades
Tom Lane wrote: For some reason I hadn't immediately twigged to the fact that your test script is just N repetitions of the exact same structure with random data. So it's not so surprising that you get random variations in behavior with different test data sets. It seems clear that our

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Tom Lane
Gary Doades [EMAIL PROTECTED] writes: Is this likely to hit me in a random fashion during normal operation, joins, sorts, order by for example? Yup, anytime you're passing data with that kind of distribution through a sort. So the options are: 1) Fix the included qsort.c code and use that

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Tom Lane
Gary Doades [EMAIL PROTECTED] writes: Ouch! That confirms my problem. I generated the random test case because it was easier than including the dump of my tables, but you can appreciate that tables 20 times the size are basically crippled when it comes to creating an index on them.

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index

2006-02-15 Thread Ron
This behavior is consistent with the pivot choosing algorithm assuming certain distribution(s) for the data. For instance, median-of-three partitioning is known to be pessimal when the data is geometrically or hyper-geometrically distributed. Also, care must be taken that sometimes is not

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Tom Lane
I wrote: Gary Doades [EMAIL PROTECTED] writes: Ouch! That confirms my problem. I generated the random test case because it was easier than including the dump of my tables, but you can appreciate that tables 20 times the size are basically crippled when it comes to creating an index on

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Tom Lane
Ron [EMAIL PROTECTED] writes: How are we choosing our pivots? See qsort.c: it looks like median of nine equally spaced inputs (ie, the 1/8th points of the initial input array, plus the end points), implemented as two rounds of median-of-three choices. With half of the data inputs zero, it's not

[HACKERS] Generating config stuff from single source

2006-02-15 Thread Peter Eisentraut
We are currently maintaining information about configuration parameters in at least three places: the documentation, guc.c, and postgresql.conf.sample. I would like to generate these from a single source. Computationally, this is not very challenging, it's just a bit of work. I imagine as

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Dann Corbit
-Original Message- From: [EMAIL PROTECTED] [mailto:pgsql-hackers- [EMAIL PROTECTED] On Behalf Of Tom Lane Sent: Wednesday, February 15, 2006 5:22 PM To: Ron Cc: pgsql-performance@postgresql.org; pgsql-hackers@postgresql.org Subject: Re: [HACKERS] qsort again (was Re: [PERFORM]

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Christopher Kings-Lynne
Ouch! That confirms my problem. I generated the random test case because it was easier than including the dump of my tables, but you can appreciate that tables 20 times the size are basically crippled when it comes to creating an index on them. I have to say that I restored a few gigabyte

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index

2006-02-15 Thread Simon Riggs
On Wed, 2006-02-15 at 19:59 -0500, Tom Lane wrote: I get amazingly stable runtimes now --- I didn't have the patience to run 100 trials, but in 30 trials I have slowest 11538 msec and fastest 11144 msec. So this code path is definitely not very sensitive to this data distribution. The

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index

2006-02-15 Thread Neil Conway
On Wed, 2006-02-15 at 18:28 -0500, Tom Lane wrote: It seems clear that our qsort.c is doing a pretty awful job of picking qsort pivots, while glibc is mostly managing not to make that mistake. I haven't looked at the glibc code yet to see what they are doing differently. glibc qsort is

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Qingqing Zhou
Tom Lane [EMAIL PROTECTED] wrote I did this 100 times and sorted the reported runtimes. I'd say this puts a considerable damper on my enthusiasm for using our qsort all the time, as was recently debated in this thread: http://archives.postgresql.org/pgsql-hackers/2005-12/msg00610.php 100

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Tom Lane
Qingqing Zhou [EMAIL PROTECTED] writes: By did this 100 times do you mean generate a sequence of at most 20*100 numbers, and for every 20 numbers, the first half are all zeros and the other half are uniform random numbers? No, I mean I ran the bit of SQL script I gave 100 separate

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Qingqing Zhou
Tom Lane [EMAIL PROTECTED] wrote Qingqing Zhou [EMAIL PROTECTED] writes: By did this 100 times do you mean generate a sequence of at most 20*100 numbers, and for every 20 numbers, the first half are all zeros and the other half are uniform random numbers? No, I mean I ran the bit

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Qingqing Zhou
Qingqing Zhou [EMAIL PROTECTED] wrote I must misunderstand something here -- I can't figure out that why the cost of the same procedure keep climbing? Ooops, I mis-intepret the sentence -- you sorted the results ... Regards, Qingqing ---(end of

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index behaviour)

2006-02-15 Thread Tom Lane
Qingqing Zhou [EMAIL PROTECTED] writes: Tom Lane [EMAIL PROTECTED] wrote No, I mean I ran the bit of SQL script I gave 100 separate times. I must misunderstand something here -- I can't figure out that why the cost of the same procedure keep climbing? No, the run cost varies randomly

Re: [HACKERS] qsort again (was Re: [PERFORM] Strange Create Index

2006-02-15 Thread Ron
At 08:21 PM 2/15/2006, Tom Lane wrote: Ron [EMAIL PROTECTED] writes: How are we choosing our pivots? See qsort.c: it looks like median of nine equally spaced inputs (ie, the 1/8th points of the initial input array, plus the end points), implemented as two rounds of median-of-three choices.

[HACKERS] Blog post on EnterpriseDB...maybe off topic

2006-02-15 Thread Christopher Kings-Lynne
http://www.flamingspork.com/blog/2006/02/16/enterprisedb-where-is-the-source/ Any comments on this? Is he referring to EnterpriseDB extensions that they don't make public? Chris ---(end of broadcast)--- TIP 1: if posting/reading through