Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-16 Thread tmp
You could add it to here -- note that if we decide it isn't worth it it'll just get removed. Which category would you recommend? Optimizer / Executor? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-08 Thread Tom Lane
Gregory Stark [EMAIL PROTECTED] writes: But I can also see Tom's reluctance. It's a fair increase in the amount of code to maintain in that file for a pretty narrow use case. On the other hand it looks like it would be all in that file. The planner wouldn't have to do anything special to set

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-06 Thread David Lee Lambert
On Thursday 04 December 2008 15:09, Gregory Stark wrote: tmp [EMAIL PROTECTED] writes: Also, it is my impression that many people use LIMIT to minimize the evaluation time of sub queries from which the outer query only needs a small subset of the sub query output. I've seen lots of

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-06 Thread Grzegorz Jaskiewicz
On 2008-12-06, at 11:29, David Lee Lambert wrote: I use ORDER BY random() LIMIT :some_small_number frequently to get a feel for data. That always builds the unrandomized relation and then sorts it. I guess an alternate path for single-table queries would be to randomly choose a block

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-06 Thread Greg Stark
It's slow because there's no way around running through the entire input. The optimization tmp is talking about wouldn't be relevant becase there is an order by clause which was precisely why I I said it was a fairly narrow use case. Most people who use limit want a specific subset even if

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-05 Thread tmp
I would tend to think it's worth it myself. I am unfortunately not familiar enough with the postgresql code base to be comfortable to provide a patch. Can I submit this optimization request to some sort of issue tracker or what should I do? -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-05 Thread Gregory Stark
tmp [EMAIL PROTECTED] writes: I would tend to think it's worth it myself. I am unfortunately not familiar enough with the postgresql code base to be comfortable to provide a patch. Can I submit this optimization request to some sort of issue tracker or what should I do? You could add it to

[HACKERS] Optimizing DISTINCT with LIMIT

2008-12-04 Thread tmp
As far as I have understood the following query SELECT DISTINCT foo FROM bar LIMIT baz is done by first sorting the input and then traversing the sorted data, ensuring uniqueness of output and stopping when the LIMIT threshold is reached. Furthermore, a part of the sort procedure is to

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-04 Thread Gregory Stark
tmp [EMAIL PROTECTED] writes: If the input is sufficiently large and the LIMIT threshold sufficiently small, maintain the DISTINCT output by hashning while traversing the input and stop when the LIMIT threshold is reached. No sorting required and *at* *most* one read of input. You mean

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-04 Thread Heikki Linnakangas
Gregory Stark wrote: tmp [EMAIL PROTECTED] writes: If the input is sufficiently large and the LIMIT threshold sufficiently small, maintain the DISTINCT output by hashning while traversing the input and stop when the LIMIT threshold is reached. No sorting required and *at* *most* one read of

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-04 Thread Gregory Stark
Heikki Linnakangas [EMAIL PROTECTED] writes: Gregory Stark wrote: Does that know to stop scanning as soon as it has seen 5 distinct values? Uhm, hm. Apparently not :( postgres=# create or replace function v(integer) returns integer as $$begin raise notice 'called %', $1; return $1; end$$

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-04 Thread Tom Lane
Heikki Linnakangas [EMAIL PROTECTED] writes: Gregory Stark wrote: You mean like this? postgres=# explain select distinct x from i limit 5; QUERY PLAN --- Limit (cost=54.50..54.51 rows=1

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-04 Thread tmp
In principle, if there are no aggregate functions, then nodeAgg could return a row immediately upon making any new entry into the hash table. Whether it's worth the code uglification is debatable ... I think it would require a third major pathway through nodeAgg. Regarding whether it's worth

Re: [HACKERS] Optimizing DISTINCT with LIMIT

2008-12-04 Thread Gregory Stark
tmp [EMAIL PROTECTED] writes: Regarding whether it's worth the effort: In each of my three past jobs (all using postgresql) I have met several queries that would fetch a small subset of a large - even huge - input. I think that types of queries are relatively common out there, but if they