Re: [PERFORM] Slow Count-Distinct Query

2014-04-06 Thread Varadharajan Mukundan
Hi Jeff, Instead what I get is the index only scan (to provide order) feeding into a > Group. > That's interesting. We tested out in two versions of Postgres (9.2 and 9.3) in different Mac machines and ended up with index-only scan only after the partial index. I remember doing a vacuum full anal

[PERFORM] Slow Count-Distinct Query

2014-04-06 Thread Jeff Janes
On Friday, April 4, 2014, Varadharajan Mukundan wrote: > Hi Jeff, > > It looks like the original emailer wrote a query that the planner is not >> smart enough to plan properly (A known limitation of that kind of query). >> He then made a bunch of changes, none of which worked. He then re-wrote

Re: [PERFORM] Slow Count-Distinct Query

2014-04-01 Thread Christopher Jackson
Hi Bricklen, Thanks again for the feedback. The concurrent refresh sounds cool. I just saw the 9.4 release is tentatively scheduled for later this year. Do you know what people have been doing for view refreshes in the meantime? Thanks On Tue, Apr 1, 2014 at 11:48 PM, Michael Paqui

Re: [PERFORM] Slow Count-Distinct Query

2014-04-01 Thread Michael Paquier
On Wed, Apr 2, 2014 at 1:22 PM, Christopher Jackson wrote: > > Hi Bricklen, > > Thanks for the feedback. I'll play around with materialized views. My > understanding is they have to be manually triggered for refresh Yep. > and there's an exclusive lock on the view while the refresh is t

Re: [PERFORM] Slow Count-Distinct Query

2014-04-01 Thread Christopher Jackson
Hi Bricklen, Thanks for the feedback. I'll play around with materialized views. My understanding is they have to be manually triggered for refresh and there's an exclusive lock on the view while the refresh is taking place. Is this your understanding as well? I'm using PG 9.3.3. If th

Re: [PERFORM] Slow Count-Distinct Query

2014-04-01 Thread bricklen
On Sun, Mar 30, 2014 at 12:45 PM, Christopher Jackson wrote: > Hi all, > > tl;dr - How can I speed up my count-distinct query? > Depending on how often you need to run that query and how important it is to you, if you are willing to accept a performance hit on INSERT/UPDATE/DELETE of the "par

Re: [PERFORM] Slow Count-Distinct Query

2014-03-31 Thread Christopher Jackson
Tom and Shawn, Thanks for the feedback. This has been helpful. It's worth noting that I was spiking this out on my local box using default memory utilization settings. I'll revisit this once we get our production box set up. It's good to know what the best practices are around the enable

Re: [PERFORM] Slow Count-Distinct Query

2014-03-31 Thread Tom Lane
Christopher Jackson writes: > tl;dr - How can I speed up my count-distinct query? EXPLAIN doesn't provide a lot of visibility into what the Aggregate plan node is doing, but in this case what it's doing is an internal sort/uniq operation to implement the DISTINCT. You didn't say what value of

Re: [PERFORM] Slow Count-Distinct Query

2014-03-31 Thread Shaun Thomas
>  tl;dr - How can I speed up my count-distinct query?   You can't. Doing a count(distinct x) is much different than a count(1), which can simply scan available indexes. To build a distinct, it has to construct an in-memory hash of every valid email, and count the distinct values therein. This

[PERFORM] Slow Count-Distinct Query

2014-03-30 Thread Christopher Jackson
Hi all, tl;dr - How can I speed up my count-distinct query? I apologize in advance if this question has been asked already. I'm finding the mailing list hard to navigate. I'm trying to speed up a query that will find a count of distinct emails with in a table using Postgres 9.3.3. The na

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-09 Thread Gorshkov
On 2011-02-03 22:48, Scott Marlowe wrote: On Thu, Feb 3, 2011 at 8:40 PM, Greg Smith wrote: Scott Marlowe wrote: Yes they're useful, but like a plastic bad covering a broken car window, they're useful because they cover something that's inherently broken. Awesome. Now we have a car anolog

Re: Does auto-analyze work on dirty writes? (was: Re: [HACKERS] [PERFORM] Slow count(*) again...)

2011-02-06 Thread Tom Lane
Mark Mielke writes: > My understanding is: > 1) Background daemon wakes up and checks whether a number of changes > have happened to the database, irrelevant of transaction boundaries. > 2) Background daemon analyzes a percentage of rows in the database for > statistical data, irrelevant of ro

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-06 Thread Nick Lello
Informix IDS supports hints as well; normally the only need for hints in this engine is when the Table/Index statistics are not being updated on a regular basis (ie: lazy DBA). On 3 February 2011 22:17, Mark Kirkwood wrote: > On 04/02/11 11:08, Josh Berkus wrote: > >> I don't think that's actual

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-06 Thread Andrew Dunstan
On 02/04/2011 02:32 AM, da...@lang.hm wrote: when a copy command is issued, I assume that there is some indication of how much data is going to follow. No of course there isn't. How would we do that with a stream like STDIN? Read the code. cheers andrew -- Sent via pgsql-performance

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-05 Thread Greg Smith
Scott Marlowe wrote: With a 24 drive RAID-10 array that can read at ~1GB/s I am almost always CPU bound during copies. This isn't wholly bad as it leaves spare IO for the rest of the machine so regular work carries on just fine. And you don't need nearly that much I/O bandwidth to reach tha

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-05 Thread Scott Marlowe
On Fri, Feb 4, 2011 at 11:37 PM, Robert Haas wrote: > On Sat, Feb 5, 2011 at 12:46 AM,   wrote: >>> Actually for me the main "con" with streaming analyze is that it adds >>> significant CPU burden to already not too fast load process. Especially if >>> it's automatically done for any load operatio

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Robert Haas
On Sat, Feb 5, 2011 at 12:46 AM, wrote: >> Actually for me the main "con" with streaming analyze is that it adds >> significant CPU burden to already not too fast load process. Especially if >> it's automatically done for any load operation performed (and I can't see >> how it can be enabled on s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread david
On Fri, 4 Feb 2011, Vitalii Tymchyshyn wrote: 04.02.11 16:33, Kenneth Marshall ???(??): In addition, the streaming ANALYZE can provide better statistics at any time during the load and it will be complete immediately. As far as passing the entire table through the ANALYZE process, a simple

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Scott Marlowe
On Fri, Feb 4, 2011 at 5:17 PM, Bruce Momjian wrote: > Mladen Gogala wrote: >> characteristic of a religious community chastising a sinner. Let me >> remind you again: all other major databases have that possibility: >> Oracle, MySQL, DB2, SQL Server and Informix. Requiring burden of proof >> abou

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Bruce Momjian
Mladen Gogala wrote: > Actually, it is not unlike a religious dogma, only stating that "hints > are bad". It even says so in the wiki. The arguments are > 1) Refusal to implement hints is motivated by distrust toward users, > citing that some people may mess things up. > Yes, they can, with a

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Bruce Momjian
Mladen Gogala wrote: > Chris Browne wrote: > > Well, the community declines to add hints until there is actual > > consensus on a good way to add hints. > > > OK. That's another matter entirely. Who should make that decision? Is > there a committee or a person who would be capable of making t

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Bruce Momjian
Greg Smith wrote: > Check out > http://www.indeed.com/jobtrends?q=postgres%2C+mysql%2C+oracle&relative=1&relative=1 > > if you want to see the real story here. Oracle has a large installed > base, but it's considered a troublesome legacy product being replaced +1 for Oracle being a "troubles

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Robert Haas
On Fri, Feb 4, 2011 at 9:38 AM, Vitalii Tymchyshyn wrote: > Actually for me the main "con" with streaming analyze is that it adds > significant CPU burden to already not too fast load process. Exactly. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Scott Marlowe
On Fri, Feb 4, 2011 at 6:05 AM, Grant Johnson wrote: > >> Yes.  And this has little to do with hints.  It has to do with years >> of development lead with THOUSANDS of engineers who can work on the >> most esoteric corner cases in their spare time.  Find the pg project a >> couple hundred software

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Kenneth Marshall
On Thu, Feb 03, 2011 at 09:05:52PM -0500, Robert Haas wrote: > On Thu, Feb 3, 2011 at 8:37 PM, wrote: > > On Thu, 3 Feb 2011, Robert Haas wrote: > > > >> On Thu, Feb 3, 2011 at 7:39 PM, ? wrote: > > Yeah, but you'll be passing the entire table through this separate > process that m

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Vitalii Tymchyshyn
04.02.11 16:33, Kenneth Marshall написав(ла): In addition, the streaming ANALYZE can provide better statistics at any time during the load and it will be complete immediately. As far as passing the entire table through the ANALYZE process, a simple counter can be used to only send the required s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Kenneth Marshall
On Thu, Feb 03, 2011 at 04:39:12PM -0800, da...@lang.hm wrote: > On Thu, 3 Feb 2011, Robert Haas wrote: > >> On Thu, Feb 3, 2011 at 3:54 PM, wrote: >>> with the current code, this is a completely separate process that knows >>> nothing about the load, so if you kick it off when you start the load

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Grant Johnson
Yes. And this has little to do with hints. It has to do with years of development lead with THOUSANDS of engineers who can work on the most esoteric corner cases in their spare time. Find the pg project a couple hundred software engineers and maybe we'll catch Oracle a little quicker. Otherw

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Torsten Zühlsdorff
Mladen Gogala schrieb: Well, the problem will not go away. As I've said before, all other databases have that feature and none of the reasons listed here convinced me that everybody else has a crappy optimizer. The problem may go away altogether if people stop using PostgreSQL. A common pr

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Віталій Тимчишин
4 лютого 2011 р. 09:32 написав: > > > when a copy command is issued, I assume that there is some indication of > how much data is going to follow. I know that it's not just 'insert > everything until the TCP connection terminates' because that would give you > no way of knowing if the copy got ev

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Fri, 4 Feb 2011, ??? wrote: 2011/2/3 If the table is not large enough to fit in ram, then it will compete for I/O, and the user will have to wait. what I'm proposing is that as the records are created, the process doing the creation makes copies of the records (either all of

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Віталій Тимчишин
2011/2/4 Mladen Gogala > Josh Berkus wrote: > >> However, since this system wasn't directly compatible with Oracle Hints, >> folks pushing for hints dropped the solution as unsatisfactory. This is >> the discussion we have every time: the users who want hints specifically >> want hints which work

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Віталій Тимчишин
2011/2/3 > > If the table is not large enough to fit in ram, then it will compete for > I/O, and the user will have to wait. > > what I'm proposing is that as the records are created, the process doing > the creation makes copies of the records (either all of them, or some of > them if not all ar

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Samuel Gendler
Neat. That was my 'you learn something every day' moment. Thanks. On Thu, Feb 3, 2011 at 9:06 PM, David Wilson wrote: > > > On Thu, Feb 3, 2011 at 6:05 PM, Robert Haas wrote: > >> >> If you want to randomly pick 10,000 rows out of all the rows that are >> going to be inserted in the table with

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread David Wilson
On Thu, Feb 3, 2011 at 6:05 PM, Robert Haas wrote: > > If you want to randomly pick 10,000 rows out of all the rows that are > going to be inserted in the table without knowing in advance how many > there will be, how do you do that? > Reservoir sampling, as the most well-known option: http://en

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: No, that's a plastic oh no! band you have. Wow, right you are. So with this type holding together my Japanese car, if it breaks and parts fall off, I'm supposed to yell "Oh, no! There goes Tokyo!", yes? -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Balti

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:56 PM, Greg Smith wrote: > Scott Marlowe wrote: >> >> It's not so much a car analogy as a plastic bad analogy. >> > > Is that like a Plastic Ono Band?  Because I think one of those is the only > thing holding the part of my bumper I smashed in the snow on right now.  I > c

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: It's not so much a car analogy as a plastic bad analogy. Is that like a Plastic Ono Band? Because I think one of those is the only thing holding the part of my bumper I smashed in the snow on right now. I could be wrong about the name. -- Greg Smith 2ndQuadrant U

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:40 PM, Greg Smith wrote: > Scott Marlowe wrote: >> >> Yes they're useful, but like a plastic bad covering a broken car window, >> they're useful because they cover something that's inherently broken. >> > > Awesome.  Now we have a car anology, with a funny typo no less.  "

Does auto-analyze work on dirty writes? (was: Re: [HACKERS] [PERFORM] Slow count(*) again...)

2011-02-03 Thread Mark Mielke
On 02/03/2011 09:45 PM, Conor Walsh wrote: My understanding is that auto-analyze will fire only after my transaction is completed, because it is a seperate daemon. If I do like so: BEGIN; COPY ...; -- Dangerously un-analyzed SELECT complicated-stuff ...; END; Auto-analyze does not benefit me,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: Yes they're useful, but like a plastic bad covering a broken car window, they're useful because they cover something that's inherently broken. Awesome. Now we have a car anology, with a funny typo no less. "Plastic bad", I love it. This is real progress toward gettin

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 7:05 PM, Robert Haas wrote: > If you want to randomly pick 10,000 rows out of all the rows that are > going to be inserted in the table without knowing in advance how many > there will be, how do you do that? Maybe you could instead just have it use some % of the rows going

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeff Davis
On Thu, 2011-02-03 at 16:50 -0500, Mladen Gogala wrote: > Chris Browne wrote: > > Well, the community declines to add hints until there is actual > > consensus on a good way to add hints. > > > OK. That's another matter entirely. Who should make that decision? Is > there a committee or a pers

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 5:39 PM, Mladen Gogala wrote: > Actually, it is not unlike a religious dogma, only stating that "hints are > bad". It even says so in the wiki. The arguments are There's been considerably more output than "hints bad! Hulk Smash!" > 1) Refusal to implement hints is motivat

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Conor Walsh
On Thu, Feb 3, 2011 at 6:33 PM, Joshua D. Drake wrote: > Well that already happens... My understanding is that auto-analyze will fire only after my transaction is completed, because it is a seperate daemon. If I do like so: BEGIN; COPY ...; -- Dangerously un-analyzed SELECT complicated-stuff ..

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:12 -0800, Conor Walsh wrote: > > I can't remember > > anyone ever complaining "ANALYZE took too long to run". I only > > remember complaints of the form "I had to remember to manually run it > > and I wish it had just happened by itself". > > Robert, > > This sounds like

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Conor Walsh
> I can't remember > anyone ever complaining "ANALYZE took too long to run".  I only > remember complaints of the form "I had to remember to manually run it > and I wish it had just happened by itself". Robert, This sounds like an argument in favor of an implicit ANALYZE after all COPY statements

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 8:37 PM, wrote: > On Thu, 3 Feb 2011, Robert Haas wrote: > >> On Thu, Feb 3, 2011 at 7:39 PM,   wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you want to

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 7:39 PM, wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you want to write the code and prove it's better than what we have now, or s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 7:39 PM, Mladen Gogala wrote: >  reality. As a matter of fact, Oracle RDBMS on the same machine will > regularly beat PgSQL in performance. > That has been my experience so far. I even posted counting query results. It sure is, but those count queries didn't run faster b

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 7:39 PM, wrote: >> Yeah, but you'll be passing the entire table through this separate >> process that may only need to see 1% of it or less on a large table. >> If you want to write the code and prove it's better than what we have >> now, or some other approach that someone

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 13:49, Jeremy Harris wrote: On 2011-02-03 21:51, Mark Kirkwood wrote: The cases I've seen in production typically involve "outgrowing" optimizer parameter settings: (e.g work_mem, effective_cache_size) as the application dataset gets bigger over time. An argument in favour of the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Grant Johnson
On PostgreSQL, the difference in no hints and hints for that one query with skewed data is that the query finishes a little faster. On some others, which shall remain nameless, it is the difference between finishing in seconds or days, or maybe never. Hints can be useful, but I can also s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeremy Harris
On 2011-02-03 21:51, Mark Kirkwood wrote: The cases I've seen in production typically involve "outgrowing" optimizer parameter settings: (e.g work_mem, effective_cache_size) as the application dataset gets bigger over time. An argument in favour of the DBMS maintaining a running estimate of s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeremy Harris
On 2011-02-03 23:29, Robert Haas wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. It doesn't sound too impossible to pass only a percentage, starting high and dropping towards 1% once the loaded size h

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Robert Haas wrote: On Thu, Feb 3, 2011 at 6:33 PM, Mladen Gogala wrote: Kevin Grittner wrote: Mladen Gogala wrote: Maybe we can agree to remove that ridiculous "we don't want hints" note from Postgresql wiki? I'd be against that. This is rehashed less frequently s

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 3:54 PM, wrote: with the current code, this is a completely separate process that knows nothing about the load, so if you kick it off when you start the load, it makes a pass over the table (competing for I/O), finishes, you contin

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
> All other databases do have that feature. I must say, this > debate gave me a good deal of stuff to think about. And, I think we're done here. The idea that the lack of hints will kill PostgreSQL is already demonstrably false. This is sounding more and more like a petulant tantrum. Folks,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Craig James
On 2/3/11 1:34 PM, Shaun Thomas wrote: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, yet in the Postgres community, they are considered bad with almost religious fervor. Postgres community is quite unique with the fatwa again

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Joshua D. Drake wrote: On Thu, 2011-02-03 at 18:33 -0500, Mladen Gogala wrote: Exactly what we don't want. Who is "we"? The majority of long term hackers. If that is so, I don't see "world domination" in the future, exactly the opposite. Database whose crea

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 6:33 PM, Mladen Gogala wrote: > Kevin Grittner wrote: >> Mladen Gogala wrote: >>> >>> Maybe we can agree to remove that ridiculous "we don't want hints" >>> note from Postgresql wiki? >>> >> >>  I'd be against that.  This is rehashed less frequently since that >> went in.  

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:33 -0500, Mladen Gogala wrote: > > > > > Exactly what we don't want. > > > Who is "we"? The majority of long term hackers. -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Cust

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Kevin Grittner wrote: Mladen Gogala wrote: Maybe we can agree to remove that ridiculous "we don't want hints" note from Postgresql wiki? I'd be against that. This is rehashed less frequently since that went in. Less wasted time and bandwidth with it there. Well, the problem

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 3:54 PM, wrote: > with the current code, this is a completely separate process that knows > nothing about the load, so if you kick it off when you start the load, it > makes a pass over the table (competing for I/O), finishes, you continue to > update the table, so it makes

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Josh Berkus wrote: I don't want to insult anybody but the whole thing does look strange. Maybe we can agree to remove that ridiculous "we don't want hints" note from Postgresql wiki? That would make it look less like , hmph, philosophical issue and more "not yet implemented" issue, especially if

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Andrew Dunstan
On 02/02/2011 07:17 PM, Greg Smith wrote: I direct anyone who thought Mladen was making a serious comment to http://www.nydailynews.com/news/politics/2009/01/08/2009-01-08_misunderestimate_tops_list_of_notable_bu-3.html if you want to get his little joke there. I plan to start using "misunde

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Justin Pitts
Thank you. It appears I owe an apology also, for jumping to that conclusion. It was rash and unfair of me. I am sorry. On Wed, Feb 2, 2011 at 5:03 PM, Mladen Gogala wrote: > Justin Pitts wrote: >>> >>> With all >>> due respect, I consider myself smarter than the optimizer.  I'm 6'4", >>> 235LBS

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Justin Pitts
> With all > due respect, I consider myself smarter than the optimizer.  I'm 6'4", 235LBS > so telling me that you disagree and that I am more stupid than a computer > program,  would not be a smart thing to do. Please, do not misunderestimate > me. I don't see computer programs make thinly veiled

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kevin Grittner
Mladen Gogala wrote: > Maybe we can agree to remove that ridiculous "we don't want hints" > note from Postgresql wiki? I'd be against that. This is rehashed less frequently since that went in. Less wasted time and bandwidth with it there. > That would make it look less like , hmph, philoso

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Josh Berkus
> I don't want to insult anybody but the whole thing does look strange. > Maybe we can agree to remove that ridiculous "we don't want hints" note > from Postgresql wiki? That would make it look less like , hmph, > philosophical issue and more "not yet implemented" issue, especially if > we have in

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Michael Glaesemann
On Feb 3, 2011, at 17:08, Josh Berkus wrote: > On 2/3/11 1:18 PM, Chris Browne wrote: >> mladen.gog...@vmsinfo.com (Mladen Gogala) writes: >>> I must say that this purist attitude is extremely surprising to >>> me. All the major DB vendors support optimizer hints, > > I don't think that's actual

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Mark Kirkwood wrote: On 04/02/11 11:08, Josh Berkus wrote: I don't think that's actually accurate. Can you give me a list of DBMSes which support hints other than Oracle? DB2 LUW (Linux, Unix, Win32 code base) has hint profiles: http://justdb2chatter.blogspot.com/2008/06/db2-hints-op

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Maciek Sakrejda wrote: The hints are there because they are definitely needed. Yet, there is a religious zeal and a fatwa against them. The opposition is philosophical, not "religious". There is no "fatwa". If you want a serious discussion, avoid inflammatory terms. I don't want to in

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
da...@lang.hm wrote: I am making the assumption that an Analyze run only has to go over the data once (a seqential scan of the table if it's >> ram for example) and gathers stats as it goes. And that's the part there's some confusion about here. ANALYZE grabs a random set of samples from the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 11:08, Josh Berkus wrote: I don't think that's actually accurate. Can you give me a list of DBMSes which support hints other than Oracle? DB2 LUW (Linux, Unix, Win32 code base) has hint profiles: http://justdb2chatter.blogspot.com/2008/06/db2-hints-optimizer-selection.html -- Se

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Maciek Sakrejda
> The hints are there because they are definitely needed. Yet, there is a > religious zeal and a fatwa against them. The opposition is philosophical, not "religious". There is no "fatwa". If you want a serious discussion, avoid inflammatory terms. --- Maciek Sakrejda | System Architect | Truviso

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Mladen Gogala wrote: Actually, I don't want Oracle hints. Oracle hints are ugly and cumbersome. I would prefer something like this: http://dev.mysql.com/doc/refman/5.0/en/index-hints.html That should also answer the question about other databases supporting hints. Sorry. I forgot that MyS

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Josh Berkus wrote: However, since this system wasn't directly compatible with Oracle Hints, folks pushing for hints dropped the solution as unsatisfactory. This is the discussion we have every time: the users who want hints specifically want hints which work exactly like Oracle's, and aren't inte

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Pavel Stehule
> In the meantime, the other databases provide hints which help me bridge the > gap. As I said before: hints are there, even if they were not meant to be > used that way. I can do things in a way that I consider very non-elegant. > The hints are there because they are definitely needed. Yet, there

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Josh Berkus
On 2/3/11 1:18 PM, Chris Browne wrote: > mladen.gog...@vmsinfo.com (Mladen Gogala) writes: >> I must say that this purist attitude is extremely surprising to >> me. All the major DB vendors support optimizer hints, I don't think that's actually accurate. Can you give me a list of DBMSes which sup

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Pavel Stehule
2011/2/3 Mladen Gogala : > Chris Browne wrote: >> >> Well, the community declines to add hints until there is actual >> consensus on a good way to add hints. >> > > OK. That's another matter entirely.   Who should make that decision? Is > there a committee or a person who would be capable of making

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Ben Chobot
On Feb 3, 2011, at 1:50 PM, Mladen Gogala wrote: > So, I will have to go back on my decision to use Postgres and re-consider > MySQL? I will rather throw away the effort invested in studying Postgres than > to risk an unfixable application downtime. I am not sure about the world > domination t

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Shaun Thomas wrote: You missed the argument. The community, or at least the devs, see hints as an ugly hack. Do I agree? Not completely, but I can definitely understand the perspective. Saying every other "vendor" has hints is really just admitting every other vendor has a crappy optimizer. Is

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 10:01, Mladen Gogala wrote: In the meantime, the fire is burning. What should the hapless owner of the database application do in the meantime? Tell the users that it will be better in the next version? As I've said before: hints are make it or break it point. Without hints, I canno

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Chris Browne wrote: Well, the community declines to add hints until there is actual consensus on a good way to add hints. OK. That's another matter entirely. Who should make that decision? Is there a committee or a person who would be capable of making that decision? Nobody has ever prop

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
On 02/03/2011 03:01 PM, Mladen Gogala wrote: As I hinted before, this is actually a purist argument which was made by someone who has never had to support a massive production database with many users for living. Our database handles 9000 transactions per second and over 200-million transacti

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kevin Grittner
Mladen Gogala wrote: > In the meantime, the fire is burning. What should the hapless > owner of the database application do in the meantime? Tell the > users that it will be better in the next version? As I've said > before: hints are make it or break it point. Without hints, I > cannot consider

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Chris Browne
mladen.gog...@vmsinfo.com (Mladen Gogala) writes: > I must say that this purist attitude is extremely surprising to > me. All the major DB vendors support optimizer hints, yet in the > Postgres community, they are considered bad with almost religious > fervor. > Postgres community is quite unique w

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Shaun Thomas wrote: On 02/03/2011 10:38 AM, Mladen Gogala wrote: It all boils down to the database. Hints, whether they're well-intentioned or not, effectively cover up bugs in the optimizer, planner, or some other approach the database is using to build its execution. Hints don't cover u

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 5:11 AM, wrote: If I am understanding things correctly, a full Analyze is going over all the data in the table to figure out patterns. No. It's going over a small, fixed-size sample which depends on default_statistics_target but

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Ross J. Reedstrom
On Thu, Feb 03, 2011 at 12:44:23PM -0500, Chris Browne wrote: > mladen.gog...@vmsinfo.com (Mladen Gogala) writes: > > Hints are not even that complicated to program. The SQL parser should > > compile the list of hints into a table and optimizer should check > > whether any of the applicable access

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Chris Browne wrote: It's worth looking back to what has already been elaborated on in the ToDo. And that precisely is what I am trying to contest. -- Mladen Gogala Sr. Oracle DBA 1500 Broadway New York, NY 10036 (212) 329-5251 http://www.vmsinfo.com The Leader in Integrated Media Intel

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Mladen Gogala wrote: With all due respect, I don't see how does the issue of hints fall into this category? As I explained, the mechanisms are already there, they're just not elegant enough. You're making some assumptions about what a more elegant mechanism would look to develop that are simp

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Chris Browne
mladen.gog...@vmsinfo.com (Mladen Gogala) writes: > Hints are not even that complicated to program. The SQL parser should > compile the list of hints into a table and optimizer should check > whether any of the applicable access methods exist in the table. If it > does - use it. If not, ignore it.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Tom Lane
Robert Haas writes: > I don't, however, agree with his contention that this is easy to > implement. It would be easy to implement something that sucked. It > would be hard to implement something that actually helped in the cases > where the existing settings aren't already sufficient. Exactly.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 11:56 AM, Bruce Momjian wrote: > The settings are currently there to better model the real world > (random_page_cost), or for testing (enable_seqscan).  They are not there > to force certain plans.  They can be used for that, but that is not > their purpose and they would no

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Tom Lane
Mladen Gogala writes: > Hints are not even that complicated to program. With all due respect, you don't know what you're talking about. regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription:

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
On 02/03/2011 10:38 AM, Mladen Gogala wrote: With all due respect, I don't see how does the issue of hints fall into this category? You have a few good arguments, and if you hadn't said this, it wouldn't have been so obvious that there was a fundamental philosophical disconnect. I asked this

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Bruce Momjian
Mladen Gogala wrote: > Hints are not even that complicated to program. The SQL parser should > compile the list of hints into a table and optimizer should check > whether any of the applicable access methods exist in the table. If it > does - use it. If not, ignore it. This looks to me like a ph

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Bruce Momjian
Mladen Gogala wrote: > Greg Smith wrote: > > Mladen Gogala wrote: > > > >> The techies at big companies are the guys who will or will not make it > >> happen. And these guys are not beginners. Appeasing them may actually > >> go a long way. > >> > > > > The PostgreSQL community isn't rea

  1   2   3   >