Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-09 Thread Gorshkov
On 2011-02-03 22:48, Scott Marlowe wrote: On Thu, Feb 3, 2011 at 8:40 PM, Greg Smithg...@2ndquadrant.com wrote: Scott Marlowe wrote: Yes they're useful, but like a plastic bad covering a broken car window, they're useful because they cover something that's inherently broken. Awesome. Now

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-06 Thread Andrew Dunstan
On 02/04/2011 02:32 AM, da...@lang.hm wrote: when a copy command is issued, I assume that there is some indication of how much data is going to follow. No of course there isn't. How would we do that with a stream like STDIN? Read the code. cheers andrew -- Sent via

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-06 Thread Nick Lello
Informix IDS supports hints as well; normally the only need for hints in this engine is when the Table/Index statistics are not being updated on a regular basis (ie: lazy DBA). On 3 February 2011 22:17, Mark Kirkwood mark.kirkw...@catalyst.net.nzwrote: On 04/02/11 11:08, Josh Berkus wrote: I

Re: Does auto-analyze work on dirty writes? (was: Re: [HACKERS] [PERFORM] Slow count(*) again...)

2011-02-06 Thread Tom Lane
Mark Mielke m...@mark.mielke.cc writes: My understanding is: 1) Background daemon wakes up and checks whether a number of changes have happened to the database, irrelevant of transaction boundaries. 2) Background daemon analyzes a percentage of rows in the database for statistical data,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-05 Thread Scott Marlowe
On Fri, Feb 4, 2011 at 11:37 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Feb 5, 2011 at 12:46 AM,  da...@lang.hm wrote: Actually for me the main con with streaming analyze is that it adds significant CPU burden to already not too fast load process. Especially if it's automatically

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-05 Thread Greg Smith
Scott Marlowe wrote: With a 24 drive RAID-10 array that can read at ~1GB/s I am almost always CPU bound during copies. This isn't wholly bad as it leaves spare IO for the rest of the machine so regular work carries on just fine. And you don't need nearly that much I/O bandwidth to reach

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Torsten Zühlsdorff
Mladen Gogala schrieb: Well, the problem will not go away. As I've said before, all other databases have that feature and none of the reasons listed here convinced me that everybody else has a crappy optimizer. The problem may go away altogether if people stop using PostgreSQL. A common

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Grant Johnson
Yes. And this has little to do with hints. It has to do with years of development lead with THOUSANDS of engineers who can work on the most esoteric corner cases in their spare time. Find the pg project a couple hundred software engineers and maybe we'll catch Oracle a little quicker.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Kenneth Marshall
On Thu, Feb 03, 2011 at 04:39:12PM -0800, da...@lang.hm wrote: On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 3:54 PM, da...@lang.hm wrote: with the current code, this is a completely separate process that knows nothing about the load, so if you kick it off when you start the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Vitalii Tymchyshyn
04.02.11 16:33, Kenneth Marshall написав(ла): In addition, the streaming ANALYZE can provide better statistics at any time during the load and it will be complete immediately. As far as passing the entire table through the ANALYZE process, a simple counter can be used to only send the required

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Kenneth Marshall
On Thu, Feb 03, 2011 at 09:05:52PM -0500, Robert Haas wrote: On Thu, Feb 3, 2011 at 8:37 PM, da...@lang.hm wrote: On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 7:39 PM, ?da...@lang.hm wrote: Yeah, but you'll be passing the entire table through this separate process

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Scott Marlowe
On Fri, Feb 4, 2011 at 6:05 AM, Grant Johnson gr...@amadensor.com wrote: Yes.  And this has little to do with hints.  It has to do with years of development lead with THOUSANDS of engineers who can work on the most esoteric corner cases in their spare time.  Find the pg project a couple

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Robert Haas
On Fri, Feb 4, 2011 at 9:38 AM, Vitalii Tymchyshyn tiv...@gmail.com wrote: Actually for me the main con with streaming analyze is that it adds significant CPU burden to already not too fast load process. Exactly. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Bruce Momjian
Greg Smith wrote: Check out http://www.indeed.com/jobtrends?q=postgres%2C+mysql%2C+oraclerelative=1relative=1 if you want to see the real story here. Oracle has a large installed base, but it's considered a troublesome legacy product being replaced +1 for Oracle being a troublesome

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Bruce Momjian
Mladen Gogala wrote: Chris Browne wrote: Well, the community declines to add hints until there is actual consensus on a good way to add hints. OK. That's another matter entirely. Who should make that decision? Is there a committee or a person who would be capable of making that

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Bruce Momjian
Mladen Gogala wrote: Actually, it is not unlike a religious dogma, only stating that hints are bad. It even says so in the wiki. The arguments are 1) Refusal to implement hints is motivated by distrust toward users, citing that some people may mess things up. Yes, they can, with and

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Scott Marlowe
On Fri, Feb 4, 2011 at 5:17 PM, Bruce Momjian br...@momjian.us wrote: Mladen Gogala wrote: characteristic of a religious community chastising a sinner. Let me remind you again: all other major databases have that possibility: Oracle, MySQL, DB2, SQL Server and Informix. Requiring burden of

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread david
On Fri, 4 Feb 2011, Vitalii Tymchyshyn wrote: 04.02.11 16:33, Kenneth Marshall ???(??): In addition, the streaming ANALYZE can provide better statistics at any time during the load and it will be complete immediately. As far as passing the entire table through the ANALYZE process, a

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-04 Thread Robert Haas
On Sat, Feb 5, 2011 at 12:46 AM, da...@lang.hm wrote: Actually for me the main con with streaming analyze is that it adds significant CPU burden to already not too fast load process. Especially if it's automatically done for any load operation performed (and I can't see how it can be enabled

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Vitalii Tymchyshyn
02.02.11 20:32, Robert Haas написав(ла): Yeah. Any kind of bulk load into an empty table can be a problem, even if it's not temporary. When you load a bunch of data and then immediately plan a query against it, autoanalyze hasn't had a chance to do its thing yet, so sometimes you get a lousy

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Vitalii Tymchyshyn wrote: 02.02.11 20:32, Robert Haas ???(??): Yeah. Any kind of bulk load into an empty table can be a problem, even if it's not temporary. When you load a bunch of data and then immediately plan a query against it, autoanalyze hasn't had a chance to

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kenneth Marshall
On Thu, Feb 03, 2011 at 02:11:58AM -0800, da...@lang.hm wrote: On Thu, 3 Feb 2011, Vitalii Tymchyshyn wrote: 02.02.11 20:32, Robert Haas ???(??): Yeah. Any kind of bulk load into an empty table can be a problem, even if it's not temporary. When you load a bunch of data and then

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jon Nelson
On Thu, Feb 3, 2011 at 7:41 AM, Kenneth Marshall k...@rice.edu wrote: On Thu, Feb 03, 2011 at 02:11:58AM -0800, da...@lang.hm wrote: On Thu, 3 Feb 2011, Vitalii Tymchyshyn wrote: 02.02.11 20:32, Robert Haas ???(??): Yeah.  Any kind of bulk load into an empty table can be a problem, even

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 4:54 AM, Vitalii Tymchyshyn tiv...@gmail.com wrote: 02.02.11 20:32, Robert Haas написав(ла): Yeah.  Any kind of bulk load into an empty table can be a problem, even if it's not temporary.  When you load a bunch of data and then immediately plan a query against it,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 5:11 AM, da...@lang.hm wrote: If I am understanding things correctly, a full Analyze is going over all the data in the table to figure out patterns. No. It's going over a small, fixed-size sample which depends on default_statistics_target but NOT on the table size.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Vitalii Tymchyshyn
03.02.11 17:31, Robert Haas написав(ла): May be introducing something like 'AutoAnalyze' threshold will help? I mean that any insert/update/delete statement that changes more then x% of table (and no less then y records) must do analyze right after it was finished. Defaults like x=50 y=1

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Greg Smith wrote: Mladen Gogala wrote: The techies at big companies are the guys who will or will not make it happen. And these guys are not beginners. Appeasing them may actually go a long way. The PostgreSQL community isn't real big on appeasing people if it's at the expense of

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Bruce Momjian
Mladen Gogala wrote: Greg Smith wrote: Mladen Gogala wrote: The techies at big companies are the guys who will or will not make it happen. And these guys are not beginners. Appeasing them may actually go a long way. The PostgreSQL community isn't real big on appeasing

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Bruce Momjian
Mladen Gogala wrote: Hints are not even that complicated to program. The SQL parser should compile the list of hints into a table and optimizer should check whether any of the applicable access methods exist in the table. If it does - use it. If not, ignore it. This looks to me like a

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
On 02/03/2011 10:38 AM, Mladen Gogala wrote: With all due respect, I don't see how does the issue of hints fall into this category? You have a few good arguments, and if you hadn't said this, it wouldn't have been so obvious that there was a fundamental philosophical disconnect. I asked

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Tom Lane
Mladen Gogala mladen.gog...@vmsinfo.com writes: Hints are not even that complicated to program. With all due respect, you don't know what you're talking about. regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 11:56 AM, Bruce Momjian br...@momjian.us wrote: The settings are currently there to better model the real world (random_page_cost), or for testing (enable_seqscan).  They are not there to force certain plans.  They can be used for that, but that is not their purpose and

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I don't, however, agree with his contention that this is easy to implement. It would be easy to implement something that sucked. It would be hard to implement something that actually helped in the cases where the existing settings aren't already

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Chris Browne
mladen.gog...@vmsinfo.com (Mladen Gogala) writes: Hints are not even that complicated to program. The SQL parser should compile the list of hints into a table and optimizer should check whether any of the applicable access methods exist in the table. If it does - use it. If not, ignore it.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Mladen Gogala wrote: With all due respect, I don't see how does the issue of hints fall into this category? As I explained, the mechanisms are already there, they're just not elegant enough. You're making some assumptions about what a more elegant mechanism would look to develop that are

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Chris Browne wrote: It's worth looking back to what has already been elaborated on in the ToDo. And that precisely is what I am trying to contest. -- Mladen Gogala Sr. Oracle DBA 1500 Broadway New York, NY 10036 (212) 329-5251 http://www.vmsinfo.com The Leader in Integrated Media

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Ross J. Reedstrom
On Thu, Feb 03, 2011 at 12:44:23PM -0500, Chris Browne wrote: mladen.gog...@vmsinfo.com (Mladen Gogala) writes: Hints are not even that complicated to program. The SQL parser should compile the list of hints into a table and optimizer should check whether any of the applicable access

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 5:11 AM, da...@lang.hm wrote: If I am understanding things correctly, a full Analyze is going over all the data in the table to figure out patterns. No. It's going over a small, fixed-size sample which depends on

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Shaun Thomas wrote: On 02/03/2011 10:38 AM, Mladen Gogala wrote: It all boils down to the database. Hints, whether they're well-intentioned or not, effectively cover up bugs in the optimizer, planner, or some other approach the database is using to build its execution. Hints don't cover

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Chris Browne
mladen.gog...@vmsinfo.com (Mladen Gogala) writes: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, yet in the Postgres community, they are considered bad with almost religious fervor. Postgres community is quite unique with

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kevin Grittner
Mladen Gogala mladen.gog...@vmsinfo.com wrote: In the meantime, the fire is burning. What should the hapless owner of the database application do in the meantime? Tell the users that it will be better in the next version? As I've said before: hints are make it or break it point. Without

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
On 02/03/2011 03:01 PM, Mladen Gogala wrote: As I hinted before, this is actually a purist argument which was made by someone who has never had to support a massive production database with many users for living. Our database handles 9000 transactions per second and over 200-million

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Chris Browne wrote: Well, the community declines to add hints until there is actual consensus on a good way to add hints. OK. That's another matter entirely. Who should make that decision? Is there a committee or a person who would be capable of making that decision? Nobody has ever

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 10:01, Mladen Gogala wrote: In the meantime, the fire is burning. What should the hapless owner of the database application do in the meantime? Tell the users that it will be better in the next version? As I've said before: hints are make it or break it point. Without hints, I

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Shaun Thomas wrote: You missed the argument. The community, or at least the devs, see hints as an ugly hack. Do I agree? Not completely, but I can definitely understand the perspective. Saying every other vendor has hints is really just admitting every other vendor has a crappy optimizer. Is

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Ben Chobot
On Feb 3, 2011, at 1:50 PM, Mladen Gogala wrote: So, I will have to go back on my decision to use Postgres and re-consider MySQL? I will rather throw away the effort invested in studying Postgres than to risk an unfixable application downtime. I am not sure about the world domination

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Pavel Stehule
2011/2/3 Mladen Gogala mladen.gog...@vmsinfo.com: Chris Browne wrote: Well, the community declines to add hints until there is actual consensus on a good way to add hints. OK. That's another matter entirely.   Who should make that decision? Is there a committee or a person who would be

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Josh Berkus
On 2/3/11 1:18 PM, Chris Browne wrote: mladen.gog...@vmsinfo.com (Mladen Gogala) writes: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, I don't think that's actually accurate. Can you give me a list of DBMSes which support

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Pavel Stehule
In the meantime, the other databases provide hints which help me bridge the gap. As I said before: hints are there, even if they were not meant to be used that way. I can do things in a way that I consider very non-elegant. The hints are there because they are definitely needed. Yet, there is

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Josh Berkus wrote: However, since this system wasn't directly compatible with Oracle Hints, folks pushing for hints dropped the solution as unsatisfactory. This is the discussion we have every time: the users who want hints specifically want hints which work exactly like Oracle's, and aren't

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Mladen Gogala wrote: Actually, I don't want Oracle hints. Oracle hints are ugly and cumbersome. I would prefer something like this: http://dev.mysql.com/doc/refman/5.0/en/index-hints.html That should also answer the question about other databases supporting hints. Sorry. I forgot that

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Maciek Sakrejda
The hints are there because they are definitely needed. Yet, there is a religious zeal and a fatwa against them. The opposition is philosophical, not religious. There is no fatwa. If you want a serious discussion, avoid inflammatory terms. --- Maciek Sakrejda | System Architect | Truviso 1065

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 11:08, Josh Berkus wrote: I don't think that's actually accurate. Can you give me a list of DBMSes which support hints other than Oracle? DB2 LUW (Linux, Unix, Win32 code base) has hint profiles: http://justdb2chatter.blogspot.com/2008/06/db2-hints-optimizer-selection.html --

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
da...@lang.hm wrote: I am making the assumption that an Analyze run only has to go over the data once (a seqential scan of the table if it's ram for example) and gathers stats as it goes. And that's the part there's some confusion about here. ANALYZE grabs a random set of samples from the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Maciek Sakrejda wrote: The hints are there because they are definitely needed. Yet, there is a religious zeal and a fatwa against them. The opposition is philosophical, not religious. There is no fatwa. If you want a serious discussion, avoid inflammatory terms. I don't want to

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Mark Kirkwood wrote: On 04/02/11 11:08, Josh Berkus wrote: I don't think that's actually accurate. Can you give me a list of DBMSes which support hints other than Oracle? DB2 LUW (Linux, Unix, Win32 code base) has hint profiles:

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Michael Glaesemann
On Feb 3, 2011, at 17:08, Josh Berkus wrote: On 2/3/11 1:18 PM, Chris Browne wrote: mladen.gog...@vmsinfo.com (Mladen Gogala) writes: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, I don't think that's actually

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Josh Berkus
I don't want to insult anybody but the whole thing does look strange. Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? That would make it look less like , hmph, philosophical issue and more not yet implemented issue, especially if we have in mind

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kevin Grittner
Mladen Gogala mladen.gog...@vmsinfo.com wrote: Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? I'd be against that. This is rehashed less frequently since that went in. Less wasted time and bandwidth with it there. That would make it look less

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Justin Pitts
With all due respect, I consider myself smarter than the optimizer.  I'm 6'4, 235LBS so telling me that you disagree and that I am more stupid than a computer program,  would not be a smart thing to do. Please, do not misunderestimate me. I don't see computer programs make thinly veiled

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Justin Pitts
Thank you. It appears I owe an apology also, for jumping to that conclusion. It was rash and unfair of me. I am sorry. On Wed, Feb 2, 2011 at 5:03 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Justin Pitts wrote: With all due respect, I consider myself smarter than the optimizer.  I'm

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Andrew Dunstan
On 02/02/2011 07:17 PM, Greg Smith wrote: I direct anyone who thought Mladen was making a serious comment to http://www.nydailynews.com/news/politics/2009/01/08/2009-01-08_misunderestimate_tops_list_of_notable_bu-3.html if you want to get his little joke there. I plan to start using

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Josh Berkus wrote: I don't want to insult anybody but the whole thing does look strange. Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? That would make it look less like , hmph, philosophical issue and more not yet implemented issue, especially if we

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 3:54 PM, da...@lang.hm wrote: with the current code, this is a completely separate process that knows nothing about the load, so if you kick it off when you start the load, it makes a pass over the table (competing for I/O), finishes, you continue to update the table,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Kevin Grittner wrote: Mladen Gogala mladen.gog...@vmsinfo.com wrote: Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? I'd be against that. This is rehashed less frequently since that went in. Less wasted time and bandwidth with it there.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:33 -0500, Mladen Gogala wrote: Exactly what we don't want. Who is we? The majority of long term hackers. -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 6:33 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Kevin Grittner wrote: Mladen Gogala mladen.gog...@vmsinfo.com wrote: Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki?  I'd be against that.  This is rehashed less

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Joshua D. Drake wrote: On Thu, 2011-02-03 at 18:33 -0500, Mladen Gogala wrote: Exactly what we don't want. Who is we? The majority of long term hackers. If that is so, I don't see world domination in the future, exactly the opposite. Database whose

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Craig James
On 2/3/11 1:34 PM, Shaun Thomas wrote: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, yet in the Postgres community, they are considered bad with almost religious fervor. Postgres community is quite unique with the fatwa

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
All other databases do have that feature. I must say, this debate gave me a good deal of stuff to think about. And, I think we're done here. The idea that the lack of hints will kill PostgreSQL is already demonstrably false. This is sounding more and more like a petulant tantrum. Folks, I

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 3:54 PM, da...@lang.hm wrote: with the current code, this is a completely separate process that knows nothing about the load, so if you kick it off when you start the load, it makes a pass over the table (competing for I/O),

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Robert Haas wrote: On Thu, Feb 3, 2011 at 6:33 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Kevin Grittner wrote: Mladen Gogala mladen.gog...@vmsinfo.com wrote: Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? I'd be

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeremy Harris
On 2011-02-03 23:29, Robert Haas wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. It doesn't sound too impossible to pass only a percentage, starting high and dropping towards 1% once the loaded size

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeremy Harris
On 2011-02-03 21:51, Mark Kirkwood wrote: The cases I've seen in production typically involve outgrowing optimizer parameter settings: (e.g work_mem, effective_cache_size) as the application dataset gets bigger over time. An argument in favour of the DBMS maintaining a running estimate of

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Grant Johnson
On PostgreSQL, the difference in no hints and hints for that one query with skewed data is that the query finishes a little faster. On some others, which shall remain nameless, it is the difference between finishing in seconds or days, or maybe never. Hints can be useful, but I can also

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 13:49, Jeremy Harris wrote: On 2011-02-03 21:51, Mark Kirkwood wrote: The cases I've seen in production typically involve outgrowing optimizer parameter settings: (e.g work_mem, effective_cache_size) as the application dataset gets bigger over time. An argument in favour of the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 7:39 PM, da...@lang.hm wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you want to write the code and prove it's better than what we have now, or some other approach that

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 7:39 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote:  reality. As a matter of fact, Oracle RDBMS on the same machine will regularly beat PgSQL in performance. That has been my experience so far. I even posted counting query results. It sure is, but those count

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 7:39 PM, da...@lang.hm wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you want to write the code and prove it's better than what we

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 8:37 PM, da...@lang.hm wrote: On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 7:39 PM,  da...@lang.hm wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Conor Walsh
I can't remember anyone ever complaining ANALYZE took too long to run.  I only remember complaints of the form I had to remember to manually run it and I wish it had just happened by itself. Robert, This sounds like an argument in favor of an implicit ANALYZE after all COPY statements,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:12 -0800, Conor Walsh wrote: I can't remember anyone ever complaining ANALYZE took too long to run. I only remember complaints of the form I had to remember to manually run it and I wish it had just happened by itself. Robert, This sounds like an argument in

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Conor Walsh
On Thu, Feb 3, 2011 at 6:33 PM, Joshua D. Drake j...@commandprompt.com wrote: Well that already happens... My understanding is that auto-analyze will fire only after my transaction is completed, because it is a seperate daemon. If I do like so: BEGIN; COPY ...; -- Dangerously un-analyzed

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 5:39 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Actually, it is not unlike a religious dogma, only stating that hints are bad. It even says so in the wiki. The arguments are There's been considerably more output than hints bad! Hulk Smash! 1) Refusal to

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeff Davis
On Thu, 2011-02-03 at 16:50 -0500, Mladen Gogala wrote: Chris Browne wrote: Well, the community declines to add hints until there is actual consensus on a good way to add hints. OK. That's another matter entirely. Who should make that decision? Is there a committee or a person who

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 7:05 PM, Robert Haas robertmh...@gmail.com wrote: If you want to randomly pick 10,000 rows out of all the rows that are going to be inserted in the table without knowing in advance how many there will be, how do you do that? Maybe you could instead just have it use some

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: Yes they're useful, but like a plastic bad covering a broken car window, they're useful because they cover something that's inherently broken. Awesome. Now we have a car anology, with a funny typo no less. Plastic bad, I love it. This is real progress toward getting

Does auto-analyze work on dirty writes? (was: Re: [HACKERS] [PERFORM] Slow count(*) again...)

2011-02-03 Thread Mark Mielke
On 02/03/2011 09:45 PM, Conor Walsh wrote: My understanding is that auto-analyze will fire only after my transaction is completed, because it is a seperate daemon. If I do like so: BEGIN; COPY ...; -- Dangerously un-analyzed SELECT complicated-stuff ...; END; Auto-analyze does not benefit me,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:40 PM, Greg Smith g...@2ndquadrant.com wrote: Scott Marlowe wrote: Yes they're useful, but like a plastic bad covering a broken car window, they're useful because they cover something that's inherently broken. Awesome.  Now we have a car anology, with a funny typo

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: It's not so much a car analogy as a plastic bad analogy. Is that like a Plastic Ono Band? Because I think one of those is the only thing holding the part of my bumper I smashed in the snow on right now. I could be wrong about the name. -- Greg Smith 2ndQuadrant

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:56 PM, Greg Smith g...@2ndquadrant.com wrote: Scott Marlowe wrote: It's not so much a car analogy as a plastic bad analogy. Is that like a Plastic Ono Band?  Because I think one of those is the only thing holding the part of my bumper I smashed in the snow on right

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: No, that's a plastic oh no! band you have. Wow, right you are. So with this type holding together my Japanese car, if it breaks and parts fall off, I'm supposed to yell Oh, no! There goes Tokyo!, yes? -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread David Wilson
On Thu, Feb 3, 2011 at 6:05 PM, Robert Haas robertmh...@gmail.com wrote: If you want to randomly pick 10,000 rows out of all the rows that are going to be inserted in the table without knowing in advance how many there will be, how do you do that? Reservoir sampling, as the most well-known

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Samuel Gendler
Neat. That was my 'you learn something every day' moment. Thanks. On Thu, Feb 3, 2011 at 9:06 PM, David Wilson david.t.wil...@gmail.comwrote: On Thu, Feb 3, 2011 at 6:05 PM, Robert Haas robertmh...@gmail.com wrote: If you want to randomly pick 10,000 rows out of all the rows that are

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Віталій Тимчишин
2011/2/3 da...@lang.hm If the table is not large enough to fit in ram, then it will compete for I/O, and the user will have to wait. what I'm proposing is that as the records are created, the process doing the creation makes copies of the records (either all of them, or some of them if not

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Віталій Тимчишин
2011/2/4 Mladen Gogala mladen.gog...@vmsinfo.com Josh Berkus wrote: However, since this system wasn't directly compatible with Oracle Hints, folks pushing for hints dropped the solution as unsatisfactory. This is the discussion we have every time: the users who want hints specifically want

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Fri, 4 Feb 2011, ??? wrote: 2011/2/3 da...@lang.hm If the table is not large enough to fit in ram, then it will compete for I/O, and the user will have to wait. what I'm proposing is that as the records are created, the process doing the creation makes copies of the records

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Віталій Тимчишин
4 лютого 2011 р. 09:32 da...@lang.hm написав: when a copy command is issued, I assume that there is some indication of how much data is going to follow. I know that it's not just 'insert everything until the TCP connection terminates' because that would give you no way of knowing if the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-02 Thread Robert Haas
On Tue, Feb 1, 2011 at 6:44 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: On 2/1/2011 6:03 PM, Andrew Dunstan wrote: Whether or not it's bad application design, it's ubiquitous, and we should make it work as best we can, IMNSHO. This often generates complaints about Postgres, and if we

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-02 Thread Mladen Gogala
Robert Haas wrote: On Tue, Feb 1, 2011 It would be pretty hard to make autoanalyze work on such tables without removing some of the performance benefits of having such tables in the first place - namely, the local buffer manager. But you could ANALYZE them by hand. Not necessarily

  1   2   >