Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Vitalii Tymchyshyn
02.02.11 20:32, Robert Haas написав(ла): Yeah. Any kind of bulk load into an empty table can be a problem, even if it's not temporary. When you load a bunch of data and then immediately plan a query against it, autoanalyze hasn't had a chance to do its thing yet, so sometimes you get a lousy

Re: [PERFORM] Which RAID Controllers to pick/avoid?

2011-02-03 Thread Glyn Astill
--- On Thu, 3/2/11, Greg Smith g...@2ndquadrant.com wrote: The 5405 and 5805 models do have a known problem where they overheat if you don't have enough cooling in the server box, with the 5805 seeming to be the bigger source of such issues.  See the reviews at

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Vitalii Tymchyshyn wrote: 02.02.11 20:32, Robert Haas ???(??): Yeah. Any kind of bulk load into an empty table can be a problem, even if it's not temporary. When you load a bunch of data and then immediately plan a query against it, autoanalyze hasn't had a chance to

[PERFORM] Get master-detail relationship metadata

2011-02-03 Thread Laszlo Nagy
Hi All, I'm working on a client program that iterates over master-detail relationships in a loop chain. Pseudo code: for row_1 in table_1: table_2 = get_details(row_1,table2) for row_2 in table_2: row_3 = get_details(row_2,table3) etc.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kenneth Marshall
On Thu, Feb 03, 2011 at 02:11:58AM -0800, da...@lang.hm wrote: On Thu, 3 Feb 2011, Vitalii Tymchyshyn wrote: 02.02.11 20:32, Robert Haas ???(??): Yeah. Any kind of bulk load into an empty table can be a problem, even if it's not temporary. When you load a bunch of data and then

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jon Nelson
On Thu, Feb 3, 2011 at 7:41 AM, Kenneth Marshall k...@rice.edu wrote: On Thu, Feb 03, 2011 at 02:11:58AM -0800, da...@lang.hm wrote: On Thu, 3 Feb 2011, Vitalii Tymchyshyn wrote: 02.02.11 20:32, Robert Haas ???(??): Yeah.  Any kind of bulk load into an empty table can be a problem, even

Re: [PERFORM] Which RAID Controllers to pick/avoid?

2011-02-03 Thread Greg Smith
Dan Birken wrote: - Is the supercap + flash memory considered superior to the BBU in practice? Is that type of system well tested? The main risk is that it's a pretty new approach. The standard BBU setup has been used for a long time now; this whole flash+supercap thing has only showed up

Re: [PERFORM] Which RAID Controllers to pick/avoid?

2011-02-03 Thread Mario Weilguni
Am 03.02.2011 00:15, schrieb Dan Birken: I'm setting up a dedicated linux postgres box with 2x300GB 15k SAS drive in a RAID 1, though if future load dictates we would like to be able to upgrade to RAID 10. The hosting provider offers the following options for a RAID controller (all are the

[PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Mark Stosberg
Each night we run over a 100,000 saved searches against PostgreSQL 9.0.x. These are all complex SELECTs using cube functions to perform a geo-spatial search to help people find adoptable pets at shelters. All of our machines in development in production have at least 2 cores in them, and I'm

Re: [PERFORM] Get master-detail relationship metadata

2011-02-03 Thread Andy Colson
On 2/3/2011 5:40 AM, Laszlo Nagy wrote: Hi All, I'm working on a client program that iterates over master-detail relationships in a loop chain. Pseudo code: for row_1 in table_1: table_2 = get_details(row_1,table2) for row_2 in table_2: row_3 = get_details(row_2,table3) etc.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 4:54 AM, Vitalii Tymchyshyn tiv...@gmail.com wrote: 02.02.11 20:32, Robert Haas написав(ла): Yeah.  Any kind of bulk load into an empty table can be a problem, even if it's not temporary.  When you load a bunch of data and then immediately plan a query against it,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 5:11 AM, da...@lang.hm wrote: If I am understanding things correctly, a full Analyze is going over all the data in the table to figure out patterns. No. It's going over a small, fixed-size sample which depends on default_statistics_target but NOT on the table size.

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Andy Colson
On 2/3/2011 9:08 AM, Mark Stosberg wrote: Each night we run over a 100,000 saved searches against PostgreSQL 9.0.x. These are all complex SELECTs using cube functions to perform a geo-spatial search to help people find adoptable pets at shelters. All of our machines in development in

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Vitalii Tymchyshyn
03.02.11 17:31, Robert Haas написав(ла): May be introducing something like 'AutoAnalyze' threshold will help? I mean that any insert/update/delete statement that changes more then x% of table (and no less then y records) must do analyze right after it was finished. Defaults like x=50 y=1

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread gnuoytr
Time for my pet meme to wiggle out of its hole (next to Phil's, and a day later). For PG to prosper in the future, it has to embrace the multi-core/processor/SSD machine at the query level. It has to. And it has to because the Big Boys already do so, to some extent, and they've realized that

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Mark Stosberg
On 02/03/2011 10:54 AM, Oleg Bartunov wrote: Mark, you could try gevel module to get structure of GIST index and look if items distributed more or less homogenous (see different levels). You can visualize index like http://www.sai.msu.su/~megera/wiki/Rtree_Index Also, if your searches are

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Greg Smith wrote: Mladen Gogala wrote: The techies at big companies are the guys who will or will not make it happen. And these guys are not beginners. Appeasing them may actually go a long way. The PostgreSQL community isn't real big on appeasing people if it's at the expense of

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Bruce Momjian
Mladen Gogala wrote: Greg Smith wrote: Mladen Gogala wrote: The techies at big companies are the guys who will or will not make it happen. And these guys are not beginners. Appeasing them may actually go a long way. The PostgreSQL community isn't real big on appeasing

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Bruce Momjian
Mladen Gogala wrote: Hints are not even that complicated to program. The SQL parser should compile the list of hints into a table and optimizer should check whether any of the applicable access methods exist in the table. If it does - use it. If not, ignore it. This looks to me like a

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
On 02/03/2011 10:38 AM, Mladen Gogala wrote: With all due respect, I don't see how does the issue of hints fall into this category? You have a few good arguments, and if you hadn't said this, it wouldn't have been so obvious that there was a fundamental philosophical disconnect. I asked

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Tom Lane
Mladen Gogala mladen.gog...@vmsinfo.com writes: Hints are not even that complicated to program. With all due respect, you don't know what you're talking about. regards, tom lane -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 11:56 AM, Bruce Momjian br...@momjian.us wrote: The settings are currently there to better model the real world (random_page_cost), or for testing (enable_seqscan).  They are not there to force certain plans.  They can be used for that, but that is not their purpose and

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I don't, however, agree with his contention that this is easy to implement. It would be easy to implement something that sucked. It would be hard to implement something that actually helped in the cases where the existing settings aren't already

Re: FW: [PERFORM] Queries becoming slow under heavy load

2011-02-03 Thread Anne Rosset
Thanks to all of you who replied and pointed NFS as a potential culprit. Our issue was that pgsql's temp dir (pgsql_tmp) was set to the default value ( $PSQL_DIR/base/pgsql_tmp/) which was located in NFS. Moving the temp dir to local disk got us a huge improvement. Anne -Original

Re: [PERFORM] Exhaustive list of what takes what locks

2011-02-03 Thread Robert Haas
On Tue, Feb 1, 2011 at 2:18 PM, Nikolas Everett nik9...@gmail.com wrote: This isn't exactly how our workload actually works.  Ours is more deadlock prone.  We have many connections all querying account and we do the migration in a transaction.  It looks as though the AccessExclusiveLock is

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Chris Browne
mladen.gog...@vmsinfo.com (Mladen Gogala) writes: Hints are not even that complicated to program. The SQL parser should compile the list of hints into a table and optimizer should check whether any of the applicable access methods exist in the table. If it does - use it. If not, ignore it.

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Aljoša Mohorović
On Thu, Feb 3, 2011 at 4:57 PM, gnuo...@rcn.com wrote: Time for my pet meme to wiggle out of its hole (next to Phil's, and a day later).  For PG to prosper in the future, it has to embrace the multi-core/processor/SSD machine at the query level.  It has to.  And it has to because the Big

Re: [PERFORM] High load,

2011-02-03 Thread Robert Haas
On Thu, Jan 27, 2011 at 5:31 AM, Michael Kohl michael.k...@tupalo.com wrote: we are running a fairly big Ruby on Rails application on Postgres 8.4. Our traffic grew quite a bit lately, and since then we are facing DB performance issues. System load occasionally explodes (around 170 yesterday

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Mladen Gogala wrote: With all due respect, I don't see how does the issue of hints fall into this category? As I explained, the mechanisms are already there, they're just not elegant enough. You're making some assumptions about what a more elegant mechanism would look to develop that are

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:57 AM, gnuo...@rcn.com wrote: Time for my pet meme to wiggle out of its hole (next to Phil's, and a day later).  For PG to prosper in the future, it has to embrace the multi-core/processor/SSD machine at the query level.  It has to.  And I'm pretty sure multi-core

Re: FW: [PERFORM] Queries becoming slow under heavy load

2011-02-03 Thread Scott Marlowe
Excellent! And you learned a bit more about how to monitor your server while you were at it. Win win! On Thu, Feb 3, 2011 at 10:40 AM, Anne Rosset aros...@collab.net wrote: Thanks to all  of you who replied and pointed NFS as a potential culprit. Our issue was that  pgsql's temp dir

Re: [PERFORM] How to best use 32 15k.7 300GB drives?

2011-02-03 Thread Robert Haas
2011/1/30 Віталій Тимчишин tiv...@gmail.com: I was thinking if a table file could be deleted if it has no single live row. And if this could be done by vacuum. In this case vacuum on table that was fully updated recently could be almost as good as cluster - any scan would skip such

Re: [PERFORM] How to best use 32 15k.7 300GB drives?

2011-02-03 Thread Dave Crooke
There is a process in Oracle which essentially allows you to do the equivalent of a CLUSTER in Postgres, but without locking the table, and so updates can continue throughout the process. It requires a bit of manual setup and fiddling (at least in Oracle 10g) this would probably scratch a lot

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Chris Browne wrote: It's worth looking back to what has already been elaborated on in the ToDo. And that precisely is what I am trying to contest. -- Mladen Gogala Sr. Oracle DBA 1500 Broadway New York, NY 10036 (212) 329-5251 http://www.vmsinfo.com The Leader in Integrated Media

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread gnuoytr
Original message Date: Thu, 3 Feb 2011 18:56:34 +0100 From: pgsql-performance-ow...@postgresql.org (on behalf of Aljoša Mohorović aljosa.mohoro...@gmail.com) Subject: Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements To: gnuo...@rcn.com

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Ross J. Reedstrom
On Thu, Feb 03, 2011 at 12:44:23PM -0500, Chris Browne wrote: mladen.gog...@vmsinfo.com (Mladen Gogala) writes: Hints are not even that complicated to program. The SQL parser should compile the list of hints into a table and optimizer should check whether any of the applicable access

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 5:11 AM, da...@lang.hm wrote: If I am understanding things correctly, a full Analyze is going over all the data in the table to figure out patterns. No. It's going over a small, fixed-size sample which depends on

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Shaun Thomas wrote: On 02/03/2011 10:38 AM, Mladen Gogala wrote: It all boils down to the database. Hints, whether they're well-intentioned or not, effectively cover up bugs in the optimizer, planner, or some other approach the database is using to build its execution. Hints don't cover

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Chris Browne
mladen.gog...@vmsinfo.com (Mladen Gogala) writes: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, yet in the Postgres community, they are considered bad with almost religious fervor. Postgres community is quite unique with

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kevin Grittner
Mladen Gogala mladen.gog...@vmsinfo.com wrote: In the meantime, the fire is burning. What should the hapless owner of the database application do in the meantime? Tell the users that it will be better in the next version? As I've said before: hints are make it or break it point. Without

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
On 02/03/2011 03:01 PM, Mladen Gogala wrote: As I hinted before, this is actually a purist argument which was made by someone who has never had to support a massive production database with many users for living. Our database handles 9000 transactions per second and over 200-million

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Chris Browne wrote: Well, the community declines to add hints until there is actual consensus on a good way to add hints. OK. That's another matter entirely. Who should make that decision? Is there a committee or a person who would be capable of making that decision? Nobody has ever

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 10:01, Mladen Gogala wrote: In the meantime, the fire is burning. What should the hapless owner of the database application do in the meantime? Tell the users that it will be better in the next version? As I've said before: hints are make it or break it point. Without hints, I

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Shaun Thomas wrote: You missed the argument. The community, or at least the devs, see hints as an ugly hack. Do I agree? Not completely, but I can definitely understand the perspective. Saying every other vendor has hints is really just admitting every other vendor has a crappy optimizer. Is

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Ben Chobot
On Feb 3, 2011, at 1:50 PM, Mladen Gogala wrote: So, I will have to go back on my decision to use Postgres and re-consider MySQL? I will rather throw away the effort invested in studying Postgres than to risk an unfixable application downtime. I am not sure about the world domination

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Pavel Stehule
2011/2/3 Mladen Gogala mladen.gog...@vmsinfo.com: Chris Browne wrote: Well, the community declines to add hints until there is actual consensus on a good way to add hints. OK. That's another matter entirely.   Who should make that decision? Is there a committee or a person who would be

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Josh Berkus
On 2/3/11 1:18 PM, Chris Browne wrote: mladen.gog...@vmsinfo.com (Mladen Gogala) writes: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, I don't think that's actually accurate. Can you give me a list of DBMSes which support

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Pavel Stehule
In the meantime, the other databases provide hints which help me bridge the gap. As I said before: hints are there, even if they were not meant to be used that way. I can do things in a way that I consider very non-elegant. The hints are there because they are definitely needed. Yet, there is

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Josh Berkus wrote: However, since this system wasn't directly compatible with Oracle Hints, folks pushing for hints dropped the solution as unsatisfactory. This is the discussion we have every time: the users who want hints specifically want hints which work exactly like Oracle's, and aren't

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Mladen Gogala wrote: Actually, I don't want Oracle hints. Oracle hints are ugly and cumbersome. I would prefer something like this: http://dev.mysql.com/doc/refman/5.0/en/index-hints.html That should also answer the question about other databases supporting hints. Sorry. I forgot that

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Maciek Sakrejda
The hints are there because they are definitely needed. Yet, there is a religious zeal and a fatwa against them. The opposition is philosophical, not religious. There is no fatwa. If you want a serious discussion, avoid inflammatory terms. --- Maciek Sakrejda | System Architect | Truviso 1065

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 11:08, Josh Berkus wrote: I don't think that's actually accurate. Can you give me a list of DBMSes which support hints other than Oracle? DB2 LUW (Linux, Unix, Win32 code base) has hint profiles: http://justdb2chatter.blogspot.com/2008/06/db2-hints-optimizer-selection.html --

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
da...@lang.hm wrote: I am making the assumption that an Analyze run only has to go over the data once (a seqential scan of the table if it's ram for example) and gathers stats as it goes. And that's the part there's some confusion about here. ANALYZE grabs a random set of samples from the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Maciek Sakrejda wrote: The hints are there because they are definitely needed. Yet, there is a religious zeal and a fatwa against them. The opposition is philosophical, not religious. There is no fatwa. If you want a serious discussion, avoid inflammatory terms. I don't want to

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Mark Kirkwood wrote: On 04/02/11 11:08, Josh Berkus wrote: I don't think that's actually accurate. Can you give me a list of DBMSes which support hints other than Oracle? DB2 LUW (Linux, Unix, Win32 code base) has hint profiles:

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Michael Glaesemann
On Feb 3, 2011, at 17:08, Josh Berkus wrote: On 2/3/11 1:18 PM, Chris Browne wrote: mladen.gog...@vmsinfo.com (Mladen Gogala) writes: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, I don't think that's actually

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: On Thu, Feb 3, 2011 at 8:57 AM, gnuo...@rcn.com wrote: Time for my pet meme to wiggle out of its hole (next to Phil's, and a day later). For PG to prosper in the future, it has to embrace the multi-core/processor/SSD machine at the query level. It has to. And

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Josh Berkus
I don't want to insult anybody but the whole thing does look strange. Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? That would make it look less like , hmph, philosophical issue and more not yet implemented issue, especially if we have in mind

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Kevin Grittner
Mladen Gogala mladen.gog...@vmsinfo.com wrote: Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? I'd be against that. This is rehashed less frequently since that went in. Less wasted time and bandwidth with it there. That would make it look less

Re: [PERFORM] Exhaustive list of what takes what locks

2011-02-03 Thread Noah Misch
On Tue, Feb 01, 2011 at 02:18:37PM -0500, Nikolas Everett wrote: Is there an exhaustive list of what takes what locks and how long they last? This documents which commands take each lock type, but it is not exhaustive: http://www.postgresql.org/docs/current/interactive/explicit-locking.html All

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Justin Pitts
With all due respect, I consider myself smarter than the optimizer.  I'm 6'4, 235LBS so telling me that you disagree and that I am more stupid than a computer program,  would not be a smart thing to do. Please, do not misunderestimate me. I don't see computer programs make thinly veiled

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Justin Pitts
Thank you. It appears I owe an apology also, for jumping to that conclusion. It was rash and unfair of me. I am sorry. On Wed, Feb 2, 2011 at 5:03 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Justin Pitts wrote: With all due respect, I consider myself smarter than the optimizer.  I'm

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Andrew Dunstan
On 02/02/2011 07:17 PM, Greg Smith wrote: I direct anyone who thought Mladen was making a serious comment to http://www.nydailynews.com/news/politics/2009/01/08/2009-01-08_misunderestimate_tops_list_of_notable_bu-3.html if you want to get his little joke there. I plan to start using

Re: [PERFORM] Which RAID Controllers to pick/avoid?

2011-02-03 Thread Joachim Worringen
Am 03.02.2011 07:45, schrieb Magnus Hagander: On Thu, Feb 3, 2011 at 07:30, Dan Birkenbir...@gmail.com wrote: - Is the linux support of the LSI and Adaptec cards comparable? Can't comment on that one, sorry. We dropped LSI in favour of Adaptec for exactly this reason. We run hundreds of

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Josh Berkus wrote: I don't want to insult anybody but the whole thing does look strange. Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? That would make it look less like , hmph, philosophical issue and more not yet implemented issue, especially if we

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 3:54 PM, da...@lang.hm wrote: with the current code, this is a completely separate process that knows nothing about the load, so if you kick it off when you start the load, it makes a pass over the table (competing for I/O), finishes, you continue to update the table,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Kevin Grittner wrote: Mladen Gogala mladen.gog...@vmsinfo.com wrote: Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? I'd be against that. This is rehashed less frequently since that went in. Less wasted time and bandwidth with it there.

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:33 -0500, Mladen Gogala wrote: Exactly what we don't want. Who is we? The majority of long term hackers. -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 6:33 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Kevin Grittner wrote: Mladen Gogala mladen.gog...@vmsinfo.com wrote: Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki?  I'd be against that.  This is rehashed less

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Joshua D. Drake wrote: On Thu, 2011-02-03 at 18:33 -0500, Mladen Gogala wrote: Exactly what we don't want. Who is we? The majority of long term hackers. If that is so, I don't see world domination in the future, exactly the opposite. Database whose

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Craig James
On 2/3/11 1:34 PM, Shaun Thomas wrote: I must say that this purist attitude is extremely surprising to me. All the major DB vendors support optimizer hints, yet in the Postgres community, they are considered bad with almost religious fervor. Postgres community is quite unique with the fatwa

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Shaun Thomas
All other databases do have that feature. I must say, this debate gave me a good deal of stuff to think about. And, I think we're done here. The idea that the lack of hints will kill PostgreSQL is already demonstrably false. This is sounding more and more like a petulant tantrum. Folks, I

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 3:54 PM, da...@lang.hm wrote: with the current code, this is a completely separate process that knows nothing about the load, so if you kick it off when you start the load, it makes a pass over the table (competing for I/O),

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mladen Gogala
Robert Haas wrote: On Thu, Feb 3, 2011 at 6:33 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Kevin Grittner wrote: Mladen Gogala mladen.gog...@vmsinfo.com wrote: Maybe we can agree to remove that ridiculous we don't want hints note from Postgresql wiki? I'd be

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeremy Harris
On 2011-02-03 23:29, Robert Haas wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. It doesn't sound too impossible to pass only a percentage, starting high and dropping towards 1% once the loaded size

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeremy Harris
On 2011-02-03 21:51, Mark Kirkwood wrote: The cases I've seen in production typically involve outgrowing optimizer parameter settings: (e.g work_mem, effective_cache_size) as the application dataset gets bigger over time. An argument in favour of the DBMS maintaining a running estimate of

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Grant Johnson
On PostgreSQL, the difference in no hints and hints for that one query with skewed data is that the query finishes a little faster. On some others, which shall remain nameless, it is the difference between finishing in seconds or days, or maybe never. Hints can be useful, but I can also

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Mark Kirkwood
On 04/02/11 13:49, Jeremy Harris wrote: On 2011-02-03 21:51, Mark Kirkwood wrote: The cases I've seen in production typically involve outgrowing optimizer parameter settings: (e.g work_mem, effective_cache_size) as the application dataset gets bigger over time. An argument in favour of the

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 7:39 PM, da...@lang.hm wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you want to write the code and prove it's better than what we have now, or some other approach that

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 7:39 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote:  reality. As a matter of fact, Oracle RDBMS on the same machine will regularly beat PgSQL in performance. That has been my experience so far. I even posted counting query results. It sure is, but those count

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread david
On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 7:39 PM, da...@lang.hm wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you want to write the code and prove it's better than what we

Re: [PERFORM] Why we don't want hints Was: Slow count(*) again...

2011-02-03 Thread Josh Berkus
http://wiki.postgresql.org/wiki/Todo#Features_We_Do_Not_Want No. 2 on the list. Heck, *I* wrote that text. I quote: Optimizer hints are used to work around problems in the optimizer and introduce upgrade and maintenance issues. We would rather have the problems reported and fixed. We have

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Robert Haas
On Thu, Feb 3, 2011 at 8:37 PM, da...@lang.hm wrote: On Thu, 3 Feb 2011, Robert Haas wrote: On Thu, Feb 3, 2011 at 7:39 PM,  da...@lang.hm wrote: Yeah, but you'll be passing the entire table through this separate process that may only need to see 1% of it or less on a large table. If you

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Conor Walsh
I can't remember anyone ever complaining ANALYZE took too long to run.  I only remember complaints of the form I had to remember to manually run it and I wish it had just happened by itself. Robert, This sounds like an argument in favor of an implicit ANALYZE after all COPY statements,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:12 -0800, Conor Walsh wrote: I can't remember anyone ever complaining ANALYZE took too long to run. I only remember complaints of the form I had to remember to manually run it and I wish it had just happened by itself. Robert, This sounds like an argument in

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Conor Walsh
On Thu, Feb 3, 2011 at 6:33 PM, Joshua D. Drake j...@commandprompt.com wrote: Well that already happens... My understanding is that auto-analyze will fire only after my transaction is completed, because it is a seperate daemon. If I do like so: BEGIN; COPY ...; -- Dangerously un-analyzed

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 5:39 PM, Mladen Gogala mladen.gog...@vmsinfo.com wrote: Actually, it is not unlike a religious dogma, only stating that hints are bad. It even says so in the wiki. The arguments are There's been considerably more output than hints bad! Hulk Smash! 1) Refusal to

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Jeff Davis
On Thu, 2011-02-03 at 16:50 -0500, Mladen Gogala wrote: Chris Browne wrote: Well, the community declines to add hints until there is actual consensus on a good way to add hints. OK. That's another matter entirely. Who should make that decision? Is there a committee or a person who

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 7:05 PM, Robert Haas robertmh...@gmail.com wrote: If you want to randomly pick 10,000 rows out of all the rows that are going to be inserted in the table without knowing in advance how many there will be, how do you do that? Maybe you could instead just have it use some

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Andy Colson
On 02/03/2011 04:56 PM, Greg Smith wrote: Scott Marlowe wrote: On Thu, Feb 3, 2011 at 8:57 AM,gnuo...@rcn.com wrote: Time for my pet meme to wiggle out of its hole (next to Phil's, and a day later). For PG to prosper in the future, it has to embrace the multi-core/processor/SSD machine at

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: Yes they're useful, but like a plastic bad covering a broken car window, they're useful because they cover something that's inherently broken. Awesome. Now we have a car anology, with a funny typo no less. Plastic bad, I love it. This is real progress toward getting

Does auto-analyze work on dirty writes? (was: Re: [HACKERS] [PERFORM] Slow count(*) again...)

2011-02-03 Thread Mark Mielke
On 02/03/2011 09:45 PM, Conor Walsh wrote: My understanding is that auto-analyze will fire only after my transaction is completed, because it is a seperate daemon. If I do like so: BEGIN; COPY ...; -- Dangerously un-analyzed SELECT complicated-stuff ...; END; Auto-analyze does not benefit me,

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:40 PM, Greg Smith g...@2ndquadrant.com wrote: Scott Marlowe wrote: Yes they're useful, but like a plastic bad covering a broken car window, they're useful because they cover something that's inherently broken. Awesome.  Now we have a car anology, with a funny typo

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: It's not so much a car analogy as a plastic bad analogy. Is that like a Plastic Ono Band? Because I think one of those is the only thing holding the part of my bumper I smashed in the snow on right now. I could be wrong about the name. -- Greg Smith 2ndQuadrant

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 8:56 PM, Greg Smith g...@2ndquadrant.com wrote: Scott Marlowe wrote: It's not so much a car analogy as a plastic bad analogy. Is that like a Plastic Ono Band?  Because I think one of those is the only thing holding the part of my bumper I smashed in the snow on right

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Greg Smith
Andy Colson wrote: Cpu's wont get faster, but HD's and SSD's will. To have one database connection, which runs one query, run fast, it's going to need multi-core support. My point was that situations where people need to run one query on one database connection that aren't in fact limited

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 9:00 PM, Greg Smith g...@2ndquadrant.com wrote: Andy Colson wrote: Cpu's wont get faster, but HD's and SSD's will.  To have one database connection, which runs one query, run fast, it's going to need multi-core support. My point was that situations where people need

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Greg Smith
Scott Marlowe wrote: No, that's a plastic oh no! band you have. Wow, right you are. So with this type holding together my Japanese car, if it breaks and parts fall off, I'm supposed to yell Oh, no! There goes Tokyo!, yes? -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Andy Colson
On 02/03/2011 10:00 PM, Greg Smith wrote: Andy Colson wrote: Cpu's wont get faster, but HD's and SSD's will. To have one database connection, which runs one query, run fast, it's going to need multi-core support. My point was that situations where people need to run one query on one

Re: [PERFORM] getting the most of out multi-core systems for repeated complex SELECT statements

2011-02-03 Thread Scott Marlowe
On Thu, Feb 3, 2011 at 9:19 PM, Andy Colson a...@squeakycode.net wrote: On 02/03/2011 10:00 PM, Greg Smith wrote: Andy Colson wrote: Cpu's wont get faster, but HD's and SSD's will. To have one database connection, which runs one query, run fast, it's going to need multi-core support. My

  1   2   >