Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
My preference is to deal with the specific value vs generic value issue. For this issue, it can affect performance even if PREPARE/EXECUTE is execute exactly once. In the last case I saw, a certain query was executing once every second, and with a specific value it would take 1 ms, and with

[HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Greg Smith
I'm happy to see we've crossed the point where the worst of the Hot Standby and Streaming Replication issues are sorted out. A look at the to-do lists: http://wiki.postgresql.org/wiki/Hot_Standby_TODO http://wiki.postgresql.org/wiki/Streaming_Replication show no Must-fix items and 5 Serious

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Gokulakannan Somasundaram
To be a bit more concrete: the typical sort of failure that you could get from broken btree operators is failure of transitivity, that is the comparators report A B and B C for some A, B, C, but do not say that A C when those two values are compared directly. I don't see any convenient

[HACKERS] Correcting Error message

2010-02-26 Thread Piyush Newe
Hi, Consider following testcase, CREATE TABLE footable(id int4, name varchar2(10)); CREATE FUNCTION foofunc(a footable, b integer DEFAULT 10) RETURNS integer AS $$ SELECT 123; $$ LANGUAGE SQL; CREATE FUNCTION foofunc(a footable, b numeric DEFAULT 10) RETURNS integer AS $$ SELECT 123; $$

Re: [HACKERS] A thought: should we run pgindent now?

2010-02-26 Thread Greg Smith
Bruce Momjian wrote: The diff is here: http://momjian.us/tmp/pgindent.diff and I checked into CVS a copy of the typedef list I used from Andrew Dunstan. Is it worthwhile to consider writing up a how to run pgindent like Bruce does page on the wiki? I've been scared off of that

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Karl Schnaitter
On Fri, Feb 26, 2010 at 12:36 AM, Gokulakannan Somasundaram gokul...@gmail.com wrote: To be a bit more concrete: the typical sort of failure that you could get from broken btree operators is failure of transitivity, that is the comparators report A B and B C for some A, B, C, but do not say

Re: [HACKERS] pg_stop_backup does not complete

2010-02-26 Thread Bernd Helmle
--On 24. Februar 2010 16:01:02 -0500 Tom Lane t...@sss.pgh.pa.us wrote: One objection to this is that it's not very clear to the user when pg_stop_backup has finished with actual work and is just waiting for the archiver, ie when is it safe to hit control-C? Maybe we should emit a backup

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Jeroen Vermeulen
Mark Mielke wrote: Re-planning a generic plan with another generic plan may generate zero benefit, with a measurable cost. More on this after... Nobody's talking about doing that any more. I proposed it initially because I didn't know about changes that made it unnecessary. All the

Re: [HACKERS] NaN/Inf fix for ECPG

2010-02-26 Thread Boszormenyi Zoltan
Tom Lane írta: Boszormenyi Zoltan z...@cybertec.at writes: Can you try whether the first patch (missing float.h from data.c) solves the problem? And together with the 2nd one? In that patch I fixed the order of float.h and math.h in nan_test.pgc, which is the opposite of the order found

[HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Greg Stark
On Fri, Feb 26, 2010 at 8:33 AM, Greg Smith g...@2ndquadrant.com wrote: I'm not sure what you might be expecting from the above combination, but what actually happens is that many of the SELECT statements on the table *that isn't even being updated* are canceled.  You see this in the logs:

Re: [HACKERS] NaN/Inf fix for ECPG

2010-02-26 Thread Rémi Zara
Le 26 févr. 2010 à 12:12, Boszormenyi Zoltan a écrit : ecpglib/data.c includes postgres_fe.h, so it should be pulling everything that's relevant from port/*, right? Michael, can we try to install the first two patches? They wouldn't hurt. I've tried patch 1 and 2, but they do not

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Greg Stark
On Fri, Feb 26, 2010 at 4:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: I feel the other one is easy. To store the hint bits inside the ItemId, in the place of size. No, we're not going there.  That breaks the fundamental page content manipulation algorithms, and falls down for tuples not yet

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Gokulakannan Somasundaram
It definitely affects current indexes. We can't completely avoid bad user functions. That is why it is important to put limits on how much damage they can do. That's the motivation for the idea I mentioned before, of double-checking visibility data in an IndexTuple before letting it survive a

Re: [HACKERS] pg_stop_backup does not complete

2010-02-26 Thread Greg Stark
On Fri, Feb 26, 2010 at 9:41 AM, Bernd Helmle maili...@oopsware.de wrote: --On 24. Februar 2010 16:01:02 -0500 Tom Lane t...@sss.pgh.pa.us wrote: One objection to this is that it's not very clear to the user when pg_stop_backup has finished with actual work and is just waiting for the

Re: [HACKERS] A thought: should we run pgindent now?

2010-02-26 Thread Bruce Momjian
Greg Smith wrote: Bruce Momjian wrote: The diff is here: http://momjian.us/tmp/pgindent.diff and I checked into CVS a copy of the typedef list I used from Andrew Dunstan. Is it worthwhile to consider writing up a how to run pgindent like Bruce does page on the wiki?

Re: [HACKERS] NaN/Inf fix for ECPG

2010-02-26 Thread Michael Meskes
On Fri, Feb 26, 2010 at 12:12:10PM +0100, Boszormenyi Zoltan wrote: Michael, can we try to install the first two patches? If I understood the rest of the thread correctly this is not needed anymore, right? Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Heikki Linnakangas
Greg Smith wrote: Attached is a tar file with some test case demo scripts that demonstrate the worst of the problems here IMHO. Thanks for that! We've been discussing this for ages, so it's nice to have a concrete example. I don't want to belittle that work because it's been important to make

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Richard Huxton
On 26/02/10 08:33, Greg Smith wrote: There are a number of HS tunables that interact with one another, and depending your priorities a few ways you can try to optimize the configuration for what I expect to be common use cases for this feature. I've written a blog entry at

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Richard Huxton
On 26/02/10 14:10, Heikki Linnakangas wrote: Ideally the standby would stash away the old pages or tuples somewhere so that it can still access them even after replaying the WAL records that remove them from the main storage. I realize that's not going to happen any time soon because it's hard

Re: [HACKERS] Assertion failure in walreceiver

2010-02-26 Thread Heikki Linnakangas
Greg Stark wrote: So do I understand this right, if you have archive_mode disabled and try to start a slave you get this error. Then when you shut down your master and set archive_mode on and bring it up and try again you'll *still* get this message because the last checkpoint will be the

Re: [HACKERS] Assertion failure in walreceiver

2010-02-26 Thread Heikki Linnakangas
Greg Stark wrote: On Thu, Feb 25, 2010 at 7:31 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Committed removal of that and the assertion. You still can't use a copy of the data directory taken right after initdb, because the initial checkpoint record has the flag set

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Gokulakannan Somasundaram
Much better to take on a simple project like enabling full sequential index scans which you claimed you had a solution for and which is in any case an important sub-problem for IOT. Greg, Well i don't think i am ready to take up a project of this size. But at the same time some

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Heikki Linnakangas
Richard Huxton wrote: On 26/02/10 08:33, Greg Smith wrote: I'm not sure what you might be expecting from the above combination, but what actually happens is that many of the SELECT statements on the table *that isn't even being updated* are canceled. You see this in the logs: Hmm - this I'd

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 05:20 AM, Jeroen Vermeulen wrote: Mark Mielke wrote: All the points about ms seem invalid to me. There are many reason why ms could increase, and many of them have nothing to do with plan efficiency. Again, re-planning due to a high ms, or a high ratio of ms, does not indicate

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Richard Huxton
On 26/02/10 14:45, Heikki Linnakangas wrote: Richard Huxton wrote: On 26/02/10 08:33, Greg Smith wrote: I'm not sure what you might be expecting from the above combination, but what actually happens is that many of the SELECT statements on the table *that isn't even being updated* are

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Alex Hunsaker bada...@gmail.com writes: Let me see if I can sum up what I was trying to say: [ this can be solved by using or avoiding prepared statements ] Not really. The place where that argument really fails is inside server-side functions: you don't get to use query submission protocol

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Heikki Linnakangas
Richard Huxton wrote: Can we not wait to cancel the transaction until *any* new lock is attempted though? That should protect all the single-statement long-running transactions that are already underway. Aggregates etc. Hmm, that's an interesting thought. You'll still need to somehow tell the

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Alex Hunsaker
On Fri, Feb 26, 2010 at 08:07, Tom Lane t...@sss.pgh.pa.us wrote: Alex Hunsaker bada...@gmail.com writes: Let me see if I can sum up what I was trying to say: [ this can be solved by using or avoiding prepared statements ] Not really.  The place where that argument really fails is inside

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Greg Stark
On Fri, Feb 26, 2010 at 2:38 PM, Gokulakannan Somasundaram gokul...@gmail.com wrote:   I have never written much code in C, and even if write it, i am sure i will receive the comment that it is a unmaintainable code.(eg: Thick index code and trailing nulls code) I definitely think

[HACKERS] Assertion failure twophase.c (testing HS/SR)

2010-02-26 Thread Erik Rijkers
9.0devel (cvs yesterday) primary+server, with this patch: extend_format_of_recovery_info_funcs_v2.patch ( http://archives.postgresql.org/pgsql-hackers/2010-02/msg02116.php ) A large (500 GB) restore left to run overnight, gave the below crash. The standby was restarted, and seems to be

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: On Fri, Feb 26, 2010 at 4:47 AM, Tom Lane t...@sss.pgh.pa.us wrote: I feel the other one is easy. To store the hint bits inside the ItemId, in the place of size. No, we're not going there. Well we were already talking about moving the hint bits to

Re: [HACKERS] Assertion failure twophase.c (testing HS/SR)

2010-02-26 Thread Tom Lane
Erik Rijkers e...@xs4all.nl writes: Is it useful to report these cases of FailedAssertion? Absolutely. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

[HACKERS] ecpg tests broken by pgindent run

2010-02-26 Thread Tom Lane
Hey Bruce, I thought the ecpg test files were supposed to be excluded from pgindenting? The buildfarm is all red, or would be if you could see it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] NaN/Inf fix for ECPG

2010-02-26 Thread Tom Lane
=?iso-8859-1?Q?R=E9mi_Zara?= remi_z...@mac.com writes: I've tried patch 1 and 2, but they do not work. The fact is that the code is not used in the backend, because strtod(NaN, endptr) works. (isnan(strtod(NaN, endptr)) is true). Hmm. So what do you get from SELECT

Re: [HACKERS] NaN/Inf fix for ECPG

2010-02-26 Thread Tom Lane
Michael Meskes mes...@postgresql.org writes: On Fri, Feb 26, 2010 at 12:12:10PM +0100, Boszormenyi Zoltan wrote: Michael, can we try to install the first two patches? If I understood the rest of the thread correctly this is not needed anymore, right? I think it would be a good idea, just to

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke m...@mark.mielke.cc writes: Will guessing at when the user can afford to wait longer improve the situation? Maybe or often, but not always. There is no way to eliminate guessing. The entire point here is that we don't know whether generating a custom plan will provide a win over

Re: [HACKERS] ecpg tests broken by pgindent run

2010-02-26 Thread Bruce Momjian
Tom Lane wrote: Hey Bruce, I thought the ecpg test files were supposed to be excluded from pgindenting? The buildfarm is all red, or would be if you could see it. Well, I am excluding them: find . -name '*.[ch]' -type f -print | egrep -v

[HACKERS] pgbouncer + psql 9.0a4

2010-02-26 Thread Garick Hamlin
I was just trying out 9.0a4 and I noticed. That I can't connect to pgbouncer with psql from 9.0a4 as a result of the set application_name changes to psql. I imagine this is really mostly pgbouncer's problem, but I couldn't figure out how if it was possible to make psql not set

Re: [HACKERS] pgbouncer + psql 9.0a4

2010-02-26 Thread Jaime Casanova
On Fri, Feb 26, 2010 at 11:26 AM, Garick Hamlin gham...@isc.upenn.edu wrote: I imagine this is really mostly pgbouncer's problem yes it is, but you can workaround it using ignore_startup_parameters parameter in pgbouncer.ini http://pgbouncer.projects.postgresql.org/doc/config.html#toc4 --

Re: [HACKERS] pgbouncer + psql 9.0a4

2010-02-26 Thread Kenneth Marshall
Hi Garick, Add an ignore_startup_parameters to your pgbouncer.ini file with application_name. Cheers, Ken On Fri, Feb 26, 2010 at 11:26:23AM -0500, Garick Hamlin wrote: I was just trying out 9.0a4 and I noticed. That I can't connect to pgbouncer with psql from 9.0a4 as a result of the set

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Richard Huxton
Replying to my own post - first sign of madness... Let's see if I've got the concepts clear here, and hopefully my thinking it through will help others reading the archives. There are two queues: 1. Cleanup on the master 2. Replay on the slave Running write queries on the master adds to both

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 1:29 AM, Alex Hunsaker bada...@gmail.com wrote: Prepared plans + exec plan (new guc/ protocol thing):  Use: not quite sure  Problems: slow because it would replan every time  Solutions: use a prepared plan with the appropriate things not parametrized...? [ aka we

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 10:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: I think this is basically a planner problem and should be fixed in the planner, not by expecting users to make significant changes in application logic in order to create an indirect effect. I would agree if I thought that

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 11:27 AM, Tom Lane t...@sss.pgh.pa.us wrote: Also, I think there is a lot of confusion here over two different issues: generic plan versus parameter-specific plan, and bad planner estimates leading to a wrong plan choice.  While the latter is certainly an issue

Re: [HACKERS] ecpg tests broken by pgindent run

2010-02-26 Thread Boszormenyi Zoltan
Bruce Momjian írta: Tom Lane wrote: Hey Bruce, I thought the ecpg test files were supposed to be excluded from pgindenting? The buildfarm is all red, or would be if you could see it. Well, I am excluding them: find . -name '*.[ch]' -type f -print | egrep -v

Re: [HACKERS] ecpg tests broken by pgindent run

2010-02-26 Thread Bruce Momjian
Boszormenyi Zoltan wrote: but if I look at the CVS history for /cvsroot/pgsql/src/interfaces/ecpg/test/expected/compat_informix-sqlda.c I see the last change was January 6, not from pgindent. My only guess is that there was some boilerplate code in the ecpg C files or outside /test

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Feb 26, 2010 at 11:27 AM, Tom Lane t...@sss.pgh.pa.us wrote: Also, I think there is a lot of confusion here over two different issues: generic plan versus parameter-specific plan, and bad planner estimates leading to a wrong plan choice.  While

Re: [HACKERS] ecpg tests broken by pgindent run

2010-02-26 Thread Bruce Momjian
Bruce Momjian wrote: Boszormenyi Zoltan wrote: but if I look at the CVS history for /cvsroot/pgsql/src/interfaces/ecpg/test/expected/compat_informix-sqlda.c I see the last change was January 6, not from pgindent. My only guess is that there was some boilerplate code in the ecpg C

[HACKERS] ProcSignalSlot vs. PGPROC

2010-02-26 Thread Markus Wanner
Hi, do I understand correctly that a BackendId is just an index into the ProcSignalSlots array and not (necessarily) the same as the index into ProcArrayStruct's procs? If yes, could these be synchronized? Why is ProcSignalSlot not part of PGPROC at all? Both are shared memory structures

Re: [HACKERS] ecpg tests broken by pgindent run

2010-02-26 Thread Andrew Dunstan
Bruce Momjian wrote: Don't look further, interfaces/ecpg/include/sqlda.h has changed by the pgindent run. Yea, it is that, and sqltypes.h and one other file I am trying to find now. I have reverted changes to three include files who's output appears in the regression

Re: [HACKERS] pgbouncer + psql 9.0a4

2010-02-26 Thread Garick Hamlin
On Fri, Feb 26, 2010 at 11:38:20AM -0500, Jaime Casanova wrote: On Fri, Feb 26, 2010 at 11:26 AM, Garick Hamlin gham...@isc.upenn.edu wrote: I imagine this is really mostly pgbouncer's problem yes it is, but you can workaround it using ignore_startup_parameters parameter in pgbouncer.ini

Re: [HACKERS] pgbouncer + psql 9.0a4

2010-02-26 Thread Jaime Casanova
On Fri, Feb 26, 2010 at 12:23 PM, Garick Hamlin gham...@isc.upenn.edu wrote: I am new to pgbouncer, sorry if this was obvious. if you never ask you never learn... nothing to be sorry about... well, besides the fact that you don't read the whole manual before trying it, but just a few months a

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Robert Haas
On Fri, Feb 26, 2010 at 10:21 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Richard Huxton wrote: Can we not wait to cancel the transaction until *any* new lock is attempted though? That should protect all the single-statement long-running transactions that are already

[HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Greg Stark
On Fri, Feb 26, 2010 at 4:43 PM, Richard Huxton d...@archonet.com wrote: Let's see if I've got the concepts clear here, and hopefully my thinking it through will help others reading the archives. There are two queues: I don't see two queues. I only see the one queue of operations which have

Re: [HACKERS] ecpg tests broken by pgindent run

2010-02-26 Thread Bruce Momjian
Andrew Dunstan wrote: Bruce Momjian wrote: Don't look further, interfaces/ecpg/include/sqlda.h has changed by the pgindent run. Yea, it is that, and sqltypes.h and one other file I am trying to find now. I have reverted changes to three include

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Fri, Feb 26, 2010 at 10:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: I think this is basically a planner problem and should be fixed in the planner, not by expecting users to make significant changes in application logic in order to create an indirect

Re: [HACKERS] ProcSignalSlot vs. PGPROC

2010-02-26 Thread Tom Lane
Markus Wanner mar...@bluegap.ch writes: do I understand correctly that a BackendId is just an index into the ProcSignalSlots array and not (necessarily) the same as the index into ProcArrayStruct's procs? If yes, could these be synchronized? Why is ProcSignalSlot not part of PGPROC at

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Gokulakannan Somasundaram
Missed the group.. On Sat, Feb 27, 2010 at 12:00 AM, Gokulakannan Somasundaram gokul...@gmail.com wrote: I definitely think thick indexes were too ambitious of a target for a first time patch. Sequential index scans is very ambitious itself despite being significantly simpler (if you have a

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Bruce Momjian
Heikki Linnakangas wrote: How to handle situations where the standby goes away for a while, such as a network outage, so that it doesn't block the master from ever cleaning up dead tuples is a concern. Yeah, that's another issue that needs to be dealt with. You'd probably need some kind

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: In the model you describe any long-lived queries on the slave cause tables in the master to bloat with dead records. Yup, same as they would do on the master. I think this model is on the roadmap but it's not appropriate for everyone and I think one of the

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 11:27 AM, Tom Lane wrote: Also, I think there is a lot of confusion here over two different issues: generic plan versus parameter-specific plan, and bad planner estimates leading to a wrong plan choice. While the latter is certainly an issue sometimes, there is no reason to

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Gokulakannan Somasundaram
IIRC, what was being talked about was shoehorning some hint bits into the line pointers by assuming that size and offset are multiples of 4. I'm not thrilled with having mutable status bits there for reliability reasons, but it could be done without breaking a lot of existing code. What I was

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke m...@mark.mielke.cc writes: Just to point out that I agree, and as per my original post, I think the only time prepared statements should be re-planned for the statistics case, is after 'analyze' has run. That sounds like a quicker solution, and a much smaller gain. After

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Bruce Momjian
Tom Lane wrote: I'm going to make an unvarnished assertion here. I believe that the notion of synchronizing the WAL stream against slave queries is fundamentally wrong and we will never be able to make it work. The information needed isn't available in the log stream and can't be made

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Josh Berkus
On 2/26/10 6:57 AM, Richard Huxton wrote: Can we not wait to cancel the transaction until *any* new lock is attempted though? That should protect all the single-statement long-running transactions that are already underway. Aggregates etc. I like this approach. Is it fragile in some

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Tom Lane
Gokulakannan Somasundaram gokul...@gmail.com writes: But Tom, can you please explain me why that broken ordering example doesn't affect the current index scans. It does. The point is that the system is set up to limit the bad consequences. You might (will) get wrong query answers, but the

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Josh Berkus
On 2/26/10 10:53 AM, Tom Lane wrote: I think that what we are going to have to do before we can ship 9.0 is rip all of that stuff out and replace it with the sort of closed-loop synchronization Greg Smith is pushing. It will probably be several months before everyone is forced to accept that,

Re: [HACKERS] Assertion failure twophase.c (testing HS/SR)

2010-02-26 Thread Heikki Linnakangas
Erik Rijkers wrote: 9.0devel (cvs yesterday) primary+server, with this patch: extend_format_of_recovery_info_funcs_v2.patch ( http://archives.postgresql.org/pgsql-hackers/2010-02/msg02116.php ) A large (500 GB) restore left to run overnight, gave the below crash. The standby was

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: On 2/26/10 10:53 AM, Tom Lane wrote: I think that what we are going to have to do before we can ship 9.0 is rip all of that stuff out and replace it with the sort of closed-loop synchronization Greg Smith is pushing. It will probably be several months

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 01:59 PM, Tom Lane wrote: ... It's walking around the problem that the idea of a generic plan is just wrong. The only time a generic plan is right, is when the specific plan would result in the same. I think that's a significant overstatement. There are a large number of

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 01:59 PM, Tom Lane wrote: Mark Mielkem...@mark.mielke.cc writes: Just to point out that I agree, and as per my original post, I think the only time prepared statements should be re-planned for the statistics case, is after 'analyze' has run. That sounds like a quicker

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Gokulakannan Somasundaram
It does. The point is that the system is set up to limit the bad consequences. You might (will) get wrong query answers, but the heap data won't get corrupted. Again Tom, if there is an update based on index scan, then it takes the tupleid and updates the wrong heap data right? The only

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke m...@mark.mielke.cc writes: There must be some way to lift the cost of planning out of the plan enumeration and selection phase, such that only plan enumeration and selection is run at execute time. In most cases, plan enumeration and selection, provided that all data required

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Greg Stark
On Fri, Feb 26, 2010 at 6:30 PM, Gokulakannan Somasundaram gokul...@gmail.com wrote: http://archives.postgresql.org/pgsql-hackers/2008-03/msg00682.php I think, the buy-in became difficult because of the code quality. Er, yeah. That's something we need to work on a bit. You should probably

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Josh Berkus
I don't see a substantial additional burden there. What I would imagine is needed is that the slave transmits a single number back --- its current oldest xmin --- and the walsender process publishes that number as its transaction xmin in its PGPROC entry on the master. If the main purpose

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Alex Hunsaker
On Fri, Feb 26, 2010 at 09:50, Robert Haas robertmh...@gmail.com wrote: On Fri, Feb 26, 2010 at 1:29 AM, Alex Hunsaker bada...@gmail.com wrote: Prepared plans + exec plan (new guc/ protocol thing):  Use: not quite sure  Problems: slow because it would replan every time  Solutions: use a

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Heikki Linnakangas
Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: On 2/26/10 10:53 AM, Tom Lane wrote: I think that what we are going to have to do before we can ship 9.0 is rip all of that stuff out and replace it with the sort of closed-loop synchronization Greg Smith is pushing. It will probably be

[HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Greg Stark
On Fri, Feb 26, 2010 at 7:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't see a substantial additional burden there.  What I would imagine is needed is that the slave transmits a single number back --- its current oldest xmin --- and the walsender process publishes that number as its

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Yeb Havinga
Tom Lane wrote: Right, but if the parameter is unknown then its distribution is also unknown. In any case that's just nitpicking, because the solution is to create a custom plan for the specific value supplied. Or are you suggesting that we should create a way for users to say here is the

[HACKERS] Re: [BUGS] BUG #5021: ts_parse doesn't recognize email addresses with underscores

2010-02-26 Thread Teodor Sigaev
Oleg, Teodor, can you look at this? I tried to fix it in wparser_def.c, but couldn't figure out how. Thanks. select distinct token as email from ts_parse('default', ' first_l...@yahoo.com ' ) where tokid = 4 Patch in attachment, it allows underscore in the middle of local part of email

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: I don't actually understand how tight synchronization on its own would solve the problem. What if the connection to the master is lost? Do you kill all queries in the standby before reconnecting? Sure. So what? They'd have been

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Joshua D. Drake
On Fri, 2010-02-26 at 12:02 -0800, Josh Berkus wrote: I don't see a substantial additional burden there. What I would imagine is needed is that the slave transmits a single number back --- its current oldest xmin --- and the walsender process publishes that number as its transaction xmin

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Aidan Van Dyk
* Greg Stark gsst...@mit.edu [100226 15:10]: On Fri, Feb 26, 2010 at 7:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't see a substantial additional burden there.  What I would imagine is needed is that the slave transmits a single number back --- its current oldest xmin --- and the

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 02:57 PM, Tom Lane wrote: Mark Mielkem...@mark.mielke.cc writes: There must be some way to lift the cost of planning out of the plan enumeration and selection phase, such that only plan enumeration and selection is run at execute time. In most cases, plan enumeration and

[HACKERS] C libpq frontend library fetchsize

2010-02-26 Thread Yeb Havinga
Hello list, I'm wondering if there would be community support for adding using the execute message with a rownum 0 in the c libpq client library, as it is used by the jdbc driver with setFetchSize. kind regards, Yeb Havinga -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Mark Mielke
On 02/26/2010 03:11 PM, Yeb Havinga wrote: Tom Lane wrote: Right, but if the parameter is unknown then its distribution is also unknown. In any case that's just nitpicking, because the solution is to create a custom plan for the specific value supplied. Or are you suggesting that we should

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: On Fri, Feb 26, 2010 at 7:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: I don't see a substantial additional burden there.  What I would imagine is needed is that the slave transmits a single number back --- its current oldest xmin --- and the walsender process

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Yeb Havinga
Mark Mielke wrote: On 02/26/2010 03:11 PM, Yeb Havinga wrote: Or instead of letting users give the distribution, gather it automatically in some plan statistics catalog? I suspect in most applications queries stay the same for months and maybe years, so after some number of iterations it is

Re: [HACKERS] Avoiding bad prepared-statement plans.

2010-02-26 Thread Tom Lane
Mark Mielke m...@mark.mielke.cc writes: Here are parts that can be done fixed: 1) Statement parsing and error checking. 2) Identification of tables and columns involved in the query. The above two are done in the parser, not the planner. 3) Query the column statistics for involved columns,

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Gokulakannan Somasundaram
It does. The point is that the system is set up to limit the bad consequences. You might (will) get wrong query answers, but the heap data won't get corrupted. Tom, if this is our goal - *can return wrong query answers, but should not corrupt the heap data.* and if we make Thick

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Josh Berkus
Well, as Heikki said, a stop-and-go WAL management approach could deal with that use-case. What I'm concerned about here is the complexity, reliability, maintainability of trying to interlock WAL application with slave queries in any sort of fine-grained fashion. This sounds a bit

[HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Greg Stark
On Fri, Feb 26, 2010 at 8:30 PM, Tom Lane t...@sss.pgh.pa.us wrote: How's it going to do that, when it has no queries at the instant of startup? Why shouldn't it have any queries at walreceiver startup? It has any xlog segments that were copied from the master and any it can find in the

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: Why shouldn't it have any queries at walreceiver startup? It has any xlog segments that were copied from the master and any it can find in the archive, it could easily reach a consistent point long before it needs to connect to the master. If you really want

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Bruce Momjian
bruce wrote: 4 The standby waiting longer than max_standby_delay to acquire a ... #4 can be controlled by max_standby_delay, where a large value only delays playback during crash recovery --- again, a rare occurance. One interesting feature is that max_standby_delay will _only_ delay

[HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Greg Stark
On Fri, Feb 26, 2010 at 9:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: There's *definitely* not going to be enough information in the WAL stream coming from a master that doesn't think it has HS slaves. We can't afford to record all that extra stuff in installations for which it's just useless

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Tom Lane
Gokulakannan Somasundaram gokul...@gmail.com writes: It does. The point is that the system is set up to limit the bad consequences. You might (will) get wrong query answers, but the heap data won't get corrupted. Again Tom, if there is an update based on index scan, then it takes the

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes: Well, as Heikki said, a stop-and-go WAL management approach could deal with that use-case. What I'm concerned about here is the complexity, reliability, maintainability of trying to interlock WAL application with slave queries in any sort of fine-grained

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Tom Lane
Greg Stark gsst...@mit.edu writes: On Fri, Feb 26, 2010 at 9:19 PM, Tom Lane t...@sss.pgh.pa.us wrote: There's *definitely* not going to be enough information in the WAL stream coming from a master that doesn't think it has HS slaves. We can't afford to record all that extra stuff in

Re: [HACKERS] A thought on Index Organized Tables

2010-02-26 Thread Gokulakannan Somasundaram
No, what generally happens is it fails to find a matching index entry at all, because the search algorithm concludes there can be no match based on the limited set of comparisons it's done. Transitivity failures lead to searching the wrong subset of the index. Actually Tom, i am not able to

Re: [HACKERS] Hot Standby query cancellation and Streaming Replication integration

2010-02-26 Thread Bruce Momjian
Dimitri Fontaine wrote: Tom Lane t...@sss.pgh.pa.us writes: Well, as Heikki said, a stop-and-go WAL management approach could deal with that use-case. What I'm concerned about here is the complexity, reliability, maintainability of trying to interlock WAL application with slave queries

  1   2   >