Re: [HACKERS] Further Hot Standby documentation required
Bruce Momjian wrote: Simon Riggs wrote: * wal_level = 'hot_standby' doesn't mention that the second parameter also needs to be set, nor is there a xref. Fixed. * wal_level doesn't describe what the impacts are on a standby if the level is changed on the primary, nor is there a caution or a warning of any kind. For example, if a standby is setup with hot_standby = on and the primary is set wal_level = archive, does the standby start working if the primary changes wal_level = hot_standby? What happens if the primary is set wal_level = hot_standby and is then changed to archive? Hmm, feels like it should rather be documented in the hot_standby setting, it affects the standby not the master after all. How about this: --- config.sgml 2 May 2010 11:32:53 - 1.274 +++ config.sgml 3 May 2010 09:06:19 - @@ -1931,6 +1933,19 @@ para Specifies whether or not you can connect and run queries during recovery, as described in xref linkend=hot-standby. + /para + para +Hot Standby requires xref linkend=guc-wal-level to be set +to literalhot_standby/ in the primary server. If it was not +while the WAL being replayed was generated, and +varnamehot_standby/ is set to literalon/, the server will +refuse to start up. Also, if varnamewal_level/ is changed on +the primary server while a standby is active in hot standby mode, and +the primary server is restarted for the setting to take effect, the +standby will shut down as soon as it sees the checkpoint record +generated at the restart of the primary server. + /para + para The default value is literaloff/literal. This parameter can only be set at server start. It only has effect during archive recovery or in standby mode. * wal_level doesn't explicitly describe that the levels are in sequence and that hot_standby is a superset of archive. It does say: The default value is literalminimal/, which writes only the information needed to recover from a crash or immediate shutdown. literalarchive/ adds logging required for WAL archiving, and literalhot_standby/ further adds information required to run read-only queries on a standby server. Want to propose a better wording? I totally agree. Also, there is the new requirement to set wal_level for PITR, which is not documented as a necessary setup step yet. Fixed. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Further Hot Standby documentation required
On Mon, 2010-05-03 at 12:17 +0300, Heikki Linnakangas wrote: * wal_level doesn't describe what the impacts are on a standby if the level is changed on the primary, nor is there a caution or a warning of any kind. For example, if a standby is setup with hot_standby = on and the primary is set wal_level = archive, does the standby start working if the primary changes wal_level = hot_standby? What happens if the primary is set wal_level = hot_standby and is then changed to archive? Hmm, feels like it should rather be documented in the hot_standby setting, it affects the standby not the master after all. Danger of action at a distance. The change is on the master, but the effect is on the standby. The person changing the master must be warned of the danger that they will bring down the standby, so it must go with the parameter, not only with the HS docs. Doesn't this new behaviour prevent startup-from-a-shutdown-checkpoint? Whether it does or not, we need it to be documented. * wal_level doesn't explicitly describe that the levels are in sequence and that hot_standby is a superset of archive. It does say: The default value is literalminimal/, which writes only the information needed to recover from a crash or immediate shutdown. literalarchive/ adds logging required for WAL archiving, and literalhot_standby/ further adds information required to run read-only queries on a standby server. Want to propose a better wording? so at least archive level changed to so a setting of either archive or hot_standby That way there is no concept of an ordering required to understand the sentence because it is explicit. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
On Fri, Apr 30, 2010 at 19:46, Andrew Dunstan and...@dunslane.net wrote: Kevin Grittner wrote: The reported source of the software seems to have gone away. I can let you have my copy, which reliably reproduces the error, so we have a good failure test case. If it's not as recent as this: http://ww2.fs.ei.tum.de/~corecode/hg/fromcvs/log/132 we might want to bring in (or at least review) what's been committed since we grabbed our copy. Some of the fixes mentioned sound like they might possibly be related. OK, there is good news on this front. When last I looked this repo was giving errors, so I worked from the copy of the code I already had. (So thanks to Kevin for prodding me to check again.) With the latest fromcvs and rcsparse packages downloaded from this site I can now make and run the 7.4 and 8.1 branches from the imported repository (I haven't checked other branches yet.) What exactly did you test? Did you run it from scratch, or did you run it on top of the existing git repo? Because AFAIK the bug isn't consisten in that if you just do a fresh migration, it often works - even with the current version we have deployed. I haven't checked that in detail though - was this included in your tests? So we need to look at how hard it would be to upgrade the scripts on git.postgresql.org, and if by doing so we can fix its versions of the back Upgrading the script is trivial. It's working off a git checkout with a specific hash,iirc, because there were no actualstable releases at the time, so if you can tell me which commit to check out, that's easy. Will this automatically fix the backbranches, or do we need to do something more than just upgrading it? branches. It would also be a good idea to have a script that regularly checks out the important branches from git and checks that they are identical to what is in CVS. That might help to increase Tom's confidence level. Yes, that would be very good. ISTR there's a script somewhere, but it didn't run without showing some issues? Particularly related to the keyword expansion stuff? If someone has such a script ready today, it should be easy to deploy it on the git server for regular checks. If not, volunteers are most welcome to write one :-) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Further Hot Standby documentation required
Simon Riggs wrote: Doesn't this new behaviour prevent startup-from-a-shutdown-checkpoint? No. * wal_level doesn't explicitly describe that the levels are in sequence and that hot_standby is a superset of archive. It does say: The default value is literalminimal/, which writes only the information needed to recover from a crash or immediate shutdown. literalarchive/ adds logging required for WAL archiving, and literalhot_standby/ further adds information required to run read-only queries on a standby server. Want to propose a better wording? so at least archive level changed to so a setting of either archive or hot_standby That way there is no concept of an ordering required to understand the sentence because it is explicit. Ok, changed along those lines. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Magnus Hagander wrote: On Fri, Apr 30, 2010 at 19:46, Andrew Dunstan and...@dunslane.net wrote: Kevin Grittner wrote: The reported source of the software seems to have gone away. I can let you have my copy, which reliably reproduces the error, so we have a good failure test case. If it's not as recent as this: http://ww2.fs.ei.tum.de/~corecode/hg/fromcvs/log/132 we might want to bring in (or at least review) what's been committed since we grabbed our copy. Some of the fixes mentioned sound like they might possibly be related. OK, there is good news on this front. When last I looked this repo was giving errors, so I worked from the copy of the code I already had. (So thanks to Kevin for prodding me to check again.) With the latest fromcvs and rcsparse packages downloaded from this site I can now make and run the 7.4 and 8.1 branches from the imported repository (I haven't checked other branches yet.) What exactly did you test? Did you run it from scratch, or did you run it on top of the existing git repo? Because AFAIK the bug isn't consisten in that if you just do a fresh migration, it often works - even with the current version we have deployed. I haven't checked that in detail though - was this included in your tests? On the version of the code I was using was fetched from the Mercurial repo. The versions I originally tested with are rcsparse-1a3116e80c2e and fromcvs-186299486bdc. The versions I have had success with are rcsparse-75d93404707d and fromcvs-c31a1dd9cbb2 On the older versions I tested, the errors appeared each time I ran the initial imports on the CVS clone I was testing with. With the latest version it has not. So we need to look at how hard it would be to upgrade the scripts on git.postgresql.org, and if by doing so we can fix its versions of the back Upgrading the script is trivial. It's working off a git checkout with a specific hash,iirc, because there were no actualstable releases at the time, so if you can tell me which commit to check out, that's easy. Will this automatically fix the backbranches, or do we need to do something more than just upgrading it? I think what we'd need to do is remove the back branches that are failing and let it reimport them completely. AFAIK this should work without disturbing the branches we didn't remove. But it should be tested rigorously before being tried. branches. It would also be a good idea to have a script that regularly checks out the important branches from git and checks that they are identical to what is in CVS. That might help to increase Tom's confidence level. Yes, that would be very good. ISTR there's a script somewhere, but it didn't run without showing some issues? Particularly related to the keyword expansion stuff? If someone has such a script ready today, it should be easy to deploy it on the git server for regular checks. If not, volunteers are most welcome to write one :-) What I have done is to set up my own mirror and I will be testing it on all the live branches with a git-ized buildfarm client. I can probably pull together a script that exports from both git and cvs and compares. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] xslt_process with more than ten parameters - patch
Hello
One Czech pg user reported problems with function xslt_process. This
function is coded with ten parameters limit.
Attached patch add support to unlimited number of parameters.
const char **
parse_params(text *paramstr)
{
char *pos;
char *pstr;
char *nvsep = =;
char *itsep = ,;
const char **params;
int nparams;
int mparams;/* max params */
pstr = text_to_cstring(paramstr);
mparams = INIT_PARAMS;
params = (const char **) palloc(INIT_PARAMS * sizeof(char *) + 1);
pos = pstr;
nparams = 0;
while (*pos != '\0')
{
if (nparams = mparams)
{
/* extend params params */
mparams += EXTEND_PARAMS;
params = (const char **) repalloc(params, mparams *
sizeof(char *) + 1);
}
params[nparams++] = pos;
pos = strstr(pos, nvsep);
if (pos != NULL)
{
*pos = '\0';
pos++;
}
else
{
/* No equal sign, so ignore this parameter */
/* We'll reset params[i] to NULL below the loop */
nparams--;
break;
}
/* since MAXPARAMS is even, we still have i MAXPARAMS */
params[nparams++] = pos;
pos = strstr(pos, itsep);
if (pos != NULL)
{
*pos = '\0';
pos++;
}
else
break;
}
params[nparams] = NULL;
return params;
}
Regards
Pavel Stehule
nolimit.diff
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
* Andrew Dunstan and...@dunslane.net [100503 09:02]: I can probably pull together a script that exports from both git and cvs and compares. HINT: It's all been done and posted to -hackers before too... Along with comparisons on on whte one-time conversions fare (parsecvs, cvs2svn/git), etc, as well as long discussion on which keyword you want expanded, and which you don't, etc. bla bla bla... Not to rain on anyone's git-parade, I'm a huge git fan, but until the busy committers, like Tom, Bruce, Heikki, Robert, Andrew, Simon, Alvaro, (and all the rest I'm missing or don't know how to spell of the top of my head) actually *all use* git (and by use, I really mean use, not just copy-n-paste from some howto mimic CVS with git guide), any planned switch is just asking for a huge backlash of it doesn't work like it used to, and bitter complaints... So, for those of you trying to plan the conversion to git, I'ld really recommend you start working with the other committers, and get them using git... Show them how much it makes their current CVS committing easier even... Let them see what git can do *for* them (and for everyone else too). And then I suspect you'll find that this whole switch will suddenly not be a big issue... The whole find the perfect conversion isn't even needed... In fact, they might even realize why Linus's method for the switch to git for the linux kernel is often the best (if not right) approach... If you want, I know a guy in Ottawa that does really fantastic git presentations... He's done them to many of the local *UGs, Is there interest from the core committers in getting one done at PGcon? a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] missing file in git repo
On Mon, May 3, 2010 at 16:03, Aidan Van Dyk ai...@highrise.ca wrote: * Andrew Dunstan and...@dunslane.net [100503 09:02]: I can probably pull together a script that exports from both git and cvs and compares. HINT: It's all been done and posted to -hackers before too... Along with comparisons on on whte one-time conversions fare (parsecvs, cvs2svn/git), etc, as well as long discussion on which keyword you want expanded, and which you don't, etc. bla bla bla... Yeah, that's the one I was referring to in my earlier mail. Not to rain on anyone's git-parade, I'm a huge git fan, but until the busy committers, like Tom, Bruce, Heikki, Robert, Andrew, Simon, Alvaro, (and all the rest I'm missing or don't know how to spell of the top of my head) actually *all use* git (and by use, I really mean use, not just copy-n-paste from some howto mimic CVS with git guide), any planned switch is just asking for a huge backlash of it doesn't work like it used to, and bitter complaints... A lot of the people you just listed use git already. In fact, most of them do. (Not in commit volume though, since AFAIK Tom doesn't). So, for those of you trying to plan the conversion to git, I'ld really recommend you start working with the other committers, and get them using git... Show them how much it makes their current CVS committing easier even... Let them see what git can do *for* them (and for everyone else too). And then I suspect you'll find that this whole switch will Been there, done that, with at least a few ;) The thing we've always agreed upon is to at least start by migrating something that's as close to our current workflow as possible to git, and *then* consider changing anything in the workflow. We're not going to change both at once. For example, we are *definitely* going to keep with posting patches on the mailinglist during any such switch, and not go to the whole pull-from-each-others-repositories method that git likes. Nor are we going to use that insane let's send every single git revision as one email causing a single patch to be a whole bunch of sequentially numbered emails method ;) If you want, I know a guy in Ottawa that does really fantastic git presentations... He's done them to many of the local *UGs, Is there interest from the core committers in getting one done at PGcon? While I appreciate the offer, that's not really the problem, I think. In fact, people who really know git tend to talk a lot about the advanced workflows that git will let you do - something we specifically *don't* want (at this time), and often comes across as an oversell. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Aidan Van Dyk ai...@highrise.ca writes: If you want, I know a guy in Ottawa that does really fantastic git presentations... He's done them to many of the local *UGs, Is there interest from the core committers in getting one done at PGcon? I'd be interested. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
On Mon, May 3, 2010 at 16:13, Tom Lane t...@sss.pgh.pa.us wrote: Aidan Van Dyk ai...@highrise.ca writes: If you want, I know a guy in Ottawa that does really fantastic git presentations... He's done them to many of the local *UGs, Is there interest from the core committers in getting one done at PGcon? I'd be interested. Hey, that comment alone is enough for me to retract my previous comment :) At least if it can be made to be focused around the PostgreSQL workflow, and not the linux kernel workflow (or whatever). -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Magnus Hagander mag...@hagander.net writes: The thing we've always agreed upon is to at least start by migrating something that's as close to our current workflow as possible to git, and *then* consider changing anything in the workflow. We're not going to change both at once. Yeah. One of the main constraints in my view is retaining our current workflow for back-patching release branches. We're not going to stop supporting those branches, and we're not going to deal with two separate repositories. So if we're to convert to a git master, it has to be able to deal with back-patches. Given that the same patch is usually textually a bit different from branch to branch, I'm not convinced that git is going to make my life easier in that respect. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
On Mon, May 3, 2010 at 16:25, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: The thing we've always agreed upon is to at least start by migrating something that's as close to our current workflow as possible to git, and *then* consider changing anything in the workflow. We're not going to change both at once. Yeah. One of the main constraints in my view is retaining our current workflow for back-patching release branches. We're not going to stop supporting those branches, and we're not going to deal with two separate repositories. So if we're to convert to a git master, it has to be able to deal with back-patches. Given that the same patch is usually textually a bit different from branch to branch, I'm not convinced that git is going to make my life easier in that respect. I doubt it will - not the way that we work (and I like the way that we work with that). It might help a little in preparing them if they're complex - but we seldom backpatch really complex patches. I think we're more focused on making sure it doesn't make things *worse* for that use-case. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
On Mon, May 3, 2010 at 10:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: Magnus Hagander mag...@hagander.net writes: The thing we've always agreed upon is to at least start by migrating something that's as close to our current workflow as possible to git, and *then* consider changing anything in the workflow. We're not going to change both at once. Yeah. One of the main constraints in my view is retaining our current workflow for back-patching release branches. We're not going to stop supporting those branches, and we're not going to deal with two separate repositories. So if we're to convert to a git master, it has to be able to deal with back-patches. Given that the same patch is usually textually a bit different from branch to branch, I'm not convinced that git is going to make my life easier in that respect. Yeah, I don't think it is. Nor do I think it will make it any harder. The main benefits I see as a committer are: - It's faster; - I can work off-line; - I can queue up patches in a branch and then drop them all into the master branch at once (assuming no conflicts, of course). This might be useful for security updates, among other things; and of course - I won't have to switch back and forth between two systems. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Aidan Van Dyk wrote: * Andrew Dunstan and...@dunslane.net [100503 09:02]: I can probably pull together a script that exports from both git and cvs and compares. HINT: It's all been done and posted to -hackers before too... Along with comparisons on on whte one-time conversions fare (parsecvs, cvs2svn/git), etc, as well as long discussion on which keyword you want expanded, and which you don't, etc. bla bla bla... Not to rain on anyone's git-parade, I'm a huge git fan, but until the busy committers, like Tom, Bruce, Heikki, Robert, Andrew, Simon, Alvaro, (and all the rest I'm missing or don't know how to spell of the top of my head) actually *all use* git (and by use, I really mean use, not just copy-n-paste from some howto mimic CVS with git guide), any planned switch is just asking for a huge backlash of it doesn't work like it used to, and bitter complaints... So, for those of you trying to plan the conversion to git, I'ld really recommend you start working with the other committers, and get them using git... Show them how much it makes their current CVS committing easier even... Let them see what git can do *for* them (and for everyone else too). And then I suspect you'll find that this whole switch will suddenly not be a big issue... The whole find the perfect conversion isn't even needed... In fact, they might even realize why Linus's method for the switch to git for the linux kernel is often the best (if not right) approach... If you want, I know a guy in Ottawa that does really fantastic git presentations... He's done them to many of the local *UGs, Is there interest from the core committers in getting one done at PGcon? I am giving a talk in Ottawa on using git with Postgres. Feel free to come and correct all my errors if you're going to be there. That is why I have been talking about it a lot, both here and on my blog. I am currently working on some stuff that would have been significantly easier if we actually had a working git repository for all the live back branches (something I do have now, I think - fingers crossed). I have also been working on making the buildfarm client work with git, one of the hurdles we need to jump over. The fact that we are being cautious about this annoys the heck out of some people. I'm sorry, but that's just the way we tend to work. If we hadn't been so cautious a year or two back when git was much less mature, we'd probably be using Subversion now and any move to git would probably not even be on the table. Pushing us on this probably won't help the cause. More likely it will get people's backs up. And I'm not sure about the wisdom of using the Linux Kernel as an example of good software practice either. And let it also be noted that git has some warts too. I've seen plenty of complaints about its command set and I've had one or two substantial complaints myself. If we were going on pure technical merit I'm not at all sure git would win the contest for a new SCM system for Postgres. I am not trying to start a debate on the relevant merits of various SCM systems, but just trying to keep things in perspective. Moving to something like git will help us a bit. Sometimes (not that often, I suspect) it will help more than a bit. None of that is to say we shouldn't move to git. Also, if you're working on features, you can use git now, and ignore CVS completely. As far as we know the existing git mirror is quite sane and reliable, on all branches from 8.2 up. That accounts for 99% of developers - the number of people doing anything on versions 7.4 to 8.1 is minuscule. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Magnus Hagander wrote: On Mon, May 3, 2010 at 16:03, Aidan Van Dyk ai...@highrise.ca wrote: * Andrew Dunstan and...@dunslane.net [100503 09:02]: I can probably pull together a script that exports from both git and cvs and compares. HINT: It's all been done and posted to -hackers before too... Along with comparisons on on whte one-time conversions fare (parsecvs, cvs2svn/git), etc, as well as long discussion on which keyword you want expanded, and which you don't, etc. bla bla bla... Yeah, that's the one I was referring to in my earlier mail. If it has been done why isn't it being run? I pulled together a quick hack, and here is what I get from my mirrors. I'm not sure why we get those diffs - it's a bit odd, although fairly insignificant. cheers andrew [and...@sophia groot]$ perl check_branches *** Mon May 3 10:00:36 2010: checking REL7_4_STABLE diff -r pg-check-cvs-REL7_4_STABLE/contrib/mysql/my2pg.pl pg-check-git-REL7_4_STABLE/contrib/mysql/my2pg.pl 43,45d42 # Revision 1.10 2003/01/07 22:18:43 momjian # Upgrade to my2pg 1.9 # *** Mon May 3 10:01:04 2010: checking REL8_0_STABLE diff -r pg-check-cvs-REL8_0_STABLE/contrib/mysql/my2pg.diff pg-check-git-REL8_0_STABLE/contrib/mysql/my2pg.diff 17,25c17 # $Log: my2pg.diff,v $ # Revision 1.1 2004/04/19 23:18:12 momjian # Update to my2pg version 1.28, add docs, update URL for newest version. # # Create diff of custom changes Tom made to the utility for CREATE # FUNCTION. # # This will make moving this utility out of CVS easier. # --- # $Log: my2pg.pl,v $ diff -r pg-check-cvs-REL8_0_STABLE/contrib/mysql/my2pg.pl pg-check-git-REL8_0_STABLE/contrib/mysql/my2pg.pl 50,57d49 # Revision 1.13 2004/04/19 23:18:12 momjian # Update to my2pg version 1.28, add docs, update URL for newest version. # # Create diff of custom changes Tom made to the utility for CREATE # FUNCTION. # # This will make moving this utility out of CVS easier. # *** Mon May 3 10:02:46 2010: checking REL8_1_STABLE *** Mon May 3 10:03:54 2010: checking REL8_2_STABLE *** Mon May 3 10:04:30 2010: checking REL8_3_STABLE *** Mon May 3 10:05:01 2010: checking REL8_4_STABLE *** Mon May 3 10:05:32 2010: checking HEAD [and...@sophia groot]$ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
* Andrew Dunstan and...@dunslane.net [100503 11:05]: If it has been done why isn't it being run? I suspect (but can only speak for myself) it's simply because to most of us using git for development, it's irrelevant... We're using it to track/build/develop, and the history and keywords aren't relevant to us tracking branch tips and makeing sure our changes work. Every once in a while we notice a conversion glich, we fix it... We've posted the results, had the little bike-shed on keywords, hand-fixed RCS files, etc, and moved on ;-) But your case of trying to automatically build/track the branch heads for the buildfarm w/ git has a lot more strict requirements of the *incremental* *conversion* *of* *CVS* than any of us had/have... Of course, improvements in that conversion mean you actually have *better* reliability than the current CVS, because you can avoid the problem of broken CVS checkouts... Of course, if the repository was git, the buildfarm wouldn't need to worry if the git repository/commit it's fetching is a good approximation of the CVS ;-) a. -- Aidan Van Dyk Create like a god, ai...@highrise.ca command like a king, http://www.highrise.ca/ work like a slave. signature.asc Description: Digital signature
Re: [HACKERS] missing file in git repo
On Mon, May 3, 2010 at 10:03 AM, Aidan Van Dyk ai...@highrise.ca wrote: Not to rain on anyone's git-parade, I'm a huge git fan, but until the busy committers, like Tom, Bruce, Heikki, Robert, Andrew, Simon, Alvaro, (and all the rest I'm missing or don't know how to spell of the top of my head) actually *all use* git (and by use, I really mean use, not just copy-n-paste from some howto mimic CVS with git guide), any planned switch is just asking for a huge backlash of it doesn't work like it used to, and bitter complaints... Heikki and I are both BIG git users, and I think Andrew, Simon, and Alvaro all use it too, though I'm not sure to what extent. Tom and Bruce don't, AFAIK, but I don't believe that either of them are waving their arms and shouting no, no, no. At least, Tom isn't: and if Bruce does, well, I know where he lives. A couple of random things I'm concerned about: - the buildfarm, obviously, has got to work - we've still got to be able to send out automatic emails to committers, and they need to not suck (I'm deliberately not specifying what exactly that means because I don't know and I don't in any case want to overspecify, but certainly suckage is imaginable) - we need to make sure that all the committers understand how to keep the history the way we want it - i.e. linear, without merges, and possibly even implement programmatic safeguards against doing anything else ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] max_standby_delay considered harmful
I've finally wrapped my head around exactly what the max_standby_delay code is doing, and I'm not happy with it. The way that code is designed is that the setting represents a maximum allowed difference between the standby server's system clock and the commit timestamps it is reading from the WAL log; whenever this difference exceeds the setting, we'll kill standby queries in hopes of catching up faster. Now, I can see the attraction of defining it that way, for certain use-cases. However, I think it is too fragile and too badly implemented to be usable in the real world; and it certainly can't be the default operating mode. There are three really fundamental problems with it: 1. The timestamps we are reading from the log might be historical, if we are replaying from archive rather than reading a live SR stream. In the current implementation that means zero grace period for standby queries. Now if your only interest is catching up as fast as possible, that could be a sane behavior, but this is clearly not the only possible interest --- in fact, if that's all you care about, why did you allow standby queries at all? 2. There could be clock skew between the master and slave servers. If the master's clock is a minute or so ahead of the slave's, again we get into a situation where standby queries have zero grace period, even though killing them won't do a darn thing to permit catchup. If the master is behind the slave then we have an artificially inflated grace period, which is going to slow down the slave. 3. There could be significant propagation delay from master to slave, if the WAL stream is being transmitted with pg_standby or some such. Again this results in cutting into the standby queries' grace period, for no defensible reason. In addition to these fundamental problems there's a fatal implementation problem: the actual comparison is not to the master's current clock reading, but to the latest commit, abort, or checkpoint timestamp read from the WAL. Thus, if the last commit was more than max_standby_delay seconds ago, zero grace time. Now if the master is really idle then there aren't going to be any conflicts anyway, but what if it's running only long-running queries? Or what happens when it was idle for awhile and then starts new queries? Zero grace period, that's what. We could possibly improve matters for the SR case by having walsender transmit the master's current clock reading every so often (probably once per activity cycle), outside the WAL stream proper. The receiver could subtract off its own clock reading in order to measure the skew, and then we could cancel queries if the de-skewed transmission time falls too far behind. However this doesn't do anything to fix the cases where we aren't reading (and caught up to) a live SR broadcast. I'm inclined to think that we should throw away all this logic and just have the slave cancel competing queries if the replay process waits more than max_standby_delay seconds to acquire a lock. This is simple, understandable, and behaves the same whether we're reading live data or not. Putting in something that tries to maintain a closed-loop maximum delay between master and slave seems like a topic for future research rather than a feature we have to have in 9.0. And in any case we'd still want the plain max delay for non-SR cases, AFAICS, because there's no sane way to use closed-loop logic in other cases. Comments? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Robert Haas wrote: Heikki and I are both BIG git users, and I think Andrew, Simon, and Alvaro all use it too, though I'm not sure to what extent. I am using it increasingly. Of course, I need to convert some of my customers A couple of random things I'm concerned about: - the buildfarm, obviously, has got to work That work is almost complete. My plan is to unveil it all in my talk at pgcon. - we need to make sure that all the committers understand how to keep the history the way we want it - i.e. linear, without merges, and possibly even implement programmatic safeguards against doing anything else That too will be part of my talk. I'm not sure how you would implement such programmatic safeguards, though. We don't have such a huge population of committers that we can't manage this by education, I think. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Andrew Dunstan and...@dunslane.net writes: Robert Haas wrote: - we need to make sure that all the committers understand how to keep the history the way we want it - i.e. linear, without merges, and possibly even implement programmatic safeguards against doing anything else That too will be part of my talk. I'm not sure how you would implement such programmatic safeguards, though. We don't have such a huge population of committers that we can't manage this by education, I think. I agree that programmatic safeguards are probably overkill, as long as we have some agreed-on, documented commit procedures. We've survived fine without any such backstops for CVS commits. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
On Mon, May 3, 2010 at 17:55, Andrew Dunstan and...@dunslane.net wrote: Robert Haas wrote: - we need to make sure that all the committers understand how to keep the history the way we want it - i.e. linear, without merges, and possibly even implement programmatic safeguards against doing anything else That too will be part of my talk. I'm not sure how you would implement such programmatic safeguards, though. We don't have such a huge population of committers that we can't manage this by education, I think. It's pretty trivial to block some of the most error-prone things. I believe it's even in there as a sample somewhere. I thought you were the one supposed to teach us this :-P Anyway, I think it's worth putting in, at least the simple ones. It's easier than cleaning up after a single mistake, most likely... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Tom Lane escribió: Andrew Dunstan and...@dunslane.net writes: Robert Haas wrote: - we need to make sure that all the committers understand how to keep the history the way we want it - i.e. linear, without merges, and possibly even implement programmatic safeguards against doing anything else That too will be part of my talk. I'm not sure how you would implement such programmatic safeguards, though. We don't have such a huge population of committers that we can't manage this by education, I think. I agree that programmatic safeguards are probably overkill, as long as we have some agreed-on, documented commit procedures. We've survived fine without any such backstops for CVS commits. It would be great to have documented procedures for fixing fat-fingered commit messages and similar problems, though. Right now we have to live with redundant commits or replies on the pgsql-committers list. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
On Mon, May 3, 2010 at 17:05, Andrew Dunstan and...@dunslane.net wrote: Magnus Hagander wrote: On Mon, May 3, 2010 at 16:03, Aidan Van Dyk ai...@highrise.ca wrote: * Andrew Dunstan and...@dunslane.net [100503 09:02]: I can probably pull together a script that exports from both git and cvs and compares. HINT: It's all been done and posted to -hackers before too... Along with comparisons on on whte one-time conversions fare (parsecvs, cvs2svn/git), etc, as well as long discussion on which keyword you want expanded, and which you don't, etc. bla bla bla... Yeah, that's the one I was referring to in my earlier mail. If it has been done why isn't it being run? Because it was showing differences and there appeared to be no real drive to fix them - at least not until the cvs import scripts were fixed. I pulled together a quick hack, and here is what I get from my mirrors. I'm not sure why we get those diffs - it's a bit odd, although fairly insignificant. I don't think we can call anything insignificant - if we want it to be identical, it needs to be identical. So we need to figure out why they're different, and correct it. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Magnus Hagander mag...@hagander.net writes: On Mon, May 3, 2010 at 17:05, Andrew Dunstan and...@dunslane.net wrote: I pulled together a quick hack, and here is what I get from my mirrors. I'm not sure why we get those diffs - it's a bit odd, although fairly insignificant. I don't think we can call anything insignificant - if we want it to be identical, it needs to be identical. So we need to figure out why they're different, and correct it. At the very least we need to understand what's causing the difference; otherwise it could be a problem that will bite us in a more serious fashion later. I agree with Andrew that this difference is not critical in itself, so it might be reasonable to not bother fixing it after we understand it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Show schema in COPY error CONTEXT strings
-BEGIN PGP SIGNED MESSAGE-
Hash: RIPEMD160
ERROR: invalid input syntax for integer: abc
CONTEXT: COPY alpha.foo, line 1, column a: abc
We're really not going to address this type of complaint on a
one-error-message-at-a-time basis. See prior discussions --- a more
realistic (and standards compliant) approach will probably involve
adding fields to the verbose form of the error message.
Pointers to previous discussions welcome. I was simply trying to
fix a specific problem I was having, but some digging shows the
problem is already solved for most (all?) other similar cases:
# insert into public.foo (id) values ('mm');
ERROR: invalid input syntax for integer: mm
LINE 1: insert into public.foo (id) values ('mm');
# update public.foo set id='mm';
ERROR: invalid input syntax for integer: mm
LINE 1: update public.foo set id='mm';
# delete from public.foo where id = 'mm';
ERROR: invalid input syntax for integer: mm
LINE 1: delete from public.foo where id = 'mm';
Yes, I realize those are technically different context cases, but
from an application point of view, the COPY case is wrong and
needs fixing.
- --
Greg Sabino Mullane g...@turnstep.com
End Point Corporation http://www.endpoint.com/
PGP Key: 0x14964AC8 201005031242
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8
-BEGIN PGP SIGNATURE-
iEYEAREDAAYFAkve/PoACgkQvJuQZxSWSsjHiQCgoPZMcnP9viWoo4KY3y/I5NiA
1N0AoNyd5Fhs8M9WRkQ1LAS58Kz8x72S
=aIY9
-END PGP SIGNATURE-
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Mon, 2010-05-03 at 11:37 -0400, Tom Lane wrote: I've finally wrapped my head around exactly what the max_standby_delay code is doing, and I'm not happy with it. Yes, I don't think I'd call it perfect yet. have the slave cancel competing queries if the replay process waits more than max_standby_delay seconds to acquire a lock. This is simple, understandable, and behaves the same whether we're reading live data or not. I have no objection, and would welcome, adding another behaviour, since that just gives us a better chance of having this feature do something useful. I'm inclined to think that we should throw away all this logic HS has been through 2 Alphas with the current behaviour and it will go through 0 Alphas with the newly proposed behaviour. At this stage of proceedings, that is extremely dangerous and I don't wish to do that. The likelihood that we replace it with something worse seems fairly high/certain: snap decision making never quite considers all angles. Phrases like throw away all this logic don't give me confidence that people that agree with that perspective would understand what they are signing up to. Putting in something that tries to maintain a closed-loop maximum delay between master and slave seems like a topic for future research rather than a feature we have to have in 9.0. And in any case we'd still want the plain max delay for non-SR cases, AFAICS, because there's no sane way to use closed-loop logic in other cases. I will be looking for ways to improve this over time. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Simon Riggs wrote: On Mon, 2010-05-03 at 11:37 -0400, Tom Lane wrote: I've finally wrapped my head around exactly what the max_standby_delay code is doing, and I'm not happy with it. Yes, I don't think I'd call it perfect yet. have the slave cancel competing queries if the replay process waits more than max_standby_delay seconds to acquire a lock. This is simple, understandable, and behaves the same whether we're reading live data or not. I have no objection, and would welcome, adding another behaviour, since that just gives us a better chance of having this feature do something useful. I'm inclined to think that we should throw away all this logic HS has been through 2 Alphas with the current behaviour and it will go through 0 Alphas with the newly proposed behaviour. At this stage of proceedings, that is extremely dangerous and I don't wish to do that. The likelihood that we replace it with something worse seems fairly high/certain: snap decision making never quite considers all angles. Phrases like throw away all this logic don't give me confidence that people that agree with that perspective would understand what they are signing up to. I'm not really sure how much serious testing outside of the small set of people mostly interested in one or another specific aspect of HS/SR has been actually done with the alphas to be honest. I just started testing HS yesterday and I already ran twice into the general issue tom is complaining about with max_standby_delay... Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Show schema in COPY error CONTEXT strings
Greg Sabino Mullane g...@turnstep.com writes: We're really not going to address this type of complaint on a one-error-message-at-a-time basis. See prior discussions --- a more realistic (and standards compliant) approach will probably involve adding fields to the verbose form of the error message. Pointers to previous discussions welcome. The most recent one I can find is the thread starting at http://archives.postgresql.org/pgsql-hackers/2009-11/msg00846.php I was simply trying to fix a specific problem I was having, but some digging shows the problem is already solved for most (all?) other similar cases: Um, no, it's not solved. There are a huge number of error messages that refer to database objects by name only, even though the name might be ambiguous. It's not reasonable to fix them one at a time, especially not in a fashion that breaks regression tests ;-). My own preference for what to do about this is to leave the primary message texts alone and add additional error-message fields for object name and schema. This would address the need without making messages uglier for the large fraction of users who don't really care; and it would also help us get closer to the SQL standard's expectations for error reporting. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Mon, 2010-05-03 at 18:54 +0200, Stefan Kaltenbrunner wrote: I'm not really sure how much serious testing outside of the small set of people mostly interested in one or another specific aspect of HS/SR has been actually done with the alphas to be honest. I just started testing HS yesterday and I already ran twice into the general issue tom is complaining about with max_standby_delay... I guarantee that if that proposal goes in, people will complain about that also. Last minute behaviour changes are bad news. I don't object to adding something, just don't take anything away. It's not like the code for it is pages long or anything. The trade off is HA or queries and two modes make sense for user choice. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Further Hot Standby documentation required
On Mon, 2010-05-03 at 10:50 +0100, Simon Riggs wrote: On Mon, 2010-05-03 at 12:17 +0300, Heikki Linnakangas wrote: * wal_level doesn't describe what the impacts are on a standby if the level is changed on the primary, nor is there a caution or a warning of any kind. For example, if a standby is setup with hot_standby = on and the primary is set wal_level = archive, does the standby start working if the primary changes wal_level = hot_standby? What happens if the primary is set wal_level = hot_standby and is then changed to archive? Hmm, feels like it should rather be documented in the hot_standby setting, it affects the standby not the master after all. Danger of action at a distance. The change is on the master, but the effect is on the standby. The person changing the master must be warned of the danger that they will bring down the standby, so it must go with the parameter, not only with the HS docs. Don't really understand why you left that bit out. Are you just leaving this for me, or is there a specific objection to adding the warning? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Aidan Van Dyk wrote: But your case of trying to automatically build/track the branch heads for the buildfarm w/ git has a lot more strict requirements of the *incremental* *conversion* *of* *CVS* than any of us had/have... Actually, the thing that has recently annoyed me most has nothing to do with the buildfarm. I wanted to use git cherry-pick to backport something and the brokenness of the back branches in the git repo made that a non-starter. Saying oh, well, if we really moved to git this wouldn't be an issue doesn't help me much. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
* Simon Riggs (si...@2ndquadrant.com) wrote: I guarantee that if that proposal goes in, people will complain about that also. Last minute behaviour changes are bad news. I don't object to adding something, just don't take anything away. It's not like the code for it is pages long or anything. I have to disagree with this. If it goes into 9.0 this way then we're signing up to support it for *years*. With something as fragile as the existing setup (as outlined by Tom), that's probably not a good idea. We've not signed up to support the existing behaviour at all yet- alpha's aren't a guarentee of what we're going to release. The trade off is HA or queries and two modes make sense for user choice. The option isn't being thrown out, it's just being made to depend on something which is alot easier to measure while still being very useful for the trade-off you're talking about. I don't really see a downside to this, to be honest. Perhaps you could speak to the specific user experience difference that you think there would be from this change? +1 from me on Tom's proposal. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Show schema in COPY error CONTEXT strings
On Mon, May 3, 2010 at 1:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: My own preference for what to do about this is to leave the primary message texts alone and add additional error-message fields for object name and schema. This would address the need without making messages uglier for the large fraction of users who don't really care; and it would also help us get closer to the SQL standard's expectations for error reporting. This might help people who use tools to parse the output, but I'm not sure that's who is having this problem. Presumably a sufficiently well-written tool can also keep track of which schema it was targeting in the first place. I have some reservations about cluttering up all of our error messages with schema names, but the status quo is pretty bad for people who have a whole bunch of nearly-identical schemas and are trying to divine to which one of them a particular error message pertains. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Show schema in COPY error CONTEXT strings
Robert Haas robertmh...@gmail.com writes: On Mon, May 3, 2010 at 1:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: My own preference for what to do about this is to leave the primary message texts alone and add additional error-message fields for object name and schema. This would address the need without making messages uglier for the large fraction of users who don't really care; and it would also help us get closer to the SQL standard's expectations for error reporting. This might help people who use tools to parse the output, but I'm not sure that's who is having this problem. If you're using psql, \set VERBOSITY verbose would presumably show you the extra fields, or we could invent a new setting that adds just these fields. Likewise you can get it in the server log if you need it. I'm not a fan of cramming more stuff into primary message texts on the theory that that's the only useful field. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Streaming replication - unable to stop the standby
I'm currently testing SR/HS in 9.0beta1 and I noticed that it seems quite easy to end up in a situation where you have a standby that seems to be stuck in: $ psql -p 5433 psql: FATAL: the database system is shutting down but not not actually shuting down ever. I ran into that a few times now (mostly because I'm trying to chase a recovery issue I hit during earlier testing) by simply having the master iterate between a pgbench run and idle while simple doing pg_ctl restart in a loop on the standby. I do vaguely recall some discussions of that but I thought the issue git settled somehow? Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication - unable to stop the standby
Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I'm currently testing SR/HS in 9.0beta1 and I noticed that it seems quite easy to end up in a situation where you have a standby that seems to be stuck in: $ psql -p 5433 psql: FATAL: the database system is shutting down but not not actually shuting down ever. I ran into that a few times now (mostly because I'm trying to chase a recovery issue I hit during earlier testing) by simply having the master iterate between a pgbench run and idle while simple doing pg_ctl restart in a loop on the standby. I do vaguely recall some discussions of that but I thought the issue git settled somehow? Hm, I haven't pushed this hard but pg_ctl stop seems to stop the standby for me. Which subprocesses of the slave postmaster are still around? Could you attach to them with gdb and get stack traces? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication - unable to stop the standby
On Mon, May 3, 2010 at 2:04 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: I'm currently testing SR/HS in 9.0beta1 and I noticed that it seems quite easy to end up in a situation where you have a standby that seems to be stuck in: $ psql -p 5433 psql: FATAL: the database system is shutting down but not not actually shuting down ever. I ran into that a few times now (mostly because I'm trying to chase a recovery issue I hit during earlier testing) by simply having the master iterate between a pgbench run and idle while simple doing pg_ctl restart in a loop on the standby. I do vaguely recall some discussions of that but I thought the issue git settled somehow? Yes - I thought it was too. Specifically, I thought I fixed it. The default mode is 'smart' shutdown, just as it is on the primary, so it won't shut down until all clients have disconnected, but it should work provided you don't leave a session somewhere. Can you describe steps to reproduce? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication - unable to stop the standby
Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I'm currently testing SR/HS in 9.0beta1 and I noticed that it seems quite easy to end up in a situation where you have a standby that seems to be stuck in: $ psql -p 5433 psql: FATAL: the database system is shutting down but not not actually shuting down ever. I ran into that a few times now (mostly because I'm trying to chase a recovery issue I hit during earlier testing) by simply having the master iterate between a pgbench run and idle while simple doing pg_ctl restart in a loop on the standby. I do vaguely recall some discussions of that but I thought the issue git settled somehow? Hm, I haven't pushed this hard but pg_ctl stop seems to stop the standby for me. Which subprocesses of the slave postmaster are still around? Could you attach to them with gdb and get stack traces? it is not always failing to shut down - it only fails sometimes - I have not exactly pinpointed yet what it is causing this but the standby is in a weird state now: * the master is currently idle * the standby has no connections at all logs from the standby: FATAL: the database system is shutting down FATAL: the database system is shutting down FATAL: replication terminated by primary server LOG: restored log file 000100190054 from archive cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory LOG: record with zero length at 19/5578 cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory FATAL: could not connect to the primary server: could not connect to server: Connection refused Is the server running on host localhost and accepting TCP/IP connections on port 5432? could not connect to server: Connection refused Is the server running on host localhost and accepting TCP/IP connections on port 5432? cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory LOG: streaming replication successfully connected to primary FATAL: the database system is shutting down the first two FATAL: the database system is shutting down are from me trying to connect using psql after i noticed that pg_ctl failed to shutdown the slave. The next thing I tried was restarting the master - which lead to the following logs and the standby noticing that and reconnecting but you cannot actually connect... process tree for the standby is: 29523 pts/2S 0:00 /home/postgres9/pginst/bin/postgres -D /mnt/space/pgdata_standby 29524 ?Ss 0:06 \_ postgres: startup process waiting for 000100190055 29529 ?Ss 0:00 \_ postgres: writer process 29835 ?Ss 0:00 \_ postgres: wal receiver process streaming 19/5578 Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Mon, 2010-05-03 at 13:13 -0400, Stephen Frost wrote: * Simon Riggs (si...@2ndquadrant.com) wrote: I guarantee that if that proposal goes in, people will complain about that also. Last minute behaviour changes are bad news. I don't object to adding something, just don't take anything away. It's not like the code for it is pages long or anything. I have to disagree with this. If it goes into 9.0 this way then we're signing up to support it for *years*. With something as fragile as the existing setup (as outlined by Tom), that's probably not a good idea. We've not signed up to support the existing behaviour at all yet- alpha's aren't a guarentee of what we're going to release. That's a great argument, either way. We will have to live with 9.0 for many years and so that's why I mention having both. Make a choice either way and we take a risk. Why? The trade off is HA or queries and two modes make sense for user choice. The option isn't being thrown out, it's just being made to depend on something which is alot easier to measure while still being very useful for the trade-off you're talking about. I don't really see a downside to this, to be honest. Perhaps you could speak to the specific user experience difference that you think there would be from this change? +1 from me on Tom's proposal. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Andrew Dunstan escribió: I pulled together a quick hack, and here is what I get from my mirrors. I'm not sure why we get those diffs - it's a bit odd, although fairly insignificant. Well, it's a $Log$ CVS keyword -- it's not surprising that it's failing to provide exactly identical output, because it's a fairly strange keyword. Note that it's the only $Log$ in the whole tree, and the reason you don't see it in 8.1 and beyond is because that contrib module was removed. Moreover, note that this particular file got imported from another CVS repo that had already expanded this Log keyword. It's not really surprising that moving it to git fails to provide a verbatim copy. If I were worried about this, I'd edit the file in all back branches and remove the changelog ... or maybe de-magicalize it by removing the first line. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication - unable to stop the standby
Robert Haas wrote: On Mon, May 3, 2010 at 2:04 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: I'm currently testing SR/HS in 9.0beta1 and I noticed that it seems quite easy to end up in a situation where you have a standby that seems to be stuck in: $ psql -p 5433 psql: FATAL: the database system is shutting down but not not actually shuting down ever. I ran into that a few times now (mostly because I'm trying to chase a recovery issue I hit during earlier testing) by simply having the master iterate between a pgbench run and idle while simple doing pg_ctl restart in a loop on the standby. I do vaguely recall some discussions of that but I thought the issue git settled somehow? Yes - I thought it was too. Specifically, I thought I fixed it. The default mode is 'smart' shutdown, just as it is on the primary, so it won't shut down until all clients have disconnected, but it should work provided you don't leave a session somewhere. Can you describe steps to reproduce? well this is basically master and standby on the same box - with the master doing short pgbench interleaved with a sleep 20, the standby is doing nothing in terms of queries and just executing pg_ctl restart in a loop(simulating a typical maintainance reboot of a standby). Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Mon, 2010-05-03 at 13:21 -0400, Robert Haas wrote: On Mon, May 3, 2010 at 11:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm inclined to think that we should throw away all this logic and just have the slave cancel competing queries if the replay process waits more than max_standby_delay seconds to acquire a lock. What if we somehow get into a situation where the replay process is waiting for a lock over and over and over again, because it keeps killing conflicting processes but something restarts them and they take locks over again? It seems hard to ensure that replay will make adequate progress with any substantially non-zero value of max_standby_delay under this definition. That is one argument against, and a reason why just one route is bad. We already have more than one way, so another option is useful -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication - unable to stop the standby
On Mon, May 3, 2010 at 2:22 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I'm currently testing SR/HS in 9.0beta1 and I noticed that it seems quite easy to end up in a situation where you have a standby that seems to be stuck in: $ psql -p 5433 psql: FATAL: the database system is shutting down but not not actually shuting down ever. I ran into that a few times now (mostly because I'm trying to chase a recovery issue I hit during earlier testing) by simply having the master iterate between a pgbench run and idle while simple doing pg_ctl restart in a loop on the standby. I do vaguely recall some discussions of that but I thought the issue git settled somehow? Hm, I haven't pushed this hard but pg_ctl stop seems to stop the standby for me. Which subprocesses of the slave postmaster are still around? Could you attach to them with gdb and get stack traces? it is not always failing to shut down - it only fails sometimes - I have not exactly pinpointed yet what it is causing this but the standby is in a weird state now: * the master is currently idle * the standby has no connections at all logs from the standby: FATAL: the database system is shutting down FATAL: the database system is shutting down FATAL: replication terminated by primary server LOG: restored log file 000100190054 from archive cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory LOG: record with zero length at 19/5578 cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory FATAL: could not connect to the primary server: could not connect to server: Connection refused Is the server running on host localhost and accepting TCP/IP connections on port 5432? could not connect to server: Connection refused Is the server running on host localhost and accepting TCP/IP connections on port 5432? cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory LOG: streaming replication successfully connected to primary FATAL: the database system is shutting down the first two FATAL: the database system is shutting down are from me trying to connect using psql after i noticed that pg_ctl failed to shutdown the slave. The next thing I tried was restarting the master - which lead to the following logs and the standby noticing that and reconnecting but you cannot actually connect... process tree for the standby is: 29523 pts/2 S 0:00 /home/postgres9/pginst/bin/postgres -D /mnt/space/pgdata_standby 29524 ? Ss 0:06 \_ postgres: startup process waiting for 000100190055 29529 ? Ss 0:00 \_ postgres: writer process 29835 ? Ss 0:00 \_ postgres: wal receiver process streaming 19/5578 uninformed-speculation Hmm. When I committed that patch to fix smart shutdown on the standby, we discussed the fact that the startup process can't simply release its locks and die at shutdown time because the locks it holds prevent other backends from seeing the database in an inconsistent state. Therefore, if we were to terminate recovery as soon as the smart shutdown request is received, we might never complete, because a backend might be waiting on a lock that will never get released. If that's really a danger scenario, then it follows that we might also fail to shut down if we can't connect to the primary, because we might not be able to replay enough WAL to release the locks the remaining backends are waiting for. That sort of looks like what is happening to you, except based on your test scenario I can't figure out where this came from: FATAL: replication terminated by primary server /uninformed-speculation ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication - unable to stop the standby
Robert Haas wrote: On Mon, May 3, 2010 at 2:22 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: Tom Lane wrote: Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes: I'm currently testing SR/HS in 9.0beta1 and I noticed that it seems quite easy to end up in a situation where you have a standby that seems to be stuck in: $ psql -p 5433 psql: FATAL: the database system is shutting down but not not actually shuting down ever. I ran into that a few times now (mostly because I'm trying to chase a recovery issue I hit during earlier testing) by simply having the master iterate between a pgbench run and idle while simple doing pg_ctl restart in a loop on the standby. I do vaguely recall some discussions of that but I thought the issue git settled somehow? Hm, I haven't pushed this hard but pg_ctl stop seems to stop the standby for me. Which subprocesses of the slave postmaster are still around? Could you attach to them with gdb and get stack traces? it is not always failing to shut down - it only fails sometimes - I have not exactly pinpointed yet what it is causing this but the standby is in a weird state now: * the master is currently idle * the standby has no connections at all logs from the standby: FATAL: the database system is shutting down FATAL: the database system is shutting down FATAL: replication terminated by primary server LOG: restored log file 000100190054 from archive cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory LOG: record with zero length at 19/5578 cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory FATAL: could not connect to the primary server: could not connect to server: Connection refused Is the server running on host localhost and accepting TCP/IP connections on port 5432? could not connect to server: Connection refused Is the server running on host localhost and accepting TCP/IP connections on port 5432? cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory cp: cannot stat `/mnt/space/wal-archive/000100190055': No such file or directory LOG: streaming replication successfully connected to primary FATAL: the database system is shutting down the first two FATAL: the database system is shutting down are from me trying to connect using psql after i noticed that pg_ctl failed to shutdown the slave. The next thing I tried was restarting the master - which lead to the following logs and the standby noticing that and reconnecting but you cannot actually connect... process tree for the standby is: 29523 pts/2S 0:00 /home/postgres9/pginst/bin/postgres -D /mnt/space/pgdata_standby 29524 ?Ss 0:06 \_ postgres: startup process waiting for 000100190055 29529 ?Ss 0:00 \_ postgres: writer process 29835 ?Ss 0:00 \_ postgres: wal receiver process streaming 19/5578 uninformed-speculation Hmm. When I committed that patch to fix smart shutdown on the standby, we discussed the fact that the startup process can't simply release its locks and die at shutdown time because the locks it holds prevent other backends from seeing the database in an inconsistent state. Therefore, if we were to terminate recovery as soon as the smart shutdown request is received, we might never complete, because a backend might be waiting on a lock that will never get released. If that's really a danger scenario, then it follows that we might also fail to shut down if we can't connect to the primary, because we might not be able to replay enough WAL to release the locks the remaining backends are waiting for. That sort of looks like what is happening to you, except based on your test scenario I can't figure out where this came from: FATAL: replication terminated by primary server as I said before I restarted the master at that point, the standby logged the above, restored 000100190054 from the archive, tried reconnecting and logged the connection refused. a few seconds later the master was up again and the standby succeeded in reconnecting. Stefan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication - unable to stop the standby
Robert Haas robertmh...@gmail.com writes: Hmm. When I committed that patch to fix smart shutdown on the standby, we discussed the fact that the startup process can't simply release its locks and die at shutdown time because the locks it holds prevent other backends from seeing the database in an inconsistent state. Therefore, if we were to terminate recovery as soon as the smart shutdown request is received, we might never complete, because a backend might be waiting on a lock that will never get released. If that's really a danger scenario, then it follows that we might also fail to shut down if we can't connect to the primary, because we might not be able to replay enough WAL to release the locks the remaining backends are waiting for. That sort of looks like what is happening to you, except based on your test scenario I can't figure out where this came from: FATAL: replication terminated by primary server I suspect you have it right, because my experiments where the standby did shut down correctly were all done with an idle master. Seems like we could go ahead and forcibly kill the startup process *once all the standby backends are gone*. There is then no need to worry about not releasing locks, and re-establishing a consistent state when we later restart is logic that we have to have anyway. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Streaming replication - unable to stop the standby
On Mon, May 3, 2010 at 2:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Hmm. When I committed that patch to fix smart shutdown on the standby, we discussed the fact that the startup process can't simply release its locks and die at shutdown time because the locks it holds prevent other backends from seeing the database in an inconsistent state. Therefore, if we were to terminate recovery as soon as the smart shutdown request is received, we might never complete, because a backend might be waiting on a lock that will never get released. If that's really a danger scenario, then it follows that we might also fail to shut down if we can't connect to the primary, because we might not be able to replay enough WAL to release the locks the remaining backends are waiting for. That sort of looks like what is happening to you, except based on your test scenario I can't figure out where this came from: FATAL: replication terminated by primary server I suspect you have it right, because my experiments where the standby did shut down correctly were all done with an idle master. Seems like we could go ahead and forcibly kill the startup process *once all the standby backends are gone*. There is then no need to worry about not releasing locks, and re-establishing a consistent state when we later restart is logic that we have to have anyway. That's exactly what we already do. The problem is that smart shutdown doesn't actually kill off the standby backends - it waits for them to exit on their own. Except, if they're blocking on a lock that's never going to get released, then they never do. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2010-04-30 at 13:41 -0500, Kevin Grittner wrote: Surely it would confuse people to see they have fewer than min_wal_segments WAL segments. That does sound like a reasonable argument, though it also applies to wal_keep_segments, so isn't an argument either way. The user will be equally confused to see fewer WAL files than they have asked to keep. The definitions of keep in my dictionary include to restrain from removal and to retain in one's possession. It defines minimum as the least quantity assignable, admissible, or possible. If I'm understanding the semantics of this GUC (which I'll grant is not a sure thing), keep does a better job of conveying the meaning, since fewer than that are initially possible, but at least that many will be *kept* once they exist. I'm sure I'll figure it out at need, but the assertions that minimum more clearly defines the purpose is shaking *my* confidence that I understand what the GUC is for. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
On Mon, May 3, 2010 at 2:54 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Simon Riggs si...@2ndquadrant.com wrote: On Fri, 2010-04-30 at 13:41 -0500, Kevin Grittner wrote: Surely it would confuse people to see they have fewer than min_wal_segments WAL segments. That does sound like a reasonable argument, though it also applies to wal_keep_segments, so isn't an argument either way. The user will be equally confused to see fewer WAL files than they have asked to keep. The definitions of keep in my dictionary include to restrain from removal and to retain in one's possession. It defines minimum as the least quantity assignable, admissible, or possible. It's really both of those things, so we could call it wal_min_keep_segments, but I think an even better name would be bikeshed_segments. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Mon, 2010-05-03 at 13:13 -0400, Stephen Frost wrote: Perhaps you could speak to the specific user experience difference that you think there would be from this change? The difference is really to do with the weight you give to two different considerations * avoid query cancellations * avoid having recovery fall behind, so that failover time is minimised Some people recognise the trade-offs and are planning multiple standby servers dedicated to different roles/objectives. Some people envisage Hot Standby as a platform for running very fast SELECTs, for which retrying the query is a reasonable possibility and for whom keeping the standby as up-to-date as possible is an important consideration from a data freshness perspective. Others view HS as a weapon against long running queries. My initial view was that the High Availability goal/role should be the default or most likely mode of operation. I would say that the current max_standby_delay favours the HA route since it specifically limits the amount by which server can fall behind. Tom's proposed behaviour (has also been proposed before) favours the avoid query cancellation route though could lead to huge amounts of lag. I'm happy to have both options because I know this is a trade-off that solution engineers want to have control of, not something we as developers can choose ahead of time. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
* Robert Haas (robertmh...@gmail.com) wrote: On Mon, May 3, 2010 at 11:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm inclined to think that we should throw away all this logic and just have the slave cancel competing queries if the replay process waits more than max_standby_delay seconds to acquire a lock. What if we somehow get into a situation where the replay process is waiting for a lock over and over and over again, because it keeps killing conflicting processes but something restarts them and they take locks over again? It seems hard to ensure that replay will make adequate progress with any substantially non-zero value of max_standby_delay under this definition. That was my first question too- but I reread what Tom wrote and came to a different conclusion: If the reply process waits more than max_standby_delay to acquire a lock, then it will kill off *everything* it runs into from that point forward, until it's done with whatever is currently available. At that point, the 'timer' would reset back to zero. When/how that timer gets reset was a question I had, but I feel like until nothing is available makes sense and is what I assumed Tom was thinking. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] max_standby_delay considered harmful
Simon, * Simon Riggs (si...@2ndquadrant.com) wrote: Tom's proposed behaviour (has also been proposed before) favours the avoid query cancellation route though could lead to huge amounts of lag. My impression of Tom's suggestion was that it would also be a maximum amount of delay which would be allowed before killing off queries- not that it would be able to wait indefinitely until no one is blocking. Based on that, I don't know that there's really much user-seen behaviour between the two, except in 'oddball' situations, where there's a time skew between the servers, or a large lag, etc, in which case I think Tom's proposal would be more likely what's 'expected', whereas what you would get with the existing implementation (zero time delay, or far too much) would be a 'gotcha'.. Thanks, Stephen signature.asc Description: Digital signature
Re: pg_start_backup and pg_stop_backup Re: [HACKERS] Re: [COMMITTERS] pgsql: Make CheckRequiredParameterValues() depend upon correct
It's really both of those things, so we could call it wal_min_keep_segments, but I think an even better name would be bikeshed_segments. Speaking from my UI perspective, I don't think users will care what we call it. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Robert Haas robertmh...@gmail.com writes: On Mon, May 3, 2010 at 11:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm inclined to think that we should throw away all this logic and just have the slave cancel competing queries if the replay process waits more than max_standby_delay seconds to acquire a lock. What if we somehow get into a situation where the replay process is waiting for a lock over and over and over again, because it keeps killing conflicting processes but something restarts them and they take locks over again? They won't be able to take locks over again, because the lock manager won't allow requests to pass a pending previous request, except in very limited circumstances that shouldn't hold here. They'll queue up behind the replay process's lock request, not in front of it. (If that isn't the case, it needs to be fixed, quite independently of this concern.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Based on that, I don't know that there's really much user-seen behaviour between the two, except in 'oddball' situations, where there's a time skew between the servers, or a large lag, etc, in which case I think Certainly that one particular case can be solved by making the servers be in time sync a prereq for HS working (in the traditional way). And by prereq I mean a user beware documentation warning. - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation http://www.endpoint.com/ PGP Key: 0x14964AC8 201005031539 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAkvfJr0ACgkQvJuQZxSWSsgSRwCgwAZpKJDqHX28y90rCx/CNXDt JGgAoO9JeoBacvTJ09UJ+o1Nek3KtcYR =gvch -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Simon, My initial view was that the High Availability goal/role should be the default or most likely mode of operation. I would say that the current max_standby_delay favours the HA route since it specifically limits the amount by which server can fall behind. I don't understand how Tom's approach would cause the slave to be further behind than the current max_standy_delay code, and I can see ways in which it would result in less delay. So, explain? The main issue with Tom's list which struck me was that max_standby_delay was linked to the system clock. HS is going to get used by a lot of PG users who aren't running time sync on their servers, or who let it get out of whack without fixing it. I'd thought that the delay was somehow based on transaction timestamps coming from the master. Keep in mind that there will be a *lot* of people using this feature, including ones without compentent available sysadmins. The lock method appeals to me simply because it would eliminate the mass cancel issues which Greg Smith was reporting every time the timer runs down. That is, it seems to me that only the oldest queries would be cancelled and not any new ones. The biggest drawback I can see to Tom's approach is possible blocking on the slave due to the lock wait from the recovery process. However, this could be managed with the new lock-waits GUC, as well as statement timeout. Overall, I think Tom's proposal gives me what I would prefer, which is degraded performance on the slave but in ways which users are used to, rather than a lot of query cancel, which will interfere with user application porting. Would the recovery lock show up in pg_locks? That would also be a good diagnostic tool. I am happy to test some of this on Amazon or GoGrid, which is what I was planning on doing anyway. P.S. can we avoid the considered harmful phrase? It carries a lot of baggage ... -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
crazy hat on --- but do I ever quit it? Andrew Dunstan and...@dunslane.net writes: We need to be thinking more now about such a contingency. Postgres use in very large installations is now at such a level that requiring a pg_dump/pg_restore is really not an option for many users. If pg_migrator is not always going to work then we need to be addressing that now, or else it is at best a stop-gap. ISTM some sort of page layout versioning scheme might be at least part of what we'll need in the medium term. Would it be on the same complexity level to support recovering WALs from previous version? On the code maintenance alone it sounds bad enough, but apart from that? The idea of course would be then to add an Hot-Standby server running next PostgreSQL version and fed from current production server. The base backup would have to either be processed by pg_migrator or we'd have to open the possibility of starting a slave from a pg_dump, which has been talked about already. The change here would be that this initial step would not be done as part of the maintenance window. Now you tell me how awful this idea really is :) Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Alvaro Herrera wrote: Andrew Dunstan escribió: I pulled together a quick hack, and here is what I get from my mirrors. I'm not sure why we get those diffs - it's a bit odd, although fairly insignificant. Well, it's a $Log$ CVS keyword -- it's not surprising that it's failing to provide exactly identical output, because it's a fairly strange keyword. Note that it's the only $Log$ in the whole tree, and the reason you don't see it in 8.1 and beyond is because that contrib module was removed. Moreover, note that this particular file got imported from another CVS repo that had already expanded this Log keyword. It's not really surprising that moving it to git fails to provide a verbatim copy. If I were worried about this, I'd edit the file in all back branches and remove the changelog ... or maybe de-magicalize it by removing the first line. Thanks for this analysis. Based in this, I'm going to exclude these two files from the test. There are other instances I can see of $Log$, but it looks like they are not affecting us. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Mon, 2010-05-03 at 15:32 -0400, Stephen Frost wrote: Simon, * Simon Riggs (si...@2ndquadrant.com) wrote: Tom's proposed behaviour (has also been proposed before) favours the avoid query cancellation route though could lead to huge amounts of lag. My impression of Tom's suggestion was that it would also be a maximum amount of delay which would be allowed before killing off queries- not that it would be able to wait indefinitely until no one is blocking. Based on that, I don't know that there's really much user-seen behaviour between the two, except in 'oddball' situations, where there's a time skew between the servers, or a large lag, etc, in which case I think Tom's proposal would be more likely what's 'expected', whereas what you would get with the existing implementation (zero time delay, or far too much) would be a 'gotcha'.. If recovery waits for max_standby_delay every time something gets in its way, it should be clear that if many things get in its way it will progressively fall behind. There is no limit to this and it can always fall further behind. It does result in fewer cancelled queries and I do understand many may like that. That is *significantly* different from how it works now. (Plus: If there really was no difference, why not leave it as is?) The bottom line is this is about conflict resolution. There is simply no way to resolve conflicts without favouring one or other of the protagonists. Whatever mechanism you come up with that favours one will, disfavour the other. I'm happy to give choices, but I'm not happy to force just one kind of conflict resolution. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] buildfarm building all live branches from git
For those who have been following along, today I reached a bit of a milestone. The buildfarm member quoll, which is running a git-ized buildfarm client due to be released in a week or so, today successfully built all the live branches, all the way back to 7.4, from git. Note, this is running from my test git repo, not the community's repo. Sadly, that means its change links will be broken - I'm not exactly sure what gets hashed to provide a commit ID in git, but the IDs don't match between these two repos. See http://www.pgbuildfarm.org/cgi-bin/show_status.pl?member=quoll. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Mon, 2010-05-03 at 15:39 -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Mon, May 3, 2010 at 11:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm inclined to think that we should throw away all this logic and just have the slave cancel competing queries if the replay process waits more than max_standby_delay seconds to acquire a lock. What if we somehow get into a situation where the replay process is waiting for a lock over and over and over again, because it keeps killing conflicting processes but something restarts them and they take locks over again? They won't be able to take locks over again, because the lock manager won't allow requests to pass a pending previous request, except in very limited circumstances that shouldn't hold here. They'll queue up behind the replay process's lock request, not in front of it. (If that isn't the case, it needs to be fixed, quite independently of this concern.) Most conflicts aren't lock-manager locks, they are snapshot conflicts, though clearly different workloads will have different characteristics. Some conflicts are buffer conflicts and the semantics of buffer cleanup locks and many other internal locks are that shared locks queue jump past exclusive lock requests. Not something we should touch, now at least. I understand that you aren't impressed by everything about the current patch but rushed changes may not help either. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Mon, May 3, 2010 at 3:39 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, May 3, 2010 at 11:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm inclined to think that we should throw away all this logic and just have the slave cancel competing queries if the replay process waits more than max_standby_delay seconds to acquire a lock. What if we somehow get into a situation where the replay process is waiting for a lock over and over and over again, because it keeps killing conflicting processes but something restarts them and they take locks over again? They won't be able to take locks over again, because the lock manager won't allow requests to pass a pending previous request, except in very limited circumstances that shouldn't hold here. They'll queue up behind the replay process's lock request, not in front of it. (If that isn't the case, it needs to be fixed, quite independently of this concern.) Well, the new backends needn't try to take the same locks as the existing backends - the point is that in the worst case this proposal means waiting max_standby_delay for EACH replay that requires taking a lock. And that might be a LONG time. One idea I had while thinking this over was to bound the maximum amount of unapplied WAL rather than the absolute amount of time lag. Now, that's a little fruity, because your WAL volume might fluctuate considerably, so you wouldn't really know how far the slave was behind the master chronologically. However, it would avoid all the time skew issues, and it would also more accurately model the idea of a bound on recovery time should we need to promote the standby to master, so maybe it works out to a win. You could still end up stuck semi-permanently behind, but never by more than N segments. Stephen's idea of a mode where we wait up to max_standby_delay for a lock but then kill everything in our path until we've caught up again is another possible way of approaching this problem, although it may lead to kill storms. Some of that may be inevitable, though: a bound on WAL lag has the same issue - if the primary is generating WAL faster than the standby can apply it, the standby will eventually decide to slaughter everything in its path. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] buildfarm building all live branches from git
On Mon, May 3, 2010 at 4:04 PM, Andrew Dunstan and...@dunslane.net wrote: For those who have been following along, today I reached a bit of a milestone. The buildfarm member quoll, which is running a git-ized buildfarm client due to be released in a week or so, today successfully built all the live branches, all the way back to 7.4, from git. Note, this is running from my test git repo, not the community's repo. Sadly, that means its change links will be broken - I'm not exactly sure what gets hashed to provide a commit ID in git, but the IDs don't match between these two repos. See http://www.pgbuildfarm.org/cgi-bin/show_status.pl?member=quoll. Thanks for working on this. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontaine dfonta...@hi-media.com wrote: Now you tell me how awful this idea really is :) I'm not sure I can count that high. :-) ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
On Mon, 2010-05-03 at 16:12 -0400, Robert Haas wrote: On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontaine dfonta...@hi-media.com wrote: Now you tell me how awful this idea really is :) I'm not sure I can count that high. :-) You don't have to... NaN Joshua D. Drake ...Robert -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Bruce Momjian wrote: As a summary, let me list the migrations pg_migrator supports: 8.3 - 8.4 8.4 - 9.0 8.3 - 9.0 Surprisingly, it is 8.3 - 8.4 that has the most restrictions because it doesn't have access to the features we added in Postgres 9.0. Tom is right that the code could be cleaned up if we removed 8.3 - 8.4, but more importantly the documentation would be clearer. I think it's extremely valuable that either 8.3 or 8.4 can be upgraded to 9.0. But let's face it: in the long run, the number of people who are going to use pg_migrator for a 8.3-8.4 migration, but that's haven't already done so, is a small user base. The feature set improvement in 8.4 had a lot of great stuff, but few that were compelling from a now I can do something completely impossible before! standpoint. As was noted recently during the Native DB replication for PG discussion over on pgsql-general last week, there are plenty of people happily running a stable 8.3 who just ignore 8.4 altogether for that reason. The replication features in 9.0 are compelling in that way though, and I expect to see plenty of upgrades to that version from both 8.3 and 8.4 installs. If that works fine right now, I would prefer to see that documented as a special case two-versions at once situation that people shouldn't necessarily expect in the future, but certainly valuable to keep going if the maintenance burden isn't so bad. Balancing out development reality with the ideal situation from the perspective of [potential|current] customers that I deal with every day, what I would prefer to see here is: 1) Commit a streamlined pg_migrator that only handles conversions with 9.0 as a target into contrib, and ship it with 9.0. Like Bruce, I had presumed that the discussion about whether that was going to happen would happen in parallel with beta (read: right now), rather than its already being too late to even consider. I think it's completely bizarre from an advocacy standpoint to even consider that you wouldn't ship such a tool with the core database, now that it's been around for long enough to have a positive track record. 2) Deprecate the pg_migrator hosted on pg_foundry as only being recommended for limited 8.3-8.4 upgrades. Essentially stop active development on the version there, and focus on the one in contrib/ instead. People who want an improved 8.3-8.4 tool can always contract with someone to backport fixes needed for their particular use case. I think we're past the point where the community at large (meaning: mainly Bruce right now) should be expected to do that, now that 9.0 is coming out, so long as 8.3 to 9.0 conversions are available too. I can't imagine suggesting to anyone that they upgrade in-place from 8.3 to 8.4 right now. Everybody I talk to who isn't already on 8.4 is delaying upgrades in anticipation of 9.0 later this year or early next. My main issues with this project continuing to be hosted in pgfoundry are: 1) Perceived lack of confidence and/or legitimacy for it as an in-place upgrade solution, which would be a terrible PR move. When people ask about in-place upgrades and I tell them there's a tool you can download for that, they look at me in terror and ask if I'm serious that it isn't just included in the core code. The improvement between answering that way and saying yes, the tool for 8.3 and 8.4 is included with the core distribution, from the perspective of selling people on adopting PostgreSQL, cannot be overstated. 2) Anyone who looks at pgfoundry for more than a few minutes walks away covered with the scent of dead projects. One reason for that is that related to how painful it is to develop there. I don't want to reignite a full anti-pgfoundry discussion here. Suffice it to say that there are many of us who just can't bear working with CVS anymore who have just given up on doing anything useful to projects hosted there. Something that's in core (and therefore included in the git conversion already being published) is much easier to work with and submit patches against. I'm already dumping git clones of the PG repo on every system I do serious work on. If each of those were then capable of generating pg_migrator patches I could submit, I would actually do that each time I use the tool for an upgrade and notice how it could be improved. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
On Mon, May 3, 2010 at 11:37 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm inclined to think that we should throw away all this logic and just have the slave cancel competing queries if the replay process waits more than max_standby_delay seconds to acquire a lock. What if we somehow get into a situation where the replay process is waiting for a lock over and over and over again, because it keeps killing conflicting processes but something restarts them and they take locks over again? It seems hard to ensure that replay will make adequate progress with any substantially non-zero value of max_standby_delay under this definition. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Further Hot Standby documentation required
Simon Riggs wrote: On Mon, 2010-05-03 at 10:50 +0100, Simon Riggs wrote: On Mon, 2010-05-03 at 12:17 +0300, Heikki Linnakangas wrote: * wal_level doesn't describe what the impacts are on a standby if the level is changed on the primary, nor is there a caution or a warning of any kind. For example, if a standby is setup with hot_standby = on and the primary is set wal_level = archive, does the standby start working if the primary changes wal_level = hot_standby? What happens if the primary is set wal_level = hot_standby and is then changed to archive? Hmm, feels like it should rather be documented in the hot_standby setting, it affects the standby not the master after all. Danger of action at a distance. The change is on the master, but the effect is on the standby. The person changing the master must be warned of the danger that they will bring down the standby, so it must go with the parameter, not only with the HS docs. Don't really understand why you left that bit out. Are you just leaving this for me, or is there a specific objection to adding the warning? Sorry, I just didn't have the time energy to figure out what to do about that. Feel free to fix as you see fit. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] missing file in git repo
Aidan Van Dyk wrote: HINT: It's all been done and posted to -hackers before too... Along with comparisons on on whte one-time conversions fare (parsecvs, cvs2svn/git), etc, as well as long discussion on which keyword you want expanded, and which you don't, etc. bla bla bla... And in some cases, even indexed to make them easier to find again due to links at http://wiki.postgresql.org/wiki/Switching_PostgreSQL_from_CVS_to_Git There's an item on there that talks about Tom's specific requests related to git backbranch workflow, from the last time this came up. Given Andrew's breakthrough with the buildfarm today, and so many developers due to be in the same place here this month, that might be something worth working through the details of live during PGCon. Most of the other steps beyond that are on Peter's plate, which isn't as concerning to me because we at least know for sure he's on-board with pushing toward the conversion. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Robert Haas wrote: On Sun, May 2, 2010 at 3:45 PM, Dimitri Fontaine dfonta...@hi-media.com wrote: Now you tell me how awful this idea really is :) I'm not sure I can count that high. :-) While I can't improve on Robert's reply, I can supply a PDF about how pg_migrator works: http://momjian.us/main/presentations/technical.html#pg_migrator -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Further Hot Standby documentation required
On Mon, 2010-05-03 at 23:28 +0300, Heikki Linnakangas wrote: Sorry, I just didn't have the time energy to figure out what to do about that. Feel free to fix as you see fit. Yeh, I sometimes feel like that towards other hacker's comments as well. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_migrator to /contrib in a later 9.0 beta
Greg Smith wrote: Bruce Momjian wrote: As a summary, let me list the migrations pg_migrator supports: 8.3 - 8.4 8.4 - 9.0 8.3 - 9.0 Surprisingly, it is 8.3 - 8.4 that has the most restrictions because it doesn't have access to the features we added in Postgres 9.0. Tom is right that the code could be cleaned up if we removed 8.3 - 8.4, but more importantly the documentation would be clearer. I think it's extremely valuable that either 8.3 or 8.4 can be upgraded to 9.0. But let's face it: in the long run, the number of people who are going to use pg_migrator for a 8.3-8.4 migration, but that's haven't already done so, is a small user base. The feature set improvement in 8.4 had a lot of great stuff, but few that were compelling from a now I can do something completely impossible before! standpoint. As was noted recently during the Native DB replication for PG discussion over on pgsql-general last week, there are plenty of people happily running a stable 8.3 who just ignore 8.4 altogether for that reason. The replication features in 9.0 are compelling in that way though, and I expect to see plenty of upgrades to that version from both 8.3 and 8.4 installs. If that works fine right now, I would prefer to see that documented as a special case two-versions at once situation that people shouldn't necessarily expect in the future, but certainly valuable to keep going if the maintenance burden isn't so bad. Until we have some kind of delete the incompatibile format step in major releases, my bet is that pg_migrator will support many previous major versions, or not work at all. It is hard to see why it would work for some major versions and not others given our current code. 2) Deprecate the pg_migrator hosted on pg_foundry as only being recommended for limited 8.3-8.4 upgrades. Essentially stop active development on the version there, and focus on the one in contrib/ instead. People who want an improved 8.3-8.4 tool can always contract with someone to backport fixes needed for their particular use case. I think we're past the point where the community at large (meaning: mainly Bruce right now) should be expected to do that, now that 9.0 is coming out, so long as 8.3 to 9.0 conversions are available too. I can't imagine suggesting to anyone that they upgrade in-place from 8.3 to 8.4 right now. Everybody I talk to who isn't already on 8.4 is delaying upgrades in anticipation of 9.0 later this year or early next. Assuming pg_migrator moves to /contrib, I don't plan on doing much to improve the pgfoundry version unless people find specific bugs. I have not released a 9.0-compatible version there for that reason. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Greg, Robert, Certainly that one particular case can be solved by making the servers be in time sync a prereq for HS working (in the traditional way). And by prereq I mean a user beware documentation warning. Last I checked, you work with *lots* of web developers and web companies. I'm sure you can see the issue with the above. Stephen's idea of a mode where we wait up to max_standby_delay for a lock but then kill everything in our path until we've caught up again is another possible way of approaching this problem, although it may lead to kill storms. Personally, I thought that the kill storms were exactly what was wrong with max_standby_delay. That is, with MSD, no matter *what* your settings or traffic are, you're going to get query cancel occasionally. I don't see the issue with Tom's approach from a wait perspective. The max wait becomes 1.001X max_standby_delay; there's no way I can think of that replay would wait longer than that. I've yet to see an explanation why it would be longer. Simon's assertion that not all operations take a conventional lock is a much more serious potential flaw. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] what is good solution for support NULL inside string_to_array function?
Hello
I understand why we don't support expression 'null'::sometype. But it
does problems with array deserialisation.
postgres=# select array_to_string(ARRAY[10,20,30,NULL,30], '|');
array_to_string
-
10|20|30|30
(1 row)
quietly removing NULL is maybe good for compatibility but is wrong for
functionality. Can we enhance function array_to_string and
string_to_array like:
CREATE OR REPLACE FUNCTION array_to_string(dta anyarray, sep text,
nullsym text)
RETURNS text AS $$
SELECT array_to_string(ARRAY(SELECT coalesce(v::text,$3)
FROM unnest($1) g(v)),$2)
$$ LANGUAGE sql;
CREATE FUNCTION
Time: 231.445 ms
postgres=# select array_to_string(ARRAY[10,20,30,NULL,30], '|', '');
array_to_string
-
10|20|30||30
(1 row)
Time: 230.879 ms
postgres=# select array_to_string(ARRAY[10,20,30,NULL,30], '|', 'NULL');
array_to_string
--
10|20|30|NULL|30
(1 row)
Time: 2.031 ms
CREATE OR REPLACE FUNCTION string_to_array(str text, sep text, nullsym text)
RETURNS text[] AS $$
SELECT ARRAY(SELECT CASE WHEN v $3 THEN v ELSE NULL END
FROM unnest(string_to_array($1,$2)) g(v))
$$ LANGUAGE sql;
CREATE FUNCTION
Time: 29.044 ms
postgres=# SELECT string_to_array('10,20,30,,40',',','');
string_to_array
{10,20,30,NULL,40}
(1 row)
postgres=# SELECT string_to_array('10,20,30,,40',',','')::int[];
string_to_array
{10,20,30,NULL,40}
(1 row)
it is correct?
other ideas?
Regards
Pavel Stehule
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] what is good solution for support NULL inside string_to_array function?
quietly removing NULL is maybe good for compatibility but is wrong for functionality. I agree. I wasn't aware of this little misfeature. Default display for NULL should be a zero-length string. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY is not working
On 4/30/2010 1:56 AM, Jaime Casanova wrote: 2010/4/30 Jaime Casanova jcasa...@systemguards.com.ec: Hi, COPY is not working on latest HEAD? regression=# select * from a; aa 32 56 (2 rows) regression=# COPY a TO '/tmp/copy_test'; COPY 0 ah! this is because COPY doesn't follow inherited tables... should it? The TRUNCATE command's behavior was changed in that regard. What TRUNCATE did in 8.3 is now TRUNCATE ONLY in 8.4. I don't see a reason why COPY should not follow suit. Jan -- Anyone who trades liberty for security deserves neither liberty nor security. -- Benjamin Franklin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] max_standby_delay considered harmful
Simon Riggs wrote: On Mon, 2010-05-03 at 13:13 -0400, Stephen Frost wrote: Perhaps you could speak to the specific user experience difference that you think there would be from this change? The difference is really to do with the weight you give to two different considerations * avoid query cancellations * avoid having recovery fall behind, so that failover time is minimised Some people recognise the trade-offs and are planning multiple standby servers dedicated to different roles/objectives. I understand Simon's point that the two behaviors have different benefits. However, I believe few users will be able to understand when to use which. As I remember, 9.0 has two behaviors: o master delays vacuum cleanup o slave delays WAL application and in 9.1 we will be adding: o slave communicates snapshots to master How would this figure into what we ultimately want in 9.1? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY is not working
Jan Wieck janwi...@yahoo.com writes: On 4/30/2010 1:56 AM, Jaime Casanova wrote: ah! this is because COPY doesn't follow inherited tables... should it? The TRUNCATE command's behavior was changed in that regard. What TRUNCATE did in 8.3 is now TRUNCATE ONLY in 8.4. I don't see a reason why COPY should not follow suit. How about because it's inappropriate for dump/reload, which is the main use-case for COPY? In any case it'd be weird for COPY FROM and COPY TO to behave differently on this point, but I don't really see how COPY FROM would do anything intelligent for inheritance. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] what is good solution for support NULL inside string_to_array function?
Josh Berkus j...@agliodbs.com writes: quietly removing NULL is maybe good for compatibility but is wrong for functionality. I agree. I wasn't aware of this little misfeature. Default display for NULL should be a zero-length string. That's just as broken as Pavel's suggestion. Unless you have something that is guaranteed distingishable from the output of any non-null value, you really can't make a significant improvement here. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] what is good solution for support NULL inside string_to_array function?
Tom Lane wrote: Default display for NULL should be a zero-length string. That's just as broken as Pavel's suggestion. Unless you have something that is guaranteed distingishable from the output of any non-null value, you really can't make a significant improvement here. Right. This is the problem we solved in CSV processing by distinguishing between quoted and unquoted values that could be null - the CSV rule is that a null value isn't quoted. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] TOAST code ignores freespace (was Tweak TOAST code)
On 5/2/2010 10:34 AM, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: Not commenting further on that patch, but I notice that when we UPDATE the toasting algorithm takes no account of the available freespace on the current block. If we are updating and the space available would make a difference to the row length chosen, it seems like it would be more beneficial to trim the row and encourage HOT updates. That doesn't strike me as a terribly good idea: it would make the behavior of TOAST significantly more difficult to predict. Also, what happens if we force a row to a smaller size and then it doesn't fit anyway (eg because someone else inserted another row on the page while we were busy doing this)? Spend even more cycles to un-toast back to the normal size, to be consistent with ordinary cross-page updates? Pretty much every previous discussion of tweaking the TOAST behavior has focused on giving the user more control (indeed, the patch you mention could be seen as doing that). What you're suggesting here would give the user less control, as well as less predictability. Correct. And on top of that, the cost/benefit of the proposed change will be extremely hard to evaluate since freespace and the value of HOT depend very much on access patterns. If we want to substantially do better, we need to use a bigger hammer. TOAST's largest performance benefit lies in the fact that it reduces the size of the main tuple, which is the data that travels in intermediate result sets throughout the executor. Reducing that size results in smaller sort sets, more in memory operations, fewer blocks seqscanned for keys and all that. Suppose we had something similar to the NULL value bitmap, specifying plain or compressed values (not TOAST references), that are moved to a shadow tuple inside the toast table. Suppose further we had some statistics about how often attributes appear in a qualification (i.e. end up in a scan key or scan filter or other parts of the qual expression list). Not sure, maybe we even want to know how often or seldom an attribute is heap_getattr()'d at all. Those don't need to be accurate counts. Small random samples will probably do. ANALYZE could evaluate those statistics and adjust the shadow storage settings per attribute accordingly. I can imagine many applications, where this would shrink the main tuples to almost nothing at all. There are for sure a lot of if's and suppose in the above and the impact of a fundamental on disk storage format change needs to be justified by a really big gain. And yes, Simon, this also depends a lot on access patterns. But if you try to gain more from TOAST, I'd look for something like this instead of making the target tuple size dynamic. Jan -- Anyone who trades liberty for security deserves neither liberty nor security. -- Benjamin Franklin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY is not working
On 5/3/2010 10:51 PM, Tom Lane wrote: Jan Wieck janwi...@yahoo.com writes: On 4/30/2010 1:56 AM, Jaime Casanova wrote: ah! this is because COPY doesn't follow inherited tables... should it? The TRUNCATE command's behavior was changed in that regard. What TRUNCATE did in 8.3 is now TRUNCATE ONLY in 8.4. I don't see a reason why COPY should not follow suit. How about because it's inappropriate for dump/reload, which is the main use-case for COPY? In any case it'd be weird for COPY FROM and COPY TO to behave differently on this point, but I don't really see how COPY FROM would do anything intelligent for inheritance. Dump would have to switch to the more consistent syntax using the COPY ONLY table ... form. As simple as that. You are right that COPY ONLY ... FROM makes no sense at all. But I would consider that the corner case, not the defining general rule. One could make exactly the save inconsistency argument about the fact that there is a SELECT ... FROM ONLY, UPDATE ONLY and DELETE FROM ONLY, but no INSERT INTO ONLY. As of right now, all commands that retrieve or affect existing rows, except for COPY, do honor the ONLY keyword to stop inheritance. And while pg_dump may be one of the main users of COPY, I think there are quite a few other. I don't really see why COPY is so special. Jan -- Anyone who trades liberty for security deserves neither liberty nor security. -- Benjamin Franklin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] what is good solution for support NULL inside string_to_array function?
2010/5/4 Tom Lane t...@sss.pgh.pa.us: Josh Berkus j...@agliodbs.com writes: quietly removing NULL is maybe good for compatibility but is wrong for functionality. I agree. I wasn't aware of this little misfeature. Default display for NULL should be a zero-length string. That's just as broken as Pavel's suggestion. Unless you have something that is guaranteed distingishable from the output of any non-null value, you really can't make a significant improvement here. I wouldn't modify current two params string_to_array and array_to_string function. So there are not any default string (maybe empty string) for NULL. My proposal is new three params functions with explicit null string definition. This cannot break compatibility and enhance functionality - It is just short cut for code from my proposal - in C this functionality can by implemented much faster. Regards Pavel regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CP949 for EUC-KR?
Hi, I'm Korean.
CP51949 is EUC-KR correct.
so, that defined code is correct too.
But in Korea, EUC-KR code is not good to use all Korean character.
In recent years, many people in Korea use the CP949 code.
MS Windows codepage also is CP949.
- Original Message -
From: Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp
To: pgsql-hackers@postgresql.org
Sent: Tuesday, April 27, 2010 7:27 PM
Subject: [HACKERS] CP949 for EUC-KR?
I heard pg_get_encoding_from_locale() failed in kor locale.
WARNING: could not determine encoding for locale kor: codeset is CP949
I found the following description in the web:
CP949 is EUC-KR, extended with UHC (Unified Hangul Code).
http://www.opensource.apple.com/source/libiconv/libiconv-13.2/libiconv/lib/cp949.h
but we define CP51949 for EUC-KR in chklocale.c.
{PG_EUC_KR, CP51949}, /* or 20949 ? */
Which is the compatible codeset with our PG_EUC_KR encoding?
949, 51949, or 20949? Should we add (or replace) CP949 for EUC-KR?
Regards,
---
Takahiro Itagaki
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
