Re: [HACKERS] [PATCH] XLogReader v2
Hi, On Thursday, July 19, 2012 07:18:08 PM Satoshi Nagayasu wrote: I agree with that we need more sophisticated way to share the code between the backend and several utilities (including xlogdump), but AFAIK, a contrib module must allow to be built *without* the core source tree. I don't think thats reasonable. The amount of code duplication required to support that usecase is just not reasonable. Especially if you want to support pre 9.3 and 9.3+. So, I agree with that we need another way to share the code between the backend and the related utilities. Any good ideas? Well, the primary patch in the above email was infrastructure to at least make reading xlog possible without much internal knowledge. But that will still leave the debugging routines... Which imo already is too much. It would mean that the users using older major version could not take advantage of new features and enhancements of the latest xlogdump, but it's not what I wanted, actually. I personally don't see that as a big problem. If xlogdump really reuses the normal infrastructure of the server the amount of code you need to backport is *way* much smaller should it ever be actually needed. Any contrib module must be able to be built with only the header files and the shared libraries when using PGXS. So, it could not assume that it has the core source tree. (If we need to assume that, I think xlogdump needs to be put into the core/bin directory.) We possibly could get away by defining an extra .a containing the necessary object files. Not nice, but... Imo it *should* be in src/bin though. Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] enhanced error fields
On 18 July 2012 19:13, Pavel Stehule pavel.steh...@gmail.com wrote:
* renamed erritem to err_generic_string
* fixed CSVlog generation
* new file /utils/error/relerror.c with axillary functions -
declarations are in utils/rel.h
Why has this revision retained none of my editorialisations? In
particular, none of the documentation updates that I made were
retained.
You also haven't included changes where I attempted to make very large
ereport statements (often with verbose use of ternary conditionals)
clearer, nor have you included my adjustments to normalise the
appearance of new code to be consistent with existing code in various
ways.
You don't have to agree with all of those things of course, but you
should have at least commented on them. I didn't spend time cleaning
things up only to have those changes ignored. I'm particularly
surprised that the documentation alterations were not retained, as you
yourself asked me to make those revisions.
/* file error location */
- if (Log_error_verbosity = PGERROR_VERBOSE)
+
{
StringInfoData msgbuf;
Why have you retained the scope here? Couldn't you have just moved the
single declaration instead?
--
Peter Geoghegan http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] embedded list v2
On 5 July 2012 02:49, Peter Geoghegan pe...@2ndquadrant.com wrote: On 28 June 2012 19:20, Andres Freund and...@2ndquadrant.com wrote: 0001-Add-embedded-list-interface.patch Looks good now? I have a few gripes. We are passed the nominal deadline. Had you planned on getting back to me this commitfest? If not, I'll shelve my review of 0002-Remove-usage-of-lib-dllist.h-and-replace-it-by-the-n.patch until commitfest 2012-09, and mark this returned with feedback. -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] enhanced error fields
2012/7/23 Peter Geoghegan pe...@2ndquadrant.com:
On 18 July 2012 19:13, Pavel Stehule pavel.steh...@gmail.com wrote:
* renamed erritem to err_generic_string
* fixed CSVlog generation
* new file /utils/error/relerror.c with axillary functions -
declarations are in utils/rel.h
Why has this revision retained none of my editorialisations? In
particular, none of the documentation updates that I made were
retained.
I am sorry, I can't do merge - I try to use diff but without success,
so I did structure changes and merge with your patch postponed
You also haven't included changes where I attempted to make very large
ereport statements (often with verbose use of ternary conditionals)
clearer, nor have you included my adjustments to normalise the
appearance of new code to be consistent with existing code in various
ways.
You don't have to agree with all of those things of course, but you
should have at least commented on them. I didn't spend time cleaning
things up only to have those changes ignored. I'm particularly
surprised that the documentation alterations were not retained, as you
yourself asked me to make those revisions.
again, I am sorry - my last patch should to define structure (files) -
because it was significant topic. When we will find a agreement, then
I'll merge changes messages, comments, doc
/* file error location */
- if (Log_error_verbosity = PGERROR_VERBOSE)
+
{
StringInfoData msgbuf;
Why have you retained the scope here? Couldn't you have just moved the
single declaration instead?
I am not sure, I have to recheck it.
Regards
Pavel
--
Peter Geoghegan http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] embedded list v2
On Monday, July 23, 2012 12:55:01 PM Peter Geoghegan wrote: On 5 July 2012 02:49, Peter Geoghegan pe...@2ndquadrant.com wrote: On 28 June 2012 19:20, Andres Freund and...@2ndquadrant.com wrote: 0001-Add-embedded-list-interface.patch Looks good now? I have a few gripes. We are passed the nominal deadline. Had you planned on getting back to me this commitfest? If not, I'll shelve my review of 0002-Remove-usage-of-lib-dllist.h-and-replace-it-by-the-n.patch until commitfest 2012-09, and mark this returned with feedback. I was actually waiting for the second review ;). Sorry for the misunderstanding. There is no rule that review cannot happen outside the commitfest, so I would appreciate if we could push this further... Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
On 07/23/2012 12:37 AM, David Fetter wrote: On Tue, Jul 17, 2012 at 06:56:50PM -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jul 16, 2012 at 3:18 PM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, while we are on the subject: hasn't this split completely broken the statistics about backend-initiated writes? Yes, it seems to have done just that. This implies that nobody has done pull-the-plug testing on either HEAD or 9.2 since the checkpointer split went in (2011-11-01), because even a modicum of such testing would surely have shown that we're failing to fsync a significant fraction of our write traffic. Furthermore, I would say that any performance testing done since then, if it wasn't looking at purely read-only scenarios, isn't worth the electrons it's written on. In particular, any performance gain that anybody might have attributed to the checkpointer splitup is very probably hogwash. This is not giving me a warm feeling about our testing practices. Is there any part of this that the buildfarm, or some other automation framework, might be able to handle? I'm not sure how you automate testing a pull-the-plug scenario. The buildfarm is not at all designed to test performance. That's why we want a performance farm. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
On Mon, Jul 23, 2012 at 08:29:16AM -0400, Andrew Dunstan wrote: On 07/23/2012 12:37 AM, David Fetter wrote: On Tue, Jul 17, 2012 at 06:56:50PM -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jul 16, 2012 at 3:18 PM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, while we are on the subject: hasn't this split completely broken the statistics about backend-initiated writes? Yes, it seems to have done just that. This implies that nobody has done pull-the-plug testing on either HEAD or 9.2 since the checkpointer split went in (2011-11-01), because even a modicum of such testing would surely have shown that we're failing to fsync a significant fraction of our write traffic. Furthermore, I would say that any performance testing done since then, if it wasn't looking at purely read-only scenarios, isn't worth the electrons it's written on. In particular, any performance gain that anybody might have attributed to the checkpointer splitup is very probably hogwash. This is not giving me a warm feeling about our testing practices. Is there any part of this that the buildfarm, or some other automation framework, might be able to handle? I'm not sure how you automate testing a pull-the-plug scenario. I have a dim memory of how the FreeBSD project was alleged to have done it, namely by rigging a serial port (yes, it was that long ago) to the power supply of another machine and randomly cycling the power. The buildfarm is not at all designed to test performance. That's why we want a performance farm. Right. Apart from hardware, what are we stalled on? Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
On 07/23/2012 08:41 AM, David Fetter wrote: The buildfarm is not at all designed to test performance. That's why we want a performance farm. Right. Apart from hardware, what are we stalled on? Software :-) I am trying to find some cycles to get something going. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
On 07/23/2012 08:29 PM, Andrew Dunstan wrote: I'm not sure how you automate testing a pull-the-plug scenario. fire up kvm or qemu instances, then kill 'em. -- Craig Ringer -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
On 07/23/2012 09:04 AM, Craig Ringer wrote: On 07/23/2012 08:29 PM, Andrew Dunstan wrote: I'm not sure how you automate testing a pull-the-plug scenario. fire up kvm or qemu instances, then kill 'em. Yeah, maybe. Knowing just when to kill them might be an interesting question. I'm also unsure how much nice cleanup the host supervisor does in such cases. VMs are wonderful things, but they aren't always the answer. I'm not saying they aren't here, just wondering. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
On 07/23/2012 09:47 PM, Andrew Dunstan wrote: On 07/23/2012 09:04 AM, Craig Ringer wrote: On 07/23/2012 08:29 PM, Andrew Dunstan wrote: I'm not sure how you automate testing a pull-the-plug scenario. fire up kvm or qemu instances, then kill 'em. Yeah, maybe. Knowing just when to kill them might be an interesting question. I'm also unsure how much nice cleanup the host supervisor does in such cases. VMs are wonderful things, but they aren't always the answer. I'm not saying they aren't here, just wondering. I've done some testing with this, and what it boils down to is that any data that made it to the virtual disk is persistent after a VM kill. Anything in dirty buffers on the VM guest is lost. It's a very close match for real hardware. I haven't tried to examine the details of the handling of virtualised disk hardware write caches, but disks should be in write-through mode anyway. A `kill -9` will clear 'em for sure, anyway, as the guest has no chance to do any cleanup. One of the great things about kvm and qemu for this sort of testing is that it's just another program. There's very little magic, and it's quite easy to test and trace. I have a qemu/kvm test harness I've been using for another project that I need to update and clean up as it'd be handy for this. It's just a matter of making the time, as it's been a busy few days. -- Craig Ringer -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Using pg_upgrade on log-shipping standby servers
On Sat, Jul 21, 2012 at 8:57 AM, Bruce Momjian br...@momjian.us wrote:
On Sat, Jul 21, 2012 at 11:24:21AM +0300, Peter Eisentraut wrote:
On fre, 2012-07-20 at 13:11 -0400, Bruce Momjian wrote:
I think the commands to run after pg_upgrade --link completes on both
primary and standby might be as easy as:
cd /u/pg/pgsql.old/data
find . -links 1 -exec cp {} /u/pgsql/data \;
Why would we want anything more complicated than this?
In practice these are on different machines, and the way the machines
are connected could vary wildly. So an automated solution might be
difficult to find.
Yeah, I was thinking of just suggesting scp as a doc example and let
users adjust that:
cd /u/pg/pgsql.old/data
find . -links 1 -exec scp {} postg...@momjian.us:/u/pgsql/data \;
Relying on the number of hard links seems very fragile. For example,
it'll break if you are using copy mode. And it won't work on Windows,
either.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.2 release schedule
On Sat, Jul 21, 2012 at 6:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: It looks like the most optimistic schedule we could possibly make, assuming no show-stopper bugs are reported, is: * Wrap beta3 on August 2 for release Monday Aug 6 * RC1 two weeks later (release Aug 20) * 9.2.0 two weeks after that (release Sept 3) Now that schedule isn't actually going to work so far as the USA contingent is concerned, because Sept 3 is Labor Day. But we could figure that there's room for one week's slip in there, either for RC1 or final, giving us a release date of Sept 10. Again, this is assuming no major bugs are found to force more slippage or additional betas/RCs. But I think it's reasonable to shoot for If All Goes Well. Comments, objections? Seems OK, but I think we need to work a little harder on evicting some things from the list of open items. I don't think all of the things listed in the blockers section really are, and I'm not sure what needs to be done about some of the things that are there. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] XLogReader v2
On Mon, Jul 23, 2012 at 3:19 AM, Andres Freund and...@2ndquadrant.com wrote: On Thursday, July 19, 2012 07:18:08 PM Satoshi Nagayasu wrote: I agree with that we need more sophisticated way to share the code between the backend and several utilities (including xlogdump), but AFAIK, a contrib module must allow to be built *without* the core source tree. I don't think thats reasonable. The amount of code duplication required to support that usecase is just not reasonable. Especially if you want to support pre 9.3 and 9.3+. It seems like the direction this is going is that the xlog reading stuff should be a library which is used by both the backend and 1 or more xlog decoding tools. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] enhanced error fields
On Mon, Jul 23, 2012 at 7:03 AM, Pavel Stehule pavel.steh...@gmail.com wrote: You don't have to agree with all of those things of course, but you should have at least commented on them. I didn't spend time cleaning things up only to have those changes ignored. I'm particularly surprised that the documentation alterations were not retained, as you yourself asked me to make those revisions. again, I am sorry - my last patch should to define structure (files) - because it was significant topic. When we will find a agreement, then I'll merge changes messages, comments, doc It seems clear that it's not reasonable to expect any more review to be done here this CommitFest, given that the changes from reviews already made haven't been incorporated into the patch. Therefore, I'm going to mark this one Returned with Feedback. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.2 release schedule
Robert Haas robertmh...@gmail.com writes: Seems OK, but I think we need to work a little harder on evicting some things from the list of open items. I don't think all of the things listed in the blockers section really are, and I'm not sure what needs to be done about some of the things that are there. I've got the libpq row processor thing. That and the CHECK NO INHERIT syntax thing are definitely release-blockers, because we won't be able to change such decisions post-release (well, we could, but the pain to benefit ratio is bad). I guess the SPGiST vs HS issue is a blocker too. A lot of the rest look like pre-existing bugs to me. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] enhanced error fields
On 23 July 2012 15:23, Robert Haas robertmh...@gmail.com wrote: It seems clear that it's not reasonable to expect any more review to be done here this CommitFest, given that the changes from reviews already made haven't been incorporated into the patch. Therefore, I'm going to mark this one Returned with Feedback. I agree that that's the right thing to do at this point. -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Checkpointer split has broken things dramatically (was Re: DELETE vs TRUNCATE explanation)
On Mon, Jul 23, 2012 at 5:41 AM, David Fetter da...@fetter.org wrote: On Mon, Jul 23, 2012 at 08:29:16AM -0400, Andrew Dunstan wrote: I'm not sure how you automate testing a pull-the-plug scenario. I have a dim memory of how the FreeBSD project was alleged to have done it, namely by rigging a serial port (yes, it was that long ago) to the power supply of another machine and randomly cycling the power. On Linux, echo b /proc/sysrq-trigger Is supposed to take it down instantly, with no flushing of dirty buffers. Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] XLogReader v2
On Monday, July 23, 2012 04:17:39 PM Robert Haas wrote: On Mon, Jul 23, 2012 at 3:19 AM, Andres Freund and...@2ndquadrant.com wrote: On Thursday, July 19, 2012 07:18:08 PM Satoshi Nagayasu wrote: I agree with that we need more sophisticated way to share the code between the backend and several utilities (including xlogdump), but AFAIK, a contrib module must allow to be built *without* the core source tree. I don't think thats reasonable. The amount of code duplication required to support that usecase is just not reasonable. Especially if you want to support pre 9.3 and 9.3+. It seems like the direction this is going is that the xlog reading stuff should be a library which is used by both the backend and 1 or more xlog decoding tools. Thats fine for the xlogreader itself - it only uses stuff from headers. The problem is that the xlog debugging/printing infrastructure is pretty much guaranteed to include just about the whole backend... Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] sortsupport for text
On Fri, Jun 15, 2012 at 6:10 PM, Tom Lane t...@sss.pgh.pa.us wrote: I would be concerned about this if it were per-sort-tuple wastage, but what I understood to be happening was that this was a single instance of an expansible buffer (per sort, perhaps, but still just one buffer). And, as you keep pointing out when it suits your argument, it's relatively uncommon to be sorting enormous values anyway. So no, I am not particularly worried about that. If you are, there are more important places to be concerned about allocation pad wastage, starting with palloc itself. And, of course, palloc doesn't use power-of-two allocation up to infinity either, as proposed here; beyond a certain limit it switches to calling malloc(), which doesn't use power-of-two allocations for large requests either, at least not in glibc - for anything 128kB or above, it uses mmap to grab something of exactly the requested size (rounded up to the OS page size) and returns it the OS unconditionally when it's freed. However, what this really boils down to is that you and Peter don't like this line of code: + tss-buflen1 = TYPEALIGN(TEXTBUFLEN, len1); What would you like it to say instead? The obvious formulation is: while (len1 tss-buflen1) tss-buflen *= 2; Or perhaps the following, which will normally be more efficient, though possibly not as efficient as what I've got there now: tss-buflen = 1 ffs(len1); Of course if len1 happens to be large enough that the high-order bit is set, the first proposal above will fail to terminate and the second one will produce 0. That can't really happen in practice because a toasted datum is limited to 1GB, of course, and I suppose a lot of code would need to be adjusted if we ever wanted to raise that limit, so maybe it's not worth worrying about it. So, is one of the above code fragments what people want? Or something else? Thanks, -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] XLogReader v2
On Mon, Jul 23, 2012 at 11:04 AM, Andres Freund and...@2ndquadrant.com wrote: On Monday, July 23, 2012 04:17:39 PM Robert Haas wrote: On Mon, Jul 23, 2012 at 3:19 AM, Andres Freund and...@2ndquadrant.com wrote: On Thursday, July 19, 2012 07:18:08 PM Satoshi Nagayasu wrote: I agree with that we need more sophisticated way to share the code between the backend and several utilities (including xlogdump), but AFAIK, a contrib module must allow to be built *without* the core source tree. I don't think thats reasonable. The amount of code duplication required to support that usecase is just not reasonable. Especially if you want to support pre 9.3 and 9.3+. It seems like the direction this is going is that the xlog reading stuff should be a library which is used by both the backend and 1 or more xlog decoding tools. Thats fine for the xlogreader itself - it only uses stuff from headers. The problem is that the xlog debugging/printing infrastructure is pretty much guaranteed to include just about the whole backend... Could that be fixed by moving the debugging routines into a separate set of files, instead of having them lumped in with the code that applies those xlog records? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] sortsupport for text
On 23 July 2012 16:09, Robert Haas robertmh...@gmail.com wrote: However, what this really boils down to is that you and Peter don't like this line of code: + tss-buflen1 = TYPEALIGN(TEXTBUFLEN, len1); I can only speak for myself, though I agree with your summary here. What would you like it to say instead? The obvious formulation is: while (len1 tss-buflen1) tss-buflen *= 2; That's what I had in mind. +1. Or perhaps the following, which will normally be more efficient, though possibly not as efficient as what I've got there now: tss-buflen = 1 ffs(len1); I'm sorry, I don't follow you here. What is ffs() ? -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] sortsupport for text
On Mon, Jul 23, 2012 at 11:34 AM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 23 July 2012 16:09, Robert Haas robertmh...@gmail.com wrote: However, what this really boils down to is that you and Peter don't like this line of code: + tss-buflen1 = TYPEALIGN(TEXTBUFLEN, len1); I can only speak for myself, though I agree with your summary here. What would you like it to say instead? The obvious formulation is: while (len1 tss-buflen1) tss-buflen *= 2; That's what I had in mind. +1. Or perhaps the following, which will normally be more efficient, though possibly not as efficient as what I've got there now: tss-buflen = 1 ffs(len1); I'm sorry, I don't follow you here. What is ffs() ? Sorry, fls, not ffs. I always get those mixed up. See src/port/fls.c -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] sortsupport for text
On 23 July 2012 16:36, Robert Haas robertmh...@gmail.com wrote: On Mon, Jul 23, 2012 at 11:34 AM, Peter Geoghegan pe...@2ndquadrant.com wrote: tss-buflen = 1 ffs(len1); I'm sorry, I don't follow you here. What is ffs() ? Sorry, fls, not ffs. I always get those mixed up. See src/port/fls.c Oh, okay. Since, I infer, we're starting from a buffer-size that's a power-of-two anyway, is there really any advantage in doing this rather than just doubling the buffer size each time? -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] sortsupport for text
On Mon, Jul 23, 2012 at 12:07 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: On 23 July 2012 16:36, Robert Haas robertmh...@gmail.com wrote: On Mon, Jul 23, 2012 at 11:34 AM, Peter Geoghegan pe...@2ndquadrant.com wrote: tss-buflen = 1 ffs(len1); I'm sorry, I don't follow you here. What is ffs() ? Sorry, fls, not ffs. I always get those mixed up. See src/port/fls.c Oh, okay. Since, I infer, we're starting from a buffer-size that's a power-of-two anyway, is there really any advantage in doing this rather than just doubling the buffer size each time? Well, if you're using a builtin fls rather than our src/port implementation, it's probably a single machine language instruction instead of a loop. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] XLogReader v2
On Monday, July 23, 2012 05:11:20 PM Robert Haas wrote: On Mon, Jul 23, 2012 at 11:04 AM, Andres Freund and...@2ndquadrant.com wrote: On Monday, July 23, 2012 04:17:39 PM Robert Haas wrote: On Mon, Jul 23, 2012 at 3:19 AM, Andres Freund and...@2ndquadrant.com wrote: On Thursday, July 19, 2012 07:18:08 PM Satoshi Nagayasu wrote: I agree with that we need more sophisticated way to share the code between the backend and several utilities (including xlogdump), but AFAIK, a contrib module must allow to be built *without* the core source tree. I don't think thats reasonable. The amount of code duplication required to support that usecase is just not reasonable. Especially if you want to support pre 9.3 and 9.3+. It seems like the direction this is going is that the xlog reading stuff should be a library which is used by both the backend and 1 or more xlog decoding tools. Thats fine for the xlogreader itself - it only uses stuff from headers. The problem is that the xlog debugging/printing infrastructure is pretty much guaranteed to include just about the whole backend... Could that be fixed by moving the debugging routines into a separate set of files, instead of having them lumped in with the code that applies those xlog records? Its a major effort. Those function use elog(), stringinfo and lots of other stuff... I am hesitant to start working on that. On the other hand - I think an in-core xlogdump would be great and sensible thing; but I can live with using my hacked up version that simply links to the backend... Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] XLogReader v2
On Mon, Jul 23, 2012 at 12:13 PM, Andres Freund and...@2ndquadrant.com wrote: Could that be fixed by moving the debugging routines into a separate set of files, instead of having them lumped in with the code that applies those xlog records? Its a major effort. Those function use elog(), stringinfo and lots of other stuff... I am hesitant to start working on that. On the other hand - I think an in-core xlogdump would be great and sensible thing; but I can live with using my hacked up version that simply links to the backend... The stringinfo thing has long been an annoyance to me. libpq has PQExpBuffer which is the exact same thing. I don't like that we have two implementations of that in two different code bases, and you have to remember to spell it right depending on where you are. I'm not sure exactly what the best way to fix that is, but it sure is a pain in the neck. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] XLogReader v2
2012/07/24 1:15, Robert Haas wrote: On Mon, Jul 23, 2012 at 12:13 PM, Andres Freund and...@2ndquadrant.com wrote: Could that be fixed by moving the debugging routines into a separate set of files, instead of having them lumped in with the code that applies those xlog records? Its a major effort. Those function use elog(), stringinfo and lots of other stuff... I am hesitant to start working on that. On the other hand - I think an in-core xlogdump would be great and sensible thing; but I can live with using my hacked up version that simply links to the backend... The stringinfo thing has long been an annoyance to me. libpq has PQExpBuffer which is the exact same thing. I don't like that we have two implementations of that in two different code bases, and you have to remember to spell it right depending on where you are. I'm not sure exactly what the best way to fix that is, but it sure is a pain in the neck. Does it make sense to make some static library which can be referred from both the backend and several client utilities, including libpq? Or just a dynamic link be preferred? Despite I do not have a clear idea right now, is it time to start thinking of it? Regards, -- Satoshi Nagayasu sn...@uptime.jp Uptime Technologies, LLC. http://www.uptime.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] XLogReader v2
On Mon, Jul 23, 2012 at 1:03 PM, Satoshi Nagayasu sn...@uptime.jp wrote: The stringinfo thing has long been an annoyance to me. libpq has PQExpBuffer which is the exact same thing. I don't like that we have two implementations of that in two different code bases, and you have to remember to spell it right depending on where you are. I'm not sure exactly what the best way to fix that is, but it sure is a pain in the neck. Does it make sense to make some static library which can be referred from both the backend and several client utilities, including libpq? Or just a dynamic link be preferred? Despite I do not have a clear idea right now, is it time to start thinking of it? IMHO, yes. I'm not sure exactly what the right way to do it is, but I think we need something along these lines. We've got some pg_dump code - in dumputils.c - that is also linked into other applications such as psql, too, which is another pile of grottiness for which we need a better solution. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] postgres 9 bind address for replication
On Mon, Jul 23, 2012 at 8:34 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Jul 21, 2012 at 1:24 AM, Adam Crews adam.cr...@gmail.com wrote: Hello, I'm sorry for cross-posting, however I originally posted this to pgsql-general list, but didnt get any replies. Then I posted to pgsql-cluster-hackers..., and now here. I’m using pg 9.1.3 on CentOS 5 and have a few slave databases setup using the built in streaming replication. On the slaves I set the “listen_addresses” config option to an ip address for a virtual alias on my network interfaces. The host has an address of 10.1.1.10, and there is a virtual alias of 10.1.1.40 that the slave postmaster binds to. When the slave makes it connection to the master to start replication the source address for the connection is the host address, not the virtual alias address. Connections appear to come from 10.1.1.10, instead of the slave postmaster address of 10.1.1.40. This seems like a bug to me. I could understand that if the postmaster is listening on all interfaces, then it should use whatever the IP is for the for the host, but in an instance where the postmaster has been configured to listen to a specific address it seems like the call to start the replication should be passed that address so connections come from the slave postmaster’s IP, instead of the host. Is there a config option that can be used to adjust this? I've looked in the docs, but haven't found one yet. Is this perhaps a bug, or lack of feature? I don't think it's a bug, because the behavior you're hoping for might not be what everyone would want in a similar situation. It might qualify as an unimplemented feature. This mailing list isn't heavily used and this seems a bit off-topic for it anyway; you might want to try a different one for further discussion of this issue. So, I think this, as Robert states, an unimplemented feature. For my situation it would be very useful to have an option to be able to specify the source address for replication. I discovered this because I bind the listen address for postgres to a single address even though the host system may have multiple addresses. I then use that single address in iptables rules on other systems. Since I expect the slave to be at a .40 address, but the replication comes from the primary address of the interface (in this case .10), my iptables rules were missing the access for the slave to connect to the master. This site http://linux-ip.net/html/routing-saddr-selection.html describes the behavior I'm seeing. How do I go about requesting a config option that would allow me to specify the source address for the replication connections? Thanks -Adam -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [BUGS] BUG #6748: sequence value may be conflict in some cases
meixiangm...@huawei.com writes: [ freshly-created sequence has wrong state after crash ] I didn't believe this at first, but sure enough, it fails just as described if you force a crash between the first and second nextval calls for the sequence. This used to work ... The change that broke it turns out to be the ALTER SEQUENCE OWNED BY call that we added to serial-column creation circa 8.2; although on closer inspection I think any ALTER SEQUENCE before the first nextval call would be problematic. The real issue is the ancient kluge in sequence creation that writes something different into the WAL log than what it leaves behind in shared buffers: /* We do not log first nextval call, so advance sequence here */ /* Note we are scribbling on local tuple, not the disk buffer */ newseq-is_called = true; newseq-log_cnt = 0; The tuple in buffers has log_cnt = 1, is_called = false, but the initial XLOG_SEQ_LOG record shows log_cnt = 0, is_called = true. So if we crash at this point, after recovery it looks like one nextval() has already been done. However, AlterSequence generates another XLOG_SEQ_LOG record based on what's in shared buffers, so after replay of that, we're back to the original state where it does not appear that any nextval() has been done. I'm of the opinion that this kluge needs to be removed; it's just insane that we're not logging the same state we leave in our buffers. To do that, we need to fix nextval() so that the first nextval call generates an xlog entry; that is, if we are changing is_called to true we ought to consider that as a reason to force an xlog entry. I think way back when we thought it was a good idea to avoid making two xlog entries during creation and immediate use of a sequence, but considering all the other xlog entries involved in creation of a sequence object, this is a pretty silly optimization. (Besides, it merely postpones the first nextval-driven xlog entry from the first to the second nextval call.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgbench -i order of vacuum
On Fri, Jul 20, 2012 at 12:26 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: On Fri, Jul 20, 2012 at 7:57 AM, Amit Kapila amit.kap...@huawei.com wrote: The command it executes is vacuum analyze .., so it will do analyze also on table which means it will collect stats corresponding to table and index. Are there stats collected on indexes? Only for expression indexes, which there aren't any of in the standard pgbench scenario. I don't see a reason not to change the ordering as you suggest. OK, done. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] postgres 9 bind address for replication
On 07/23/2012 02:23 PM, Adam Crews wrote: On Mon, Jul 23, 2012 at 8:34 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Jul 21, 2012 at 1:24 AM, Adam Crews adam.cr...@gmail.com wrote: Hello, I'm sorry for cross-posting, however I originally posted this to pgsql-general list, but didnt get any replies. Then I posted to pgsql-cluster-hackers..., and now here. I’m using pg 9.1.3 on CentOS 5 and have a few slave databases setup using the built in streaming replication. On the slaves I set the “listen_addresses” config option to an ip address for a virtual alias on my network interfaces. The host has an address of 10.1.1.10, and there is a virtual alias of 10.1.1.40 that the slave postmaster binds to. When the slave makes it connection to the master to start replication the source address for the connection is the host address, not the virtual alias address. Connections appear to come from 10.1.1.10, instead of the slave postmaster address of 10.1.1.40. This seems like a bug to me. I could understand that if the postmaster is listening on all interfaces, then it should use whatever the IP is for the for the host, but in an instance where the postmaster has been configured to listen to a specific address it seems like the call to start the replication should be passed that address so connections come from the slave postmaster’s IP, instead of the host. Is there a config option that can be used to adjust this? I've looked in the docs, but haven't found one yet. Is this perhaps a bug, or lack of feature? I don't think it's a bug, because the behavior you're hoping for might not be what everyone would want in a similar situation. It might qualify as an unimplemented feature. This mailing list isn't heavily used and this seems a bit off-topic for it anyway; you might want to try a different one for further discussion of this issue. So, I think this, as Robert states, an unimplemented feature. For my situation it would be very useful to have an option to be able to specify the source address for replication. I discovered this because I bind the listen address for postgres to a single address even though the host system may have multiple addresses. I then use that single address in iptables rules on other systems. Since I expect the slave to be at a .40 address, but the replication comes from the primary address of the interface (in this case .10), my iptables rules were missing the access for the slave to connect to the master. This site http://linux-ip.net/html/routing-saddr-selection.html describes the behavior I'm seeing. How do I go about requesting a config option that would allow me to specify the source address for the replication connections? You just have :-) You could just add an iptables rule redirecting .10 packets on port 5432 (or whatever you're using) appropriately. We don't have any provision for binding the local end of any connection AFAIK. So the first question is Do we want to? and the second is If yes, when and how? I don't see that replication should be a special case - if this is worth providing for it should be applicable to all clients, ISTM. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [patch] libpq one-row-at-a-time API
Here is a simple test program that takes a SELECT
query, reads it and outputs a COPY-formatted stream
to standard output, to simulate some activity.
It operates on 3 modes, specified by commant-line switches:
-f Load full resultset at once - old way.
-s Single-Row mode using PQgetResult().
-z Single-Row mode using PQgetRowData().
It is compiled with 2 different libpqs that correspond to
single-row-modeX branches in my github repo:
rowdump1 - libpq with rowBuf + PQgetRowData(). rowBuf is
required for PQgetRowData.
[ https://github.com/markokr/postgres/tree/single-row-mode1 ]
rowdump2 - Plain libpq patched for single-row mode.
No PQgetRowData() here.
[ https://github.com/markokr/postgres/tree/single-row-mode2 ]
Notes:
* Hardest part is picking realistic queries that matter.
It's possible to construct artificial queries that make
results go either way.
* It does not make sense for compare -f with others. But it
does make sense to compare -f from differently patched libpqs
to detect any potential slowdowns.
* The time measured is User Time of client process.
---
QUERY: select 1,200,30,rpad('x',30,'z') from
generate_series(1,500)
./rowdump1 -f: 3.90 3.90 3.93 avg: 3.91
./rowdump2 -f: 4.03 4.13 4.05 avg: 4.07
./rowdump1 -s: 6.26 6.33 6.49 avg: 6.36
./rowdump2 -s: 7.48 7.46 7.50 avg: 7.48
./rowdump1 -z: 2.88 2.90 2.79 avg: 2.86
QUERY: select
rpad('x',10,'z'),rpad('x',20,'z'),rpad('x',30,'z'),rpad('x',40,'z'),rpad('x',50,'z'),rpad('x',60,'z')
from generate_series(1,300)
./rowdump1 -f: 6.29 6.36 6.14 avg: 6.26
./rowdump2 -f: 6.79 6.69 6.72 avg: 6.73
./rowdump1 -s: 7.71 7.72 7.80 avg: 7.74
./rowdump2 -s: 8.14 8.16 8.57 avg: 8.29
./rowdump1 -z: 6.45 5.15 5.16 avg: 5.59
QUERY: select
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100
from generate_series(1,80)
./rowdump1 -f: 5.68 5.66 5.72 avg: 5.69
./rowdump2 -f: 5.69 5.84 5.67 avg: 5.73
./rowdump1 -s: 7.68 7.76 7.67 avg: 7.70
./rowdump2 -s: 7.57 7.54 7.62 avg: 7.58
./rowdump1 -z: 2.78 2.82 2.72 avg: 2.77
QUERY: select 1000,rpad('x', 400, 'z'),rpad('x', 4000, 'z') from
generate_series(1,10)
./rowdump1 -f: 2.71 2.66 2.58 avg: 2.65
./rowdump2 -f: 3.11 3.14 3.16 avg: 3.14
./rowdump1 -s: 2.64 2.61 2.64 avg: 2.63
./rowdump2 -s: 3.15 3.11 3.11 avg: 3.12
./rowdump1 -z: 2.53 2.51 2.46 avg: 2.50
---
Test code attached. Please play with it.
By this test, both rowBuf and PQgetRowData() look good.
--
marko
pg1 = /opt/apps/pgsql92mode1
pg2 = /opt/apps/pgsql92mode2
CFLAGS = -O -g -Wall
all: rowdump1 rowdump2
rowdump1: rowdump.c
$(CC) -I$(pg1)/include $(CFLAGS) -o $@ $ -L$(pg1)/lib
-Wl,-rpath=$(pg1)/lib -lpq -DHAVE_ROWDATA
rowdump2: rowdump.c
$(CC) -I$(pg2)/include $(CFLAGS) -o $@ $ -L$(pg2)/lib
-Wl,-rpath=$(pg2)/lib -lpq
clean:
rm -f rowdump1 rowdump2
xtest.sh
Description: Bourne shell script
#include stdio.h
#include stdlib.h
#include string.h
#include unistd.h
#include getopt.h
#include libpq-fe.h
struct Context {
PGconn *db;
int count;
char *buf;
int buflen;
int bufpos;
};
static void die(PGconn *db, const char *msg)
{
if (db)
fprintf(stderr, %s: %s\n, msg, PQerrorMessage(db));
else
fprintf(stderr, %s\n, msg);
exit(1);
}
static void out_flush(struct Context *ctx)
{
int out;
if (!ctx-buf)
return;
out = write(1, ctx-buf, ctx-bufpos);
if (out != ctx-bufpos)
die(NULL, failed to write file);
ctx-bufpos = 0;
ctx-buflen = 0;
free(ctx-buf);
ctx-buf = NULL;
}
static void out_char(struct Context *ctx, char c)
{
if (ctx-bufpos + 1 ctx-buflen) {
if (!ctx-buf) {
ctx-buflen = 16;
ctx-buf = malloc(ctx-buflen);
if (!ctx-buf)
die(NULL, failed to allocate buffer);
} else {
ctx-buflen *= 2;
ctx-buf = realloc(ctx-buf, ctx-buflen);
if (!ctx-buf)
die(NULL, failed to resize buffer);
}
}
ctx-buf[ctx-bufpos++] = c;
}
static void proc_value(struct Context *ctx, const char *val, int vlen)
{
int i;
char c;
for (i = 0; i vlen; i++) {
c = val[i];
switch (c) {
case '\\':
out_char(ctx, '\\');
out_char(ctx, '\\');
break;
case '\t':
out_char(ctx, '\\');
out_char(ctx, 't');
break;
case '\n':
out_char(ctx, '\\');
out_char(ctx, 'n');
break;
case '\r':
out_char(ctx, '\\');
out_char(ctx, 'r');
break;
default:
out_char(ctx, c);
break;
}
}
}
static void proc_row(struct Context *ctx, PGresult *res, int tup)
{
int n = PQnfields(res);
const char *val;
int
Re: [HACKERS] 9.2 release schedule
On Mon, Jul 23, 2012 at 10:29:06AM -0400, Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: Seems OK, but I think we need to work a little harder on evicting some things from the list of open items. I don't think all of the things listed in the blockers section really are, and I'm not sure what needs to be done about some of the things that are there. I've got the libpq row processor thing. That and the CHECK NO INHERIT syntax thing are definitely release-blockers, because we won't be able to change such decisions post-release (well, we could, but the pain to benefit ratio is bad). I guess the SPGiST vs HS issue is a blocker too. A lot of the rest look like pre-existing bugs to me. The only preexisting issues listed under Blockers for 9.2 are GiST indexes vs fuzzy comparisons used by geometric types and Should we fix tuple limit handling, or redefine 9.x behavior as correct?. Also, I'm not sure what exactly the keepalives item indicates. Whether every regression deserves to block the release is, of course, a separate question. I think WAL files which were restored from the archive are archived again is the thorniest regression, and we don't yet have a patch. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Using pg_upgrade on log-shipping standby servers
On mån, 2012-07-23 at 10:08 -0400, Robert Haas wrote: Relying on the number of hard links seems very fragile. For example, it'll break if you are using copy mode. And it won't work on Windows, either. pg_upgrade could remember the list of files that the user would need to copy to the remote server (i.e., the list of files pg_upgrade itself copied or linked) and write that to a file. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [patch] libpq one-row-at-a-time API
On Mon, Jul 23, 2012 at 2:05 PM, Marko Kreen mark...@gmail.com wrote:
Here is a simple test program that takes a SELECT
query, reads it and outputs a COPY-formatted stream
to standard output, to simulate some activity.
It operates on 3 modes, specified by commant-line switches:
-f Load full resultset at once - old way.
-s Single-Row mode using PQgetResult().
-z Single-Row mode using PQgetRowData().
It is compiled with 2 different libpqs that correspond to
single-row-modeX branches in my github repo:
rowdump1 - libpq with rowBuf + PQgetRowData(). rowBuf is
required for PQgetRowData.
[ https://github.com/markokr/postgres/tree/single-row-mode1 ]
rowdump2 - Plain libpq patched for single-row mode.
No PQgetRowData() here.
[ https://github.com/markokr/postgres/tree/single-row-mode2 ]
Notes:
* Hardest part is picking realistic queries that matter.
It's possible to construct artificial queries that make
results go either way.
* It does not make sense for compare -f with others. But it
does make sense to compare -f from differently patched libpqs
to detect any potential slowdowns.
* The time measured is User Time of client process.
---
QUERY: select 1,200,30,rpad('x',30,'z') from
generate_series(1,500)
./rowdump1 -f: 3.90 3.90 3.93 avg: 3.91
./rowdump2 -f: 4.03 4.13 4.05 avg: 4.07
./rowdump1 -s: 6.26 6.33 6.49 avg: 6.36
./rowdump2 -s: 7.48 7.46 7.50 avg: 7.48
./rowdump1 -z: 2.88 2.90 2.79 avg: 2.86
QUERY: select
rpad('x',10,'z'),rpad('x',20,'z'),rpad('x',30,'z'),rpad('x',40,'z'),rpad('x',50,'z'),rpad('x',60,'z')
from generate_series(1,300)
./rowdump1 -f: 6.29 6.36 6.14 avg: 6.26
./rowdump2 -f: 6.79 6.69 6.72 avg: 6.73
./rowdump1 -s: 7.71 7.72 7.80 avg: 7.74
./rowdump2 -s: 8.14 8.16 8.57 avg: 8.29
./rowdump1 -z: 6.45 5.15 5.16 avg: 5.59
QUERY: select
1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100
from generate_series(1,80)
./rowdump1 -f: 5.68 5.66 5.72 avg: 5.69
./rowdump2 -f: 5.69 5.84 5.67 avg: 5.73
./rowdump1 -s: 7.68 7.76 7.67 avg: 7.70
./rowdump2 -s: 7.57 7.54 7.62 avg: 7.58
./rowdump1 -z: 2.78 2.82 2.72 avg: 2.77
QUERY: select 1000,rpad('x', 400, 'z'),rpad('x', 4000, 'z') from
generate_series(1,10)
./rowdump1 -f: 2.71 2.66 2.58 avg: 2.65
./rowdump2 -f: 3.11 3.14 3.16 avg: 3.14
./rowdump1 -s: 2.64 2.61 2.64 avg: 2.63
./rowdump2 -s: 3.15 3.11 3.11 avg: 3.12
./rowdump1 -z: 2.53 2.51 2.46 avg: 2.50
---
Test code attached. Please play with it.
By this test, both rowBuf and PQgetRowData() look good.
I agree on performance grounds. It's important for libpq to be fast.
It seems odd (but maybe ok) that you have to set the single row mode
on the connection only to have the server reset it whenever you call a
send function: maybe rename to PQsetResultSingleRowMode?
Does PQgetRowData() break the ability to call PQgetvalue() against the
result as well as other functions like PQgetisnull()? If so, I
object.
merlin
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [patch] libpq one-row-at-a-time API
On Tue, Jul 24, 2012 at 12:27 AM, Merlin Moncure mmonc...@gmail.com wrote: On Mon, Jul 23, 2012 at 2:05 PM, Marko Kreen mark...@gmail.com wrote: It seems odd (but maybe ok) that you have to set the single row mode on the connection only to have the server reset it whenever you call a send function: maybe rename to PQsetResultSingleRowMode? Server does not reset it, it's purely client-side flag. It is reset by next PQsend*/PQexec* call. If there are several resultsets sent by server for one query, they all share the same setting. I don't think extra-long function names that describe exactly the function behavior are improvement over shorter but inexact names, as you need to read the docs anyway to know the real behavior. But its a matter of taste, so it can be renamed if people want it. Alternative is to create parallel PQsend* functions that set the flag. It gets rid of the possibility of any extra activity between PQsend and PQsetSingleRowMode(). But it seems messy to do that just for single-row-mode, instead it makes sense to have PQsend* that takes a flags argument to tune it's behavior. But that is worth doing only if we have more flags than just one. Does PQgetRowData() break the ability to call PQgetvalue() against the result as well as other functions like PQgetisnull()? If so, I object. I don't get what are you objecting to. The PQgetRowData() is called instead of PQgetResult() and it returns zero-row PGresult for row header and list of PGdataValues that point to actual data. You call the value functions, they don't crash, but as the result has zero rows (the data is not copied to it) they don't do anything useful either. The whole point of the API is to avoid the unnecessary copying. You can mix the calls to PQgetRowData() and PQgetResult() during one resultset, it works fine although does not seem that useful. I guess you fear that some code can unexpectedly see new behavior, and that exactly why the row-callback API needs to be scrapped - it allowed such possibility. But with PQsetSingleRowMode and PQgetRowData, the new behavior is seen only by new code that clearly expects it, so I don't see what the problem is. -- marko -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Tab completion for INHERIT and NO INHERIT
I was playing around with partitioning recently, and was annoyed that tab completion doesn't work for setting the table to (dis)inherit. It seemed easy enough to add. For the NO INHERIT case, it would be nice to complete only table names currently being inherited by the first table, but I don't know how to make that happen. Cheers, Jeff complete_inherit_v1.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] postgres 9 bind address for replication
On Jul 23, 2012, at 11:45 AM, Andrew Dunstan and...@dunslane.net wrote: On 07/23/2012 02:23 PM, Adam Crews wrote: On Mon, Jul 23, 2012 at 8:34 AM, Robert Haas robertmh...@gmail.com wrote: On Sat, Jul 21, 2012 at 1:24 AM, Adam Crews adam.cr...@gmail.com wrote: Hello, I'm sorry for cross-posting, however I originally posted this to pgsql-general list, but didnt get any replies. Then I posted to pgsql-cluster-hackers..., and now here. I’m using pg 9.1.3 on CentOS 5 and have a few slave databases setup using the built in streaming replication. On the slaves I set the “listen_addresses” config option to an ip address for a virtual alias on my network interfaces. The host has an address of 10.1.1.10, and there is a virtual alias of 10.1.1.40 that the slave postmaster binds to. When the slave makes it connection to the master to start replication the source address for the connection is the host address, not the virtual alias address. Connections appear to come from 10.1.1.10, instead of the slave postmaster address of 10.1.1.40. This seems like a bug to me. I could understand that if the postmaster is listening on all interfaces, then it should use whatever the IP is for the for the host, but in an instance where the postmaster has been configured to listen to a specific address it seems like the call to start the replication should be passed that address so connections come from the slave postmaster’s IP, instead of the host. Is there a config option that can be used to adjust this? I've looked in the docs, but haven't found one yet. Is this perhaps a bug, or lack of feature? I don't think it's a bug, because the behavior you're hoping for might not be what everyone would want in a similar situation. It might qualify as an unimplemented feature. This mailing list isn't heavily used and this seems a bit off-topic for it anyway; you might want to try a different one for further discussion of this issue. So, I think this, as Robert states, an unimplemented feature. For my situation it would be very useful to have an option to be able to specify the source address for replication. I discovered this because I bind the listen address for postgres to a single address even though the host system may have multiple addresses. I then use that single address in iptables rules on other systems. Since I expect the slave to be at a .40 address, but the replication comes from the primary address of the interface (in this case .10), my iptables rules were missing the access for the slave to connect to the master. This site http://linux-ip.net/html/routing-saddr-selection.html describes the behavior I'm seeing. How do I go about requesting a config option that would allow me to specify the source address for the replication connections? You just have :-) You could just add an iptables rule redirecting .10 packets on port 5432 (or whatever you're using) appropriately. We don't have any provision for binding the local end of any connection AFAIK. So the first question is Do we want to? and the second is If yes, when and how? I don't see that replication should be a special case - if this is worth providing for it should be applicable to all clients, ISTM. cheers andrew I've worked around the issue for now by just allowing the host address instead of the postmaster ip address in my iptables. I agree that if implemented it should be an option available to all clients, not just the replicator. To give a little background, here's what I'm doing. In my environment we have dedicated ip addresses to specific postmasters, then sometimes run multiple postmasters on the same server. Our data files are stored on san storage and can be moved between different servers. With this setup I've abstracted my database from the underlying hardware and can move the postmasters between hosts with minimal downtime. (3 minutes to stop the db, unmount the storage, mount on the new host, start the postmaster). The ip travels to the new host as a virtual interface along with the database files. To make sure clients can't access a db they are not supposed to, I control access via iptables and pg_hba. I can now do system patching with little impact to the application. Thanks, -Adam -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] isolation check takes a long time
On fre, 2012-07-20 at 13:15 -0400, Andrew Dunstan wrote: On 07/19/2012 09:54 AM, Andrew Dunstan wrote: Meanwhile, I would like to remove the prepared_transactions test from the main isolation schedule, and add a new Make target which runs that test explicitly. Is there any objection to that? Here's the patch for that. Why was this backpatched to 9.1? That doesn't seem appropriate. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
