[HACKERS] CF3+4 (was Re: Parallel query execution)
At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? I know various people (myself included) have been trying to keep CF3 moving, e.g. sending followup mail, adjusting patch status, etc. I want to help, but I don't know what's wrong. What are the committers working on, and what is the status of the Ready for commiter patches? Is the problem that the patches marked Ready aren't, in fact, ready? Or is it lack of feedback from authors? Or something else? Would it help at all to move all pending items (i.e. anything less than ready) from CF3 to CF4, just so that the committers have only one list to look at, while reviewers can work on the other? Only psychological, but maybe that's better than the current situation? -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] passing diff options to pg_regress
Hi Peter, Idea is really very good. About the patch: Patch looks good to me. Applied cleanly on latest sources. make / make install / make check / initdb everything works well. Tested with few options and it is working well. However, I think you need to add this in docs. Letting people know about this environment variable to make use of that. Thanks On Wed, Jan 16, 2013 at 1:21 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 14, 2013 at 6:57 AM, Peter Eisentraut pete...@gmx.net wrote: I sometimes find it useful to view a regression test difference using other diff options, such as -u -w or more context. There is currently no easy way to accomplish that. I suggest allowing to override the diff options using an environment variable, such as PG_REGRESS_DIFF_OPTS. The patch is very small. Great idea. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Jeevan B Chalke Senior Software Engineer, RD EnterpriseDB Corporation The Enterprise PostgreSQL Company Phone: +91 20 30589500 Website: www.enterprisedb.com EnterpriseDB Blog: http://blogs.enterprisedb.com/ Follow us on Twitter: http://www.twitter.com/enterprisedb This e-mail message (and any attachment) is intended for the use of the individual or entity to whom it is addressed. This message contains information from EnterpriseDB Corporation that may be privileged, confidential, or exempt from disclosure under applicable law. If you are not the intended recipient or authorized to receive this for the intended recipient, any use, dissemination, distribution, retention, archiving, or copying of this communication is strictly prohibited. If you have received this e-mail in error, please notify the sender immediately by reply e-mail and delete this message.
Re: [HACKERS] Parallel query execution
On Tue, Jan 15, 2013 at 11:07 PM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: There are still 34 items needing attention in CF3. I suggest that, if you have some spare time, your help would be very much appreciated there. The commitfest that started on Jan 15th has 65 extra items. Anything currently listed in CF3 can rightfully be considered to be part of CF4, too. In case you hadn't noticed, we've totally lost control of the CF process. Quite aside from the lack of progress on closing CF3, major hackers who should know better are submitting significant new feature patches now, despite our agreement in Ottawa that nothing big would be accepted after CF3. At this point I'd bet against releasing 9.3 during 2013. I have been skimming the commitfest application, and unlike some of the previous commitfests a huge number of patches have had review at some point in time, but probably need more...so looking for the red Nobody in the 'reviewers' column probably understates the shortage of review. I'm curious what the qualitative feelings are on patches or clusters thereof and what kind of review would be helpful in clearing the field. -- fdr -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] system administration functions with hardcoded superuser checks
2013/1/15 Peter Eisentraut pete...@gmx.net: On 12/18/12 12:09 PM, Peter Eisentraut wrote: There are some system administration functions that have hardcoded superuser checks, specifically: pg_reload_conf pg_rotate_logfile pg_read_file pg_read_file_all pg_read_binary_file pg_read_binary_file_all pg_stat_file pg_ls_dir Some of these are useful in monitoring or maintenance tools, and the hardcoded superuser checks require that these tools run with maximum privileges. Couldn't we just install these functions without default privileges and allow users to grant privileges as necessary? This is still being debated, but just for the heck of it, here is a patch for how this implementation would look like. Even though it gives flexibility of system configuration, it seems to me too less-grained access control because above function accept arbitrary path name, thus, privilege to execution of these function also allows to reference arbitrary access. Here are two type of access controls. One is subject-verb-object type; that describes subject's permitted actions on a particular object. GRANT/REVOKE command specifies which objects are scope of this privilege. SELinux is also SVO type. On the other hand, second is subject-verb type. Superuser privilege is applied independent from the object. I never heard a root user who cannot perform as superuser on /etc directory, for example. I think, it is a reasonable design that above functions right now requires superuser privilege because it can take arbitrary pathname. My preference is, above functions (and others that takes pathname arguments) check SVO style permissions, instead of hardcoded superuser privilege. For example, is it a senseless idea to have a mapping table between database user and operating system user, then call access(2) to check whether mapped os user have privilege to access this file? Thanks, -- KaiGai Kohei kai...@kaigai.gr.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: Review of pg_basebackup and pg_receivexlog to use non-blocking socket communication, was: Re: [HACKERS] Re: [BUGS] BUG #7534: walreceiver takes long time to detect n/w breakdown
On 07.01.2013 16:23, Boszormenyi Zoltan wrote: Since my other patch against pg_basebackup is now committed, this patch doesn't apply cleanly, patch rejects 2 hunks. The fixed up patch is attached. Now that I look at this a high-level perspective, why are we only worried about timeouts in the Copy-mode and when connecting? The initial checkpoint could take a long time too, and if the server turns into a black hole while the checkpoint is running, pg_basebackup will still hang. Then again, a short timeout on that phase would be a bad idea, because the checkpoint can indeed take a long time. In streaming replication, the keep-alive messages carry additional information, the timestamps and WAL locations, so a keepalive makes sense at that level. But otherwise, aren't we just trying to reimplement TCP keepalives? TCP keepalives are not perfect, but if we want to have an application level timeout, it should be implemented in the FE/BE protocol. I don't think we need to do anything specific to pg_basebackup. The user can simply specify TCP keepalive settings in the connection string, like with any libpq program. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 12:03 AM, Bruce Momjian br...@momjian.us wrote: On Tue, Jan 15, 2013 at 11:01:04PM +, Simon Riggs wrote: On 15 January 2013 22:55, Bruce Momjian br...@momjian.us wrote: Why is this being discussed now? It is for 9.4 and will take months. I didn't think there was a better time. We don't usually discuss features during beta testing. Bruce, there are many, many patches on the queue. How will we ever get to beta testing if we begin open ended discussions on next release? If we can't finish what we've started for 9.3, why talk about 9.4? Yes, its a great topic for discussion, but there are better times. Like when? I don't remember a policy of not discussing things now. Does anyone else remember this? Are you saying feature discussion is only between commit-fests? Is this written down anywhere? I only remember beta-time as a time not to discuss features. We kind of do - when in a CF we should do reviewing of existing patches, when outside a CF we should do discussions and work on new features. It's on http://wiki.postgresql.org/wiki/CommitFest. It doesn't specifically say do this and don't do htat, but it says focus on review and discussing things that will happen that far ahead is definitely not focusing on review. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] CF Progress or the lack thereof
On 2013-01-16 02:07:29 -0500, Tom Lane wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: There are still 34 items needing attention in CF3. I suggest that, if you have some spare time, your help would be very much appreciated there. The commitfest that started on Jan 15th has 65 extra items. Anything currently listed in CF3 can rightfully be considered to be part of CF4, too. In case you hadn't noticed, we've totally lost control of the CF process. Quite aside from the lack of progress on closing CF3, major hackers who should know better are submitting significant new feature patches now, despite our agreement in Ottawa that nothing big would be accepted after CF3. At this point I'd bet against releasing 9.3 during 2013. To be honest I don't think new patches coming in late are the actual problem though. If you look at the CF progress in December and January there have been rather few non-trivial commits of patches that haven't been authored by committers: - auto updatable views (Dean via Tom) - autovacuum truncate lock (Jan via Kevin) - minimal recovery.conf (Zoltan via Magnus) I might have missed one or two, but thats about it. There's quite some smaller patches going through but thats all coming directly from on-list conversations. I benefit from that, no question, but its still frustrating for others. At this point it feels like reviewing doesn't really help all that much. There's a noticeable amount of patches in Ready for Committer state and while one or two might not have that status rightfully thats made up by others that are ready but not marked as such. While I think I put a fair amount of time into reviewing I am happy to increase that, but if it just means patches are lingering around anyway, why bother? Now that sounds a bit like I am blaming committers. I am not. Its their own time and they all got other stuff they need (but not necessarily like) to to do and thats absolutely fair. But we still need a way to at least partially solve this issue. Regards, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 9:21 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Not sure. One start might be to actually start having commitfest managers. I haven't actually checked numbers, but my feeling is that last round most commitfests (maybe not all) had a commitfest manager that worked (usually hard) at keeping things moving, whereas this round most (or all?) commitfests have been unmanaged. I'm not sure if that's the reason for it, but it's a feeling I have.. I know various people (myself included) have been trying to keep CF3 moving, e.g. sending followup mail, adjusting patch status, etc. I want to help, but I don't know what's wrong. What are the committers working on, and what is the status of the Ready for commiter patches? Is the problem that the patches marked Ready aren't, in fact, ready? Or is it lack of feedback from authors? Or something else? I'm not one of the committers that pick up the most patches, but from reading messages on the lists I think fairly often patches that are marked as ready, aren't. Sometimes they require a small change, which is fine, but more often than not once it hits a committer it ends up with a lot of feedback requiring rather extensive changes. As in it technical works, but it's better to do it in a different way. I'm not sure how to catch those better. Would it help at all to move all pending items (i.e. anything less than ready) from CF3 to CF4, just so that the committers have only one list to look at, while reviewers can work on the other? Only psychological, but maybe that's better than the current situation? No, it would. They should've been bounced to the next commitfest when the last one closed. The problem was we never closed it... Tom also wrote: In case you hadn't noticed, we've totally lost control of the CF process. Quite aside from the lack of progress on closing CF3, major hackers who should know better are submitting significant new feature patches now, despite our agreement in Ottawa that nothing big would be accepted after CF3. At this point I'd bet against releasing 9.3 during 2013. Well, if we said that, why don't we just single out those patches, and bounce them right now. Before people put more work into it. We also talked about the one-patch-one-review. Did someone ever check if that worked out - did we get that spread, or did we end up with the same ratio as last time? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 16 January 2013 08:21, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Totally lost control is an overstatement. The current situation is that there are clearly more patches than people working on them; a situation that we've been in, with various degrees for all of the last 9 years of my involvement with PostgreSQL. AFAIK there was no CF manager assigned to Nov2012 CF, so not surprising it is off track. Not sure we ever agreed a process for assigning CF mgrs, so again, not surprising it is off track. I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Curious buildfarm failures (fwd)
Hi, On 2013-01-16 01:28:09 -0500, Tom Lane wrote: It's a compiler bug. Gah. Not again. Not that I am surprised, but still. icc 11.1 apparently thinks that this loop in doPickSplit: (Why does it think it needs to prefetch an array it's only going to write into? Is IA64's cache hardware really that stupid?) I think it is. They had that strange model of putting all the intelligence into the compiler and make the hardware relatively dumb. Worked well I'd say. And it makes use of IA64's bizarre scheme for software-unrolling loops, which I am going to do my darnedest to forget now that I've learned it; ... Diagnosis: icc 11.1 is not ready for prime time. Consider me impressed. I tried to see what went wrong from a code generation POV with the original bug but I have to admit I gave up with only an inkling of an idea what it could be (I think it miscalculate the starting offset when copying the whole 'result' memory at once). That instruction set is just plain too crazy for my brain. The consequence seems to be icc on IA-64 in general is not ready for prime time... I doubt they are still investigating significant resources into it anyway. I shall now retire with a glass of wine and attempt to forget everything I just learned about IA64. What a bizarre architecture ... The explanation/problem alone leaves me in want of something stronger, only its 1pm here... Nice work, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 1:12 PM, Simon Riggs si...@2ndquadrant.com wrote: On 16 January 2013 08:21, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Totally lost control is an overstatement. The current situation is that there are clearly more patches than people working on them; a situation that we've been in, with various degrees for all of the last 9 years of my involvement with PostgreSQL. AFAIK there was no CF manager assigned to Nov2012 CF, so not surprising it is off track. Not sure we ever agreed a process for assigning CF mgrs, so again, not surprising it is off track. I don't think we have a process for it, and that's perhaps something we need to discuss for the future. I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. If he's willing to do that, then +1 from me. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] I broke pg_upgrade for GiST
I just realized that my patch that turned XLogRecPtr into a uint64
changed the on-disk format of GiST indexes, because the NSN field in the
page header is an XLogRecPtr. Oops. Fortunately that's easy to fix. I
avoided the same issue with LSNs by continuing to use the old two-field
struct in the on-disk format, we just need to do the same for the NSN
field in GiST. Patch attached.
This got me thinking: although the problem is immediately visible when
you try to run queries on an upgraded gist index - you get incorrect
results - the test procedure explained in TESTING file and performed by
tests.h won't catch this.
We're never going to have full coverage of all possible
incompatibilities, but it would be nice to have at least a simple test
suite that creates an index using each indexam, and checks that they
still work after upgrade.
- Heikki
diff --git a/src/backend/access/gist/gist.c b/src/backend/access/gist/gist.c
index 95d33c8..dcaa6f3 100644
--- a/src/backend/access/gist/gist.c
+++ b/src/backend/access/gist/gist.c
@@ -240,7 +240,7 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
{
/* save old rightlink and NSN */
oldrlink = GistPageGetOpaque(page)-rightlink;
- oldnsn = GistPageGetOpaque(page)-nsn;
+ oldnsn = GistPageGetNSN(page);
dist-buffer = buffer;
dist-block.blkno = BufferGetBlockNumber(buffer);
@@ -364,7 +364,7 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
* F_FOLLOW_RIGHT flags ensure that scans will follow the
* rightlinks until the downlinks are inserted.
*/
- GistPageGetOpaque(ptr-page)-nsn = oldnsn;
+ GistPageSetNSN(ptr-page, oldnsn);
}
START_CRIT_SECTION();
@@ -473,7 +473,7 @@ gistplacetopage(Relation rel, Size freespace, GISTSTATE *giststate,
{
Page leftpg = BufferGetPage(leftchildbuf);
- GistPageGetOpaque(leftpg)-nsn = recptr;
+ GistPageSetNSN(leftpg, recptr);
GistClearFollowRight(leftpg);
PageSetLSN(leftpg, recptr);
@@ -561,7 +561,7 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace, GISTSTATE *giststate)
}
if (stack-blkno != GIST_ROOT_BLKNO
- stack-parent-lsn GistPageGetOpaque(stack-page)-nsn)
+ stack-parent-lsn GistPageGetNSN(stack-page))
{
/*
* Concurrent split detected. There's no guarantee that the
@@ -707,8 +707,7 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace, GISTSTATE *giststate)
*/
}
else if (GistFollowRight(stack-page) ||
- stack-parent-lsn
- GistPageGetOpaque(stack-page)-nsn)
+ stack-parent-lsn GistPageGetNSN(stack-page))
{
/*
* The page was split while we momentarily unlocked the
@@ -793,7 +792,7 @@ gistFindPath(Relation r, BlockNumber child, OffsetNumber *downlinkoffnum)
if (GistFollowRight(page))
elog(ERROR, concurrent GiST page split was incomplete);
- if (top-parent top-parent-lsn GistPageGetOpaque(page)-nsn
+ if (top-parent top-parent-lsn PageXLogRecPtrGet(GistPageGetOpaque(page)-nsn)
GistPageGetOpaque(page)-rightlink != InvalidBlockNumber /* sanity check */ )
{
/*
diff --git a/src/backend/access/gist/gistget.c b/src/backend/access/gist/gistget.c
index 79996a2..3300fec 100644
--- a/src/backend/access/gist/gistget.c
+++ b/src/backend/access/gist/gistget.c
@@ -263,7 +263,7 @@ gistScanPage(IndexScanDesc scan, GISTSearchItem *pageItem, double *myDistances,
*/
if (!XLogRecPtrIsInvalid(pageItem-data.parentlsn)
(GistFollowRight(page) ||
- pageItem-data.parentlsn opaque-nsn)
+ pageItem-data.parentlsn GistPageGetNSN(page))
opaque-rightlink != InvalidBlockNumber /* sanity check */ )
{
/* There was a page split, follow right link to add pages */
diff --git a/src/backend/access/gist/gistvacuum.c b/src/backend/access/gist/gistvacuum.c
index dcad36e..b5be676 100644
--- a/src/backend/access/gist/gistvacuum.c
+++ b/src/backend/access/gist/gistvacuum.c
@@ -114,7 +114,7 @@ pushStackIfSplited(Page page, GistBDItem *stack)
GISTPageOpaque opaque = GistPageGetOpaque(page);
if (stack-blkno != GIST_ROOT_BLKNO !XLogRecPtrIsInvalid(stack-parentlsn)
- (GistFollowRight(page) || stack-parentlsn opaque-nsn)
+ (GistFollowRight(page) || stack-parentlsn GistPageGetNSN(page))
opaque-rightlink != InvalidBlockNumber /* sanity check */ )
{
/* split page detected, install right link to the stack */
diff --git a/src/backend/access/gist/gistxlog.c b/src/backend/access/gist/gistxlog.c
index e3a213e..c18065a 100644
--- a/src/backend/access/gist/gistxlog.c
+++ b/src/backend/access/gist/gistxlog.c
@@ -66,7 +66,7 @@ gistRedoClearFollowRight(XLogRecPtr lsn, XLogRecord *record, int block_index,
*/
if (lsn = PageGetLSN(page))
{
- GistPageGetOpaque(page)-nsn = lsn;
+ GistPageSetNSN(page, lsn);
GistClearFollowRight(page);
PageSetLSN(page, lsn);
@@ -271,7 +271,7 @@ gistRedoPageSplitRecord(XLogRecPtr lsn, XLogRecord *record)
if (newpage-header-blkno == GIST_ROOT_BLKNO)
{
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 2:07 AM, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: There are still 34 items needing attention in CF3. I suggest that, if you have some spare time, your help would be very much appreciated there. The commitfest that started on Jan 15th has 65 extra items. Anything currently listed in CF3 can rightfully be considered to be part of CF4, too. In case you hadn't noticed, we've totally lost control of the CF process. Quite aside from the lack of progress on closing CF3, major hackers who should know better are submitting significant new feature patches now, despite our agreement in Ottawa that nothing big would be accepted after CF3. At this point I'd bet against releasing 9.3 during 2013. Or we could reject all of those patches. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 6:52 AM, Magnus Hagander mag...@hagander.net wrote: On Wed, Jan 16, 2013 at 12:03 AM, Bruce Momjian br...@momjian.us wrote: On Tue, Jan 15, 2013 at 11:01:04PM +, Simon Riggs wrote: On 15 January 2013 22:55, Bruce Momjian br...@momjian.us wrote: Why is this being discussed now? It is for 9.4 and will take months. I didn't think there was a better time. We don't usually discuss features during beta testing. Bruce, there are many, many patches on the queue. How will we ever get to beta testing if we begin open ended discussions on next release? If we can't finish what we've started for 9.3, why talk about 9.4? Yes, its a great topic for discussion, but there are better times. Like when? I don't remember a policy of not discussing things now. Does anyone else remember this? Are you saying feature discussion is only between commit-fests? Is this written down anywhere? I only remember beta-time as a time not to discuss features. We kind of do - when in a CF we should do reviewing of existing patches, when outside a CF we should do discussions and work on new features. It's on http://wiki.postgresql.org/wiki/CommitFest. It doesn't specifically say do this and don't do htat, but it says focus on review and discussing things that will happen that far ahead is definitely not focusing on review. Bruce is evidently under the impression that he's no longer under any obligation to review or commit other people's patches, or participate in the CommitFest process in any way. I believe that he has not committed a significant patch written by someone else in several years. If the committers on the core team aren't committed to the process, it doesn't stand much chance of working. The fact that I have been completely buried for the last six months is perhaps not helping, either, but even at the very low level of engagement I've been at recently, I've still done more reviews (a few) than patch submissions (none). I view it as everyone's responsibility to maintain a similar balance in their own work. And some people are, but not enough, especially among the committers. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
At 2013-01-16 13:08:27 +0100, mag...@hagander.net wrote: One start might be to actually start having commitfest managers. (I'm skipping over this point, since Craig's nomination as CF manager is being discussed elsewhere in this thread.) As in it technical works, but it's better to do it in a different way. I'm not sure how to catch those better. It may be best to return the patch with feedback in that case (i.e. fail fast) rather than fix the problem. With some advice about the right way to do things, the original author or someone else can do the remaining work before a committer looks at it again. If that works, then everyone involved will have a better idea of where each patch stands, rather than having the opaque backlog we do now. In the best case, more people can get some hands-on experience in what it takes to prepare properly committable patches, and thus improve any reviews they do in future. I can only speak for myself, but I would be happy to help polish up patches like this. I don't know if this will help (and I haven't checked to see if it's been discussed before), but perhaps it's worth trying? Right now there seems to be very little else we non-committers can do to help directly. Aside: I thought about importing all CF patches into separate branches in one git repository, so that one doesn't have to play hunt-the-wumpus to find the latest version of a patch, and it's easier to submit small fixes, easier to reject or fix patches that don't apply, easier to fix whitespace errors once and for all (or even run pg_indent), etc. If any committers think it would be helpful, please let me know. We also talked about the one-patch-one-review. Did someone ever check if that worked out - did we get that spread, or did we end up with the same ratio as last time? Here's a breakdown based purely on the names from the CF page (i.e. I didn't check archives to see who actually posted reviews, and didn't take into account reviews posted without updating the CF page). First, authors: 1 Abhijit Menon-Sen 1 Alexander Korotkov, Oleg Bartunov 1 Alexander Lakhin 1 Amit Kapila, Hari Babu 1 Asif Rehman 1 Atri Sharma 1 cjk 1 Dean Rasheed 1 Etsuro Fujita 1 Greg Smith 1 Heikki Linnakangas 1 Jameison Martin 1 Jan Wieck 1 Jeff Janes 1 Jeremy Evans 1 Joachim Wieland 1 Josh Kupershmidt 1 Kevin Grittner 1 Kyotaro Horiguichi 1 Lars Kanis 1 Marko Tiikkaja, Joel Jacobson 1 Oskari Saarenmaa 1 Phil Sorber 1 Robert Haas 1 Satoshi Nagayasu 1 Vik Reykja 1 Will Leinweber 2 Álvaro Herrera 2 Andres Freund 2 Fujii Masao 2 Kyotaro Horiguchi 2 Peter Eisentraut 3 Amit Kapila 3 Dimitri Fontaine 3 Pavel Stehule 3 Tomas Vondra 3 Zoltán Böszörményi 7 Alexander Korotkov 13 Karl O. Pinc Second, reviewers: 1 Alastair Turner 1 Ali Dar 1 Amit Khandekar 1 Andrew Dunstan 1 Asif Rehman 1 Dimitri Fontaine 1 Etsuro Fujita 1 Hari Babu 1 Heikki Linnakangas 1 Ibrar Ahmed 1 Jeevan Chalke 1 Josh Berkus 1 Josh Kupershmidt 1 KaiGai Kohei 1 Kevin Grittner 1 Kyotaro Horiguchi 1 Magnus Hagander 1 Michael Paquier 1 Muhammad Usama 1 Piyush Newe 1 Simon Riggs 1 Vik Reykja 1 Zoltán Böszörményi 2 Abhijit Menon-Sen 2 Andres Freund 2 Fujii Masao 2 Karl O. Pinc 2 Marko Tiikkaja 2 Noah Misch 2 Shigeru Hanada 2 Tom Lane 3 Peter Geoghegan 3 Tomas Vondra 4 Jeff Davis 4 Pavel Stehule 4 Robert Haas 6 Amit Kapila 9 Peter Eisentraut 10 Nobody I haven't looked too carefully, but it seems to me that the people who reviewed patches reviewed more than the number they submitted, but also many people didn't review anything. -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
* Claudio Freire (klaussfre...@gmail.com) wrote: Well, there's the fault in your logic. It won't be as linear. I really don't see how this has become so difficult to communicate. It doesn't have to be linear. We're currently doing massive amounts of parallel processing by hand using partitioning, tablespaces, and client-side logic to split up the jobs. It's certainly *much* faster than doing it in a single thread. It's also faster with 10 processes going than 5 (we've checked). With 10 going, we've hit the FC fabric limit (and these are spinning disks in the SAN, not SSDs). I'm also sure it'd be much slower if all 10 processes were trying to read data through a single process that's reading from the I/O system. We've got some processes which essentially end up doing that and we don't come anywhere near the total FC fabric bandwidth when just scanning through the system because, at that point, you do hit the limits of how fast the individual drive sets can provide data. To be clear- I'm not suggesting that we would parallelize a SeqScan node and have the nodes above it be single-threaded. As I said upthread- we want to parallelize reading and processing the data coming in. Perhaps at some level that works out to not change how we actually *do* seqscans at all and instead something higher in the plan tree just creates multiple of them on independent threads, but it's still going to end up being parallel I/O in the end. I'm done with this thread for now- as brought up, we need to focus on getting 9.3 out the door. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Parallel query execution
* Tom Lane (t...@sss.pgh.pa.us) wrote: In case you hadn't noticed, we've totally lost control of the CF process. I concur. Quite aside from the lack of progress on closing CF3, major hackers who should know better are submitting significant new feature patches now, despite our agreement in Ottawa that nothing big would be accepted after CF3. For my small part, it wasn't my intent to drop a contentious patch at the end. I had felt it was pretty minor and relatively simple. My arguments regarding the popen patch were simply that it didn't address one of the use-cases that I was hoping to. I'll hold off on working on the compressed transport for now in favor of doing reviews and trying to help get 9.3 wrapped up. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Parallel query execution
* Daniel Farina (dan...@heroku.com) wrote: I have been skimming the commitfest application, and unlike some of the previous commitfests a huge number of patches have had review at some point in time, but probably need more...so looking for the red Nobody in the 'reviewers' column probably understates the shortage of review. I've been frustrated by that myself. I realize we don't want to duplicate work but I'm really starting to think that having the Reviewers column has turned out to actually work against us. I'm curious what the qualitative feelings are on patches or clusters thereof and what kind of review would be helpful in clearing the field. I haven't been thrilled with the patches that I've looked at but they've also been ones that hadn't been reviewed before, so perhaps that's what should be expected. It'd be neat if we had some idea of what committers were actively working on and keep off of *those*, but keep working on the ones which aren't being worked by a committer currently. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
2013-01-16 14:18 keltezéssel, Abhijit Menon-Sen írta: At 2013-01-16 13:08:27 +0100, mag...@hagander.net wrote: One start might be to actually start having commitfest managers. (I'm skipping over this point, since Craig's nomination as CF manager is being discussed elsewhere in this thread.) As in it technical works, but it's better to do it in a different way. I'm not sure how to catch those better. It may be best to return the patch with feedback in that case (i.e. fail fast) rather than fix the problem. With some advice about the right way to do things, the original author or someone else can do the remaining work before a committer looks at it again. If that works, then everyone involved will have a better idea of where each patch stands, rather than having the opaque backlog we do now. In the best case, more people can get some hands-on experience in what it takes to prepare properly committable patches, and thus improve any reviews they do in future. I can only speak for myself, but I would be happy to help polish up patches like this. I don't know if this will help (and I haven't checked to see if it's been discussed before), but perhaps it's worth trying? Right now there seems to be very little else we non-committers can do to help directly. Aside: I thought about importing all CF patches into separate branches in one git repository, so that one doesn't have to play hunt-the-wumpus to find the latest version of a patch, and it's easier to submit small fixes, easier to reject or fix patches that don't apply, easier to fix whitespace errors once and for all (or even run pg_indent), etc. If any committers think it would be helpful, please let me know. We also talked about the one-patch-one-review. Did someone ever check if that worked out - did we get that spread, or did we end up with the same ratio as last time? Here's a breakdown based purely on the names from the CF page (i.e. I didn't check archives to see who actually posted reviews, and didn't take into account reviews posted without updating the CF page). First, authors: 1 Abhijit Menon-Sen 1 Alexander Korotkov, Oleg Bartunov 1 Alexander Lakhin 1 Amit Kapila, Hari Babu 1 Asif Rehman 1 Atri Sharma 1 cjk 1 Dean Rasheed 1 Etsuro Fujita 1 Greg Smith 1 Heikki Linnakangas 1 Jameison Martin 1 Jan Wieck 1 Jeff Janes 1 Jeremy Evans 1 Joachim Wieland 1 Josh Kupershmidt 1 Kevin Grittner 1 Kyotaro Horiguichi 1 Lars Kanis 1 Marko Tiikkaja, Joel Jacobson 1 Oskari Saarenmaa 1 Phil Sorber 1 Robert Haas 1 Satoshi Nagayasu 1 Vik Reykja 1 Will Leinweber 2 Álvaro Herrera 2 Andres Freund 2 Fujii Masao 2 Kyotaro Horiguchi 2 Peter Eisentraut 3 Amit Kapila 3 Dimitri Fontaine 3 Pavel Stehule 3 Tomas Vondra 3 Zoltán Böszörményi It should be 2 submitted patches. The PQconninfo() was a subfeature of the pg_basebackup patch, Magnus added it to the commitfest separately. 7 Alexander Korotkov 13 Karl O. Pinc Second, reviewers: 1 Alastair Turner 1 Ali Dar 1 Amit Khandekar 1 Andrew Dunstan 1 Asif Rehman 1 Dimitri Fontaine 1 Etsuro Fujita 1 Hari Babu 1 Heikki Linnakangas 1 Ibrar Ahmed 1 Jeevan Chalke 1 Josh Berkus 1 Josh Kupershmidt 1 KaiGai Kohei 1 Kevin Grittner 1 Kyotaro Horiguchi 1 Magnus Hagander 1 Michael Paquier 1 Muhammad Usama 1 Piyush Newe 1 Simon Riggs 1 Vik Reykja 1 Zoltán Böszörményi I reviewed 2 patches: https://commitfest.postgresql.org/action/patch_view?id=1005 https://commitfest.postgresql.org/action/patch_view?id=1006 2 Abhijit Menon-Sen 2 Andres Freund 2 Fujii Masao 2 Karl O. Pinc 2 Marko Tiikkaja 2 Noah Misch 2 Shigeru Hanada 2 Tom Lane 3 Peter Geoghegan 3 Tomas Vondra 4 Jeff Davis 4 Pavel Stehule 4 Robert Haas 6 Amit Kapila 9 Peter Eisentraut 10 Nobody I haven't looked too carefully, but it seems to me that the people who reviewed patches reviewed more than the number they submitted, but also many people didn't review anything. -- Abhijit -- -- Zoltán Böszörményi Cybertec Schönig Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt, Austria Web: http://www.postgresql-support.de http://www.postgresql.at/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Event Triggers: adding information
On Wed, Jan 9, 2013 at 11:58 AM, Alvaro Herrera
alvhe...@2ndquadrant.com wrote:
I made some changes to this, and I think the result (attached) is
cleaner overall.
Now, this review is pretty much unfinished as far as I am concerned;
mainly I've been trying to figure out how it all works and improving
some stuff as I go, and I haven't looked at the complete patch yet. We
might very well doing some things quite differently; for example I'm not
really sure of the way we handle CommandTags, or the way we do object
lookups at the event trigger stage, only to have it repeated later when
the actual deletion takes place. In particular, since event_trigger.c
does not know the lock strength that's going to be taken at deletion, it
only grabs ShareUpdateExclusiveLock; so there is lock escalation here
which could lead to deadlocks. This might need to be rethought. I
added a comment somewhere, noting that perhaps it's ProcessUtility's job
to set the object classId (or at least the ObjectType) at the same time
the TargetOid is being set, rather than have event_trigger.c figure it
out from the parse node. And so on. I haven't looked at plpgsql or
pg_dump yet, either.
I think this points to a couple of problems: this patch isn't
well-enough thought out, and it's got several features jammed into a
single patch. This should really be split up into several patches and
each one submitted separately.
I think that ddl_command_trace is an unnecessary frammish that should
be ripped out entirely. It is easy enough to accomplish the same
thing with ddl_command_start and ddl_command_end, and so we're just
increasing the number of cases we have to support in the future for no
real benefit.
We've been discussing on IM the handling of DROP in general. The
current way it works is that the object OID is reported only if the
toplevel command specifies to drop a single object (so no OID if you do
DROP TABLE foo,bar); and this is all done by standard_ProcessUtility.
This seems bogus to me, and it's also problematic for things like DROP
SCHEMA, as well as DROP OWNED BY (which, as noted upthread, is not
handled at all but should of course be). I think the way to do it is
have performDeletion and performMultipleDeletions (dependency.c) call
into the event trigger code, by doing something like this:
1. look up all objects to drop (i.e. resolve the list of objects
specified by the user, and complete with objects dependent on those)
2. iterate over this list, firing DROP at-start event triggers
3. do the actual deletion of objects (i.e. deleteOneObject, I think)
4. iterate again over the list firing DROP at-end event triggers
This gets right back to an argument Dimitri and I have been having
since v1 of this patch, which is whether these are command triggers or
event triggers. I think we ought to support both, but they are not
the same thing. I think the design you are proposing is just fine for
an event called sql_drop, but it makes no sense for an event called
ddl_command_end, which needs to be called once per DDL statement - at
the end. Not once per object dropped.
We would have one or more event triggers being called with a context of
TOPLEVEL (for objects directly mentioned in the command), and then some
more calls with a context of CASCADE. Exactly how should DROP OWNED BY
be handled is not clear; perhaps we should raise one TOPLEVEL event for
the objects directly owned by the role? Maybe we should just do CASCADE
for all objects? This is not clear yet.
TOPLEVEL is supposed to correspond to a complete SQL statement entered
by the user:
typedef enum
{
PROCESS_UTILITY_TOPLEVEL, /* toplevel
interactive command */
PROCESS_UTILITY_QUERY, /* a complete query,
but not toplevel */
PROCESS_UTILITY_SUBCOMMAND, /* a piece of a query */
PROCESS_UTILITY_GENERATED /* internally
generated node query node */
} ProcessUtilityContext;
IMHO, DROP OWNED BY probably shouldn't fire ddl_command_start/end.
I excluded it from ddl_command_start in the initial commit because
it's such a weird case, and it didn't seem to cleanly fit in the
framework. Now, that could be changed, but surely that should be a
separate patch if we're going to do it rather than lumped in with a
bunch of other dubious changes, and more than that, it just
underscores, again, the fact that we're pouring a lot of effort into
fleshing out ddl_command_start/end instead of doing what we probably
should be doing, which is adding events that are actually targeting
what we want to do, like sql_drop.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
Simon Riggs escribió: On 16 January 2013 08:21, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Totally lost control is an overstatement. The current situation is that there are clearly more patches than people working on them; a situation that we've been in, with various degrees for all of the last 9 years of my involvement with PostgreSQL. AFAIK there was no CF manager assigned to Nov2012 CF, so not surprising it is off track. Not sure we ever agreed a process for assigning CF mgrs, so again, not surprising it is off track. FWIW, the September 2012 commitfest also went without a manager *for a month*. I started doing the job at that point because it was pretty clear to me that there was a serious problem. And when I finally closed it (just before the Nov. 2012 one was about to start), it was my intention to manage the next one as well; however somebody else told me they wanted that job, so I stepped aside. That was, probably, a bad decision on my part, and I blame myself for not keeping a closer eye on things that I intuitively knew were going wrong. I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. Seconded. I particularly like the fact that Craig is not already a PG developer, so he's not going to be working on his own patches. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. Seconded. I particularly like the fact that Craig is not already a PG developer, so he's not going to be working on his own patches. So when can he start? :D Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] pg_dump transaction's read-only mode
On Wed, Jan 16, 2013 at 1:21 AM, Pavan Deolasee pavan.deola...@gmail.comwrote: On Wed, Jan 9, 2013 at 6:42 AM, Gurjeet Singh singh.gurj...@gmail.comwrote: I have updated the commitfest submission to link to the correct patch email. Thanks Gurjeet. I initially thought that this patch deserves accompanying documentation because pg_dump's serializable transaction may error out because of a conflict. But the following line in the docs [1] confirms otherwise: read-only transactions will never have serialization conflicts So no doc patch necessary :) Can you please mark the patch as Ready for committer if you think that way ? Done. -- Gurjeet Singh http://gurjeet.singh.im/
Re: [HACKERS] Parallel query execution
On 01/15/2013 11:32 PM, Bruce Momjian wrote: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck in parallelism would be: 1. Parallel sort 2. Parallel aggregation (for commutative aggregates) 3. Parallel nested loop join (especially for expression joins, like GIS) parallel data load? :/ We have that in pg_restore, and I thinnk we are getting parallel dump in 9.3, right? Unfortunately, I don't see it in the last 9.3 commit-fest. Is it still being worked on? I am about half way through reviewing it. Unfortunately paid work take precedence over unpaid work. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
At 2013-01-16 09:02:45 -0500, sfr...@snowman.net wrote: So when can he start? :D Also, what should he start with? CF3 as it stands today, or CF4 with all of the pending patches moved from CF3, immense though the result may be? I slightly prefer the latter, so that we're all on the same page when it comes to seeing what needs to be done. (If others agree, I can do some patch-moving, unless some special privileges are needed to do it.) -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/16/2013 10:02 PM, Stephen Frost wrote: * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. Seconded. I particularly like the fact that Craig is not already a PG developer, so he's not going to be working on his own patches. So when can he start? :D Hey, I've had a few toy patches accepted... but yeah, I'm more interested in build systems, QA, docs, etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On 01/16/2013 08:12 PM, Simon Riggs wrote: I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. I'm happy to step up and help out. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pkg-config files for libpq and ecpg
On Tue, Jan 15, 2013 at 02:16:01PM -0500, Peter Eisentraut wrote: A side issue that arose: libecpg_compat is linked with libpq, but doesn't seem to use it. This was added many years ago in cd75f94dafd43358305811b7576ad75d889097e3, but it doesn't appear to be required anymore. Needs some checking. libecpg_compat doesn't directly need libpq IIRC, but it does indirectly There are a couple functions from libecpg that are used in libecpg_compat and libecpg itself does need libpq. Michael -- Michael Meskes Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org) Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org Jabber: michael.meskes at gmail dot com VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
* Abhijit Menon-Sen (a...@2ndquadrant.com) wrote: Also, what should he start with? CF3 as it stands today, or CF4 with all of the pending patches moved from CF3, immense though the result may be? I slightly prefer the latter, so that we're all on the same page when it comes to seeing what needs to be done. I'd leave it up to him to decide, but I think the general thought was to move it all to one place. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Curious buildfarm failures (fwd)
Hi, On Wed, 16 Jan 2013, Andres Freund wrote: On 2013-01-16 01:28:09 -0500, Tom Lane wrote: It's a compiler bug. Thanks for investigating. But I'm not sure there is any way other way for me other than switching to gcc, because intel stopped providing their IA64 version of compilers free of charge even for non-commercial/educational people: http://software.intel.com/en-us/intel-software-development-tools-for-intel-itanium-processors (their website is also a bit of maze, so I don't see anywhere the updated versions of the 10.0, 10.1, 11.1 compilers that I have) So unless somebody suggest otherwise, i'm going to switch to gcc on this buildfarm. Cheers, Sergey * Sergey E. Koposov, PhD, Research Associate Institute of Astronomy, University of Cambridge Madingley road, CB3 0HA, Cambridge, UK Tel: +44-1223-337-551 Web: http://www.ast.cam.ac.uk/~koposov/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Curious buildfarm failures (fwd)
On 2013-01-16 14:41:47 +, Sergey Koposov wrote: Hi, On Wed, 16 Jan 2013, Andres Freund wrote: On 2013-01-16 01:28:09 -0500, Tom Lane wrote: It's a compiler bug. Thanks for investigating. But I'm not sure there is any way other way for me other than switching to gcc, because intel stopped providing their IA64 version of compilers free of charge even for non-commercial/educational people: http://software.intel.com/en-us/intel-software-development-tools-for-intel-itanium-processors (their website is also a bit of maze, so I don't see anywhere the updated versions of the 10.0, 10.1, 11.1 compilers that I have) So unless somebody suggest otherwise, i'm going to switch to gcc on this buildfarm. What about switching to -O1 of 11.1? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Curious buildfarm failures (fwd)
On 01/16/2013 09:41 AM, Sergey Koposov wrote: So unless somebody suggest otherwise, i'm going to switch to gcc on this buildfarm. If you switch compiler it should be a new buildfarm animal. (That just means re-registering so you get a new name/secret pair.) We have provision for upgrading the OS version and the compiler version, but changing which OS or which compiler is used requires a new animal - it's too great a discontinuity. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Curious buildfarm failures (fwd)
On Wed, 16 Jan 2013, Andres Freund wrote: So unless somebody suggest otherwise, i'm going to switch to gcc on this buildfarm. What about switching to -O1 of 11.1? I don't know. It is up to -hackers to decide. I think that icc on ia64 have shown bugginess time after time. But if you think that buildfarm with icc 11.1 -O1 carry more information than say running gcc, i can still run icc. S * Sergey E. Koposov, PhD, Research Associate Institute of Astronomy, University of Cambridge Madingley road, CB3 0HA, Cambridge, UK Tel: +44-1223-337-551 Web: http://www.ast.cam.ac.uk/~koposov/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] log_lock_waits to identify transaction's relation
On 16 January 2013 03:47, Stephen Frost sfr...@snowman.net wrote: Simon, * Simon Riggs (si...@2ndquadrant.com) wrote: Attached patch passes through further information about the lock wait, so we can display the following message instead LOG: process %d acquired %s on transaction %u on relation %u of database %u after %ld.%03d ms I love this idea. Please take these comments as an initial/quick review because I'd really like to see this get in. It's there as a way to prove FKlocks works, or not, and to help that get committed. It's possible we reject this in this CF, since I see it as low priority anyway. A couple quick notes regarding the patch- what does GetXactLockTableRelid really provide..? The ability to access a static variable in a different module. It doesn't provide anything other than that, This patch does use it outside of lmgr.c, where XactLockTableRelid is defined. Second, do we really need to describeXact bool to DescribeLockTag..? Strikes me as unnecessary- if the information is available, include it. The information isn't available, which is why I didn't include it. Comments explain that the additional information is only available within the backend that was waiting. That's sufficient for stats, but not enough for an extended multi-backend deadlock report. Lastly, I really don't like the changes made to XactLockTableWait() and friends. They had a very clearly defined and simple goal previously and the additional parameter seems to muddy things a bit, particularly without any comments about what it's all about or why it's there. I understand that you're trying to pass the necessary information down to where the log is generated, but it doesn't feel quite right. There is an API change to XactLockTableWait() to pass the relid. That part is essential to any solution. There are 2 possible designs for this... (1) Simple - we remember the relid we are waiting on in a backend-only variable so it can be used later when generating stats. (2) Complex - record the additional information within the unused fields of the locktag of a TRANSACTION lock, so they will be stored and accessible by all, within the shared lock table itself. This would require changing the hash function for transaction lock tags so that only the key and not the additional payload items get hashed. That seemed a much more contentious change. Also, what about VirtualXactLockTableWait()..? Should that have a similar treatment? That isn't needed, IMHO. We wait on Xids whenever we see them on heap tuples. Vxids aren't recorded anywhere, so they don't offer the same blockage, which is basically for visibility checks on index creation. That is more obvious. One might argue its needed as well, but I would see it as a separate patch. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [PATCH] Compile without warning with gcc's -Wtype-limits, -Wempty-body
On 1/15/13 6:36 AM, Andres Freund wrote: I just think that the price of fixing a single Assert() that hasn't changed in years where the variable isn't likely to ever get signed is acceptable. Well, once you get past that one change you proposed, you will also find pg_standby.c: In function 'SetWALFileNameForCleanup': pg_standby.c:348:3: error: comparison of unsigned expression = 0 is always true [-Werror=type-limits] (which, curiously, is the only one that clang complains about). I don't like removing safety checks from code when there is no other mechanism that could make up for it somehow. I think the best practice at the moment, as with most gcc -Wextra warnings, is to manually check them once in a while. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 10:33 AM, Stephen Frost sfr...@snowman.net wrote: * Claudio Freire (klaussfre...@gmail.com) wrote: Well, there's the fault in your logic. It won't be as linear. I really don't see how this has become so difficult to communicate. It doesn't have to be linear. We're currently doing massive amounts of parallel processing by hand using partitioning, tablespaces, and client-side logic to split up the jobs. It's certainly *much* faster than doing it in a single thread. It's also faster with 10 processes going than 5 (we've checked). With 10 going, we've hit the FC fabric limit (and these are spinning disks in the SAN, not SSDs). I'm also sure it'd be much slower if all 10 processes were trying to read data through a single process that's reading from the I/O system. We've got some processes which essentially end up doing that and we don't come anywhere near the total FC fabric bandwidth when just scanning through the system because, at that point, you do hit the limits of how fast the individual drive sets can provide data. Well... just closing then (to let people focus on 9.3's CF), that's a level of hardware I haven't had experience with, but seems to behave much different than regular (big and small) RAID arrays. In any case, perhaps tablespaces are a hint here: if nodes are working on different tablespaces, there's an indication that they *can* be parallelized efficiently. That could be fleshed out on a parallel execution node, but for that to work the whole execution engine needs to be thread-safe (or it has to fork). It won't be easy. It's best to concentrate on lower-hanging fruits, like sorting and aggregates. Now back to the CF. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [sepgsql 1/3] add name qualified creation label
On Tue, Jan 15, 2013 at 3:02 PM, Kohei KaiGai kai...@kaigai.gr.jp wrote: This patch adds sepgsql the feature of name qualified creation label. Background, on creation of a certain database object, sepgsql assigns a default security label according to the security policy that has a set of rules to determine a label of new object. Usually, a new object inherits its parent (e.g table is a parent of column) object's label, unless it has a particular type_transition rule in the policy. Type_transition rule allows to describe a particular security label as default label of new object towards a pair of client and parent object. For example, the below rule says columns constructed under the table labeled as sepgsql_table_t by client with staff_t will have staff_column_t, instead of table's label. TYPE_TRANSITION staff_t sepgsql_table_t:db_column staff_column_t; Recently, this rule was enhanced to take 5th argument for object name; that enables to special case handling exceptionally. It was originally designed to describe default security labels for files in /etc directory, because many application put its own configuration files here, thus, traditional type_transition rule was poor to describe all the needed defaults. On the other hand, we can port this concept of database system also. One example is temporary objects being constructed under the pg_temp schema. If we could assign a special default label on this, it allows unprivileged users (who cannot create persistent tables) to create temporary tables that has no risk of information leak to other users. Otherwise, we may be able to assign a special security label on system columns and so on. From the perspective of implementation on sepgsql side, all we need to do is replace old security_compute_create_raw() interface by new security_compute_create_name_raw(). If here is no name qualified type_transition rules, it performs as if existing API, so here is no backword compatible issue. This patch can be applied on the latest master branch. This looks OK on a quick once-over, but should it update the documentation somehow? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] json api WIP patch
On Mon, Jan 14, 2013 at 11:02 PM, Andrew Dunstan and...@dunslane.net wrote: After a couple of iterations, some performance enhancements to the json parser and lexer have ended up with a net performance improvement over git tip. On our test rig, the json parse test runs at just over 13s per 1 parses on git tip and approx 12.55s per 1 parses with the attached patch. Truth be told, I think the lexer changes have more than paid for the small cost of the switch to an RD parser. But since the result is a net performance win PLUS some enhanced functionality, I think we should be all good. Yeah, that sounds great. Thanks for putting in the effort. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Curious buildfarm failures (fwd)
Sergey Koposov kopo...@ast.cam.ac.uk writes: On Wed, 16 Jan 2013, Andres Freund wrote: What about switching to -O1 of 11.1? I don't know. It is up to -hackers to decide. I think that icc on ia64 have shown bugginess time after time. But if you think that buildfarm with icc 11.1 -O1 carry more information than say running gcc, i can still run icc. I think the reason that this bug doesn't manifest at -O1 is that then icc doesn't attempt to do any loop unrolling/vectorizing. So that's a big chunk of potential optimization bugs we'd be dodging. It's hard to say whether that renders the test worthless in comparison with what people would try to do in production. Should we recommend that people not try to use -O2 or higher with icc on IA64? IMO it's important that we have some icc members in the buildfarm, just because it's useful to see a different compiler's take on warnings. We do have some icc-on-mainstream-Intel members, but not many. Perhaps Sergey should use 10.1, which so far appears to not have so many bugs. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 01:08:27PM +0100, Magnus Hagander wrote: On Wed, Jan 16, 2013 at 9:21 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2013-01-16 02:07:29 -0500, t...@sss.pgh.pa.us wrote: In case you hadn't noticed, we've totally lost control of the CF process. What can we do to get it back on track? Not sure. One start might be to actually start having commitfest managers. I haven't actually checked numbers, but my feeling is that last round most commitfests (maybe not all) had a commitfest manager that worked (usually hard) at keeping things moving, whereas this round most (or all?) commitfests have been unmanaged. I'm not sure if that's the reason for it, but it's a feeling I have.. Agreed. I want to help, but I don't know what's wrong. What are the committers working on, and what is the status of the Ready for commiter patches? Is the problem that the patches marked Ready aren't, in fact, ready? Or is it lack of feedback from authors? Or something else? I'm not one of the committers that pick up the most patches, but from reading messages on the lists I think fairly often patches that are marked as ready, aren't. Sometimes they require a small change, which is fine, but more often than not once it hits a committer it ends up with a lot of feedback requiring rather extensive changes. That's an intrinsic hazard of non-committer reviews. All the people who are truly skilled at distinguishing ready patches from not-ready patches are already committers, but a non-committer review catching even one important problem adds value. I'd rather have more people getting involved in the process with high-effort reviews, however imperfect. As in it technical works, but it's better to do it in a different way. I'm not sure how to catch those better. That's even harder, because it's subjective. Committers often disagree on such matters. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 08:42:29AM -0500, Stephen Frost wrote: * Daniel Farina (dan...@heroku.com) wrote: I have been skimming the commitfest application, and unlike some of the previous commitfests a huge number of patches have had review at some point in time, but probably need more...so looking for the red Nobody in the 'reviewers' column probably understates the shortage of review. I've been frustrated by that myself. I realize we don't want to duplicate work but I'm really starting to think that having the Reviewers column has turned out to actually work against us. That column tells the CF manager whom to browbeat. Without a CF manager, a stale entry can indeed make a patch look under-control when it isn't. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Teaching pg_receivexlog to follow timeline switches
On 15.01.2013 20:22, Fujii Masao wrote: On Tue, Jan 15, 2013 at 11:05 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Now that a standby server can follow timeline switches through streaming replication, we should do teach pg_receivexlog to do the same. Patch attached. I made one change to the way START_STREAMING command works, to better support this. When a standby server reaches the timeline it's streaming from the master, it stops streaming, fetches any missing timeline history files, and parses the history file of the latest timeline to figure out where to continue. However, I don't want to parse timeline history files in pg_receivexlog. Better to keep it simple. So instead, I modified the server-side code for START_STREAMING to return the next timeline's ID at the end, and used that in pg_receivexlog. I also modifed BASE_BACKUP to return not only the start XLogRecPtr, but also the corresponding timeline ID. Otherwise we might try to start streaming from wrong timeline if you issue a BASE_BACKUP at the same moment the server switches to a new timeline. When pg_receivexlog switches timeline, what to do with the partial file on the old timeline? When the timeline changes in the middle of a WAL segment, the segment old the old timeline is only half-filled. For example, when timeline changes from 1 to 2, you'll have this in pg_xlog: 00010006 00010007 00010008 00020008 0002.history The segment 00010008 is only half-filled, as the timeline changed in the middle of that segment. The beginning portion of that file is duplicated in 00020008, with the timeline-changing checkpoint record right after the duplicated portion. When we stream that with pg_receivexlog, and hit the timeline switch, we'll have this situation in the client: 00010006 00010007 00010008.partial What to do with the partial file? One option is to rename it to 00010008. However, if you then kill pg_receivexlog before it has finished streaming a full segment from the new timeline, on restart it will try to begin streaming WAL segment 00010009, because it sees that segment 00010008 is already completed. That'd be wrong. Can't we rename .partial file safely after we receive a full segment of the WAL file with new timeline and the same logid/segmentid? I'd prefer to leave the .partial suffix in place, as the segment really isn't complete. It doesn't make a difference when you recover to the latest timeline, but if you have a more complicated scenario with multiple timelines that are still alive, ie. there's a server still actively generating WAL on that timeline, you'll easily get confused. As an example, imagine that you have a master server, and one standby. You maintain a WAL archive for backup purposes with pg_receivexlog, connected to the standby. Now, for some reason, you get a split-brain situation and the standby server is promoted with new timeline 2, while the real master is still running. The DBA notices the problem, and kills the standby and pg_receivexlog. He deletes the XLOG files belonging to timeline 2 in pg_receivexlog's target directory, and re-points pg_recevexlog to the master while he re-builds the standby server from backup. At that point, pg_receivexlog will start streaming from the end of the zero-padded segment, not knowing that it was partial, and you have a hole in the archived WAL stream. Oops. The DBA could avoid that by also removing the last WAL segment on timeline 1, the one that was partial. But it's really not obvious that there's anything wrong with that segment. Keeping the .partial suffix makes it clear. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] log_lock_waits to identify transaction's relation
* Simon Riggs (si...@2ndquadrant.com) wrote: A couple quick notes regarding the patch- what does GetXactLockTableRelid really provide..? The ability to access a static variable in a different module. It doesn't provide anything other than that, It isn't actually necessary for that currently though, is it? All calls to it appeared to be inside the same module. Do you anticipate that needing to change in the future? The information isn't available, which is why I didn't include it. Comments explain that the additional information is only available within the backend that was waiting. That's sufficient for stats, but not enough for an extended multi-backend deadlock report. Right, sorry, to be clear, it seemed that we could simply detect if the information is available and print it out if it is- and not print it if it isn't. The additional boolean variable looked to be unnecessary, or is there a point where OidIsValid(GetXactLockTableRelid()) will be true but the additional information shouldn't be printed or isn't actually available? There is an API change to XactLockTableWait() to pass the relid. That part is essential to any solution. I'm wondering if we could pass in something both more generic and with more information- perhaps a simple string which is constructed when we have more information and is then included in the log message which is generated? Perhaps something which includes both the relname and the OID? Could that be used across more than just these kinds of locks? Of course, on the flip side, I understand that we don't want to spend a lot of time building strings and whatnot during all of these code paths; would be nice if we could defer that until we're about to log. Regardless, I do think it'd be good to update the comments to indicate that the relid is being passed in solely for the purpose of being available to be included in the log, if necessary. (2) Complex - record the additional information within the unused fields of the locktag of a TRANSACTION lock, so they will be stored and accessible by all, within the shared lock table itself. This would require changing the hash function for transaction lock tags so that only the key and not the additional payload items get hashed. That seemed a much more contentious change. As the lock table is itself contended for greatly, I agree that we probably don't want to go changing it. It would be nice if we had a way to associate additional meta-data with why a lock is being attempted though and for logging it. That isn't needed, IMHO. We wait on Xids whenever we see them on heap tuples. Vxids aren't recorded anywhere, so they don't offer the same blockage, which is basically for visibility checks on index creation. That is more obvious. One might argue its needed as well, but I would see it as a separate patch. Alright. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] pg_trgm partial-match
On Fri, Nov 23, 2012 at 2:11 AM, Fujii Masao masao.fu...@gmail.com wrote: On Mon, Nov 19, 2012 at 10:56 AM, Tomas Vondra t...@fuzzy.cz wrote: I've done a quick review of the current patch: Thanks for the commit! As Alexander pointed out upthread, another infrastructure patch is required before applying this patch. So I will implement the infra patch first. I marked this patch as Returned with Feedback because unfortunately I don't have enough time to revise the patch. I will retry this maybe in 9.4dev phase. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Materialized views WIP patch
Kevin Grittner kgri...@mail.com writes: I've been struggling with two areas: - pg_dump sorting for MVs which depend on other MVs Surely that should fall out automatically given that the dependency is properly expressed in pg_depend? If you mean you're trying to get it to cope with circular dependencies between MVs, it might take some work on the pg_dump side, but plain ordering shouldn't require new code. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Materialized views WIP patch
Tom Lane wrote: Kevin Grittner kgri...@mail.com writes: I've been struggling with two areas: - pg_dump sorting for MVs which depend on other MVs Surely that should fall out automatically given that the dependency is properly expressed in pg_depend? If you mean you're trying to get it to cope with circular dependencies between MVs, it might take some work on the pg_dump side, but plain ordering shouldn't require new code. The *definitions* sort properly, but what I'm trying to do is define them WITH NO DATA and load data after all the COPY statements for tables. If mva is referenced by mvb, the goal is the REFRESH mva, build its indexes before running REFRESH for mvb and building its indexes. To do things in any other order does't seem to me to leave things after restore in the same state they were in at the time of the dump. So I should have been a little more verbose describing the problem: pg_dump sorting of REFRESH and CREATE INDEX steps for MVs which depend on other MVs. Last night I found why my previous attempts had been failing -- I was trying to build the dependencies at the wrong point in the dump process, after the sorts had already been done. Now that I've spotted that fundamental flaw, I think I can get this out of the way without too much more fanfare. I kept thinking I had something wrong in the detail of my approach, while the problem was at a much higher level. Where I really need someone to hit me upside the head with a clue-stick is the code I added to the bottom of RelationBuildDesc() in relcache.c. The idea is that on first access to an unlogged MV, to detect that the heap has been replaced by the init fork, set relisvalid to false, and make the heap look normal again. I couldn't see any way to do that which wasn't a kludge, and I can't figure out how to deal with relcache properly in implementing that kludge. Either a tip about the right way to work the kludge, or a suggestion for a less kludgy alternative would be welcome. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Teaching pg_receivexlog to follow timeline switches
On Thu, Jan 17, 2013 at 1:08 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: On 15.01.2013 20:22, Fujii Masao wrote: On Tue, Jan 15, 2013 at 11:05 PM, Heikki Linnakangas hlinnakan...@vmware.com wrote: Now that a standby server can follow timeline switches through streaming replication, we should do teach pg_receivexlog to do the same. Patch attached. I made one change to the way START_STREAMING command works, to better support this. When a standby server reaches the timeline it's streaming from the master, it stops streaming, fetches any missing timeline history files, and parses the history file of the latest timeline to figure out where to continue. However, I don't want to parse timeline history files in pg_receivexlog. Better to keep it simple. So instead, I modified the server-side code for START_STREAMING to return the next timeline's ID at the end, and used that in pg_receivexlog. I also modifed BASE_BACKUP to return not only the start XLogRecPtr, but also the corresponding timeline ID. Otherwise we might try to start streaming from wrong timeline if you issue a BASE_BACKUP at the same moment the server switches to a new timeline. When pg_receivexlog switches timeline, what to do with the partial file on the old timeline? When the timeline changes in the middle of a WAL segment, the segment old the old timeline is only half-filled. For example, when timeline changes from 1 to 2, you'll have this in pg_xlog: 00010006 00010007 00010008 00020008 0002.history The segment 00010008 is only half-filled, as the timeline changed in the middle of that segment. The beginning portion of that file is duplicated in 00020008, with the timeline-changing checkpoint record right after the duplicated portion. When we stream that with pg_receivexlog, and hit the timeline switch, we'll have this situation in the client: 00010006 00010007 00010008.partial What to do with the partial file? One option is to rename it to 00010008. However, if you then kill pg_receivexlog before it has finished streaming a full segment from the new timeline, on restart it will try to begin streaming WAL segment 00010009, because it sees that segment 00010008 is already completed. That'd be wrong. Can't we rename .partial file safely after we receive a full segment of the WAL file with new timeline and the same logid/segmentid? I'd prefer to leave the .partial suffix in place, as the segment really isn't complete. It doesn't make a difference when you recover to the latest timeline, but if you have a more complicated scenario with multiple timelines that are still alive, ie. there's a server still actively generating WAL on that timeline, you'll easily get confused. As an example, imagine that you have a master server, and one standby. You maintain a WAL archive for backup purposes with pg_receivexlog, connected to the standby. Now, for some reason, you get a split-brain situation and the standby server is promoted with new timeline 2, while the real master is still running. The DBA notices the problem, and kills the standby and pg_receivexlog. He deletes the XLOG files belonging to timeline 2 in pg_receivexlog's target directory, and re-points pg_recevexlog to the master while he re-builds the standby server from backup. At that point, pg_receivexlog will start streaming from the end of the zero-padded segment, not knowing that it was partial, and you have a hole in the archived WAL stream. Oops. The DBA could avoid that by also removing the last WAL segment on timeline 1, the one that was partial. But it's really not obvious that there's anything wrong with that segment. Keeping the .partial suffix makes it clear. Thanks for elaborating the reason why .partial suffix should be kept. I agree that keeping the .partial suffix would be safer. Regards, -- Fujii Masao -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Materialized views WIP patch
On 16 January 2013 05:40, Kevin Grittner kgri...@mail.com wrote: Here is a new version of the patch, with most issues discussed in previous posts fixed. I've been struggling with two areas: - pg_dump sorting for MVs which depend on other MVs - proper handling of the relisvalid flag for unlogged MVs after recovery Some weirdness: postgres=# CREATE VIEW v_test2 AS SELECT 1 moo; CREATE VIEW postgres=# CREATE MATERIALIZED VIEW mv_test2 AS SELECT moo, 2*moo FROM v_test2 UNION ALL SELECT moo, 3*moo FROM v_test2; SELECT 2 postgres=# \d+ mv_test2 Materialized view public.mv_test2 Column | Type | Modifiers | Storage | Stats target | Description --+-+---+-+--+- moo | integer | | plain | | ?column? | integer | | plain | | View definition: SELECT *SELECT* 1.moo, *SELECT* 1.?column?; Has OIDs: no The weirdness I refer you to is the view definition. This does not occur with a straightforward UNION. This does not occur with a regular view: postgres=# CREATE VIEW v_test3 AS SELECT moo, 2*moo FROM v_test2 UNION ALL SELECT moo, 3*moo FROM v_test2; CREATE VIEW postgres=# \d+ v_test3 View public.v_test3 Column | Type | Modifiers | Storage | Description --+-+---+-+- moo | integer | | plain | ?column? | integer | | plain | View definition: SELECT v_test2.moo, 2 * v_test2.moo FROM v_test2 UNION ALL SELECT v_test2.moo, 3 * v_test2.moo FROM v_test2; -- Thom
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 01:37:28PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:32 PM, Bruce Momjian br...@momjian.us wrote: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck in parallelism would be: 1. Parallel sort 2. Parallel aggregation (for commutative aggregates) 3. Parallel nested loop join (especially for expression joins, like GIS) parallel data load? :/ We have that in pg_restore, and I think we are getting parallel dump in 9.3, right? Unfortunately, I don't see it in the last 9.3 commit-fest. Is it still being worked on? Not exactly, I meant something like being able to use parallel processing when doing INSERT or COPY directly in core. If there is a parallel processing infrastructure, it could also be used for such write operations. I agree that the cases mentioned by Josh are far more appealing though... I am not sure how a COPY could be easily parallelized, but I supposed it could be done as part of the 1GB segment feature. People have complained that COPY is CPU-bound, so it might be very interesting to see if we could offload some of that parsing overhead to a child. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 01:48:29AM -0300, Alvaro Herrera wrote: Bruce Momjian escribió: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck in parallelism would be: 1. Parallel sort 2. Parallel aggregation (for commutative aggregates) 3. Parallel nested loop join (especially for expression joins, like GIS) parallel data load? :/ We have that in pg_restore, and I thinnk we are getting parallel dump in 9.3, right? Unfortunately, I don't see it in the last 9.3 commit-fest. Is it still being worked on? It's in the previous-to-last commitfest. IIRC that patch required review and testing from people with some Windows background. There are still 34 items needing attention in CF3. I suggest that, if you have some spare time, your help would be very much appreciated there. The commitfest that started on Jan 15th has 65 extra items. Anything currently listed in CF3 can rightfully be considered to be part of CF4, too. Wow, I had no idea we were that far behind. I have avoided commit-fest work because I often travel so might leave the items abandoned, and I try to do cleanup of items that never make the commit-fest --- I thought that was something that needed doing too, and I rarely can complete that task. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 08:11:06AM -0500, Robert Haas wrote: We kind of do - when in a CF we should do reviewing of existing patches, when outside a CF we should do discussions and work on new features. It's on http://wiki.postgresql.org/wiki/CommitFest. It doesn't specifically say do this and don't do htat, but it says focus on review and discussing things that will happen that far ahead is definitely not focusing on review. Bruce is evidently under the impression that he's no longer under any obligation to review or commit other people's patches, or participate in the CommitFest process in any way. I believe that he has not committed a significant patch written by someone else in several years. If the committers on the core team aren't committed to the process, it doesn't stand much chance of working. I assume you know I was the most frequent committer of other people's patches for years before the commit-fests started, so I thought I would move on to other things. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 09:05:39AM -0500, Andrew Dunstan wrote: On 01/15/2013 11:32 PM, Bruce Momjian wrote: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck in parallelism would be: 1. Parallel sort 2. Parallel aggregation (for commutative aggregates) 3. Parallel nested loop join (especially for expression joins, like GIS) parallel data load? :/ We have that in pg_restore, and I thinnk we are getting parallel dump in 9.3, right? Unfortunately, I don't see it in the last 9.3 commit-fest. Is it still being worked on? I am about half way through reviewing it. Unfortunately paid work take precedence over unpaid work. Do you think it will make it into 9.3? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Materialized views WIP patch
Thom Brown wrote: Some weirdness: postgres=# CREATE VIEW v_test2 AS SELECT 1 moo; CREATE VIEW postgres=# CREATE MATERIALIZED VIEW mv_test2 AS SELECT moo, 2*moo FROM v_test2 UNION ALL SELECT moo, 3*moo FROM v_test2; SELECT 2 postgres=# \d+ mv_test2 Materialized view public.mv_test2 Column | Type | Modifiers | Storage | Stats target | Description --+-+---+-+--+- moo | integer | | plain | | ?column? | integer | | plain | | View definition: SELECT *SELECT* 1.moo, *SELECT* 1.?column?; You are very good at coming up with these, Thom! Will investigate. Can you confirm that *selecting* from the MV works as you would expect; it is just the presentation in \d+ that's a problem? Thanks, -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Materialized views WIP patch
On 16 January 2013 17:20, Kevin Grittner kgri...@mail.com wrote: Thom Brown wrote: Some weirdness: postgres=# CREATE VIEW v_test2 AS SELECT 1 moo; CREATE VIEW postgres=# CREATE MATERIALIZED VIEW mv_test2 AS SELECT moo, 2*moo FROM v_test2 UNION ALL SELECT moo, 3*moo FROM v_test2; SELECT 2 postgres=# \d+ mv_test2 Materialized view public.mv_test2 Column | Type | Modifiers | Storage | Stats target | Description --+-+---+-+--+- moo | integer | | plain | | ?column? | integer | | plain | | View definition: SELECT *SELECT* 1.moo, *SELECT* 1.?column?; You are very good at coming up with these, Thom! Will investigate. Can you confirm that *selecting* from the MV works as you would expect; it is just the presentation in \d+ that's a problem? Yes, nothing wrong with using the MV, or refreshing it: postgres=# TABLE mv_test2; moo | ?column? -+-- 1 |2 1 |3 (2 rows) postgres=# SELECT * FROM mv_test2; moo | ?column? -+-- 1 |2 1 |3 (2 rows) postgres=# REFRESH MATERIALIZED VIEW mv_test2; REFRESH MATERIALIZED VIEW But a pg_dump of the MV has the same issue as the view definition: -- -- Name: mv_test2; Type: MATERIALIZED VIEW; Schema: public; Owner: thom; Tablespace: -- CREATE MATERIALIZED VIEW mv_test2 ( moo, ?column? ) AS SELECT *SELECT* 1.moo, *SELECT* 1.?column? WITH NO DATA; -- Thom
Re: [HACKERS] Parallel query execution
On 01/16/2013 12:20 PM, Bruce Momjian wrote: On Wed, Jan 16, 2013 at 09:05:39AM -0500, Andrew Dunstan wrote: On 01/15/2013 11:32 PM, Bruce Momjian wrote: On Wed, Jan 16, 2013 at 01:28:18PM +0900, Michael Paquier wrote: On Wed, Jan 16, 2013 at 1:22 PM, Josh Berkus j...@agliodbs.com wrote: Claudio, Stephen, It really seems like the areas where we could get the most bang for the buck in parallelism would be: 1. Parallel sort 2. Parallel aggregation (for commutative aggregates) 3. Parallel nested loop join (especially for expression joins, like GIS) parallel data load? :/ We have that in pg_restore, and I thinnk we are getting parallel dump in 9.3, right? Unfortunately, I don't see it in the last 9.3 commit-fest. Is it still being worked on? I am about half way through reviewing it. Unfortunately paid work take precedence over unpaid work. Do you think it will make it into 9.3? Yes, I hope it will. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. +1, although I'll suggest that we should have *two* CF managers for this one to keep the workload manageable. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+CF4 WAS: Parallel query execution
I assume you know I was the most frequent committer of other people's patches for years before the commit-fests started, so I thought I would move on to other things. Why would you think that? Given the volume of incoming patches, we need more committers than ever. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+CF4 WAS: Parallel query execution
On Wed, Jan 16, 2013 at 09:50:07AM -0800, Josh Berkus wrote: I assume you know I was the most frequent committer of other people's patches for years before the commit-fests started, so I thought I would move on to other things. Why would you think that? Given the volume of incoming patches, we need more committers than ever. Well, I usually do stuff no one wants to do, and it seems we have people doing this. Also, I had my hand in deciding lots of things when I was committing all those patches in the past, so I thought others should get the chance. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+CF4 WAS: Parallel query execution
Well, I usually do stuff no one wants to do, and it seems we have people doing this. Also, I had my hand in deciding lots of things when I was committing all those patches in the past, so I thought others should get the chance. Well, we clearly don't have *enough* people committing patches. So we could really use your help. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Materialized views WIP patch
Kevin Grittner kgri...@mail.com writes: Tom Lane wrote: Surely that should fall out automatically given that the dependency is properly expressed in pg_depend? The *definitions* sort properly, but what I'm trying to do is define them WITH NO DATA and load data after all the COPY statements for tables. If mva is referenced by mvb, the goal is the REFRESH mva, build its indexes before running REFRESH for mvb and building its indexes. To do things in any other order does't seem to me to leave things after restore in the same state they were in at the time of the dump. Ah. Can't you treat this using the same pg_dump infrastructure as for the data for an ordinary table? The dependencies made for the TableDataInfo object might be a bit different, but after that it seems like the sort logic ought to be happy. Where I really need someone to hit me upside the head with a clue-stick is the code I added to the bottom of RelationBuildDesc() in relcache.c. The idea is that on first access to an unlogged MV, to detect that the heap has been replaced by the init fork, set relisvalid to false, and make the heap look normal again. Hmm. I agree that relcache.c has absolutely no business doing that, but not sure what else to do instead. Seems like it might be better done at first touch of the MV in the parser, rewriter, or planner --- but the fact that I can't immediately decide which of those is right makes me feel that it's still too squishy. I'm also wondering about locking issues there. Obviously you don't want more than one backend trying to rebuild the MV. Do we really need unlogged MVs in the first iteration? Seems like that's adding a whole bunch of new issues, when you have quite enough already without that. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Materialized views WIP patch
Do we really need unlogged MVs in the first iteration? Seems like that's adding a whole bunch of new issues, when you have quite enough already without that. While I think there is strong user demand for unlogged MVs, if we can get MVs without unlogged ones for 9.3, I say go for that. We'll add unlogged in 9.4. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH 3/5] Split out xlog reading into its own module called xlogreader
Andres Freund wrote: The way xlog reading was done up to now made it impossible to use that nontrivial code outside of xlog.c although it is useful for different purposes like debugging wal (xlogdump) and decoding wal back into logical changes. I have pushed this part after some more editorialization. Most notably, (I think,) I removed the #ifdef FRONTEND piece. I think this should be part of the pg_xlogdump patch. Sorry about this. Also I changed XLogReaderAllocate() to not receive an initial read starting point, because that seemed rather pointless. Both xlog.c and the submitted pg_xlogdump use a non-invalid RecPtr in their first call AFAICS. There was a bug in xlog.c's usage of XLogReadRecord: it was calling ereport() on the errorbuf when the return value was NULL, which is not always sane because some errors do not set the errorbuf. I fixed that, and I documented it more clearly in xlogreader.h (I hope). I made some other minor stylistic changes, reworded a few comments, etc. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
Stephen Frost sfr...@snowman.net writes: * Abhijit Menon-Sen (a...@2ndquadrant.com) wrote: Also, what should he start with? CF3 as it stands today, or CF4 with all of the pending patches moved from CF3, immense though the result may be? I slightly prefer the latter, so that we're all on the same page when it comes to seeing what needs to be done. I'd leave it up to him to decide, but I think the general thought was to move it all to one place. The original intention, per agreement at the last dev meeting, https://wiki.postgresql.org/wiki/PgCon_2012_Developer_Meeting#CommitFest_Schedule was that we'd have a triage discussion between CF3 and CF4 to try to figure out which remaining big patches had a realistic chance of getting committed during CF4. The ones that didn't could then be deferred to 9.4 without first sucking a lot of time away from the ones that could get in. If we decide to fold CF3 and CF4 together, either we lose that step (which would make me sad, it seems like a good idea) or we need to figure another way to work it into the process. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
If we decide to fold CF3 and CF4 together, either we lose that step (which would make me sad, it seems like a good idea) or we need to figure another way to work it into the process. Well, we should have the triage discussion ASAP then. We were really supposed to have it a week ago. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] COPY .. COMPRESSED
On 1/15/13 2:53 PM, Stephen Frost wrote: You're right, to clarify, for *file_fdw*, which is a backend-only operation, the popen patch is great (thought I made that clear before). I would think that if we get writable FDWs, you would want file_fdw to go through zlib so that it can write directly to the file. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
* Bruce Momjian (br...@momjian.us) wrote: I am not sure how a COPY could be easily parallelized, but I supposed it could be done as part of the 1GB segment feature. People have complained that COPY is CPU-bound, so it might be very interesting to see if we could offload some of that parsing overhead to a child. COPY can certainly be CPU bound but before we can parallelize that usefully we need to solve the problem around extent locking when trying to do multiple COPY's to the same table. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] [PATCH] COPY .. COMPRESSED
* Peter Eisentraut (pete...@gmx.net) wrote: On 1/15/13 2:53 PM, Stephen Frost wrote: You're right, to clarify, for *file_fdw*, which is a backend-only operation, the popen patch is great (thought I made that clear before). I would think that if we get writable FDWs, you would want file_fdw to go through zlib so that it can write directly to the file. With the popen patch, I expect it could be defined as '|gzip myfile.gz'.. I believe that patch did that. It'd be ideal to add support for that to file_fdw also, certainly. That shouldn't be hard as file_fdw is essentially a wrapper around backend COPY already and this would just be adding a few additional options to pass through. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] CF3+4
Josh Berkus j...@agliodbs.com writes: If we decide to fold CF3 and CF4 together, either we lose that step (which would make me sad, it seems like a good idea) or we need to figure another way to work it into the process. Well, we should have the triage discussion ASAP then. We were really supposed to have it a week ago. Well, I don't want to just say hey, it's triage time! while we're still actively working on the CF. Personally, the reason I've accomplished nothing since early December on CF3 is lack of bandwidth --- bug fixing, email-answering, and non-PG Red Hat work have consumed all my time that wasn't eaten by holiday distractions. I'm hoping to get something done soon towards committing immediately-committable CF entries, but if we're instead focusing on patches that require triage discussions, that's likely to not happen. Looking back at the original schedule agreement, I see I misremembered it a bit; we actually had the idea for two rounds of which-patches-have- a-chance: CF1: June 15 - July 15 CF2: Sept 15 - Oct 15 CF3: Nov 15 - Dec 15 Planning Week - Dec 8-15 CF4.1: Jan 15 - Feb 15 Final Triage: Feb 1-7 but both of those discussion weeks were mid-CF, and I now think that that's got little chance of working because of bandwidth considerations. We certainly totally forgot about the first one. I think a realistic answer might be to admit that we've slipped quite a bit. Set the end date of CF3 to perhaps end of January, do triage the first week of February, and then start CF4 after that, about three or four weeks later than planned. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
On Wed, Jan 16, 2013 at 03:13:50PM -0500, Tom Lane wrote: I think a realistic answer might be to admit that we've slipped quite a bit. Set the end date of CF3 to perhaps end of January, do triage the first week of February, and then start CF4 after that, about three or four weeks later than planned. This makes sense. Things have just not worked out as planned and we should adjust. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
2013/1/16 Stephen Frost sfr...@snowman.net: * Bruce Momjian (br...@momjian.us) wrote: I am not sure how a COPY could be easily parallelized, but I supposed it could be done as part of the 1GB segment feature. People have complained that COPY is CPU-bound, so it might be very interesting to see if we could offload some of that parsing overhead to a child. COPY can certainly be CPU bound but before we can parallelize that usefully we need to solve the problem around extent locking when trying to do multiple COPY's to the same table. Probably update any related indexes and constraint checking should be paralellized. Regards Pavel Thanks, Stephen -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Event Triggers: adding information
Robert Haas robertmh...@gmail.com writes: I think this points to a couple of problems: this patch isn't well-enough thought out, and it's got several features jammed into a single patch. This should really be split up into several patches and each one submitted separately. Ok. Now I want to talk about our process a little. That's a 2 paragraphs diversion, after that it's getting back to technical matters. There's a difference between it's not the way I would have done it and the author didn't think about what he's doing. That's also the reason why it's very hard to justify sending a polished enough patch as a non commiter. And then this patch is like the next one in a long series that is lasting for about 2 years now, and spliting it is just more work for everybody, and then you take the risk that the next commiter who looks at the patch prefers to see a complete enough view of the goal you're trying to reach. I think that ddl_command_trace is an unnecessary frammish that should be ripped out entirely. It is easy enough to accomplish the same thing with ddl_command_start and ddl_command_end, and so we're just increasing the number of cases we have to support in the future for no real benefit. I think you might want to review the use case behing ddl_command_trace, that has been asked by who users wanted to see their use case supported in some easier way than just what you're talking about here. This gets right back to an argument Dimitri and I have been having since v1 of this patch, which is whether these are command triggers or event triggers. I think we ought to support both, but they are not the same thing. I think the design you are proposing is just fine for an event called sql_drop, but it makes no sense for an event called ddl_command_end, which needs to be called once per DDL statement - at the end. Not once per object dropped. What I think you're missing here is the proposal flying around to have drop operation get back to ProcessUtility so that C hooks and event triggers both can have at it. It's been debated on the list earlier, way before we talked about event triggers, also as a way to clean up things, IIRC. I agree that if we keep the code as it is to implement cascading behavior, a sql_drop event makes more sense. TOPLEVEL is supposed to correspond to a complete SQL statement entered by the user: Having the DROP command get back to ProcessUtility would allow us to have the nested events as GENERATED, but CASCADE would allow an easier way to just match that if that's what users are interested into. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 10:06:51PM +0100, Pavel Stehule wrote: 2013/1/16 Stephen Frost sfr...@snowman.net: * Bruce Momjian (br...@momjian.us) wrote: I am not sure how a COPY could be easily parallelized, but I supposed it could be done as part of the 1GB segment feature. People have complained that COPY is CPU-bound, so it might be very interesting to see if we could offload some of that parsing overhead to a child. COPY can certainly be CPU bound but before we can parallelize that usefully we need to solve the problem around extent locking when trying to do multiple COPY's to the same table. Probably update any related indexes and constraint checking should be paralellized. Wiki updated: https://wiki.postgresql.org/wiki/Parallel_Query_Execution -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
2013/1/16 Bruce Momjian br...@momjian.us: Wiki updated: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Could we add CTE to that opportunities list? I think that some kind of queries in CTE queries could be easilly parallelized. []s -- Dickson S. Guedes mail/xmpp: gue...@guedesoft.net - skype: guediz http://guedesoft.net - http://www.postgresql.org.br -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Event Triggers: adding information
On Wed, Jan 16, 2013 at 4:16 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Ok. Now I want to talk about our process a little. That's a 2 paragraphs diversion, after that it's getting back to technical matters. There's a difference between it's not the way I would have done it and the author didn't think about what he's doing. That's also the reason why it's very hard to justify sending a polished enough patch as a non commiter. There's no rule that anyone's got to agree with my opinion on anything, but the idea that a patch should do one thing and do it well is not a novel concept on this mailing list, however much you may feel otherwise. This patch: - adds ddl_command_trace and ddl_command_end events - causes those events to be called not only for actual SQL commands but also for things that happen, at present, to go through the same code path - adds additional magic variables to PL/pgsql to expose new information not previously exposed - adds CONTEXT as an additional event-trigger-filter variable, along with TAG In my mind, that's four or five separate patches. You don't have to agree, but that's what I think, and I'm not going to apologize for thinking it. Moreover, I think you will struggle to find examples of patches committed in the last three years that made as many different, only loosely-related changes as you're proposing for this one. As for the patch not being well-thought-out, I'm sticking by that, too. Alvaro's comments about lock escalations should be enough to tickle your alarm bells, but there are plenty of other problems here, too. I think you might want to review the use case behing ddl_command_trace, that has been asked by who users wanted to see their use case supported in some easier way than just what you're talking about here. That argument carries no water with me. You're asking every PostgreSQL user in the universe to carry the overhead of a feature that 90% of them will not use. That is mighty expensive syntactic sugar. I am not talking about code-complexity, either. It is pretty obvious that there is run-time overhead of having this feature. I am not aware of any case where we have accepted run-time overhead for a feature not mandated by the SQL standard. Given the rest of what you have here, it is quite simple to arrange for the same function to be called after a create or alter and before a drop. Being able to do that in one command instead of two is not a sufficient reason to add another event type. This gets right back to an argument Dimitri and I have been having since v1 of this patch, which is whether these are command triggers or event triggers. I think we ought to support both, but they are not the same thing. I think the design you are proposing is just fine for an event called sql_drop, but it makes no sense for an event called ddl_command_end, which needs to be called once per DDL statement - at the end. Not once per object dropped. What I think you're missing here is the proposal flying around to have drop operation get back to ProcessUtility so that C hooks and event triggers both can have at it. It's been debated on the list earlier, way before we talked about event triggers, also as a way to clean up things, IIRC. I'm very much opposed to that proposal, as I am to your proposal to expose internal and generated events to users. Recursing into ProcessUtility is a nasty, messy hook that is responsible for subtle bugs and locking problems in our current DDL implementation. We should be trying to refactor that to clean it up, not exposing it as a user-visible detail. I do NOT want a future refactoring effort to clean up race conditions and duplicate name lookups in the DDL code to be blocked because someone is relying on the details of the existing implementation in their event trigger implementation. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 07:57:01PM -0200, Dickson S. Guedes wrote: 2013/1/16 Bruce Momjian br...@momjian.us: Wiki updated: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Could we add CTE to that opportunities list? I think that some kind of queries in CTE queries could be easilly parallelized. I added CTEs with joins. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4 (was Re: Parallel query execution)
On Wed, Jan 16, 2013 at 12:36 PM, Josh Berkus j...@agliodbs.com wrote: I would like to nominate Craig Ringer to be independent CF mgr for Jan2013 CF. +1, although I'll suggest that we should have *two* CF managers for this one to keep the workload manageable. That has never worked before, so I'm reluctant to start now. Either one person does all the work and the other person still gets half the credit, or neither person does any work because they're hoping the other person has it covered. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] COPY .. COMPRESSED
On Mon, Jan 14, 2013 at 11:28 AM, Stephen Frost sfr...@snowman.net wrote: * Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost sfr...@snowman.net writes: I do like the idea of a generalized answer which just runs a user-provided command on the server but that's always going to require superuser privileges. The design that was being kicked around allowed pipes to be used on the client side too, ie \copy foo to '| gzip ...'. That form would not require any special privileges, and might be thought preferable for another reason too: it offloads the work from the server. It's a different use-case which, imv, is really already trivially covered: psql -c 'COPY foo TO STDOUT;' | gzip myfile.gz While there is no option currently for having the server do the compression before sending the data over the wire. +1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] COPY .. COMPRESSED
On Tue, Jan 15, 2013 at 3:37 PM, Christopher Browne cbbro...@gmail.com wrote:
That points towards a fix that involves having a set of non-arbitrary commands
that we allow plain users to use.
Hmm. There's an interesting thought...
How about having a pg_filters table in pg_catalog which allows capturing
labels and names of known-to-be-safe binary filters:
insert into pg_filters (label, location)
values
('zcat', '/usr/bin/zcat'),
('bzip2', '/usr/bin/bzip2'),
('bunzip2', '/usr/bin/bunzip2');
And then having some capability to grant permissions to roles to use
these filters.
I suspect that's going to be less efficient than using a compression
library that's linked into the backend, because you have to copy all
the data through the kernel to another process and back. And it's
certainly a lot more complex.
If it greatly broadened the applicability of this feature I might
think it was worthwhile, but I can't see that it does. I suspect that
supporting zlib, which we already linked against, would cater to
something well upwards of 90% of the use cases here. Sure, there are
other things, but zlib is very widely used and bzip2 IME is far too
slow to be taken seriously for this kind of application. The
additional space savings that you get for the additional CPU
investment is typically small, and if you really need it, having to
un-gzip and re-bzip2 on the client is always an option. If you're
using bzip2 you obviously have CPU time to burn.
At any rate, I think it would be good to avoid letting our desire for
infinite flexibility get in the way of doing something useful.
--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Teaching pg_receivexlog to follow timeline switches
Fujii Masao masao.fu...@gmail.com writes: Thanks for elaborating the reason why .partial suffix should be kept. I agree that keeping the .partial suffix would be safer. +1 to both points. So +2 I guess :) Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] COPY .. COMPRESSED
On Wed, Jan 16, 2013 at 5:15 PM, Robert Haas robertmh...@gmail.com wrote:
On Tue, Jan 15, 2013 at 3:37 PM, Christopher Browne cbbro...@gmail.com
wrote:
That points towards a fix that involves having a set of non-arbitrary
commands
that we allow plain users to use.
Hmm. There's an interesting thought...
How about having a pg_filters table in pg_catalog which allows capturing
labels and names of known-to-be-safe binary filters:
insert into pg_filters (label, location)
values
('zcat', '/usr/bin/zcat'),
('bzip2', '/usr/bin/bzip2'),
('bunzip2', '/usr/bin/bunzip2');
And then having some capability to grant permissions to roles to use
these filters.
I suspect that's going to be less efficient than using a compression
library that's linked into the backend, because you have to copy all
the data through the kernel to another process and back. And it's
certainly a lot more complex.
More complex, certainly.
By spawning a separate process, we'd get benefit of multicore CPUs,
so I'm not sure I agree that it's necessarily slower.
At any rate, I think it would be good to avoid letting our desire for
infinite flexibility get in the way of doing something useful.
Oh, agreed. I was actively thinking of the cooler bits of this pointing more
towards 9.4 than 9.3.
--
When confronted by a difficult problem, solve it by reducing it to the
question, How would the Lone Ranger handle this?
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: pgbench - aggregation of info written into log
Hi,
I'm looking into this as a committer. It seems that this is the
newest patch and the reviewer(Pavel) stated that it is ready for
commit. However, I noticed that this patch adds a Linux/UNIX only
feature(not available on Windows). So I would like to ask cores if
this is ok or not.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp
Hi,
attached is a v5 of this patch. Details below:
On 8.12.2012 16:33, Andres Freund wrote:
Hi Tomas,
On 2012-11-27 14:55:59 +0100, Pavel Stehule wrote:
attached is a v4 of the patch. There are not many changes, mostly some
simple tidying up, except for handling the Windows.
After a quick look I am not sure what all the talk about windows is
about? instr_time.h seems to provide all you need, even for windows? The
only issue of gettimeofday() for windows seems to be that it is that its
not all that fast an not too high precision, but that shouldn't be a
problem in this case?
Could you expand a bit on the problems?
As explained in the previous message, this is an existing problem with
unavailable timestamp. I'm not very fond of adding Linux-only features,
but fixing that was not the goal of this patch.
* I had a problem with doc
The current patch has conflict markers in the sgml source, there seems
to have been some unresolved merge. Maybe that's all that causes the
errors?
Whats your problem with setting up the doc toolchain?
Yeah, my fault because of incomplete merge. But the real culprit was a
missing refsect2. Fixed.
issues:
* empty lines with invisible chars (tabs) + and sometimes empty lines
after and before {}
Fixed (I've removed the lines etc.)
* adjustment of start_time
+ * the desired interval */
+ while (agg-start_time
+ agg_interval INSTR_TIME_GET_DOUBLE(now))
+
agg-start_time = agg-start_time + agg_interval;
can skip one interval - so when transaction time will be larger or
similar to agg_interval - then results can be strange. We have to know
real length of interval
Could you post a patch that adresses these issues?
So, in the end I've rewritten the section advancing the start_time. Now
it works so that when skipping an interval (because of a very long
transaction), it will print lines even for those empty intervals.
So for example with a transaction file containing a single query
SELECT pg_sleep(1.5);
and an interval length of 1 second, you'll get something like this:
1355009677 0 0 0 0 0
1355009678 1 1501028 2253085056784 1501028 1501028
1355009679 0 0 0 0 0
1355009680 1 1500790 2252370624100 1500790 1500790
1355009681 1 1500723 2252169522729 1500723 1500723
1355009682 0 0 0 0 0
1355009683 1 1500719 2252157516961 1500719 1500719
1355009684 1 1500703 2252109494209 1500703 1500703
1355009685 0 0 0 0 0
which is IMHO the best way to deal with this.
I've fixed several minor issues, added a few comments.
regards
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Event Triggers: adding information
Dimitri Fontaine dimi...@2ndquadrant.fr writes: There's a difference between it's not the way I would have done it and the author didn't think about what he's doing. That's also the reason why it's very hard to justify sending a polished enough patch as a non commiter. And then this patch is like the next one in a long series that is lasting for about 2 years now, and spliting it is just more work for everybody, and then you take the risk that the next commiter who looks at the patch prefers to see a complete enough view of the goal you're trying to reach. What was discussed at the last dev meeting was assigning a committer to each large patch to start with, which would reduce the risk of the goalposts moving that way. It seems to me that Robert's at least unofficially taken that role for event triggers. You should be happy, because if I were reviewing it I'd likely bounce the whole thing. I'm not convinced this will *ever* be a stable feature that doesn't create more problems than it fixes. What I think you're missing here is the proposal flying around to have drop operation get back to ProcessUtility so that C hooks and event triggers both can have at it. I've seen no such proposal, and it seems like a nonstarter just from the idea. dependency.c doesn't have a syntax tree to describe each object that it finds to drop; creating one, and then doing a lookup to re-find the object, is just going to make drops hugely slower and buggier. Not to mention the large amount of code that would have to be added and maintained. Not to mention that the objects dependency.c works with aren't necessarily all that interesting from the user's level --- for instance, do you really want to see each column default expression dropped individually? Not to mention that the permissions considerations are different from a standalone DROP. The bigger picture there is that it's taken us years, and multiple major iterations, to get cascaded drops to work properly and reliably. I'm disinclined to rip that code open and rewrite it completely; let alone add hooks that might inject random operations at any point in the process. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] COPY .. COMPRESSED
On Tue, Jan 15, 2013 at 4:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: I find the argument that this supports compression-over-the-wire to be quite weak, because COPY is only one form of bulk data transfer, and one that a lot of applications don't ever use. If we think we need to support transmission compression for ourselves, it ought to be integrated at the wire protocol level, not in COPY. Just to not look like I'm rejecting stuff without proposing alternatives, here is an idea about a backwards-compatible design for doing that: we could add an option that can be set in the connection request packet. Say, transmission_compression = gzip. But presumably this would transparently compress at one end and decompress at the other end, which is again a somewhat different use case. To get compressed output on the client side, you have to decompress and recompress. Maybe that's OK, but it's not quite the same thing. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] COPY .. COMPRESSED
* Bruce Momjian (br...@momjian.us) wrote: On Wed, Jan 16, 2013 at 06:19:09PM -0500, Robert Haas wrote: But presumably this would transparently compress at one end and decompress at the other end, which is again a somewhat different use case. To get compressed output on the client side, you have to decompress and recompress. Maybe that's OK, but it's not quite the same thing. Is there a TODO here? For some reason, I thought there was a patch... Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Re: Privileges for INFORMATION_SCHEMA.SCHEMATA (was Re: [DOCS] Small clarification in 34.41. schemata)
2013/1/15 Tom Lane t...@sss.pgh.pa.us: Casey Allen Shobe ca...@shobe.info writes: On Wed, Jan 9, 2013 at 8:56 PM, Tom Lane t...@sss.pgh.pa.us wrote: However, it seems to me that this behavior is actually wrong for our purposes, as it represents a too-literal reading of the spec. The SQL standard has no concept of privileges on schemas, only ownership. We do have privileges on schemas, so it seems to me that the consistent thing would be for this view to show any schema that you either own or have some privilege on. IMHO, schemata should follow the standard as it does today. Other platforms have privileges on schemas as well, and this sort of thing seems to fall into the same bucket as other platform compatibilities outside the scope of what the standard thinks about, which means you use pg_catalog to access that information rather than information_schema, which should be expected to work consistently on all platforms that implement it. Meh. To me, standards compliance requires that if you have created a SQL-compliant database, you'd better see spec-compliant output from the information schema. As soon as you do something outside the standard (in this instance, grant some privileges on a schema), it becomes a judgment call whether and how that should affect what you see in the information schema. It may be that the current behavior of this view is actually the best thing, but a standards-compliance argument doesn't do anything to convince me. regards, tom lane My original assumption here was that the documentation [1] was in need of clarification. On the other hand the current output of information_schema.schemata isn't quite I was expecting, which would be as Tom writes: the consistent thing would be for this view to show any schema that you either own or have some privilege on. As it stands, the only way of extracting a list of visible schemas from PostgreSQL's information_schema (i.e. without relying on PostgreSQL-specific system functions) is doing something like this: SELECT DISTINCT(table_schema) FROM information_schema.tables Digging about a bit [2], it seems the only other RDBMSes with a fully-fledged information_schema are Microsoft SQL Server and MySQL. I don't have access to SQL Server; the documentation [3] says Returns one row for each schema in the current database, which also strikes me as incorrect (can someone confirm this behaviour?). For MySQL, the documentation [4] indicates that their implementation shows all schemas (in MySQL: databases) visible to the current user, and I've confirmed this behaviour with MySQL 5.5. Personally I'd support modifying PostgreSQL's information_schema.schemata to show all schemas the current user owns/has privileges on, providing it's not an egregious violation of the SQL standard. It seems I'm not the only user who has been stymied by this issue [5][6][7]; also, resolving it would also make it consistent with MySQL's output [8] [1] http://www.postgresql.org/docs/9.2/static/infoschema-schemata.html [2] http://en.wikipedia.org/wiki/Information_schema [3] http://msdn.microsoft.com/en-us/library/ms182642.aspx [4] http://dev.mysql.com/doc/refman/5.5/en/schemata-table.html [5] http://www.postgresql.org/message-id/CAFjNrYv4MrkbXi-usroCqNiaSyEAzvJ7GjtsEJW2RK7-R=8...@mail.gmail.com [6] http://www.postgresql.org/message-id/200612211146.kblbklqa001...@wwwmaster.postgresql.org [7] http://www.postgresql.org/message-id/50aff3fe.4030...@gmail.com [8] Not that I'm claiming MySQL's implementation is authoritative or anything Regards Ian Lawrence Barwick -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PATCH] COPY .. COMPRESSED
On Wed, Jan 16, 2013 at 06:19:09PM -0500, Robert Haas wrote: On Tue, Jan 15, 2013 at 4:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: I find the argument that this supports compression-over-the-wire to be quite weak, because COPY is only one form of bulk data transfer, and one that a lot of applications don't ever use. If we think we need to support transmission compression for ourselves, it ought to be integrated at the wire protocol level, not in COPY. Just to not look like I'm rejecting stuff without proposing alternatives, here is an idea about a backwards-compatible design for doing that: we could add an option that can be set in the connection request packet. Say, transmission_compression = gzip. But presumably this would transparently compress at one end and decompress at the other end, which is again a somewhat different use case. To get compressed output on the client side, you have to decompress and recompress. Maybe that's OK, but it's not quite the same thing. Is there a TODO here? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: pgbench - aggregation of info written into log
On 01/16/2013 05:59 PM, Tatsuo Ishii wrote: Hi, I'm looking into this as a committer. It seems that this is the newest patch and the reviewer(Pavel) stated that it is ready for commit. However, I noticed that this patch adds a Linux/UNIX only feature(not available on Windows). So I would like to ask cores if this is ok or not. I haven't been following the thread, but if the complaint is that Windows doesn't have accurate high-resolution timers, which is what it kinda looks like from the rest of your message, then it's not true. Every version since Windows2000 has had QueryPerformanceCounter()/QueryPerformanceFrequency(). And we use it: see src/include/portability/instr_time.h If that's not the problem, then can someone please point me at the message that sets the problem out clearly, or else just recap it? cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Event Triggers: adding information
On Wed, Jan 16, 2013 at 6:14 PM, Tom Lane t...@sss.pgh.pa.us wrote: What was discussed at the last dev meeting was assigning a committer to each large patch to start with, which would reduce the risk of the goalposts moving that way. It seems to me that Robert's at least unofficially taken that role for event triggers. You should be happy, because if I were reviewing it I'd likely bounce the whole thing. I'm not convinced this will *ever* be a stable feature that doesn't create more problems than it fixes. And speaking of the goalposts moving... I don't think that's the problem, here. Rather, I think the problem is that the design is ardently refusing to move. It might be a slight overstatement to say that every review I've ever posted for this patch has complained about design decisions that expose implementation details to the user that we might want to change later, but not by much. And yet, two years on, we've got proposals on the table to artificially force *more* things through ProcessUtility(). There's no particularly consistency to which things do and don't go through that function today, and no reason whatsoever to try to force everything to go through there. I agree with everything you say in the portion of the email I didn't quote, and I'm pretty sure I've made similar points more than once in the past. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: pgbench - aggregation of info written into log
I'm looking into this as a committer. It seems that this is the
newest patch and the reviewer(Pavel) stated that it is ready for
commit. However, I noticed that this patch adds a Linux/UNIX only
feature(not available on Windows). So I would like to ask cores if
this is ok or not.
I haven't been following the thread, but if the complaint is that
Windows doesn't have accurate high-resolution timers, which is what it
kinda looks like from the rest of your message, then it's not
true. Every version since Windows2000 has had
QueryPerformanceCounter()/QueryPerformanceFrequency(). And we use it:
see src/include/portability/instr_time.h
In my understanding the problem is not related to resolution.
If that's not the problem, then can someone please point me at the
message that sets the problem out clearly, or else just recap it?
It seems instr_time.h on Windows simply does not provide current
timestamp. From pgbench.c:
/*
* if transaction finished, record the time it took in the log
*/
if (logfile commands[st-state + 1] == NULL)
{
instr_time now;
instr_time diff;
double usec;
INSTR_TIME_SET_CURRENT(now);
diff = now;
INSTR_TIME_SUBTRACT(diff, st-txn_begin);
usec = (double) INSTR_TIME_GET_MICROSEC(diff);
#ifndef WIN32
/* This is more than we really ought to know about
instr_time */
fprintf(logfile, %d %d %.0f %d %ld %ld\n,
st-id, st-cnt, usec, st-use_file,
(long) now.tv_sec, (long) now.tv_usec);
#else
/* On Windows, instr_time doesn't provide a timestamp
anyway */
fprintf(logfile, %d %d %.0f %d 0 0\n,
st-id, st-cnt, usec, st-use_file);
#endif
}
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: pgbench - aggregation of info written into log
On 01/16/2013 06:48 PM, Tatsuo Ishii wrote:
I'm looking into this as a committer. It seems that this is the
newest patch and the reviewer(Pavel) stated that it is ready for
commit. However, I noticed that this patch adds a Linux/UNIX only
feature(not available on Windows). So I would like to ask cores if
this is ok or not.
I haven't been following the thread, but if the complaint is that
Windows doesn't have accurate high-resolution timers, which is what it
kinda looks like from the rest of your message, then it's not
true. Every version since Windows2000 has had
QueryPerformanceCounter()/QueryPerformanceFrequency(). And we use it:
see src/include/portability/instr_time.h
In my understanding the problem is not related to resolution.
If that's not the problem, then can someone please point me at the
message that sets the problem out clearly, or else just recap it?
It seems instr_time.h on Windows simply does not provide current
timestamp. From pgbench.c:
/*
* if transaction finished, record the time it took in the log
*/
if (logfile commands[st-state + 1] == NULL)
{
instr_time now;
instr_time diff;
double usec;
INSTR_TIME_SET_CURRENT(now);
diff = now;
INSTR_TIME_SUBTRACT(diff, st-txn_begin);
usec = (double) INSTR_TIME_GET_MICROSEC(diff);
#ifndef WIN32
/* This is more than we really ought to know about
instr_time */
fprintf(logfile, %d %d %.0f %d %ld %ld\n,
st-id, st-cnt, usec, st-use_file,
(long) now.tv_sec, (long) now.tv_usec);
#else
/* On Windows, instr_time doesn't provide a timestamp
anyway */
fprintf(logfile, %d %d %.0f %d 0 0\n,
st-id, st-cnt, usec, st-use_file);
#endif
}
This might be way more than we want to do, but there is an article that
describes some techniques for doing what seems to be missing (AIUI):
http://msdn.microsoft.com/en-us/magazine/cc163996.aspx
cheers
andrew
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Tuesday, January 15, 2013, Stephen Frost wrote: * Gavin Flower (gavinflo...@archidevsys.co.nz javascript:;) wrote: How about being aware of multiple spindles - so if the requested data covers multiple spindles, then data could be extracted in parallel. This may, or may not, involve multiple I/O channels? Yes, this should dovetail with partitioning and tablespaces to pick up on exactly that. I'd rather not have the benefits of parallelism be tied to partitioning if we can help it. Hopefully implementing parallelism in core would result in something more transparent than that. Cheers, Jeff
Re: [HACKERS] Parallel query execution
On Tuesday, January 15, 2013, Gavin Flower wrote: On 16/01/13 11:14, Bruce Momjian wrote: I mentioned last year that I wanted to start working on parallelism: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Years ago I added thread-safety to libpq. Recently I added two parallel execution paths to pg_upgrade. The first parallel path allows execution of external binaries pg_dump and psql (to restore). The second parallel path does copy/link by calling fork/thread-safe C functions. I was able to do each in 2-3 days. I believe it is time to start adding parallel execution to the backend. We already have some parallelism in the backend: effective_io_concurrency and helper processes. I think it is time we start to consider additional options. Parallelism isn't going to help all queries, in fact it might be just a small subset, but it will be the larger queries. The pg_upgrade parallelism only helps clusters with multiple databases or tablespaces, but the improvements are significant. I have summarized my ideas by updating our Parallel Query Execution wiki page: https://wiki.postgresql.org/wiki/Parallel_Query_Execution Please consider updating the page yourself or posting your ideas to this thread. Thanks. Hmm... How about being aware of multiple spindles - so if the requested data covers multiple spindles, then data could be extracted in parallel. This may, or may not, involve multiple I/O channels? effective_io_concurrency does this for bitmap scans. I thought there was a patch in the commitfest to extend this to ordinary index scans, but now I can't find it. But it still doesn't give you CPU parallelism. The nice thing about CPU parallelism is that it usually brings some amount of IO parallelism for free, while the reverse less likely to be so. Cheers, Jeff
Re: [HACKERS] review: pgbench - aggregation of info written into log
It seems instr_time.h on Windows simply does not provide current
timestamp. From pgbench.c:
/*
* if transaction finished, record the time it took in the log
*/
if (logfile commands[st-state + 1] == NULL)
{
instr_time now;
instr_time diff;
double usec;
INSTR_TIME_SET_CURRENT(now);
diff = now;
INSTR_TIME_SUBTRACT(diff, st-txn_begin);
usec = (double) INSTR_TIME_GET_MICROSEC(diff);
#ifndef WIN32
/* This is more than we really ought to know about
instr_time */
fprintf(logfile, %d %d %.0f %d %ld %ld\n,
st-id, st-cnt, usec, st-use_file,
(long) now.tv_sec, (long) now.tv_usec);
#else
/* On Windows, instr_time doesn't provide a timestamp
anyway */
fprintf(logfile, %d %d %.0f %d 0 0\n,
st-id, st-cnt, usec, st-use_file);
#endif
}
This might be way more than we want to do, but there is an article
that describes some techniques for doing what seems to be missing
(AIUI):
http://msdn.microsoft.com/en-us/magazine/cc163996.aspx
Even this would be doable, I'm afraid it may not fit in 9.3 if we
think about the current status of CF. So our choice would be:
1) Postpone the patch to 9.4
2) Commit the patch in 9.3 without Windows support
I personally am ok with #2. We traditionally avoid particular paltform
specific features on PostgreSQL. However I think the policiy could be
losen for contrib staffs. Also pgbench is just a client program. We
could always use pgbench on UNIX/Linux if we truely need the feature.
What do you think?
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] string escaping in tutorial/syscat.source
On Tuesday, January 15, 2013, Josh Kupershmidt wrote: On Tue, Jan 15, 2013 at 6:35 PM, Jeff Janes jeff.ja...@gmail.comjavascript:; wrote: Do you propose back-patching this? You could argue that this is a bug in 9.1 and 9.2. Before that, they generate deprecation warnings, but do not give the wrong answer. I think that backpatching to 9.1 would be reasonable, though I won't complain if the fix is only applied to HEAD. If it is only to be applied to HEAD, or only to 9.1, 9.2, and HEAD, then this part seems to be unnecessary and I think should be removed (setting a value to its default is more likely to cause confusion than remove confusion): SET standard_conforming_strings TO on; and the corresponding reset as well. Well, it may be unnecessary for people who use the modern default standard_conforming_strings. But some people have kept standard_conforming_strings=off in recent versions because they have old code which depends on this. OK, I didn't anticipate them doing that in their default (i.e. postgresql.conf) but of course they might do that. I've marked it ready for committer. Thanks, Jeff
Re: [HACKERS] review: pgbench - aggregation of info written into log
On 01/16/2013 08:05 PM, Tatsuo Ishii wrote:
It seems instr_time.h on Windows simply does not provide current
timestamp. From pgbench.c:
/*
* if transaction finished, record the time it took in the log
*/
if (logfile commands[st-state + 1] == NULL)
{
instr_time now;
instr_time diff;
double usec;
INSTR_TIME_SET_CURRENT(now);
diff = now;
INSTR_TIME_SUBTRACT(diff, st-txn_begin);
usec = (double) INSTR_TIME_GET_MICROSEC(diff);
#ifndef WIN32
/* This is more than we really ought to know about
instr_time */
fprintf(logfile, %d %d %.0f %d %ld %ld\n,
st-id, st-cnt, usec, st-use_file,
(long) now.tv_sec, (long) now.tv_usec);
#else
/* On Windows, instr_time doesn't provide a timestamp
anyway */
fprintf(logfile, %d %d %.0f %d 0 0\n,
st-id, st-cnt, usec, st-use_file);
#endif
}
This might be way more than we want to do, but there is an article
that describes some techniques for doing what seems to be missing
(AIUI):
http://msdn.microsoft.com/en-us/magazine/cc163996.aspx
Even this would be doable, I'm afraid it may not fit in 9.3 if we
think about the current status of CF. So our choice would be:
1) Postpone the patch to 9.4
2) Commit the patch in 9.3 without Windows support
I personally am ok with #2. We traditionally avoid particular paltform
specific features on PostgreSQL. However I think the policiy could be
losen for contrib staffs. Also pgbench is just a client program. We
could always use pgbench on UNIX/Linux if we truely need the feature.
What do you think?
Fair enough, I was just trying to point out alternatives. We have
committed platform-specific features before now. I hope it doesn't just
get left like this, though.
cheers
andrew
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] CF3+4
On 16 January 2013 19:28, Tom Lane t...@sss.pgh.pa.us wrote: Stephen Frost sfr...@snowman.net writes: * Abhijit Menon-Sen (a...@2ndquadrant.com) wrote: Also, what should he start with? CF3 as it stands today, or CF4 with all of the pending patches moved from CF3, immense though the result may be? I slightly prefer the latter, so that we're all on the same page when it comes to seeing what needs to be done. I'd leave it up to him to decide, but I think the general thought was to move it all to one place. The original intention, per agreement at the last dev meeting, https://wiki.postgresql.org/wiki/PgCon_2012_Developer_Meeting#CommitFest_Schedule was that we'd have a triage discussion between CF3 and CF4 to try to figure out which remaining big patches had a realistic chance of getting committed during CF4. The ones that didn't could then be deferred to 9.4 without first sucking a lot of time away from the ones that could get in. Sorry to correct you but that was not the original intention/agreement. Robert made that suggestion, and I opposed it, saying it was too early and suggesting triage week for first week of Feb instead. My recollection, sitting opposite you, was that you agreed, as did many others. I agree we need triage, and have no problem if you lead that. But lets wait until early Feb, please. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] review: pgbench - aggregation of info written into log
This might be way more than we want to do, but there is an article that describes some techniques for doing what seems to be missing (AIUI): http://msdn.microsoft.com/en-us/magazine/cc163996.aspx Even this would be doable, I'm afraid it may not fit in 9.3 if we think about the current status of CF. So our choice would be: 1) Postpone the patch to 9.4 2) Commit the patch in 9.3 without Windows support I personally am ok with #2. We traditionally avoid particular paltform specific features on PostgreSQL. However I think the policiy could be losen for contrib staffs. Also pgbench is just a client program. We could always use pgbench on UNIX/Linux if we truely need the feature. What do you think? Fair enough, I was just trying to point out alternatives. We have committed platform-specific features before now. I hope it doesn't just get left like this, though. Yeah, I hope someone pick this up and propose as a TODO item. In the mean time, I'm going to commit the patch without Windows support unless there's objection. -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese: http://www.sraoss.co.jp -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Enabling Checksums
New version of checksums patch. Changes: * rebased * removed two duplicate lines; apparently the result of a bad merge * Added heap page to WAL chain when logging an XLOG_HEAP2_VISIBLE to avoid torn page issues updating PD_ALL_VISIBLE. This is the most significant change. * minor comment cleanup No open issues that I'm aware of with the patch itself. Greg appears to have made some progress on the automated corruption tester. Note to reviewers: I also have a patch out to remove PD_ALL_VISIBLE entirely. The code around PD_ALL_VISIBLE is quite tricky (with or without this patch), so if the PD_ALL_VISIBLE patch is committed first then it will make reviewing this patch easier. Regardless, the second patch to be committed will need to be rebased on top of the first. Regards, Jeff Davis replace-tli-with-checksums-20130116.patch.gz Description: GNU Zip compressed data checksums-20130116.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] log_lock_waits to identify transaction's relation
On 16 January 2013 16:12, Stephen Frost sfr...@snowman.net wrote: * Simon Riggs (si...@2ndquadrant.com) wrote: A couple quick notes regarding the patch- what does GetXactLockTableRelid really provide..? The ability to access a static variable in a different module. It doesn't provide anything other than that, It isn't actually necessary for that currently though, is it? All calls to it appeared to be inside the same module. Do you anticipate that needing to change in the future? OK, will change. The information isn't available, which is why I didn't include it. Comments explain that the additional information is only available within the backend that was waiting. That's sufficient for stats, but not enough for an extended multi-backend deadlock report. Right, sorry, to be clear, it seemed that we could simply detect if the information is available and print it out if it is- and not print it if it isn't. The additional boolean variable looked to be unnecessary, or is there a point where OidIsValid(GetXactLockTableRelid()) will be true but the additional information shouldn't be printed or isn't actually available? You might be reporting a deadlock in another backend when you are waiting on a transaction yourself. If we don't address that we'll end up reporting the wrong info in a way that makes it look real and probably never even notice. There is an API change to XactLockTableWait() to pass the relid. That part is essential to any solution. I'm wondering if we could pass in something both more generic and with more information- perhaps a simple string which is constructed when we have more information and is then included in the log message which is generated? Perhaps something which includes both the relname and the OID? Could that be used across more than just these kinds of locks? Of course, on the flip side, I understand that we don't want to spend a lot of time building strings and whatnot during all of these code paths; would be nice if we could defer that until we're about to log. Keeping it as an Oid is better, since we don't always need the string. Regardless, I do think it'd be good to update the comments to indicate that the relid is being passed in solely for the purpose of being available to be included in the log, if necessary. OK, will add. -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel query execution
On Wed, Jan 16, 2013 at 10:04 PM, Jeff Janes jeff.ja...@gmail.com wrote: Hmm... How about being aware of multiple spindles - so if the requested data covers multiple spindles, then data could be extracted in parallel. This may, or may not, involve multiple I/O channels? effective_io_concurrency does this for bitmap scans. I thought there was a patch in the commitfest to extend this to ordinary index scans, but now I can't find it. I never pushed it to the CF since it interacts so badly with the kernel. I was thinking about pushing the small part that is a net win in all cases, the back-sequential patch, but that's independent of any spindle count. It's more related to rotating media and read request merges than it is to multiple spindles or parallelization. The kernel guys basically are waiting for me to patch the kernel. I think I convinced our IT guy at the office to lend me a machine for tests... so it might happen soon. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
