Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Amit Kapila
From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Tom Lane Sent: Monday, July 16, 2012 3:06 AM It might have accidentally failed to fail if tested on a compiler that gives a full 32 bits to enum ForkNumber, but there absolutely would be padding

Re: [HACKERS] [patch] libpq one-row-at-a-time API

2012-07-16 Thread Marko Kreen
On Mon, Jul 16, 2012 at 4:11 AM, Tom Lane t...@sss.pgh.pa.us wrote: Marko Kreen mark...@gmail.com writes: Now, looking at the problem with some perspective, the solution is obvious: when in single-row mode, the PQgetResult() must return proper PGresult for that single row. And everything else

Re: [HACKERS] [patch] libpq one-row-at-a-time API

2012-07-16 Thread Tom Lane
Marko Kreen mark...@gmail.com writes: On Mon, Jul 16, 2012 at 4:11 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm starting to look at this patch now. I think we could drop the PQgetRowData() API: it complicates matters for little gain that I can see. The argument for it was to avoid the cost of

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Robert Haas
On Sun, Jul 15, 2012 at 5:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: We could fairly cheaply dodge the problem with padding after ForkNumber if we were to zero out the whole request array at shmem initialization, so that any such pad bytes are guaranteed zero. However, padding in

Re: [HACKERS] [patch] libpq one-row-at-a-time API

2012-07-16 Thread Marko Kreen
On Mon, Jul 16, 2012 at 4:33 PM, Tom Lane t...@sss.pgh.pa.us wrote: Marko Kreen mark...@gmail.com writes: On Mon, Jul 16, 2012 at 4:11 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm starting to look at this patch now. I think we could drop the PQgetRowData() API: it complicates matters for little

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, Jul 15, 2012 at 5:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: We could fairly cheaply dodge the problem with padding after ForkNumber if we were to zero out the whole request array at shmem initialization, so that any such pad bytes are

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Heikki Linnakangas
On 16.07.2012 18:24, Robert Haas wrote: On Sun, Jul 15, 2012 at 5:36 PM, Tom Lanet...@sss.pgh.pa.us wrote: We could fairly cheaply dodge the problem with padding after ForkNumber if we were to zero out the whole request array at shmem initialization, so that any such pad bytes are guaranteed

Re: [HACKERS] [patch] libpq one-row-at-a-time API

2012-07-16 Thread Tom Lane
Marko Kreen mark...@gmail.com writes: On Mon, Jul 16, 2012 at 4:33 PM, Tom Lane t...@sss.pgh.pa.us wrote: Mm. I still think we should drop it, because it's still a dangerous API that's not necessary for the principal benefit of this feature. Yes, it is a secondary feature, but it fits the

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Tom Lane
I wrote: Robert Haas robertmh...@gmail.com writes: So I'm having a hard time understanding under what imaginable set of circumstances this might break. Padding inside RelFileNodeBackend would break it, because ForwardFsyncRequest copies the rnode as a struct. So that's why I'm asking

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Robert Haas
On Sun, Jul 15, 2012 at 2:29 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think what we ought to do is bite the bullet and refactor the representation of the pendingOps table. What I'm thinking about is reducing the hash key to just RelFileNodeBackend + ForkNumber, so that there's one hashtable

Re: [HACKERS] [patch] libpq one-row-at-a-time API

2012-07-16 Thread Marko Kreen
On Mon, Jul 16, 2012 at 6:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: Marko Kreen mark...@gmail.com writes: On Mon, Jul 16, 2012 at 4:33 PM, Tom Lane t...@sss.pgh.pa.us wrote: Mm. I still think we should drop it, because it's still a dangerous API that's not necessary for the principal benefit

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 11:57 AM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Robert Haas robertmh...@gmail.com writes: So I'm having a hard time understanding under what imaginable set of circumstances this might break. Padding inside RelFileNodeBackend would break it, because

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sun, Jul 15, 2012 at 2:29 PM, Tom Lane t...@sss.pgh.pa.us wrote: Yeah, you have a point there. It's not real clear that switching fsync from off to on is an operation that we can make any guarantees about, short of executing something like the code

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Jul 16, 2012 at 11:57 AM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, I'd be a lot happier about assuming that bare RelFileNode contains no padding, because that's at least got all the fields the same type. So that brings us back to the question of

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 11:44 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I wouldn't rely on that, though. I wouldn't be surprised if there was some debugging flag or similar that initialized all pages to random values or 0xdeadbeef or something, before handing them out to

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 12:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sun, Jul 15, 2012 at 2:29 PM, Tom Lane t...@sss.pgh.pa.us wrote: Yeah, you have a point there. It's not real clear that switching fsync from off to on is an operation that we can

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: The documentation on MacOS X isn't quite as explicit, but I'd still be astonished if we found any other behavior. TBH, I'd be kind of surprised if this is the only place in our code base that relies on the initial contents of shared memory being

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Jul 16, 2012 at 12:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Uh, that's exactly what's under discussion. Not sending useless fsync requests when fsync is off is just one part of it; a part that happens to be quite useful for some test

Re: [HACKERS] Getting rid of pre-assignment of index names in CREATE TABLE LIKE

2012-07-16 Thread Robert Haas
On Sat, Jul 14, 2012 at 4:02 PM, Tom Lane t...@sss.pgh.pa.us wrote: In bug #6734 we have a complaint about a longstanding misfeature of CREATE TABLE LIKE. Ordinarily, this command doesn't select names for copied indexes, but leaves that to be done at runtime by DefineIndex. But if it's

Re: [HACKERS] Getting rid of pre-assignment of index names in CREATE TABLE LIKE

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Sat, Jul 14, 2012 at 4:02 PM, Tom Lane t...@sss.pgh.pa.us wrote: I suggested that we could dodge the problem by allowing IndexStmt to carry a comment to be attached to the new index, and thereby avoid needing an explicit COMMENT command. Attached

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 12:36 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jul 16, 2012 at 12:08 PM, Tom Lane t...@sss.pgh.pa.us wrote: Uh, that's exactly what's under discussion. Not sending useless fsync requests when fsync is off is just one part

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: In my view, the elephant in the room here is that it's dramatically inefficient for every backend to send an fsync request on every block write. Yeah. This was better before the decision was taken to separate bgwriter from checkpointer; before that,

Re: [HACKERS] Getting rid of pre-assignment of index names in CREATE TABLE LIKE

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 12:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sat, Jul 14, 2012 at 4:02 PM, Tom Lane t...@sss.pgh.pa.us wrote: I suggested that we could dodge the problem by allowing IndexStmt to carry a comment to be attached to the new

Re: [HACKERS] Getting rid of pre-assignment of index names in CREATE TABLE LIKE

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: The problem isn't confined to CREATE TABLE LIKE; it's a widespread design flaw that will likely take years of work to clean up completely. I don't think that's a reason not to commit your change though; it fixes a bug and is an incremental

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 12:27 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: The documentation on MacOS X isn't quite as explicit, but I'd still be astonished if we found any other behavior. TBH, I'd be kind of surprised if this is the only place in our code

Re: [HACKERS] Getting rid of pre-assignment of index names in CREATE TABLE LIKE

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 1:10 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: The problem isn't confined to CREATE TABLE LIKE; it's a widespread design flaw that will likely take years of work to clean up completely. I don't think that's a reason not to commit

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 12:57 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: In my view, the elephant in the room here is that it's dramatically inefficient for every backend to send an fsync request on every block write. Yeah. This was better before the

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Tom Lane
I wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jul 16, 2012 at 11:57 AM, Tom Lane t...@sss.pgh.pa.us wrote: So that brings us back to the question of why this code is supporting fsync requests for local relations in the first place. Couldn't we have it ignore those, and then only

Re: [HACKERS] Synchronous Standalone Master Redoux

2012-07-16 Thread Robert Haas
On Sat, Jul 14, 2012 at 7:54 PM, Josh Berkus j...@agliodbs.com wrote: So, here's the core issue with degraded mode. I'm not mentioning this to block any patch anyone has, but rather out of a desire to see someone address this core problem with some clever idea I've not thought of. The problem

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Unfortunately, there are lots of important operations (like bulk loading, SELECT * FROM bigtable, and VACUUM notverybigtable) that inevitably end up writing out their own dirty buffers. And even when the background writer does write something, it's

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 2:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Jul 16, 2012 at 11:57 AM, Tom Lane t...@sss.pgh.pa.us wrote: So that brings us back to the question of why this code is supporting fsync requests for local relations

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: At any rate, I'm somewhat less convinced that the split was a good idea than I was when we did it, mostly because we haven't really gone anywhere with it subsequently. BTW, while we are on the subject: hasn't this split completely broken the statistics

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 3:18 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: At any rate, I'm somewhat less convinced that the split was a good idea than I was when we did it, mostly because we haven't really gone anywhere with it subsequently. BTW, while we

Re: [HACKERS] [PERFORM] DELETE vs TRUNCATE explanation

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Yes, it seems to have done just that. The comment for ForwardFsyncRequest is a few bricks short of a load too: ... Line 2 seems to have been mechanically changed from background writer to checkpointer, but of course it should still say background

Re: [HACKERS] Closing out the June commitfest

2012-07-16 Thread Josh Berkus
Trim trailing NULL columns https://commitfest.postgresql.org/action/patch_view?id=840 Feh, put my name on this one for a performance test/functionality review. Someone else needs to do the code review though. None of the three above seem to me to be blocking further work, so I don't have a

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Mon, Jul 16, 2012 at 2:47 PM, Tom Lane t...@sss.pgh.pa.us wrote: So I started to do this, and immediately hit a roadblock: although it's clear that we can discard any attempt to fsync a backend-local relation, it's not so clear that we don't need to

[HACKERS] b-tree index search algorithms

2012-07-16 Thread Samuel Vogel
Hello, I'm currently on a university research project if performance could be increased by substituting different inter-node search algorithms instead of the currently used binary search. But I'm having troubles understanding how the general b-tree implementation (nbtree.h) is used to

Re: [HACKERS] Using pg_upgrade on log-shipping standby servers

2012-07-16 Thread Jeff Davis
On Tue, 2012-07-10 at 11:50 -0400, Bruce Momjian wrote: I don't think we can assume that because pg_upgrade was run on the master and standby that they are binary identical, can we? Technically the user file are identical, but the system catalogs and WAL might be different, hence my

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 7:17 PM, Tom Lane t...@sss.pgh.pa.us wrote: The attached patch covers everything discussed in this thread, except for the buggy handling of stats, which I think should be fixed in a separate patch since it's only relevant to 9.2+. With respect to this chunk: + * We do

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: With respect to this chunk: + * We do not need to go through this dance for temp relations, though, because + * we never make WAL entries for temp rels, and so a temp rel poses no threat + * to the health of a regular rel that has taken over

Re: [HACKERS] b-tree index search algorithms

2012-07-16 Thread Tom Lane
Samuel Vogel s...@muel-vogel.de writes: I'm currently on a university research project if performance could be increased by substituting different inter-node search algorithms instead of the currently used binary search. Hm, what have you got in mind exactly? But I'm having troubles

Re: [HACKERS] CompactCheckpointerRequestQueue versus pad bytes

2012-07-16 Thread Robert Haas
On Mon, Jul 16, 2012 at 9:58 PM, Tom Lane t...@sss.pgh.pa.us wrote: BTW, I wonder whether the code that checks for relfilenode conflict when selecting a pg_class or relfilenode OID tries both file naming conventions? If not, should we make it do so? I don't believe it does, nor do I see what

Re: [HACKERS] [v9.3] Row-Level Security

2012-07-16 Thread Robert Haas
On Sun, Jul 15, 2012 at 5:52 AM, Kohei KaiGai kai...@kaigai.gr.jp wrote: The attached patch is a revised version of row-level security feature. I added a feature to invalidate plan cache if user-id was switched between planner and optimizer. It enabled to generate more optimized plan than the

Re: [HACKERS] [v9.3] Row-Level Security

2012-07-16 Thread Kohei KaiGai
2012/7/17 Robert Haas robertmh...@gmail.com: On Sun, Jul 15, 2012 at 5:52 AM, Kohei KaiGai kai...@kaigai.gr.jp wrote: The attached patch is a revised version of row-level security feature. I added a feature to invalidate plan cache if user-id was switched between planner and optimizer. It

Re: [HACKERS] [PATCH] lock_timeout and common SIGALRM framework

2012-07-16 Thread Alvaro Herrera
Excerpts from Tom Lane's message of vie jul 13 18:23:31 -0400 2012: Boszormenyi Zoltan z...@cybertec.at writes: Try SET deadlock_timeout = 0; Actually, when I try that I get ERROR: 0 is outside the valid range for parameter deadlock_timeout (1 .. 2147483647) So I don't see any