Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-01-26 Thread Andres Freund
On 2013-01-25 13:48:50 +0900, Michael Paquier wrote: All the comments are addressed in version 8 attached, except for the hashtable part, which requires some heavy changes. On Thu, Jan 24, 2013 at 3:41 AM, Andres Freund and...@2ndquadrant.comwrote: On 2013-01-15 18:16:59 +0900, Michael

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-01-28 Thread Andres Freund
On 2013-01-27 07:54:43 +0900, Michael Paquier wrote: On Sun, Jan 27, 2013 at 1:52 AM, Andres Freund and...@2ndquadrant.comwrote: On 2013-01-25 13:48:50 +0900, Michael Paquier wrote: As far as I understand that code its purpose is to enforce that all potential users have an up2date

Re: [HACKERS] logical changeset generation v4 - Heikki's thoughts about the patch state

2013-01-28 Thread Andres Freund
On 2013-01-26 16:20:33 -0500, Steve Singer wrote: On 13-01-24 11:15 AM, Steve Singer wrote: On 13-01-24 06:40 AM, Andres Freund wrote: Fair enough. I am also working on a user of this infrastructure but that doesn't help you very much. Steve Singer seemed to make some stabs at writing

Re: [HACKERS] logical changeset generation v4 - Heikki's thoughts about the patch state

2013-01-28 Thread Andres Freund
seconded. I *really* would love to see a more technical review, its hard to see issues after spending that much time in a certain worldview... Thanks! Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql

Re: [HACKERS] logical changeset generation v4

2013-01-28 Thread Andres Freund
On 2013-01-27 12:28:21 -0500, Steve Singer wrote: On 13-01-22 11:30 AM, Andres Freund wrote: Hi, I pushed a new rebased version (the xlogreader commit made it annoying to merge). The main improvements are * way much coherent code internally for intializing logical rep * explicit control

Re: [HACKERS] Re: logical changeset generation v4 - Heikki's thoughts about the patch state

2013-01-28 Thread Andres Freund
On 2013-01-28 11:59:52 +0200, Heikki Linnakangas wrote: On 24.01.2013 00:30, Andres Freund wrote: Also, while the apply side surely isn't benchmarkable without any being submitted, the changeset generation can very well be benchmarked. A very, very adhoc benchmark: -c max_wal_senders=10

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-01-28 Thread Andres Freund
Hi, On 2013-01-28 20:31:48 +0900, Michael Paquier wrote: On Mon, Jan 28, 2013 at 7:39 PM, Andres Freund and...@2ndquadrant.comwrote: On 2013-01-27 07:54:43 +0900, Michael Paquier wrote: I think you're misunderstanding how this part works a bit. We don't acquire locks on the table itself

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2013-01-28 Thread Andres Freund
On 2013-01-28 20:50:21 +0900, Michael Paquier wrote: On Mon, Jan 28, 2013 at 8:44 PM, Andres Freund and...@anarazel.de wrote: Another argument that would be enough for a rejection of this patch by a committer is the problem of invalid toast indexes that cannot be removed up cleanly

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-01-28 Thread Andres Freund
. And if you don't vacuum the whole heap you can't lower relfrozenxid. So changing freeze_min_age doesn't help at all to avoid anti-wraparound vacuums. Am I missing something? Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-01-28 Thread Andres Freund
On 2013-01-28 08:15:29 -0800, Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: On 2013-01-29 00:11:12 +1100, Josh Berkus wrote: So I think we need to sort by age(relfrozenxid) in tables that are over the anti-wraparound limit. Given your code that doesn't seem

Re: [HACKERS] logical changeset generation v4

2013-01-28 Thread Andres Freund
On 2013-01-28 12:23:02 +0100, Andres Freund wrote: On 2013-01-27 12:28:21 -0500, Steve Singer wrote: On 13-01-22 11:30 AM, Andres Freund wrote: Hi, I pushed a new rebased version (the xlogreader commit made it annoying to merge). The main improvements are * way much coherent

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-01-30 Thread Andres Freund
-table vacuums happen more frequently than anti-freeze vacuums anyway the cost of actual anti-freeze vacuums, should they happen because of a too busy autovacuum, aren't a problem in itself. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-01-30 Thread Andres Freund
On 2013-01-30 05:39:29 -0800, Kevin Grittner wrote: Andres Freund and...@2ndquadrant.com wrote: On 2013-01-29 16:09:52 +1100, Josh Berkus wrote: I have to admit, I fail to see why this is a good idea. There isn't much of an efficiency bonus in freezing early (due to hint bits

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-01-30 Thread Andres Freund
On 2013-01-30 14:58:24 +0100, Andres Freund wrote: So reducing vacuum_freeze_min_age not only helps minimize the writes that are needed when autovacuum needs to scan the entire heap, but also decreases the frequency of those full-table scans. But it increases the amount of pages

Re: backend hangs at immediate shutdown (Re: [HACKERS] Back-branch update releases coming in a couple weeks)

2013-01-30 Thread Andres Freund
(). Yeah, it's a known hazard that quickdie() operates like that. What about not translating those? The messages are static and all memory needed by postgres should be pre-allocated. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development

Re: [HACKERS] autovacuum not prioritising for-wraparound tables

2013-01-30 Thread Andres Freund
-- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] backend hangs at immediate shutdown

2013-01-30 Thread Andres Freund
mighty ugly. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] huge tlb support

2012-07-09 Thread Andres Freund
|VM_INSERTPAGE|VM_MIXEDMAP|VM_SAO| \ VM_HUGETLB|VM_SHARED|VM_MAYSHARE) -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

[HACKERS] [PATCH] XLogReader v2

2012-07-19 Thread Andres Freund
an xlogdump utility in core/contrib would be a good idea now that it can be done without a huge amount of code duplication. I plan to check Satoshi-san's version of xlogdump whether I can crib some of the commandline interface and some code from there. Greetings, Andres -- Andres Freund

Re: [HACKERS] postgres 8.4.9: running out of memory (malloc fails) when running a transaction that runs a LOT of selects

2012-07-20 Thread Andres Freund
keys, or anything like that? After youve got that out of memory message, the log should show a list of memory contexts with the amount of memory allocated in each. Could you include that in a mail? Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com

Re: [HACKERS] postgres 8.4.9: running out of memory (malloc fails) when running a transaction that runs a LOT of selects

2012-07-20 Thread Andres Freund
Hi, On Friday, July 20, 2012 11:56:09 AM Benedikt Grundmann wrote: I also noticed just know that all TopMemoryContext's after the first one look significantly different. They contain large PortalMemory sections. Are those related to cursors? Yes. Andres -- Andres Freund

Re: [HACKERS] [PATCH] XLogReader v2

2012-07-23 Thread Andres Freund
by defining an extra .a containing the necessary object files. Not nice, but... Imo it *should* be in src/bin though. Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] embedded list v2

2012-07-23 Thread Andres Freund
On Monday, July 23, 2012 12:55:01 PM Peter Geoghegan wrote: On 5 July 2012 02:49, Peter Geoghegan pe...@2ndquadrant.com wrote: On 28 June 2012 19:20, Andres Freund and...@2ndquadrant.com wrote: 0001-Add-embedded-list-interface.patch Looks good now? I have a few gripes. We

Re: [HACKERS] [PATCH] XLogReader v2

2012-07-23 Thread Andres Freund
On Monday, July 23, 2012 04:17:39 PM Robert Haas wrote: On Mon, Jul 23, 2012 at 3:19 AM, Andres Freund and...@2ndquadrant.com wrote: On Thursday, July 19, 2012 07:18:08 PM Satoshi Nagayasu wrote: I agree with that we need more sophisticated way to share the code between the backend

Re: [HACKERS] [PATCH] XLogReader v2

2012-07-23 Thread Andres Freund
On Monday, July 23, 2012 05:11:20 PM Robert Haas wrote: On Mon, Jul 23, 2012 at 11:04 AM, Andres Freund and...@2ndquadrant.com wrote: On Monday, July 23, 2012 04:17:39 PM Robert Haas wrote: On Mon, Jul 23, 2012 at 3:19 AM, Andres Freund and...@2ndquadrant.com wrote: On Thursday, July

Re: [HACKERS] canceling autovacuum task woes

2012-07-24 Thread Andres Freund
or not, the current logging is wholly inadequate. Very, very, very quick guess: The relation extension lock? Thoughts? Anybody else have this problem? I have seen spuriously high occurances of that message before, but I never really investigated it. Andres -- Andres Freund http://www.2ndQuadrant.com

Re: [HACKERS] New WAL code dumps core trivially on replay of bad data

2012-08-20 Thread Andres Freund
--- it should not have anything to do with malloc'ing new space. But anyway, this is not what happened in my example. If the memory is big enough (128kb) it will be mmap'ed into place. In that case overcommiting applies before the pages have been brought in. Greetings, Andres -- Andres

Re: [HACKERS] huge tlb support

2012-08-21 Thread Andres Freund
and move on. He is talking about transparent hugepages not hugepages afaics. Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] huge tlb support

2012-08-21 Thread Andres Freund
On Tuesday, August 21, 2012 05:56:58 PM Robert Haas wrote: On Tue, Aug 21, 2012 at 11:31 AM, Andres Freund and...@2ndquadrant.com wrote: On Tuesday, August 21, 2012 05:30:28 PM Robert Haas wrote: On Thu, Aug 16, 2012 at 10:53 PM, David Gould da...@sonic.net wrote: A warning, on RHEL 6.1

Re: [HACKERS] plperl crash with Debian 6 (64 bit), pl/perlu, libwww and https

2012-08-23 Thread Andres Freund
}' 01301880 000180006207 Note the 8'th bit being unset in SigCgt and set in SigIgn. Thats SIGFPE... Not sure how relevant this really is, but it could cause errors to be ignored... Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-23 Thread Andres Freund
On Thursday, August 23, 2012 12:17:22 PM Andres Freund wrote: Hi, While debugging an instance of this bug I noticed that plperlu always removes the SIGFPE handler and sets it to ignore: andres@awork2:~$ psql -p 5435 -U postgres -h /var/run/postgresql test Timing is on. psql (9.1devel

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-23 Thread Andres Freund
On Friday, August 24, 2012 06:55:04 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On Thursday, August 23, 2012 12:17:22 PM Andres Freund wrote: While debugging an instance of this bug I noticed that plperlu always removes the SIGFPE handler and sets it to ignore

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-23 Thread Andres Freund
On Friday, August 24, 2012 06:55:04 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On Thursday, August 23, 2012 12:17:22 PM Andres Freund wrote: While debugging an instance of this bug I noticed that plperlu always removes the SIGFPE handler and sets it to ignore

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-23 Thread Andres Freund
On Friday, August 24, 2012 07:19:42 AM Andres Freund wrote: On Friday, August 24, 2012 06:55:04 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On Thursday, August 23, 2012 12:17:22 PM Andres Freund wrote: While debugging an instance of this bug I noticed that plperlu

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-23 Thread Andres Freund
On Friday, August 24, 2012 07:33:01 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: Um ... how exactly can that happen, if the signal is now ignored? My man 2 signal tells me: According to POSIX, the behavior of a process is undefined after it ignores a SIGFPE, SIGILL

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-24 Thread Andres Freund
On Friday, August 24, 2012 04:53:36 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: ./pod/perl581delta.pod: At startup Perl blocks the SIGFPE signal away since there isn't much Perl can do about it. Previously this blocking was in effect also for programs executed from

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-24 Thread Andres Freund
On Friday, August 24, 2012 05:09:18 PM Andrew Dunstan wrote: On 08/24/2012 10:58 AM, Andres Freund wrote: On Friday, August 24, 2012 04:53:36 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: ./pod/perl581delta.pod: At startup Perl blocks the SIGFPE signal away since

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-24 Thread Andres Freund
On Friday, August 24, 2012 04:53:36 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: ./pod/perl581delta.pod: At startup Perl blocks the SIGFPE signal away since there isn't much Perl can do about it. Previously this blocking was in effect also for programs executed from

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-24 Thread Andres Freund
On Saturday, August 25, 2012 12:15:00 AM Alex Hunsaker wrote: On Fri, Aug 24, 2012 at 4:10 PM, Andres Freund and...@2ndquadrant.com wrote: We probably should workaround that bug anyway given that its a pretty trivial DOS using only a trusted language and it will take quite some time

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-24 Thread Andres Freund
On Saturday, August 25, 2012 06:38:09 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: Doing a pqsignal(SIGFPE, FloatExceptionHandler) after PERL_SYS_INIT3 seems to work. Is that acceptable? Surely that's breaking perl's expectations, to more or less the same degree they're

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-08-26 Thread Andres Freund
On Saturday, August 25, 2012 06:38:09 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: Doing a pqsignal(SIGFPE, FloatExceptionHandler) after PERL_SYS_INIT3 seems to work. Is that acceptable? Surely that's breaking perl's expectations, to more or less the same degree they're

Re: [HACKERS] Use of systable_beginscan_ordered in event trigger patch

2012-08-28 Thread Andres Freund
it again. +1 Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] splitting htup.h

2012-08-29 Thread Andres Freund
, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] rows modified in current transaction

2012-08-30 Thread Andres Freund
don't think we currently expose that. txid_current_snapshot(), txis_visible_in_snapshot() may work. Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] splitting *_desc routines

2012-08-30 Thread Andres Freund
to pile more and more utilities into the main postgres binary though. Note the ugliness some the testing tools in src/backend go through just to link to a few files... Yuck. Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training

Re: [HACKERS] rows modified in current transaction

2012-08-30 Thread Andres Freund
On Thursday, August 30, 2012 06:09:43 PM Andres Freund wrote: On Thursday, August 30, 2012 06:06:59 PM Robert Haas wrote: On Thu, Aug 30, 2012 at 10:36 AM, Miroslav Šimulčík simulcik.m...@gmail.com wrote: is there any way to check if row have already been modified by the current

Re: [HACKERS] HEAD crashes on windows when doing VACUUM ANALYZE

2012-08-30 Thread Andres Freund
lead to a different error, but ... Postgres does install a handler for it. What happens if you run SELECT -2147483648/-1;? Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2012-09-03 Thread Andres Freund
into PQconnectdb(). PQServer PQstartServer(const char **keywords, const char **values); or whatever seems to be more future proof especially considering the possibility that this will grow into something more featureful. Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2012-09-03 Thread Andres Freund
On Monday, September 03, 2012 10:54:23 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On Monday, September 03, 2012 10:23:52 PM Tom Lane wrote: I'm reluctantly coming to the conclusion that we can't pass these parameters through the regular libpq connection string mechanism

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2012-09-03 Thread Andres Freund
On Tuesday, September 04, 2012 06:20:59 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I can see why that would be nice, but is it really realistic? Don't we expect some more diligence in applications using this against letting such a child continue to run after ctrl-c

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-09-05 Thread Andres Freund
On Sunday, August 26, 2012 06:10:02 PM Andres Freund wrote: On Saturday, August 25, 2012 06:38:09 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: Doing a pqsignal(SIGFPE, FloatExceptionHandler) after PERL_SYS_INIT3 seems to work. Is that acceptable? Surely that's

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2012-09-05 Thread Andres Freund
On Tuesday, September 04, 2012 12:11:28 PM Amit Kapila wrote: On Tuesday, September 04, 2012 11:00 AM Andres Freund wrote: On Tuesday, September 04, 2012 06:20:59 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I can see why that would be nice, but is it really realistic

Re: [HACKERS] plperl sigfpe reset can crash the server

2012-09-05 Thread Andres Freund
On Wednesday, September 05, 2012 07:15:52 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On Sunday, August 26, 2012 06:10:02 PM Andres Freund wrote: On Saturday, August 25, 2012 06:38:09 AM Tom Lane wrote: Surely that's breaking perl's expectations, to more or less the same

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2012-09-07 Thread Andres Freund
happened. Something like DoBackgroundWork(int forThisManySeconds). Of course, for that to work, we'd need to have resumable vacuum. I like the idea of keeping everything single threaded. To me this path seems to be the best way to never get the feature at all... Andres -- Andres Freund

Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2012-09-09 Thread Andres Freund
have to wait. Yep, thats the case I was talking about upthread ;) On Monday, September 03, 2012 11:04:15 PM Andres Freund wrote: Don't we expect some more diligence in applications using this against letting such a child continue to run after ctrl-c/SIGTERMing e.g. pg_dump in comparison

Re: [HACKERS] embedded list v2

2012-09-09 Thread Andres Freund
that the list implementation works. Thanks! Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] [PATCH] XLogReader v2

2012-09-09 Thread Andres Freund
started doing anything about it? I can redo a version but before we agree on the strategy for logging error handling the only thing that would change is the cuddly braces and the IDENTIFCATION tags... Thanks! Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development

Re: [HACKERS] [PATCH] XLogReader v2

2012-09-09 Thread Andres Freund
On Sunday, September 09, 2012 08:40:38 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On Tuesday, September 04, 2012 09:33:54 PM Alvaro Herrera wrote: * There are way too many #ifdef VERBOSE_DEBUG stuff for my taste. It might look better if you had macros

[HACKERS] [RFC][PATCH] wal decoding, attempt #2

2012-09-14 Thread Andres Freund
or even the general direction of the implementation? Questions? Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

[HACKERS] [PATCH 7/8] Make InvalidateSystemCaches public

2012-09-14 Thread Andres Freund
Pieces of this are in commit: make relfilenode lookup (tablespace, relfilenode --- src/backend/utils/cache/inval.c | 2 +- src/include/utils/inval.h | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c index

[HACKERS] [PATCH 4/8] add simple xlogdump tool

2012-09-14 Thread Andres Freund
--- src/bin/Makefile| 2 +- src/bin/xlogdump/Makefile | 25 src/bin/xlogdump/xlogdump.c | 334 3 files changed, 360 insertions(+), 1 deletion(-) create mode 100644 src/bin/xlogdump/Makefile create mode 100644

[HACKERS] [PATCH 2/8] Add minimal binary heap implementation

2012-09-14 Thread Andres Freund
This is basically untested. --- src/backend/lib/Makefile | 2 +- src/backend/lib/simpleheap.c | 255 +++ src/include/lib/simpleheap.h | 91 +++ 3 files changed, 347 insertions(+), 1 deletion(-) create mode 100644

Re: [HACKERS] embedded list v2

2012-09-14 Thread Andres Freund
dlist_get_head() or such anyway... I wondered whether the solution for that would be to remove the variants that check for an empty list (except an Assert). Not a full review, just some things that struck me in a quick scan... Thanks! Andres -- Andres Freund http://www.2ndQuadrant.com

[HACKERS] [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

2012-09-14 Thread Andres Freund
Features: - streaming reading/writing - filtering - reassembly of records Reusing the ReadRecord infrastructure in situations where the code that wants to do so is not tightly integrated into xlog.c is rather hard and would require changes to rather integral parts of the recovery code which

[HACKERS] [PATCH 5/8] Add a new syscache to fetch a pg_class entry via (reltablespace, relfilenode)

2012-09-14 Thread Andres Freund
This patch is problematic because formally indexes used by syscaches needs to be unique, this one is not though because of 0/InvalidOids relfilenode entries for nailed/shared catalog entries. Those values cannot be sensibly queries from the catalog anyway though (the relmapper infrastructure

[HACKERS] [PATCH 6/8] Log enough data into the wal to reconstruct logical changes from it if wal_level=logical

2012-09-14 Thread Andres Freund
This adds a new wal_level value 'logical' Missing cases: - heap_multi_insert - primary key changes for updates - no primary key - LOG_NEWPAGE --- src/backend/access/heap/heapam.c| 135 +--- src/backend/access/transam/xlog.c | 1 +

[HACKERS] [PATCH 1/8] Add embedded list interface (header only)

2012-09-14 Thread Andres Freund
Adds a single and a double linked list which can easily embedded into other datastructures and can be used without any additional allocations. Problematic: It requires USE_INLINE to be used. It could be remade to fallback to to externally defined functions if that is not available but that

[HACKERS] git tree

2012-09-14 Thread Andres Freund
://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/xlog- decoding-rebasing-cf2 That branch will be regularly rebased to a new master,fixes/new features, and a pgindent run over the new files... Greetings, Andres -- Andres Freund http://www

Re: [HACKERS] embedded list v2

2012-09-15 Thread Andres Freund
Hi Tom, On Saturday, September 15, 2012 07:32:45 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On Friday, September 14, 2012 10:48:35 PM Tom Lane wrote: Instead let's provide a macro for an empty list value, so that you can do something like static ilist_d_head

Re: [HACKERS] git tree

2012-09-15 Thread Andres Freund
On Saturday, September 15, 2012 03:14:32 AM Andres Freund wrote: That branch will be regularly rebased to a new master,fixes/new features, and a pgindent run over the new files... I fixed up the formatting of the new stuff (xlogreader, ilist are submitted separately, no point in doing anything

Re: [HACKERS] [COMMITTERS] pgsql: Properly set relpersistence for fake relcache entries.

2012-09-15 Thread Andres Freund
these issues. While CreateCheckPoint() itself treats that kind of checkpoint similarly to a shutdown checkpoint it doesn't pass that similarity to BufferSync (via CheckPointGuts-CheckPointBuffers). I hope I missed something ... Greetings, Andres -- Andres Freund http://www

Re: [HACKERS] embedded list v2

2012-09-16 Thread Andres Freund
On Saturday, September 15, 2012 07:21:44 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On Saturday, September 15, 2012 07:32:45 AM Tom Lane wrote: Well, actually, that just brings us to the main point which is: I do not believe that circular links are a good design choice

Re: [HACKERS] embedded list v2

2012-09-16 Thread Andres Freund
Hi, On Sunday, September 16, 2012 04:23:14 PM Andres Freund wrote: What do you think about something like: typedef struct dlist_iter { /* * Use a union with equivalent storage as dlist_node to make it possible to * initialize the struct inside a macro without multiple

[HACKERS] Add pg_relation_by_filenode(reltbspc, filenode) admin function

2012-09-16 Thread Andres Freund
Now that I proposed a new syscache upthread its easily possible to provide pg_relation_by_filenode which I wished for multiple times in the past when looking at filesystem activity and wondering which table does what. You can sortof get the same result via SELECT oid FROM ( SELECT oid,

[HACKERS] [PATCH 1/2] Add a new relmapper.c function RelationMapFilenodeToOid that acts as a reverse of RelationMapOidToFilenode

2012-09-16 Thread Andres Freund
--- src/backend/utils/cache/relmapper.c | 53 + src/include/utils/relmapper.h | 2 ++ 2 files changed, 55 insertions(+) diff --git a/src/backend/utils/cache/relmapper.c b/src/backend/utils/cache/relmapper.c index 6f21495..771f34d 100644 ---

[HACKERS] [PATCH 2/2] Add a new function pg_relation_by_filenode to lookup up a relation given the tablespace and the filenode OIDs

2012-09-16 Thread Andres Freund
This requires the previously added RELFILENODE syscache. --- doc/src/sgml/func.sgml | 23 - src/backend/utils/adt/dbsize.c | 78 ++ src/include/catalog/pg_proc.h | 2 ++ src/include/utils/builtins.h | 1 + 4 files changed, 103

Re: [HACKERS] [PATCH 2/2] Add a new function pg_relation_by_filenode to lookup up a relation given the tablespace and the filenode OIDs

2012-09-16 Thread Andres Freund
On Monday, September 17, 2012 12:35:32 AM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: This requires the previously added RELFILENODE syscache. [ raised eyebrow... ] There's a RELFILENODE syscache? I don't see one, and I doubt it would work given that the contents

Re: [HACKERS] Re: [COMMITTERS] pgsql: Properly set relpersistence for fake relcache entries.

2012-09-17 Thread Andres Freund
the possibility of being really bad we were pretty lucky... Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

2012-09-17 Thread Andres Freund
Hi Heikki, On Monday, September 17, 2012 09:40:17 AM Heikki Linnakangas wrote: On 15.09.2012 03:39, Andres Freund wrote: Features: - streaming reading/writing - filtering - reassembly of records Reusing the ReadRecord infrastructure in situations where the code that wants to do so

Re: [HACKERS] [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

2012-09-17 Thread Andres Freund
On Monday, September 17, 2012 10:30:35 AM Heikki Linnakangas wrote: On 17.09.2012 11:12, Andres Freund wrote: On Monday, September 17, 2012 09:40:17 AM Heikki Linnakangas wrote: On 15.09.2012 03:39, Andres Freund wrote: 2. We should focus on reading WAL, I don't see the point of mixing WAL

Re: [HACKERS] [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

2012-09-17 Thread Andres Freund
On Monday, September 17, 2012 11:07:28 AM Andres Freund wrote: On Monday, September 17, 2012 10:30:35 AM Heikki Linnakangas wrote: On 17.09.2012 11:12, Andres Freund wrote: On Monday, September 17, 2012 09:40:17 AM Heikki Linnakangas wrote: If you don't want the capability to forward

Re: [HACKERS] [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

2012-09-17 Thread Andres Freund
On Monday, September 17, 2012 12:55:47 PM Heikki Linnakangas wrote: On 17.09.2012 13:01, Andres Freund wrote: On Monday, September 17, 2012 11:07:28 AM Andres Freund wrote: On Monday, September 17, 2012 10:30:35 AM Heikki Linnakangas wrote: On 17.09.2012 11:12, Andres Freund wrote

Re: [HACKERS] [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

2012-09-17 Thread Andres Freund
On Monday, September 17, 2012 01:50:33 PM Heikki Linnakangas wrote: On 17.09.2012 14:42, Andres Freund wrote: On Monday, September 17, 2012 12:55:47 PM Heikki Linnakangas wrote: On 17.09.2012 13:01, Andres Freund wrote: On Monday, September 17, 2012 11:07:28 AM Andres Freund wrote

Re: [HACKERS] [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

2012-09-17 Thread Andres Freund
On Monday, September 17, 2012 12:52:32 PM Heikki Linnakangas wrote: On 17.09.2012 12:07, Andres Freund wrote: On Monday, September 17, 2012 10:30:35 AM Heikki Linnakangas wrote: The user of the facility doesn't need to be aware of record boundaries, that's the responsibility of the facility

Re: [HACKERS] [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

2012-09-17 Thread Andres Freund
On Monday, September 17, 2012 04:08:01 PM Tom Lane wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: On 17.09.2012 13:01, Andres Freund wrote: It seems we would need one additional callback for both approaches like: -error(severity, format, ...) For both to avoid having to draw

Re: [HACKERS] [PATCH 3/8] Add support for a generic wal reading facility dubbed XLogReader

2012-09-17 Thread Andres Freund
On Monday, September 17, 2012 04:18:28 PM Heikki Linnakangas wrote: On 17.09.2012 17:08, Tom Lane wrote: Heikki Linnakangashlinnakan...@vmware.com writes: On 17.09.2012 13:01, Andres Freund wrote: It seems we would need one additional callback for both approaches like: -error(severity

Re: [HACKERS] [COMMITTERS] pgsql: Fix bufmgr so CHECKPOINT_END_OF_RECOVERY behaves as a shutdown c

2012-09-17 Thread Andres Freund
of recovery - a shutdown checkpoint WAL record is written but the buffer manager had been altered to treat end of recovery as a normal checkpoint. This bug exacerbates the bufmgr relpersistence bug. Bug spotted by Andres Freund, patch by me. I am confused by this patch. It seems to me

Re: [HACKERS] [COMMITTERS] pgsql: Fix bufmgr so CHECKPOINT_END_OF_RECOVERY behaves as a shutdown c

2012-09-19 Thread Andres Freund
if such a reason existed, there wouldn't be anything to read, because the backing files are unlinked before WAL replay begins. Back then I thought that resetting the relation by copying the init fork might use the buffer cache. It doesn't atm... Andres -- Andres Freund http

Re: [HACKERS] XLogInsert scaling, revisited

2012-09-20 Thread Andres Freund
has? Or do we have to use a GUC for that? Several platforms support sysconf(_SC_NPROCESSORS_CONF) although after a quick look it doesn't seem to be standardized. A guc initialized to that or falling back to 4 or so? Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] [COMMITTERS] pgsql: Properly set relpersistence for fake relcache entries.

2012-09-20 Thread Andres Freund
On Monday, September 17, 2012 03:58:37 PM Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: Btw, I played with this some more on Saturday and I think, while definitely a bad bug, the actual consequences aren't as bad as at least I initially feared. Fake relcache entries

Re: [HACKERS] [COMMITTERS] pgsql: Properly set relpersistence for fake relcache entries.

2012-09-21 Thread Andres Freund
On Friday, September 21, 2012 03:30:31 PM Marko Tiikkaja wrote: On 9/20/12 11:55 PM, Andres Freund wrote: On Monday, September 17, 2012 03:58:37 PM Tom Lane wrote: OK, that explains why we've not seen a blizzard of trouble reports. Still seems like a good idea to fix it ASAP, though

Re: [HACKERS] External Replication

2012-09-21 Thread Andres Freund
=postgresql.git;a=commitdiff;h=3855968f328918b6cd1401dd11d109d471a54d40 and 3a0e4d36ebd7f477822d5bae41ba121a40d22ccc Look into earlier discussions around event/command triggers why putting stuff plainly into ProcessUtility is not all that helpful... Greetings, Andres -- Andres Freund

Re: [HACKERS] External Replication

2012-09-21 Thread Andres Freund
triggers started recently. - Are these command triggers currently in the 9.2.0 code base or is it in a alpha 9.2.1xxx? Its not in 9.2 and will only be in 9.3 Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services

Re: [HACKERS] [RFC][PATCH] wal decoding, attempt #2 - Design Documents

2012-09-22 Thread Andres Freund
feedback! Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services //-*- mode: adoc -*- = High Level Design for Logical Replication in Postgres = :copyright: PostgreSQL Global Development Group 2012 :author: Andres

[HACKERS] DROP INDEX CONCURRENTLY is not really concurrency safe leaves around undroppable indexes

2012-09-24 Thread Andres Freund
-- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] DROP INDEX CONCURRENTLY is not really concurrency safe leaves around undroppable indexes

2012-09-24 Thread Andres Freund
On Monday, September 24, 2012 01:27:54 PM Andres Freund wrote: Problem 2: undroppable indexes: Session 1: CREATE TABLE test_drop_concurrently(id serial primary key, data int); CREATE INDEX test_drop_concurrently_data ON test_drop_concurrently(data); BEGIN; EXPLAIN ANALYZE SELECT * FROM

Re: [HACKERS] pg_reorg in core?

2012-09-24 Thread Andres Freund
while you also have longrunning queries... 2. no support for concurrent on system tables (not easy for shared catalogs) 3. no support for the indexes of exlusion constraints (not hard I think) Greetings, Andres -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] DROP INDEX CONCURRENTLY is not really concurrency safe leaves around undroppable indexes

2012-09-24 Thread Andres Freund
Hi, On Monday, September 24, 2012 01:27:54 PM Andres Freund wrote: Hi, Problem 1: concurrency: Testcase: Session 1: CREATE TABLE test_drop_concurrently(id serial primary key, data int); INSERT INTO test_drop_concurrently(data) SELECT * FROM generate_series(1, 10); CREATE INDEX

Re: [HACKERS] DROP INDEX CONCURRENTLY is not really concurrency safe leaves around undroppable indexes

2012-09-24 Thread Andres Freund
On Monday, September 24, 2012 01:37:59 PM Andres Freund wrote: On Monday, September 24, 2012 01:27:54 PM Andres Freund wrote: Problem 2: undroppable indexes: Session 1: CREATE TABLE test_drop_concurrently(id serial primary key, data int); CREATE INDEX test_drop_concurrently_data

Re: [HACKERS] pg_reorg in core?

2012-09-25 Thread Andres Freund
On Tuesday, September 25, 2012 04:37:05 AM Michael Paquier wrote: On Tue, Sep 25, 2012 at 8:13 AM, Andres Freund and...@2ndquadrant.comwrote: On Tuesday, September 25, 2012 12:55:35 AM Josh Berkus wrote: On 9/24/12 3:43 PM, Simon Riggs wrote: On 24 September 2012 17:36, Josh Berkus j

<    4   5   6   7   8   9   10   11   12   13   >