[HACKERS] pg_dump bug in 9.4beta2 and HEAD

2014-08-13 Thread Joachim Wieland
I'm seeing an assertion failure with "pg_dump -c --if-exists" which is not ready to handle BLOBs it seems: pg_dump: pg_backup_archiver.c:472: RestoreArchive: Assertion `mark != ((void *)0)' failed. To reproduce: $ createdb test $ pg_dump -c --if-exists test (works, dumps empty database) $ psql

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Joachim Wieland
On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner < ste...@kaltenbrunner.cc> wrote: > > What might make sense is something like pg_dump_restore which would have > > no intermediate storage at all, just pump the data etc from one source > > to another in parallel. But I pity the poor guy who ha

Re: [HACKERS] Optimizing pglz compressor

2013-03-06 Thread Joachim Wieland
On Tue, Mar 5, 2013 at 8:32 AM, Heikki Linnakangas wrote: > With these tweaks, I was able to make pglz-based delta encoding perform > roughly as well as Amit's patch. Out of curiosity, do we know how pglz compares with other algorithms, e.g. lz4 ? -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] Materialized view assertion failure in HEAD

2013-03-05 Thread Joachim Wieland
On Tue, Mar 5, 2013 at 9:11 AM, Kevin Grittner wrote: > Will investigate. > You don't have default_with_oids = on, do you? No, this was a default install with a default postgresql.conf -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: h

[HACKERS] Materialized view assertion failure in HEAD

2013-03-05 Thread Joachim Wieland
I'm getting an assertion failure in HEAD with materialized views, see below for backtrace. To reproduce, just run make installcheck, dump the regression database and then restore it, the server crashes during restore. (gdb) bt #0 0x7f283a366425 in raise () from /lib/x86_64-linux-gnu/libc.so.

Re: [HACKERS] posix_fadvise missing in the walsender

2013-02-20 Thread Joachim Wieland
On Wed, Feb 20, 2013 at 4:54 PM, Robert Haas wrote: > On Tue, Feb 19, 2013 at 5:48 PM, Simon Riggs wrote: >> I agree with Merlin and Joachim - if we have the call in one place, we >> should have it in both. > > We might want to assess whether we even want to have it one place. > I've seen cases w

Re: [HACKERS] posix_fadvise missing in the walsender

2013-02-19 Thread Joachim Wieland
On Tue, Feb 19, 2013 at 5:48 PM, Simon Riggs wrote: In access/transam/xlog.c we give the OS buffer caching a hint that we won't need a WAL file any time soon with posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); > > I agree with Merlin and Joachim - if we have t

[HACKERS] posix_fadvise missing in the walsender

2013-02-17 Thread Joachim Wieland
In access/transam/xlog.c we give the OS buffer caching a hint that we won't need a WAL file any time soon with posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); before closing the WAL file, but only if we don't have walsenders. That's reasonable because the walsender will reopen that sam

Re: [HACKERS] Logical decoding & exported base snapshot

2012-12-11 Thread Joachim Wieland
On Tue, Dec 11, 2012 at 6:52 PM, Andres Freund wrote: > One problem I see is that while exporting a snapshot solves the > visibility issues of the table's contents it does not protect against > schema changes. I am not sure whether thats a problem. > > If somebody runs a CLUSTER or something like

Re: [HACKERS] parallel pg_dump

2012-12-08 Thread Joachim Wieland
On Sat, Dec 8, 2012 at 3:05 PM, Bruce Momjian wrote: > On Sat, Dec 8, 2012 at 11:13:30AM -0500, Andrew Dunstan wrote: > > I am working on it when I get a chance, but keep getting hammered. > > I'd love somebody else to review it too. > > FYI, I will be posting pg_upgrade performance numbers usin

Re: [HACKERS] [PATCH] pg_dump: Sort overloaded functions in deterministic order

2012-10-17 Thread Joachim Wieland
On Wed, Oct 17, 2012 at 5:43 PM, Alvaro Herrera wrote: > (I tested the new pg_dump with 8.2 and HEAD and also verified it passes > pg_upgrade's "make check". I didn't test with other server versions.) I also tested against 8.3 and 8.4 since 8.4 is the version that introduced pg_get_function_iden

Re: [HACKERS] Add FET to Default and Europe.txt

2012-10-08 Thread Joachim Wieland
On Mon, Oct 8, 2012 at 4:00 PM, Tom Lane wrote: > We can't just refuse to deal with this ambiguity. We have to have some > very-low-pain way to install settings that will please those large > fractions of our user base. Moreover, if that very-low-pain way isn't > the exact same way it's been don

[HACKERS] Review for pg_dump: Sort overloaded functions in deterministic order

2012-09-25 Thread Joachim Wieland
Patch looks good, all concerns that had been raised previously have been addressed in this version of the patch. The only thing that IMO needs to change is a stylistic issue: if (fout->remoteVersion >= 80200) { [...] (fout->remoteVersion >= 80400) ? "pg_catalog.pg_get_function_identity_ar

Re: [HACKERS] "could not open relation with OID" errors after promoting the standby to master

2012-05-24 Thread Joachim Wieland
On Tue, May 22, 2012 at 9:50 AM, Robert Haas wrote: > Hmm.  I think that if you do it this way, the minimum recovery point > won't be respected, which could leave you with a corrupted database. > Now, if all the WAL files that you need are present in pg_xlog anyway, > then they ought to get replay

Re: [HACKERS] "could not open relation with OID" errors after promoting the standby to master

2012-05-17 Thread Joachim Wieland
On Wed, May 16, 2012 at 11:38 PM, Alvaro Herrera wrote: > Well, that is not surprising in itself -- InitTempTableNamespace calls > RemoveTempRelations to cleanup from a possibly crashed previous backend > with the same ID.  So that part of the backtrace looks normal to me > (unless there is someth

[HACKERS] "could not open relation with OID" errors after promoting the standby to master

2012-05-15 Thread Joachim Wieland
I've switched servers yesterday night and the previous slave is now the master. This is 9.0.6 (originally) / 9.0.7 (now) on Linux. Now I'm seeing a bunch of ERROR: could not open relation with OID 1990987633 STATEMENT: create temp table seen_files (fileid integer) Interestingly enough, 90% of

Re: [HACKERS] patch for parallel pg_dump

2012-04-04 Thread Joachim Wieland
On Tue, Apr 3, 2012 at 11:04 AM, Robert Haas wrote: > OK, but it seems like a pretty fragile assumption that none of the > workers will ever manage to emit any other error messages.  We don't > rely on this kind of assumption in the backend, which is a lot > better-structured and less spaghetti-li

Re: [HACKERS] parallel pg_dump

2012-04-04 Thread Joachim Wieland
On Tue, Apr 3, 2012 at 9:26 AM, Andrew Dunstan wrote: > First, either the creation of the destination directory needs to be delayed > until all the sanity checks have passed and we're sure we're actually going > to write something there, or it needs to be removed if we error exit before > anything

Re: [HACKERS] patch for parallel pg_dump

2012-04-03 Thread Joachim Wieland
On Tue, Apr 3, 2012 at 9:34 AM, Robert Haas wrote: > On Sun, Apr 1, 2012 at 12:35 PM, Joachim Wieland wrote: >> Unfortunately this is not really the case. What is being moved out of >> pg_backup_archiver.c and into parallel.c is either the shutdown logic >> that has been ap

Re: [HACKERS] patch for parallel pg_dump

2012-03-28 Thread Joachim Wieland
On Wed, Mar 28, 2012 at 1:46 PM, Robert Haas wrote: > I'm wondering if we really need this much complexity around shutting > down workers.  I'm not sure I understand why we need both a "hard" and > a "soft" method of shutting them down.  At least on non-Windows > systems, it seems like it would be

Re: [HACKERS] patch for parallel pg_dump

2012-03-28 Thread Joachim Wieland
On Thu, Mar 29, 2012 at 2:46 AM, Andrew Dunstan wrote: > But your patch hasn't got rid of them, and so it's declared twice. There is > no pgpipe.h, BTW, it's declared in port.h. If VC2005 doesn't complain about > the double declaration then that's a bug in the compiler, IMNSHO. Doesn't it > even i

Re: [HACKERS] patch for parallel pg_dump

2012-03-28 Thread Joachim Wieland
On Wed, Mar 28, 2012 at 5:19 PM, Andrew Dunstan wrote: > First hurdle: It doesn't build under Windows/mingw-w64: > >   parallel.c:40:12: error: static declaration of 'pgpipe' follows >   non-static declaration Strange, I'm not seeing this but I'm building with VC2005. What happens is that you're

Re: [HACKERS] patch for parallel pg_dump

2012-03-25 Thread Joachim Wieland
On Fri, Mar 23, 2012 at 11:11 AM, Alvaro Herrera wrote: > Are you going to provide a rebased version? Rebased version attached, this patch also includes Robert's earlier suggestions. parallel_pg_dump_5.diff.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] patch for parallel pg_dump

2012-03-23 Thread Joachim Wieland
On Fri, Mar 23, 2012 at 11:11 AM, Alvaro Herrera wrote: > Are you going to provide a rebased version? Yes, working on that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch for parallel pg_dump

2012-03-17 Thread Joachim Wieland
On Fri, Mar 16, 2012 at 12:06 AM, Robert Haas wrote: >> Good. The only exit handler I've seen so far is >> pgdump_cleanup_at_exit. If there's no other one, is it okay to remove >> all of this stacking functionality (see on_exit_nicely_index / >> MAX_ON_EXIT_NICELY) from dumputils.c and just define

[HACKERS] double free in current HEAD's pg_dump

2012-03-17 Thread Joachim Wieland
There's a double free in the current HEAD's pg_dump. Fix attached. diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 2b0a5ff..57a6ccb 100644 *** a/src/bin/pg_dump/pg_dump.c --- b/src/bin/pg_dump/pg_dump.c *** dumpBlobs(Archive *fout, void *arg) *** 2372,2379

Re: [HACKERS] patch for parallel pg_dump

2012-03-14 Thread Joachim Wieland
On Wed, Mar 14, 2012 at 4:39 PM, Andrew Dunstan wrote: > I've just started looking at the patch, and I'm curious to know why it > didn't follow the pattern of parallel pg_restore which created a new worker > for each table rather than passing messages to looping worker threads as > this appears to

Re: [HACKERS] patch for parallel pg_dump

2012-03-14 Thread Joachim Wieland
On Wed, Mar 14, 2012 at 2:02 PM, Robert Haas wrote: > I think we should get rid of die_horribly(), and instead have arrange > to always clean up AH via an on_exit_nicely hook. Good. The only exit handler I've seen so far is pgdump_cleanup_at_exit. If there's no other one, is it okay to remove all

Re: [HACKERS] patch for parallel pg_dump

2012-03-13 Thread Joachim Wieland
On Tue, Mar 13, 2012 at 1:53 PM, Robert Haas wrote: > What I mean is that the function ArchiveEntry() is defined in > pg_backup_archiver.c, and it takes an argument called relpages, and > the string "relpages" does not appear anywhere else in that file. Uhm, that's kinda concerning, isn't it... f

Re: [HACKERS] patch for parallel pg_dump

2012-02-23 Thread Joachim Wieland
On Thu, Feb 16, 2012 at 1:29 PM, Robert Haas wrote: > Can you provide an updated patch? Robert, updated patch is attached. parallel_pg_dump_2.diff.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] patch for parallel pg_dump

2012-02-08 Thread Joachim Wieland
On Wed, Feb 8, 2012 at 1:21 PM, Robert Haas wrote: >> In my patch I dealt with exactly the same problem for the error >> handler by creating a separate function that has a static variable (a >> pointer to the ParallelState). The value is set and retrieved through >> the same function, so yes, it's

Re: [HACKERS] patch for parallel pg_dump

2012-02-07 Thread Joachim Wieland
On Tue, Feb 7, 2012 at 4:59 PM, Robert Haas wrote: > It turns out that (as you anticipated) there are some problems with my > previous proposal. I assume you're talking to Tom, as my powers of anticipation are actually quite limited... :-) > This is not > quite enough to get rid of g_conn, but i

Re: [HACKERS] patch for parallel pg_dump

2012-02-03 Thread Joachim Wieland
On Fri, Feb 3, 2012 at 7:52 AM, Robert Haas wrote: > If you're OK with that much I'll go do it. Sure, go ahead! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch for parallel pg_dump

2012-02-02 Thread Joachim Wieland
On Wed, Feb 1, 2012 at 12:24 PM, Robert Haas wrote: > I think we're more-or-less proposing to rename "Archive" to > "Connection", aren't we? > > And then ArchiveHandle can store all the things that aren't related to > a specific connection. How about something like that: (Hopefully you'll come up

Re: [HACKERS] patch for parallel pg_dump

2012-01-31 Thread Joachim Wieland
On Tue, Jan 31, 2012 at 9:05 AM, Robert Haas wrote: > And just for added fun and excitement, they all have inconsistent > naming conventions and inadequate documentation. > > I think if we need more refactoring in order to support multiple > database connections, we should go do that refactoring.

Re: [HACKERS] patch for parallel pg_dump

2012-01-30 Thread Joachim Wieland
On Mon, Jan 30, 2012 at 12:20 PM, Robert Haas wrote: > But the immediate problem is that pg_dump.c is heavily reliant on > global variables, which isn't going to fly if we want this code to use > threads on Windows (or anywhere else).  It's also bad style. Technically, since most of pg_dump.c dum

Re: [HACKERS] patch for parallel pg_dump

2012-01-29 Thread Joachim Wieland
On Fri, Jan 27, 2012 at 4:57 PM, Robert Haas wrote: > But even if you do know that subclassing > is intended, that doesn't prove that the particular Archive object is > always going to be an ArchiveHandle under the hood.  If it is, why not > just pass it as an ArchiveHandle to begin with? I know

Re: [HACKERS] Sending notifications from the master to the standby

2012-01-10 Thread Joachim Wieland
On Tue, Jan 10, 2012 at 11:55 AM, Tom Lane wrote: > Simon Riggs writes: > [ Tom sketches a design ] > Seems a bit overcomplicated.  I was just thinking of having walreceiver > note the WAL endpoint at the instant of receipt of a notify message, > and not release the notify message to the slave ri

Re: [HACKERS] Sending notifications from the master to the standby

2012-01-10 Thread Joachim Wieland
On Tue, Jan 10, 2012 at 12:00 AM, Tom Lane wrote: > So this design is non-optimal both for existing uses and for the > proposed new uses, which means nobody will like it.  You could > ameliorate #1 by adding a GUC that determines whether NOTIFY actually > writes WAL, but that's pretty ugly.  In an

Re: [HACKERS] pg_restore --no-post-data and --post-data-only

2011-12-08 Thread Joachim Wieland
On Tue, Nov 15, 2011 at 6:14 PM, Andrew Dunstan wrote: > Updated version with pg_restore included is attached. The patch applies with some fuzz by now but compiles without errors or warnings. The feature just works, it is not adding a lot of new code, basically it parses the given options and th

Re: [HACKERS] synchronized snapshots

2011-09-28 Thread Joachim Wieland
Hi Marko, On Wed, Sep 28, 2011 at 2:29 AM, Marko Tiikkaja wrote: > In a sequence such as this: > >  BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE; >  INSERT INTO foo VALUES (-1); >  SELECT pg_export_snapshot(); > > the row added to "foo" is not visible in the exported snapshot.  If that's > the

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 6:09 PM, Jim Nasby wrote: > I suspect that all the other cases of BEGIN failing would be syntax errors, so > you would immediately know in testing that something was wrong. A missing file > is definitely not a syntax error, so we can't really depend on user testing > to en

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 6:41 AM, Florian Weimer wrote: > * Simon Riggs: > >> I don't see the need to change the BEGIN command, which is SQL >> Standard. We don't normally do that. > > Some language bindings treat BEGIN specially, so it might be difficult > to use this feature. It's true, the comm

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 3:47 AM, Heikki Linnakangas wrote: > On 15.08.2011 04:31, Joachim Wieland wrote: >> >> The one thing that it does not implement is leaving the transaction in >> an aborted state if the BEGIN TRANSACTION command failed for an >> invalid snapshot

Re: [HACKERS] Snapshot synchronization, again...

2011-02-28 Thread Joachim Wieland
On Mon, Feb 28, 2011 at 6:38 PM, Robert Haas wrote: >> Remember that it's not only about saving shared memory, it's also >> about making sure that the snapshot reflects a state of the database >> that has actually existed at some point in the past. Furthermore, we >> can easily invalidate a snapsh

Re: [HACKERS] Snapshot synchronization, again...

2011-02-27 Thread Joachim Wieland
On Sun, Feb 27, 2011 at 3:04 PM, Heikki Linnakangas wrote: >> Why exactly, Heikki do you think the hash is more troublesome? > It just feels wrong to rely on cryptography just to save some shared memory. Remember that it's not only about saving shared memory, it's also about making sure that the

Re: [HACKERS] Snapshot synchronization, again...

2011-02-22 Thread Joachim Wieland
On Tue, Feb 22, 2011 at 3:34 PM, Heikki Linnakangas wrote: > On 22.02.2011 16:29, Robert Haas wrote: >> On Tue, Feb 22, 2011 at 8:58 AM, Heikki Linnakangas >>  wrote: >>> No, the hash is stored in shared memory. The hash of the garbage has to >>> match. >> >> Oh.  Well that's really silly.  At th

Re: [HACKERS] Snapshot synchronization, again...

2011-02-21 Thread Joachim Wieland
Hi, On Mon, Feb 21, 2011 at 4:56 PM, Alvaro Herrera wrote: > What's the reason for this restriction? >        if (databaseId != MyDatabaseId) >                ereport(ERROR, >                        (errcode(ERRCODE_INVALID_PARAMETER_VALUE), >                         errmsg("cannot import snapsho

Re: [HACKERS] Snapshot synchronization, again...

2011-02-19 Thread Joachim Wieland
On Sat, Feb 19, 2011 at 9:17 PM, Peter Eisentraut wrote: > The only consideration against MD5 might be > that it would make us look quite lame.  We should probably provide > builtin SHA1 and SHA2 functions for this and other reasons. In this particular case however the checksum is never exposed t

Re: [HACKERS] Snapshot synchronization, again...

2011-02-18 Thread Joachim Wieland
On Fri, Feb 18, 2011 at 8:57 PM, Alvaro Herrera wrote: > 1. why are you using the expansible char array stuff instead of using > the StringInfo facility? > > 2. is md5 the most appropriate digest for this?  If you need a > cryptographically secure hash, do we need something stronger?  If not, > wh

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-08 Thread Joachim Wieland
On Tue, Feb 8, 2011 at 8:31 PM, Itagaki Takahiro wrote: > On Tue, Feb 8, 2011 at 13:34, Robert Haas wrote: >> So how close are we to having a committable version of this?  Should >> we push this out to 9.2? > > I think so. The feature is pretty attractive, but more works are required: >  * Re-bas

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-07 Thread Joachim Wieland
Hi Jaime, thanks for your review! On Sun, Feb 6, 2011 at 2:12 PM, Jaime Casanova wrote: > code review: > > something i found, and is a very simple one, is this warning (there's > a similar issue in _StartMasterParallel with the buf variable) > """ > pg_backup_directory.c: In function ‘_EndMaster

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-04 Thread Joachim Wieland
On Thu, Feb 3, 2011 at 11:46 PM, Itagaki Takahiro wrote: > I think we have 2 important technical issues here: >  * The consistency is not perfect. Each transaction is started >   with small delays in step 1, but we cannot guarantee no other >   transaction between them. This is exactly where the

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-01 Thread Joachim Wieland
On Sun, Jan 30, 2011 at 5:26 PM, Robert Haas wrote: > The parallel pg_dump portion of this patch (i.e. the still-uncommitted > part) no longer applies.  Please rebase. Here is a rebased version with some minor changes as well. I haven't tested it on Windows now but will do so as soon as the Unix

Re: [HACKERS] Snapshot synchronization, again...

2011-01-30 Thread Joachim Wieland
Here is a new version of the patch incorporating most of Noah's suggestions. The patch now also adds documentation. Since I couldn't really find a suitable section to document the two new functions, I added a new one for now. Any better ideas where it should go? On Thu, Jan 20, 2011 at 1:37 AM, No

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Joachim Wieland
On Thu, Jan 20, 2011 at 6:07 AM, Heikki Linnakangas wrote: >> It's part of the overall idea to make sure files are not inadvertently >> exchanged between different backups and that a file is not truncated. >> In the future I'd also like to add a checksum to the TOC so that a >> backup can be check

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-19 Thread Joachim Wieland
On Wed, Jan 19, 2011 at 7:47 AM, Heikki Linnakangas wrote: >> Here are the latest patches all of them also rebased to current HEAD. >> Will update the commitfest app as well. > > What's the idea of storing the file sizes in the toc file? It looks like > it's not used for anything. It's part of th

Re: [HACKERS] Snapshot synchronization, again...

2011-01-18 Thread Joachim Wieland
Noah, thank you for this excellent review. I have taken over most (allmost all) of your suggestions (except for the documentation which is still missing). Please check the updated & attached patch for details. On Fri, Jan 14, 2011 at 10:13 PM, Noah Misch wrote: > "" is a valid

Re: [HACKERS] Snapshot synchronization, again...

2011-01-07 Thread Joachim Wieland
On Thu, Dec 30, 2010 at 7:31 AM, Joachim Wieland wrote: > What I am proposing now is the following: > > We return snapshot information as a chunk of data to the client. At > the same time however, we set a checksum in shared memory to protect > against modification of the snapsho

Re: [HACKERS] Snapshot synchronization, again...

2010-12-31 Thread Joachim Wieland
On Fri, Dec 31, 2010 at 8:28 AM, Alvaro Herrera wrote: > A backend can have any number of snapshots registered, and those don't > allow GlobalXmin to advance. Consider an open cursor, for example. > Even if the rest of the transaction is read committed, the snapshot > registered by the open curso

Re: [HACKERS] Snapshot synchronization, again...

2010-12-30 Thread Joachim Wieland
On Thu, Dec 30, 2010 at 9:49 AM, Florian Pflug wrote: > On Dec30, 2010, at 13:31 , Joachim Wieland wrote: >> We return snapshot information as a chunk of data to the client. At >> the same time however, we set a checksum in shared memory to protect >> against modificati

Re: [HACKERS] Snapshot synchronization, again...

2010-12-30 Thread Joachim Wieland
On Thu, Dec 30, 2010 at 9:40 AM, Alvaro Herrera wrote: >> Disadvantage of b: It doesn't allow a snapshot to be installed on a >> different server. It requires a serializable open transaction to hold >> the snapshot. > > Why does it require a serializable transaction?  You could simply > register t

[HACKERS] Snapshot synchronization, again...

2010-12-30 Thread Joachim Wieland
The snapshot synchronization discussion from the parallel pg_dump patch somehow ended without a clear way to go forward. Let me sum up what has been brought up and propose a short- and longterm solution. Summary: Passing snapshot sync information can be done either: a) by returning complete sna

Re: [HACKERS] page compression

2010-12-28 Thread Joachim Wieland
On Tue, Dec 28, 2010 at 10:10 AM, Andy Colson wrote: > I know its been discussed before, and one big problem is license and patent > problems. > > Would this project be a problem: > > http://oldhome.schmorp.de/marc/liblzf.html It looks like even liblzf is not going to be accepted. I have proposed

Re: [HACKERS] directory archive format for pg_dump

2010-12-16 Thread Joachim Wieland
On Thu, Dec 16, 2010 at 12:48 PM, Heikki Linnakangas wrote: > As soon as we have parallel pg_dump, the next big thing is going to be > parallel dump of the same table using multiple processes. Perhaps we should > prepare for that in the directory archive format, by allowing the data of a > single

Re: [HACKERS] Hot Standby: too many KnownAssignedXids

2010-12-15 Thread Joachim Wieland
On Tue, Dec 7, 2010 at 3:42 AM, Heikki Linnakangas wrote: > Ok, I've committed this patch now. I can confirm that I could continue replaying the logfiles on the standby host with this patch. Thanks a lot, Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make ch

Re: [HACKERS] directory archive format for pg_dump

2010-12-07 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 2:52 PM, Heikki Linnakangas wrote: > Ok, committed, with some small cleanup since the last patch I posted. > > Could you update the directory-format patch on top of the committed version, > please? Thanks for committing the first part. Here is the updated and rebased direct

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Joachim Wieland
On Sun, Dec 5, 2010 at 9:27 PM, Robert Haas wrote: > On Sun, Dec 5, 2010 at 9:04 PM, Andrew Dunstan wrote: >> Why not just say give me the snapshot currently held by process ? >> >> And please, not temp files if possible. > > As far as I'm aware, the full snapshot doesn't normally exist in >

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-02 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 9:33 PM, Tom Lane wrote: > In particular, this issue *has* been discussed before, and there was a > consensus that preserving dump consistency was a requirement.  I don't > think that Joachim gets to bypass that decision just by submitting a > patch that ignores it. I am no

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-02 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 12:56 PM, Josh Berkus wrote: > Now, if only I could think of some way to write a parallel dump to a set of > pipes, I'd be in heaven. What exactly are you trying to accomplish with the pipes? Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) T

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-02 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 6:19 AM, Heikki Linnakangas wrote: > I don't see the point of the sort-by-relpages code. The order the objects > are dumped should be irrelevant, as long as you obey the restrictions > dictated by dependencies. Or is it only needed for the multiple-target-dirs > feature? Fra

Re: [HACKERS] directory archive format for pg_dump

2010-12-01 Thread Joachim Wieland
On Wed, Dec 1, 2010 at 9:05 AM, Heikki Linnakangas wrote: > Forgot attachment. This is also available in the above git repo. I have quickly checked your modifications, on the one hand I like the reduction of functions, I would have said that we have AH around all the time and so we could just all

Re: [HACKERS] Suggested "easy" TODO: pg_dump --from-list

2010-11-24 Thread Joachim Wieland
On Wed, Nov 24, 2010 at 9:38 AM, Andrew Dunstan wrote: > It would be unique, but a pain in the neck for users to get. Robert's idea > will have more traction with users. Whatever approach we use, we need to think about the use case where 1% of the objects should be dumped but should also make sur

Re: [HACKERS] Suggested "easy" TODO: pg_dump --from-list

2010-11-24 Thread Joachim Wieland
On Wed, Nov 24, 2010 at 1:15 AM, Tom Lane wrote: > Nope ... those strings are just helpful comments, they aren't really > guaranteed to be unique identifiers.  In any case, it seems unlikely > that a user could expect to get the more complicated cases exactly right > other than by consulting "pg_d

Re: [HACKERS] Hot Standby: too many KnownAssignedXids

2010-11-23 Thread Joachim Wieland
On Tue, Nov 23, 2010 at 8:45 AM, Heikki Linnakangas wrote: > On 19.11.2010 23:46, Joachim Wieland wrote: >> >> FATAL:  too many KnownAssignedXids. head: 0, tail: 0, nxids: 9978, >> pArray->maxKnownAssignedXids: 6890 > > Hmm, that's a lot of entries in KnownAssig

Re: [HACKERS] Suggested "easy" TODO: pg_dump --from-list

2010-11-23 Thread Joachim Wieland
On Tue, Nov 23, 2010 at 10:24 PM, Andrew Dunstan wrote: > Well, very little about pg_dump is very [E], IMNSHO. The question in my mind > here is what format the list file will take. For example, how would we > specify a function? Would we need to specify all the argument types (or at > least the I

Re: [HACKERS] Hot Standby: too many KnownAssignedXids

2010-11-22 Thread Joachim Wieland
On Sun, Nov 21, 2010 at 11:48 PM, Fujii Masao wrote: > -- > If you suspect a bug in Hot Standby, please set >        trace_recovery_messages = DEBUG2 > in postgresql.conf and repeat the action > > Always useful to know > * max_connections > * current number of sessions > * whether we h

Re: [HACKERS] directory archive format for pg_dump

2010-11-19 Thread Joachim Wieland
Hi Jose, 2010/11/19 José Arthur Benetasso Villanova : > The dir format generated in my database 60 files, with different > sizes, and it looks very confusing. Is it possible to use the same > trick as pigz and pbzip2, creating a concatenated file of streams? What pigz is parallelizing is the actu

Re: [HACKERS] directory archive format for pg_dump

2010-11-19 Thread Joachim Wieland
On Fri, Nov 19, 2010 at 11:53 PM, Tom Lane wrote: > Dimitri Fontaine writes: > > I think I'd like to see a separate patch for the new compression > > support. Sorry about that, I realize that's extra work… > > That part of the patch is likely to get rejected outright anyway, > so I *strongly* re

Re: [HACKERS] directory archive format for pg_dump

2010-11-19 Thread Joachim Wieland
Hi Dimitri, thanks for reviewing my patch! On Fri, Nov 19, 2010 at 2:44 PM, Dimitri Fontaine wrote: > I think I'd like to see a separate patch for the new compression > support. Sorry about that, I realize that's extra work… I guess it wouldn't be a very big deal but I also doubt that it makes

[HACKERS] Hot Standby: too many KnownAssignedXids

2010-11-19 Thread Joachim Wieland
Hi, I am seeing the following here on 9.0.1 on Linux x86-64: LOG: redo starts at 1F8/FC00E978 FATAL: too many KnownAssignedXids CONTEXT: xlog redo insert: rel 1663/16384/18373; tid 3829898/23 and this is the complete history: postgres was running as HS in foreground, Ctrl-C'ed it for a rest

Re: [HACKERS] host name support in pg_hba.conf

2010-10-05 Thread Joachim Wieland
On Tue, Oct 5, 2010 at 3:41 PM, Peter Eisentraut wrote: > The reason why I think this is semi-important and not just cosmetic is > that (for some reason that is not entirely clear to me) clients > connecting to localhost end up appearing to the server as ::1 on a lot > of machines I use which are

Re: [HACKERS] Patch for PKST timezone

2010-05-16 Thread Joachim Wieland
On Wed, May 12, 2010 at 12:39 AM, Tom Lane wrote: > Joachim Wieland writes: >> Good we have found that inconsistency now, so let's add PKST. > > OK, done.  BTW, I notice that PKT was labeled "(not in zic)", which > is not the case, per this discussion.  I see

Re: [HACKERS] Patch for PKST timezone

2010-05-11 Thread Joachim Wieland
On Tue, May 11, 2010 at 10:45 AM, Heikki Linnakangas wrote: > I don't think we want to include all timezone names in the default > config, timezone abbreviations aren't always unique for example. But we > should include PKST because we already include PKT; it would be nasty > for an application to

Re: [HACKERS] a faster compression algorithm for pg_dump

2010-04-10 Thread Joachim Wieland
On Fri, Apr 9, 2010 at 5:51 AM, Greg Stark wrote: > Linking against as an option isn't nearly as bad since the user > compiling it can choose whether to include the restricted feature or > not. That's what we do with readline. However it's not nearly as > attractive when it restricts what file for

[HACKERS] a faster compression algorithm for pg_dump

2010-04-08 Thread Joachim Wieland
I'd like to revive the discussion about offering another compression algorithm than zlib to at least pg_dump. There has been a previous discussion here: http://archives.postgresql.org/pgsql-performance/2009-08/msg00053.php and it ended without any real result. The results so far were: - There ex

Re: [HACKERS] message clarifications

2010-04-03 Thread Joachim Wieland
On Sat, Apr 3, 2010 at 9:02 PM, Robert Haas wrote: > On Apr 3, 2010, at 11:13 AM, Tom Lane wrote: >> Peter Eisentraut writes: >>> The following messages from the postgres catalog either appear to be >>> internal errors that should be marked differently, or they are in my >>> estimation unintelli

[HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Joachim Wieland
People have been talking about a parallel version of pg_dump a few times already. I have been working on some proof-of-concept code for this feature every now and then and I am planning to contribute this for 9.1. There are two main issues with a parallel version of pg_dump: The first one is that

Re: [HACKERS] five-key syscaches

2010-03-29 Thread Joachim Wieland
On Mon, Mar 29, 2010 at 12:32 AM, Robert Haas wrote: > Per previous discussion, PFA a patch to change the maximum number of > keys for a syscache from 4 to 5. > > http://archives.postgresql.org/pgsql-hackers/2010-02/msg01105.php > > This is intended for application to 9.1, and is supporting > infr

Re: [HACKERS] 9.0 release notes done

2010-03-22 Thread Joachim Wieland
On Sat, Mar 20, 2010 at 5:02 AM, Bruce Momjian wrote: > Interestingly the 9.0 release notes contain 201 items, while the 8.4 > release notes contained 314 items. Is the following pg_dump change covered by the release notes? I couldn't find it. It was the last committed patch from the 2010-01 comm

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-28 Thread Joachim Wieland
On Sun, Feb 28, 2010 at 8:47 PM, Josh Berkus wrote: > 1) Automated retry of cancelled queries on the slave.  I have no idea > how hard this would be to implement, but it makes the difference between > writing lots of exception-handling code for slave connections > (unacceptable) to just slow respo

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-28 Thread Joachim Wieland
On Sun, Feb 28, 2010 at 2:54 PM, Greg Stark wrote: > Really? I think we get lots of suprised wows from the field from the > idea that a long-running read-only query can cause your database to > bloat. I think the only reason that's obvious to us is that we've been > grappling with that problem for

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-17 Thread Joachim Wieland
On Tue, Feb 16, 2010 at 11:41 PM, Tom Lane wrote: > Joachim Wieland writes: >> [ listen/notify patch ] > > I found a number of implementation problems having to do with wraparound > behavior and error recovery.  I think they're all fixed, but any > remaining bugs are p

[HACKERS] Listen/Notify payload and interfaces

2010-02-17 Thread Joachim Wieland
This one is for the maintainers of the various postgresql interfaces: With the new listen/notify implementation we can send a payload along with the notification. This has been in the protocol already since years and there is no change needed for libpq. However we need to adapt the various interfa

Re: [HACKERS] LISTEN/NOTIFY and notification timing guarantees

2010-02-16 Thread Joachim Wieland
On Tue, Feb 16, 2010 at 1:31 PM, Kevin Grittner wrote: > Tom Lane  wrote: >> We could adopt the historical policy of sending self-notifies >> pre-commit, but that doesn't seem tremendously appetizing from the >> standpoint of transactional integrity. > > But one traditional aspect of transactional

Re: [HACKERS] LISTEN/NOTIFY and notification timing guarantees

2010-02-16 Thread Joachim Wieland
On Tue, Feb 16, 2010 at 6:20 AM, Tom Lane wrote: > Another possibility is to force a ProcessIncomingNotifies scan to occur > before we reach ReadyForQuery if we sent any notifies in the > just-finished transaction --- but that won't help if there are > uncommitted messages in front of ours. What

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-15 Thread Joachim Wieland
On Mon, Feb 15, 2010 at 1:48 PM, Simon Riggs wrote: > On Mon, 2010-02-15 at 12:59 +0100, Joachim Wieland wrote: >> I have tested it already. The point where it currently fails is the >> following line: >> >>       qe->xid = GetCurrentTransactionId(); > > That

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-15 Thread Joachim Wieland
On Sun, Feb 14, 2010 at 11:44 PM, Simon Riggs wrote: > Next set of questions > > * Will this work during Hot Standby now? The barrier was that it wrote > to a table and so we could not allow that. ISTM this new version can and > should work with Hot Standby. Can you test that and if so, remove the

Re: [HACKERS] LISTEN/NOTIFY and notification timing guarantees

2010-02-15 Thread Joachim Wieland
On Mon, Feb 15, 2010 at 3:31 AM, Tom Lane wrote: > I'm not sure how probable it is that applications might be coded in a > way that relies on the properties lost according to point #2 or #3. Your observations are all correct as far as I can tell. One question regarding #2: Is a client applicatio

Re: [HACKERS] Parameter name standby_mode

2010-02-12 Thread Joachim Wieland
On Fri, Feb 12, 2010 at 8:59 AM, Heikki Linnakangas wrote: > Agreed. I've changed it now so that if primary_conninfo is not set, it > doesn't try to establish a streaming connection. If you want to get the > connection information from environment variables, you can use > primary_conninfo=''. Why

  1   2   >