[HACKERS] pg_dump bug in 9.4beta2 and HEAD

2014-08-13 Thread Joachim Wieland
I'm seeing an assertion failure with pg_dump -c --if-exists which is not ready to handle BLOBs it seems: pg_dump: pg_backup_archiver.c:472: RestoreArchive: Assertion `mark != ((void *)0)' failed. To reproduce: $ createdb test $ pg_dump -c --if-exists test (works, dumps empty database) $ psql

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Joachim Wieland
On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: What might make sense is something like pg_dump_restore which would have no intermediate storage at all, just pump the data etc from one source to another in parallel. But I pity the poor guy who has to

Re: [HACKERS] Optimizing pglz compressor

2013-03-06 Thread Joachim Wieland
On Tue, Mar 5, 2013 at 8:32 AM, Heikki Linnakangas hlinnakan...@vmware.com wrote: With these tweaks, I was able to make pglz-based delta encoding perform roughly as well as Amit's patch. Out of curiosity, do we know how pglz compares with other algorithms, e.g. lz4 ? -- Sent via

[HACKERS] Materialized view assertion failure in HEAD

2013-03-05 Thread Joachim Wieland
I'm getting an assertion failure in HEAD with materialized views, see below for backtrace. To reproduce, just run make installcheck, dump the regression database and then restore it, the server crashes during restore. (gdb) bt #0 0x7f283a366425 in raise () from

Re: [HACKERS] Materialized view assertion failure in HEAD

2013-03-05 Thread Joachim Wieland
On Tue, Mar 5, 2013 at 9:11 AM, Kevin Grittner kgri...@ymail.com wrote: Will investigate. You don't have default_with_oids = on, do you? No, this was a default install with a default postgresql.conf -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] posix_fadvise missing in the walsender

2013-02-20 Thread Joachim Wieland
On Wed, Feb 20, 2013 at 4:54 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Feb 19, 2013 at 5:48 PM, Simon Riggs si...@2ndquadrant.com wrote: I agree with Merlin and Joachim - if we have the call in one place, we should have it in both. We might want to assess whether we even want to

Re: [HACKERS] posix_fadvise missing in the walsender

2013-02-19 Thread Joachim Wieland
On Tue, Feb 19, 2013 at 5:48 PM, Simon Riggs si...@2ndquadrant.com wrote: In access/transam/xlog.c we give the OS buffer caching a hint that we won't need a WAL file any time soon with posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); I agree with Merlin and Joachim - if we have

[HACKERS] posix_fadvise missing in the walsender

2013-02-17 Thread Joachim Wieland
In access/transam/xlog.c we give the OS buffer caching a hint that we won't need a WAL file any time soon with posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); before closing the WAL file, but only if we don't have walsenders. That's reasonable because the walsender will reopen that

Re: [HACKERS] Logical decoding exported base snapshot

2012-12-11 Thread Joachim Wieland
On Tue, Dec 11, 2012 at 6:52 PM, Andres Freund and...@2ndquadrant.com wrote: One problem I see is that while exporting a snapshot solves the visibility issues of the table's contents it does not protect against schema changes. I am not sure whether thats a problem. If somebody runs a CLUSTER

Re: [HACKERS] parallel pg_dump

2012-12-08 Thread Joachim Wieland
On Sat, Dec 8, 2012 at 3:05 PM, Bruce Momjian br...@momjian.us wrote: On Sat, Dec 8, 2012 at 11:13:30AM -0500, Andrew Dunstan wrote: I am working on it when I get a chance, but keep getting hammered. I'd love somebody else to review it too. FYI, I will be posting pg_upgrade performance

Re: [HACKERS] [PATCH] pg_dump: Sort overloaded functions in deterministic order

2012-10-17 Thread Joachim Wieland
On Wed, Oct 17, 2012 at 5:43 PM, Alvaro Herrera alvhe...@2ndquadrant.com wrote: (I tested the new pg_dump with 8.2 and HEAD and also verified it passes pg_upgrade's make check. I didn't test with other server versions.) I also tested against 8.3 and 8.4 since 8.4 is the version that introduced

Re: [HACKERS] Add FET to Default and Europe.txt

2012-10-08 Thread Joachim Wieland
On Mon, Oct 8, 2012 at 4:00 PM, Tom Lane t...@sss.pgh.pa.us wrote: We can't just refuse to deal with this ambiguity. We have to have some very-low-pain way to install settings that will please those large fractions of our user base. Moreover, if that very-low-pain way isn't the exact same

[HACKERS] Review for pg_dump: Sort overloaded functions in deterministic order

2012-09-25 Thread Joachim Wieland
Patch looks good, all concerns that had been raised previously have been addressed in this version of the patch. The only thing that IMO needs to change is a stylistic issue: if (fout-remoteVersion = 80200) { [...] (fout-remoteVersion = 80400) ?

Re: [HACKERS] could not open relation with OID errors after promoting the standby to master

2012-05-24 Thread Joachim Wieland
On Tue, May 22, 2012 at 9:50 AM, Robert Haas robertmh...@gmail.com wrote: Hmm.  I think that if you do it this way, the minimum recovery point won't be respected, which could leave you with a corrupted database. Now, if all the WAL files that you need are present in pg_xlog anyway, then they

Re: [HACKERS] could not open relation with OID errors after promoting the standby to master

2012-05-17 Thread Joachim Wieland
On Wed, May 16, 2012 at 11:38 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: Well, that is not surprising in itself -- InitTempTableNamespace calls RemoveTempRelations to cleanup from a possibly crashed previous backend with the same ID.  So that part of the backtrace looks normal to me

[HACKERS] could not open relation with OID errors after promoting the standby to master

2012-05-15 Thread Joachim Wieland
I've switched servers yesterday night and the previous slave is now the master. This is 9.0.6 (originally) / 9.0.7 (now) on Linux. Now I'm seeing a bunch of ERROR: could not open relation with OID 1990987633 STATEMENT: create temp table seen_files (fileid integer) Interestingly enough, 90% of

Re: [HACKERS] parallel pg_dump

2012-04-04 Thread Joachim Wieland
On Tue, Apr 3, 2012 at 9:26 AM, Andrew Dunstan and...@dunslane.net wrote: First, either the creation of the destination directory needs to be delayed until all the sanity checks have passed and we're sure we're actually going to write something there, or it needs to be removed if we error exit

Re: [HACKERS] patch for parallel pg_dump

2012-04-04 Thread Joachim Wieland
On Tue, Apr 3, 2012 at 11:04 AM, Robert Haas robertmh...@gmail.com wrote: OK, but it seems like a pretty fragile assumption that none of the workers will ever manage to emit any other error messages.  We don't rely on this kind of assumption in the backend, which is a lot better-structured and

Re: [HACKERS] patch for parallel pg_dump

2012-04-03 Thread Joachim Wieland
On Tue, Apr 3, 2012 at 9:34 AM, Robert Haas robertmh...@gmail.com wrote: On Sun, Apr 1, 2012 at 12:35 PM, Joachim Wieland j...@mcknight.de wrote: Unfortunately this is not really the case. What is being moved out of pg_backup_archiver.c and into parallel.c is either the shutdown logic that has

Re: [HACKERS] patch for parallel pg_dump

2012-03-28 Thread Joachim Wieland
On Wed, Mar 28, 2012 at 5:19 PM, Andrew Dunstan and...@dunslane.net wrote: First hurdle: It doesn't build under Windows/mingw-w64:   parallel.c:40:12: error: static declaration of 'pgpipe' follows   non-static declaration Strange, I'm not seeing this but I'm building with VC2005. What happens

Re: [HACKERS] patch for parallel pg_dump

2012-03-28 Thread Joachim Wieland
On Thu, Mar 29, 2012 at 2:46 AM, Andrew Dunstan and...@dunslane.net wrote: But your patch hasn't got rid of them, and so it's declared twice. There is no pgpipe.h, BTW, it's declared in port.h. If VC2005 doesn't complain about the double declaration then that's a bug in the compiler, IMNSHO.

Re: [HACKERS] patch for parallel pg_dump

2012-03-28 Thread Joachim Wieland
On Wed, Mar 28, 2012 at 1:46 PM, Robert Haas robertmh...@gmail.com wrote: I'm wondering if we really need this much complexity around shutting down workers.  I'm not sure I understand why we need both a hard and a soft method of shutting them down.  At least on non-Windows systems, it seems

Re: [HACKERS] patch for parallel pg_dump

2012-03-25 Thread Joachim Wieland
On Fri, Mar 23, 2012 at 11:11 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Are you going to provide a rebased version? Rebased version attached, this patch also includes Robert's earlier suggestions. parallel_pg_dump_5.diff.gz Description: GNU Zip compressed data -- Sent via

Re: [HACKERS] patch for parallel pg_dump

2012-03-23 Thread Joachim Wieland
On Fri, Mar 23, 2012 at 11:11 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Are you going to provide a rebased version? Yes, working on that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

[HACKERS] double free in current HEAD's pg_dump

2012-03-17 Thread Joachim Wieland
There's a double free in the current HEAD's pg_dump. Fix attached. diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 2b0a5ff..57a6ccb 100644 *** a/src/bin/pg_dump/pg_dump.c --- b/src/bin/pg_dump/pg_dump.c *** dumpBlobs(Archive *fout, void *arg) *** 2372,2379

Re: [HACKERS] patch for parallel pg_dump

2012-03-17 Thread Joachim Wieland
On Fri, Mar 16, 2012 at 12:06 AM, Robert Haas robertmh...@gmail.com wrote: Good. The only exit handler I've seen so far is pgdump_cleanup_at_exit. If there's no other one, is it okay to remove all of this stacking functionality (see on_exit_nicely_index / MAX_ON_EXIT_NICELY) from dumputils.c

Re: [HACKERS] patch for parallel pg_dump

2012-03-14 Thread Joachim Wieland
On Wed, Mar 14, 2012 at 2:02 PM, Robert Haas robertmh...@gmail.com wrote: I think we should get rid of die_horribly(), and instead have arrange to always clean up AH via an on_exit_nicely hook. Good. The only exit handler I've seen so far is pgdump_cleanup_at_exit. If there's no other one, is

Re: [HACKERS] patch for parallel pg_dump

2012-03-14 Thread Joachim Wieland
On Wed, Mar 14, 2012 at 4:39 PM, Andrew Dunstan aduns...@postgresql.org wrote: I've just started looking at the patch, and I'm curious to know why it didn't follow the pattern of parallel pg_restore which created a new worker for each table rather than passing messages to looping worker threads

Re: [HACKERS] patch for parallel pg_dump

2012-03-13 Thread Joachim Wieland
On Tue, Mar 13, 2012 at 1:53 PM, Robert Haas robertmh...@gmail.com wrote: What I mean is that the function ArchiveEntry() is defined in pg_backup_archiver.c, and it takes an argument called relpages, and the string relpages does not appear anywhere else in that file. Uhm, that's kinda

Re: [HACKERS] patch for parallel pg_dump

2012-02-23 Thread Joachim Wieland
On Thu, Feb 16, 2012 at 1:29 PM, Robert Haas robertmh...@gmail.com wrote: Can you provide an updated patch? Robert, updated patch is attached. parallel_pg_dump_2.diff.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] patch for parallel pg_dump

2012-02-08 Thread Joachim Wieland
On Wed, Feb 8, 2012 at 1:21 PM, Robert Haas robertmh...@gmail.com wrote: In my patch I dealt with exactly the same problem for the error handler by creating a separate function that has a static variable (a pointer to the ParallelState). The value is set and retrieved through the same

Re: [HACKERS] patch for parallel pg_dump

2012-02-07 Thread Joachim Wieland
On Tue, Feb 7, 2012 at 4:59 PM, Robert Haas robertmh...@gmail.com wrote: It turns out that (as you anticipated) there are some problems with my previous proposal. I assume you're talking to Tom, as my powers of anticipation are actually quite limited... :-) This is not quite enough to get

Re: [HACKERS] patch for parallel pg_dump

2012-02-03 Thread Joachim Wieland
On Fri, Feb 3, 2012 at 7:52 AM, Robert Haas robertmh...@gmail.com wrote: If you're OK with that much I'll go do it. Sure, go ahead! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch for parallel pg_dump

2012-02-02 Thread Joachim Wieland
On Wed, Feb 1, 2012 at 12:24 PM, Robert Haas robertmh...@gmail.com wrote: I think we're more-or-less proposing to rename Archive to Connection, aren't we? And then ArchiveHandle can store all the things that aren't related to a specific connection. How about something like that: (Hopefully

Re: [HACKERS] patch for parallel pg_dump

2012-01-31 Thread Joachim Wieland
On Tue, Jan 31, 2012 at 9:05 AM, Robert Haas robertmh...@gmail.com wrote: And just for added fun and excitement, they all have inconsistent naming conventions and inadequate documentation. I think if we need more refactoring in order to support multiple database connections, we should go do

Re: [HACKERS] patch for parallel pg_dump

2012-01-30 Thread Joachim Wieland
On Mon, Jan 30, 2012 at 12:20 PM, Robert Haas robertmh...@gmail.com wrote: But the immediate problem is that pg_dump.c is heavily reliant on global variables, which isn't going to fly if we want this code to use threads on Windows (or anywhere else).  It's also bad style. Technically, since

Re: [HACKERS] patch for parallel pg_dump

2012-01-29 Thread Joachim Wieland
On Fri, Jan 27, 2012 at 4:57 PM, Robert Haas robertmh...@gmail.com wrote: But even if you do know that subclassing is intended, that doesn't prove that the particular Archive object is always going to be an ArchiveHandle under the hood.  If it is, why not just pass it as an ArchiveHandle to

Re: [HACKERS] Sending notifications from the master to the standby

2012-01-10 Thread Joachim Wieland
On Tue, Jan 10, 2012 at 12:00 AM, Tom Lane t...@sss.pgh.pa.us wrote: So this design is non-optimal both for existing uses and for the proposed new uses, which means nobody will like it.  You could ameliorate #1 by adding a GUC that determines whether NOTIFY actually writes WAL, but that's

Re: [HACKERS] Sending notifications from the master to the standby

2012-01-10 Thread Joachim Wieland
On Tue, Jan 10, 2012 at 11:55 AM, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: [ Tom sketches a design ] Seems a bit overcomplicated.  I was just thinking of having walreceiver note the WAL endpoint at the instant of receipt of a notify message, and not release

Re: [HACKERS] pg_restore --no-post-data and --post-data-only

2011-12-08 Thread Joachim Wieland
On Tue, Nov 15, 2011 at 6:14 PM, Andrew Dunstan and...@dunslane.net wrote: Updated version with pg_restore included is attached. The patch applies with some fuzz by now but compiles without errors or warnings. The feature just works, it is not adding a lot of new code, basically it parses the

Re: [HACKERS] synchronized snapshots

2011-09-28 Thread Joachim Wieland
Hi Marko, On Wed, Sep 28, 2011 at 2:29 AM, Marko Tiikkaja marko.tiikk...@2ndquadrant.com wrote: In a sequence such as this:  BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE;  INSERT INTO foo VALUES (-1);  SELECT pg_export_snapshot(); the row added to foo is not visible in the exported

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 3:47 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 15.08.2011 04:31, Joachim Wieland wrote: The one thing that it does not implement is leaving the transaction in an aborted state if the BEGIN TRANSACTION command failed for an invalid snapshot

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 6:41 AM, Florian Weimer fwei...@bfk.de wrote: * Simon Riggs: I don't see the need to change the BEGIN command, which is SQL Standard. We don't normally do that. Some language bindings treat BEGIN specially, so it might be difficult to use this feature. It's true,

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 6:09 PM, Jim Nasby j...@nasby.net wrote: I suspect that all the other cases of BEGIN failing would be syntax errors, so you would immediately know in testing that something was wrong. A missing file is definitely not a syntax error, so we can't really depend on user

Re: [HACKERS] Snapshot synchronization, again...

2011-02-28 Thread Joachim Wieland
On Mon, Feb 28, 2011 at 6:38 PM, Robert Haas robertmh...@gmail.com wrote: Remember that it's not only about saving shared memory, it's also about making sure that the snapshot reflects a state of the database that has actually existed at some point in the past. Furthermore, we can easily

Re: [HACKERS] Snapshot synchronization, again...

2011-02-27 Thread Joachim Wieland
On Sun, Feb 27, 2011 at 3:04 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Why exactly, Heikki do you think the hash is more troublesome? It just feels wrong to rely on cryptography just to save some shared memory. Remember that it's not only about saving shared memory, it's

Re: [HACKERS] Snapshot synchronization, again...

2011-02-22 Thread Joachim Wieland
On Tue, Feb 22, 2011 at 3:34 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 22.02.2011 16:29, Robert Haas wrote: On Tue, Feb 22, 2011 at 8:58 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com  wrote: No, the hash is stored in shared memory. The hash of the

Re: [HACKERS] Snapshot synchronization, again...

2011-02-21 Thread Joachim Wieland
Hi, On Mon, Feb 21, 2011 at 4:56 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: What's the reason for this restriction?        if (databaseId != MyDatabaseId)                ereport(ERROR,                        (errcode(ERRCODE_INVALID_PARAMETER_VALUE),                        

Re: [HACKERS] Snapshot synchronization, again...

2011-02-19 Thread Joachim Wieland
On Sat, Feb 19, 2011 at 9:17 PM, Peter Eisentraut pete...@gmx.net wrote: The only consideration against MD5 might be that it would make us look quite lame.  We should probably provide builtin SHA1 and SHA2 functions for this and other reasons. In this particular case however the checksum is

Re: [HACKERS] Snapshot synchronization, again...

2011-02-18 Thread Joachim Wieland
On Fri, Feb 18, 2011 at 8:57 PM, Alvaro Herrera alvhe...@commandprompt.com wrote: 1. why are you using the expansible char array stuff instead of using the StringInfo facility? 2. is md5 the most appropriate digest for this?  If you need a cryptographically secure hash, do we need something

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-08 Thread Joachim Wieland
On Tue, Feb 8, 2011 at 8:31 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: On Tue, Feb 8, 2011 at 13:34, Robert Haas robertmh...@gmail.com wrote: So how close are we to having a committable version of this?  Should we push this out to 9.2? I think so. The feature is pretty attractive,

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-07 Thread Joachim Wieland
Hi Jaime, thanks for your review! On Sun, Feb 6, 2011 at 2:12 PM, Jaime Casanova ja...@2ndquadrant.com wrote: code review: something i found, and is a very simple one, is this warning (there's a similar issue in _StartMasterParallel with the buf variable) pg_backup_directory.c: In

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-04 Thread Joachim Wieland
On Thu, Feb 3, 2011 at 11:46 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: I think we have 2 important technical issues here:  * The consistency is not perfect. Each transaction is started   with small delays in step 1, but we cannot guarantee no other   transaction between them.

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-01 Thread Joachim Wieland
On Sun, Jan 30, 2011 at 5:26 PM, Robert Haas robertmh...@gmail.com wrote: The parallel pg_dump portion of this patch (i.e. the still-uncommitted part) no longer applies.  Please rebase. Here is a rebased version with some minor changes as well. I haven't tested it on Windows now but will do so

Re: [HACKERS] Snapshot synchronization, again...

2011-01-30 Thread Joachim Wieland
Here is a new version of the patch incorporating most of Noah's suggestions. The patch now also adds documentation. Since I couldn't really find a suitable section to document the two new functions, I added a new one for now. Any better ideas where it should go? On Thu, Jan 20, 2011 at 1:37 AM,

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Joachim Wieland
On Thu, Jan 20, 2011 at 6:07 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: It's part of the overall idea to make sure files are not inadvertently exchanged between different backups and that a file is not truncated. In the future I'd also like to add a checksum to the TOC so

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-19 Thread Joachim Wieland
On Wed, Jan 19, 2011 at 7:47 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Here are the latest patches all of them also rebased to current HEAD. Will update the commitfest app as well. What's the idea of storing the file sizes in the toc file? It looks like it's not used

Re: [HACKERS] Snapshot synchronization, again...

2011-01-18 Thread Joachim Wieland
Noah, thank you for this excellent review. I have taken over most (allmost all) of your suggestions (except for the documentation which is still missing). Please check the updated attached patch for details. On Fri, Jan 14, 2011 at 10:13 PM, Noah Misch n...@leadboat.com wrote:

Re: [HACKERS] Snapshot synchronization, again...

2011-01-07 Thread Joachim Wieland
On Thu, Dec 30, 2010 at 7:31 AM, Joachim Wieland j...@mcknight.de wrote: What I am proposing now is the following: We return snapshot information as a chunk of data to the client. At the same time however, we set a checksum in shared memory to protect against modification of the snapshot

Re: [HACKERS] Snapshot synchronization, again...

2010-12-31 Thread Joachim Wieland
On Fri, Dec 31, 2010 at 8:28 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: A backend can have any number of snapshots registered, and those don't allow GlobalXmin to advance. Consider an open cursor, for example. Even if the rest of the transaction is read committed, the snapshot

[HACKERS] Snapshot synchronization, again...

2010-12-30 Thread Joachim Wieland
The snapshot synchronization discussion from the parallel pg_dump patch somehow ended without a clear way to go forward. Let me sum up what has been brought up and propose a short- and longterm solution. Summary: Passing snapshot sync information can be done either: a) by returning complete

Re: [HACKERS] Snapshot synchronization, again...

2010-12-30 Thread Joachim Wieland
On Thu, Dec 30, 2010 at 9:40 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Disadvantage of b: It doesn't allow a snapshot to be installed on a different server. It requires a serializable open transaction to hold the snapshot. Why does it require a serializable transaction?  You could

Re: [HACKERS] Snapshot synchronization, again...

2010-12-30 Thread Joachim Wieland
On Thu, Dec 30, 2010 at 9:49 AM, Florian Pflug f...@phlo.org wrote: On Dec30, 2010, at 13:31 , Joachim Wieland wrote: We return snapshot information as a chunk of data to the client. At the same time however, we set a checksum in shared memory to protect against modification of the snapshot

Re: [HACKERS] page compression

2010-12-28 Thread Joachim Wieland
On Tue, Dec 28, 2010 at 10:10 AM, Andy Colson a...@squeakycode.net wrote: I know its been discussed before, and one big problem is license and patent problems. Would this project be a problem: http://oldhome.schmorp.de/marc/liblzf.html It looks like even liblzf is not going to be accepted.

Re: [HACKERS] directory archive format for pg_dump

2010-12-16 Thread Joachim Wieland
On Thu, Dec 16, 2010 at 12:48 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: As soon as we have parallel pg_dump, the next big thing is going to be parallel dump of the same table using multiple processes. Perhaps we should prepare for that in the directory archive format, by

Re: [HACKERS] Hot Standby: too many KnownAssignedXids

2010-12-15 Thread Joachim Wieland
On Tue, Dec 7, 2010 at 3:42 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok, I've committed this patch now. I can confirm that I could continue replaying the logfiles on the standby host with this patch. Thanks a lot, Joachim -- Sent via pgsql-hackers mailing list

Re: [HACKERS] directory archive format for pg_dump

2010-12-07 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 2:52 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Ok, committed, with some small cleanup since the last patch I posted. Could you update the directory-format patch on top of the committed version, please? Thanks for committing the first part. Here

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Joachim Wieland
On Sun, Dec 5, 2010 at 9:27 PM, Robert Haas robertmh...@gmail.com wrote: On Sun, Dec 5, 2010 at 9:04 PM, Andrew Dunstan and...@dunslane.net wrote: Why not just say give me the snapshot currently held by process ? And please, not temp files if possible. As far as I'm aware, the full

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-02 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 6:19 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I don't see the point of the sort-by-relpages code. The order the objects are dumped should be irrelevant, as long as you obey the restrictions dictated by dependencies. Or is it only needed for the

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-02 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 12:56 PM, Josh Berkus j...@agliodbs.com wrote: Now, if only I could think of some way to write a parallel dump to a set of pipes, I'd be in heaven. What exactly are you trying to accomplish with the pipes? Joachim -- Sent via pgsql-hackers mailing list

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-02 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 9:33 PM, Tom Lane t...@sss.pgh.pa.us wrote: In particular, this issue *has* been discussed before, and there was a consensus that preserving dump consistency was a requirement.  I don't think that Joachim gets to bypass that decision just by submitting a patch that

Re: [HACKERS] directory archive format for pg_dump

2010-12-01 Thread Joachim Wieland
On Wed, Dec 1, 2010 at 9:05 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Forgot attachment. This is also available in the above git repo. I have quickly checked your modifications, on the one hand I like the reduction of functions, I would have said that we have AH around

Re: [HACKERS] Suggested easy TODO: pg_dump --from-list

2010-11-24 Thread Joachim Wieland
On Wed, Nov 24, 2010 at 1:15 AM, Tom Lane t...@sss.pgh.pa.us wrote: Nope ... those strings are just helpful comments, they aren't really guaranteed to be unique identifiers.  In any case, it seems unlikely that a user could expect to get the more complicated cases exactly right other than by

Re: [HACKERS] Suggested easy TODO: pg_dump --from-list

2010-11-24 Thread Joachim Wieland
On Wed, Nov 24, 2010 at 9:38 AM, Andrew Dunstan and...@dunslane.net wrote: It would be unique, but a pain in the neck for users to get. Robert's idea will have more traction with users. Whatever approach we use, we need to think about the use case where 1% of the objects should be dumped but

Re: [HACKERS] Suggested easy TODO: pg_dump --from-list

2010-11-23 Thread Joachim Wieland
On Tue, Nov 23, 2010 at 10:24 PM, Andrew Dunstan and...@dunslane.net wrote: Well, very little about pg_dump is very [E], IMNSHO. The question in my mind here is what format the list file will take. For example, how would we specify a function? Would we need to specify all the argument types (or

Re: [HACKERS] Hot Standby: too many KnownAssignedXids

2010-11-23 Thread Joachim Wieland
On Tue, Nov 23, 2010 at 8:45 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 19.11.2010 23:46, Joachim Wieland wrote: FATAL:  too many KnownAssignedXids. head: 0, tail: 0, nxids: 9978, pArray-maxKnownAssignedXids: 6890 Hmm, that's a lot of entries in KnownAssignedXids

Re: [HACKERS] Hot Standby: too many KnownAssignedXids

2010-11-22 Thread Joachim Wieland
On Sun, Nov 21, 2010 at 11:48 PM, Fujii Masao masao.fu...@gmail.com wrote: -- If you suspect a bug in Hot Standby, please set        trace_recovery_messages = DEBUG2 in postgresql.conf and repeat the action Always useful to know * max_connections * current number of sessions *

[HACKERS] Hot Standby: too many KnownAssignedXids

2010-11-19 Thread Joachim Wieland
Hi, I am seeing the following here on 9.0.1 on Linux x86-64: LOG: redo starts at 1F8/FC00E978 FATAL: too many KnownAssignedXids CONTEXT: xlog redo insert: rel 1663/16384/18373; tid 3829898/23 and this is the complete history: postgres was running as HS in foreground, Ctrl-C'ed it for a

Re: [HACKERS] directory archive format for pg_dump

2010-11-19 Thread Joachim Wieland
Hi Dimitri, thanks for reviewing my patch! On Fri, Nov 19, 2010 at 2:44 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: I think I'd like to see a separate patch for the new compression support. Sorry about that, I realize that's extra work… I guess it wouldn't be a very big deal but I also

Re: [HACKERS] directory archive format for pg_dump

2010-11-19 Thread Joachim Wieland
On Fri, Nov 19, 2010 at 11:53 PM, Tom Lane t...@sss.pgh.pa.us wrote: Dimitri Fontaine dimi...@2ndquadrant.fr writes: I think I'd like to see a separate patch for the new compression support. Sorry about that, I realize that's extra work… That part of the patch is likely to get rejected

Re: [HACKERS] directory archive format for pg_dump

2010-11-19 Thread Joachim Wieland
Hi Jose, 2010/11/19 José Arthur Benetasso Villanova jose.art...@gmail.com: The dir format generated in my database 60 files, with different sizes, and it looks very confusing. Is it possible to use the same trick as pigz and pbzip2, creating a concatenated file of streams? What pigz is

Re: [HACKERS] host name support in pg_hba.conf

2010-10-05 Thread Joachim Wieland
On Tue, Oct 5, 2010 at 3:41 PM, Peter Eisentraut pete...@gmx.net wrote: The reason why I think this is semi-important and not just cosmetic is that (for some reason that is not entirely clear to me) clients connecting to localhost end up appearing to the server as ::1 on a lot of machines I

Re: [HACKERS] Patch for PKST timezone

2010-05-16 Thread Joachim Wieland
On Wed, May 12, 2010 at 12:39 AM, Tom Lane t...@sss.pgh.pa.us wrote: Joachim Wieland j...@mcknight.de writes: Good we have found that inconsistency now, so let's add PKST. OK, done.  BTW, I notice that PKT was labeled (not in zic), which is not the case, per this discussion.  I seem to recall

Re: [HACKERS] Patch for PKST timezone

2010-05-11 Thread Joachim Wieland
On Tue, May 11, 2010 at 10:45 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I don't think we want to include all timezone names in the default config, timezone abbreviations aren't always unique for example. But we should include PKST because we already include PKT; it would

Re: [HACKERS] a faster compression algorithm for pg_dump

2010-04-10 Thread Joachim Wieland
On Fri, Apr 9, 2010 at 5:51 AM, Greg Stark gsst...@mit.edu wrote: Linking against as an option isn't nearly as bad since the user compiling it can choose whether to include the restricted feature or not. That's what we do with readline. However it's not nearly as attractive when it restricts

[HACKERS] a faster compression algorithm for pg_dump

2010-04-08 Thread Joachim Wieland
I'd like to revive the discussion about offering another compression algorithm than zlib to at least pg_dump. There has been a previous discussion here: http://archives.postgresql.org/pgsql-performance/2009-08/msg00053.php and it ended without any real result. The results so far were: - There

Re: [HACKERS] message clarifications

2010-04-03 Thread Joachim Wieland
On Sat, Apr 3, 2010 at 9:02 PM, Robert Haas robertmh...@gmail.com wrote: On Apr 3, 2010, at 11:13 AM, Tom Lane t...@sss.pgh.pa.us wrote: Peter Eisentraut pete...@gmx.net writes: The following messages from the postgres catalog either appear to be internal errors that should be marked

Re: [HACKERS] five-key syscaches

2010-03-29 Thread Joachim Wieland
On Mon, Mar 29, 2010 at 12:32 AM, Robert Haas robertmh...@gmail.com wrote: Per previous discussion, PFA a patch to change the maximum number of keys for a syscache from 4 to 5. http://archives.postgresql.org/pgsql-hackers/2010-02/msg01105.php This is intended for application to 9.1, and is

[HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Joachim Wieland
People have been talking about a parallel version of pg_dump a few times already. I have been working on some proof-of-concept code for this feature every now and then and I am planning to contribute this for 9.1. There are two main issues with a parallel version of pg_dump: The first one is

Re: [HACKERS] 9.0 release notes done

2010-03-22 Thread Joachim Wieland
On Sat, Mar 20, 2010 at 5:02 AM, Bruce Momjian br...@momjian.us wrote: Interestingly the 9.0 release notes contain 201 items, while the 8.4 release notes contained 314 items. Is the following pg_dump change covered by the release notes? I couldn't find it. It was the last committed patch from

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-28 Thread Joachim Wieland
On Sun, Feb 28, 2010 at 2:54 PM, Greg Stark gsst...@mit.edu wrote: Really? I think we get lots of suprised wows from the field from the idea that a long-running read-only query can cause your database to bloat. I think the only reason that's obvious to us is that we've been grappling with that

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-28 Thread Joachim Wieland
On Sun, Feb 28, 2010 at 8:47 PM, Josh Berkus j...@agliodbs.com wrote: 1) Automated retry of cancelled queries on the slave.  I have no idea how hard this would be to implement, but it makes the difference between writing lots of exception-handling code for slave connections (unacceptable) to

[HACKERS] Listen/Notify payload and interfaces

2010-02-17 Thread Joachim Wieland
This one is for the maintainers of the various postgresql interfaces: With the new listen/notify implementation we can send a payload along with the notification. This has been in the protocol already since years and there is no change needed for libpq. However we need to adapt the various

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-17 Thread Joachim Wieland
On Tue, Feb 16, 2010 at 11:41 PM, Tom Lane t...@sss.pgh.pa.us wrote: Joachim Wieland j...@mcknight.de writes: [ listen/notify patch ] I found a number of implementation problems having to do with wraparound behavior and error recovery.  I think they're all fixed, but any remaining bugs

Re: [HACKERS] LISTEN/NOTIFY and notification timing guarantees

2010-02-16 Thread Joachim Wieland
On Tue, Feb 16, 2010 at 6:20 AM, Tom Lane t...@sss.pgh.pa.us wrote: Another possibility is to force a ProcessIncomingNotifies scan to occur before we reach ReadyForQuery if we sent any notifies in the just-finished transaction --- but that won't help if there are uncommitted messages in front

Re: [HACKERS] LISTEN/NOTIFY and notification timing guarantees

2010-02-16 Thread Joachim Wieland
On Tue, Feb 16, 2010 at 1:31 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Tom Lane  wrote: We could adopt the historical policy of sending self-notifies pre-commit, but that doesn't seem tremendously appetizing from the standpoint of transactional integrity. But one traditional

Re: [HACKERS] LISTEN/NOTIFY and notification timing guarantees

2010-02-15 Thread Joachim Wieland
On Mon, Feb 15, 2010 at 3:31 AM, Tom Lane t...@sss.pgh.pa.us wrote: I'm not sure how probable it is that applications might be coded in a way that relies on the properties lost according to point #2 or #3. Your observations are all correct as far as I can tell. One question regarding #2: Is a

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-15 Thread Joachim Wieland
On Sun, Feb 14, 2010 at 11:44 PM, Simon Riggs si...@2ndquadrant.com wrote: Next set of questions * Will this work during Hot Standby now? The barrier was that it wrote to a table and so we could not allow that. ISTM this new version can and should work with Hot Standby. Can you test that and

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-15 Thread Joachim Wieland
On Mon, Feb 15, 2010 at 1:48 PM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2010-02-15 at 12:59 +0100, Joachim Wieland wrote: I have tested it already. The point where it currently fails is the following line:       qe-xid = GetCurrentTransactionId(); That's a shame. So it will never

Re: [HACKERS] Parameter name standby_mode

2010-02-12 Thread Joachim Wieland
On Fri, Feb 12, 2010 at 8:59 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Agreed. I've changed it now so that if primary_conninfo is not set, it doesn't try to establish a streaming connection. If you want to get the connection information from environment variables, you

  1   2   >