[HACKERS] pg_dump bug in 9.4beta2 and HEAD

2014-08-13 Thread Joachim Wieland
I'm seeing an assertion failure with "pg_dump -c --if-exists" which is not ready to handle BLOBs it seems: pg_dump: pg_backup_archiver.c:472: RestoreArchive: Assertion `mark != ((void *)0)' failed. To reproduce: $ createdb test $ pg_dump -c --if-exists test (works, dumps empty database) $ psql

Re: [HACKERS] patch for parallel pg_dump

2012-01-29 Thread Joachim Wieland
On Fri, Jan 27, 2012 at 4:57 PM, Robert Haas wrote: > But even if you do know that subclassing > is intended, that doesn't prove that the particular Archive object is > always going to be an ArchiveHandle under the hood.  If it is, why not > just pass it as an ArchiveHandle to begin with? I know

Re: [HACKERS] patch for parallel pg_dump

2012-01-30 Thread Joachim Wieland
On Mon, Jan 30, 2012 at 12:20 PM, Robert Haas wrote: > But the immediate problem is that pg_dump.c is heavily reliant on > global variables, which isn't going to fly if we want this code to use > threads on Windows (or anywhere else).  It's also bad style. Technically, since most of pg_dump.c dum

Re: [HACKERS] patch for parallel pg_dump

2012-01-31 Thread Joachim Wieland
On Tue, Jan 31, 2012 at 9:05 AM, Robert Haas wrote: > And just for added fun and excitement, they all have inconsistent > naming conventions and inadequate documentation. > > I think if we need more refactoring in order to support multiple > database connections, we should go do that refactoring.

Re: [HACKERS] patch for parallel pg_dump

2012-02-02 Thread Joachim Wieland
On Wed, Feb 1, 2012 at 12:24 PM, Robert Haas wrote: > I think we're more-or-less proposing to rename "Archive" to > "Connection", aren't we? > > And then ArchiveHandle can store all the things that aren't related to > a specific connection. How about something like that: (Hopefully you'll come up

Re: [HACKERS] patch for parallel pg_dump

2012-02-03 Thread Joachim Wieland
On Fri, Feb 3, 2012 at 7:52 AM, Robert Haas wrote: > If you're OK with that much I'll go do it. Sure, go ahead! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch for parallel pg_dump

2012-02-07 Thread Joachim Wieland
On Tue, Feb 7, 2012 at 4:59 PM, Robert Haas wrote: > It turns out that (as you anticipated) there are some problems with my > previous proposal. I assume you're talking to Tom, as my powers of anticipation are actually quite limited... :-) > This is not > quite enough to get rid of g_conn, but i

Re: [HACKERS] patch for parallel pg_dump

2012-02-08 Thread Joachim Wieland
On Wed, Feb 8, 2012 at 1:21 PM, Robert Haas wrote: >> In my patch I dealt with exactly the same problem for the error >> handler by creating a separate function that has a static variable (a >> pointer to the ParallelState). The value is set and retrieved through >> the same function, so yes, it's

Re: [HACKERS] patch for parallel pg_dump

2012-02-23 Thread Joachim Wieland
On Thu, Feb 16, 2012 at 1:29 PM, Robert Haas wrote: > Can you provide an updated patch? Robert, updated patch is attached. parallel_pg_dump_2.diff.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] patch for parallel pg_dump

2012-03-13 Thread Joachim Wieland
On Tue, Mar 13, 2012 at 1:53 PM, Robert Haas wrote: > What I mean is that the function ArchiveEntry() is defined in > pg_backup_archiver.c, and it takes an argument called relpages, and > the string "relpages" does not appear anywhere else in that file. Uhm, that's kinda concerning, isn't it... f

Re: [HACKERS] patch for parallel pg_dump

2012-03-14 Thread Joachim Wieland
On Wed, Mar 14, 2012 at 2:02 PM, Robert Haas wrote: > I think we should get rid of die_horribly(), and instead have arrange > to always clean up AH via an on_exit_nicely hook. Good. The only exit handler I've seen so far is pgdump_cleanup_at_exit. If there's no other one, is it okay to remove all

Re: [HACKERS] patch for parallel pg_dump

2012-03-14 Thread Joachim Wieland
On Wed, Mar 14, 2012 at 4:39 PM, Andrew Dunstan wrote: > I've just started looking at the patch, and I'm curious to know why it > didn't follow the pattern of parallel pg_restore which created a new worker > for each table rather than passing messages to looping worker threads as > this appears to

[HACKERS] double free in current HEAD's pg_dump

2012-03-17 Thread Joachim Wieland
There's a double free in the current HEAD's pg_dump. Fix attached. diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 2b0a5ff..57a6ccb 100644 *** a/src/bin/pg_dump/pg_dump.c --- b/src/bin/pg_dump/pg_dump.c *** dumpBlobs(Archive *fout, void *arg) *** 2372,2379

Re: [HACKERS] patch for parallel pg_dump

2012-03-17 Thread Joachim Wieland
On Fri, Mar 16, 2012 at 12:06 AM, Robert Haas wrote: >> Good. The only exit handler I've seen so far is >> pgdump_cleanup_at_exit. If there's no other one, is it okay to remove >> all of this stacking functionality (see on_exit_nicely_index / >> MAX_ON_EXIT_NICELY) from dumputils.c and just define

Re: [HACKERS] patch for parallel pg_dump

2012-03-23 Thread Joachim Wieland
On Fri, Mar 23, 2012 at 11:11 AM, Alvaro Herrera wrote: > Are you going to provide a rebased version? Yes, working on that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch for parallel pg_dump

2012-03-25 Thread Joachim Wieland
On Fri, Mar 23, 2012 at 11:11 AM, Alvaro Herrera wrote: > Are you going to provide a rebased version? Rebased version attached, this patch also includes Robert's earlier suggestions. parallel_pg_dump_5.diff.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] patch for parallel pg_dump

2012-03-28 Thread Joachim Wieland
On Wed, Mar 28, 2012 at 5:19 PM, Andrew Dunstan wrote: > First hurdle: It doesn't build under Windows/mingw-w64: > >   parallel.c:40:12: error: static declaration of 'pgpipe' follows >   non-static declaration Strange, I'm not seeing this but I'm building with VC2005. What happens is that you're

Re: [HACKERS] patch for parallel pg_dump

2012-03-28 Thread Joachim Wieland
On Thu, Mar 29, 2012 at 2:46 AM, Andrew Dunstan wrote: > But your patch hasn't got rid of them, and so it's declared twice. There is > no pgpipe.h, BTW, it's declared in port.h. If VC2005 doesn't complain about > the double declaration then that's a bug in the compiler, IMNSHO. Doesn't it > even i

Re: [HACKERS] patch for parallel pg_dump

2012-03-28 Thread Joachim Wieland
On Wed, Mar 28, 2012 at 1:46 PM, Robert Haas wrote: > I'm wondering if we really need this much complexity around shutting > down workers.  I'm not sure I understand why we need both a "hard" and > a "soft" method of shutting them down.  At least on non-Windows > systems, it seems like it would be

Re: [HACKERS] patch for parallel pg_dump

2012-04-03 Thread Joachim Wieland
On Tue, Apr 3, 2012 at 9:34 AM, Robert Haas wrote: > On Sun, Apr 1, 2012 at 12:35 PM, Joachim Wieland wrote: >> Unfortunately this is not really the case. What is being moved out of >> pg_backup_archiver.c and into parallel.c is either the shutdown logic >> that has been ap

Re: [HACKERS] parallel pg_dump

2012-04-04 Thread Joachim Wieland
On Tue, Apr 3, 2012 at 9:26 AM, Andrew Dunstan wrote: > First, either the creation of the destination directory needs to be delayed > until all the sanity checks have passed and we're sure we're actually going > to write something there, or it needs to be removed if we error exit before > anything

Re: [HACKERS] patch for parallel pg_dump

2012-04-04 Thread Joachim Wieland
On Tue, Apr 3, 2012 at 11:04 AM, Robert Haas wrote: > OK, but it seems like a pretty fragile assumption that none of the > workers will ever manage to emit any other error messages.  We don't > rely on this kind of assumption in the backend, which is a lot > better-structured and less spaghetti-li

Re: [HACKERS] parallel pg_dump

2012-12-08 Thread Joachim Wieland
On Sat, Dec 8, 2012 at 3:05 PM, Bruce Momjian wrote: > On Sat, Dec 8, 2012 at 11:13:30AM -0500, Andrew Dunstan wrote: > > I am working on it when I get a chance, but keep getting hammered. > > I'd love somebody else to review it too. > > FYI, I will be posting pg_upgrade performance numbers usin

Re: [HACKERS] Logical decoding & exported base snapshot

2012-12-11 Thread Joachim Wieland
On Tue, Dec 11, 2012 at 6:52 PM, Andres Freund wrote: > One problem I see is that while exporting a snapshot solves the > visibility issues of the table's contents it does not protect against > schema changes. I am not sure whether thats a problem. > > If somebody runs a CLUSTER or something like

Re: [HACKERS] Suggested "easy" TODO: pg_dump --from-list

2010-11-24 Thread Joachim Wieland
On Wed, Nov 24, 2010 at 1:15 AM, Tom Lane wrote: > Nope ... those strings are just helpful comments, they aren't really > guaranteed to be unique identifiers.  In any case, it seems unlikely > that a user could expect to get the more complicated cases exactly right > other than by consulting "pg_d

Re: [HACKERS] Suggested "easy" TODO: pg_dump --from-list

2010-11-24 Thread Joachim Wieland
On Wed, Nov 24, 2010 at 9:38 AM, Andrew Dunstan wrote: > It would be unique, but a pain in the neck for users to get. Robert's idea > will have more traction with users. Whatever approach we use, we need to think about the use case where 1% of the objects should be dumped but should also make sur

Re: [HACKERS] directory archive format for pg_dump

2010-12-01 Thread Joachim Wieland
On Wed, Dec 1, 2010 at 9:05 AM, Heikki Linnakangas wrote: > Forgot attachment. This is also available in the above git repo. I have quickly checked your modifications, on the one hand I like the reduction of functions, I would have said that we have AH around all the time and so we could just all

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-02 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 6:19 AM, Heikki Linnakangas wrote: > I don't see the point of the sort-by-relpages code. The order the objects > are dumped should be irrelevant, as long as you obey the restrictions > dictated by dependencies. Or is it only needed for the multiple-target-dirs > feature? Fra

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-02 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 12:56 PM, Josh Berkus wrote: > Now, if only I could think of some way to write a parallel dump to a set of > pipes, I'd be in heaven. What exactly are you trying to accomplish with the pipes? Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) T

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-02 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 9:33 PM, Tom Lane wrote: > In particular, this issue *has* been discussed before, and there was a > consensus that preserving dump consistency was a requirement.  I don't > think that Joachim gets to bypass that decision just by submitting a > patch that ignores it. I am no

Re: [HACKERS] WIP patch for parallel pg_dump

2010-12-05 Thread Joachim Wieland
On Sun, Dec 5, 2010 at 9:27 PM, Robert Haas wrote: > On Sun, Dec 5, 2010 at 9:04 PM, Andrew Dunstan wrote: >> Why not just say give me the snapshot currently held by process ? >> >> And please, not temp files if possible. > > As far as I'm aware, the full snapshot doesn't normally exist in >

Re: [HACKERS] directory archive format for pg_dump

2010-12-07 Thread Joachim Wieland
On Thu, Dec 2, 2010 at 2:52 PM, Heikki Linnakangas wrote: > Ok, committed, with some small cleanup since the last patch I posted. > > Could you update the directory-format patch on top of the committed version, > please? Thanks for committing the first part. Here is the updated and rebased direct

Re: [HACKERS] Hot Standby: too many KnownAssignedXids

2010-12-15 Thread Joachim Wieland
On Tue, Dec 7, 2010 at 3:42 AM, Heikki Linnakangas wrote: > Ok, I've committed this patch now. I can confirm that I could continue replaying the logfiles on the standby host with this patch. Thanks a lot, Joachim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make ch

Re: [HACKERS] directory archive format for pg_dump

2010-12-16 Thread Joachim Wieland
On Thu, Dec 16, 2010 at 12:48 PM, Heikki Linnakangas wrote: > As soon as we have parallel pg_dump, the next big thing is going to be > parallel dump of the same table using multiple processes. Perhaps we should > prepare for that in the directory archive format, by allowing the data of a > single

Re: [HACKERS] page compression

2010-12-28 Thread Joachim Wieland
On Tue, Dec 28, 2010 at 10:10 AM, Andy Colson wrote: > I know its been discussed before, and one big problem is license and patent > problems. > > Would this project be a problem: > > http://oldhome.schmorp.de/marc/liblzf.html It looks like even liblzf is not going to be accepted. I have proposed

[HACKERS] Snapshot synchronization, again...

2010-12-30 Thread Joachim Wieland
The snapshot synchronization discussion from the parallel pg_dump patch somehow ended without a clear way to go forward. Let me sum up what has been brought up and propose a short- and longterm solution. Summary: Passing snapshot sync information can be done either: a) by returning complete sna

Re: [HACKERS] Snapshot synchronization, again...

2010-12-30 Thread Joachim Wieland
On Thu, Dec 30, 2010 at 9:40 AM, Alvaro Herrera wrote: >> Disadvantage of b: It doesn't allow a snapshot to be installed on a >> different server. It requires a serializable open transaction to hold >> the snapshot. > > Why does it require a serializable transaction?  You could simply > register t

Re: [HACKERS] Snapshot synchronization, again...

2010-12-30 Thread Joachim Wieland
On Thu, Dec 30, 2010 at 9:49 AM, Florian Pflug wrote: > On Dec30, 2010, at 13:31 , Joachim Wieland wrote: >> We return snapshot information as a chunk of data to the client. At >> the same time however, we set a checksum in shared memory to protect >> against modificati

Re: [HACKERS] Snapshot synchronization, again...

2010-12-31 Thread Joachim Wieland
On Fri, Dec 31, 2010 at 8:28 AM, Alvaro Herrera wrote: > A backend can have any number of snapshots registered, and those don't > allow GlobalXmin to advance. Consider an open cursor, for example. > Even if the rest of the transaction is read committed, the snapshot > registered by the open curso

Re: [HACKERS] Snapshot synchronization, again...

2011-01-07 Thread Joachim Wieland
On Thu, Dec 30, 2010 at 7:31 AM, Joachim Wieland wrote: > What I am proposing now is the following: > > We return snapshot information as a chunk of data to the client. At > the same time however, we set a checksum in shared memory to protect > against modification of the snapsho

Re: [HACKERS] Snapshot synchronization, again...

2011-01-18 Thread Joachim Wieland
Noah, thank you for this excellent review. I have taken over most (allmost all) of your suggestions (except for the documentation which is still missing). Please check the updated & attached patch for details. On Fri, Jan 14, 2011 at 10:13 PM, Noah Misch wrote: > "" is a valid

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-19 Thread Joachim Wieland
On Wed, Jan 19, 2011 at 7:47 AM, Heikki Linnakangas wrote: >> Here are the latest patches all of them also rebased to current HEAD. >> Will update the commitfest app as well. > > What's the idea of storing the file sizes in the toc file? It looks like > it's not used for anything. It's part of th

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-01-20 Thread Joachim Wieland
On Thu, Jan 20, 2011 at 6:07 AM, Heikki Linnakangas wrote: >> It's part of the overall idea to make sure files are not inadvertently >> exchanged between different backups and that a file is not truncated. >> In the future I'd also like to add a checksum to the TOC so that a >> backup can be check

[HACKERS] posix_fadvise missing in the walsender

2013-02-17 Thread Joachim Wieland
In access/transam/xlog.c we give the OS buffer caching a hint that we won't need a WAL file any time soon with posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); before closing the WAL file, but only if we don't have walsenders. That's reasonable because the walsender will reopen that sam

Re: [HACKERS] posix_fadvise missing in the walsender

2013-02-19 Thread Joachim Wieland
On Tue, Feb 19, 2013 at 5:48 PM, Simon Riggs wrote: In access/transam/xlog.c we give the OS buffer caching a hint that we won't need a WAL file any time soon with posix_fadvise(openLogFile, 0, 0, POSIX_FADV_DONTNEED); > > I agree with Merlin and Joachim - if we have t

Re: [HACKERS] posix_fadvise missing in the walsender

2013-02-20 Thread Joachim Wieland
On Wed, Feb 20, 2013 at 4:54 PM, Robert Haas wrote: > On Tue, Feb 19, 2013 at 5:48 PM, Simon Riggs wrote: >> I agree with Merlin and Joachim - if we have the call in one place, we >> should have it in both. > > We might want to assess whether we even want to have it one place. > I've seen cases w

[HACKERS] Materialized view assertion failure in HEAD

2013-03-05 Thread Joachim Wieland
I'm getting an assertion failure in HEAD with materialized views, see below for backtrace. To reproduce, just run make installcheck, dump the regression database and then restore it, the server crashes during restore. (gdb) bt #0 0x7f283a366425 in raise () from /lib/x86_64-linux-gnu/libc.so.

Re: [HACKERS] Materialized view assertion failure in HEAD

2013-03-05 Thread Joachim Wieland
On Tue, Mar 5, 2013 at 9:11 AM, Kevin Grittner wrote: > Will investigate. > You don't have default_with_oids = on, do you? No, this was a default install with a default postgresql.conf -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: h

Re: [HACKERS] Optimizing pglz compressor

2013-03-06 Thread Joachim Wieland
On Tue, Mar 5, 2013 at 8:32 AM, Heikki Linnakangas wrote: > With these tweaks, I was able to make pglz-based delta encoding perform > roughly as well as Amit's patch. Out of curiosity, do we know how pglz compares with other algorithms, e.g. lz4 ? -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] Allowing parallel pg_restore from pipe

2013-04-24 Thread Joachim Wieland
On Wed, Apr 24, 2013 at 4:05 PM, Stefan Kaltenbrunner < ste...@kaltenbrunner.cc> wrote: > > What might make sense is something like pg_dump_restore which would have > > no intermediate storage at all, just pump the data etc from one source > > to another in parallel. But I pity the poor guy who ha

[HACKERS] Review for pg_dump: Sort overloaded functions in deterministic order

2012-09-25 Thread Joachim Wieland
Patch looks good, all concerns that had been raised previously have been addressed in this version of the patch. The only thing that IMO needs to change is a stylistic issue: if (fout->remoteVersion >= 80200) { [...] (fout->remoteVersion >= 80400) ? "pg_catalog.pg_get_function_identity_ar

Re: [HACKERS] Add FET to Default and Europe.txt

2012-10-08 Thread Joachim Wieland
On Mon, Oct 8, 2012 at 4:00 PM, Tom Lane wrote: > We can't just refuse to deal with this ambiguity. We have to have some > very-low-pain way to install settings that will please those large > fractions of our user base. Moreover, if that very-low-pain way isn't > the exact same way it's been don

Re: [HACKERS] [PATCH] pg_dump: Sort overloaded functions in deterministic order

2012-10-17 Thread Joachim Wieland
On Wed, Oct 17, 2012 at 5:43 PM, Alvaro Herrera wrote: > (I tested the new pg_dump with 8.2 and HEAD and also verified it passes > pg_upgrade's "make check". I didn't test with other server versions.) I also tested against 8.3 and 8.4 since 8.4 is the version that introduced pg_get_function_iden

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-03 Thread Joachim Wieland
On Wed, Feb 3, 2010 at 2:05 AM, Jeff Davis wrote: >> Thanks, very well spotted... Actually the same is true for LISTEN... I >> have reworked the patch to do the changes to listenChannels only in >> the post-commit functions. > > I'm worried that this creates the opposite problem: that a LISTEN > t

Re: [HACKERS] Pathological regexp match

2010-02-09 Thread Joachim Wieland
On Mon, Feb 8, 2010 at 6:07 PM, David E. Wheeler wrote: > On Feb 8, 2010, at 5:15 AM, Magnus Hagander wrote: > >> The text is about 180Kb. PostgreSQL takes ~40 seconds without the >> patch, ~36 seconds with it, to extract the match from it. Perl takes >> 0.016 seconds. > > Obviously we need to sup

Re: [HACKERS] synchronized snapshots

2010-02-10 Thread Joachim Wieland
Hi Markus, On Fri, Feb 5, 2010 at 6:29 PM, Markus Wanner wrote: > > So, let's first concentrate on the intended use case: allowing parallel > pg_dump. To me it seems like a pragmatic and quick solution, however, I'm > not sure if requiring superuser privileges is acceptable. http://www.postgresq

[HACKERS] Parameter name standby_mode

2010-02-10 Thread Joachim Wieland
We want to teach people that Hot Standby and Streaming Replication are two different features. However, Streaming Replication calls its main parameter "standby_mode" which reminds more of Hot Standby than of Streaming Replication. People could also run a warm standby without streaming replication,

Re: [HACKERS] Parameter name standby_mode

2010-02-10 Thread Joachim Wieland
On Wed, Feb 10, 2010 at 12:16 PM, Heikki Linnakangas wrote: > If they want to implement the warm standby using the (new) built-in > logic to keep retrying restore_command, they would set > standby_mode='on'. standby_mode='on' doesn't imply streaming replication. > > If you want to use pg_standby o

Re: [HACKERS] Parameter name standby_mode

2010-02-11 Thread Joachim Wieland
On Fri, Feb 12, 2010 at 7:28 AM, Fujii Masao wrote: > Yeah, even if primary_conninfo is not given, the standby tries to invoke > walreceiver by using the another connection settings (environment variables > or defaults). This is intentional behavior, and would make the setup of SR > easier. So I'd

Re: [HACKERS] Parameter name standby_mode

2010-02-12 Thread Joachim Wieland
On Fri, Feb 12, 2010 at 8:59 AM, Heikki Linnakangas wrote: > Agreed. I've changed it now so that if primary_conninfo is not set, it > doesn't try to establish a streaming connection. If you want to get the > connection information from environment variables, you can use > primary_conninfo=''. Why

Re: [HACKERS] LISTEN/NOTIFY and notification timing guarantees

2010-02-15 Thread Joachim Wieland
On Mon, Feb 15, 2010 at 3:31 AM, Tom Lane wrote: > I'm not sure how probable it is that applications might be coded in a > way that relies on the properties lost according to point #2 or #3. Your observations are all correct as far as I can tell. One question regarding #2: Is a client applicatio

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-15 Thread Joachim Wieland
On Sun, Feb 14, 2010 at 11:44 PM, Simon Riggs wrote: > Next set of questions > > * Will this work during Hot Standby now? The barrier was that it wrote > to a table and so we could not allow that. ISTM this new version can and > should work with Hot Standby. Can you test that and if so, remove the

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-15 Thread Joachim Wieland
On Mon, Feb 15, 2010 at 1:48 PM, Simon Riggs wrote: > On Mon, 2010-02-15 at 12:59 +0100, Joachim Wieland wrote: >> I have tested it already. The point where it currently fails is the >> following line: >> >>       qe->xid = GetCurrentTransactionId(); > > That

Re: [HACKERS] LISTEN/NOTIFY and notification timing guarantees

2010-02-16 Thread Joachim Wieland
On Tue, Feb 16, 2010 at 6:20 AM, Tom Lane wrote: > Another possibility is to force a ProcessIncomingNotifies scan to occur > before we reach ReadyForQuery if we sent any notifies in the > just-finished transaction --- but that won't help if there are > uncommitted messages in front of ours. What

Re: [HACKERS] LISTEN/NOTIFY and notification timing guarantees

2010-02-16 Thread Joachim Wieland
On Tue, Feb 16, 2010 at 1:31 PM, Kevin Grittner wrote: > Tom Lane  wrote: >> We could adopt the historical policy of sending self-notifies >> pre-commit, but that doesn't seem tremendously appetizing from the >> standpoint of transactional integrity. > > But one traditional aspect of transactional

[HACKERS] Listen/Notify payload and interfaces

2010-02-17 Thread Joachim Wieland
This one is for the maintainers of the various postgresql interfaces: With the new listen/notify implementation we can send a payload along with the notification. This has been in the protocol already since years and there is no change needed for libpq. However we need to adapt the various interfa

Re: [HACKERS] Listen / Notify - what to do when the queue is full

2010-02-17 Thread Joachim Wieland
On Tue, Feb 16, 2010 at 11:41 PM, Tom Lane wrote: > Joachim Wieland writes: >> [ listen/notify patch ] > > I found a number of implementation problems having to do with wraparound > behavior and error recovery.  I think they're all fixed, but any > remaining bugs are p

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-28 Thread Joachim Wieland
On Sun, Feb 28, 2010 at 2:54 PM, Greg Stark wrote: > Really? I think we get lots of suprised wows from the field from the > idea that a long-running read-only query can cause your database to > bloat. I think the only reason that's obvious to us is that we've been > grappling with that problem for

Re: [HACKERS] Re: Hot Standby query cancellation and Streaming Replication integration

2010-02-28 Thread Joachim Wieland
On Sun, Feb 28, 2010 at 8:47 PM, Josh Berkus wrote: > 1) Automated retry of cancelled queries on the slave.  I have no idea > how hard this would be to implement, but it makes the difference between > writing lots of exception-handling code for slave connections > (unacceptable) to just slow respo

Re: [HACKERS] 9.0 release notes done

2010-03-22 Thread Joachim Wieland
On Sat, Mar 20, 2010 at 5:02 AM, Bruce Momjian wrote: > Interestingly the 9.0 release notes contain 201 items, while the 8.4 > release notes contained 314 items. Is the following pg_dump change covered by the release notes? I couldn't find it. It was the last committed patch from the 2010-01 comm

Re: [HACKERS] five-key syscaches

2010-03-29 Thread Joachim Wieland
On Mon, Mar 29, 2010 at 12:32 AM, Robert Haas wrote: > Per previous discussion, PFA a patch to change the maximum number of > keys for a syscache from 4 to 5. > > http://archives.postgresql.org/pgsql-hackers/2010-02/msg01105.php > > This is intended for application to 9.1, and is supporting > infr

[HACKERS] Parallel pg_dump for 9.1

2010-03-29 Thread Joachim Wieland
People have been talking about a parallel version of pg_dump a few times already. I have been working on some proof-of-concept code for this feature every now and then and I am planning to contribute this for 9.1. There are two main issues with a parallel version of pg_dump: The first one is that

Re: [HACKERS] message clarifications

2010-04-03 Thread Joachim Wieland
On Sat, Apr 3, 2010 at 9:02 PM, Robert Haas wrote: > On Apr 3, 2010, at 11:13 AM, Tom Lane wrote: >> Peter Eisentraut writes: >>> The following messages from the postgres catalog either appear to be >>> internal errors that should be marked differently, or they are in my >>> estimation unintelli

[HACKERS] a faster compression algorithm for pg_dump

2010-04-08 Thread Joachim Wieland
I'd like to revive the discussion about offering another compression algorithm than zlib to at least pg_dump. There has been a previous discussion here: http://archives.postgresql.org/pgsql-performance/2009-08/msg00053.php and it ended without any real result. The results so far were: - There ex

Re: [HACKERS] a faster compression algorithm for pg_dump

2010-04-10 Thread Joachim Wieland
On Fri, Apr 9, 2010 at 5:51 AM, Greg Stark wrote: > Linking against as an option isn't nearly as bad since the user > compiling it can choose whether to include the restricted feature or > not. That's what we do with readline. However it's not nearly as > attractive when it restricts what file for

Re: [HACKERS] Patch for PKST timezone

2010-05-11 Thread Joachim Wieland
On Tue, May 11, 2010 at 10:45 AM, Heikki Linnakangas wrote: > I don't think we want to include all timezone names in the default > config, timezone abbreviations aren't always unique for example. But we > should include PKST because we already include PKT; it would be nasty > for an application to

Re: [HACKERS] Patch for PKST timezone

2010-05-16 Thread Joachim Wieland
On Wed, May 12, 2010 at 12:39 AM, Tom Lane wrote: > Joachim Wieland writes: >> Good we have found that inconsistency now, so let's add PKST. > > OK, done.  BTW, I notice that PKT was labeled "(not in zic)", which > is not the case, per this discussion.  I see

[HACKERS] "could not open relation with OID" errors after promoting the standby to master

2012-05-15 Thread Joachim Wieland
I've switched servers yesterday night and the previous slave is now the master. This is 9.0.6 (originally) / 9.0.7 (now) on Linux. Now I'm seeing a bunch of ERROR: could not open relation with OID 1990987633 STATEMENT: create temp table seen_files (fileid integer) Interestingly enough, 90% of

Re: [HACKERS] "could not open relation with OID" errors after promoting the standby to master

2012-05-17 Thread Joachim Wieland
On Wed, May 16, 2012 at 11:38 PM, Alvaro Herrera wrote: > Well, that is not surprising in itself -- InitTempTableNamespace calls > RemoveTempRelations to cleanup from a possibly crashed previous backend > with the same ID.  So that part of the backtrace looks normal to me > (unless there is someth

Re: [HACKERS] "could not open relation with OID" errors after promoting the standby to master

2012-05-24 Thread Joachim Wieland
On Tue, May 22, 2012 at 9:50 AM, Robert Haas wrote: > Hmm.  I think that if you do it this way, the minimum recovery point > won't be respected, which could leave you with a corrupted database. > Now, if all the WAL files that you need are present in pg_xlog anyway, > then they ought to get replay

Re: [HACKERS] host name support in pg_hba.conf

2010-10-05 Thread Joachim Wieland
On Tue, Oct 5, 2010 at 3:41 PM, Peter Eisentraut wrote: > The reason why I think this is semi-important and not just cosmetic is > that (for some reason that is not entirely clear to me) clients > connecting to localhost end up appearing to the server as ::1 on a lot > of machines I use which are

[HACKERS] Hot Standby: too many KnownAssignedXids

2010-11-19 Thread Joachim Wieland
Hi, I am seeing the following here on 9.0.1 on Linux x86-64: LOG: redo starts at 1F8/FC00E978 FATAL: too many KnownAssignedXids CONTEXT: xlog redo insert: rel 1663/16384/18373; tid 3829898/23 and this is the complete history: postgres was running as HS in foreground, Ctrl-C'ed it for a rest

Re: [HACKERS] directory archive format for pg_dump

2010-11-19 Thread Joachim Wieland
Hi Dimitri, thanks for reviewing my patch! On Fri, Nov 19, 2010 at 2:44 PM, Dimitri Fontaine wrote: > I think I'd like to see a separate patch for the new compression > support. Sorry about that, I realize that's extra work… I guess it wouldn't be a very big deal but I also doubt that it makes

Re: [HACKERS] directory archive format for pg_dump

2010-11-19 Thread Joachim Wieland
On Fri, Nov 19, 2010 at 11:53 PM, Tom Lane wrote: > Dimitri Fontaine writes: > > I think I'd like to see a separate patch for the new compression > > support. Sorry about that, I realize that's extra work… > > That part of the patch is likely to get rejected outright anyway, > so I *strongly* re

Re: [HACKERS] directory archive format for pg_dump

2010-11-19 Thread Joachim Wieland
Hi Jose, 2010/11/19 José Arthur Benetasso Villanova : > The dir format generated in my database 60 files, with different > sizes, and it looks very confusing. Is it possible to use the same > trick as pigz and pbzip2, creating a concatenated file of streams? What pigz is parallelizing is the actu

Re: [HACKERS] Hot Standby: too many KnownAssignedXids

2010-11-22 Thread Joachim Wieland
On Sun, Nov 21, 2010 at 11:48 PM, Fujii Masao wrote: > -- > If you suspect a bug in Hot Standby, please set >        trace_recovery_messages = DEBUG2 > in postgresql.conf and repeat the action > > Always useful to know > * max_connections > * current number of sessions > * whether we h

Re: [HACKERS] Suggested "easy" TODO: pg_dump --from-list

2010-11-23 Thread Joachim Wieland
On Tue, Nov 23, 2010 at 10:24 PM, Andrew Dunstan wrote: > Well, very little about pg_dump is very [E], IMNSHO. The question in my mind > here is what format the list file will take. For example, how would we > specify a function? Would we need to specify all the argument types (or at > least the I

Re: [HACKERS] Hot Standby: too many KnownAssignedXids

2010-11-23 Thread Joachim Wieland
On Tue, Nov 23, 2010 at 8:45 AM, Heikki Linnakangas wrote: > On 19.11.2010 23:46, Joachim Wieland wrote: >> >> FATAL:  too many KnownAssignedXids. head: 0, tail: 0, nxids: 9978, >> pArray->maxKnownAssignedXids: 6890 > > Hmm, that's a lot of entries in KnownAssig

Re: [HACKERS] Snapshot synchronization, again...

2011-01-30 Thread Joachim Wieland
Here is a new version of the patch incorporating most of Noah's suggestions. The patch now also adds documentation. Since I couldn't really find a suitable section to document the two new functions, I added a new one for now. Any better ideas where it should go? On Thu, Jan 20, 2011 at 1:37 AM, No

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-01 Thread Joachim Wieland
On Sun, Jan 30, 2011 at 5:26 PM, Robert Haas wrote: > The parallel pg_dump portion of this patch (i.e. the still-uncommitted > part) no longer applies.  Please rebase. Here is a rebased version with some minor changes as well. I haven't tested it on Windows now but will do so as soon as the Unix

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-04 Thread Joachim Wieland
On Thu, Feb 3, 2011 at 11:46 PM, Itagaki Takahiro wrote: > I think we have 2 important technical issues here: >  * The consistency is not perfect. Each transaction is started >   with small delays in step 1, but we cannot guarantee no other >   transaction between them. This is exactly where the

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-07 Thread Joachim Wieland
Hi Jaime, thanks for your review! On Sun, Feb 6, 2011 at 2:12 PM, Jaime Casanova wrote: > code review: > > something i found, and is a very simple one, is this warning (there's > a similar issue in _StartMasterParallel with the buf variable) > """ > pg_backup_directory.c: In function ‘_EndMaster

Re: [HACKERS] pg_dump directory archive format / parallel pg_dump

2011-02-08 Thread Joachim Wieland
On Tue, Feb 8, 2011 at 8:31 PM, Itagaki Takahiro wrote: > On Tue, Feb 8, 2011 at 13:34, Robert Haas wrote: >> So how close are we to having a committable version of this?  Should >> we push this out to 9.2? > > I think so. The feature is pretty attractive, but more works are required: >  * Re-bas

Re: [HACKERS] Snapshot synchronization, again...

2011-02-18 Thread Joachim Wieland
On Fri, Feb 18, 2011 at 8:57 PM, Alvaro Herrera wrote: > 1. why are you using the expansible char array stuff instead of using > the StringInfo facility? > > 2. is md5 the most appropriate digest for this?  If you need a > cryptographically secure hash, do we need something stronger?  If not, > wh

Re: [HACKERS] Snapshot synchronization, again...

2011-02-19 Thread Joachim Wieland
On Sat, Feb 19, 2011 at 9:17 PM, Peter Eisentraut wrote: > The only consideration against MD5 might be > that it would make us look quite lame.  We should probably provide > builtin SHA1 and SHA2 functions for this and other reasons. In this particular case however the checksum is never exposed t

Re: [HACKERS] Snapshot synchronization, again...

2011-02-21 Thread Joachim Wieland
Hi, On Mon, Feb 21, 2011 at 4:56 PM, Alvaro Herrera wrote: > What's the reason for this restriction? >        if (databaseId != MyDatabaseId) >                ereport(ERROR, >                        (errcode(ERRCODE_INVALID_PARAMETER_VALUE), >                         errmsg("cannot import snapsho

Re: [HACKERS] Snapshot synchronization, again...

2011-02-22 Thread Joachim Wieland
On Tue, Feb 22, 2011 at 3:34 PM, Heikki Linnakangas wrote: > On 22.02.2011 16:29, Robert Haas wrote: >> On Tue, Feb 22, 2011 at 8:58 AM, Heikki Linnakangas >>  wrote: >>> No, the hash is stored in shared memory. The hash of the garbage has to >>> match. >> >> Oh.  Well that's really silly.  At th

Re: [HACKERS] Snapshot synchronization, again...

2011-02-27 Thread Joachim Wieland
On Sun, Feb 27, 2011 at 3:04 PM, Heikki Linnakangas wrote: >> Why exactly, Heikki do you think the hash is more troublesome? > It just feels wrong to rely on cryptography just to save some shared memory. Remember that it's not only about saving shared memory, it's also about making sure that the

Re: [HACKERS] Snapshot synchronization, again...

2011-02-28 Thread Joachim Wieland
On Mon, Feb 28, 2011 at 6:38 PM, Robert Haas wrote: >> Remember that it's not only about saving shared memory, it's also >> about making sure that the snapshot reflects a state of the database >> that has actually existed at some point in the past. Furthermore, we >> can easily invalidate a snapsh

Re: [HACKERS] synchronized snapshots

2011-08-15 Thread Joachim Wieland
On Mon, Aug 15, 2011 at 3:47 AM, Heikki Linnakangas wrote: > On 15.08.2011 04:31, Joachim Wieland wrote: >> >> The one thing that it does not implement is leaving the transaction in >> an aborted state if the BEGIN TRANSACTION command failed for an >> invalid snapshot

  1   2   >