Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Fujii Masao
On Thu, Oct 27, 2011 at 11:57 PM, Magnus Hagander wrote: > On Thu, Oct 27, 2011 at 16:54, Tom Lane wrote: >> Magnus Hagander writes: >>> On Thu, Oct 27, 2011 at 13:19, Heikki Linnakangas >>> wrote: On 27.10.2011 14:09, Fujii Masao wrote: > Yes. But that sounds unuserfriendly. Padding t

Re: [HACKERS] Your review of pg_receivexlog/pg_basebackup

2011-10-27 Thread Fujii Masao
On Thu, Oct 27, 2011 at 11:14 PM, Magnus Hagander wrote: > Here's a version that does this. Turns out this requires a lot less > code than what was previously in there, which is always nice. > > We still need to solve the other part which is how to deal with the > partial files on restore. But thi

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Magnus Hagander
On Oct 28, 2011 5:19 AM, "Bruce Momjian" wrote: > > Stephen Frost wrote: > > > > > Regarding pg_dumpall and pg_restore, I'm pretty sure both of those can > > > > be configured to connect to other databases instead, including for > > > > globals. > > > > > > Well, please show me the code, because t

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Magnus Hagander
On Oct 28, 2011 5:22 AM, "Tom Lane" wrote: > > Bruce Momjian writes: > > Stephen Frost wrote: > >> Yes, they would have removed it because they didn't want it. As I > >> recall, part of the agreement to create an extra database by default was > >> that it could be removed if users didn't want it

Re: [HACKERS] Review: [PL/pgSQL] %TYPE and array declaration - second patch

2011-10-27 Thread Pavel Stehule
Hello 2011/10/28 Tom Lane : > Pavel Stehule writes: >> there is final Wojciech's patch > this is just small note about length of this patch. This patch was significantly smaller then he solved problem with derivate types for compound types - it should to solve problem described in this thread h

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Tom Lane
Bruce Momjian writes: > Robert Haas wrote: >> that if you're doing something to the database that someone might >> object to, you oughtn't be doing it to the postgres database either. >> You should create a database just for pg_upgrade's use and install its >> crap in there. > It installs crap in

Re: [HACKERS] Review: [PL/pgSQL] %TYPE and array declaration - second patch

2011-10-27 Thread Tom Lane
Pavel Stehule writes: > there is final Wojciech's patch I looked this over a little bit, but I don't see an answer to the question I put on the commitfest entry: why is this being done in plpgsql, and not somewhere in the core code? The core has also got the concept of %TYPE, and could stand to

[HACKERS] Include commit identifier in version() function

2011-10-27 Thread pasman pasmański
Hi. I think, it be useful to include in version() function a hexadecimal identifier of commit, for fast checkout to it in git. -- pasman -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/p

Re: [HACKERS] Hot Standby startup with overflowed snapshots

2011-10-27 Thread Robert Haas
On Thu, Oct 27, 2011 at 6:55 PM, Simon Riggs wrote: > It seems cheap to add in a call to LogStandbySnapshot() after each > call to pg_stop_backup(). > > Does anyone think this case is worth adding code for? Seems like one > more thing to break. Why at that particular time? It would maybe nice if

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Bruce Momjian
Robert Haas wrote: > On Thu, Oct 27, 2011 at 11:35 PM, Bruce Momjian wrote: > >> What about creating a new, single-purpose database in the source > >> cluster and then removing it again after we're done? > > > > That is not a problem --- I can easily use template1. > > Huh? > > You just said upt

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Robert Haas
On Thu, Oct 27, 2011 at 11:35 PM, Bruce Momjian wrote: >> What about creating a new, single-purpose database in the source >> cluster and then removing it again after we're done? > > That is not a problem --- I can easily use template1. Huh? You just said upthread that you didn't want to use tem

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > > Stephen Frost wrote: > >> Yes, they would have removed it because they didn't want it. As I > >> recall, part of the agreement to create an extra database by default was > >> that it could be removed if users didn't want it. Turning around and > >> the

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Tom Lane
Bruce Momjian writes: > Stephen Frost wrote: >> Yes, they would have removed it because they didn't want it. As I >> recall, part of the agreement to create an extra database by default was >> that it could be removed if users didn't want it. Turning around and >> then saying "but things won't w

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Bruce Momjian
Stephen Frost wrote: -- Start of PGP signed section. > * Bruce Momjian (br...@momjian.us) wrote: > > Well, you would have to remove it _after_ you did the pg_upgrade. Right > > now if you do a normal dump/restore upgrade, you also have to re-remove > > the postgres database. We don't have any mec

Re: [HACKERS] Add socket dir to pg_config..?

2011-10-27 Thread Tom Lane
Stephen Frost writes: > Was just wondering if we might want to include the default socket > directory that was compiled in as part of the pg_config output..? [ shrug... ] We don't report the compiled-in port number, which is considerably more critical. And we don't report changes in any of

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Stephen Frost
* Bruce Momjian (br...@momjian.us) wrote: > Well, you would have to remove it _after_ you did the pg_upgrade. Right > now if you do a normal dump/restore upgrade, you also have to re-remove > the postgres database. We don't have any mechanism to drop a database > as part of pg_dumpall's restore i

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Bruce Momjian
Stephen Frost wrote: -- Start of PGP signed section. > * Bruce Momjian (br...@momjian.us) wrote: > > I have not seen enough demand to make this a user-visible configuration. > > We can just tell them to create a postgres database. Frankly, they > > would have had to _remove_ the postgres database

Re: [HACKERS] Hot Standby startup with overflowed snapshots

2011-10-27 Thread Chris Redekop
Sorry..."designed" was poor choice of words, I meant "not unexpected". Doing the checkpoint right after pg_stop_backup() looks like it will work perfectly for me, so thanks for all your help! On a side note I am sporadically seeing another error on hotstandby startup. I'm not terribly concerned

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Stephen Frost
* Bruce Momjian (br...@momjian.us) wrote: > I have not seen enough demand to make this a user-visible configuration. > We can just tell them to create a postgres database. Frankly, they > would have had to _remove_ the postgres database after initdb for it not > to be there, and they are instruct

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Bruce Momjian
Stephen Frost wrote: -- Start of PGP signed section. > * Bruce Momjian (br...@momjian.us) wrote: > > So, it is going to be confusing to support both databases because there > > is the cleanup details I have to document if I use template1. > > Presumably there's some other database in the system be

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Stephen Frost
* Bruce Momjian (br...@momjian.us) wrote: > So, it is going to be confusing to support both databases because there > is the cleanup details I have to document if I use template1. Presumably there's some other database in the system besides template1 if postgres doesn't exist.. Would it be possib

Re: [HACKERS] pg_upgrade if 'postgres' database is dropped

2011-10-27 Thread Bruce Momjian
Robert Haas wrote: > On Tue, Oct 4, 2011 at 12:11 PM, Heikki Linnakangas > wrote: > > pg_upgrade doesn't work if the 'postgres' database has been dropped in the > > old cluster: > > > > ~/pgsql.master$ bin/pg_upgrade -b ~/pgsql.91stable/bin -B bin/ -d > > ~/pgsql.91stable/data -D data-upgraded/ >

[HACKERS] Add socket dir to pg_config..?

2011-10-27 Thread Stephen Frost
All, Was just wondering if we might want to include the default socket directory that was compiled in as part of the pg_config output..? Thanks, Stephen signature.asc Description: Digital signature

Re: [HACKERS] pg_dumpall Sets Roll default_tablespace Before Creating Tablespaces

2011-10-27 Thread Florian Pflug
On Oct27, 2011, at 23:02 , Bruce Momjian wrote: > Florian Pflug wrote: >> On Oct21, 2011, at 16:42 , Phil Sorber wrote: >>> If you did want to make them immutable, I also like Florian's idea of >>> a dependency graph. This would make the dumps less readable though. >> >> Hm, I kinda reversed my op

Re: [HACKERS] Hot Standby startup with overflowed snapshots

2011-10-27 Thread Simon Riggs
On Thu, Oct 27, 2011 at 10:09 PM, Chris Redekop wrote: > hrmz, still basically the same behaviour.  I think it might be a *little* > better with this patch.  Before when under load it would start up quickly > maybe 2 or 3 times out of 10 attemptswith this patch it might be up to 4 > or 5 time

Re: [HACKERS] (PATCH) Adding CORRESPONDING (NULL error)

2011-10-27 Thread Kerem Kat
On Thu, Oct 27, 2011 at 23:20, Tom Lane wrote: > I wrote: >> Kerem Kat writes: >>> Union with NULL error persists without the corresponding patch. Here >>> is the output from postgres without the patch: > >>> SELECT a FROM (SELECT 1 a) foo >>> UNION >>> SELECT a FROM (SELECT NULL a) foo2; > >>> E

Re: [HACKERS] Hot Standby startup with overflowed snapshots

2011-10-27 Thread Chris Redekop
hrmz, still basically the same behaviour. I think it might be a *little* better with this patch. Before when under load it would start up quickly maybe 2 or 3 times out of 10 attemptswith this patch it might be up to 4 or 5 times out of 10...ish...or maybe it was just fluke *shrug*. I'm stil

Re: [HACKERS] (PATCH) Adding CORRESPONDING (NULL error)

2011-10-27 Thread Tom Lane
Kerem Kat writes: > On Thu, Oct 27, 2011 at 23:20, Tom Lane wrote: >> BTW, just to clarify: although that case fails, the case Erik was >> complaining of does work in unmodified Postgres: >> ... >> and I agree with him that it should still work with CORRESPONDING. > That is by design, because CO

Re: [HACKERS] (PATCH) Adding CORRESPONDING (NULL error)

2011-10-27 Thread Tom Lane
I wrote: > Kerem Kat writes: >> Union with NULL error persists without the corresponding patch. Here >> is the output from postgres without the patch: >> SELECT a FROM (SELECT 1 a) foo >> UNION >> SELECT a FROM (SELECT NULL a) foo2; >> ERROR: failed to find conversion function from unknown to i

Re: [HACKERS] pg_dumpall Sets Roll default_tablespace Before Creating Tablespaces

2011-10-27 Thread Bruce Momjian
Florian Pflug wrote: > On Oct21, 2011, at 16:42 , Phil Sorber wrote: > > If you did want to make them immutable, I also like Florian's idea of > > a dependency graph. This would make the dumps less readable though. > > Hm, I kinda reversed my opinion on that, though - i.e., I no longer think > tha

Re: [HACKERS] (PATCH) Adding CORRESPONDING (NULL error)

2011-10-27 Thread Tom Lane
Kerem Kat writes: > Union with NULL error persists without the corresponding patch. Here > is the output from postgres without the patch: > SELECT a FROM (SELECT 1 a) foo > UNION > SELECT a FROM (SELECT NULL a) foo2; > ERROR: failed to find conversion function from unknown to integer Yeah, thi

[HACKERS] fun with unlogged tables

2011-10-27 Thread Robert Haas
One of the optimizations that I did for 9.1 was to make transactions that touch only temporary and/or unlogged tables always commit asynchronously, because if the database crashes the table contents will be blown away in their entirety, and whether or not the commit made it down to disk won't matte

[HACKERS] portal with hold

2011-10-27 Thread fschmidt
I am posting to beg for the implementation of a "with hold" feature for portals, similar to what available for cursors. This is needed by the JDBC driver to implement Java's Result.HOLD_CURSORS_OVER_COMMIT which is needed to make Java's setFetchSize() work which is needed to read large result sets

Re: [HACKERS] out-of-order caution

2011-10-27 Thread Tom Lane
"Kevin Grittner" writes: > (2) They *can* get a serialization failure involving just two > transactions: a read and a write. Only if you ignore the difference between SELECT FOR UPDATE/SHARE and plain SELECT. I think calling the former a "read" is a conceptual error to start with. It has the s

Re: [HACKERS] out-of-order caution

2011-10-27 Thread Tom Lane
"Kevin Grittner" writes: > Simon Riggs wrote: >> On Thu, Oct 27, 2011 at 4:41 PM, Kevin Grittner >> wrote: >>> | It is possible for a SELECT command using ORDER BY and FOR >>> | UPDATE/SHARE to return rows out of order. This is because ORDER >>> | BY is applied first. >> I think it should say t

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Florian Pflug
On Oct27, 2011, at 16:30 , Simon Riggs wrote: > On Thu, Oct 27, 2011 at 3:03 PM, Florian Pflug wrote: > >>> I think you make a good case for doing this. >>> >>> However, I'm concerned that moving LogStandbySnapshot() in a backpatch >>> seems more risky than it's worth. We could easily introduce

Re: [HACKERS] (PATCH) Adding CORRESPONDING (NULL error)

2011-10-27 Thread Kerem Kat
Hi, Union with NULL error persists without the corresponding patch. Here is the output from postgres without the patch: SELECT a FROM (SELECT 1 a) foo UNION SELECT a FROM (SELECT NULL a) foo2; ERROR: failed to find conversion function from unknown to integer It is thrown from parse_coerce.c:c

Re: [HACKERS] out-of-order caution

2011-10-27 Thread Kevin Grittner
Robert Haas wrote: > Simple test case: > > rhaas=# create table oops (a int); > CREATE TABLE > rhaas=# insert into oops values (1), (2), (3), (4); > INSERT 0 4 > rhaas=# begin; > BEGIN > rhaas=# update oops set a = 5 where a = 2; > UPDATE 1 > > In another session: > > rhaas=# select * from oops

Re: [HACKERS] out-of-order caution

2011-10-27 Thread Robert Haas
On Thu, Oct 27, 2011 at 1:51 PM, Kevin Grittner wrote: > Simon Riggs wrote: >> On Thu, Oct 27, 2011 at 4:41 PM, Kevin Grittner >> wrote: >>> On the docs page for the SELECT statement, there is a caution >>> which starts with: >>> >>> | It is possible for a SELECT command using ORDER BY and FOR >

Re: [HACKERS] out-of-order caution

2011-10-27 Thread Kevin Grittner
Simon Riggs wrote: > On Thu, Oct 27, 2011 at 4:41 PM, Kevin Grittner > wrote: >> On the docs page for the SELECT statement, there is a caution >> which starts with: >> >> | It is possible for a SELECT command using ORDER BY and FOR >> | UPDATE/SHARE to return rows out of order. This is because OR

Re: [HACKERS] Bug in walsender when calling out to do_pg_stop_backup (and others?)

2011-10-27 Thread Bruce Momjian
Greg Jaskiewicz wrote: > > On 19 Oct 2011, at 18:28, Florian Pflug wrote: > > All the other flags which indicate cancellation reasons are set from signal > > handers, I believe. We could of course mark as ClientConnectionLostPending > > as volatile just to be consistent. Not sure whether that's

Re: [HACKERS] Hot Standby startup with overflowed snapshots

2011-10-27 Thread Simon Riggs
On Thu, Oct 27, 2011 at 5:26 PM, Chris Redekop wrote: > Thanks for the patch Simon, but unfortunately it does not resolve the issue > I am seeing.  The standby still refuses to finish starting up until long > after all clients have disconnected from the primary (>10 minutes).  I do > see your new

Re: [HACKERS] plpgsql versus long ELSIF chains

2011-10-27 Thread Heikki Linnakangas
On 27.10.2011 19:18, Tom Lane wrote: So really what needs to be done here is to make ELSIF chains explicit in the parsetree representation, and handle them via looping not recursion at runtime. This will also make it a lot easier to do the grammar via left-recursion. So I'm going to go off and

Re: [HACKERS] plpgsql versus long ELSIF chains

2011-10-27 Thread David E. Wheeler
On Oct 27, 2011, at 9:18 AM, Tom Lane wrote: > So I'm going to go off and do that, but I wonder whether anyone thinks > this is sufficiently important to back-patch. I'm inclined to think > that back-patching isn't a good idea, because changing the > representation of PLpgSQL_stmt_if will break (

Re: [HACKERS] Hot Standby startup with overflowed snapshots

2011-10-27 Thread Chris Redekop
Thanks for the patch Simon, but unfortunately it does not resolve the issue I am seeing. The standby still refuses to finish starting up until long after all clients have disconnected from the primary (>10 minutes). I do see your new log statement on startup, but only once - it does not repeat.

[HACKERS] plpgsql versus long ELSIF chains

2011-10-27 Thread Tom Lane
Some off-list discussion found that the cause of this problem: http://archives.postgresql.org/pgsql-general/2011-10/msg00879.php was an attempt to write a plpgsql IF-ELSIF-ELSIF-ELSIF-ELSIF-...-ELSE statement with five thousand branches. Putting aside the wisdom of doing that, it seems like the pa

Re: [HACKERS] out-of-order caution

2011-10-27 Thread Simon Riggs
On Thu, Oct 27, 2011 at 4:41 PM, Kevin Grittner wrote: > On the docs page for the SELECT statement, there is a caution which > starts with: > > | It is possible for a SELECT command using ORDER BY and FOR > | UPDATE/SHARE to return rows out of order. This is because ORDER BY > | is applied first.

[HACKERS] out-of-order caution

2011-10-27 Thread Kevin Grittner
On the docs page for the SELECT statement, there is a caution which starts with: | It is possible for a SELECT command using ORDER BY and FOR | UPDATE/SHARE to return rows out of order. This is because ORDER BY | is applied first. Is this risk limited to queries running in READ COMMITTED transa

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Simon Riggs
On Thu, Oct 27, 2011 at 3:13 PM, Tom Lane wrote: > However, the > obvious next question is whether those other modules don't need to be > changed also, and if not why not. Good point. StartupSubtrans() is also changed by this patch, since it will be supplied with an earlier initialisation value

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Dimitri Fontaine
Magnus Hagander writes: > On Thu, Oct 27, 2011 at 13:19, Heikki Linnakangas > wrote: >> Perhaps we should add automatic padding in the server, though. It wouldn't >> take much code in the server, and would make life easier for people writing >> their scripts. Maybe even have the backend check for

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Magnus Hagander
On Thu, Oct 27, 2011 at 16:54, Tom Lane wrote: > Magnus Hagander writes: >> On Thu, Oct 27, 2011 at 13:19, Heikki Linnakangas >> wrote: >>> On 27.10.2011 14:09, Fujii Masao wrote: Yes. But that sounds unuserfriendly. Padding the WAL file manually is easy-to-do for a user? > >> I'd defi

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Tom Lane
Magnus Hagander writes: > On Thu, Oct 27, 2011 at 13:19, Heikki Linnakangas > wrote: >> On 27.10.2011 14:09, Fujii Masao wrote: >>> Yes. But that sounds unuserfriendly. Padding the WAL file manually >>> is easy-to-do for a user? > I'd definitely want to avoid anything that requires pg_receivexlo

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Simon Riggs
On Thu, Oct 27, 2011 at 3:03 PM, Florian Pflug wrote: >> I think you make a good case for doing this. >> >> However, I'm concerned that moving LogStandbySnapshot() in a backpatch >> seems more risky than it's worth. We could easily introduce a new bug >> into what we would all agree is a complex

Re: [HACKERS] Your review of pg_receivexlog/pg_basebackup

2011-10-27 Thread Magnus Hagander
On Wed, Oct 26, 2011 at 09:52, Heikki Linnakangas wrote: > (CC'ing pgsql-hackers, this started as an IM discussion yesterday but really > belongs in the archives) > > On 25.10.2011 23:52, Magnus Hagander wrote: >>> >>> There's a tiny chance to get incomplete xlog files with pg_receivexlog if >>> y

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Tom Lane
Simon Riggs writes: > On Thu, Oct 27, 2011 at 12:36 PM, Robert Haas wrote: >> On Thu, Oct 27, 2011 at 5:37 AM, Simon Riggs wrote: >>> It's much easier to understand that StartupCLOG() is actually a no-op >>> and that we need to trim the clog at the end of recovery in all cases. >> If it's a no-

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Florian Pflug
On Oct27, 2011, at 15:51 , Simon Riggs wrote: > On Thu, Oct 27, 2011 at 12:29 AM, Florian Pflug wrote: >> Here's what I image CreateCheckPoint() should look like: >> >> 1) LogStandbySnapshot() and fill out oldestActiveXid >> 2) Fill out REDO >> 3) Wait for concurrent commits >> 4) Fill out nextXi

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Simon Riggs
On Thu, Oct 27, 2011 at 12:29 AM, Florian Pflug wrote: > Per my theory about the cause of the problem in my other mail, I think you > might see StartupCLOG failures even during crash recovery, provided that > wal_level was set to hot_standby when the primary crashed. Here's how > > 1) We start a

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Simon Riggs
On Thu, Oct 27, 2011 at 12:36 PM, Robert Haas wrote: > On Thu, Oct 27, 2011 at 5:37 AM, Simon Riggs wrote: >> On Thu, Oct 27, 2011 at 4:36 AM, Tom Lane wrote: >>> Robert Haas writes: On Wed, Oct 26, 2011 at 12:16 PM, Simon Riggs wrote: > This fixes both the subtrans and clog bug

[HACKERS] Hot Standby startup with overflowed snapshots

2011-10-27 Thread Simon Riggs
Chris Redekop's recent report of slow startup for Hot Standby has made me revisit the code there. Although there isn't a bug, there is a missed opportunity for starting up faster which could be the source of Chris' annoyance. The following patch allows a faster startup in some circumstances. The

Re: [HACKERS] (PATCH) Adding CORRESPONDING (NULL error)

2011-10-27 Thread Erik Rijkers
(pgsql 9.2devel (25 oct) with your latest CORRESPONDING patch; linux x86_64 GNU/Linux 2.6.18-274.3.1.el5) Hi, here is another peculiarity, which I think is a bug: -- first without CORRESPONDING: $ psql -Xaf null.sql select 1 a , 2 b union all select null a, 4 b ; a |

Re: [HACKERS] isolationtester and invalid permutations

2011-10-27 Thread Robert Haas
On Tue, Oct 25, 2011 at 9:13 PM, Alvaro Herrera wrote: > Instead of simply aborting a spec that specifies running commands on > blocked sessions (what we call an invalid permutation), it seems more > useful to report the problem, cleanup the sessions, and continue with > the next permutation. > >

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Florian Pflug
On Oct27, 2011, at 08:57 , Heikki Linnakangas wrote: > On 27.10.2011 02:29, Florian Pflug wrote: >> Per my theory about the cause of the problem in my other mail, I think you >> might see StartupCLOG failures even during crash recovery, provided that >> wal_level was set to hot_standby when the pri

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Robert Haas
On Thu, Oct 27, 2011 at 7:19 AM, Heikki Linnakangas wrote: > On 27.10.2011 14:09, Fujii Masao wrote: >> >> On Thu, Oct 27, 2011 at 7:48 PM, Magnus Hagander >>  wrote: >>> >>> I'm rewriting the handling of partial files per the other thread >>> started by Heikki. The idea is that there will be an a

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Robert Haas
On Thu, Oct 27, 2011 at 5:37 AM, Simon Riggs wrote: > On Thu, Oct 27, 2011 at 4:36 AM, Tom Lane wrote: >> Robert Haas writes: >>> On Wed, Oct 26, 2011 at 12:16 PM, Simon Riggs wrote: This fixes both the subtrans and clog bugs in one patch. >> >>> I don't see the point of changing StartupCL

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Magnus Hagander
On Thu, Oct 27, 2011 at 13:19, Heikki Linnakangas wrote: > On 27.10.2011 14:09, Fujii Masao wrote: >> >> On Thu, Oct 27, 2011 at 7:48 PM, Magnus Hagander >>  wrote: >>> >>> I'm rewriting the handling of partial files per the other thread >>> started by Heikki. The idea is that there will be an act

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Heikki Linnakangas
On 27.10.2011 14:09, Fujii Masao wrote: On Thu, Oct 27, 2011 at 7:48 PM, Magnus Hagander wrote: I'm rewriting the handling of partial files per the other thread started by Heikki. The idea is that there will be an actual .partial file in there when pg_receivexlog has ended, and you have to deal

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Fujii Masao
On Thu, Oct 27, 2011 at 7:48 PM, Magnus Hagander wrote: > On Thu, Oct 27, 2011 at 12:29, Fujii Masao wrote: >> On Thu, Oct 27, 2011 at 6:25 PM, Fujii Masao wrote: >>> On Thu, Oct 27, 2011 at 5:18 PM, Magnus Hagander >>> wrote: Not sure I follow. When we arrive at PQgetCopyData() there sho

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Magnus Hagander
On Thu, Oct 27, 2011 at 12:29, Fujii Masao wrote: > On Thu, Oct 27, 2011 at 6:25 PM, Fujii Masao wrote: >> On Thu, Oct 27, 2011 at 5:18 PM, Magnus Hagander wrote: >>> Not sure I follow. When we arrive at PQgetCopyData() there should be >>> nothing buffered, and if the end of stream happens there

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Fujii Masao
On Thu, Oct 27, 2011 at 6:25 PM, Fujii Masao wrote: > On Thu, Oct 27, 2011 at 5:18 PM, Magnus Hagander wrote: >> Not sure I follow. When we arrive at PQgetCopyData() there should be >> nothing buffered, and if the end of stream happens there it returns >> -1, and we exit, no? So where is the data

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Heikki Linnakangas
On 27.10.2011 02:29, Florian Pflug wrote: Per my theory about the cause of the problem in my other mail, I think you might see StartupCLOG failures even during crash recovery, provided that wal_level was set to hot_standby when the primary crashed. Here's how 1) We start a checkpoint, and get as

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Simon Riggs
On Thu, Oct 27, 2011 at 4:36 AM, Tom Lane wrote: > Robert Haas writes: >> On Wed, Oct 26, 2011 at 12:16 PM, Simon Riggs wrote: >>> This fixes both the subtrans and clog bugs in one patch. > >> I don't see the point of changing StartupCLOG() to be an empty >> function and adding a new function Tr

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Fujii Masao
On Thu, Oct 27, 2011 at 5:18 PM, Magnus Hagander wrote: > Not sure I follow. When we arrive at PQgetCopyData() there should be > nothing buffered, and if the end of stream happens there it returns > -1, and we exit, no? So where is the data that's lost? > > I do realize we don't actually fsync() a

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Magnus Hagander
On Thu, Oct 27, 2011 at 10:12, Fujii Masao wrote: > On Thu, Oct 27, 2011 at 4:49 PM, Magnus Hagander wrote: >> On Thu, Oct 27, 2011 at 09:46, Fujii Masao wrote: >>> On Thu, Oct 27, 2011 at 4:40 PM, Magnus Hagander >>> wrote: On Thu, Oct 27, 2011 at 09:29, Fujii Masao wrote: > On Thu,

Re: [HACKERS] Hot Backup with rsync fails at pg_clog if under load

2011-10-27 Thread Heikki Linnakangas
On 27.10.2011 09:57, Heikki Linnakangas wrote: My suggestion is to fix the CLOG problem in that same way that you fixed the SUBTRANS problem, i.e. by moving LogStandbySnapshot() to before CheckPointGuts(). Here's what I image CreateCheckPoint() should look like: 1) LogStandbySnapshot() and fill

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Fujii Masao
On Thu, Oct 27, 2011 at 4:49 PM, Magnus Hagander wrote: > On Thu, Oct 27, 2011 at 09:46, Fujii Masao wrote: >> On Thu, Oct 27, 2011 at 4:40 PM, Magnus Hagander wrote: >>> On Thu, Oct 27, 2011 at 09:29, Fujii Masao wrote: On Thu, Oct 27, 2011 at 3:29 AM, Magnus Hagander wrote: >

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Magnus Hagander
On Thu, Oct 27, 2011 at 09:46, Fujii Masao wrote: > On Thu, Oct 27, 2011 at 4:40 PM, Magnus Hagander wrote: >> On Thu, Oct 27, 2011 at 09:29, Fujii Masao wrote: >>> On Thu, Oct 27, 2011 at 3:29 AM, Magnus Hagander >>> wrote: I've applied this version with a few more minor changes that Hei

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Fujii Masao
On Thu, Oct 27, 2011 at 4:40 PM, Magnus Hagander wrote: > On Thu, Oct 27, 2011 at 09:29, Fujii Masao wrote: >> On Thu, Oct 27, 2011 at 3:29 AM, Magnus Hagander wrote: >>> I've applied this version with a few more minor changes that Heikki found. >> >> Cool! >> >> When I tried pg_receivexlog and

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Magnus Hagander
On Thu, Oct 27, 2011 at 09:29, Fujii Masao wrote: > On Thu, Oct 27, 2011 at 3:29 AM, Magnus Hagander wrote: >> I've applied this version with a few more minor changes that Heikki found. > > Cool! > > When I tried pg_receivexlog and checked the contents of streamed WAL file by > xlogdump, I found

Re: [HACKERS] Updated version of pg_receivexlog

2011-10-27 Thread Fujii Masao
On Thu, Oct 27, 2011 at 3:29 AM, Magnus Hagander wrote: > I've applied this version with a few more minor changes that Heikki found. Cool! When I tried pg_receivexlog and checked the contents of streamed WAL file by xlogdump, I found that recent WAL records that walsender has already sent don't