Re: [PATCHES] PITR Archival
On Tue, 2004-06-15 at 16:34, Simon Riggs wrote: I enclose a working set of context diff patches and new files to make PITR archiving work, as of cvstip (NOW). As of now, this patch is invalidated by recent changes. Don't try to run it, it doesn't even start. There's good news coming later in this email update :) I'm now in the middle of reworking this, 1-3 days, I guess. There is one bug: shutdown doesn't work quite right. I haven't fixed this because I've spent too long trying to decipher how pgstat did a clean shutdown, discovering now that it didn't and that has now been patched...something similar is required for pgarch, but I'm out of time now...leaving time for discussion of this lot... The code that doesn't work appears to be the Archiver startup code, so that effectively requires me to rework the startup/shutdown code that was slightly flaky, as mentioned above. I originally cloned the pgstat code, which as I observed was possible to improve uponthat having been done (a good thing) should just leave the task as a re-clone (especially since that par of the code I can almost recite by now). I would welcome one of the Win32 crew having an eye through the code, to suggest any areas to avoid/pay close attention to. I don't want to invalidate any other work, seeing as I'm last to show now at this party. If we get this done smoothly, I reckon I can have some PITR recovery control done by beta freeze. The good news: ...before recent changes, I had a working version of the streaming recovery code and recovered 3 test databases. I would note that the recovery speed was quite impressive, even on my 700Mhz dev pc, so I have confidence that this will work well on the real deal. How this works: when startup enters recovery, if you have supplied a file: DataDir/recovery.conf, then it will read this to get a recovery command line. startup executes this command each time recovery requests a new xlog from archive, using only one file at a time. Result: you will be able to recover, no matter how many xlogs you have and how little disk space you have (in comparison). This will allow you to write very short scripts to perform - infinite recovery, just like Oracle 9i and DB2 8.1 - integration with tapes and backup software - automated standby databases, where an active database feeds logs to a passive database that is permanently InRecovery, yet can be brought on-line within a few seconds if the link drops. I haven't yet written the code to stop at-a-point-in-time, but I remain ever optimistic. I won't slow down the patch waiting for that... ...and I take it the doc freeze is not the same time as the code freeze.. Best regards, Simon Riggs ---(end of broadcast)--- TIP 2: you can get off all lists at once with the unregister command (send unregister YourEmailAddressHere to [EMAIL PROTECTED])
[PATCHES] PITR Archival
I enclose a working set of context diff patches and new files to make
PITR archiving work, as of cvstip (NOW).
You'll see the new options in the postgresql.conf...though you may wish
to use archive_debug = true as well, when testing.
There is one bug: shutdown doesn't work quite right. I haven't fixed
this because I've spent too long trying to decipher how pgstat did a
clean shutdown, discovering now that it didn't and that has now been
patched...something similar is required for pgarch, but I'm out of time
now...leaving time for discussion of this lot...
I'm looking to have this lot committed asap, cos my fingers are starting
to catch the bitrot now. :)
I have a considerable amount still to learn about CVS, diff and patch,
so anybody wanting to spend 10-15 mins on the phone with me would
greatly enhance my chances of helping patch my patch, when the bugs roll
in.
If we get this done smoothly, I reckon I can have some PITR recovery
control done by beta freeze.
Best regards, Simon Riggs
? Makefile.global
? pitr_arch_v3.01.patch
? pitr_v3.1.patch
? backend/postgres
? backend/catalog/postgres.bki
? backend/catalog/postgres.description
? backend/postmaster/pgarch.c
? backend/postmaster/postmaster.c.new
? backend/utils/mb/conversion_procs/conversion_create.sql
? backend/utils/mb/conversion_procs/ascii_and_mic/libascii_and_mic.so.0.0
? backend/utils/mb/conversion_procs/cyrillic_and_mic/libcyrillic_and_mic.so.0.0
? backend/utils/mb/conversion_procs/euc_cn_and_mic/libeuc_cn_and_mic.so.0.0
? backend/utils/mb/conversion_procs/euc_jp_and_sjis/libeuc_jp_and_sjis.so.0.0
? backend/utils/mb/conversion_procs/euc_kr_and_mic/libeuc_kr_and_mic.so.0.0
? backend/utils/mb/conversion_procs/euc_tw_and_big5/libeuc_tw_and_big5.so.0.0
? backend/utils/mb/conversion_procs/latin2_and_win1250/liblatin2_and_win1250.so.0.0
? backend/utils/mb/conversion_procs/latin_and_mic/liblatin_and_mic.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_ascii/libutf8_and_ascii.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_big5/libutf8_and_big5.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_cyrillic/libutf8_and_cyrillic.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_euc_cn/libutf8_and_euc_cn.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_euc_jp/libutf8_and_euc_jp.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_euc_kr/libutf8_and_euc_kr.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_euc_tw/libutf8_and_euc_tw.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_gb18030/libutf8_and_gb18030.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_gbk/libutf8_and_gbk.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_iso8859/libutf8_and_iso8859.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_iso8859_1/libutf8_and_iso8859_1.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_johab/libutf8_and_johab.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_sjis/libutf8_and_sjis.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_tcvn/libutf8_and_tcvn.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_uhc/libutf8_and_uhc.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_win1250/libutf8_and_win1250.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_win1256/libutf8_and_win1256.so.0.0
? backend/utils/mb/conversion_procs/utf8_and_win874/libutf8_and_win874.so.0.0
? bin/initdb/initdb
? bin/initlocation/initlocation
? bin/ipcclean/ipcclean
? bin/pg_config/pg_config
? bin/pg_controldata/pg_controldata
? bin/pg_ctl/pg_ctl
? bin/pg_dump/pg_dump
? bin/pg_dump/pg_dumpall
? bin/pg_dump/pg_restore
? bin/pg_resetxlog/pg_resetxlog
? bin/psql/psql
? bin/scripts/clusterdb
? bin/scripts/createdb
? bin/scripts/createlang
? bin/scripts/createuser
? bin/scripts/dropdb
? bin/scripts/droplang
? bin/scripts/dropuser
? bin/scripts/vacuumdb
? include/pg_config.h
? include/pgarch.h
? include/stamp-h
? interfaces/ecpg/compatlib/libecpg_compat.so.1.1
? interfaces/ecpg/ecpglib/libecpg.so.4.2
? interfaces/ecpg/pgtypeslib/libpgtypes.so.1.2
? interfaces/ecpg/preproc/ecpg
? interfaces/libpq/libpq.so.3.2
? pl/plpgsql/src/libplpgsql.so.1.0
? port/pg_config_paths.h
? timezone/zic
Index: backend/access/nbtree/nbtsort.c
===
RCS file: /projects/cvsroot/pgsql-server/src/backend/access/nbtree/nbtsort.c,v
retrieving revision 1.82
diff -c -r1.82 nbtsort.c
*** backend/access/nbtree/nbtsort.c 2 Jun 2004 17:28:17 - 1.82
--- backend/access/nbtree/nbtsort.c 15 Jun 2004 15:19:48 -
***
*** 67,72
--- 67,73
#include miscadmin.h
#include storage/smgr.h
#include utils/tuplesort.h
+ #include access/xlog.h
/*
***
*** 220,235
wstate.index = btspool-index;
/*
! * We need to log index creation in WAL iff WAL archiving is enabled
* AND it's not a temp index.
- *
- * XXX when WAL archiving is actually supported, this test will likely
- * need to change; and the hardwired extern is cruddy anyway ...
*/
{
! extern char XLOG_archive_dir[];
Re: [PATCHES] PITR Archival
On Tue, 2004-06-15 at 16:34, Simon Riggs wrote: I enclose a working set of context diff patches and new files to make PITR archiving work, as of cvstip (NOW). You'll see the new options in the postgresql.conf...though you may wish to use archive_debug = true as well, when testing. There is one bug: shutdown doesn't work quite right. I haven't fixed this because I've spent too long trying to decipher how pgstat did a clean shutdown, discovering now that it didn't and that has now been patched...something similar is required for pgarch, but I'm out of time now...leaving time for discussion of this lot... The patch creates a further child process of postmaster, the archiver. Archiver code is similar, but not the same, as pgstat.c, and by analogy lives in src/backend/postmaster/pgarch.c with a matching pgarch.h in src/include (just as with pgstat.h) At various points, 5 processes are involved, all of which are identified clearly in the debugging messages... - backend: sends signal to postmaster to say xlog is ready to archive - postmaster: catches and resends to archiver - archiver: then calls system(3) to invoke a user-defined archival task - bgwriter: later cleans up archive_status at checkpoint time Please note: you will need to initdb to make this work Also note: archive_mode is not designed to be turned on/off frequently. It is possible to confuse it if you turn it on, then restart with it off, then turn it on again. That is likely to create a hole in the archive history of xlogs and you will not be able to recover correctly. You may also note that the design has changed substantially from many earlier design postings...all of this is based on community input to rationalise behaviour and to streamline code. If you wish to test recovery, you should: - do a full physical backup of DataDir, while postmaster is UP - archive all xlogs when disaster strikes - restore backup of DataDir - restore all archived xlogs to pg_xlog - startup postmaster (and watch...) Further work is still required to make it STOP recovering at a predetermined point in time Best regards, Simon Riggs ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [PATCHES] PITR Archival
On Tue, Jun 15, 2004 at 04:34:30PM +0100, Simon Riggs wrote: I have a considerable amount still to learn about CVS, diff and patch, so anybody wanting to spend 10-15 mins on the phone with me would greatly enhance my chances of helping patch my patch, when the bugs roll in. Can't help you with the phone thingie, but if you want to see what's in a patch and be able to edit it nicely, I suggest you use meld. It's a python dual-pane GTK display, really nice. Too slow for real coding (on my machine that is) but real good for seeing what you changed. Watch out for use of Ctrl, Shift and Alt -- very handy. Can use CVS as well, or you can use two source trees. -- Alvaro Herrera (alvherre[a]dcc.uchile.cl) There is evil in the world. There are dark, awful things. Occasionally, we get a glimpse of them. But there are dark corners; horrors almost impossible to imagine... even in our worst nightmares. (Van Helsing, Dracula A.D. 1972) ---(end of broadcast)--- TIP 8: explain analyze is your friend
Re: [PATCHES] PITR Archival
Your patch has been added to the PostgreSQL unapplied patches list at: http://momjian.postgresql.org/cgi-bin/pgpatches I will try to apply it once it is reviewed. --- Simon Riggs wrote: I enclose a working set of context diff patches and new files to make PITR archiving work, as of cvstip (NOW). You'll see the new options in the postgresql.conf...though you may wish to use archive_debug = true as well, when testing. There is one bug: shutdown doesn't work quite right. I haven't fixed this because I've spent too long trying to decipher how pgstat did a clean shutdown, discovering now that it didn't and that has now been patched...something similar is required for pgarch, but I'm out of time now...leaving time for discussion of this lot... I'm looking to have this lot committed asap, cos my fingers are starting to catch the bitrot now. :) I have a considerable amount still to learn about CVS, diff and patch, so anybody wanting to spend 10-15 mins on the phone with me would greatly enhance my chances of helping patch my patch, when the bugs roll in. If we get this done smoothly, I reckon I can have some PITR recovery control done by beta freeze. Best regards, Simon Riggs [ Attachment, skipping... ] [ Attachment, skipping... ] [ Attachment, skipping... ] ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly -- Bruce Momjian| http://candle.pha.pa.us [EMAIL PROTECTED] | (610) 359-1001 + If your life is a hard drive, | 13 Roberts Road + Christ can be your backup.| Newtown Square, Pennsylvania 19073 ---(end of broadcast)--- TIP 8: explain analyze is your friend
