Re: [HACKERS] RServ patch to support multiple slaves (sorta)
On Wednesday 25 June 2003 08:42 am, Alvaro Herrera wrote: On Wed, Jun 25, 2003 at 11:11:35AM -0400, Bruce Momjian wrote: Tom Lane wrote: Bruce Momjian [EMAIL PROTECTED] writes: Patch applied. Thanks. Michael A Nachbaur wrote: Attached is a patch that provides *VERY* limited support for multiple slave servers. I haven't tested it very well, so use at your own risk (and I recommend against using it in production). It sounded to me like that patch was intended for comment, not for application. Yes, this was my original intent. If anyone else thought it was worthy enough to go into CVS, then great, but mainly I wanted a few more pairs of eyes to look it over. He said it wasn't all he wanted to do with the code, but that it did work. With so few rserv patches, it seems like something we should get in, but maybe not? Other comments? I am not sure myself. I don't remember the patch right now, but it seemed to me the patch didn't have anything to do with multiple slaves anyway... When was it posted? I can't find it in the archives... (it'd be nice to have the date of the original message in the attribution when you quote other people, that way it's much easier to find it in the archives) All my patch does is allow you to limit what tables you replicate from a slave. In this way, SlaveA can replicate tables X, Y and Z, while SlaveB can replicate tables M, N and O. I have a single master database, and different authentication databases at key areas of my infrastructure (mail authentication, web server configuration, etc). I was getting errors when trying to replicate SlaveA just after adding SlaveB, because the necessary tables didn't exist on SlaveA. Some 2 years ago I wrote a patch for multiple slaves and it worked reasonably well... I wasn't too much in the Postgres world those days so I didn't submit it. If I can get to my CVS archive I'll extract it and post for review. That'd be great. My patch, like I said in my original post (06/19/2003 07:36pm PST), is just a beginning, and I'm not even 100% sure it'll work reliably. Although I'm an experienced Perl programmer, I'm not as familar with PostgreSQL's internals as I'd like to be (e.g. I tread lightly when it comes to the pg_* tables). If someone else has better support, I'd much rather a) take the code and run, and b) not have to do the same myself since I have too many items on my task list as it is. -- Michael A Nachbaur [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
Re: [HACKERS] RServ patch to support multiple slaves (sorta)
On Thursday 19 June 2003 07:36 pm, Michael A Nachbaur wrote: Attached is a patch that provides *VERY* limited support for multiple slave servers. I haven't tested it very well, so use at your own risk (and I recommend against using it in production). snip Okay, I just did some more testing, and found out that my code actually doesn't work. Consider the attached test case (if run with -delete, it will delete the master, slavea - slavec databases that it creates). Anyway, it looks like it replicates the A table just fine, and the slaveb and slavec databases replicate just fine, but the SyncID was incremented by the SlaveA replication, and therefore b and c never get updated. I don't know enough about how the RServ code works right now to fix this right away. Any ideas? Or should I just figure it out for myself? (I know everyone is busy getting ready for the feature freeze) -- Michael A Nachbaur [EMAIL PROTECTED] replicate-testcase.sh Description: application/shellscript ---(end of broadcast)--- TIP 7: don't forget to increase your free space map settings
[HACKERS] RServ patch to support multiple slaves (sorta)
Attached is a patch that provides *VERY* limited support for multiple slave
servers. I haven't tested it very well, so use at your own risk (and I
recommend against using it in production).
Basically, I have a central database server that has 4 summary tables inside
it replicated to a remote slave (these database tables are for my mail server
authentication, so these are replicated to another server tuned for many
connections, and so I don't have postgres connections opened straight to my
back-end database server).
Unfortunately, I also wanted to implement a replication database server for
hot-backups. I realized, too late, that the replication process is pretty
greedy and will try to replicate all tables marked as a MasterAddTable.
To make a long story, I made a patch to RServ.pm and Replicate that allows you
to specify, on the command line, a list of tables that you want to
replicate...it'll ignore all others.
I haven't finished, since this has to be integrated with CleanLog for
instance, but this should (and does) suffice for the moment.
I have yet to test it with two slaves, but at least my mail server replication
database now works (it was failing every time it tried to replicate, for a
variety of reasons).
Anyone have any suggestions on how to improve on this? (or, if someone more
familiar with this code wants to take the ball and run with it, you're
welcome to).
--
Michael A Nachbaur [EMAIL PROTECTED]
diff -pur rserv/Replicate rserv-new/Replicate
--- rserv/Replicate 2003-06-19 19:15:09.0 -0700
+++ rserv-new/Replicate 2003-06-19 19:14:23.0 -0700
@@ -32,6 +32,7 @@ if (defined($opt_help) || (scalar(@ARGV)
my $master = $ARGV[0] || master;
my $slave = $ARGV[1] || slave;
+my $tables = $#ARGV 2 ? undef : { map {($_, undef)} @ARGV[2..$#ARGV] };
my $server = 0;
my $minfo = dbname=$master;
@@ -56,7 +57,7 @@ SyncSync($mconn, $sconn);
my $outf = new IO::File;
open $outf, $snapshot;
print \n Prepare Snapshot\n\n if ($verbose);
-$res = PrepareSnapshot($mconn, $outf, $server);
+$res = PrepareSnapshot($mconn, $outf, $server, $tables);
close $outf;
die \n ERROR\n if $res 0;
if ($res == 0)
@@ -68,7 +69,7 @@ if ($res == 0)
my $inpf = new IO::File;
open $inpf, $snapshot;
print \n Apply Snapshot\n\n if ($verbose);
-$res = ApplySnapshot($sconn, $inpf);
+$res = ApplySnapshot($sconn, $inpf, $tables);
close $inpf;
die \n ERROR\n if $res 0;
diff -pur rserv/RServ.pm rserv-new/RServ.pm
--- rserv/RServ.pm 2003-06-19 19:14:59.0 -0700
+++ rserv-new/RServ.pm 2003-06-19 19:34:47.0 -0700
@@ -19,7 +19,7 @@ my %Stables = ();
sub PrepareSnapshot
{
- my ($conn, $outf, $server) = @_; # (@_[0], @_[1], @_[2]);
+ my ($conn, $outf, $server, $onlytables) = @_; # (@_[0], @_[1], @_[2]);
my $result = $conn-exec(BEGIN);
if ($result-resultStatus ne PGRES_COMMAND_OK)
@@ -52,6 +52,10 @@ sub PrepareSnapshot
while (@row = $result-fetchrow)
{
# printf $row[0], $row[1], $row[2]\n;
+ if (ref($onlytables) eq 'HASH') {
+ next unless (exists $onlytables-{$row[1]});
+ $onlytables-{$row[1]} = $row[0] unless ($onlytables-{$row[1]});
+ }
push @{$Mtables{$row[0]}}, $row[1], $row[2];
}
@@ -232,7 +236,7 @@ sub GetSYNCID
sub CleanLog
{
- my ($conn, $howold) = @_; # (@_[0], @_[1]);
+ my ($conn, $howold, $onlytables) = @_; # (@_[0], @_[1]);
my $result = $conn-exec(BEGIN);
if ($result-resultStatus ne PGRES_COMMAND_OK)
@@ -274,6 +278,11 @@ sub CleanLog
my $alist = join(',', keys %active);
my $sinfo = logid $maxid;
$sinfo .= and logid not in ($alist) if $alist ne '';
+ #if (ref($onlytables) eq 'HASH') {
+ # foreach my $onlytable (keys %{$onlytables}) {
+ # $sinfo
+ # }
+ #}
$sql = delete from _RSERV_LOG_ where .
logtime now() - '$howold second'::interval and $sinfo;
@@ -302,7 +311,7 @@ sub CleanLog
sub ApplySnapshot
{
- my ($conn, $inpf) = @_; # (@_[0], @_[1]);
+ my ($conn, $inpf, $onlytables) = @_; # (@_[0], @_[1]);
my $result = $conn-exec(BEGIN);
if ($result-resultStatus ne PGRES_COMMAND_OK)
@@ -336,6 +345,10 @@ sub ApplySnapshot
while (@row = $result-fetchrow)
{
# printf %s %s\n, $row[1], $row[0];
+ if (ref($onlytables) eq 'HASH') {
+ next unless (exists $onlytables-{$row[1]});
+ $onlytables-{$row[1]} = $row[0] unless ($onlytables-{$row[1]});
+ }
push @{$Stables{$row[1]}}, $row[0], $row[2], $row[3];
}
Only in rserv-new: RServ.pm~
Only in rserv-new: .RServ.pm.swp
---(end of broadcast)---
TIP 5: Have you checked our extensive FAQ?
http://www.postgresql.org/docs/faqs/FAQ.html
Re: [HACKERS] RBLs ... I'm tired of spam ...
Install SpamAssassin, and let it figure it out for you. It uses a whole list of RBLs and uses them to score a message as spam, instead of just blanket-denying messages from those SMTP servers. It works quite well. On Tuesday 27 May 2003 01:41 pm, Marc G. Fournier wrote: *Way* off topic ... but I'm tired of processing through 300 messages nightly of which 10 are stuff that need to be approved for the lists, and 290 are trash ... What are ppl using / trusting out there as far as Free RBLs are concerned? ---(end of broadcast)--- TIP 3: if posting/reading through Usenet, please send an appropriate subscribe-nomail command to [EMAIL PROTECTED] so that your message can get through to the mailing list cleanly -- Michael A Nachbaur [EMAIL PROTECTED] ---(end of broadcast)--- TIP 4: Don't 'kill -9' the postmaster
