-- [EMAIL PROTECTED]
> Nate,
>
> We are using EMC and BCV's each night to create both backups and a
> reporting database. Your SA has it WRONG.
>
> 1) Put all tablespaces in hot backup mode
> 2) sleep 5
> 3) split the bcv
> 4) return tablespaces to normal
>
> The reason that this is important has nothing in the world to do with EMC
> and everything to do with Oracle. Since switching the database files
> into and out of hot backup causes an immediate write to the file headers
> locking or unlocking the SCN. Consequently your backups are inconsistent
> and consequently worthless. The reason is that when you enter hotbackup
> mode all of the file SCN's are frozen giving Oracle a known starting
> point for recovery's sake. By switching back to normal mode the RDBMS
> has no idea of which transactions are in the files and which are not. It
> assumes that it is seeing an instance crash and will only use the on-line
> redo. Your DB will come up, but don't expect it to stay that way, ours
> crashed with "block corruption" 45 minutes later. Believe me, we tried
> it due to a misinformed EMC person(that doesn't happen does it?) & failed.
Problem there is that to get the oracle database that handles
the export back online you have automate a recover until current
(or check the current numbers and do a recover until the number).
Especially if the existing database is running parallel server
and the new one isn't it's a pain.
Also: don't know if it's been cleared up in 9, but we used
go through some gyrations to make the split off copy of the
database use a different network name so that the backup server
didn't come up answering the production database's queries.
Another luvly thing about the recover is that Oracle8 (may have
changed) opens the redo's w/ O_READ | O_WRITE. If the production
system exports them NFS read only (which makes sense to them)
you have to copy the files locally in order to replay them.
Doesn't sound like much but if you are trying to catch up to
a heavily loaded database it can make a big difference.
The attached aren't the prettiest code in the universe but
at least worked.
--
Steven Lembark 2930 W. Palmer
Workhorse Computing Chicago, IL 60647
+1 800 762 1582
##############################################################################
#
# $Author: lems1 $
# $Date: 1998/11/12 07:06:53 $
# $Revision: 1.8 $
# $Name: $
#
##############################################################################
package Substandard;
#use strict;
use vars qw( $VERSION @ISA @EXPORT );
require Exporter;
@ISA = qw( Exporter );
# Items to export into callers namespace by default. Note: do not export
# names by default without a very good reason. Use EXPORT_OK instead.
# Do not simply export all your public functions/methods/constants.
@EXPORT = qw();
@EXPORT_OK = qw( send recv chat flush timeout clear_error );
$VERSION = '0.01';
##############################################################################
use Carp;
use Symbol;
use IPC::Open3;
# dealing with sub-processes requires some kind of boilerplate
# for defunct children. callers can munge this to whatever
# they like. this is pretty minimal but avoids major nasties.
$SIG{PIPE} = sub { print STDERR 'SIGPIPE handled' };
# create a new svrmgrl object
sub new
{
# create anon hash and bless it into the package requested.
my $self = bless {}, shift;
$self->{isopen} = 0;
$self->{usable} = 0;
$self->{error} = 1;
$self->{command} = shift;
return $self unless $self->{command};
# use file descriptors with open3. open3 requires existing symbol table
# entries for the file descriptors.
$self->{fd_write} = gensym();
$self->{fd_read} = gensym();
$self->{fd_err} = gensym();
# fork off the command, opening its stdout, stdin & stderr into the
# file descriptors we hand it. if the pid isn't returned then
# give up.
$self->{error} = 2;
$self->{pid} = open3( $self->{fd_write}, $self->{fd_read}, $self->{fd_err},
$self->{command} );
return $self unless $self->{pid};
$self->{timeout} = 30; # standard timeout for selects
# select requires bit masks for the file descriptors to check. this
# sets up the bit masks for each of the file descriptors returned
# by open3.
$self->{select_write} = $self->{select_read} = $self->{select_err} = '';
vec( $self->{select_write}, fileno($self->{fd_write}), 1 ) = 1;
vec( $self->{select_read}, fileno($self->{fd_read}), 1 ) = 1;
vec( $self->{select_err}, fileno($self->{fd_err}), 1 ) = 1;
# at this point the object is both open and usable.
# future errors will normally leave the object open
# and unusable ({useable} == 0) with the error = $!.
$self->{isopen} = 1; # open for use.
$self->{usable} = 1; # clean, so far...
$self->{error} = 0; # none, yet...
# hand the popluated hash reference back to the caller.
$self
}
# close the item down after to goes out of scope.
# alwasys returns zero.
sub DESTROY
{
my $self = shift;
$self->flush if( $self->{isopen} && $self{usable} );
close $self->{fd_err};
close $self->{fd_read};
close $self->{fd_write};
$self->{isopen} = 0; # not open
$self->{usable} = 0; # not usable
$self->{error} = 0; # no way to handle errors anyway.
}
# get a message back from the server manaer
sub select_and_sysread
{
my( $fd, $sel, $time ) = @_;
my $result = '';
my ( $i, $j, $k );
while( (select($i=@$sel[0], $j=@$sel[1], $k=@$sel[2], $time) )[0] > 0 )
{
my $buffer = '';
if( sysread $fd, $buffer, 8192 )
{
if( $! )
{
confess "failed sysread: $!";
}
elsif( $buffer =~ m/\S/sg )
{
# if we find at least one non-space character on the
line then
# split it up and store the result.
$result .= $buffer;
$time = 1;
}
}
}
[ split m/\n/, $result ]
}
sub recv
{
my $self = shift;
$self = shift unless ref $self;
# return empty list unless the object is in a useful
# state for reading.
return () unless $self->{isopen} && $self->{usable};
my $fd = $self->{fd_err};
my $sel = [ ( undef, undef, $self->{select_err} ) ];
my $err = select_and_sysread($fd, $sel, 0 );
my $fd = $self->{fd_read};
my $sel = [ ( $self->{select_read}, undef, undef ) ];
my $time = $self->{timeout};
my $out = select_and_sysread( $fd, $sel, 30 );
$self->{reply} = [ ( @$err, @$out ) ];
@{ $self->{reply} }
}
# remove any pending input from the input stream
# and discard it.
sub flush
{
my $self = shift;
# set the object to use polling then grab, discard any input.
my $time = $self->{timeout};
$self->{timeout} = 0;
# suck up any data remaining in the pipeline
$self->recv;
# reset the timeout.
$self->{timeout} = $time;
# reply set to nothing after the flush.
$self->{reply} = [];
0
}
# send a message to the svrmgrl.
sub send
{
my $self = shift;
$self = shift unless ref $self;
return scalar @_ unless $self->{isopen};
my $timeout = $self->{timeout};
LOOP:
for( ; @_ ; $timeout = 0 )
{
# same basic issues as in recv, above: make local copy of
# the select bit mask, $writeable is 0 or 1 because there is only
# one item in the select list. if the write fd is not ready to
# accept data then quit.
my $write = $self->{select_write} ;
my ( $writeable, $timeleft ) = select( undef, $write, undef, $timeout
);
last LOOP unless $writeable;
# lines are written as-is to allow better control.
# for automatic line breaks, etc, use chat.
my $message = shift;
syswrite $self->{fd_write}, $message, length $message;
}
# scalar @_ will give the number of un-written messages.
# zero return indicates success. caller can use the
# negative of this value to show list of messages unsent
# (e.g., if this returns 2 then @list[-2] was the first
# unsent message).
scalar @_;
}
# pass a set of messages to the server manager, recording the
# result of each line. this is different from send which
# ignores any replys to multiple-line messages. this puts
# the sent messages onto the same list as the replys. mainly
# useful for logging.
sub chat
{
my $self = shift;
$self = shift unless ref $self;
my @result = ();
LOOP:
for( @_ )
{
# break up multi-line input into single lines. makes
# the mixed input and output easier on the eye...
for( split m/\n/, $_ )
{
next unless m/\S/;
s/\t/ /g;
push @result, "> $_";
if( $self->send("$_\n") )
{
push @result, "ERROR: SEND FAILED";
}
else
{
push @result, $self->recv();
}
}
}
@result
}
sub timeout
{
$self = shift;
$self->{timeout} = @_ ? shift : 1 ;
}
sub clear_error
{
my $self = shift;
$self->{error} = '';
$self->{usable} = 1;
}
# Autoload methods go after =cut, and are processed by the autosplit program.
1;
__END__
# Below is the stub of documentation for your module. You better edit it!
=head1 NAME
Svrmgrl - KwikHak perl extension to control a svrmgrl process.
=head1 Synopsis
use Svrmgrl;
new create new object
send @ send message(s) to svrmgrl. each scaler is
sent as is to the server.
recv recieve a message from the server.
chat @ send message(s) to svrmgrl. each scalar is
broken up on newlines with the messages sent
and replys stored in the return list.
quit explicit close.
=head1 DESCRIPTION
Blah blah blah.
=head1 AUTHOR
Steven Lembark <[EMAIL PROTECTED]>
=head1 SEE ALSO
perl(1).
=cut
##############################################################################
#
# $Author: lems1 $
# $Date: 1998/11/12 07:07:20 $
# $Revision: 2.6 $
# $Name: $
#
##############################################################################
package Svrmgrl;
#use strict;
use vars qw($VERSION @ISA @EXPORT);
require Exporter;
@ISA = qw( Exporter Substandard );
# Items to export into callers namespace by default. Note: do not export
# names by default without a very good reason. Use EXPORT_OK instead.
# Do not simply export all your public functions/methods/constants.
@EXPORT = qw();
@EXPORT_OK = qw();
$VERSION = '0.01';
use lib '/dbi-processing/lib/site-perl';
use Muchoruno; # parallel execution and logging.
use Substandard; # sub process management.
##############################################################################
use Carp;
# Preloaded methods go here.
# create a new svrmgrl object as Svrmgrl->new( ORACLE_SID, connect_string );
# Substandard takes care of the fork and communication.
# dealing with subprocesses and popen requries handling
# SIGPIPE's. this goes here rather than Substandard because
# we know how we are logging the data, Substandard doesn't.
$SIG{PIPE} = sub { postlog( 'notice', 'DOA process detected via SIGPIPE' ); exit 0; };
sub new
{
# set up environment so that the sub-process created via
# substandard will be run properly. the first argument
# from the caller. perl puts the package id on the stack.
my $package = shift;
my $sid = shift;
fubar "bogus Svrmgrl create: no ORACLE_SID", 0
unless $sid;
my $connect = @_ ? shift : 'internal';
# set up the oracle environment.
# now find the ORACLE_HOME from the /etc/oratab for the
# ORACLE_SID we are using. grep the first entry from
# /etc/oratab with the ORACLE_SID, split it apart on
# colons and take the second entry.
fubar "failed to open oratab: $!"
unless open ORATAB, "< /etc/oratab";
my $home = ( split m/:/, (grep m/^$sid/, <ORATAB>)[0] )[1] ;
close ORATAB;
fubar "no ORACLE_HOME entry for $ENV{ORACLE_SID} found in /etc/oratab"
unless $home;
# set up shell environment
$ENV{ORACLE_SID} = $sid;
$ENV{ORACLE_HOME} = $home;
# TWO_TASK shouldn't be set for internal connects should be set
# to the ORACLE_SID for others.
if( $connect eq 'internal' )
{
delete $ENV{TWO_TASK} if exists $ENV{TWO_TASK};
}
else
{
$ENV{TWO_TASK} = $ENV{ORACLE_SID};
}
postlog( 'debug', 'Svrmgrl', qx(echo "TWO_TASK == '\$TWO_TASK'") );
# the server manager command will come from $ORACLE_HOME/bin,
# and must be adjusted for users running this as superuser.
my $svrmgrl = "$ENV{ORACLE_HOME}/bin/svrmgrl";
my $command = $> ? $svrmgrl : "/usr/local/bin/su -p oracle --command
$svrmgrl";
# at this point it seems reasonable to try and fork off the
# svrmgrl process.
postlog(
'info', 'Svrmgrl',
"oracle_sid = $ENV{ORACLE_SID}",
"oracle_home = $ENV{ORACLE_HOME}",
"connect id = $connect",
"command = $command",
);
# create new Substandard object.
# which is then blessed into this package.
# this means we have to clean up after the function.
# could also use Substandard::new $package, $command;
my $self = Substandard->new( $command );
bless $self, $package;
# make logging prettier with multiple sessions open.
# pid is used in various places for tracking things.
$self->{sid} = $sid;
# default select timeout
$self->{timeout} = 60;
# make tracking syslog entrys for multiple-connect
# chat's manageable.
$self->{logid} = $connect eq 'internal' ? $sid : "$sid:$connect" ;
# show off the rest of our object
postlog(
'info', 'Svrmgrl',
"process id = $self->{pid}",
"logging id = $self->{logid}",
"timeout = $self->{timeout}",
);
# make the connection. uses Svrmgrl->chat which logs
# the send and recv to debug and chat to info.
$self->chat( "connect $connect" );
# no reply from oracle, may be a sync issue on the
# recv. try it again one time to be sure...
$self->recv
unless( @{$self->{reply}} );
# warn about connect failures.
postlog( 'notice', 'Svrmgrl', 'may have failed on local connect',
@{$self->{reply}} )
unless( @{$self->{reply}} && grep m/Connect/, @{$self->{reply}} );
## used for testing only...
##
## # this has caused us more problems than anything else...
## # just to be sure, have svrmgrl show what it sees the
## # TWO_TASK environment variable set to via echoing it
## # from a shell escape w/in svrmgrl.
##
## postlog( 'debug', 'Svrmgrl', $self->chat( q(!echo "TWO_TASK is set to
'$TWO_TASK'") );
# hand back the popluated hash reference to our caller.
$self
}
# automatic quit for out-of-scope variables.
sub DESTROY
{
my $self = shift;
$self->quit;
postlog( 'debug', 'final words', Substandard->recv( $self ) );
Substandard->DESTROY( $self );
0
}
# quit svrmgrl and save last words in $self->{reply}.
# when the variable goes out of scope then DESTROY
# will handle cleaning it up and any required cleanups
sub quit
{
my $self = shift;
$self->send( "quit\n" );
$self->{usable} = 0;
@{ $self->{reply} }
}
##############################################################################
# send, recv & chat simply log what passes through them and then call
# the Substandard versions.
##############################################################################
sub send
{
my $self = shift;
postlog( 'debug', "Svrmgrl ($self->{logid}) : send", @_ );
Substandard->send( $self, @_ )
}
sub recv
{
my $self = shift;
postlog( 'debug', "Svrmgrl ($self->{logid}) : recv", Substandard->recv($self,
@_) );
@{$self->{reply}}
}
sub chat
{
my $self = shift;
postlog( 'info', "Svrmgrl ($self->{logid}) : chat >", @_ );
Substandard->chat( $self, @_ );
postlog( 'info', "Svrmgrl ($self->{logid}) : chat <", @{$self->{reply}} );
@{ $self->{reply} }
}
# make use happy
1
__END__
# Below is the stub of documentation for your module. You better edit it!
=head1 NAME
Svrmgrl - KwikHak perl extension to control a svrmgrl process.
=head1 Synopsis
use Svrmgrl;
new create new object
quit explicit close, saves reply in object and sets usable
flag to zero.
DESTROY automatically calls quit when object goes out of scope.
send
recv
chat
call Substandard->blah with logging using the
$self->{logid} text..
=head1 DESCRIPTION
Connection to Svrmgrl.
=head1 AUTHOR
Steven Lembark <[EMAIL PROTECTED]>
=head1 SEE ALSO
perl(1).
=cut
#!/usr/local/bin/perl -w
##############################################################################
#
# $Author: lems1 $
# $File: $
# $Date: 1998/11/12 07:08:45 $
# $Revision: 6.6 $
# $Name: $
#
# see below for pod.
##############################################################################
##############################################################################
# command line arguments.
#
# switches:
#
# -r == recover only. skips all of the volume group operations,
# use to test code or after doctoring the remote logs, etc,
# to avoid failures due to abberrations.
#
# -q == quit after sync, useful for testing recovery portion without
# having to resync each time. can run with -q first to sync the
# db and later with -r to replay the logs.
#
# take arguments:
#
# -i == which instince to restart. must be key of %dbinfo.
#
# -t == cutoff time for replay. default is 3600 seconds (1hr).
#
# -T == time of day to proceed through before giving up replaying
# logs. may cause playback that runs over cutoff time.
##############################################################################
##############################################################################
# perl configuration
##############################################################################
# CPAN modules
use Carp; # error message handling
use File::Basename; # file parsing
use File::Copy; # copy files by name
# getopts( 'i:t:qr' );
use Getopt::Std;
use vars qw( $opt_i $opt_q $opt_r $opt_t );
# homegrown modules
use lib '/dbi-processing/lib/site-perl';
use Muchoruno; # initsyslog(), postlog(), fubar()
use Svrmgrl; # svrmgrl object for talking to orace.
use Pidfile; # pidfile object for semaphore file.
# perl output configs
$\ = "\n"; # output newlines after strings on output.
$, = "\n"; # output newlines between list elements on
output.
##############################################################################
# user configurable parameters
# anything site-specific should be here.
##############################################################################
##############################################################################
# user environment
# needed for lvm operations. this will normally be on the path
# of anyone running as superuser (e.g., root) but can't hurt.
$ENV{PATH} = "/sbin:/usr/sbin:/usr/local/bin:/usr/bin:/usr/sbin";
##############################################################################
# oracle configuration
# data used to manage specific instances.
%dbinfo =
(
bcvs =>
{
# string to connect into the production database after the sync is
# complete. this is used to query the production systems v$log_history
# for archive files during the local system startup -- when the tables
# are not yet accessable locally. note that if the link to produciton
# is changed then this needs to be updated also.
remote => 'dbwh/dbwh@prod',
# where to look for archive source files. archdir is the local
# directory into which the files will be copied, $archsrc is where
# the production copies of the archive logs will be nfs mounted.
# simplest solutin os to nfs mount the various production archive
# dir's next to the archdir, makes the archsrc file glob easier.
archdir => '/u01/app/oracle/admin/prod/arch/',
archsrc => '/u01/app/oracle/admin/prod/arch/../?arch/',
# this will be used to store the Svrmgrl object for this
# sid once it's created. there is no syntatic necessity
# for putting this here but it makes debugging easier.
svrmgrl => 0,
# this is used to find the change numbers in the log file.
# the logfile contains lines formatted as:
# /^After: <change number> <changeid>$/
# which are scanned for the maximum change number. this
# is the change number that the local database must be
# recovered through in order to be fully recovered. defnining
# maxchange here is syntatic sugar to avoid undefined variable
# warnings; it is not necessary for correct execution.
changeid => 'prod',
maxchange => 0,
# this is issued to oracle as-is, line by line to start up the
# oracle instance after the sync is complete.
startup =>
[
q{ startup nomount
pfile='/u01/app/oracle/admin/bcvs/pfile/initbcvs_recover_734.ora'; },
q{ @/u01/app/oracle/admin/bcvs/pfile/cr_controlfile.sql; },
],
# this is issued to oracle as-is after the startup is complete
# to manage user id's, etc.
housekeep =>
[
q{ alter user sysadm identified by mdasys; },
q{ alter user dbwh identified by whse; },
q{ grant connect, bscs_role to dbwh; },
],
},
# see above for item definitions.
wrtx =>
{
remote => 'dbwh/dbwh@rtx',
archdir => '/u01/app/oracle/admin/wrtx/arch/',
archsrc => '/u01/app/oracle/admin/wrtx/arch/../?arch/',
svrmgrl => 0,
changeid => 'rtx',
maxchange => 0,
startup =>
[
q{ startup nomount
pfile='/u01/app/oracle/admin/wrtx/pfile/initwrtx_recover734.ora'; },
q{ @/u01/app/oracle/admin/wrtx/pfile/cr_controlfile.sql; },
],
housekeep =>
[
q{ alter user sysadm identified by mdasys; },
q{ alter user dbwh identified by whse; },
q{ grant connect, rtx_select to dbwh; },
],
},
);
# for confienence -- saves using keys %dbinfo all over the place
@dbnames = sort keys %dbinfo;
# query to determine archive file name from thread and change numbers
# during database recovery. this should return a single value, the
# archive file name.
#
# the THREAD and CHANGE tokens will be replaced with the
# thread and change numbers requested by oracle.
#
# newlines will be stripped out of it before sending to oracle to save
# log space.
#
# beware of perl metachar's or use single quotes (i.e., q{}).
my $logquery =
q{
select
archive_name
from
v$log_history
where
thread# = THREAD
and
low_change# <= CHANGE
and
high_change# >= CHANGE
;
};
##############################################################################
# disk files used to manage execution
# touch this file to stop replaying oracle logs and bring
# the database on line. this avoids political issues with
# issuing signals to the program.
$abortfile = '/dbi-processing/run/bcvs-replay-abort';
# if this is present the job will post a log entry and exit
# without doing anything. this allows the operators to skip
# an execution w/o having to be root and play with crontabs.
$norunfile = '/dbi-processing/run/DO-NOT-PERFORM-SYNC';
# where to put the log file of the remote execution. used mainly
# for debugging failures and re-running syncs to test parsing of
# the remote text.
$logfile = '/dbi-processing/log/bcvs-sync.log';
# where to look for process id file used as semaphore for
# package alredy running. pidfiles live here and look like
# "$rundir/sync-$ORACLE_SID.pid".
$rundir = '/dbi-processing/run/';
##############################################################################
# specific to remote execution for the emc sync operation
# this is the thing that gets run on the remote system to perform
# the actual sync and split and where it gets run. the user must
# have a .rhosts on the remote system. this has nothing at all to
# do with the local oracle user id used in the connects, though
# they may be the same.
$remote_host = 'uranus';
$remote_user = 'dbwh';
$remote_script = '/opt/symmapps/dwh_35_bcv.sh';
##############################################################################
# end of user configurable parameters
##############################################################################
##############################################################################
# oracle database control
#
# these all take the oracle sid as their sole argument and get the rest
# from %dbinfo using it.
##############################################################################
sub oracle_shutdown
{
my $sid = shift;
my $svr = $dbinfo{$sid}{svrmgrl};
postlog( 'info', 'shutdown', $svr->chat('shutdown abort') );
# quick check: make sure the sgadefprod.dbs isn't there. whatever
# this thing is it causes problems if left behind...
-f "$ENV{ORACLE_HOME}/dbs/sgadef$ENV{ORACLE_SID}.dbs"
and fubar "found the sgadef$ENV{ORACLE_SID}.dbs";
# some file system housekeeping. this can be done
# either before or after the sync.
-f "$ENV{ORACLE_HOME}/dbs/sgadef$ENV{ORACLE_SID}.dbs"
and fubar "found the sgadef$ENV{ORACLE_SID}.dbs";
# make sure writable, empty logs exists for oracle
# these will be created in the local directory.
for( 'recover.lst','shutdown.lst','startup.lst' )
{
next unless -e;
unlink or fubar "failed to unlink $_: $!";
open JUNK, "> $_" or fubar "failed to open $_: $!";
close JUNK or fubar "failed to close $_: $!";
chmod 0777, $_ or fubar "failed to chmod $_: $!";
}
}
# whatever needs to be done to tidy things up after the recover
# is complete.
sub oracle_housekeep
{
my $sid = shift;
my $svr = $dbinfo{$sid}{svrmgrl};
postlog( 'info', 'housekeep', $svr->chat(@{$dbinfo{$sid}{housekeep}}) );
}
# restart the database by bringing it up and creating a new control file.
sub oracle_restart
{
my $sid = shift;
my $svr = $dbinfo{$sid}{svrmgrl};
my $dir = $dbinfo{$sid}{archdir};
postlog( 'info', "restart $svr->{sid}" );
{ # clean up any stale archive files
postlog( 'info', "clean up $dir" );
my @stale = glob( "$dbinfo{$sid}{archdir}/*" );
unlink @stale if @stale;
}
# need to make sure it's down in the first place then
# feed these verbatum into oracle -- startup, file specs,
# .sql scripts, whatever. the oracle_shutdown does some
# housekeeping and sanity checks ot make sure the database
# seems restartable on the way back in. that should be
# run before the volume group operations to make sure that
# everything's done properly.
$svr->chat( 'shutdown abort;', @{$dbinfo{$sid}{startup}} );
}
# replay the logs and put the database online.
#
# note: this is the messiest part of the whole procecure.
#
# once the database has been sync'd the first step in bringing it
# up again is to start it in oracle `recovery' mode and replay logs
# of any xactions in process during the split. after that the
# instance is shut down one more time and restarted with the normal
# pfile. this leaves the database up and running.
#
# steps for restarting the database in svrmgrl:
#
# connect internal
# startup nomount pfile='$sync_pfile';
# @$sync_sql;
# recover automatic until change '$max_change'
# using backup controlfile;
#
# <interactive dialog for recover automatic goes on here>
#
# alter database open resetlogs;
sub oracle_replay
{
my $sid = shift;
postlog( 'info', "beginning log replay for $sid" );
# save typing, speeds things up on frequently accessed values.
my $svr = $dbinfo{$sid}{svrmgrl};
my $dir = "$dbinfo{$sid}{archdir}/";
my $src = "$dbinfo{$sid}{archsrc}/";
# check that the database is up at all -- might not be
# if in log replay due to a failed restart.
oracle_restart $sid
if grep { /not open/ } $svr->chat( "desc user_tables;" );
# remove this file if it exists, doublecheck
# if the file sill exists and die if it does.
-e $abortfile && unlink $abortfile;
fubar "failed to unlink $abortfile: $!", 0
if -e $abortfile;
# this is used to query the production system's v$log_history for
# archive file names corresponding to thread and change numbers
# requested by the archive recover operation.
my $pluto = Svrmgrl->new( $sid, $dbinfo{$sid}{remote} );
# deal with long restarts.
#
# cutoff says how long to replay before checking if we have passed
# the maxseq in the files. $maxchange is the change number in
# effect right after the database is taken out of hot backup mode.
# if changes are played back to this number then the database can
# safely be started. taken together the two values say that we can
# gracefully exit the replay loop if we are overtime and the database
# is restatable.
my $maxchange = $dbinfo{$sid}{maxchange};
my $cutoff = time + $maxtime;
my $restartable = 0;
my $overtime = 0;
postlog(
'info', 'cutoff',
"cutoff time is $cutoff",
"current time is " . time,
"safe change no. $maxchange"
);
# need to increase the timeout to allow for log replay in the
# chat sections. scale of miuntes is gross overkill but it is
# less painful to wait than to fail.
$svr->{timeout} = 300;
$pluto->{timeout} = 300;
# start recovering the database.
# this generates the first thread, change number message used to
# get the first logfile. after that feeding the log file names
# into svrmgrl gets successive requests. the requests are fed
# into a query againsed production's v$log_history to get the
# required log file. when the query returns no further rows
# then CANCEL is fed into the recover and the database is started.
$svr->chat( "recover automatic until cancel using backup controlfile;\n" );
# this will loop until either a fatal error or successful recovery is
# found in $svr->{reply} fatal errors will exit here via fubar.
LOOP:
for(;;)
{
# sanity checks first: requested abort, running overtime and
# it's safe to restart.
# if this file exists then stop playing back the logs and
# try to bring the database on line. this is used in cases
# where there is too much log data to reply it -- either the
# sync took too long or new logs are geing generated faster
# than we can replay them.
if( -e $abortfile )
{
postlog(
'notice', 'restart',
'',
"found file $abortfile",
'user requested stop of log replay.',
'stopping log replay and bringing the database on line
now.',
'',
);
postlog(
'notice', 'restart',
'',
"change value $maxchange not yet reached",
"during log replay. the database may not",
"start up properly.",
"please escalate any restart errors.",
'',
) unless $restartable;
# exit the replay loop and fall through to
# database startup commands.
last LOOP;
}
# notice whether we have passed the cutoff time and post
# a warning if necessary. checking $overtime avoids
# posting the message a zillion times and making a mess
# of the logs.
if( ! $overtime && time > $cutoff )
{
local $now = time;
postlog(
'notice', 'restart',
'',
"past replay cutoff time of $cutoff." ,
"current time is $now.",
'log replay will stop automatically after',
"change number $maxchange is reached",
'and the database can safely be restarted',
'',
);
++$overtime;
}
# if we have passed the cutoff time and the startup can
# be safely stopped then post a notice the fact and give up.
if( $overtime && $restartable )
{
postlog(
'notice', 'restart',
'',
'this is not an error',
'time limit for log replay exeeeded and',
'required logs have been applied to the database.',
'log replay ending, the database will now start up',
'please escalate any errors during the startup',
'',
);
last LOOP;
}
# at this point it seems reasonable to go ahead replaying
# this log file.
# search for oracle messages in the reply. if there
# are none then something is wrong...
my @reply = grep { /ORA-\d+/ } @{ $svr->{reply} };
# this is currently a bit on the paranoid side. this could
# silently retry and deliver a warning message. for now it
# seems safer to put up the first message in case of a problem.
unless( @reply )
{
postlog(
'info', 'restart',
'empty reply from local bcvs instance in restart',
'retrying now.',
);
# try it again...
$svr->recv;
# if we have data then tell the operators not to worry,
otherwise
# tell them to escalate the issue.
if( @{$svr->{reply}} )
{
postlog(
'info', 'restart',
'retry has data.',
'execution continuing normally.',
'escalation is not necessary.',
);
}
else
{
postlog(
'notice', 'restart',
'',
'empty reply from production after retrying
query',
'please escalate',
'execution will continue and may post this
message again,',
'this only needs to be escalated once',
'',
);
}
next LOOP;
}
# use an empty reply to flag failing to query for a file name
# in the section below. this should be filled in by chat in
# section that queries production for the file name -- see
# $log_query below.
$pluto->flush;
# manage the reply from feeding oracle the recover automatic
# and file name(s) required for startup. this should either
# get the change and thread from $svr->{reply} or issue a sensible,
# fatal message on detecting a problem and die. any non-
# numbered message (lacking ORA- in front of it) can be safely
# ignored at this point -- will normally only be the Log applied
# message handled below.
REPLY:
for( @reply )
{
# this is good news, everything below this (elsif or else)
# in this if-ladder is bad news and should at the very least
# post a notice into the logs.
if( m/ORA-00279/ )
{
# extract thread and change number from oracle message.
my ( $change, $thread ) =
/Change\s+(\d+).*thread\s+(\d+)/;
postlog( 'info', 'change number', $change );
postlog( 'info', 'thread number', $thread );
# this is very, very bad news...
unless( $thread && $change )
{
postlog(
'notice', 'replay',
'',
'unable to detect change/thread in
oracle reply',
'please escalate',
'line scanned is shown below',
$_,
'',
);
fubar "failed parsing ORA-00279 message", 0;
}
# so far, so good...
if( ! $restartable && $change >= $maxchange )
{
postlog(
'notice', 'maxchange',
'',
'this is not an error',
"now applying change $change.",
'log replay can be aborted and the
database',
'safely restarted if time limits
require.',
'replay is continuing normally at this
time.',
'',
);
++$restartable;
}
# at this point the change looks reasonable. if it is
# above the required max_change# from prod at the end
# of hot backup then the database can safely be
restarted
# without replaying more logs.
# just to keep track of what's going on...
postlog(
'debug', 'replay',
"time = " . time,
"cutoff = $cutoff",
"overtime = $overtime",
"change = $change",
"maxchange = $maxchange",
"$restartable = $restartable",
);
# this needs to jive with the definition of $logquery
# in the user-configurable section at the top.
stripping
# excess white space neatens up the logs.
my $query = $logquery;
$query =~ s/THREAD/$thread/gs;
$query =~ s/CHANGE/$change/gs;
$query =~ s/\s+/ /gs;
# the file will be on a line by itself in the
$pluto->{reply}.
for( $pluto->{reply} = [] ; @{$pluto->{reply}} == 0 ; )
{
$pluto->chat( $query );
}
# done scanning the reply, fall through to the portion
that
# extracts the file name, copies the file and feeds
the name
# back to oracle.
last REPLY;
}
elsif( /ORA-00289/ )
{
# ignore oracle's suggesions. they may be wrong
# anyway.
next REPLY;
}
elsif( /ORA-(00283|01507)/ )
{
# whole thing failed due to oracle internal error
postlog(
'notice', 'replay',
'',
'oracle internal error during database
restart',
'please escalate',
'oracle error status shown below',
@{$svr->{reply}},
'',
);
fubar( 'oracle failure dectected during restart', 0 );
}
elsif( /ORA-00936/ )
{
# bad syntax -- probably a coding error in this
program.
postlog(
'notice', 'FATAL',
'',
'bad syntax in file name query to production',
'please escalate',
'oracle error status shown below',
@{$svr->{reply}},
'',
);
fubar( 'oracle failure dectected during restart', 0 );
}
elsif( /ORA-OO278/ )
{
# the log file is no longer required for this replay.
# this isn't necessary fatal. oracle sometimes gives
# this message when thread change numbers overlap and
# it can skip a file in the replay sequence. however,
# is also can be caused by a logic loop failing to
# update $svr->{reply} and feeding the same file into
oracle
# over-and-over-and... thing to notice in the logs
# is whether the same file keeps repeating or a single
# file name shows up one time.
postlog(
'notice', 'replay',
'',
'this is not an error by itself',
'please escalate only if it repeats',
'possible thread/change detection loop',
'oracle reply shown below',
$_,
'',
);
}
elsif( /ORA-01503/ )
{
# failed to create the control file. this is fatal.
postlog(
'notice', 'replay',
'',
'failed to create control file',
'please escalate',
'oracle error message shown below',
$_,
'',
);
fubar( 'oracle failure detected during restart', 0 );
}
elsif( /ORA-01565/ )
{
# missing one of the .dbf files. causes include
missing
# or bad soft link in /u02/oradata/prod, missing
logical
# volume in /dev/bcvgXXX or missing volume group.
#
# either way, this is fatal.
#
# as a quick check, look for the .dbf file in
/dev/bcvg*
# to give a more useful error message.
my $file = m/'.*'/;
$base = basename( $file );
my @list = glob( "/dev/bcvg*/$base" );
postlog(
'notice', 'replay',
'',
'failed to locate .dbf file',
'please escalate',
'oracle reply shown below',
$_,
'',
);
postlog( 'notice', 'restart', "$file not on disk or
bad soft link" )
unless -e $file;
postlog( 'notice', 'restart', 'checking for volume
groups gives', @list );
fubar( 'oracle failure detected during restart', 0 );
}
elsif( /ORA-00310/ )
{
# mismatch between the sequence number requested by
# production and the one we give it.
postlog(
'notice', 'replay',
'',
'sequence mis-match in reply from production',
'please escalate',
'oracle reply shown below',
$_,
'',
);
fubar( 'unable to determine required log sequence', 0
);
}
elsif( /ORA-01034/ || /ORA-01507/ )
{
# this is fatal if it repeats. it means that oracle
did
# not start up properly and is not available for the
replay
# operation. repeat means that the retry failed.
postlog(
'notice', 'replay',
'',
'failed to start up oracle for archive log
replay',
'please escalate if this message repeats.',
'oracle reply shown below',
$_,
'',
);
oracle_restart $sid;
$svr->chat( "recover automatic until cancel using
backup controlfile;\n" );
next LOOP;
}
else
{
# oracle message we don't know how to deal with,
notice the
# fact. slowly but surely this should produce fewer
and
# fewer notices...
postlog(
'notice', 'replay',
'',
'oracle reply unmatched by the if-ladder:',
'this is not an error by itself',
'please do not escalate',
'oracle reply shown below',
$_,
'',
);
}
}
# at this point we should have stored a reply from querying prod for
# the file name into $pluto->{reply}. if this is still a reference
# to an empty array then we never ran chat or got nothing back in the
# section above and cannot proceed.
fubar "empty reply from production oracle, check debug logs"
unless @{ $pluto->{reply} };
# snag the file name from the reply.
# the split removes white space. $file is a list because
# that's what map wants to return.
#
# if $file is empty then we have reached the end of our
# replay logs and should cancel out of the recovery and
# bring the datbase on line. issuing the CANCEL command
# should return the media recovery complete message --
# which exits this loop.
my ( $file ) = map { split } grep { m{/\d+arch\d+\.arc} } @{
$pluto->{reply} };
# snag the file name from the reply.
# the split removes white space. $file is a list because
# that's what map wants to return.
#
# if $file is empty then we have reached the end of our
# replay logs and should cancel out of the recovery and
# bring the datbase on line. issuing the CANCEL command
# should return the media recovery complete message --
# which exits this loop.
if( $file )
{
postlog( 'info', 'archive file ', $file );
# copy the file from the NFS mounted source directory into
# the local directory. this is necessary because oracle
# opens the archive file for writing even though it only
# reads the file and the NFS volumes are mouted read-only.
my ( $srcfile ) = glob( $src . basename($file) );
# this copy has to be done as oracle since the root user
# is mapped to `nobody' on the remote system.
my $command = "su -p oracle --command 'cp $srcfile $file
2>&1;'";
postlog( 'debug', 'copy', $file, $srcfile, $command,
qx($command) );
# doublecheck that the source and dest files match, keep
# in the logs.
postlog( 'notice', 'restart', "oddity: $srcfile not the same
size as $file after copy" )
unless -s $srcfile == -s $file;
# feed the file name into svrmgrl, this generates the
# next reply processed above.
#
# also double-check that the log seems applied properly and
# post a notice if it doesn't seem to be.
$svr->chat( $file );
if( grep { /Log applied/ } @{ $svr->{reply} } )
{
postlog( 'info', 'restart', 'Log applied' );
}
else
{
postlog( 'notice', 'restart', 'failed to find "Log
applied" reply', @{$svr->{reply}} );
}
# remove archive log file after it is used.
unlink $file;
}
else
{
# finished with recovery, exit this loop and fall
# through to the database startup.
last LOOP;
}
}
# got this far means it's time to try restarting the database.
# first step is to stop the recovery.
# then open the database for use.
#
# this will post a notice if it doesn't find a succesful
# startup and then keep going -- no telling how many ways
# Oracle has to not tell us that it succeeded in the startup.
# set a more sensable timeout for the remaining operations
$svr->{timeout} = 60;
postlog( 'info', 'finished applying log files' );
$svr->chat( 'CANCEL' );
$svr->chat( 'alter database open resetlogs;' );
postlog(
'notice', 'restart',
'',
'failed to detect successful restart',
'after cancelling log replay.',
'this is not fatal by istelf.',
'please escalate any startup errors.',
'final message shown below:',
@{$svr->{reply}},
'',
) unless grep { /Statement processed/ } @{$svr->{reply}};
# close the connection here. perl will garbage collect
# the rest later on.
$pluto->quit
}
##############################################################################
# volume group / unix system management
#
# these all require rootly powers.
##############################################################################
sub volume_group_deactivate
{
postlog( 'info', 'deactivate bcvg* volume groups' );
my $oops = 0;
# by this time oracle should be shut down and oracle should be the only
# thing using any of these logical volumes. q.e.d., there's no harm in
# zapping anything that still has one of the lv's open...
qx{ fuser -k /dev/bcvg*/*.dbf };
# now that the database is down and the lv's are unused
# we can safely inactivate the volume groups.
for( glob('/dev/bcvg*') )
{
postlog( 'info', 'deactivate', $_ );
system( "vgchange -a n $_" )
and ++$oops;
}
$oops
}
sub volume_group_activate
{
postlog( 'info', 'activate bcvg* volume groups' );
# should remain zero throughout. nonzero return from
# any of the vgchange operations ups this. nonzero
# return will post an error and re-run the sync.
my $oops = 0;
# re-actiavte the vg's -- since the source of our hardware replication
# (pluto/neptune) is a cluster we need to "vgchange -c n" in order to
# de-clusterify the group before activating it on hyde (which doesn't have
# clustering configured). after that the volumes can be activated.
for( glob('/dev/bcvg*') )
{
postlog( 'info', 'activate ', $_ );
if( 0xff00 & system("vgchange -c n $_") )
{
postlog( 'notice', 'vg activate', "failed vgchange -n $_",
'please escalate', );
++$oops;
}
elsif( 0xff00 & system("vgchange -a y $_") )
{
postlog( 'notice', 'vg activate', "failed to activate $_",
'please escalate', );
++$oops;
}
else
{
postlog( 'debug', "$_ succeeded" );
}
}
$oops
}
sub nfs_mounts
{
# generate a hash of nfs mounts required for oracle
# restart from /etc/fstab.
open FSTAB, "< /etc/fstab"
or fubar "unable to open /etc/fstab: $!";
my %nfsmounts = map { (+split)[1], 1 } grep { m{/oracle/.+\s+nfs} } <FSTAB>;
close FSTAB
or fubar "very wierd: unable to close FSTAB: $!";
fubar "failed to find nfs mounts in fstab"
unless keys %nfsmounts;
postlog( 'debug', 'nfs require', sort keys %nfsmounts );
# next step is to remove the mounted file systems from the list.
open MNTTAB, "< /etc/mnttab"
or fubar "unable to open /etc/mnttab: $!";
for( map { (+split)[1] } grep { m{/oracle/.+\s+nfs} } <MNTTAB> )
{
postlog( 'debug', 'nfs mounted', $_ );
delete $nfsmounts{$_} if exists $nfsmounts{$_};
}
close MNTTAB
or fubar "wierd, unable to close /etc/mnttab: $!";
# whatever's left needs to be mounted
postlog( 'debug', 'nfs mount ', sort keys %nfsmounts );
# quick check: assume same number of items means
# everything required is mounted.
for( sort keys %nfsmounts )
{
qx( mount $_ );
fubar "failed to mount $_"
if $? >> 8 ;
}
}
##############################################################################
# remote execution on unanus & log checking.
##############################################################################
sub bcvs_data_sync
{
postlog( 'info', 'beginning remote execution on uranus' );
my $oops = 0;
# run the remote command.
# output is saved in the logfile for later perusal or replay.
# this is the logfile used with the -r option to this program.
# the logfile contents are put into the syslogs when they
# are examined in the bcvs_log_check, below.
#
# to allow tail -f of the logfile without having to restart the
# thing every time the log is appended to. this will also help
# check sync times...
$command = qq{ su -p $remote_user --command "remsh $remote_host
'$remote_script' > $logfile 2>&1" };
system $command and ++$oops;
# make sure other folks can read the thing.
chmod 0644, $logfile;
$oops
}
sub bcvs_log_check
{
postlog( 'info', 'log check', "checking $logfile" );
open LOGFILE, "< $logfile"
or fubar "failed to open log file: $!";
my @remote = <LOGFILE>;
close LOGFILE;
# log the entire uranus session in the debug logs.
# this can get rather large and needs to be summarized in the info logs.
chomp @remote;
postlog( 'debug', 'remote', @remote );
# whatever checks we like or information that can be gleaned
# from the logs should be extracted here for later use.
# all of the sync scripts have a 'ERROR' in the error messages, none
# of the standard messages do.
my @errors = grep { /^ERROR/i } @remote;
my @success = grep { /Split command successful/ } @remote;
if( @success )
{
# believe that it worked, notify of nonfatal erors.
# errors are probably non-fatal but just to be sure
# anything further should be escalated.
postlog( 'info', 'log check', 'detected successful split' );
if( @errors )
{
postlog(
'notice', 'log check',
'',
'detected possible error on split.',
'this is not fatal by itself.',
'please escalate any further warnings or errors',
'',
);
}
}
else
{
# hope that it worked, notify if possible failure.
# any errors are immediately fatal.
if( @errors )
{
postlog(
'notice', 'log check',
'',
'detected likekly fatal errors during sync',
'please escalate this error',
'execution aborting',
'',
);
fubar( "errors and no sync in logs", 0 );
}
else
{
postlog(
'notice', 'log check',
'',
'failed to detect successful split',
'this is not fatal by itself',
'please escalate.',
'execution continuing',
'',
);
}
}
postlog( 'info', 'sync errors', @errors ) if @errors;
# now to find the max change numbers. this is used in the
# replay section to decide if the database can be safely started.
for( @dbnames )
{
my $sid = $_;
my $changeid = $dbinfo{$sid}{changeid};
for( grep { /^After:.+$changeid/ } @remote )
{
postlog( 'info', 'change no', $_ );
my $change = ( +split )[1];
$dbinfo{$sid}{maxchange} = $change
if $change > $dbinfo{$sid}{maxchange};
}
}
for( @dbnames )
{
postlog( 'info', 'maxchange', "$_ : $dbinfo{$_}{maxchange}" );
}
# return the error status, might be able to return something
# more useful...
scalar @errors
}
##############################################################################
# real execution begins here
##############################################################################
# use local5 for testing, local3 for normal execution.
initsyslog( 'BCVS SYNC', 'local3' );
# there is no way to run this w/o being root since it has to
# su several times during the operation and normally will
# have to perform nfs, LVM and mount operations.
fubar "$0 REQUIRES SUPERUSER ACCESS. ABORTING..."
if $>;
# check for norun file. quit now if it exists.
if( -e $norunfile )
{
postlog(
'notice', 'norunfile',
'',
"found file $norunfile",
"execution is aborting before it starts",
"THIS PROGRAM WILL NOT RUN UNTIL THE",
"FILE LISTED ABOVE IS REMOVED",
"please escalate unless the file was placed there",
"by operators at the warehouse staff request.",
'',
);
exit 0;
}
##############################################################################
# command line arguments.
#
# switches:
#
# -r == recover only. skips all of the volume group operations,
# use to test code or after doctoring the remote logs, etc,
# to avoid failures due to abberrations.
#
# -q == quit after sync, useful for testing recovery portion without
# having to resync each time. can run with -q first to sync the
# db and later with -r to replay the logs.
#
# take arguments:
#
# -i == which instince to restart. must be key of %dbinfo.
#
# -t == cutoff time for replay. default is 3600 seconds (1hr).
#
# -T == time of day to proceed through before giving up replaying
# logs. may cause playback that runs over cutoff time.
##############################################################################
postlog( 'debug', 'ARGV', @ARGV );
getopts( 'i:t:qr' );
# decide which database to act on. mainly useful with -r to
# deal with oddities in starting only one of the databases.
# since @dbnames is the list of currently valid instances if
# $opt_i isn't on the list then the exectuion is fubar.
if( $opt_i )
{
fubar( "bad instance given on command line: $opt_i", 0 )
unless grep { /$opt_i/ } @dbnames;
@dbnames = ( $opt_i );
}
postlog( 'notice', 'syncing', @dbnames );
# -t cutoff (seconds) will stop log replay after this much time
# if the maxchange has been reached.
$maxtime = $opt_t ? $opt_t : 3600;
# -T HHMM (military time) will stop log replay after finding
# a log with this or later timestamp.
$logcutoff = $opt_T ? $opt_T : 0;
# keep multiple syncs from running at once, etc.
# after this point either we have a `lock' on the sync process or will exit.
# the pidfile will be removed when $pidfile goes out of scope.
# nobogus will remove a file that is empty or for a defunct process.
# at this point the local svrmgrl connections are necessary
# regardless.
for( @dbnames )
{
my $pidfilename = "$rundir/sync-$_.pid";
$dbinfo{$_}{pidfile} = Pidfile->new( $pidfilename, 'nobogus' );
fubar "Failed to create $pidfilename"
if $dbinfo{$_}{pidfile}->{status};
$dbinfo{$_}{svrmgrl} = Svrmgrl->new( $_ );
}
if( $opt_r )
{
postlog( 'info', 'replay-log mode, no remote execution or mounts' );
# extract any useful information or success/failure messages
# from the log file. when running this with -r this will
# re-check the existing (possibly edited) log before
# replaying it. this will make no attempt to handle NFS
# volumes, activate volume groups or the like. its only
# purpose is in handling problems caused by mis-managed
# interaction with oracle during the startup procedure.
#
# note: this notes all error-like problems in the log.
# it may be necessary to edit the log and remove these
# by hand before re-running this. the original log is
# preserved in the debug syslogs if the complete, original
# source is ever requried.
bcvs_log_check;
for( @dbnames )
{
oracle_restart $_
if( grep { /Connected to an idle instance/ }
@{$dbinfo{$_}{svrmgrl}->{reply}} );
}
}
else
{
# normal execution goes here: manage the nfs volumes and volume
# groups, actually force the remote execution and check the logs
# for failures.
# check that the required nfs volumes are mounted. this is
# required to make sure the various archive logs are available
# to us locally.
nfs_mounts;
# stop oracle. this has to be done in any case, replaying logs
# or running the sync for real.
for( @dbnames )
{
oracle_shutdown $_;
}
# fly in the ointment: the sync may fail and we may not really
# know it until the volume group activation fails. soooo... the
# whole thing gets wrapped into a loop. the deactivate needs to
# be in here beacuase the activate may partly succeed before
# finding out that one of the emc syncs failed.
while( volume_group_deactivate || bcvs_data_sync || bcvs_log_check ||
volume_group_activate )
{
postlog( 'notice', 'rerunning sync after failed log check' );
postlog( 'notice', 'please escalate if this messages repeats' );
}
# start up the oracle instance. only need this in here because the
# log replay doesn't shut down the database.
for( @dbnames )
{
oracle_restart $_;
}
# this portion *can* take a while, soooooo...
print STDERR "\a\n\a\n** WAKEUP **";
}
# USE THIS FOR DEBUGGING ONLY!!!
exit 0 if $opt_q;
# by this point the sync & split operation has completed
# successfully and the volume groups are on line. now
# comes the fun part: restarting oracle...
# this is the messy part: recover the database using archive
# log files from the production system. then use housekeep to
# tidy up after the restart. anything not specifically
# about the startup or revovery should go here (e.g.,
# cleaning up passwords, assigning roles).
#
# the housekeeping complete, we can safely remove the pidfile.
# following processes can now enjoy their feeding frenzy on the data.
for( @dbnames )
{
oracle_replay $_;
oracle_housekeep $_;
$dbinfo{$_}{pidfile}->remove;
}
# finito
postlog( 'notice', '', '>> FINISHED PACKAGE BCVS SYNC <<' );
0
##############################################################################
__END__
##############################################################################
# documentation.
# this can be processed into man, html, etc, via pod2* programs.
=head1 NAME
sync_bcvs_oracle
=head1 Synopsis
use: sync_bcvs_oracle [-r];
messages are logged via syslog; all stdout text can be safely discarded.
switches:
-r == recover only. skips all of the volume group operations,
use to test code or after doctoring the remote logs, etc,
to avoid failures due to abberrations.
-q == quit after sync, useful for testing recovery portion without
having to resync each time. can run with -q first to sync the
db and later with -r to replay the logs.
take arguments:
-i == which instince to restart. must be key of %dbinfo.
-t == cutoff time for replay. default is 3600 seconds (1hr).
-T == time of day to proceed through before giving up replaying
logs. may cause playback that runs over cutoff time.
=head1 DESCRIPTION
sync_bcvs_oracle performs a sync of the bcvs oracle instance on hyde. This is
performed using a combination of this control script, scripts on the unix node
"neptune" and the emc mirroring capability. After sutting down the local
oracle instnace, this runs scripts on uranus to put the production system into
hot-backup mode, sync the mirror and then split it. When the remote portion
is complete hyde has a copy of the database in hot backup mode. The
difficulty here is that the mirrored database is basically in the same state
it would have been if HP-UX crashed during a hot-backup session. This
requires a recovery from archive log files to get the database back on line.
The messiest portion of the procedure is contained in the subroutine
oracle_restart. This is responsible for getting the oracle instance `bcvs'
back on line. Doing this requires a fair amout of interaction with both the
local copy of oracle and the running instance in production.
All useful logging is done via syslog, normally to the `local3' service.
debug level logs will contain the entire transcript of oracle `chat' calls and
all values returned from external sources (e.g., remote execution, oracle).
The info level logging contains all values fed into oracle for processing
(e.g., archive log file names) and should provide sufficient information to
determine if the operation was successful. The notice level logging will only
output start, completion and error messages.
uses Muchoruno.pm for logging, fubar.
uses Svrmgrl.pm for connections to background svrmgrl programs.
uses Pidfile.pm for pid-containing semaphore file.
=head1 AUTHOR
Steven Lembark <[EMAIL PROTECTED]>
=head1 SEE ALSO
perl(1).
=cut
