Docs pretty much as posted on hackers recently.

SGML compiles cleanly...

Comments/changes welcome.

-- 
  Simon Riggs             
  EnterpriseDB   http://www.enterprisedb.com
Index: doc/src/sgml/backup.sgml
===================================================================
RCS file: /projects/cvsroot/pgsql/doc/src/sgml/backup.sgml,v
retrieving revision 2.83
diff -c -r2.83 backup.sgml
*** doc/src/sgml/backup.sgml	17 Aug 2006 23:04:02 -0000	2.83
--- doc/src/sgml/backup.sgml	14 Sep 2006 18:11:59 -0000
***************
*** 1187,1192 ****
--- 1187,1498 ----
    </sect2>
   </sect1>
  
+  <sect1 id="warm-standby">
+   <title>Warm Standby Servers for High Availability</title>
+ 
+   <indexterm zone="backup">
+    <primary>Warm Standby</primary>
+   </indexterm>
+ 
+   <indexterm zone="backup">
+    <primary>PITR Standby</primary>
+   </indexterm>
+ 
+   <indexterm zone="backup">
+    <primary>Standby Server</primary>
+   </indexterm>
+ 
+   <indexterm zone="backup">
+    <primary>Log Shipping</primary>
+   </indexterm>
+ 
+   <indexterm zone="backup">
+    <primary>Witness Server</primary>
+   </indexterm>
+ 
+   <indexterm zone="backup">
+    <primary>STONITH</primary>
+   </indexterm>
+ 
+   <indexterm zone="backup">
+    <primary>High Availability</primary>
+   </indexterm>
+ 
+   <para>
+    Continuous Archiving can be used to create a High Availability (HA)
+    cluster configuration with one or more Standby Servers ready to take
+    over operations in the case that the Primary Server fails. This
+    capability is more widely known as Warm Standby Log Shipping.
+   </para>
+ 
+   <para>
+    The Primary and Standby Server work together to provide this capability,
+    though the servers are only loosely coupled. The Primary Server operates
+    in Continuous Archiving mode, while the Standby Server operates in a
+    continuous Recovery mode, reading the WAL files from the Primary. No
+    changes to the database tables are required to enable this capability,
+    so it offers a low administration overhead in comparison with other
+    replication approaches. This configuration also has a very low
+    performance impact on the Primary server.
+   </para>
+ 
+   <para>
+    Directly moving WAL or "log" records from one database server to another
+    is typically described as Log Shipping. PostgreSQL implements file-based
+    Log Shipping, meaning WAL records are batched one file at a time. WAL
+    files can be shipped easily and cheaply over any distance, whether it be
+    to an adjacent system, another system on the same site or another system
+    on the far side of the globe. The bandwidth required for this technique
+    varies according to the transaction rate of the Primary Server.
+    Record-based Log Shipping is also possible with custom-developed
+    procedures, discussed in a later section. Future developments are likely
+    to include options for synchronous and/or integrated record-based log
+    shipping.
+   </para>
+ 
+   <para>
+    It should be noted that the log shipping is asynchronous, i.e. the WAL
+    records are shipped after transaction commit. As a result there can be a
+    small window of data loss, should the Primary Server suffer a
+    catastrophic failure. The window of data loss is minimised by the use of
+    the archive_timeout parameter, which can be set as low as a few seconds
+    if required. A very low setting can increase the bandwidth requirements
+    for file shipping.
+   </para>
+ 
+   <para>
+    The Standby server is not available for access, since it is continually
+    performing recovery processing. Recovery performance is sufficiently
+    good that the Standby will typically be only minutes away from full
+    availability once it has been activated. As a result, we refer to this
+    capability as a Warm Standby configuration that offers High
+    Availability. Restoring a server from an archived base backup and
+    rollforward can take considerably longer and so that technique only
+    really offers a solution for Disaster Recovery, not HA.
+   </para>
+ 
+   <para>
+    Other mechanisms for High Availability replication are available, both
+    commercially and as open-source software.  
+   </para>
+ 
+   <para>
+    In general, log shipping between servers running different release
+    levels will not be possible. It is the policy of the PostgreSQL Worldwide
+    Development Group not to make changes to disk formats during minor release
+    upgrades, so it is likely that running different minor release levels 
+    on Primary and Standby servers will work successfully. However, no
+    formal support for that is offered and you are advised not to allow this
+    to occur over long periods.
+   </para>
+ 
+   <sect2 id="warm-standby-planning">
+    <title>Planning</title>
+ 
+    <para>
+     On the Standby server all tablespaces and paths will refer to similarly
+     named mount points, so it is important to create the Primary and Standby
+     servers so that they are as similar as possible, at least from the
+     perspective of the database server. Furthermore, any CREATE TABLESPACE
+     commands will be passed across as-is, so any new mount points must be
+     created on both servers before they are used on the Primary. Hardware
+     need not be the same, but experience shows that maintaining two
+     identical systems is easier than maintaining two dissimilar ones over
+     the whole lifetime of the application and system.
+    </para>
+ 
+    <para>
+     There is no special mode required to enable a Standby server. The
+     operations that occur on both Primary and Standby servers are entirely
+     normal continuous archiving and recovery tasks. The primary point of
+     contact between the two database servers is the archive of WAL files
+     that both share: Primary writing to the archive, Standby reading from
+     the archive. Care must be taken to ensure that WAL archives for separate
+     servers do not become mixed together or confused.
+    </para>
+ 
+    <para>
+     The magic that makes the two loosely coupled servers work together is
+     simply a restore_command that waits for the next WAL file to be archived
+     from the Primary. The restore_command is specified in the recovery.conf
+     file on the Standby Server. Normal recovery processing would request a
+     file from the WAL archive, causing an error if the file was unavailable.
+     For Standby processing it is normal for the next file to be unavailable,
+     so we must be patient and wait for it to appear. A waiting
+     restore_command can be written as a custom script that loops after
+     polling for the existence of the next WAL file. There must also be some
+     way to trigger failover, which should interrupt the restore_command,
+     break the loop and return a file not found error to the Standby Server.
+     This then ends recovery and the Standby will then come up as a normal
+     server.
+    </para>
+ 
+    <para>
+     Sample code for the C version of the restore_command would be be:
+ <programlisting>
+ triggered = false;
+ while (!NextWALFileReady() && !triggered)
+ {
+     sleep(100000L);         // wait for ~0.1 sec
+     if (CheckForExternalTrigger())
+         triggered = true;
+ }
+ if (!triggered)
+         CopyWALFileForRecovery();
+ </programlisting>
+    </para>
+ 
+    <para>
+     PostgreSQL does not provide the system software required to identify a
+     failure on the Primary and notify the Standby system and then the
+     Standby database server. Many such tools exist and are well integrated
+     with other aspects of a system failover, such as ip address migration.
+    </para>
+ 
+    <para>
+     Triggering failover is an important part of planning and design. The
+     restore_command is executed in full once for each WAL file. The process
+     running the restore_command is therefore created and dies for each file,
+     so there is no daemon or server process and so we cannot use signals and
+     a signal handler. A more permanent notification is required to trigger
+     the failover. It is possible to use a simple timeout facility,
+     especially if used in conjunction with a known archive_timeout setting
+     on the Primary. This is somewhat error prone since a network or busy
+     Primary server might be sufficient to initiate failover. A notification
+     mechanism such as the explicit creation of a trigger file is less error
+     prone, if this can be arranged.
+    </para>
+   </sect2>
+ 
+   <sect2 id="warm-standby-config">
+    <title>Implementation</title>
+ 
+    <para>
+     The short procedure for configuring a Standby Server is as follows. For
+     full details of each step, refer to previous sections as noted.
+     <orderedlist>
+      <listitem>
+       <para>
+        Set up Primary and Standby systems as near identically as possible,
+        including two identical copies of PostgreSQL at same release level.
+       </para>
+      </listitem>
+      <listitem>
+       <para>
+        Set up Continuous Archiving from the Primary to a WAL archive located
+        in a directory on the Standby Server. Ensure that both <xref
+        linkend="guc-archive-command"> and <xref linkend="guc-archive-timeout">
+        are set. (See <xref linkend="backup-archiving-wal">)
+       </para>
+      </listitem>
+      <listitem>
+       <para>
+        Make a Base Backup of the Primary Server. (See <xref
+        linkend="backup-base-backup">)
+       </para>
+      </listitem>
+      <listitem>
+       <para>
+        Begin recovery on the Standby Server from the local WAL archive,
+        using a recovery.conf that specifies a restore_command that waits as
+        described previously. (See <xref linkend="backup-pitr-recovery">)
+       </para>
+      </listitem>
+     </orderedlist>
+    </para>
+ 
+    <para>
+     Recovery treats the WAL Archive as read-only, so once a WAL file has
+     been copied to the Standby system it can be copied to tape at the same
+     time as it is being used by the Standby database server to recover.
+     Thus, running a Standby Server for High Availability can be performed at
+     the same time as files are stored for longer term Disaster Recovery
+     purposes. 
+    </para>
+ 
+    <para>
+     For testing purposes, it is possible to run both Primary and Standby
+     servers on the same system. This does not provide any worthwhile
+     improvement on server robustness, nor would it be described as HA.
+    </para>
+   </sect2>
+ 
+   <sect2 id="warm-standby-failover">
+    <title>Failover</title>
+ 
+    <para>
+     If the Primary Server fails then the Standby Server should take begin
+     failover procedures.
+    </para>
+ 
+    <para>
+     If the Standby Server fails then no failover need take place. If the
+     Standby Server can be restarted, then the recovery process can also be
+     immediately restarted, taking advantage of Restartable Recovery.
+    </para>
+ 
+    <para>
+     If the Primary Server fails and then immediately restarts, you must have
+     a mechanism for informing it that it is no longer the Primary. This is
+     sometimes known as STONITH (Should the Other Node In The Head), which is
+     necessary to avoid situations where both systems think they are the
+     Primary, which can lead to confusion and ultimately data loss.
+    </para>
+ 
+    <para>
+     Many failover systems use just two systems, the Primary and the Standby,
+     connected by some kind of heartbeat mechanism to continually verify the
+     connectivity between the two and the viability of the Primary. It is
+     also possible to use a third system, known as a Witness Server to avoid
+     some problems of inappropriate failover, but the additional complexity
+     may not be worthwhile unless it is set-up with sufficient care and
+     rigorous testing.
+    </para>
+ 
+    <para>
+     At the instant that failover takes place to the Standby, we have only a
+     single server in operation. This is known as a degenerate state.
+     The former Standby is now the Primary, but the former Primary is down 
+     and may stay down. We must now fully re-create a Standby server, 
+     either on the former Primary system when it comes up, or on a third, 
+     possibly new, system. Once complete the Primary and Standby can be 
+     considered to have switched roles. Some people choose to use a third 
+     server to provide additional protection across the failover interval, 
+     though clearly this complicates the system configuration and 
+     operational processes (and this can also act as a Witness Server).
+    </para>
+ 
+    <para>
+     So, switching from Primary to Standby Server can be fast, but requires
+     some time to re-prepare the failover cluster. Regular switching from
+     Primary to Standby is encouraged, since it allows the regular downtime
+     one each system required to maintain HA. This also acts as a test of the
+     failover so that it definitely works when you really need it. Written
+     administration procedures are advised.
+    </para>
+   </sect2>
+ 
+   <sect2 id="warm-standby-record">
+    <title>Implementing Record-based Log Shipping</title>
+ 
+    <para>
+     The main features for Log Shipping in this release are based around the
+     file-based Log Shipping described above. It is also possible to
+     implement record-based Log Shipping using the pg_xlogfile_name_offset()
+     function, though this requires custom development.
+    </para>
+ 
+    <para>
+     An external program can call pg_xlogfile_name_offset() to find out the
+     filename and the exact byte offset within it of the latest WAL pointer.
+     If the external program regularly polls the server it can find out how
+     far forward the pointer has moved. It can then access the WAL file
+     directly and copy those bytes across to a less up-to-date copy on a
+     Standby Server.
+    </para>
+   </sect2>
+  </sect1>
+ 
   <sect1 id="migration">
    <title>Migration Between Releases</title>
  
---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

               http://archives.postgresql.org

Reply via email to