Re: [jira] Commented: (DERBY-239) Need a online backup feature that does not block update operations when online backup is in progress.

Wed, 05 Oct 2005 20:08:19 -0700
Attached is the functional/design spec based on the input I got from everyone on this topic. I think, most of the issues are covered in this doc, except the backup of jars stored inside the database. 


Thanks
-suresh


Title: Online Backup 












Online Backup Functional/Design Spec



 



Overview 



 



Support real-time online backup mechanism that does not block database
operations when backup
is in progress.
When backup is running, currently Derby
only permits reads from the database. By adding support to permit writes, it
will be easier for the users to schedule backups without being concerned about
the impact of backup operation on the applications that are accessing the
database.



 



Introduction



 



New online backup mechanism will not change the way database
can be restored from the backup. In the current system, if there any non-logged
operations when backup starts, they will be rolled back on restore because database
is frozen at the start of the backup. To support real-time online backups that
do not block user operations, it is necessary to have the transaction log for all
the operations that happen after the backup starts to bring the database to the
consistent state during restore.  Because
a transaction that has non-logged operations can commit when backup is in
progress. To make a consistent backup of the database, there should not be any
pending transactions with non-logged operations. 



 



JDBC



 



Backup operations are support through system procedures. They
can be called using JDBC program or through IJ. 



Existing Backup Procedures:



After this enhancement, existing backup procedures (SYSCS_UTIL.SYSCS_BACKUP_DATABASE
 ...etc) will allow writes to the
database. 



Information about existing online backup procedures can be
found in the Derby admin guide. Derby Admin Guide . These
backup procedures will WAIT for the transaction
with non-logged operations to commit before proceeding with the backup.   



New Procedures:



Two new procedures are added to make backup wait/return error,
if any non-logged operations are pending at the start of the backup.



 



Full
backup: 







SYSCS_UTIL.SYSCS_BACKUP_DATABASE(IN
BACKUPDIR VARCHAR(), 



                                  IN WAIT_FOR_NONLOGGED_OPERATIONS
SMALLINT) 



 

Full backup with log
archive mode: 



SYSCS_UTIL.SYSCS_BACKUP_DATABASE_AND_ENABLE_LOG_ARCHIVE_MODE


     (IN BACKUPDIR VARCHAR(32672),




      IN SMALLINT DELETE_ARCHIVED_LOG_FILES,


      IN
WAIT_FOR_NONLOGGED_OPERATIONS SMALLINT)) 





Arguments



 



BACKUPDIR:



An input argument of type VARCHAR(32672)
that specifies the full system path to the database directory to be backed up.



 



DELETE_ARCHIVED_LOG_FILES:



A non-zero input argument of type SMALLINT will make backup
procedures to delete existing online archived log files that were created
before this backup will be deleted. The log files are deleted only after a
successful backup.



 



WAIT_FOR_NONLOGGED_OPERATIONS : 

A non-zero input argument of type SMALLINT will make backup procedures to wait
for the non-logged operation that are currently running to commit, while an
argument of 0 will make backup fail if there are any  uncommitted non-logged operations.



 



Design



 



Current Online backup copies all the database files (log, seg0  ...Etc) to the specified location when the database
is frozen (writes are blocked). 



In the new scheme, goal is to copy the database information to
the backup location, without blocking any user operation for the whole duration
of the backup.  Following sections discuss
how the information is copied into the backup. 



 



Containers:



 



At the start of the backup, all the containers that need to
be backed up are found by finding all the files in the data segment (seg0). Then
each container is copied to the backup by reading through the page cache
instead of the direct file copy. 



 



·       
Individual pages are latched, when they are being
copied to the backup to prevent modifications to the page. 



·       
Any new containers created when backup is in
progress will be recreated on restore using the information from the transaction
log. 



·       
Containers that were in the process of creation
will not be backed up, if the creation is not complete. Container cache will
make sure that backup can not get handle to  the containers that are being created.  



·       
If committed drop occurs on a container that is
being backed up, backup will wait for the committed drop to complete and then copy
the stub to the backup. 



·       
Committed container stubs that are created after
the checkpoint for the backup will not be deleted  until the backup is complete. A
checkpoint that occurs after the backup garbage collects them. 



·       
 All
requests to open containers in non-logged mode will be automatically converted
to logged mode after the backup starts. 



   



Log files:



Transaction log is used to bring the database to the
consistent state on restore from the backup. A checkpoint is made at the start
of the backup; this checkpoint will be starting point for the recovery from the
backup. Log files are copied to the backup using regular file copy.  



 



    
 
  • Backup
     Checkpoint information in the log control files is copied to the backup
     location at the start. 
    • 
 
  • All
     the transaction log files starting from the backup checkpoint to the log file
     that is active at the end of the backup are copied into the backup. 
    • 
 
  • State
     of the database in the backup depends on last log file that goes into the back
     up.  After all the containers are
     copied to the backup , last log file that goes
     into the backup is determined.  
    • 
 
  • Database
     engine wills not delete the log files that are to yet to be backed up,
     when backup is running even if they are no longer required for regular
     crash recovery.  
    • 
 
  • Checkpoints
     can occur in parallel when the backup is progress.   
    • 




 



Jar files:



 



Storing Jar files in the database is not a logged operations
and they don’t go through the page cache; they will be copied to the backup
directly as it is done now.  I am not
sure at the moment, if that will lead to any issues.  Any input about jar files will be helpful. 



 



Other files:



    
 
  • Database
     bootstrap file Service.Properties will be to be copied to the backup.  Need to check if any synchronization is
     required to copy this file.
    • 
 
  • Copy
     the BACKUP.HISTORY file at the end of the backup.
    • 
 
  •  Lock files should not be copied to the
     backup. 
    • 
 
  • Tmp directory
     should not be backed up. 
    • 




 



 



Testing:



Most of the current backup tests are single threaded. I will
add new functional tests case that will execute read/writes on the database,
when backup is running.  Some of the
testing scenarios:



1.     
Test the new procedure calls.           



2.     
Checkpoint in parallel to the backup.



3.     
DML operations in parallel to the backup. 



4.     
Grow the container after it is backed up / parallel.



5.     
Non-logged operations in parallel to the backup and
pending when backup starts.  



6.     
DDL operations in parallel to the backup.



7.     
In-Place compress in parallel to the backup.



8.     
….etc.



 



 



  References:



 



·        
Derby Admin Guide Backup
Section



·        
Derby-Dev
Thread



 



 



 



·       
Version 1. 10/5/2005 :  [EMAIL PROTECTED]



 



 



 






























					Reply via email to