[
http://issues.apache.org/jira/browse/DERBY-1156?page=comments#action_12422000 ]
Suresh Thalamati commented on DERBY-1156:
-----------------------------------------
How to handle crash/recovery when switching to new encryption key ?
After re-encrypting all the containers in the database with a new
encryption key. A final switch has to be made for the database to use
the new encryption key to encrypt the transaction log and the data.
Problem is how to perform the switch in such a way that if there is crash,
system will work with the old encryption key or the new encryption key.
Next boot of the database should not require both the keys to recovery
from a crash.
Log is encrypted, so recovery has to know the correct encryption
key to decrypt the transaction log records. Note that even the
checkpoint log record in the transaction log is encrypted.
I think an algorithm using a flag in the log control file
(CHECK_POINT_WITH_NEW_KEY) tha is written on check-point and
another flag (derby.storage.reencryptionLog) in service.properties
to track that re-encryption in progress and also to remember the first
log file that contains the log with the new encryption key.
During the switch to new-encryption key , old copy of service.properties and
the verify.key files saved , so that incase of crash, system will revert back
to the old version of these files.
Incase of a crash during re-encryption ; On recovery reading from
the log that contains log records encrypted with new key
including the commit log record of re-encryption is avoided by
deleting the log file. Because the commit log record is written to
the last log file, re-encryption transaction will be rolled back bringing
the database to state it was before re-encryption started.
Persistent Values/files :
CHECK_POINT_WITH_NEW_KEY to the log.ctrl file
derby.storage.reencryptionLog property to the service.properties.
service.properties.old. verify.key.old to keep track of old version
of the encryption properties.
Following is the description of the re-encryption of the database algorithm:
Re-encryption is performed after the recovery during the RawStore boot:
1) Perform a check point, this will make sure old transaction log
will not be read after the re-encryption (Logged with OLD KEY)
2) Begin Transaction (Logged with OLD Key).
3) Re-encrypt all containers with the new key. (this action is
logged with OLD KEY).
5) Switch to a new Log File and set the encryption engine for the log
to be the one with the new encryption key.
(After this point the transaction log will be encrypted with
the New Encryption Key).
6) update the service.properties with the property
derby.storage.reencryptionLog = (the last log file number)
(this property will help in tracking that re-encryption in progress
and also to delete the last log file in-case of crash)
7) copy service.properties after excluding derby.storage.reencryptionLog
property
to service.properties.old and and verify.key into verify.key.old
8) End Transaction (logged with NEW KEY)
9) update the service.properties with new encryption key/password information.
10) update the verify.key file with the new external key incase of external
encryption key.
11) Perform a checkpoint and also mark a flag that checkpoint is done
with a new key in the log.ctrl file. CHECK_POINT_WITH_NEW_KEY = true.
(IF WE REACHED THIS FAR MEANS RE-ENCRYPTION IS SUCCESSFUL).
Now perform the cleanup:
performCleanup ()
{
12) Update the derby.storage.reencryptionLog = 0 ( re-encryption complete ,
only cleanup remaining)
13) Cleanup copies of the container files encrypted with old encryption key.
14) Update the log.control file with CHECK_POINT_WITH_NEW_KEY = false.
15) Remove the service.properties.old and verify.key.old files.
16) Remove derby.storage.reencryptionLog property from the service.properties
file.
}
If all the steps above are successful, database is ready to use the new
encryption key.
On Recovery :
1)
String renEncryptionLogProp = find(derby.storage.ReEncryptionLog );
if( reEncryptioLogProp != null ) {
reEncyrptionLog = Long.valueof(renEncryptionLog);
} else {
// This may not re-encryption recovery case or it crashed
// before the derby.storage.ReEncryptionLog was set.
// database will continue to use the old key and recovery will
// rollback if there were any changed by re-encryption. No special
// handling required.
}
if ( reEncryptionLog > 0 ) {
// renecryption was in progress
if (CHECK_POINT_WITH_NEW_KEY == true)
{
// crashed after the checkpoint with the new key ,
// this is as good as re-encryption is complete.
1) call performCleanup();
2) verify the user entered key against the new verify.key and
proceed with recovery.
}else
{
// re-encryption was not complete, rollback and
// bring the database to the state it was before
// re-encryption started.
1) delete(Blow up) the log file(derby.storage.ReEncryptionLog)
where log records with new encryption key is written if
the file exists. (so that we don't see any log with new key.
2) revert to the the old versions of service.properties and
verify.key, this will also remove the
derby.storage.ReEncryptionLog property.
3) verify the user entered old key/password using the
verify.key/or the key in the service.properties. And then
proceed with recovery.
4) recovery will undo the partially completed re-encryption,
this will bring the containers back to versions of
the old encryption key.
}
} else
{
if (reEncryptionLog == 0)
{
// crash occurred during cleanup; re-encryption was complete.
call performcleanup();
}
}
If the above algorithm works, Other minor issues:
1) instead of having service.properties.old file name the specific properties
as new or old and set them accordingly.
2) instead of storing the log file to delete incase of crash in the
as a property (derby.storage.ReEncryptionLog) in the
service.properties. Store it in the log.ctrl file; There is only one LONG
spare remaining , not sure if worth using for this case.
I would really appreciate if some one can take a look at the above algorithm
and verify if it really works or if there are any suggestions solve in
different way, that will be great too.
Thanks
-suresh
> allow the encrypting of an existing unencrypted db and allow the
> re-encrypting of an existing encrypted db
> ----------------------------------------------------------------------------------------------------------
>
> Key: DERBY-1156
> URL: http://issues.apache.org/jira/browse/DERBY-1156
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.1.2.3
> Reporter: Mike Matrigali
> Assigned To: Suresh Thalamati
> Priority: Minor
> Fix For: 10.2.0.0
>
> Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff,
> reencrypt_3.diff, reencrypt_4.diff
>
>
> encrypted database to be re-encrypted with a new password.
> Here are some ideas for an initial implementation.
> The easiest way to do this is to make sure we have exclusive access to the
> data and that no log is required in the new copy of the db. I want to avoid
> the log as it also is encrypted. Here is my VERY high level plan:
> 1) Force exclusive access by putting all the work in the low level store,
> offline boot method. We will do redo recovery as usual, but at the end
> there will be an entry point to do the copy/encrypt operation.
> copy/encrypt process:
> 0) The request to encrypt/re-encrypt the db will be handled with a new set
> of url flags passed into store at boot time. The new flags will provide
> the same inputs as the current encrypt flags. So at high level the
> request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags".
> TODO - provide exact new flag syntax.
> 1) Open a transaction do all logged work to do the encryption. All logging
> will be done with existing encryption.
> 2) Copy and encrypt every db file in the database. The target files will
> be in the data directory. There will be a new suffix to track the new
> files, similar to the current process used for handling drop table in
> a transaction consistent manner without logging the entire table to the
> log.
> Entire encrypted destination file is guaranteed synced to disk before
> transaction commits. I don't think this part needs to be logged.
> Files will be read from the cache using existing mechanism and written
> directly into new encrypted files (new encrypted data does not end up in
> the cache).
> 3) Switch encrypted files for old files. Do this under a new log operation
> so the process can be correctly rolled back if the encrypt db operation
> transaction fails. Rollback will do file at a time switches, no reading
> of encrypted data is necessary.
> 4) log a "change encryption of db" log record, but do not update
> system.properties with the change.
> 5) commit transaction.
> 6) update system.properties and sync changes.
> 7) TODO - need someway to handle crash between steps 5 and 6.
> 6) checkpoint all data, at this point guaranteed that there is no outstanding
> transaction, so after checkpoint is done there is no need for the log.
> ISSUES:
> o there probably should be something that catches a request to encrypt to
> whatever db was already encrypted with.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
