My reading of the proposal is that it should work. I've thought about
this edge case a lot and have not come up with a better approach. This
is really a very small window, so definitely pick anything that is
simpler (to understand/and code) over other solutions. The description
is sort of high level now, if you want more feedback before submitting
a patch once you get deeper in the code and can post more detail to
the algorithm I will respond to whatever is posted.
I was sort of expecting old/new named properties in service.properties
vs. different versions of the file. Again if different files are easier
that is fine with me.
As usual with these tiny crash point recovery cases, reproducible
testing is a pain. I am ok with using the "sane" only debug flag
method used in few other places in the store code. The obvious
test points seem to be:
1) right before commit of reencryption (everything should be backed out
and db only accessible with old key)
2) right after commit but before service.properties is updated on disk
with new encryption info.
maybe others - basically make sure to test each new path you describe in
the recovery code described below.
Another good case might be doing the whole thing twice on the same db to
make sure we don't leave stuff around from the first time around. ie.
encrypt unencrypted-> then reencrypt with a new key (with various
crashes the second time).
/mikem
Suresh Thalamati (JIRA) wrote:
[ http://issues.apache.org/jira/browse/DERBY-1156?page=comments#action_12422000 ]
Suresh Thalamati commented on DERBY-1156:
-----------------------------------------
How to handle crash/recovery when switching to new encryption key ?
After re-encrypting all the containers in the database with a new
encryption key. A final switch has to be made for the database to use
the new encryption key to encrypt the transaction log and the data.
Problem is how to perform the switch in such a way that if there is crash,
system will work with the old encryption key or the new encryption key.
Next boot of the database should not require both the keys to recovery
from a crash.
Log is encrypted, so recovery has to know the correct encryption
key to decrypt the transaction log records. Note that even the
checkpoint log record in the transaction log is encrypted.
I think an algorithm using a flag in the log control file
(CHECK_POINT_WITH_NEW_KEY) tha is written on check-point and
another flag (derby.storage.reencryptionLog) in service.properties
to track that re-encryption in progress and also to remember the first
log file that contains the log with the new encryption key.
During the switch to new-encryption key , old copy of service.properties and
the verify.key files saved , so that incase of crash, system will revert back
to the old version of these files.
Incase of a crash during re-encryption ; On recovery reading from
the log that contains log records encrypted with new key
including the commit log record of re-encryption is avoided by
deleting the log file. Because the commit log record is written to
the last log file, re-encryption transaction will be rolled back bringing
the database to state it was before re-encryption started.
Persistent Values/files :
CHECK_POINT_WITH_NEW_KEY to the log.ctrl file
derby.storage.reencryptionLog property to the service.properties.
service.properties.old. verify.key.old to keep track of old version
of the encryption properties.
Following is the description of the re-encryption of the database algorithm:
Re-encryption is performed after the recovery during the RawStore boot:
1) Perform a check point, this will make sure old transaction log
will not be read after the re-encryption (Logged with OLD KEY)
2) Begin Transaction (Logged with OLD Key).
3) Re-encrypt all containers with the new key. (this action is
logged with OLD KEY).
5) Switch to a new Log File and set the encryption engine for the log
to be the one with the new encryption key.
(After this point the transaction log will be encrypted with
the New Encryption Key).
6) update the service.properties with the property
derby.storage.reencryptionLog = (the last log file number)
(this property will help in tracking that re-encryption in progress
and also to delete the last log file in-case of crash)
7) copy service.properties after excluding derby.storage.reencryptionLog property
to service.properties.old and and verify.key into verify.key.old
8) End Transaction (logged with NEW KEY)
9) update the service.properties with new encryption key/password information.
10) update the verify.key file with the new external key incase of external
encryption key.
11) Perform a checkpoint and also mark a flag that checkpoint is done
with a new key in the log.ctrl file. CHECK_POINT_WITH_NEW_KEY = true.
(IF WE REACHED THIS FAR MEANS RE-ENCRYPTION IS SUCCESSFUL).
Now perform the cleanup:
performCleanup ()
{
12) Update the derby.storage.reencryptionLog = 0 ( re-encryption complete ,
only cleanup remaining)
13) Cleanup copies of the container files encrypted with old encryption key.
14) Update the log.control file with CHECK_POINT_WITH_NEW_KEY = false.
15) Remove the service.properties.old and verify.key.old files.
16) Remove derby.storage.reencryptionLog property from the service.properties
file.
}
If all the steps above are successful, database is ready to use the new
encryption key.
On Recovery :
1)
String renEncryptionLogProp = find(derby.storage.ReEncryptionLog );
if( reEncryptioLogProp != null ) {
reEncyrptionLog = Long.valueof(renEncryptionLog);
} else {
// This may not re-encryption recovery case or it crashed
// before the derby.storage.ReEncryptionLog was set.
// database will continue to use the old key and recovery will
// rollback if there were any changed by re-encryption. No special
// handling required.
}
if ( reEncryptionLog > 0 ) {
// renecryption was in progress
if (CHECK_POINT_WITH_NEW_KEY == true)
{
// crashed after the checkpoint with the new key ,
// this is as good as re-encryption is complete.
1) call performCleanup();
2) verify the user entered key against the new verify.key and
proceed with recovery.
}else
{
// re-encryption was not complete, rollback and
// bring the database to the state it was before
// re-encryption started.
1) delete(Blow up) the log file(derby.storage.ReEncryptionLog)
where log records with new encryption key is written if
the file exists. (so that we don't see any log with new key.
2) revert to the the old versions of service.properties and
verify.key, this will also remove the
derby.storage.ReEncryptionLog property.
3) verify the user entered old key/password using the
verify.key/or the key in the service.properties. And then
proceed with recovery.
4) recovery will undo the partially completed re-encryption,
this will bring the containers back to versions of
the old encryption key.
}
} else
{
if (reEncryptionLog == 0)
{
// crash occurred during cleanup; re-encryption was complete.
call performcleanup();
}
}
If the above algorithm works, Other minor issues:
1) instead of having service.properties.old file name the specific properties
as new or old and set them accordingly.
2) instead of storing the log file to delete incase of crash in the
as a property (derby.storage.ReEncryptionLog) in the
service.properties. Store it in the log.ctrl file; There is only one LONG
spare remaining , not sure if worth using for this case.
I would really appreciate if some one can take a look at the above algorithm
and verify if it really works or if there are any suggestions solve in
different way, that will be great too.
Thanks
-suresh
allow the encrypting of an existing unencrypted db and allow the re-encrypting
of an existing encrypted db
----------------------------------------------------------------------------------------------------------
Key: DERBY-1156
URL: http://issues.apache.org/jira/browse/DERBY-1156
Project: Derby
Issue Type: Improvement
Components: Store
Affects Versions: 10.1.2.3
Reporter: Mike Matrigali
Assigned To: Suresh Thalamati
Priority: Minor
Fix For: 10.2.0.0
Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff,
reencrypt_3.diff, reencrypt_4.diff
encrypted database to be re-encrypted with a new password.
Here are some ideas for an initial implementation.
The easiest way to do this is to make sure we have exclusive access to the
data and that no log is required in the new copy of the db. I want to avoid
the log as it also is encrypted. Here is my VERY high level plan:
1) Force exclusive access by putting all the work in the low level store,
offline boot method. We will do redo recovery as usual, but at the end
there will be an entry point to do the copy/encrypt operation.
copy/encrypt process:
0) The request to encrypt/re-encrypt the db will be handled with a new set
of url flags passed into store at boot time. The new flags will provide
the same inputs as the current encrypt flags. So at high level the
request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags".
TODO - provide exact new flag syntax.
1) Open a transaction do all logged work to do the encryption. All logging
will be done with existing encryption.
2) Copy and encrypt every db file in the database. The target files will
be in the data directory. There will be a new suffix to track the new
files, similar to the current process used for handling drop table in
a transaction consistent manner without logging the entire table to the log.
Entire encrypted destination file is guaranteed synced to disk before
transaction commits. I don't think this part needs to be logged.
Files will be read from the cache using existing mechanism and written
directly into new encrypted files (new encrypted data does not end up in
the cache).
3) Switch encrypted files for old files. Do this under a new log operation
so the process can be correctly rolled back if the encrypt db operation
transaction fails. Rollback will do file at a time switches, no reading
of encrypted data is necessary.
4) log a "change encryption of db" log record, but do not update
system.properties with the change.
5) commit transaction.
6) update system.properties and sync changes.
7) TODO - need someway to handle crash between steps 5 and 6.
6) checkpoint all data, at this point guaranteed that there is no outstanding
transaction, so after checkpoint is done there is no need for the log.
ISSUES:
o there probably should be something that catches a request to encrypt to
whatever db was already encrypted with.
