Hello
Thanks a lot for the patch, I think in terms of functionality, the patch
provides very straightforward functionalities regarding key management. In
terms of documentation, I think the patch is still lacking some pieces of
information that kind of prevent people from fully understanding how KMS works
and how it can be used and why, (at least that is the impression I got from the
zoom meeting recordings :p). I spent some time today revisiting the
key-management documentation in the patch and rephrase and restructure it
based on my current understanding of latest KMS design. I mentioned all 3
application level keys that we have agreed and emphasize on explaining the SQL
level encryption key because that is the key that can be used right now. Block
and WAL levels keys we can add here more information once they are actually
used in the TDE development.
Please see below the KMS documentation that I have revised and I hope it will
be more clear and easier for people to understand KMS. Feel free to make
adjustments. Please note that we use the term "wrap" and "unwrap" a lot in our
past discussions. Originally we used the terms within a context involving Key
encryption keys (KEK). For example, "KMS wraps a master key with KEK". Later,
we used the same term in a context involving encrypting user secret /password.
For example, "KMS wraps a user secret with SQL key". In my opinion, both make
sense but it may be confusing to people having the same term used differently.
So in my revision below, the terms "wrap" and "unwrap" refer to encrypting or
decrypting user secret / password as they are used in "pg_wrap() and
pg_unwrap()". I use the terms "encapsulate" and "restore" when KEK is used to
encrypt or decrypt a key.
Chapter 32: Encryption Key Management
----------------------------------------------
PostgreSQL supports internal Encryption Key Management System, which is
designed to manage the life cycles of cryptographic keys within the PostgreSQL
system. This includes dealing with their generation, storage, usage and
rotation.
Encryption Key Management is enabled when PostgreSQL is build with
--with-openssl and cluster passphrase command is specified during initdb. The
cluster passphrase provided by --cluster-passphrase-command option during
initdb and the one generated by cluster_passphrase_command in the
postgresql.conf must match, otherwise, the database cluster will not start up.
32.1 Key Generations and Derivations
------------------------------------------
When cluster_passphrase_command option is specified to the initdb, the process
will derive the cluster passphrase into a Key Encryption Key (KEK) and a HMAC
Key using key derivation protocol before the actual generation of application
level cryptographic level keys.
-Key Encryption Key (KEK)
KEK is primarily used to encapsulate or restore a given application level
cryptographic key
-HMAC Key
HMAC key is used to compute the HASH of a given application level cryptographic
key for integrity check purposes
These 2 keys are not stored physically within the PostgreSQL cluster as they
are designed to be derived from the correctly configured cluster passphrase.
Encryption Key Management System currently manages 3 application level
cryptographic keys that have different purposes and usages within the
PostgreSQL system and these are generated using pg_strong_random() after KEK
and HMAC key derivation during initdb process.
The 3 keys are:
-SQL Level Key
SQL Level Key is used to wrap and unwrap a user secret / passphrase via
pg_wrap() and pg_unwrap() SQL functions. These 2 functions are designed to be
used in conjunction with the cryptographic functions provided by pgcrypto
extension to perform column level encryption/decryption without having to
supply a clear text user secret or passphrase that is required by many pgcrypto
functions as input. Please refer to [Wrap and Unwrap User Secret section] for
usage examples.
-Block Level Key
Block Level Key is primarily used to encrypt / decrypt buffers as part of the
Transparent Data Encryption (TDE) feature
-WAL Level Key
WAL Level Key is primarily used to encrypt / decrypt WAL files as part of the
Transparent Data Encryption (TDE) feature
The 3 application level keys above will be encapsulated and hashed using KEK
and HMAC key mentioned above before they are physically stored to pg_cryptokeys
directory within the cluster.
32.1. Key Initialization
-------------------------
When a PostgreSQL cluster with encryption key management enabled is started,
the cluster_passphrase_command parameter in postgresql.conf will be evaluated
and the cluster passphrase will be derived into KEK and HMAC Key in similar
ways as initdb.
After that, the 3 encapsulated application level cryptographic keys will be
retrieved from pg_cryptokeys directory to be restored and integrity-checked by
the key management system using the derived KEK and HMAC key. If this process
fails, it is likely that the cluster passphrase supplied to the cluster is not
the same as that supplied to the initdb process. The cluster will refuse to
start in this case and user has to manually correct the cluster passphrase.
32.2. Wrap and Unwrap User Secret
----------------------------------------
Encryption key management system provides pg_wrap() and pg_unwrap SQL functions
(listed in Table 9.97) to perform wrap and unwrap operations on user secret
with the SQL level encryption key. The SQL level encryption key is one of the 3
application level keys generated during initdb process when cluster_passphrase
is supplied.
When pg_wrap() and pg_unwrap() functions are invoked, SQL level encryption key
will internally be used to perform the encryption and decryption operation with
HMAC-based integrity check. From user's point of view, he or she is not aware
of the actual SQL level encryption key used internally by both wrap functions
One possible use case is to combine pg_wrap() and pg_unwrap() with pgcrypto.
User wraps the user encryption secret with pg_wrap function and passes the
wrapped encryption secret to pg_unwrap function for the pgcrypto encryption
functions. The wrapped secret can be stored in the application server or
somewhere secured and should be obtained promptly for cryptographic operation
with pgcrypto.
Here is an example that shows how to encrypt and decrypt data together with
wrap and unwrap functions:
=# SELECT pg_wrap('my secret passward');
pg_wrap
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
\xb2c89f76f04f95d029f179e0fc3df4ed7254127b5562a9e27d42d1cd037c942dea65ce7c0750c520fa4f4e90481c9eb7e1e42a068248c262c1a6f25c6eab64303b1154ccc9a14361223641aab4a7aabe
(1 row)
Once wrapping the user key, user can encrypt and decrypt user data using the
wrapped user key together with the key unwrap functions:
=# INSERT INTO tbl
VALUES (pgp_sym_encrypt('secret data',
pg_unwrap('\xb2c89f76f04f95d029f179e0fc3df4ed7254127b5562a9e27d42d1cd037c942dea65ce7c0750c520fa4f4e90481c9eb7e1e42a068248c262c1a6f25c6eab64303b1154ccc9a14361223641aab4a7aabe')));
INSERT 1
=# SELECT * FROM tbl;
col
--------------------------------------------------------------------------------------------------------------------------------------------------------------
\xc30d04070302a199ee38bea0320b75d23c01577bb3ffb315d67eecbeca3e40e869cea65efbf0b470f805549af905f94d94c447fbfb8113f585fc86b30c0bd784b10c9857322dc00d556aa8de14
(1 row)
=# SELECT pgp_sym_decrypt(col,
pg_unwrap('\xb2c89f76f04f95d029f179e0fc3df4ed7254127b5562a9e27d42d1cd037c942dea65ce7c0750c520fa4f4e90481c9eb7e1e42a068248c262c1a6f25c6eab64303b1154ccc9a14361223641aab4a7aabe'))
as col
FROM tbl;
col
--------------
secret data
(1 row)
The data 'secret data' is practically encrypted by the user secret 'my secret
passward' but using wrap and unwrap functions user don't need to know the
actual user secret during operation.
32.3. Key Rotation Process
------------------------------
Encryption keys in general are not interminable, the longer the same key is in
use, the chance of it being breached increases. Performing key rotation on a
regular basis help meet standardized security practices such as PCI-DSS and it
is a good practice in security to limit the number of encrypted bytes available
for a specific key version. The key lifetimse are based on key length, key
strength, algorithm and total number of bytes enciphered. The key management
systems provides a efficient method to perform key rotation.
Please be aware that the phrase "key rotation" here only refers to the rotation
of KEK and HMAC keys. The 3 application level encryption keys (SQL, Block and
WAL levels) are not rotated; they will in fact be the same before and after a
"key rotation." This can be justified because the actual keys are never stored
anywhere physically, presented to user or captured in logging. What is being
rotated here is the KEK and HMAC keys who are responsible for encapsulating and
restoring the actual application level encryption keys.
Since both KEK and HMAC keys are derived from a cluster passphrase, the "key
rotation" ultimately refers to the rotation of cluster passphrase and deriving
a new KEK and HMAC keys from the new cluster passphrase. The new set of KEK and
HMAC keys can then be used to encapsulate all 3 application level encryptions
keys and store the new results in pg_cryptokeys directory.
To rotate the cluster passphrase, user firstly needs to update
cluster_passphrase_command in the postgresql.conf and then execute
pg_rotate_cluster_passphrase() SQL function to initiate the rotation.
Cary Huang
-------------
HighGo Software Inc. (Canada)
mailto:[email protected]
http://www.highgo.ca
---- On Mon, 30 Mar 2020 21:30:19 -0700 Masahiko Sawada
<mailto:[email protected]> wrote ----
On Tue, 31 Mar 2020 at 09:36, Cary Huang <mailto:[email protected]> wrote:
>
> Hi
> I had a look on kms_v9 patch and have some comments
>
> --> pg_upgrade.c
> keys are copied correctly, but as pg_upgrade progresses further, it will try
> to start the new_cluster from "issue_warnings_and_set_wal_level()" function,
> which is called after key copy. The new cluster will fail to start due to the
> mismatch between cluster_passphrase_command and the newly copied keys. This
> causes pg_upgrade to always finish with failure. We could move
> "copy_master_encryption_key()" to be called after
> "issue_warnings_and_set_wal_level()" and this will make pg_upgrade to finish
> with success, but user will still have to manually correct the
> "cluster_passphrase_command" param on the new cluster in order for it to
> start up correctly. Should pg_upgrade also take care of copying
> "cluster_passphrase_command" param from old to new cluster after it has
> copied the encryption keys so users don't have to do this step? If the
> expectation is for users to manually correct "cluster_passphrase_command"
> param after successful pg_upgrade and key copy, then there should be a
> message to remind the users to do so.
I think both the old cluster and the new cluster must be initialized
with the same passphrase at initdb. Specifying the different
passphrase command to the new cluster at initdb and changing it after
pg_upgrade doesn't make sense. Also I don't think we need to copy
cluster_passphrase_command same as other GUC parameters.
I've changed the patch so that pg_upgrade copies the crypto keys only
if both new and old cluster enable the key management. User must
specify the same passphrase command to both old and new cluster, which
is not cumbersome, I think. I also added the description about this to
the doc.
>
> -->Kmgr.c
> + /*
> + * If there is only temporary directory, it means that the previous
> + * rotation failed after wrapping the all internal keys by the new
> + * passphrase. Therefore we use the new cluster passphrase.
> + */
> + if (stat(KMGR_DIR, &st) != 0)
> + {
> + ereport(DEBUG1,
> + (errmsg("both directories %s and %s exist, use the newly wrapped keys",
> + KMGR_DIR, KMGR_TMP_DIR)));
>
> I think the error message should say "there is only temporary directory
> exist" instead of "both directories exist"
You're right. Fixed.
I've attached the new version patch.
Regards,
--
Masahiko Sawada http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
