Hello Folks,

I would like to publish a proposal regarding improvements to Ceph
data-at-rest encryption mechanism. Adam Kupczyk and I worked
on that in last weeks.

Initially we considered several architectural approaches and made
several iterations of discussions with Intel storage group. The proposal
is condensed description of the solution we see as the most promising

We are open to any comments and questions.

Adam Kupczyk
Radoslaw Zarzynski


Data at-rest encryption is mechanism for protecting data center
operator from revealing content of physical carriers.

Ceph already implements a form of at rest encryption. It is performed
through dm-crypt as intermediary layer between OSD and its physical
storage. The proposed at rest encryption mechanism will be orthogonal
and, in some ways, superior to already existing solution.


* Radoslaw Zarzynski (Mirantis)
* Adam Kupczyk (Mirantis)

Interested Parties

If you are interested in contributing to this blueprint, or want to be
a "speaker" during the Summit session, list your name here.

Name (Affiliation)
Name (Affiliation)

Current Status

Current data at rest encryption is achieved through dm-crypt placed
under OSD’s filestore. This solution is a generic one and cannot
leverage Ceph-specific characteristics. The best example is that
encryption is done multiple times - one time for each replica. Another
issue is lack of granularity - either OSD encrypts nothing, or OSD
encrypts everything (with dm-crypt on).

Cryptographic keys are stored on filesystem of storage node that hosts
OSDs. Changing them require redeploying the OSDs.

The best way to address those issues seems to be introducing
encryption into Ceph OSD.

Detailed Description

In addition to the currently available solution, Ceph OSD would
accommodate encryption component placed in the replication mechanisms.

Data incoming from Ceph clients would be encrypted by primary OSD. It
would replicate ciphertext to non-primary members of an acting set.
Data sent to Ceph client would be decrypted by OSD handling read
operation. This allows to:
* perform only one encryption per write,
* achieve per-pool key granulation for both key and encryption itself.

Unfortunately, having always and everywhere the same key for a given
pool is unacceptable - it would make cluster migration and key change
extremely burdensome process. To address those issues crypto key
versioning would be introduced. All RADOS objects inside single
placement group stored on a given OSD would use the same crypto key
version. The same PG on other replica may use different version of the
same, per pool-granulated key.

In typical case ciphertext data transferred from OSD to OSD can be
used without change. This is when both OSDs have the same crypto key
version for given placement group. In rare cases when crypto keys are
different (key change or transition period) receiving OSD will recrypt
with local key versions.

For compression to be effective it must be done before encryption. Due
to that encryption may be applied differently for replication pools
and EC pools. Replicated pools do not implement compression; for those
pools encryption is applied right after data enters OSD. For EC pools
encryption is applied after compressing. When compression will be
implemented for replicated pools, it must be placed before encryption.

Ceph currently has thin abstraction layer over block ciphers
(CryptoHandler, CryptoKeyHandler). We want to extend this API to
introduce initialization vectors, chaining modes and asynchronous
operations. Implementation of this API may be based on AF_ALG kernel
interface. This assures the ability to use hardware accelerations
already implemented in Linux kernel. Moreover, due to working on
bigger chunks (dm-crypt operates on 512 byte long sectors) the raw
encryption performance may be even higher.

The encryption process must not impede random reads and random writes
to RADOS objects. Solution for this is to create encryption/decryption
process that will be applicable for arbitrary data range. This can be
done most easily by applying chaining mode that doesn’t impose
dependencies between subsequent data chunks. Good candidates are
CTR[1] and XTS[2].

Encryption-related metadata would be stored in extended attributes.

In order to coordinate encryption across acting set, all replicas will
share information about crypto key versions they use. Real
cryptographic keys never be stored permanently by Ceph OSD. Instead,
it would be gathered from monitors. Key management improvements will
be addressed in separate task based on dedicated proposal [3].




Work items

Coding tasks
* Extended Crypto API (CryptoHandler, CryptoKeyHandler).
* Encryption for replicated pools.
* Encryption for EC pools.
* Key management.

Build / release tasks
* Unit tests for extended Crypto API.
* Functional tests for encrypted replicated pools.
* Functional tests for encrypted EC pools.

Documentation tasks
* Document extended Crypto API.
* Document migration procedures.
* Document crypto key creation and versioning.
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Reply via email to