Hello! During the TDE discussions at pgconf.dev, I promised some benchmarks about performance characteristics regarding different WAL encryption options, to provide solid data before deciding on an approach.
I ended up benchmarking and optimizing different variants more than I initially planned, but now I have some conclusions that I think are interesting to share. I will only focus on two variants that make sense from a performance perspective, per page or per record AES-CTR-256. Other encryption modes / integrations all result in significantly decreased performance (or no visible benefit for added complexity). For example AES-GCM can be implemented in both versions, but because GCM is slower, the end result will be slower. I can also share numbers about GCM/XTS and other possibilities if people are interested, but for a "simple implementation in the core", I think it should be safe to assume that we want to go with CTR. So the two approaches are: 1. ctr_pagelevel: when we flush or read a wal page, we encrypt/decrypt it. This is simple, encrypts absolutely everything, but does everything in the critical section holding the WAL lock. 2. ctr_perrecord: encryption happens when we write a WAL record. Each record has to hold an additional IV, since at this point we can't rely on the LSN. It parallelizes encryption at the cost of increased complexity in the backend. Only the record data is encrypted, headers stay plaintext, which slightly simplifies command-line tools. I attached (one version) of both patches. All patches I used for testing are minimal: encryption is hardcoded, uses a hardcoded key baked into the source, there are no options to disable it or change settings, only the minimum required for performance testing and validation (making sure that we indeed encrypt the WAL). The naive implementation of perrecord is clearly slower than plaintext/pagelevel, however, after optimizing both versions, we can see the advantages of it: pagelevel performance degrades in scenarios with many clients and high WAL throughput, as the lock contention increases. In my tests, I focused on write performance. Read performance (recovery, replication) is a different topic, and my implementations for both could be optimized better - I didn't try to measure and improve them yet. I used 3 test scenarios: * sysbench oltp_read_write * sysbench oltp_write_only * A scenario in which each session repeats the following: 1. select 100 random IDs (locally, no SQL queries, as ids are serial and continuous) 2. execute an UPDATE SET c=c+1 WHERE id IN (id list); For testing, I used an AMD Threadripper 3970X (32 cores) and an Intel Optane P5800X for storage, which doesn't have the same throttling limitations as consumer SSDs, the tests weren't limited by IO. In the read_write scenario, there's no difference between the 2 encryption implementation and unencrypted master at all. There's some noise, but we can't notice the effect of encryption at all. In the write_only scenario, there's still no difference between master and ctr_perrecord, but ctr_pagelevel starts to fall behind at 32/64 threads. The difference disappears at 128 threads, as at that point master performance also degrades in my test setup. In the update_inlist scenario which increases WAL throughput more quickly, it already falls behind the other two at 8 threads, and remains there for all following runs: everywhere except in the single threaded case we can see a clear 20% performance drop. Similarly to the write_only scenario, there's no measurable difference between master and ctr_perrecord. This is in line with my earlier observations during less detailed benchmarking, that if we want to guarantee good WAL encryption performance, we have to go with per record encryption. The results also depend on the exact hardware used for the tests, scenario, and so on, but I think the general point of it is clear: the more work we do in the critical section, the worse the performance loss can be in certain configurations. Going with a choice that adds no additional workload there seems like a generally safer choice to me. Please share any feedback / improvement ideas for the patches, or different approaches we could test, I can continue benchmarking more scenarios / prototypes if anybody has ideas. Unless there's a different conclusion, I'll proceed with sharing a first prototype of a TDE proposal using the per record approach in the following weeks. I also attached the actual results of a comparison run as a CSV file.Hello! During the TDE discussions at pgconf.dev, I promised some benchmarks about performance characteristics regarding different WAL encryption options, to provide solid data before deciding on an approach. I ended up benchmarking and optimizing different variants a lot more than I initially planned, but now I have some conclusions that I think are interesting to share. I will only focus on two variants that make sense from a performance perspective, per page or per record AES-CTR-256. Other encryption modes / integrations all result in significantly decreased performance (or no visible benefit for added complexity). For example AES-GCM can be implemented in both versions, but because GCM is slower, the end result will be slower. I can also share numbers about GCM/XTR and other possibilities if people are interested, but for a "simple implementation in the core", I think it should be safe to assume that we want to go with CTR. So the two approaches are: 1. ctr_pagelevel: when we flush or read a wal page, we encrypt/decrypt it. This is simple, encrypts absolutely everything, but does everything in the critical section holding the WAL lock. 2. ctr_perrecord: encryption happens when we write a WAL record. Each record has to hold an additional IV, since at this point we can't rely on the LSN. Parallelizes encryption, at the cost of increasing complexity. Only the record data is encrypted, headers stay plaintext. I attached (one version) of both patches. All patches I used for testing are minimal: encryption is hardcoded, uses a hardcoded key baked into the source, there are no options to disable it or change settings, only the minimum required for performance testing and validation (making sure that we indeed encrypt the WAL). The naive implementation of perrecord is clearly slower than plaintext/pagelevel, however, after optimizing both versions, we can see the advantages of it: pagelevel performance degrades in scenarios with many clients and high WAL throughput, as the lock contention increases. In my tests, I focused on write performance. Read performance (recovery, replication) is a different topic, and my implementations for both could be optimized better - I didn't try to measure and improve them yet. I used 3 test scenarios: * sysbench oltp_read_write * sysbench oltp_write_only * A scenario in which each session repeats the following: 1. select 100 random IDs (locally, no SQL queries, as ids are serial and continuous) 2. execute an UPDATE SET c=c+1 WHERE id IN (id list); For testing, I used an AMD Threadripper 3970X (32 cores) and an Intel Optane P5800X for storage, which doesn't have the same throttling limitations as consumer SSDs, the tests weren't limited by IO. In the read_write scenario, there's no difference between the 2 encryption implementation and unencrypted master at all. There's some noise, but we can't notice the effect of encryption at all. In the write_only scenario, there's still no difference between master and ctr_perrecord, but ctr_pagelevel starts to fall behind at 32/64 threads. The difference disappears at 128 threads, as at that point master performance also degrades in my test setup. In the update_inlist scenario which increases WAL throughput more quickly, it already falls behind the other two at 8 threads, and remains there for all following runs: everywhere except in the single threaded case we can see a clear 20% performance drop. Similarly to the write_only scenario, there's no measurable difference between master and ctr_perrecord. This is in line with my earlier observations during less detailed benchmarking, that if we want to guarantee good WAL encryption performance, we have to go with per record encryption. The results also depend on the exact hardware used for the tests, scenario, and so on, but I think the general point of it is clear: the more work we do in the critical section, the worse the performance loss can be in certain configurations. Going with a choice that adds no additional workload there seems like a generally safer choice to me. Please share any feedback / improvement ideas for the patches, or different approaches we could test, I can continue benchmarking more scenarios / prototypes if anybody has ideas. Unless there's a different conclusion, I'll proceed with sharing a first prototype of a TDE proposal using the per record approach in the following weeks. I also attached the actual results of a comparison run as a CSV file.
nocfbot-0001-wal_ctr-per-record-AES-256-CTR-WAL-encrypti.patch
Description: Binary data
nocfbot-0001-wal_pagelevel-page-level-AES-256-CTR-WAL-en.patch
Description: Binary data
workload,threads,master_tps,perpage_tps,perrecord_tps,perpage_vs_master_pct,perrecord_vs_master_pct read_write,1,51.29,48.76,50.72,-4.93,-1.11 read_write,8,1952.38,2003.24,2002.50,2.61,2.57 read_write,32,16048.35,16761.09,16861.1,4.44,5.06 read_write,64,20206.66,19657.02,21187.07,-2.72,4.85 read_write,128,18430.83,18311.25,18067.96,-0.65,-1.97 write_only,1,2929.6,3075.29,2897.49,4.97,-1.1 write_only,8,24925.86,24355.15,23562.66,-2.29,-5.47 write_only,32,44286.44,38454.49,46805.01,-13.17,5.69 write_only,64,44237.47,36657.56,44193.35,-17.13,-0.1 write_only,128,30799.56,29392.12,29425.72,-4.57,-4.46 inlist_update,1,3576.14,3512.72,3403.79,-1.77,-4.82 inlist_update,8,21697.73,16831.56,21628.58,-22.43,-0.32 inlist_update,32,26272.75,20594.12,25870.31,-21.61,-1.53 inlist_update,64,26514.75,20317.26,25895.03,-23.37,-2.34 inlist_update,128,25550.13,20395.19,25329.01,-20.18,-0.87
