Re: TLS/SSL overhead

[onmstester onmstester]

Thank you,



I should have mention hardware and software which I used in this experiment:

CPU: one Intel Xeon silver 4210 10 core 2.2G

Network: 1Gb

OS: Ubuntu 20.04.2 LTS

Java: 1.8.0_321 Oracle

Apache Cassandra 4.0.1


Data model is a single table:



 text partitionKey,   15chars

 int clusterKey,        8 digits

 text simpleColumn  1200 chars

key: (partitionkey, clusterKey)




Generated keys and Cassandra ssl config is the same as dzone article: 
https://dzone.com/articles/setting-up-a-cassandra-cluster-with-ssl#

server_encryption_options:


3


internode_encryption: all


4


keystore: /opt/cassandra/conf/certs/cassandra.keystore


5


keystore_password: cassandra


6


truststore: /opt/cassandra/conf/certs/cassandra.truststore


7


truststore_password: cassandra


8


# More advanced defaults below:


9


protocol: TLS


10


​


11


client_encryption_options:


12


enabled: true


13


# If enabled and optional is set to true encrypted and unencrypted 
connections are handled.


14


optional: false


15


keystore: /opt/cassandra/conf/certs/cassandra.keystore


16


keystore_password: cassandra


17


truststore: /opt/cassandra/conf/certs/cassandra.truststore


18


truststore_password: cassandra


19


require_client_auth: true


20


protocol: TLS




Cassandra Configs other than default:

Max Heap: 31GB

G1 gc almost tuned for write throughput (90%):

Separate physical disk drive for commitlog and data

commitlog compression (lz4) + sstable compression (flush lz4 + compaction: zstd)

internode_compression: all




Client side: datastax-oss 4.13 with client protocol encryption, 10 threads/1000 
async insert



And the benchmark result for single node cluster, which is the only scenario 
that i could validate with multiple repeats:



Scenario


Write/sec


Node CPU usage (other resources < 10% utilized)


No_SSL


115K


90%



Client_SSL


112K


90%







So the overhead was 2.5% for client SSL on single node cluster with default SSL 
configs.




Honestly, I'm not very satisfied with accuracy of my benchmarks because I could 
not use all CPU resources on multi node cluster with RF > 1 and throughput was 
almost the same for both SSL and non-SSL configurations on those scenarios (I 
asked community for help on that matter but still no luck). 



Eric, for the sake of making it a blog post, its not a comprehensive, accurate 
experiment to rely on as i explained, but anyway the information i provided 
above is all i got so far. If you need more information or have suggestions on 
improving these experiments, please let me know.



Daemeon, output packets from client (which use lz4 compresion) are about 400 
bytes and from netstat -s/tcp part: while 16M segments sent/1000 of them 
retransmitted


Best Regards



Sent using https://www.zoho.com/mail/






 On Mon, 07 Feb 2022 06:50:16 +0330 daemeon reiydelle  
wrote 



the % numbers seen high for a clean network and a reasonable fast client. The 
5% really not reasonable. No jumbo frames? No network retries (netstats)?







Daemeon Reiydelle

email: mailto:daeme...@gmail.com

San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle



"Why is it so hard to rhyme either Life or Love?" - Sondheim









On Sun, Feb 6, 2022 at 6:06 PM Dinesh Joshi  wrote:

I wish there was an easy answer to this question. Like you pointed out it is 
hardware dependent but software stack plays a big part. For instance, the JVM 
you're running makes a difference too. Cassandra comes with netty and IIRC we 
include tcnative which accelerates TLS. You could also slip Amazon's Corretto 
Crypto Provider into your runtime. I am not suggesting using everything all at 
once but a combination of libraries, runtimes, JVM, OS, cipher suites can make 
a big difference. Therefore it is best to try it out on your stack.



Typically modern hardware has accelerators for common encryption algorithms. If 
the software stack enables you to optimally take advantage of the hardware then 
you could see very little to no impact on latencies.



Cassandra maintains persistent connections therefore the visible impact is on 
connection establishment time (TLS handshake is expensive). Encryption will 
make thundering herd problems worse. You should watch out for those two issues.



Dinesh





On Feb 5, 2022, at 3:53 AM, onmstester onmstester  
wrote:



Hi, 



Anyone measured impact of wire encryption using TLS 
(client_encryption/server_encryption) on cluster latency/throughput? 

It may be dependent on Hardware or even data model but I already did some sort 
of measurements and got to 2% for client encryption and 3-5% for client + 
server encryption and wanted to validate that with community.



Best Regards



Sent using https://www.zoho.com/mail/

Re: TLS/SSL overhead

[daemeon reiydelle]

the % numbers seen high for a clean network and a reasonable fast client.
The 5% really not reasonable. No jumbo frames? No network retries
(netstats)?



*Daemeon Reiydelle*

*email: daeme...@gmail.com *
*San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle*

*"Why is it so hard to rhyme either Life or Love?" - Sondheim*


On Sun, Feb 6, 2022 at 6:06 PM Dinesh Joshi  wrote:

> I wish there was an easy answer to this question. Like you pointed out it
> is hardware dependent but software stack plays a big part. For instance,
> the JVM you're running makes a difference too. Cassandra comes with netty
> and IIRC we include tcnative which accelerates TLS. You could also slip
> Amazon's Corretto Crypto Provider into your runtime. I am not suggesting
> using everything all at once but a combination of libraries, runtimes, JVM,
> OS, cipher suites can make a big difference. Therefore it is best to try it
> out on your stack.
>
> Typically modern hardware has accelerators for common encryption
> algorithms. If the software stack enables you to optimally take advantage
> of the hardware then you could see very little to no impact on latencies.
>
> Cassandra maintains persistent connections therefore the visible impact is
> on connection establishment time (TLS handshake is expensive). Encryption
> will make thundering herd problems worse. You should watch out for those
> two issues.
>
> Dinesh
>
>
> On Feb 5, 2022, at 3:53 AM, onmstester onmstester 
> wrote:
>
> Hi,
>
> Anyone measured impact of wire encryption using TLS
> (client_encryption/server_encryption) on cluster latency/throughput?
> It may be dependent on Hardware or even data model but I already did some
> sort of measurements and got to 2% for client encryption and 3-5% for
> client + server encryption and wanted to validate that with community.
>
> Best Regards
>
> Sent using Zoho Mail 
>
>
>
>

Re: TLS/SSL overhead

[Dinesh Joshi]

I wish there was an easy answer to this question. Like you pointed out it is 
hardware dependent but software stack plays a big part. For instance, the JVM 
you're running makes a difference too. Cassandra comes with netty and IIRC we 
include tcnative which accelerates TLS. You could also slip Amazon's Corretto 
Crypto Provider into your runtime. I am not suggesting using everything all at 
once but a combination of libraries, runtimes, JVM, OS, cipher suites can make 
a big difference. Therefore it is best to try it out on your stack.

Typically modern hardware has accelerators for common encryption algorithms. If 
the software stack enables you to optimally take advantage of the hardware then 
you could see very little to no impact on latencies.

Cassandra maintains persistent connections therefore the visible impact is on 
connection establishment time (TLS handshake is expensive). Encryption will 
make thundering herd problems worse. You should watch out for those two issues.

Dinesh


> On Feb 5, 2022, at 3:53 AM, onmstester onmstester  wrote:
> 
> Hi, 
> 
> Anyone measured impact of wire encryption using TLS 
> (client_encryption/server_encryption) on cluster latency/throughput? 
> It may be dependent on Hardware or even data model but I already did some 
> sort of measurements and got to 2% for client encryption and 3-5% for client 
> + server encryption and wanted to validate that with community.
> 
> Best Regards
> 
> Sent using Zoho Mail 
> 
> 

Re: TLS/SSL overhead

[Erick Ramirez]

The 3-5% penalty range is consistent with what other users have reported
over the years but I'm sorry that I can't seem to find the
threads/references so my response is unfortunately anecdotal.

More importantly, would you be interested in sharing your data? It would be
great to feature it as a blog post and I'm sure a lot of users are going to
be very interested. It doesn't have to be a polished write up and we've got
other contributors who'd be happy to help with the draft if that's a
concern. Cheers!