Re: TLS/SSL overhead

[daemeon reiydelle]

the % numbers seen high for a clean network and a reasonable fast client.
The 5% really not reasonable. No jumbo frames? No network retries
(netstats)?



*Daemeon Reiydelle*

*email: daeme...@gmail.com *
*San Francisco 1.415.501.0198/Skype daemeon.c.m.reiydelle*

*"Why is it so hard to rhyme either Life or Love?" - Sondheim*


On Sun, Feb 6, 2022 at 6:06 PM Dinesh Joshi  wrote:

> I wish there was an easy answer to this question. Like you pointed out it
> is hardware dependent but software stack plays a big part. For instance,
> the JVM you're running makes a difference too. Cassandra comes with netty
> and IIRC we include tcnative which accelerates TLS. You could also slip
> Amazon's Corretto Crypto Provider into your runtime. I am not suggesting
> using everything all at once but a combination of libraries, runtimes, JVM,
> OS, cipher suites can make a big difference. Therefore it is best to try it
> out on your stack.
>
> Typically modern hardware has accelerators for common encryption
> algorithms. If the software stack enables you to optimally take advantage
> of the hardware then you could see very little to no impact on latencies.
>
> Cassandra maintains persistent connections therefore the visible impact is
> on connection establishment time (TLS handshake is expensive). Encryption
> will make thundering herd problems worse. You should watch out for those
> two issues.
>
> Dinesh
>
>
> On Feb 5, 2022, at 3:53 AM, onmstester onmstester 
> wrote:
>
> Hi,
>
> Anyone measured impact of wire encryption using TLS
> (client_encryption/server_encryption) on cluster latency/throughput?
> It may be dependent on Hardware or even data model but I already did some
> sort of measurements and got to 2% for client encryption and 3-5% for
> client + server encryption and wanted to validate that with community.
>
> Best Regards
>
> Sent using Zoho Mail 
>
>
>
>

Re: TLS/SSL overhead

[Dinesh Joshi]

I wish there was an easy answer to this question. Like you pointed out it is 
hardware dependent but software stack plays a big part. For instance, the JVM 
you're running makes a difference too. Cassandra comes with netty and IIRC we 
include tcnative which accelerates TLS. You could also slip Amazon's Corretto 
Crypto Provider into your runtime. I am not suggesting using everything all at 
once but a combination of libraries, runtimes, JVM, OS, cipher suites can make 
a big difference. Therefore it is best to try it out on your stack.

Typically modern hardware has accelerators for common encryption algorithms. If 
the software stack enables you to optimally take advantage of the hardware then 
you could see very little to no impact on latencies.

Cassandra maintains persistent connections therefore the visible impact is on 
connection establishment time (TLS handshake is expensive). Encryption will 
make thundering herd problems worse. You should watch out for those two issues.

Dinesh


> On Feb 5, 2022, at 3:53 AM, onmstester onmstester  wrote:
> 
> Hi, 
> 
> Anyone measured impact of wire encryption using TLS 
> (client_encryption/server_encryption) on cluster latency/throughput? 
> It may be dependent on Hardware or even data model but I already did some 
> sort of measurements and got to 2% for client encryption and 3-5% for client 
> + server encryption and wanted to validate that with community.
> 
> Best Regards
> 
> Sent using Zoho Mail 
> 
> 

Re: Running enablefullquerylog crashes cassandra

[Jeff Jirsa]

That looks like a nodetool stack - can you check the Cassandra log for 
corresponding error? 

> On Feb 6, 2022, at 12:52 AM, Gil Ganz  wrote:
> 
> 
> Hey
> I'm trying to enable full query log on cassandra 4.01 node and it's causing 
> cassandra to shutdown
> 
> nodetool enablefullquerylog --path /mnt/fql_data
> 
> Cassandra has shutdown.
> error: null
> -- StackTrace --
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:267)
> at 
> sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222)
> at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
> at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
> at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown 
> Source)
> at 
> javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
> at 
> javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
> at com.sun.proxy.$Proxy6.enableFullQueryLogger(Unknown Source)
> at 
> org.apache.cassandra.tools.NodeProbe.enableFullQueryLogger(NodeProbe.java:1836)
> at 
> org.apache.cassandra.tools.nodetool.EnableFullQueryLog.execute(EnableFullQueryLog.java:62)
> at 
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:358)
> at 
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:343)
> at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:246)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:84)
> 
> /mnt/fql_data is owned by cassandra user
> Doesn't matter if directory is empty or not
> 
> contents of cassandra.yaml 
> 
> full_query_logging_options:
>  log_dir: /mnt/fql_data
>  roll_cycle: HOURLY
>  block: true
>  max_queue_weight: 268435456
>  max_log_size: 34359738368
> # archive command is "/path/to/script.sh %path" where %path is replaced 
> with the file being rolled:
> # archive_command:
> # max_archive_retries: 10
> 
> No errors in the log, last couple of lines are 
> 
> INFO  [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334 
> BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll 
> cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log 
> size:34359738368 Archive command:
> INFO  [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335 
> BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested
> 
> I noticed there is a similiar bug 
> https://issues.apache.org/jira/browse/CASSANDRA-17136  but I also tried 
> setting disk_failure_policy to ignore, same thing.
> Has Anyone encountered something similar? 
> 
> 
> 
> 
> 
> gil

Running enablefullquerylog crashes cassandra

[Gil Ganz]

Hey
I'm trying to enable full query log on cassandra 4.01 node and it's causing
cassandra to shutdown

nodetool enablefullquerylog --path /mnt/fql_data

Cassandra has shutdown.
error: null
-- StackTrace --
java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at
sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:222)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at
javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
at
javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
at
javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
at com.sun.proxy.$Proxy6.enableFullQueryLogger(Unknown Source)
at
org.apache.cassandra.tools.NodeProbe.enableFullQueryLogger(NodeProbe.java:1836)
at
org.apache.cassandra.tools.nodetool.EnableFullQueryLog.execute(EnableFullQueryLog.java:62)
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.runInternal(NodeTool.java:358)
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:343)
at org.apache.cassandra.tools.NodeTool.execute(NodeTool.java:246)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:84)

/mnt/fql_data is owned by cassandra user
Doesn't matter if directory is empty or not

contents of cassandra.yaml

full_query_logging_options:
 log_dir: /mnt/fql_data
 roll_cycle: HOURLY
 block: true
 max_queue_weight: 268435456
 max_log_size: 34359738368
# archive command is "/path/to/script.sh %path" where %path is replaced
with the file being rolled:
# archive_command:
# max_archive_retries: 10

No errors in the log, last couple of lines are

INFO  [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,334
BinLog.java:420 - Attempting to configure bin log: Path: /mnt/fql_data Roll
cycle: HOURLY Blocking: true Max queue weight: 268435456 Max log
size:34359738368 Archive command:
INFO  [RMI TCP Connection(10)-127.0.0.1] 2022-02-06 08:41:50,335
BinLog.java:433 - Cleaning directory: /mnt/fql_data as requested

I noticed there is a similiar bug
https://issues.apache.org/jira/browse/CASSANDRA-17136  but I also tried
setting disk_failure_policy to ignore, same thing.
Has Anyone encountered something similar?





gil