[
https://issues.apache.org/jira/browse/CASSANDRA-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17724628#comment-17724628
]
Stefan Miklosovic commented on CASSANDRA-18541:
-----------------------------------------------
The provided graphs do not make sense to me but maybe I am interpreting them
wrong. When authsuccess rps is compared to cpu usage, it seems like more auth
success (more auth requests) are causing more stress on cpu which I would say
makes sense in general as some work has to be done right?
These options you set (..._validity_in_ms) caches the credentials so it does
not go to the disk to read them from the table. That does not mean that the
work will not be done. One potential bottleneck might be
PasswordAuthenticator.authenticate which calls checkpw method which uses BCrypt
which compares a hash from table (hashed password) with a password a client
provided to log in.
What BCrypt does is that it takes the hash from the table (cached), it parses
the number of rounds and other details and it will hash the plaintext password
with so and so many rounds. Then it will compare the hash in db with just
hashed password.
If "cassandra" role has e.g. this hash in system_auth.roles
"$2a$10$8XhBMxLGA3px/U0nHOczFOXxUNDcVOrD4czN6zRJHgpaUympsemgW"
"2a" in "$2a" is salt version, "10" in "$10" is number of salting round.
This can be a number from 4 to 30. Check CassandraRoleManager.getGensaltRounds
method.
More salting rounds you do, more CPU intensive it will be. I am afraid the
complexity of this raises exponentially. If you do like 25 rounds it will start
to eat too much CPU and requests may timeout. 10 is sweet spot between
complexity and security but if you throw 150 requests on it all of a sudden it
may look like what you see.
> AUTH requests use too much resources
> ------------------------------------
>
> Key: CASSANDRA-18541
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18541
> Project: Cassandra
> Issue Type: Bug
> Reporter: Yury Vidineev
> Priority: Normal
> Attachments: Screenshot_20230520_000633.png,
> Screenshot_20230520_000654.png
>
>
> Hello. I see unexpected CPU usage in a rare situation that may be worth
> digging into.
> We have C* 4.0.9 on Debian running on Java 11.0.18.
> It's a small cluster of 3 nodes on commodity hardware (6 cores CPU, 32 Gb
> RAM, 2 x 512 Gb SSD NVME).
> This ring has about 35 clients using Datastax Java Driver for Apache
> Cassandra.
> In the driver connection settings, we use the following:
> CONNECTION_POOL_LOCAL_SIZE = 400
> CONNECTION_POOL_REMOTE_SIZE = 100
>
> And for some reason, from time to time, it causes hundreds of AUTH requests
> per second that leads to an enormous CPU usage.
> And yes, it's easy not to use these settings in the driver, leaving defaults
> that don't produce such an amount of AUTHs. But isn't it weird that ~150 AUTH
> rps consume ~1200% CPU?
> Please see attached graphs.
> I have the following in the settings:
> authenticator: PasswordAuthenticator
> authorizer: CassandraAuthorizer
> roles_validity_in_ms: 600000
> permissions_validity_in_ms: 600000
> credentials_validity_in_ms: 600000
> Please let me know if I can provide any other necessary information.
> Thanks for your work. Cassandra is amazing :)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
