The cassandra user requires QUORUM consistency to be achieved for authentication. Normal users only require ONE. I suspect your system_auth keyspace has an RF of 1, and the node that owns the cassandra users data is down.
Steps to recover: 1. Turn off authentication on all the nodes 2. Restart the nodes and make sure they are UN 3. Alter system_auth to have a higher RF than 1 (3 is probably appropriate) 4. Turn auth back on and restart 5. Create a new user and use that from now on.
