That error message sounds like one of the nodes timed out in the paxos
propose stage. You can check the system.log and gc.log and see if you
can find anything unusual in them, such as network errors, out of sync
clocks or long stop-the-world GC pauses.
BTW, since you said you want it to be fast, I think it's worth
mentioning that LWT comes with additional cost and is much slower than a
straight forward INSERT/UPDATE. You should avoid using it if possible.
For example, if all of the Cassandra clients (samba servers) are running
on the same machine, it may be far more efficient to use a lock than LWT.
On 11/04/2023 18:18, Ralph Boehme wrote:
Hi folks!
Ralph here from the Samba team.
I'm currently doing research into Opensource distributed NoSQL
key/value stores to be used by Samba as an more scalable alternative
to Samba's own homegrown distributed key/value store called "ctdb" [1].
As an Opensource implementation of the SMB filesharing protocol from
Microsoft, we have some specific requirements wrt to database behaviour:
- fast
- fast
- fast
- highly consistent, iow linearizable
We got away without a linearizable database as historically the SMB
protocol and the SMB client implementations were built around the
assumption that handle and session state at the server could be lost
due to events like process or server crashes and client would
implement a best effort strategy to recover client state.
Modern SMB3 offers stronger guarantees which require a strongly
consistent ie linearizable database.
While prototyping a Python module for our pluggable database client in
Samba I ran into the following issue with Cassandra:
File "cassandra/cluster.py", line 2618, in
cassandra.cluster.Session.execute
File "cassandra/cluster.py", line 4901, in
cassandra.cluster.ResponseFuture.result
cassandra.protocol.ErrorMessageSub: <Error from server: code=1700
[Unknown] message="CAS operation result is unknown - proposal accepted
by 1 but not a quorum.">
This happens when executing the following LWT:
f'''
INSERT INTO {dbname} (key, guid, owner, refcount)
VALUES (?, ?, ?, ?)
IF NOT EXISTS
''')
This is the first time I'm running Cassandra. I've just setup a three
node test cluster and everything looks ok:
# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 172.18.200.21 360,09 KiB 16 100,0%
4590f3a6-4ca5-466f-a24d-edc54afa36f0 rack1
UN 172.18.200.23 326,92 KiB 16 100,0%
9175fd4e-4d84-4899-878a-dd5266132ff8 rack1
UN 172.18.200.22 335,32 KiB 16 100,0%
35e05369-cc8a-4642-b98d-a5fcc326502f rack1
Can anyone shed some light on what I might be doing wrong?
Thanks!
-slow
[1] <https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba>
