Transaction failed because of timeout, retry failed because of the first try actually succeeded.
Hi everyone,
I recently encountered a problem wrt light weight transaction. My query is
to insert a row to a table if the row doesn't exist. It goes like this:
Insert Into mytable (key, col1, col2) Value("key1", 1, 2) If Not Exist
My case is the driver somehow gets time out from waiting for coordinator to
response, but the transaction actually succeeded. So my code retry the
query and failed.
This is not an idempotent write, so the retry might be a bad idea. And
honestly this is not a cassandra issue. But i wonder if anyone in the
community ever had this problem before and how would you recommend to solve
it?
Thanks
--
come on
Re: Too many keyspaces causes cql connection to time out ?
so i guess i have to 1) increase the heap size or 2) reduce the number of keyspaces/column families. Thanks for you confirmation. On Tue, May 24, 2016 at 10:08 AM, Eric Stevens wrote: > Large numbers of tables is generally recommended against. Each table has > a fixed on-heap memory overhead, and by your description it sounds like you > might have as many as 12,000 total tables when you start running into > trouble. > > With such a small heap to begin with, you've probably used up most of the > available heap just with managing the tables. This is supported by the big > STW pauses you're observing in Cassandra's logs. > > On Tue, May 24, 2016 at 11:04 AM Justin Lin > wrote: > >> We are exploring cassandra's limit by creating a lot of keyspaces with >> moderate number of column families (roughly 40 - 50) per keyspace and we >> have a problem after we reach certain amount of keyspaces, that cqlsh >> starts to time out when connecting to cassandra. >> >> This is our cassandra setup. We have one cassandra running locally and we >> assign 4GB memory to jvm, set the memtable_allocation_type to be onheap and >> use default memtable_heap_space_in_mb, which i believe is 2GB. The >> cassandra version is 2.1.9. >> >> So after we create more than 250 keyspaces, cqlsh starts to times out >> when connecting to cassandra in most case. (Sometimes it still can connect >> to cassandra). And from cassandra log, we can see it takes roughly 3 >> seconds to do gc when there is an incoming connection. And the gc is the >> only difference between the timeout connection and the successful >> connection. So we suspect this Stop-The-World GC might block the connection >> until it times out. This is the log that i think is relevant. >> >> INFO 20160524-060930.028882 :: Initializing >> sandbox_20160524_t06_09_18.table1 >> >> INFO 20160524-060933.908008 :: G1 Young Generation GC in 551ms. G1 Eden >> Space: 98112 -> 0; G1 Old Gen: 2811821584 -> 3034119696; >> >> INFO 20160524-060933.908043 :: G1 Old Generation GC in 2631ms. G1 Old >> Gen: 3034119696 -> 2290099032; >> >> We suspect the issue might relate to the reported bug as well: >> https://issues.apache.org/jira/browse/CASSANDRA-9291 >> but not really sure about it. >> >> Sorry for the setup, so my question is >> 1) is the connection timeout related to gc or the tomestone in >> system.schame_keyspaces table? >> 2) how can we fix this problem? >> 3) I did some tests by dropping a bunch of keyspaces after timing out and >> it seems to fix the problem as i never got another time out. Is this the >> only way to fix it? >> >> Thanks a lot for your help. >> >> -- >> come on >> > -- come on
Too many keyspaces causes cql connection to time out ?
We are exploring cassandra's limit by creating a lot of keyspaces with moderate number of column families (roughly 40 - 50) per keyspace and we have a problem after we reach certain amount of keyspaces, that cqlsh starts to time out when connecting to cassandra. This is our cassandra setup. We have one cassandra running locally and we assign 4GB memory to jvm, set the memtable_allocation_type to be onheap and use default memtable_heap_space_in_mb, which i believe is 2GB. The cassandra version is 2.1.9. So after we create more than 250 keyspaces, cqlsh starts to times out when connecting to cassandra in most case. (Sometimes it still can connect to cassandra). And from cassandra log, we can see it takes roughly 3 seconds to do gc when there is an incoming connection. And the gc is the only difference between the timeout connection and the successful connection. So we suspect this Stop-The-World GC might block the connection until it times out. This is the log that i think is relevant. INFO 20160524-060930.028882 :: Initializing sandbox_20160524_t06_09_18.table1 INFO 20160524-060933.908008 :: G1 Young Generation GC in 551ms. G1 Eden Space: 98112 -> 0; G1 Old Gen: 2811821584 -> 3034119696; INFO 20160524-060933.908043 :: G1 Old Generation GC in 2631ms. G1 Old Gen: 3034119696 -> 2290099032; We suspect the issue might relate to the reported bug as well: https://issues.apache.org/jira/browse/CASSANDRA-9291 but not really sure about it. Sorry for the setup, so my question is 1) is the connection timeout related to gc or the tomestone in system.schame_keyspaces table? 2) how can we fix this problem? 3) I did some tests by dropping a bunch of keyspaces after timing out and it seems to fix the problem as i never got another time out. Is this the only way to fix it? Thanks a lot for your help. -- come on
