Large numbers of tables is generally recommended against. Each table has a fixed on-heap memory overhead, and by your description it sounds like you might have as many as 12,000 total tables when you start running into trouble.
With such a small heap to begin with, you've probably used up most of the available heap just with managing the tables. This is supported by the big STW pauses you're observing in Cassandra's logs. On Tue, May 24, 2016 at 11:04 AM Justin Lin <linjianfeng...@gmail.com> wrote: > We are exploring cassandra's limit by creating a lot of keyspaces with > moderate number of column families (roughly 40 - 50) per keyspace and we > have a problem after we reach certain amount of keyspaces, that cqlsh > starts to time out when connecting to cassandra. > > This is our cassandra setup. We have one cassandra running locally and we > assign 4GB memory to jvm, set the memtable_allocation_type to be onheap and > use default memtable_heap_space_in_mb, which i believe is 2GB. The > cassandra version is 2.1.9. > > So after we create more than 250 keyspaces, cqlsh starts to times out when > connecting to cassandra in most case. (Sometimes it still can connect to > cassandra). And from cassandra log, we can see it takes roughly 3 seconds > to do gc when there is an incoming connection. And the gc is the only > difference between the timeout connection and the successful connection. So > we suspect this Stop-The-World GC might block the connection until it times > out. This is the log that i think is relevant. > > INFO 20160524-060930.028882 :: Initializing > sandbox_20160524_t06_09_18.table1 > > INFO 20160524-060933.908008 :: G1 Young Generation GC in 551ms. G1 Eden > Space: 222298112 -> 0; G1 Old Gen: 2811821584 -> 3034119696; > > INFO 20160524-060933.908043 :: G1 Old Generation GC in 2631ms. G1 Old > Gen: 3034119696 -> 2290099032; > > We suspect the issue might relate to the reported bug as well: > https://issues.apache.org/jira/browse/CASSANDRA-9291 > but not really sure about it. > > Sorry for the setup, so my question is > 1) is the connection timeout related to gc or the tomestone in > system.schame_keyspaces table? > 2) how can we fix this problem? > 3) I did some tests by dropping a bunch of keyspaces after timing out and > it seems to fix the problem as i never got another time out. Is this the > only way to fix it? > > Thanks a lot for your help. > > -- > come on >
