Hi Vladimir, Thanks for the response.
When I see *"**com.datastax.driver.core.ControlConnection"* exceptions, I see that keyspaces and CF are created. But when I create CF with large number of columns(2400 cols) quickly one after the other(with 2 seconds gap between CREATE TABLE queries), I get schema agreement timeout errors (* com.datastax.driver.core.Cluster | Error while waiting for schema agreement). *This happens even with a clean slate(empty data directory), just after creating 4 keyspaces. Timeout is set to 30 seconds. Please note that CREATE TABLE queries are NOT fired in parallel. I wait for 1 query to complete(with schema agreement) before firing another one. I want to dig deeper into what all things happen in C* at time of CF creation to understand more about the limitation of number of keyspaces which can be created. Can you please point me to the corresponding source code? Specifically if you can also point me to the this 1MB per CF thingy, it would be great. Best Regards, Saumitra On Mon, Dec 19, 2016 at 11:41 PM, Vladimir Yudovin <vla...@winguzone.com> wrote: > Hi, > > *Question*: Does C* reads some schema/metadata on calling cqlsh, which is > causing timeout with large number of keyspaces? > A lot ). cqlsh reads schemas, cluster topology, each node tokens, etc. You > can just capture TCP port 9042 (unless you use SSL) and view all > negotiation between cqlsh and node. > > > *Question*: Can a single C* cluster of 5 nodes(32gb/8cpu each) support > upto 500 keyspaces each having 25 CFs. What kind of issues I can expect? > You have 500*25 = 12500 tables, it's huge number. Each CF takes at least > 1M of heap memory. So it needs 12G heap only for starting usage. Make test > on one-two node cluster. > > > *Question*: What is the effect of below exception? > Is keyspaces created despite exception or no? > > Best regards, Vladimir Yudovin, > *Winguzone <https://winguzone.com?from=list> - Cloud Cassandra Hosting* > > > ---- On Mon, 19 Dec 2016 10:24:20 -0500 *Saumitra S > <saumitra.srivast...@gmail.com <saumitra.srivast...@gmail.com>>* wrote > ---- > > Hi All, > > I have a 2 node cluster(32gb ram/8cpu) running 3.0.10 and I created 50 > keyspaces in it. Each keyspace has 25 CF. Column count in each CF ranges > between 5 to 30. > > I am getting few issues once keyspace count reaches ~50. > > *Issue 1:* > > When I try to use cqlsh, I get timeout. > > *$ cqlsh `hostname -i`* > *Connection error: ('Unable to connect to any servers', {'10.0.20.220': > OperationTimedOut('errors=None, last_host=None',)})* > > If I increase connect timeout, I am able to access cluster through cqlsh > > *$ cqlsh --connect-timeout 20 `hostname -i //this works fine* > > *Question: *Does C* reads some schema/metadata on calling cqlsh, which is > causing timeout with large number of keyspaces? > > > *Issue 2:* > > If I create keyspaces which have 3 large CF(each having around 2500 cols), > then I start to see schema agreement timeout in my logs. I have set schema > agreement timeout to 30 seconds in driver. > > *2016-12-13 08:37:02.733 | gbd-std-01 | WARN | cluster2-worker-194 | > com.datastax.driver.core.Cluster | Error while waiting for schema agreement* > > *Question:* Can a single C* cluster of 5 nodes(32gb/8cpu each) support > upto 500 keyspaces each having 25 CFs. What kind of issues I can expect? > > > *Issue 3:* > > I am creating keyspaces and CFs through datastax driver. I see following > exception in my log after reaching *~50 keyspaces.* > > *Question: *What is the effect of below exception? > > 2016-12-19 13:55:35.615 | gbd-std-01 | ERROR | cluster1-worker-147 | > *com.datastax.driver.core.ControlConnection > | [Control connection] Unexpected error while refreshing schema* > *java.util.concurrent.ExecutionException: > com.datastax.driver.core.exceptions.OperationTimedOutException: > [gbd-cass-20.ec2-east1.hidden.com/10.0.20.220 > <http://gbd-cass-20.ec2-east1.hidden.com/10.0.20.220>] Operation timed out* > at com.google.common.util.concurrent.AbstractFuture$ > Sync.getValue(AbstractFuture.java:299) ~[com.google.guava.guava-18.0. > jar:na] > at com.google.common.util.concurrent.AbstractFuture$ > Sync.get(AbstractFuture.java:286) ~[com.google.guava.guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[com.google.guava.guava-18.0.jar:na] > at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at > com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at com.datastax.driver.core.SchemaParser$V3SchemaParser. > fetchSystemRows(SchemaParser.java:632) ~[com.datastax.cassandra. > cassandra-driver-core-3.0.0.jar:na] > at com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at > com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at > com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at com.datastax.driver.core.Cluster$Manager$ > SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570) > [com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at com.datastax.driver.core.ExceptionCatchingRunnable.run( > ExceptionCatchingRunnable.java:32) [com.datastax.cassandra. > cassandra-driver-core-3.0.0.jar:na] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_45] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_45] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] > Caused by: com.datastax.driver.core.exceptions.OperationTimedOutException: > [gbd-cass-20.ec2-east1.hidden.com/10.0.20.220] Operation timed out > at com.datastax.driver.core.DefaultResultSetFuture.onTimeout( > DefaultResultSetFuture.java:209) ~[com.datastax.cassandra. > cassandra-driver-core-3.0.0.jar:na] > at > com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:1260) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at > io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:581) > ~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final] > at io.netty.util.HashedWheelTimer$HashedWheelBucket. > expireTimeouts(HashedWheelTimer.java:655) ~[io.netty.netty-common-4.0. > 33.Final.jar:4.0.33.Final] > at > io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:367) > ~[io.netty.netty-common-4.0.33.Final.jar:4.0.33.Final] > ... 1 common frames omitted > 2016-12-19 13:55:39.885 | gbd-std-01 | ERROR | cluster2-worker-124 | > *com.datastax.driver.core.ControlConnection > | [Control connection] Unexpected error while refreshing schema* > *java.util.concurrent.ExecutionException: > com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout > during read query at consistency ONE (1 responses were required but only 0 > replica responded)* > at com.google.common.util.concurrent.AbstractFuture$ > Sync.getValue(AbstractFuture.java:299) ~[com.google.guava.guava-18.0. > jar:na] > at com.google.common.util.concurrent.AbstractFuture$ > Sync.get(AbstractFuture.java:286) ~[com.google.guava.guava-18.0.jar:na] > at > com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) > ~[com.google.guava.guava-18.0.jar:na] > at com.datastax.driver.core.SchemaParser.get(SchemaParser.java:467) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at > com.datastax.driver.core.SchemaParser.access$400(SchemaParser.java:30) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at com.datastax.driver.core.SchemaParser$V3SchemaParser. > fetchSystemRows(SchemaParser.java:632) ~[com.datastax.cassandra. > cassandra-driver-core-3.0.0.jar:na] > at com.datastax.driver.core.SchemaParser.refresh(SchemaParser.java:56) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at > com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:341) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at > com.datastax.driver.core.ControlConnection.refreshSchema(ControlConnection.java:306) > ~[com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at com.datastax.driver.core.Cluster$Manager$ > SchemaRefreshRequestDeliveryCallback$1.runMayThrow(Cluster.java:2570) > [com.datastax.cassandra.cassandra-driver-core-3.0.0.jar:na] > at com.datastax.driver.core.ExceptionCatchingRunnable.run( > ExceptionCatchingRunnable.java:32) [com.datastax.cassandra. > cassandra-driver-core-3.0.0.jar:na] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_45] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_45] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_45] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] > > > > Best Regards, > Saumitra > > >
