Re: Keyspace/CF creation Timeouts
What is the error getting logged in system.log ? On 25 October 2016 at 15:31, Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > No, I am not trying to create many column families. > I already have 500 CF in my cluster and now I am trying to create one KS > and 3 CF. > > write_request_timeout_in_ms = 1 -> 10 seconds > > On Tue, Oct 25, 2016 at 3:00 PM, Surbhi Gupta > wrote: > >> As you have many keyspaces and column family to be created that might be >> the reason that within the stipulated time response is not coming back and >> u r seeing time out. >> Can you pls check write_request_timeout_in_ms also? >> >> On 25 October 2016 at 14:55, Edward Capriolo >> wrote: >> >>> I do not believe the ConsistencyLevel matters for schema changes. In >>> recent versions request_timeout_in_ms has been replaced by N variables >>> which allow different timeout values for different types of operations. >>> >>> You seem to have both a lot of keyspaces and column families. It seems >>> likely that you have a large cluster since you have mentioned multiple data >>> centers. Many people have similar problems namely: the operation that >>> changes the schema will timeout at the client level, but the cluster will >>> eventually carry out the change. >>> >>> Many people seem to be writing their schema changing code to issue the >>> request, ignore the response (in the case of timeout) and then use a >>> command like "describe schema|cluster" to confirm the change propagation. >>> Generally it is viewed as a annoying problem that people deal with. >>> >>> On Tue, Oct 25, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada < >>> jaibheem...@gmail.com> wrote: >>> 1. Yes, all nodes are up and running, 2. We are using the Local_QUORUM. On Tue, Oct 25, 2016 at 1:28 PM, Surbhi Gupta >>> > wrote: > 1. Make sure all nodes are up and running while you are trying to > create the Keyspaces and Column Family. > 2. What is the write consistency level u r using? > > > On 25 October 2016 at 13:18, Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> wrote: > >> Hello All, >> >> I have recently started noticing timeouts while creating KS/CF. this >> is happening with increase in no.of keyspaces. >> >> Does anyone have an idea what to look for? as I don't see any error >> or exception in the logs. >> or is there some kind of parameter change required? >> >> C* Version: 2.1.16 >> Total KS: 170 >> Total CF: 500 >> Total DC : 8 >> >> request_timeout_in_ms: 1 >> > > >>> >> >
Re: Keyspace/CF creation Timeouts
No, I am not trying to create many column families. I already have 500 CF in my cluster and now I am trying to create one KS and 3 CF. write_request_timeout_in_ms = 1 -> 10 seconds On Tue, Oct 25, 2016 at 3:00 PM, Surbhi Gupta wrote: > As you have many keyspaces and column family to be created that might be > the reason that within the stipulated time response is not coming back and > u r seeing time out. > Can you pls check write_request_timeout_in_ms also? > > On 25 October 2016 at 14:55, Edward Capriolo > wrote: > >> I do not believe the ConsistencyLevel matters for schema changes. In >> recent versions request_timeout_in_ms has been replaced by N variables >> which allow different timeout values for different types of operations. >> >> You seem to have both a lot of keyspaces and column families. It seems >> likely that you have a large cluster since you have mentioned multiple data >> centers. Many people have similar problems namely: the operation that >> changes the schema will timeout at the client level, but the cluster will >> eventually carry out the change. >> >> Many people seem to be writing their schema changing code to issue the >> request, ignore the response (in the case of timeout) and then use a >> command like "describe schema|cluster" to confirm the change propagation. >> Generally it is viewed as a annoying problem that people deal with. >> >> On Tue, Oct 25, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada < >> jaibheem...@gmail.com> wrote: >> >>> 1. Yes, all nodes are up and running, >>> 2. We are using the Local_QUORUM. >>> >>> On Tue, Oct 25, 2016 at 1:28 PM, Surbhi Gupta >>> wrote: >>> 1. Make sure all nodes are up and running while you are trying to create the Keyspaces and Column Family. 2. What is the write consistency level u r using? On 25 October 2016 at 13:18, Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Hello All, > > I have recently started noticing timeouts while creating KS/CF. this > is happening with increase in no.of keyspaces. > > Does anyone have an idea what to look for? as I don't see any error or > exception in the logs. > or is there some kind of parameter change required? > > C* Version: 2.1.16 > Total KS: 170 > Total CF: 500 > Total DC : 8 > > request_timeout_in_ms: 1 > >>> >> >
Re: Keyspace/CF creation Timeouts
As you have many keyspaces and column family to be created that might be the reason that within the stipulated time response is not coming back and u r seeing time out. Can you pls check write_request_timeout_in_ms also? On 25 October 2016 at 14:55, Edward Capriolo wrote: > I do not believe the ConsistencyLevel matters for schema changes. In > recent versions request_timeout_in_ms has been replaced by N variables > which allow different timeout values for different types of operations. > > You seem to have both a lot of keyspaces and column families. It seems > likely that you have a large cluster since you have mentioned multiple data > centers. Many people have similar problems namely: the operation that > changes the schema will timeout at the client level, but the cluster will > eventually carry out the change. > > Many people seem to be writing their schema changing code to issue the > request, ignore the response (in the case of timeout) and then use a > command like "describe schema|cluster" to confirm the change propagation. > Generally it is viewed as a annoying problem that people deal with. > > On Tue, Oct 25, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> wrote: > >> 1. Yes, all nodes are up and running, >> 2. We are using the Local_QUORUM. >> >> On Tue, Oct 25, 2016 at 1:28 PM, Surbhi Gupta >> wrote: >> >>> 1. Make sure all nodes are up and running while you are trying to create >>> the Keyspaces and Column Family. >>> 2. What is the write consistency level u r using? >>> >>> >>> On 25 October 2016 at 13:18, Jai Bheemsen Rao Dhanwada < >>> jaibheem...@gmail.com> wrote: >>> Hello All, I have recently started noticing timeouts while creating KS/CF. this is happening with increase in no.of keyspaces. Does anyone have an idea what to look for? as I don't see any error or exception in the logs. or is there some kind of parameter change required? C* Version: 2.1.16 Total KS: 170 Total CF: 500 Total DC : 8 request_timeout_in_ms: 1 >>> >>> >> >
Re: Keyspace/CF creation Timeouts
I do not believe the ConsistencyLevel matters for schema changes. In recent versions request_timeout_in_ms has been replaced by N variables which allow different timeout values for different types of operations. You seem to have both a lot of keyspaces and column families. It seems likely that you have a large cluster since you have mentioned multiple data centers. Many people have similar problems namely: the operation that changes the schema will timeout at the client level, but the cluster will eventually carry out the change. Many people seem to be writing their schema changing code to issue the request, ignore the response (in the case of timeout) and then use a command like "describe schema|cluster" to confirm the change propagation. Generally it is viewed as a annoying problem that people deal with. On Tue, Oct 25, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > 1. Yes, all nodes are up and running, > 2. We are using the Local_QUORUM. > > On Tue, Oct 25, 2016 at 1:28 PM, Surbhi Gupta > wrote: > >> 1. Make sure all nodes are up and running while you are trying to create >> the Keyspaces and Column Family. >> 2. What is the write consistency level u r using? >> >> >> On 25 October 2016 at 13:18, Jai Bheemsen Rao Dhanwada < >> jaibheem...@gmail.com> wrote: >> >>> Hello All, >>> >>> I have recently started noticing timeouts while creating KS/CF. this is >>> happening with increase in no.of keyspaces. >>> >>> Does anyone have an idea what to look for? as I don't see any error or >>> exception in the logs. >>> or is there some kind of parameter change required? >>> >>> C* Version: 2.1.16 >>> Total KS: 170 >>> Total CF: 500 >>> Total DC : 8 >>> >>> request_timeout_in_ms: 1 >>> >> >> >
Re: Keyspace/CF creation Timeouts
1. Yes, all nodes are up and running, 2. We are using the Local_QUORUM. On Tue, Oct 25, 2016 at 1:28 PM, Surbhi Gupta wrote: > 1. Make sure all nodes are up and running while you are trying to create > the Keyspaces and Column Family. > 2. What is the write consistency level u r using? > > > On 25 October 2016 at 13:18, Jai Bheemsen Rao Dhanwada < > jaibheem...@gmail.com> wrote: > >> Hello All, >> >> I have recently started noticing timeouts while creating KS/CF. this is >> happening with increase in no.of keyspaces. >> >> Does anyone have an idea what to look for? as I don't see any error or >> exception in the logs. >> or is there some kind of parameter change required? >> >> C* Version: 2.1.16 >> Total KS: 170 >> Total CF: 500 >> Total DC : 8 >> >> request_timeout_in_ms: 1 >> > >
Re: Keyspace/CF creation Timeouts
1. Make sure all nodes are up and running while you are trying to create the Keyspaces and Column Family. 2. What is the write consistency level u r using? On 25 October 2016 at 13:18, Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Hello All, > > I have recently started noticing timeouts while creating KS/CF. this is > happening with increase in no.of keyspaces. > > Does anyone have an idea what to look for? as I don't see any error or > exception in the logs. > or is there some kind of parameter change required? > > C* Version: 2.1.16 > Total KS: 170 > Total CF: 500 > Total DC : 8 > > request_timeout_in_ms: 1 >
Keyspace/CF creation Timeouts
Hello All, I have recently started noticing timeouts while creating KS/CF. this is happening with increase in no.of keyspaces. Does anyone have an idea what to look for? as I don't see any error or exception in the logs. or is there some kind of parameter change required? C* Version: 2.1.16 Total KS: 170 Total CF: 500 Total DC : 8 request_timeout_in_ms: 1
