date:20171012

Re: Best approach to prepare to shutdown a cassandra node

2017-10-12 Thread Anshu Vajpayee

Why are you killing when we have nodetool stopdaemon ?

On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas 
wrote:

> That's what I thought.
>
> Thanks!
>
> 2017-10-12 14:26 GMT-03:00 Hannu Kröger :
>
>> Hi,
>>
>> Drain should be enough.  It stops accepting writes and after that
>> cassandra can be safely shut down.
>>
>> Hannu
>>
>> On 12 October 2017 at 20:24:41, Javier Canillas (
>> javier.canil...@gmail.com) wrote:
>>
>> Hello everyone,
>>
>> I have some time working with Cassandra, but every time I need to
>> shutdown a node (for any reason like upgrading version or moving instance
>> to another host) I see several errors on the client applications (yes, I'm
>> using the official java driver).
>>
>> By the way, I'm starting C* as a stand-alone process
>> ,
>> and C* version is 3.11.0.
>>
>> The way I have implemented the shutdown process is something like the
>> following:
>>
>> *# Drain all information from commitlog into sstables*
>>
>> *bin/nodetool drain*
>>
>>
>> *cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
>> '{print $2}'`*
>> *if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
>> *echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
>> cassandra)"*
>> *kill $cassandra_pid*
>>
>> *echo -n "+ Checking it is down. "*
>> *counter=10*
>> *while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
>> 2>&1 ]*
>> *do*
>> *echo -n ". "*
>> *((counter--))*
>> *sleep 1s*
>> *done*
>> *echo ""*
>> *if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
>> *echo "+ Its down."*
>> *else*
>> *echo "- Killing Cassandra."*
>> *kill -9 $cassandra_pid*
>> *fi*
>> *else*
>> *echo "Care there was a problem finding Cassandra PID"*
>> *fi*
>>
>> Should I add at the beginning the following lines?
>>
>> echo "shutdowing cassandra gracefully with: nodetool disable gossip"
>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
>> echo "shutdowing cassandra gracefully with: nodetool disable binary
>> protocol"
>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
>> echo "shutdowing cassandra gracefully with: nodetool thrift"
>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>>
>> The shutdown log is the following:
>>
>> *WARN  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
>> StorageService.java:321 - Stopping gossip by operator request*
>> *INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
>> Gossiper.java:1532 - Announcing shutdown*
>> *INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
>> StorageService.java:2268 - Node /10.254.169.36  state
>> jump to shutdown*
>> *INFO  [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
>> Server.java:176 - Stop listening for CQL clients*
>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
>> StorageService.java:1442 - DRAINING: starting drain process*
>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
>> HintsService.java:220 - Paused hints dispatch*
>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
>> Gossiper.java:1532 - Announcing shutdown*
>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
>> StorageService.java:2268 - Node /127.0.0.1  state jump to
>> shutdown*
>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
>> MessagingService.java:984 - Waiting for messaging service to quiesce*
>> *INFO  [ACCEPT-/192.168.6.174 ] 2017-10-12
>> 14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
>> the accept() thread*
>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
>> HintsService.java:220 - Paused hints dispatch*
>> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
>> StorageService.java:1442 - DRAINED*
>>
>> Disabling Gossip seemed a good idea, but watching the logs, it may use it
>> to gracefully telling the other nodes he is going down, so I don't know if
>> it's good or bad idea.
>>
>> Disabling Thrift and Binary protocol should only avoid new connections,
>> but the one stablished and running should be attempted to finish.
>>
>> Any thoughts or comments?
>>
>> Thanks
>>
>> Javier.
>>
>>
>>
>


-- 
*Regards,*
*Anshu *

Re: Best approach to prepare to shutdown a cassandra node

2017-10-12 Thread Javier Canillas

That's what I thought.

Thanks!

2017-10-12 14:26 GMT-03:00 Hannu Kröger :

> Hi,
>
> Drain should be enough.  It stops accepting writes and after that
> cassandra can be safely shut down.
>
> Hannu
>
> On 12 October 2017 at 20:24:41, Javier Canillas (javier.canil...@gmail.com)
> wrote:
>
> Hello everyone,
>
> I have some time working with Cassandra, but every time I need to shutdown
> a node (for any reason like upgrading version or moving instance to another
> host) I see several errors on the client applications (yes, I'm using the
> official java driver).
>
> By the way, I'm starting C* as a stand-alone process
> ,
> and C* version is 3.11.0.
>
> The way I have implemented the shutdown process is something like the
> following:
>
> *# Drain all information from commitlog into sstables*
>
> *bin/nodetool drain*
>
>
> *cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
> '{print $2}'`*
> *if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
> *echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
> cassandra)"*
> *kill $cassandra_pid*
>
> *echo -n "+ Checking it is down. "*
> *counter=10*
> *while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
> 2>&1 ]*
> *do*
> *echo -n ". "*
> *((counter--))*
> *sleep 1s*
> *done*
> *echo ""*
> *if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
> *echo "+ Its down."*
> *else*
> *echo "- Killing Cassandra."*
> *kill -9 $cassandra_pid*
> *fi*
> *else*
> *echo "Care there was a problem finding Cassandra PID"*
> *fi*
>
> Should I add at the beginning the following lines?
>
> echo "shutdowing cassandra gracefully with: nodetool disable gossip"
> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
> echo "shutdowing cassandra gracefully with: nodetool disable binary
> protocol"
> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
> echo "shutdowing cassandra gracefully with: nodetool thrift"
> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>
> The shutdown log is the following:
>
> *WARN  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
> StorageService.java:321 - Stopping gossip by operator request*
> *INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
> Gossiper.java:1532 - Announcing shutdown*
> *INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
> StorageService.java:2268 - Node /10.254.169.36  state
> jump to shutdown*
> *INFO  [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
> Server.java:176 - Stop listening for CQL clients*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
> StorageService.java:1442 - DRAINING: starting drain process*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
> HintsService.java:220 - Paused hints dispatch*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
> Gossiper.java:1532 - Announcing shutdown*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
> StorageService.java:2268 - Node /127.0.0.1  state jump to
> shutdown*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
> MessagingService.java:984 - Waiting for messaging service to quiesce*
> *INFO  [ACCEPT-/192.168.6.174 ] 2017-10-12
> 14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
> the accept() thread*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
> HintsService.java:220 - Paused hints dispatch*
> *INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
> StorageService.java:1442 - DRAINED*
>
> Disabling Gossip seemed a good idea, but watching the logs, it may use it
> to gracefully telling the other nodes he is going down, so I don't know if
> it's good or bad idea.
>
> Disabling Thrift and Binary protocol should only avoid new connections,
> but the one stablished and running should be attempted to finish.
>
> Any thoughts or comments?
>
> Thanks
>
> Javier.
>
>
>

Re: Best approach to prepare to shutdown a cassandra node

2017-10-12 Thread Hannu Kröger

Hi,

Drain should be enough.  It stops accepting writes and after that cassandra
can be safely shut down.

Hannu

On 12 October 2017 at 20:24:41, Javier Canillas (javier.canil...@gmail.com)
wrote:

Hello everyone,

I have some time working with Cassandra, but every time I need to shutdown
a node (for any reason like upgrading version or moving instance to another
host) I see several errors on the client applications (yes, I'm using the
official java driver).

By the way, I'm starting C* as a stand-alone process
,
and C* version is 3.11.0.

The way I have implemented the shutdown process is something like the
following:

*# Drain all information from commitlog into sstables*

*bin/nodetool drain*


*cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
'{print $2}'`*
*if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
*echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
cassandra)"*
*kill $cassandra_pid*

*echo -n "+ Checking it is down. "*
*counter=10*
*while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
2>&1 ]*
*do*
*echo -n ". "*
*((counter--))*
*sleep 1s*
*done*
*echo ""*
*if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
*echo "+ Its down."*
*else*
*echo "- Killing Cassandra."*
*kill -9 $cassandra_pid*
*fi*
*else*
*echo "Care there was a problem finding Cassandra PID"*
*fi*

Should I add at the beginning the following lines?

echo "shutdowing cassandra gracefully with: nodetool disable gossip"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
echo "shutdowing cassandra gracefully with: nodetool disable binary
protocol"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
echo "shutdowing cassandra gracefully with: nodetool thrift"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift

The shutdown log is the following:

*WARN  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
StorageService.java:321 - Stopping gossip by operator request*
*INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
Gossiper.java:1532 - Announcing shutdown*
*INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
StorageService.java:2268 - Node /10.254.169.36  state
jump to shutdown*
*INFO  [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
Server.java:176 - Stop listening for CQL clients*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
StorageService.java:1442 - DRAINING: starting drain process*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
HintsService.java:220 - Paused hints dispatch*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
Gossiper.java:1532 - Announcing shutdown*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
StorageService.java:2268 - Node /127.0.0.1  state jump to
shutdown*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
MessagingService.java:984 - Waiting for messaging service to quiesce*
*INFO  [ACCEPT-/192.168.6.174 ] 2017-10-12
14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
the accept() thread*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
HintsService.java:220 - Paused hints dispatch*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
StorageService.java:1442 - DRAINED*

Disabling Gossip seemed a good idea, but watching the logs, it may use it
to gracefully telling the other nodes he is going down, so I don't know if
it's good or bad idea.

Disabling Thrift and Binary protocol should only avoid new connections, but
the one stablished and running should be attempted to finish.

Any thoughts or comments?

Thanks

Javier.

Best approach to prepare to shutdown a cassandra node

2017-10-12 Thread Javier Canillas

Hello everyone,

I have some time working with Cassandra, but every time I need to shutdown
a node (for any reason like upgrading version or moving instance to another
host) I see several errors on the client applications (yes, I'm using the
official java driver).

By the way, I'm starting C* as a stand-alone process
,
and C* version is 3.11.0.

The way I have implemented the shutdown process is something like the
following:

*# Drain all information from commitlog into sstables*

*bin/nodetool drain*


*cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
'{print $2}'`*
*if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
*echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
cassandra)"*
*kill $cassandra_pid*

*echo -n "+ Checking it is down. "*
*counter=10*
*while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
2>&1 ]*
*do*
*echo -n ". "*
*((counter--))*
*sleep 1s*
*done*
*echo ""*
*if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
*echo "+ Its down."*
*else*
*echo "- Killing Cassandra."*
*kill -9 $cassandra_pid*
*fi*
*else*
*echo "Care there was a problem finding Cassandra PID"*
*fi*

Should I add at the beginning the following lines?

echo "shutdowing cassandra gracefully with: nodetool disable gossip"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
echo "shutdowing cassandra gracefully with: nodetool disable binary
protocol"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
echo "shutdowing cassandra gracefully with: nodetool thrift"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift

The shutdown log is the following:

*WARN  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
StorageService.java:321 - Stopping gossip by operator request*
*INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
Gossiper.java:1532 - Announcing shutdown*
*INFO  [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
StorageService.java:2268 - Node /10.254.169.36  state
jump to shutdown*
*INFO  [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
Server.java:176 - Stop listening for CQL clients*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
StorageService.java:1442 - DRAINING: starting drain process*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
HintsService.java:220 - Paused hints dispatch*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
Gossiper.java:1532 - Announcing shutdown*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
StorageService.java:2268 - Node /127.0.0.1  state jump to
shutdown*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
MessagingService.java:984 - Waiting for messaging service to quiesce*
*INFO  [ACCEPT-/192.168.6.174 ] 2017-10-12
14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
the accept() thread*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
HintsService.java:220 - Paused hints dispatch*
*INFO  [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
StorageService.java:1442 - DRAINED*

Disabling Gossip seemed a good idea, but watching the logs, it may use it
to gracefully telling the other nodes he is going down, so I don't know if
it's good or bad idea.

Disabling Thrift and Binary protocol should only avoid new connections, but
the one stablished and running should be attempted to finish.

Any thoughts or comments?

Thanks

Javier.

Re: Schema Mismatch Issue in Production

2017-10-12 Thread Nitan Kainth

Rolling restart after making DDL changes saves us. We saw this because of race 
condition in our app servers, but it could happen for various other reasons 
like node overloaded, network etc

Sent from my iPhone

> On Oct 12, 2017, at 3:46 AM, Carlos Rolo  wrote:
> 
> Which version are you running? I got stuck in a similar situation (With a lot 
> more nodes) and the only way to make it good was to stop the whole cluster, 
> start nodes 1 by 1.
> 
> 
> 
> Regards,
> 
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>  
> Pythian - Love your data
> 
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: 
> linkedin.com/in/carlosjuzarterolo 
> Mobile: +351 918 918 100 
> www.pythian.com
> 
>> On Thu, Oct 12, 2017 at 5:53 AM, Pradeep Chhetri  
>> wrote:
>> Hello everyone,
>> 
>> We had some issues yesterday in our 3 nodes cluster where the application 
>> tried to create the same table twice quickly and cluster became unstable.
>> 
>> Temporarily, we reduced it to single node cluster which gave us some relief.
>> 
>> Now when we are trying to bootstrap a new node and add it to cluster. we're 
>> seeing schema mismatch issue. 
>> 
>> # nodetool status
>> Datacenter: datacenter1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  AddressLoad   Tokens   Owns (effective)  Host ID 
>>   Rack
>> UN  10.42.247.173  3.07 GiB   256  100.0%
>> dffc39e5-d4ba-4b10-872e-0e3cc10f5e08  rack1
>> UN  10.42.209.245  2.25 GiB   256  100.0%
>> 9b99d5d8-818e-4741-9533-259d0fc0e16d  rack1
>> 
>> root@cassandra-2:~# nodetool describecluster
>> Cluster Information:
>> Name: sa-cassandra
>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Schema versions:
>> e2275d0f-a5fc-39d9-8f11-268b5e9dc295: [10.42.209.245]
>> 
>> 5f5f66f5-d6aa-3b90-b674-e08811d4d412: [10.42.247.173]
>> 
>> Freshly bootstrapped node - 10.42.247.173 
>> Single node from original cluster - 10.42.209.245
>> 
>> I read 
>> https://docs.datastax.com/en/dse-trblshoot/doc/troubleshooting/schemaDisagree.html
>>  and tried restarting the new node but it didnt help.
>> 
>> Please do suggest. We are facing this issue in production.
>> 
>> Thank you.
> 
> 
> --
> 
> 
> 
>

Re: [RELEASE] Apache Cassandra 3.11.1 released

2017-10-12 Thread Lutaya Shafiq Holmes

sounds good

On 10/11/17, Russell Bateman  wrote:
> I got it on Linux Mint via the Update Manager this morning already.
>
>
> On 10/11/2017 07:33 AM, Lucas Benevides wrote:
>> Hello Michael Schuler,
>>
>> When will this version become available for upgrade from apt-get? I
>> visited the address http://www.apache.org/dist/cassandra/debian and
>> there was no version 3111.
>>
>> To me it is easier to upgrade the nodes this way as I am in a lab, not
>> in a production site.
>>
>> Thanks in advance,
>> Lucas Benevides
>>
>>
>> 2017-10-10 18:14 GMT-03:00 Michael Shuler > >:
>>
>> The Cassandra team is pleased to announce the release of Apache
>> Cassandra version 3.11.1.
>>
>> Apache Cassandra is a fully distributed database. It is the right
>> choice
>> when you need scalability and high availability without compromising
>> performance.
>>
>> http://cassandra.apache.org/
>>
>> Downloads of source and binary distributions are listed in our
>> download
>> section:
>>
>> http://cassandra.apache.org/download/
>> 
>>
>> This version is a bug fix release[1] on the 3.11 series. As always,
>> please pay attention to the release notes[2] and Let us know[3] if
>> you
>> were to encounter any problem.
>>
>> Enjoy!
>>
>> [1]: (CHANGES.txt) https://goo.gl/QFBuPn
>> [2]: (NEWS.txt) https://goo.gl/vHd41x
>> [3]: https://issues.apache.org/jira/browse/CASSANDRA
>> 
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> 
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>>
>>
>
>


-- 
Lutaaya Shafiq
Web: www.ronzag.com | i...@ronzag.com
Mobile: +256702772721 | +256783564130
Twitter: @lutayashafiq
Skype: lutaya5
Blog: lutayashafiq.com
http://www.fourcornersalliancegroup.com/?a=shafiqholmes

"The most beautiful people we have known are those who have known defeat,
known suffering, known struggle, known loss and have found their way out of
the depths. These persons have an appreciation, a sensitivity and an
understanding of life that fills them with compassion, gentleness and a
deep loving concern. Beautiful people do not just happen." - *Elisabeth
Kubler-Ross*

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Schema Mismatch Issue in Production

2017-10-12 Thread Pradeep Chhetri

Got the cluster to converge on same schema by restarting just the one
having different version.

Thanks.

On Thu, Oct 12, 2017 at 2:23 PM, Pradeep Chhetri 
wrote:

> Hi Carlos,
>
> Thank you for the reply.
>
> I am running 3.9 cassandra version.
>
> I am also not sure what affect does this have on the applications talking
> to the cassandra.
>
> So there is no way to converge the cluster schema version without downtime.
>
> Thank you.
>
> On Thu, Oct 12, 2017 at 2:16 PM, Carlos Rolo  wrote:
>
>> Which version are you running? I got stuck in a similar situation (With a
>> lot more nodes) and the only way to make it good was to stop the whole
>> cluster, start nodes 1 by 1.
>>
>>
>>
>> Regards,
>>
>> Carlos Juzarte Rolo
>> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>>
>> Pythian - Love your data
>>
>> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
>> *linkedin.com/in/carlosjuzarterolo
>> *
>> Mobile: +351 918 918 100
>> www.pythian.com
>>
>> On Thu, Oct 12, 2017 at 5:53 AM, Pradeep Chhetri 
>> wrote:
>>
>>> Hello everyone,
>>>
>>> We had some issues yesterday in our 3 nodes cluster where the
>>> application tried to create the same table twice quickly and cluster became
>>> unstable.
>>>
>>> Temporarily, we reduced it to single node cluster which gave us some
>>> relief.
>>>
>>> Now when we are trying to bootstrap a new node and add it to cluster.
>>> we're seeing schema mismatch issue.
>>>
>>> # nodetool status
>>> Datacenter: datacenter1
>>> ===
>>> Status=Up/Down
>>> |/ State=Normal/Leaving/Joining/Moving
>>> --  AddressLoad   Tokens   Owns (effective)  Host ID
>>>Rack
>>> UN  10.42.247.173  3.07 GiB   256  100.0%
>>> dffc39e5-d4ba-4b10-872e-0e3cc10f5e08  rack1
>>> UN  10.42.209.245  2.25 GiB   256  100.0%
>>> 9b99d5d8-818e-4741-9533-259d0fc0e16d  rack1
>>>
>>> root@cassandra-2:~# nodetool describecluster
>>> Cluster Information:
>>> Name: sa-cassandra
>>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> Schema versions:
>>> e2275d0f-a5fc-39d9-8f11-268b5e9dc295: [10.42.209.245]
>>>
>>> 5f5f66f5-d6aa-3b90-b674-e08811d4d412: [10.42.247.173]
>>>
>>> Freshly bootstrapped node - 10.42.247.173
>>> Single node from original cluster - 10.42.209.245
>>>
>>> I read https://docs.datastax.com/en/dse-trblshoot/doc/troubles
>>> hooting/schemaDisagree.html and tried restarting the new node but it
>>> didnt help.
>>>
>>> Please do suggest. We are facing this issue in production.
>>>
>>> Thank you.
>>>
>>
>>
>> --
>>
>>
>>
>>
>

Re: Schema Mismatch Issue in Production

2017-10-12 Thread Pradeep Chhetri

Hi Carlos,

Thank you for the reply.

I am running 3.9 cassandra version.

I am also not sure what affect does this have on the applications talking
to the cassandra.

So there is no way to converge the cluster schema version without downtime.

Thank you.

On Thu, Oct 12, 2017 at 2:16 PM, Carlos Rolo  wrote:

> Which version are you running? I got stuck in a similar situation (With a
> lot more nodes) and the only way to make it good was to stop the whole
> cluster, start nodes 1 by 1.
>
>
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
> *linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +351 918 918 100
> www.pythian.com
>
> On Thu, Oct 12, 2017 at 5:53 AM, Pradeep Chhetri 
> wrote:
>
>> Hello everyone,
>>
>> We had some issues yesterday in our 3 nodes cluster where the application
>> tried to create the same table twice quickly and cluster became unstable.
>>
>> Temporarily, we reduced it to single node cluster which gave us some
>> relief.
>>
>> Now when we are trying to bootstrap a new node and add it to cluster.
>> we're seeing schema mismatch issue.
>>
>> # nodetool status
>> Datacenter: datacenter1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  AddressLoad   Tokens   Owns (effective)  Host ID
>>  Rack
>> UN  10.42.247.173  3.07 GiB   256  100.0%
>> dffc39e5-d4ba-4b10-872e-0e3cc10f5e08  rack1
>> UN  10.42.209.245  2.25 GiB   256  100.0%
>> 9b99d5d8-818e-4741-9533-259d0fc0e16d  rack1
>>
>> root@cassandra-2:~# nodetool describecluster
>> Cluster Information:
>> Name: sa-cassandra
>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Schema versions:
>> e2275d0f-a5fc-39d9-8f11-268b5e9dc295: [10.42.209.245]
>>
>> 5f5f66f5-d6aa-3b90-b674-e08811d4d412: [10.42.247.173]
>>
>> Freshly bootstrapped node - 10.42.247.173
>> Single node from original cluster - 10.42.209.245
>>
>> I read https://docs.datastax.com/en/dse-trblshoot/doc/troubles
>> hooting/schemaDisagree.html and tried restarting the new node but it
>> didnt help.
>>
>> Please do suggest. We are facing this issue in production.
>>
>> Thank you.
>>
>
>
> --
>
>
>
>

Re: Schema Mismatch Issue in Production

2017-10-12 Thread Carlos Rolo

Which version are you running? I got stuck in a similar situation (With a
lot more nodes) and the only way to make it good was to stop the whole
cluster, start nodes 1 by 1.



Regards,

Carlos Juzarte Rolo
Cassandra Consultant / Datastax Certified Architect / Cassandra MVP

Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin:
*linkedin.com/in/carlosjuzarterolo
*
Mobile: +351 918 918 100
www.pythian.com

On Thu, Oct 12, 2017 at 5:53 AM, Pradeep Chhetri 
wrote:

> Hello everyone,
>
> We had some issues yesterday in our 3 nodes cluster where the application
> tried to create the same table twice quickly and cluster became unstable.
>
> Temporarily, we reduced it to single node cluster which gave us some
> relief.
>
> Now when we are trying to bootstrap a new node and add it to cluster.
> we're seeing schema mismatch issue.
>
> # nodetool status
> Datacenter: datacenter1
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  AddressLoad   Tokens   Owns (effective)  Host ID
>  Rack
> UN  10.42.247.173  3.07 GiB   256  100.0%
> dffc39e5-d4ba-4b10-872e-0e3cc10f5e08  rack1
> UN  10.42.209.245  2.25 GiB   256  100.0%
> 9b99d5d8-818e-4741-9533-259d0fc0e16d  rack1
>
> root@cassandra-2:~# nodetool describecluster
> Cluster Information:
> Name: sa-cassandra
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> e2275d0f-a5fc-39d9-8f11-268b5e9dc295: [10.42.209.245]
>
> 5f5f66f5-d6aa-3b90-b674-e08811d4d412: [10.42.247.173]
>
> Freshly bootstrapped node - 10.42.247.173
> Single node from original cluster - 10.42.209.245
>
> I read https://docs.datastax.com/en/dse-trblshoot/doc/troubles
> hooting/schemaDisagree.html and tried restarting the new node but it
> didnt help.
>
> Please do suggest. We are facing this issue in production.
>
> Thank you.
>

-- 


--

Re: Best approach to prepare to shutdown a cassandra node

Re: Best approach to prepare to shutdown a cassandra node

Re: Best approach to prepare to shutdown a cassandra node

Best approach to prepare to shutdown a cassandra node

Re: Schema Mismatch Issue in Production

Re: [RELEASE] Apache Cassandra 3.11.1 released

Re: Schema Mismatch Issue in Production

Re: Schema Mismatch Issue in Production

Re: Schema Mismatch Issue in Production

9 matches

Site Navigation

Mail list logo

Footer information