Re: CDC and schema disagreement

2024-09-20 Thread Štefan Miklošovič
led` in the cassandra.yaml file is set > to `false` on at least one node in the cluster, and then the `ALTER > TABLE ... WITH cdc=...` statement was run against that node, the cluster > will end up in the schema disagreement state. At this stage, a rolling > restart will bring the schema

CDC and schema disagreement

2024-09-20 Thread Bowen Song via user
chema disagreement state. At this stage, a rolling restart will bring the schema back in sync, but the changes made to the `cdc` table property will be lost. On Cassandra 4.1.6, the same procedure doesn't cause visible schema disagreement in the `nodetool describecluster` command's

Re: Schema disagreement

2018-05-01 Thread Gábor Auth
Hi, On Tue, May 1, 2018 at 10:27 PM Gábor Auth wrote: > One or two years ago I've tried the CDC feature but switched off... maybe > is it a side effect of switched off CDC? How can I fix it? :) > Okay, I've worked out. Updated the schema of the affected keyspaces on the new nodes with 'cdc=fals

Re: Schema disagreement

2018-05-01 Thread Gábor Auth
Hi, On Tue, May 1, 2018 at 7:40 PM Gábor Auth wrote: > What can I do? Any suggestion? :( > Okay, I've diffed the good and the bad system_scheme tables. The only difference is the `cdc` field in three keyspaces (in `tables` and `views`): - the value of `cdc` field on the good node is `False` - t

Re: Schema disagreement

2018-05-01 Thread Gábor Auth
Hi, On Mon, Apr 30, 2018 at 11:11 PM Gábor Auth wrote: > On Mon, Apr 30, 2018 at 11:03 PM Ali Hubail > wrote: > >> What steps have you performed to add the new DC? Have you tried to follow >> certain procedures like this? >> >> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/ops

Re: Schema disagreement

2018-04-30 Thread Gábor Auth
Hi, On Mon, Apr 30, 2018 at 11:03 PM Ali Hubail wrote: > What steps have you performed to add the new DC? Have you tried to follow > certain procedures like this? > > https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html > Yes, exactly. :/ Bye, Gábor Auth

Re: Schema disagreement

2018-04-30 Thread Ali Hubail
"user@cassandra.apache.org" , cc Subject Re: Schema disagreement Hi, On Mon, Apr 30, 2018 at 11:39 AM Gábor Auth wrote: 've just tried to add a new DC and new node to my cluster (3 DCs and 10 nodes) and the new node has a different schema version: Is it normal? Node is marked dow

Re: Schema disagreement

2018-04-30 Thread Gábor Auth
Hi, On Mon, Apr 30, 2018 at 11:39 AM Gábor Auth wrote: > 've just tried to add a new DC and new node to my cluster (3 DCs and 10 > nodes) and the new node has a different schema version: > Is it normal? Node is marked down but doing a repair successfully? WARN [MigrationStage:1] 2018-04-30 20

Schema disagreement

2018-04-30 Thread Gábor Auth
Hi, I've just tried to add a new DC and new node to my cluster (3 DCs and 10 nodes) and the new node has a different schema version: Cluster Information: Name: cluster Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch Partitioner: org.apache.cassandra.dht.Murmur3P

Re: Schema Disagreement vs Nodetool resetlocalschema

2016-09-11 Thread Jens Rantil
Hi Michael, Did you ever get an answer on this? I'm curious to hear for future reference. Thanks, Jens On Monday, June 20, 2016, Michael Fong wrote: > Hi, > > > > We have recently encountered several schema disagreement issue while > upgrading Cassandra. In one o

Schema Disagreement vs Nodetool resetlocalschema

2016-06-19 Thread Michael Fong
Hi, We have recently encountered several schema disagreement issue while upgrading Cassandra. In one of the cases, the 2-node cluster idled for over 30 minutes and their schema remain unsynced. Due to other logic flows, Cassandra cannot be restarted, and hence we need to come up an alternative

Re: What are problems with schema disagreement

2015-07-06 Thread Robert Coli
On Mon, Jul 6, 2015 at 1:30 PM, John Wong wrote: > But is there a problem with letting schema disagreement running for a long > time? > It depends on what the nature of the desynch is, but generally speaking there may be. If you added a column or a columnfamily, and one node didn&#

Re: What are problems with schema disagreement

2015-07-06 Thread John Wong
Thanks. Yeah we typically restart the nodes in the minor version to force resync. But is there a problem with letting schema disagreement running for a long time? Thanks. John On Mon, Jul 6, 2015 at 2:29 PM, Robert Coli wrote: > > > On Thu, Jul 2, 2015 at 9:31 PM, John Wong wrote

Re: What are problems with schema disagreement

2015-07-06 Thread Robert Coli
certain time, which led me to believe something happened internal, > although that was a poor wild guess. > > But is it safe to be okay with schema disagreement? I worry about data > consistency if I let it sit too long. > In general one shouldn't run with schema disagreement

Re: What are problems with schema disagreement

2015-07-02 Thread John Wong
was a poor wild guess. But is it safe to be okay with schema disagreement? I worry about data consistency if I let it sit too long. Thanks. John > > On Jul 2, 2015, at 9:37 PM, John Wong wrote: > > > > Hi. > > > > Here is a schema disagreement we encountered. &

Re: What are problems with schema disagreement

2015-07-02 Thread graham sanderson
gt; We deployed an application which would send a schema update (DDL=auto). We > found this prod cluster had 3 schema difference. Other existing applications > were fine, so some people were curious what if we left this problem alone > until off hours. > > Is there any concerns wi

What are problems with schema disagreement

2015-07-02 Thread John Wong
Hi. Here is a schema disagreement we encountered. Schema versions: b6467059-5897-3cc1-9ee2-73f31841b0b0: [10.0.1.100, 10.0.1.109] c8971b2d-0949-3584-aa87-0050a4149bbd: [10.0.1.55, 10.0.1.16, 10.0.1.77] c733920b-2a31-30f0-bca1-45a8c9130a2c: [10.0.1.221] We deployed an

RE: Cassandra schema disagreement

2014-08-12 Thread Demeyer Jonathan
Jonathan [mailto:jonathan.deme...@macq.eu] Sent: mardi 12 août 2014 11:03 To: user@cassandra.apache.org Subject: Cassandra schema disagreement Hello, I have a cluster running and I'm trying to change the schema on it. Altough it succeeds on one cluster (a test one), on another it keeps creatin

Cassandra schema disagreement

2014-08-12 Thread Demeyer Jonathan
Hello, I have a cluster running and I'm trying to change the schema on it. Altough it succeeds on one cluster (a test one), on another it keeps creating two separate schema versions (both are 2 DC configuration; the cluster where it goes wrong end up with a schema version on each DC). I use ap

RE: Schema disagreement errors

2014-05-13 Thread Donald Smith
@cassandra.apache.org Subject: Schema disagreement errors We have recently started seeing a lot of Schema Disagreement errors. We are using Cassandra 2.0.6 with Oracle Java 1.7. I went through the Cassandra FAQ and followed the below steps: * nodetool disablethrift * nodetool disablegossip

Re: Schema disagreement errors

2014-05-13 Thread Robert Coli
On Tue, May 13, 2014 at 5:11 PM, Donald Smith < donald.sm...@audiencescience.com> wrote: > I too have noticed that after doing “nodetool flush” (or “nodetool > drain”), the commit logs are still there. I think they’re NEW (empty) > commit logs, but I may be wrong. Anyone know? > Assuming they ar

Re: Schema disagreement errors

2014-05-13 Thread Vincent Mallet
Hey Gaurav, You should consider moving to 2.0.7 which fixes a bunch of these schema disagreement problems. You could also play around with nodetool resetlocalschema on the nodes that are behind, but be careful with that one. I'd go with 2.0.7 first for sure. Thanks, Vince. On Mon, M

Re: Schema disagreement errors

2014-05-13 Thread Duncan Sands
Hi Gaurav, a schema versioning bug was fixed in 2.0.7. Best wishes, Duncan. On 12/05/14 21:31, Gaurav Sehgal wrote: We have recently started seeing a lot of Schema Disagreement errors. We are using Cassandra 2.0.6 with Oracle Java 1.7. I went through the Cassandra FAQ and followed the below

Re: Schema disagreement errors

2014-05-12 Thread Laing, Michael
Upgrade to 2.0.7 fixed this for me. You can also try 'nodetool resetlocalschema' on disagreeing nodes. This worked temporarily for me in 2.0.6. ml On Mon, May 12, 2014 at 3:31 PM, Gaurav Sehgal wrote: > We have recently started seeing a lot of Schema Disagreement errors. W

Schema disagreement errors

2014-05-12 Thread Gaurav Sehgal
We have recently started seeing a lot of Schema Disagreement errors. We are using Cassandra 2.0.6 with Oracle Java 1.7. I went through the Cassandra FAQ and followed the below steps: - nodetool disablethrift - nodetool disablegossip - nodetool drain - 'kill '.

Re: Schema disagreement under normal conditions, ALTER TABLE hangs

2013-11-28 Thread Josh Dzielak
Thanks Rob. Let me add one thing in case someone else finds this thread - Restarting the nodes did not in and of itself get the schema disagreement resolved. We had to run the ALTER TABLE command individually on each of the disagreeing nodes once they came back up. On Tuesday, November 26

Re: Schema disagreement under normal conditions, ALTER TABLE hangs

2013-11-26 Thread Robert Coli
On Mon, Nov 25, 2013 at 6:42 PM, Josh Dzielak wrote: > Recently we had a strange thing happen. Altering schema (gc_grace_seconds) > for a column family resulted in a schema disagreement. 3/4 of nodes got it, > 1/4 didn't. There was no partition at the time, nor was there mul

Schema disagreement under normal conditions, ALTER TABLE hangs

2013-11-25 Thread Josh Dzielak
Recently we had a strange thing happen. Altering schema (gc_grace_seconds) for a column family resulted in a schema disagreement. 3/4 of nodes got it, 1/4 didn't. There was no partition at the time, nor was there multiple schema updates issued. Going to the nodes with stale schema and tryi

Re: Cannot resolve schema disagreement

2013-05-09 Thread srmore
st chance of success is : >> >> 1) dump the existing schema from a good node >> 2) nodetool drain on all nodes >> 3) stop cluster >> 4) move schema and migration CF tables out of the way on all nodes >> 5) start cluster >> 6) re-load schema, being careful

Re: Cannot resolve schema disagreement

2013-05-09 Thread aaron morton
> 6) re-load schema, being careful to explicitly check for schema > agreement on all nodes between schema modifying statements > > In many/most cases of schema disagreement, people try the FAQ approach > and it doesn't work and they end up being forced to do the above > anyway. In general if you can tolerate the downtime, you should save > yourself the effort and just do the above process. > > =Rob >

Re: Cannot resolve schema disagreement

2013-05-09 Thread srmore
t; 2) nodetool drain on all nodes > 3) stop cluster > 4) move schema and migration CF tables out of the way on all nodes > 5) start cluster > 6) re-load schema, being careful to explicitly check for schema > agreement on all nodes between schema modifying statements > > In many/mo

Re: Cannot resolve schema disagreement

2013-05-09 Thread Robert Coli
f the way on all nodes 5) start cluster 6) re-load schema, being careful to explicitly check for schema agreement on all nodes between schema modifying statements In many/most cases of schema disagreement, people try the FAQ approach and it doesn't work and they end up being forced to do the

Cannot resolve schema disagreement

2013-05-08 Thread srmore
Hello, I have a cluster of 4 nodes and two of them are on different schema. I tried to run the commands described in the FAQ section but no luck ( http://wiki.apache.org/cassandra/FAQ#schema_disagreement) . After running the commands, I get back to the same issue. Cannot afford to lose the data s

Re: Schema Disagreement after migration from 1.0.6 to 1.1.4

2012-09-05 Thread Martin Koch
x.com/docs/1.1/install/upgrading). > > > > After bringing up 1.1.4 there are no errors in the log, but the cluster > now suffers from schema disagreement > > > > [default@unknown] describe cluster; > > Cluster Information: > >Snitch: org.apache.cassandra.loc

Re: Schema Disagreement after migration from 1.0.6 to 1.1.4

2012-09-05 Thread Omid Aladini
ding). > > After bringing up 1.1.4 there are no errors in the log, but the cluster now > suffers from schema disagreement > > [default@unknown] describe cluster; > Cluster Information: >Snitch: org.apache.cassandra.locator.SimpleSnitch >Partitioner: org.apache.cassan

Re: Schema Disagreement after migration from 1.0.6 to 1.1.4

2012-09-05 Thread Edward Sargisson
://www.datastax.com/docs/1.1/install/upgrading). After bringing up 1.1.4 there are no errors in the log, but the cluster now suffers from schema disagreement [default@unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner

Schema Disagreement after migration from 1.0.6 to 1.1.4

2012-09-05 Thread Martin Koch
the log, but the cluster now suffers from schema disagreement [default@unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: 59adb24e-f3cd-3e02-97f0-5b395827453f

Re: How schema disagreement can be fixed faster on 1.0.10 cluster ?

2012-07-26 Thread Tyler Hobbs
I know you specified 1.0.10, but C* 1.1 solves this problem: http://www.datastax.com/dev/blog/the-schema-management-renaissance On Thu, Jul 26, 2012 at 7:29 AM, Mateusz Korniak < mateusz-li...@ant.gliwice.pl> wrote: > Hi ! > We got into schema disagreement situation on 1.0.10 hav

How schema disagreement can be fixed faster on 1.0.10 cluster ?

2012-07-26 Thread Mateusz Korniak
Hi ! We got into schema disagreement situation on 1.0.10 having 250GB of compressed data per node. Following http://wiki.apache.org/cassandra/FAQ#schema_disagreement after node restart looks like it is replaying all schema changes one be one , right ? As we did a lot of them during cluster

Re: Couldn't detect any schema definitions in local storage - after handling schema disagreement according to FAQ

2012-05-21 Thread aaron morton
ll? The approach you followed was the correct one. I've updated the wiki to say the errors are expected. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 19/05/2012, at 6:34 AM, Piavlo wrote: > Hi, > > I had a schema disagree

Couldn't detect any schema definitions in local storage - after handling schema disagreement according to FAQ

2012-05-18 Thread Piavlo
Hi, I had a schema disagreement problem in cassandra 1.0.9 cluster, where one node had different schema version. So I followed the faq at http://wiki.apache.org/cassandra/FAQ#schema_disagreement disabled gossip, disabled thrift, drained and finally stopped the cassandra process, on startup

Re: Schema disagreement in 1.0.2

2011-12-15 Thread blafrisch
context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Schema-disagreement-in-1-0-2-tp7098609p7099003.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Schema disagreement in 1.0.2

2011-12-15 Thread blafrisch
/cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Schema-disagreement-in-1-0-2-tp7098609p7098609.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: Schema disagreement issue in 1.0.0

2011-10-20 Thread aaron morton
Looks like a bug, patch is here https://issues.apache.org/jira/browse/CASSANDRA-3391 Until it is fixed avoid using CompositeType in the key_validator_class and blow away the Schema and Migrations SSTables. Cheers - Aaron Morton Freelance Developer @aaronmorton http://www.thel

Schema disagreement issue in 1.0.0

2011-10-20 Thread Tamil selvan R.S
Im facing the following issue with Cassandra 1.0 set up. The same works for 0.8.7 # cassandra-cli -h x.x.x.x -f RTSCFs.sch Connected to: "Real Time Stats" on x.x.x.x/9160 Authenticated to keyspace: Stats 39c3e120-fa24-11e0--61d449114eff Waiting for schema agreement... The schema has not settl

Re: How to solve this kind of schema disagreement...

2011-08-10 Thread aaron morton
I don't have time to look into the reasons for that error, but that does not sound good. It kind of sounds like there are multiple migration chains out there in the cluster. This could come from apply changes to different nodes at the same time. Is this a prod system ? If not I would shut it d

Re: How to solve this kind of schema disagreement...

2011-08-09 Thread Dikang Gu
And a lot of "not apply" logs. DEBUG [MigrationStage:1] 2011-08-10 11:36:29,376 DefinitionsUpdateVerbHandler.java (line 70) Applying AddColumnFamily from /192.168.1.9 DEBUG [MigrationStage:1] 2011-08-10 11:36:29,376 DefinitionsUpdateVerbHandler.java (line 80) Migration not applied Previous ver

Re: How to solve this kind of schema disagreement...

2011-08-09 Thread Dikang Gu
Hi Aaron, I set the log level to be DEBUG, and find a lot of forceFlush debug info in the log: DEBUG [StreamStage:1] 2011-08-10 11:31:56,345 ColumnFamilyStore.java (line 725) forceFlush requested but everything is clean DEBUG [StreamStage:1] 2011-08-10 11:31:56,345 ColumnFamilyStore.java (line

Re: How to solve this kind of schema disagreement...

2011-08-09 Thread aaron morton
um. There has got to be something stopping the migration from completing. Turn the logging up to DEBUG before starting and look for messages from MigrationManager.java Provide all the log messages from Migration.java on the 1.27 node Cheers - Aaron Morton Freelance Cassandra

Re: How to solve this kind of schema disagreement...

2011-08-07 Thread Dikang Gu
Hi Aaron, I repeat the whole procedure: 1. kill the cassandra instance on 1.27. 2. rm the data/system/Migrations-g-* 3. rm the data/system/Schema-g-* 4. bin/cassandra to start the cassandra. Now, the migration seems stop and I do not find any error in the system.log yet. The ring looks good: [

Re: How to solve this kind of schema disagreement...

2011-08-07 Thread aaron morton
did you check the logs in 1.27 for errors ? Could you be seeing this ? https://issues.apache.org/jira/browse/CASSANDRA-2867 Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 7 Aug 2011, at 16:24, Dikang Gu wrote: > I restart both

Re: How to solve this kind of schema disagreement...

2011-08-06 Thread Dikang Gu
I restart both nodes, and deleted the shcema* and migration* and restarted them. The current cluster looks like this: [default@unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema v

Re: How to solve this kind of schema disagreement...

2011-08-06 Thread aaron morton
After there restart you what was in the logs for the 1.27 machine from the Migration.java logger ? Some of the messages will start with "Applying migration" You should have shut down both of the nodes, then deleted the schema* and migration* system sstables, then restarted one of them and wat

Re: How to solve this kind of schema disagreement...

2011-08-06 Thread Dikang Gu
I have tried this, but the schema still does not agree in the cluster: [default@unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: UNREACHABLE: [192.168.1.28] 75eece10-b

Re: Schema Disagreement

2011-08-05 Thread Yi Yang
Thanks Aaron. On Aug 2, 2011, at 3:04 AM, aaron morton wrote: > Hang on, using brain now. > > That is triggering a small bug in the code see > https://issues.apache.org/jira/browse/CASSANDRA-2984 > > For not just remove the column meta data. > > Cheers > > - > Aaron Morton >

Re: Schema Disagreement

2011-08-05 Thread Yi Yang
Thanks Jonathan. On Aug 2, 2011, at 8:28 PM, Jonathan Ellis wrote: > Have you seen http://wiki.apache.org/cassandra/FAQ#schema_disagreement ? > > On Tue, Aug 2, 2011 at 10:25 PM, Dikang Gu wrote: >> I also encounter the schema disagreement in my 0.8.1 cluster today… >>

Re: How to solve this kind of schema disagreement...

2011-08-05 Thread Benoit Perroud
Based on http://wiki.apache.org/cassandra/FAQ#schema_disagreement, 75eece10-bf48-11e0--4d205df954a7 own the majority, so shutdown and remove the schema* and migration* sstables from both 192.168.1.28 and 192.168.1.27 2011/8/5 Dikang Gu : > [default@unknown] describe cluster; > Cluster Informa

How to solve this kind of schema disagreement...

2011-08-05 Thread Dikang Gu
[default@unknown] describe cluster; Cluster Information: Snitch: org.apache.cassandra.locator.SimpleSnitch Partitioner: org.apache.cassandra.dht.RandomPartitioner Schema versions: 743fe590-bf48-11e0--4d205df954a7: [192.168.1.28] 75eece10-bf48-11e0--4d205df954a7: [192.168.1.9, 192.1

Re: Schema Disagreement

2011-08-03 Thread aaron morton
apache.org/cassandra/FAQ#schema_disagreement ? >> >> On Tue, Aug 2, 2011 at 10:25 PM, Dikang Gu wrote: >>> I also encounter the schema disagreement in my 0.8.1 cluster today… >>> >>> The disagreement occurs when I create a column family using the hector api,

Re: Schema Disagreement

2011-08-02 Thread Dikang Gu
5 PM, Dikang Gu wrote: > > I also encounter the schema disagreement in my 0.8.1 cluster today… > > > > The disagreement occurs when I create a column family using the hector api, > > and I found the following errors in my cassandra/system.log > > ERROR [poo

Re: Schema Disagreement

2011-08-02 Thread Jonathan Ellis
Have you seen http://wiki.apache.org/cassandra/FAQ#schema_disagreement ? On Tue, Aug 2, 2011 at 10:25 PM, Dikang Gu wrote: > I also encounter the schema disagreement in my 0.8.1 cluster today… > > The disagreement occurs when I create a column family using the hector api, > and

Re: Schema Disagreement

2011-08-02 Thread Dikang Gu
I also encounter the schema disagreement in my 0.8.1 cluster today… The disagreement occurs when I create a column family using the hector api, and I found the following errors in my cassandra/system.log ERROR [pool-2-thread-99] 2011-08-03 11:21:18,051 Cassandra.java (line 3378) Internal error

Re: Schema Disagreement

2011-08-02 Thread aaron morton
Hang on, using brain now. That is triggering a small bug in the code see https://issues.apache.org/jira/browse/CASSANDRA-2984 For not just remove the column meta data. Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 2 Aug 2011

Re: Schema Disagreement

2011-08-02 Thread aaron morton
What do you see when you run describe cluster; in the cassandra-cli ? Whats the exact error you get and is there anything in the server side logs ? Have you added other CF's before adding this one ? Did the schema agree before starting this statement? I ran the statement below on the current tr

Re: Schema Disagreement

2011-08-01 Thread Dikang Gu
I thought the schema disagree problem was already solved in 0.8.1... On possible solution is to decommission the disagree node and rejoin it. On Tue, Aug 2, 2011 at 8:01 AM, Yi Yang wrote: > Dear all, > > I'm always meeting mp with schema disagree problems while trying to create > a column fam

Schema Disagreement

2011-08-01 Thread Yi Yang
Dear all, I'm always meeting mp with schema disagree problems while trying to create a column family like this, using cassandra-cli: create column family sd with column_type = 'Super' and key_validation_class = 'UUIDType' and comparator = 'LongType' and subcomparator = 'UTF8Type