Hi Robert,
Some of your nodes did not accept the new schema, or several alter commands
were issued on different nodes concurrently.
Repair cannot fix that as it works on data, not on schema.
To fix your issue, roll restart your whole cluster and all nodes should get
back in agreement on the schema.
Cheers,
On Tue, Dec 6, 2016 at 3:46 PM Robert Sicoie <robert.sic...@gmail.com>
wrote:
> Any suggestion is more than welcome.
> The fact is that the same alter commands worked in two other environments
> and now fails in the thirds environment. No downtime is accepted.
>
> Thanks,
> Robert
>
> Robert Sicoie
>
> On Tue, Dec 6, 2016 at 4:11 PM, Robert Sicoie <robert.sic...@gmail.com>
> wrote:
>
> Hi guys,
>
> I am running cassandra 3.0.5 on 5 nodes, NetworkTopologyStrategy, factor 2
> I had to add new 3 columns to an existing type and I've done it through
> cqlsh.
> After one of the alter commands I've got
>
> *OperationTimedOut: errors={'x.x.x.x': 'Request timed out while waiting
> for schema agreement. See Session.execute[_async](timeout) and
> Cluster.max_schema_agreement_wait.'}, last_host=x.x.x.x*
>
> I've checked with describe schema and the new fields were in place, so I
> suppose that cql tool timed out.
>
> After this I've noticed that the application (connecting through datastax
> java driver, quorum consistency level) started to throw
>
> *com.datastax.driver.core.exceptions.ReadFailureException: Cassandra
> failure during read query at consistency QUORUM (2 responses were required
> but only 0 replica responded, 1 failed)*
>
>
> I thought that this might be related to the schema changed and that repair
> would force schema propagation so I run nodetool repair on the box on which
> I trigged the schema alteration.
>
> Then I got
>
> *... Validation failed in /x.x.x.y (progress: 0%)*
>
> on this node and
>
> *ERROR [CompactionExecutor:37434] 2016-12-06 14:03:38,543
> CassandraDaemon.java:195 - Exception in thread
> Thread[CompactionExecutor:37434,1,main]*
> *java.lang.AssertionError: null*
> * at
> org.apache.cassandra.db.rows.ComplexColumnData$Builder.addCell(ComplexColumnData.java:246)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:613)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:539)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:220)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:516)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> *....*
> * at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]*
> * at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]*
> * at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]*
> * at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]*
> * at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]*
>
> On the node /x.x.x.y
>
> Do you any suggestion?
> Thank you in advance,
> Robert
>
>
> --
-----------------
Alexander Dejanovski
France
@alexanderdeja
Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
