Any suggestion is more than welcome.
The fact is that the same alter commands worked in two other environments
and now fails in the thirds environment. No downtime is accepted.
Thanks,
Robert
Robert Sicoie
On Tue, Dec 6, 2016 at 4:11 PM, Robert Sicoie <robert.sic...@gmail.com>
wrote:
> Hi guys,
>
> I am running cassandra 3.0.5 on 5 nodes, NetworkTopologyStrategy, factor 2
> I had to add new 3 columns to an existing type and I've done it through
> cqlsh.
> After one of the alter commands I've got
>
> *OperationTimedOut: errors={'x.x.x.x': 'Request timed out while waiting
> for schema agreement. See Session.execute[_async](timeout) and
> Cluster.max_schema_agreement_wait.'}, last_host=x.x.x.x*
>
> I've checked with describe schema and the new fields were in place, so I
> suppose that cql tool timed out.
>
> After this I've noticed that the application (connecting through datastax
> java driver, quorum consistency level) started to throw
>
> *com.datastax.driver.core.exceptions.ReadFailureException: Cassandra
> failure during read query at consistency QUORUM (2 responses were required
> but only 0 replica responded, 1 failed)*
>
>
> I thought that this might be related to the schema changed and that repair
> would force schema propagation so I run nodetool repair on the box on which
> I trigged the schema alteration.
>
> Then I got
>
> *... Validation failed in /x.x.x.y (progress: 0%)*
>
> on this node and
>
> *ERROR [CompactionExecutor:37434] 2016-12-06 14:03:38,543
> CassandraDaemon.java:195 - Exception in thread
> Thread[CompactionExecutor:37434,1,main]*
> *java.lang.AssertionError: null*
> * at
> org.apache.cassandra.db.rows.ComplexColumnData$Builder.addCell(ComplexColumnData.java:246)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:613)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:539)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:220)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:516)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> *....*
> * at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
> ~[apache-cassandra-3.0.5.jar:3.0.5]*
> * at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_60]*
> * at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_60]*
> * at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_60]*
> * at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_60]*
> * at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]*
>
> On the node /x.x.x.y
>
> Do you any suggestion?
> Thank you in advance,
> Robert
>
