Thank you Alexander,
The restart did the trick.
Thank you,
Robert
Robert Sicoie
On Tue, Dec 6, 2016 at 5:00 PM, Alexander Dejanovski <a...@thelastpickle.com
> wrote:
> Hi Robert,
>
> Some of your nodes did not accept the new schema, or several alter
> commands were issued on different nodes concurrently.
> Repair cannot fix that as it works on data, not on schema.
>
> To fix your issue, roll restart your whole cluster and all nodes should
> get back in agreement on the schema.
>
> Cheers,
>
> On Tue, Dec 6, 2016 at 3:46 PM Robert Sicoie <robert.sic...@gmail.com>
> wrote:
>
>> Any suggestion is more than welcome.
>> The fact is that the same alter commands worked in two other environments
>> and now fails in the thirds environment. No downtime is accepted.
>>
>> Thanks,
>> Robert
>>
>> Robert Sicoie
>>
>> On Tue, Dec 6, 2016 at 4:11 PM, Robert Sicoie <robert.sic...@gmail.com>
>> wrote:
>>
>> Hi guys,
>>
>> I am running cassandra 3.0.5 on 5 nodes, NetworkTopologyStrategy, factor 2
>> I had to add new 3 columns to an existing type and I've done it through
>> cqlsh.
>> After one of the alter commands I've got
>>
>> *OperationTimedOut: errors={'x.x.x.x': 'Request timed out while waiting
>> for schema agreement. See Session.execute[_async](timeout) and
>> Cluster.max_schema_agreement_wait.'}, last_host=x.x.x.x*
>>
>> I've checked with describe schema and the new fields were in place, so I
>> suppose that cql tool timed out.
>>
>> After this I've noticed that the application (connecting through datastax
>> java driver, quorum consistency level) started to throw
>>
>> *com.datastax.driver.core.exceptions.ReadFailureException: Cassandra
>> failure during read query at consistency QUORUM (2 responses were required
>> but only 0 replica responded, 1 failed)*
>>
>>
>> I thought that this might be related to the schema changed and that
>> repair would force schema propagation so I run nodetool repair on the box
>> on which I trigged the schema alteration.
>>
>> Then I got
>>
>> *... Validation failed in /x.x.x.y (progress: 0%)*
>>
>> on this node and
>>
>> *ERROR [CompactionExecutor:37434] 2016-12-06 14:03:38,543
>> CassandraDaemon.java:195 - Exception in thread
>> Thread[CompactionExecutor:37434,1,main]*
>> *java.lang.AssertionError: null*
>> * at
>> org.apache.cassandra.db.rows.ComplexColumnData$Builder.addCell(ComplexColumnData.java:246)
>> ~[apache-cassandra-3.0.5.jar:3.0.5]*
>> * at
>> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:613)
>> ~[apache-cassandra-3.0.5.jar:3.0.5]*
>> * at
>> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:539)
>> ~[apache-cassandra-3.0.5.jar:3.0.5]*
>> * at
>> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:220)
>> ~[apache-cassandra-3.0.5.jar:3.0.5]*
>> * at
>> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:159)
>> ~[apache-cassandra-3.0.5.jar:3.0.5]*
>> * at
>> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
>> ~[apache-cassandra-3.0.5.jar:3.0.5]*
>> * at org.apache.cassandra.db.rows.Row$Merger.merge(Row.java:516)
>> ~[apache-cassandra-3.0.5.jar:3.0.5]*
>> *....*
>> * at
>> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263)
>> ~[apache-cassandra-3.0.5.jar:3.0.5]*
>> * at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[na:1.8.0_60]*
>> * at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[na:1.8.0_60]*
>> * at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[na:1.8.0_60]*
>> * at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> [na:1.8.0_60]*
>> * at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]*
>>
>> On the node /x.x.x.y
>>
>> Do you any suggestion?
>> Thank you in advance,
>> Robert
>>
>>
>> --
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
