[
https://issues.apache.org/jira/browse/CASSANDRA-13661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073107#comment-16073107
]
Rekha Joshi edited comment on CASSANDRA-13661 at 7/4/17 6:56 AM:
-----------------------------------------------------------------
Hi,
The cassandra cluster had following changes this week
>Scrub
>Repair
>Creation of new tables in a keyspace
>Dropping of two columns from a cassandra table.
The application error was the normal indexing it was able to do just prior to
these changes is failing now, giving ReadTimeoutException.
We increased request_timeout to 100000 just to see if that makes any
difference.It does not help.
*ReadTimeoutException*
2017-07-04T03:45:10,966+0000 ERROR
com.intuit.profile.writer.cassandra.index.IndexAppend[:91] - Error while
appending the index 'Cassandra timeout during read query at consistency ONE (1
responses were required but only 0 replica responded)'
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout
during read query at consistency ONE (1 responses were required but only 0
replica responded)
at
com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:89)
at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:217)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55)
at
com.intuit.profile.writer.cassandra.index.IndexService.readIdentifiables(IndexService.java:33)
at
com.intuit.profile.writer.cassandra.index.IndexAppend.run(IndexAppend.java:70)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra
timeout during read query at
At the cassandra log we see the continuous runtimeexception happening -
java.lang.RuntimeException:
30623431613136352d656433372d343939322d393066342d366632313961393530353062 is not
defined as a collection.
*java.lang.RuntimeException:
30623431613136352d656433372d343939322d393066342d366632313961393530353062 is not
defined as a collection*
INFO [MemtableFlushWriter:9] 2017-07-03 21:32:31,209 Memtable.java:382 -
Completed flushing
/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1376205-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1499140925239,
position=21125505)
ERROR [CompactionExecutor:9] 2017-07-03 21:32:31,245 CassandraDaemon.java:231 -
Exception in thread Thread[CompactionExecutor:9,1,main]
java.lang.RuntimeException:
30623431613136352d656433372d343939322d393066342d366632313961393530353062 is not
defined as a collection
at
org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.composites.CompoundSparseCellNameType$WithCollection.compare(CompoundSparseCellNameType.java:287)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.composites.CompoundSparseCellNameType$WithCollection.compare(CompoundSparseCellNameType.java:204)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.ArrayBackedSortedColumns.binarySearch(ArrayBackedSortedColumns.java:426)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.ArrayBackedSortedColumns.binarySearch(ArrayBackedSortedColumns.java:408)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.ArrayBackedSortedColumns.getColumn(ArrayBackedSortedColumns.java:272)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.LazilyCompactedRow$Reducer.reduce(LazilyCompactedRow.java:274)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.LazilyCompactedRow$Reducer.reduce(LazilyCompactedRow.java:219)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:205)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.1.jar:na]
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
~[guava-16.0.1.jar:na]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.1.jar:na]
at
org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:174)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:193)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
INFO [CompactionExecutor:9] 2017-07-03 21:32:31,247 CompactionTask.java:141 -
Compacting
[SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376203-Data.db'),
SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376204-Data.db'),
SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376205-Data.db'),
SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376201-Data.db'),
SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376202-Data.db')]
I have looked at both https://issues.apache.org/jira/browse/CASSANDRA-6332 and
https://issues.apache.org/jira/browse/CASSANDRA-5202
And that has not helped.Please let me know soon.
Thanks
Rekha
was (Author: rekhajoshm):
Hi,
The cassandra cluster had following changes this week
>Scrub
>Repair
>Creation of new tables in a keyspace
>Dropping of two columns from a cassandra table.
The application error was the normal indexing it was able to do just prior to
these changes, it is failing to do now, giving ReadTimeoutException.
We increased request_timeout to 100000 just to see if that makes any
difference.It does not help.
*ReadTimeoutException*
2017-07-04T03:45:10,966+0000 ERROR
com.intuit.profile.writer.cassandra.index.IndexAppend[:91] - Error while
appending the index 'Cassandra timeout during read query at consistency ONE (1
responses were required but only 0 replica responded)'
com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra timeout
during read query at consistency ONE (1 responses were required but only 0
replica responded)
at
com.datastax.driver.core.exceptions.ReadTimeoutException.copy(ReadTimeoutException.java:89)
at
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at
com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:217)
at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55)
at
com.intuit.profile.writer.cassandra.index.IndexService.readIdentifiables(IndexService.java:33)
at
com.intuit.profile.writer.cassandra.index.IndexAppend.run(IndexAppend.java:70)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.datastax.driver.core.exceptions.ReadTimeoutException: Cassandra
timeout during read query at
At the cassandra log we see the continuous runtimeexception happening -
java.lang.RuntimeException:
30623431613136352d656433372d343939322d393066342d366632313961393530353062 is not
defined as a collection.
*java.lang.RuntimeException:
30623431613136352d656433372d343939322d393066342d366632313961393530353062 is not
defined as a collection*
INFO [MemtableFlushWriter:9] 2017-07-03 21:32:31,209 Memtable.java:382 -
Completed flushing
/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-1376205-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1499140925239,
position=21125505)
ERROR [CompactionExecutor:9] 2017-07-03 21:32:31,245 CassandraDaemon.java:231 -
Exception in thread Thread[CompactionExecutor:9,1,main]
java.lang.RuntimeException:
30623431613136352d656433372d343939322d393066342d366632313961393530353062 is not
defined as a collection
at
org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.composites.CompoundSparseCellNameType$WithCollection.compare(CompoundSparseCellNameType.java:287)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.composites.CompoundSparseCellNameType$WithCollection.compare(CompoundSparseCellNameType.java:204)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.ArrayBackedSortedColumns.binarySearch(ArrayBackedSortedColumns.java:426)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.ArrayBackedSortedColumns.binarySearch(ArrayBackedSortedColumns.java:408)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.ArrayBackedSortedColumns.getColumn(ArrayBackedSortedColumns.java:272)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.LazilyCompactedRow$Reducer.reduce(LazilyCompactedRow.java:274)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.LazilyCompactedRow$Reducer.reduce(LazilyCompactedRow.java:219)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:205)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.1.jar:na]
at
com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
~[guava-16.0.1.jar:na]
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
~[guava-16.0.1.jar:na]
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
~[guava-16.0.1.jar:na]
at
org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:174)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:193)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:197)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
~[cassandra-all-2.1.15.1423.jar:2.1.15.1423]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
INFO [CompactionExecutor:9] 2017-07-03 21:32:31,247 CompactionTask.java:141 -
Compacting
[SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376203-Data.db'),
SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376204-Data.db'),
SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376205-Data.db'),
SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376201-Data.db'),
SSTableReader(path='/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-1376202-Data.db')]
I have looked at both https://issues.apache.org/jira/browse/CASSANDRA-6332 and
https://issues.apache.org/jira/browse/CASSANDRA-5202
And that has not helped.Please let me know soon.
Thanks
Rekha
> java.lang.RuntimeException:
> 30623431613136352d656433372d343939322d393066342d366632313961393530353062 is
> not defined as a collection
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-13661
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13661
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Reporter: Rekha Joshi
> Priority: Blocker
>
> ERROR [CompactionExecutor:9] 2017-07-03 21:20:31,221 CassandraDaemon.java:231
> - Exception in thread Thread[CompactionExecutor:9,1,main]
> java.lang.RuntimeException:
> 30623431613136352d656433372d343939322d393066342d366632313961393530353062 is
> not defined as a collection
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
