On Wed, Mar 22, 2017 at 4:34 PM, Daniel Miranda <dan...@cobli.co> wrote:
> I found out the problem is conditioned to having the row cache enabled. > Whenever a query would return an empty result set in a particular table, it > would fail instead with the exception being thrown in all all nodes. > > Disabling the row cache for that particular table fixes the problem. > Re-enabling has not caused the issue again yet. I do not have row cache > saving enabled, and the issue persisted between node restarts, which it's > somewhat strange. > > I couldn't reproduce the issue with any other tables. I can produce a > sanitized dump of the SSTable if a developer desires. > > -- > Regards, > Daniel > > On Wed, Mar 22, 2017 at 2:33 PM Daniel Miranda <dan...@cobli.co> wrote: > >> Thank you for the pointer Michael, I'll try to investigate if this is the >> same bug I am seeing. >> >> I am afraid it might not be, since I'm observing the error periodically, >> not just during compactions, and the traceback seems different. >> >> Regards, >> Daniel >> >> On Wed, Mar 22, 2017 at 1:27 PM Michael Shuler <mich...@pbandjelly.org> >> wrote: >> >> Possibly https://issues.apache.org/jira/browse/CASSANDRA-12336, which >> shows fixed in 3.0.9, 3.8. There are a couple related bug reports listed >> on there, which you might investigate, as well. >> >> -- >> Kind regards, >> Michael >> >> On 03/22/2017 11:21 AM, Daniel Miranda wrote: >> > Greetings, >> > >> > Recently I've started to see the an assertion (traceback follows at the >> > end of the message) causing exceptions in a 3-node Cassandra 3.5 cluster >> > (running on Ubuntu 14.04 on Amazon EC2). It seems to happen in all >> > nodes. Repairs run fine without indicating any errors. >> > >> > I can't seem to find any information about it from someone else or any >> > bug reports. >> > >> > Should I bother running an SSTable scrub? Is it a known issue that is >> > fixed in subsequent versions? >> > >> > Thanks in advance, >> > Daniel >> > >> > --- >> > WARN [SharedPool-Worker-1] 2017-03-16 18:54:35,587 >> > AbstractLocalAwareExecutorService.java:169 - Uncaught exception on >> > thread Thread[SharedPool-Worker-1,5,main]: {} >> > java.lang.AssertionError: null >> > at >> > org.apache.cassandra.db.rows.UnfilteredRowIterators.concat( >> UnfilteredRowIterators.java:157) >> > ~[apache-cassandra-3.5.jar:3.5] >> > at >> > org.apache.cassandra.db.SinglePartitionReadCommand.getThroughCache( >> SinglePartitionReadCommand.java:420) >> > ~[apache-cassandra-3.5.jar:3.5] >> > at >> > org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage( >> SinglePartitionReadCommand.java:324) >> > ~[apache-cassandra-3.5.jar:3.5] >> > at >> > org.apache.cassandra.db.ReadCommand.executeLocally( >> ReadCommand.java:366) >> > ~[apache-cassandra-3.5.jar:3.5] >> > at >> > org.apache.cassandra.service.StorageProxy$ >> LocalReadRunnable.runMayThrow(StorageProxy.java:1797) >> > ~[apache-cassandra-3.5.jar:3.5] >> > at >> > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run( >> StorageProxy.java:2466) >> > ~[apache-cassandra-3.5.jar:3.5] >> > at >> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> > ~[na:1.8.0_111] >> > at >> > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ >> ice$FutureTask.run(AbstractLocalAwareExecutorService.java:164) >> > ~[apache-cassandra-3.5.jar:3.5] >> > at >> > org.apache.cassandra.concurrent.AbstractLocalAwareExecutorServ >> ice$LocalSessionFutureTask.run(AbstractLocalAwareExecutorServ >> ice.java:136) >> > [apache-cassandra-3.5.jar:3.5] >> > at >> > org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) >> > [apache-cassandra-3.5.jar:3.5] >> > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111] >> > -- >> > *Daniel Miranda* >> > >> > DevOps Engineering Intern >> > (11) 991959845 >> > www.cobli.co <http://www.cobli.co/> >> >> I would strongly advice not using the rowcache in the tick-tock series. It is not on by default and as a result I believe not heavily used. I can not find reference to it but a few months ago on this list someone pointed out that it was not doing anything in there version at all. More trouble then it is worth IMHO just use the standard caching.
