[
https://issues.apache.org/jira/browse/CASSANDRA-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14220863#comment-14220863
]
Jacek Lewandowski commented on CASSANDRA-8332:
----------------------------------------------
I can see the problem with dropping keyspace right after write-intensive
operations in 2.0.11.
The way to reproduce:
1. clean data directory
2. start C*
3. run subsequent trials
It happens after the second trial or later - depending on the amount of data
being written during a single trial. However, I couldn't reproduce it after a
single trial.
Now, the trial is:
1. create keyspace and table (bigint pk, 5 x text)
2. write *b* batches of *n* inserts in parallel (using Java Driver 2.1.2,
protocol V2) with *p* workers
3. drop keyspace
4. assert that the keyspace was really dropped by getting metadata from Cluster
object
The procedure break in step 1, because asserting Cluster metadata is useless -
it says the keyspace is missing. However, in the next trial, when it tries to
create the keyspace again, it says that the keyspace already exists. Notice,
that "drop keyspace" statement doesn't throw any exception, either in Java
Driver or in CQLSH.
I tried different values for b, n and p, for example b=50k, n=50, p=6 and each
text field is 30 bytes long (10 x 24-bit utf characters)
System log shows various errors. Some of them occur even during step 2 in the
second trial:
{noformat}
ERROR 13:01:00,422 Exception in thread Thread[CompactionExecutor:4,1,main]
java.lang.RuntimeException: java.io.FileNotFoundException:
/var/lib/cassandra/data/ks/tab/ks-tab-jb-2-Data.db (No such file or directory)
at
org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52)
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1366)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1172)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1184)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:272)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:278)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException:
/var/lib/cassandra/data/ks/tab/ks-tab-jb-2-Data.db (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
at
org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)
at
org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:34)
at
org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:48)
... 17 more
{noformat}
Very simiar errors are added to the system log when the keyspace is tried to be
dropped. However, in this case I also noticed the following exceptions:
{noformat}
INFO 12:43:35,006 Compaction interrupted:
Compaction@3e922a80-7599-3e0e-b804-6b32581a18d9(ks, tab, 1579024/110473935)bytes
ERROR 12:43:35,007 Unexpected error during query
java.lang.RuntimeException: java.util.concurrent.ExectionException:
java.lang.RuntimeException: Tried to hard link to file that does not exist
/var/lib/cassandra/data/ks/tab/ks-tab-jb-10-Statistics.db
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:413)
at
org.apache.cassandra.service.MigrationManager.announce(MigrationManager.java:285)
at
org.apache.cassandra.service.MigrationManager.announceKeyspaceDrop(MigrationManager.java:259)
at
org.apache.cassandra.cql3.statements.DropKeyspaceStatement.announceMigration(DropKeyspaceStatement.java:62)
at
org.apache.cassandra.cql3.statements.SchemaAlteringStatement.execute(SchemaAlteringStatement.java:79)
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:158)
at
com.datastax.bdp.cassandra.cql3.DseQueryHandler$StatementExecution.execute(DseQueryHandler.java:448)
at
com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithTiming(DseQueryHandler.java:190)
at
com.datastax.bdp.cassandra.cql3.DseQueryHandler.executeOperationWithAuditLogging(DseQueryHandler.java:223)
at
com.datastax.bdp.cassandra.cql3.DseQueryHandler.process(DseQueryHandler.java:103)
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:119)
at
org.apache.cassandra.transport.Message$Dispatcher.messageReceived(Message.java:321)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
at
org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by:java.util.concurrent.ExecutionException: java.lang.RuntimeException:
Tried to hard link to file that does not exist
/var/lib/cassandra/data/ks/tab/ks-tab-jb-10-Statistics.db
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:409)
... 19 more
Caused by: java.lang.RuntimeException: Tried to hard link to file that does not
exist /var/lib/cassandra/data/ks/tab/ks-tab-jb-10-Statistics.db
at org.apache.cassandra.io.util.FileUtils.createHardLink(FileUtils.java:77)
at
org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:1226)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1822)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1855)
at org.apache.cassandra.db.DefsTables.dropColumnFamily(DefsTables.java:404)
at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:288)
at org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:184)
at
org.apache.cassandra.service.MigrationManager$2.runMayThrow(MigrationManager.java:303)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
... 3 more
{noformat}
and
{noformat}
java.lang.AssertionError
at
org.apache.cassandra.io.compress.CompressionParameters.setLiveMetadata(CompressionParameters.java:108)
at
org.apache.cassandra.io.sstable.SSTableReader.getCompressionMetadata(SSTableReader.java:631)
at
org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1366)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:67)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1172)
at
org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1184)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:272)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:278)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:126)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Notice, that there are no "null" in the exception. Also - this exception was
thrown a little bit later (few seconds) after "drop keyspace" command was
tried.
> Null pointer after droping keyspace
> -----------------------------------
>
> Key: CASSANDRA-8332
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8332
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Lohfink
> Assignee: T Jake Luciani
> Priority: Minor
> Fix For: 2.1.3
>
>
> After dropping keyspace, sometimes I see this in logs:
> {code}
> ERROR 03:40:29 Exception in thread Thread[CompactionExecutor:2,1,main]
> java.lang.AssertionError: null
> at
> org.apache.cassandra.io.compress.CompressionParameters.setLiveMetadata(CompressionParameters.java:108)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getCompressionMetadata(SSTableReader.java:1142)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1896)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:68)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1681)
> ~[main/:na]
> at
> org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:1693)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:181)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getScanners(WrappingCompactionStrategy.java:320)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:340)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:151)
> ~[main/:na]
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> ~[main/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:75)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> ~[main/:na]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:233)
> ~[main/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_71]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_71]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
> {code}
> Minor issue since doesn't really affect anything, but the error makes it look
> like somethings wrong. Seen on 2.1 branch
> (1b21aef8152d96a180e75f2fcc5afad9ded6c595), not sure how far back (may be
> post 2.1.2).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
