[ https://issues.apache.org/jira/browse/CASSANDRA-6829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Shuler resolved CASSANDRA-6829. --------------------------------------- Resolution: Not a Problem [~odpeer] I'm closing this as not a problem with Cassandra, itself. Do seek some help on the Cassandra IRC channel or mailing list, if you need some help with your cluster - I know this ticket is elderly, so I hope that you made some progress in the last months :) > nodes sporadically shutting down > -------------------------------- > > Key: CASSANDRA-6829 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6829 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Windows Azure VMs. > The VMs OS is SUSE Enterprise. I striped 2 logical volumes for each VM, one > for data and one for commitlog, and formatted them as XFS. > Oracle Java 1.7_45 > Datastax Enterprise 4.0 (Cassandra version 2.0.5.22) > Reporter: Oded Peer > > I deployed a Datastax 4.0 Cassandra cluster in Windows Azure and started load > tests. After a while some of the nodes announce shutdown and stop responding > to client requests. > The error preceding the shutdown is "FSWriteError in > /mnt/dsedata/lib/cassandra/poc/event_log/poc-event_log-tmp-jb-581-Data.db" > "Caused by: java.io.IOException: Input/output error". > The storage I'm using in my VMs is Azure Blob storage. The VMs OS is SUSE > Enterprise. I striped 2 logical volumes for each VM, one for data and one > for commitlog, and formatted them as XFS. > I am using Oracle Java 1.7_45 > Restarting the Cassandra process resolves the problem for a short while > (minutes) afterwards the problem occurs again. > I noticed that it happens only in tmp files of a specific table. See the > errors from 3 random nodes: > (1) ERROR [CompactionExecutor:48] 2014-03-09 11:38:45,188 > CassandraDaemon.java (line 192) Exception in thread > Thread[CompactionExecutor:48,1,main] > FSWriteError in > /mnt/dsedata/lib/cassandra/poc/event_log/poc-event_log-tmp-jb-409-Data.db > (2) ERROR [CompactionExecutor:37] 2014-03-10 10:04:30,828 > CassandraDaemon.java (line 196) Exception in thread > Thread[CompactionExecutor:37,1,main] > FSWriteError in > /mnt/dsedata/lib/cassandra/poc/event_log/poc-event_log-tmp-jb-946-Data.db > (3) ERROR [CompactionExecutor:48] 2014-03-10 10:23:39,248 > CassandraDaemon.java (line 196) Exception in thread > Thread[CompactionExecutor:48,1,main] > FSWriteError in > /mnt/dsedata/lib/cassandra/poc/event_log/poc-event_log-tmp-jb-874-Data.db > The table is a wide-row table created as: > CREATE TABLE event_log ( > time_slice bigint, > distribution_key int, > event_id text, > ... 300 columns ... > PRIMARY KEY ((time_slice, distribution_key), event_id) > ) compaction={'class': 'SizeTieredCompactionStrategy'} AND > compression={'sstable_compression': 'LZ4Compressor'}; > CREATE INDEX EVENT_LOG_2IX ON event_log (event_id); > 'time_slice' represents a 5 minute time-period such as yyyyMMddHHmm where > 'mm' is between 00 and 55 with increments of 5. > The Data files under the 'data' directory got to be very big in a very short > time after the test started. > For example: > 1.5G Mar 10 10:50 > /mnt/dsedata/lib/cassandra/poc/event_log/poc-event_log-jb-968-Data.db > 3.0G Mar 10 11:41 > /mnt/dsedata/lib/cassandra/poc/event_log/poc-event_log-tmp-jb-970-Data.db > Full stack trace: > ERROR [CompactionExecutor:37] 2014-03-10 10:04:30,828 CassandraDaemon.java > (line 196) Exception in thread Thread[CompactionExecutor:37,1,main] > FSWriteError in > /mnt/dsedata/lib/cassandra/poc/event_log/poc-event_log-tmp-jb-946-Data.db > at > org.apache.cassandra.io.compress.CompressedSequentialWriter.close(CompressedSequentialWriter.java:270) > at > org.apache.cassandra.io.sstable.SSTableWriter.close(SSTableWriter.java:356) > at > org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:324) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:204) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) > at > org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:197) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.io.IOException: Input/output error > at sun.nio.ch.FileDispatcherImpl.force0(Native Method) > at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:76) > at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:376) > at > org.apache.cassandra.io.compress.CompressionMetadata$Writer.close(CompressionMetadata.java:366) > at > org.apache.cassandra.io.compress.CompressedSequentialWriter.close(CompressedSequentialWriter.java:266) > ... 13 more -- This message was sent by Atlassian JIRA (v6.2#6252)