Hi Bhaskar, Can you check your limits using 'ulimit -a'? The default is 1024, which needs to be increased if you have not done so already.
Here you will find a list of recommended production settings: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html Mark On Tue, Jul 8, 2014 at 5:30 AM, Bhaskar Singhal <bhaskarsing...@yahoo.com> wrote: > Hi, > > I am using Cassandra 2.0.7 (with default settings and 16GB heap on quad > core ubuntu server with 32gb ram) and trying to ingest 1MB values using > cassandra-stress. It works fine for a while(1600secs) but after ingesting > around 120GB data, I start getting the following error: > Operation [70668] retried 10 times - error inserting key 0070668 > ((TTransportException): java.net.SocketException: Broken pipe) > > The cassandra server is still running but in the system.log I see the > below mentioned errors. > > ERROR [COMMIT-LOG-ALLOCATOR] 2014-07-07 22:39:23,617 CassandraDaemon.java > (line 198) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] > java.lang.NoClassDefFoundError: > org/apache/cassandra/db/commitlog/CommitLog$4 > at > org.apache.cassandra.db.commitlog.CommitLog.handleCommitError(CommitLog.java:374) > at > org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:116) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.lang.ClassNotFoundException: > org.apache.cassandra.db.commitlog.CommitLog$4 > at java.net.URLClassLoader$1.run(URLClassLoader.java:363) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > ... 4 more > Caused by: java.io.FileNotFoundException: > /path/2.0.7/cassandra/build/classes/main/org/apache/cassandra/db/commitlog/CommitLog$4.class > (Too many open files) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.<init>(FileInputStream.java:146) > at > sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1086) > at sun.misc.Resource.cachedInputStream(Resource.java:77) > at sun.misc.Resource.getByteBuffer(Resource.java:160) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:436) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > ... 10 more > ERROR [FlushWriter:7] 2014-07-07 22:39:24,924 CassandraDaemon.java (line > 198) Exception in thread Thread[FlushWriter:7,5,main] > FSWriteError in > /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db > at > org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475) > at > org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212) > at > org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301) > at > org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:417) > at > org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:350) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.io.FileNotFoundException: > /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db > (Too many open files) > at java.io.FileOutputStream.open(Native Method) > at java.io.FileOutputStream.<init>(FileOutputStream.java:221) > at java.io.FileOutputStream.<init>(FileOutputStream.java:110) > at > org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:466) > ... 9 more > > There are around 9685 open files by the Cassandra server process (using > lsof), 3938 commit log segments in /cassandra/commitlog and around 572 > commit log segments deleted during the course of the test. > > I am wondering what is causing Cassandra to open so many files, is the > flushing slow? or something else? > > I tried increasing the flush writers, but that didn't help. > > > Regards, > Bhaskar > > > CREATE KEYSPACE "Keyspace1" WITH replication = { > 'class': 'SimpleStrategy', > 'replication_factor': '1' > }; > > CREATE TABLE "Standard1" ( > key blob, > "C0" blob, > PRIMARY KEY (key) > ) WITH COMPACT STORAGE AND > bloom_filter_fp_chance=0.010000 AND > caching='KEYS_ONLY' AND > comment='' AND > dclocal_read_repair_chance=0.000000 AND > gc_grace_seconds=864000 AND > index_interval=128 AND > read_repair_chance=0.100000 AND > replicate_on_write='true' AND > populate_io_cache_on_flush='false' AND > default_time_to_live=0 AND > speculative_retry='NONE' AND > memtable_flush_period_in_ms=0 AND > compaction={'class': 'SizeTieredCompactionStrategy'} AND > compression={}; > >
