Re: TTransportException (java.net.SocketException: Broken pipe)
I have not seen the issue after changing the commit log segment size to 1024MB. tpstats output: Pool Name Active Pending Completed Blocked All time blocked ReadStage 0 0 0 0 0 RequestResponseStage 0 0 0 0 0 MutationStage 32 40 2526143 0 0 ReadRepairStage 0 0 0 0 0 ReplicateOnWriteStage 0 0 0 0 0 GossipStage 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage 0 0 3 0 0 MemoryMeter 0 0 24752 0 0 MemtablePostFlusher 1 19 12939 0 0 FlushWriter 6 10 12442 1 2940 MiscStage 0 0 0 0 0 PendingRangeCalculator 0 0 1 0 0 commitlog_archiver 0 0 0 0 0 InternalResponseStage 0 0 0 0 0 HintedHandoff 0 0 0 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 0 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0 On Saturday, 19 July 2014 1:32 AM, Robert Coli rc...@eventbrite.com wrote: On Mon, Jul 7, 2014 at 9:30 PM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: I am using Cassandra 2.0.7 (with default settings and 16GB heap on quad core ubuntu server with 32gb ram) 16GB of heap will lead to significant GC pauses, and probably will not improve total performance versus 8gb heap. I continue to maintain that your problem is that you are writing faster than you can flush. Paste the output of nodetool tpstats? =Rob
Re: TTransportException (java.net.SocketException: Broken pipe)
On Mon, Jul 21, 2014 at 8:07 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: I have not seen the issue after changing the commit log segment size to 1024MB. Yes... your insanely over-huge commitlog will be contained in fewer files if you increase the size of segments that will not make it any less of an insanely over-huge commitlog which indicates systemic failure in your application's use of Cassandra. Congratulations on masking your actual issue with your configuration change. Pool NameActive Pending Completed Blocked All time blocked FlushWriter 610 12442 1 2940 1/4 attempts to flush blocked waiting for resources, and you have 6 actives flushes and 10 pending, because YOU'RE WRITING TOO FAST. As a meta aside, I am unlikely to respond to further questions of yours which do not engage with what I have now told you three or four times, that YOU'RE WRITING TOO FAST. =Rob
Re: TTransportException (java.net.SocketException: Broken pipe)
On Mon, Jul 7, 2014 at 9:30 PM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: I am using Cassandra 2.0.7 (with default settings and 16GB heap on quad core ubuntu server with 32gb ram) 16GB of heap will lead to significant GC pauses, and probably will not improve total performance versus 8gb heap. I continue to maintain that your problem is that you are writing faster than you can flush. Paste the output of nodetool tpstats? =Rob
Re: TTransportException (java.net.SocketException: Broken pipe)
Are you still seeing the same exceptions about too many open files? On Thu, Jul 17, 2014 at 6:28 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: Even after changing ulimits and moving to the recommended production settings, we are still seeing the same issue. root@lnx148-76:~# cat /proc/17663/limits Limit Soft Limit Hard Limit Units Max cpu time unlimitedunlimitedseconds Max file size unlimitedunlimitedbytes Max data size unlimitedunlimitedbytes Max stack size8388608 unlimitedbytes Max core file size0unlimitedbytes Max resident set unlimitedunlimitedbytes Max processes 256502 256502 processes Max open files4096 4096 files Max locked memory 6553665536bytes Max address space unlimitedunlimitedbytes Max file locksunlimitedunlimitedlocks Max pending signals 256502 256502 signals Max msgqueue size 819200 819200 bytes Max nice priority 00 Max realtime priority 00 Max realtime timeout unlimitedunlimitedus Regards, Bhaskar On Thursday, 10 July 2014 12:09 AM, Robert Coli rc...@eventbrite.com wrote: On Tue, Jul 8, 2014 at 10:17 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: But I am wondering why does Cassandra need to keep 3000+ commit log segment files open? Because you are writing faster than you can flush to disk. =Rob
Re: TTransportException (java.net.SocketException: Broken pipe)
Yes, I am. lsof lists around 9000 open file handles.. and there were around 3000 commitlog segments. On Thursday, 17 July 2014 1:24 PM, Benedict Elliott Smith belliottsm...@datastax.com wrote: Are you still seeing the same exceptions about too many open files? On Thu, Jul 17, 2014 at 6:28 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: Even after changing ulimits and moving to the recommended production settings, we are still seeing the same issue. root@lnx148-76:~# cat /proc/17663/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 256502 256502 processes Max open files 4096 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 256502 256502 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Regards, Bhaskar On Thursday, 10 July 2014 12:09 AM, Robert Coli rc...@eventbrite.com wrote: On Tue, Jul 8, 2014 at 10:17 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: But I am wondering why does Cassandra need to keep 3000+ commit log segment files open? Because you are writing faster than you can flush to disk. =Rob
Re: TTransportException (java.net.SocketException: Broken pipe)
Even after changing ulimits and moving to the recommended production settings, we are still seeing the same issue. root@lnx148-76:~# cat /proc/17663/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes 256502 256502 processes Max open files 4096 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 256502 256502 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us Regards, Bhaskar On Thursday, 10 July 2014 12:09 AM, Robert Coli rc...@eventbrite.com wrote: On Tue, Jul 8, 2014 at 10:17 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: But I am wondering why does Cassandra need to keep 3000+ commit log segment files open? Because you are writing faster than you can flush to disk. =Rob
Re: TTransportException (java.net.SocketException: Broken pipe)
We have these precise settings but are still seeing the broken pipe exception in our gc logs. Any clues? Sent from my iPhone On Jul 8, 2014, at 1:17 PM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: Thanks Mark. Yes the 1024 is the limit. I haven't changed it as per the recommended production settings. But I am wondering why does Cassandra need to keep 3000+ commit log segment files open? Regards, Bhaskar On Tuesday, 8 July 2014 1:50 PM, Mark Reddy mark.re...@boxever.com wrote: Hi Bhaskar, Can you check your limits using 'ulimit -a'? The default is 1024, which needs to be increased if you have not done so already. Here you will find a list of recommended production settings: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html Mark On Tue, Jul 8, 2014 at 5:30 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: Hi, I am using Cassandra 2.0.7 (with default settings and 16GB heap on quad core ubuntu server with 32gb ram) and trying to ingest 1MB values using cassandra-stress. It works fine for a while(1600secs) but after ingesting around 120GB data, I start getting the following error: Operation [70668] retried 10 times - error inserting key 0070668 ((TTransportException): java.net.SocketException: Broken pipe) The cassandra server is still running but in the system.log I see the below mentioned errors. ERROR [COMMIT-LOG-ALLOCATOR] 2014-07-07 22:39:23,617 CassandraDaemon.java (line 198) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.lang.NoClassDefFoundError: org/apache/cassandra/db/commitlog/CommitLog$4 at org.apache.cassandra.db.commitlog.CommitLog.handleCommitError(CommitLog.java:374) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:116) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.ClassNotFoundException: org.apache.cassandra.db.commitlog.CommitLog$4 at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 4 more Caused by: java.io.FileNotFoundException: /path/2.0.7/cassandra/build/classes/main/org/apache/cassandra/db/commitlog/CommitLog$4.class (Too many open files) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:146) at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1086) at sun.misc.Resource.cachedInputStream(Resource.java:77) at sun.misc.Resource.getByteBuffer(Resource.java:160) at java.net.URLClassLoader.defineClass(URLClassLoader.java:436) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ... 10 more ERROR [FlushWriter:7] 2014-07-07 22:39:24,924 CassandraDaemon.java (line 198) Exception in thread Thread[FlushWriter:7,5,main] FSWriteError in /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475) at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212) at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301) at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:417) at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:350) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db (Too many open files) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.init(FileOutputStream.java:221) at java.io.FileOutputStream.init(FileOutputStream.java:110) at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:466) ... 9 more There are around 9685 open files by the Cassandra server process (using lsof
Re: TTransportException (java.net.SocketException: Broken pipe)
On Tue, Jul 8, 2014 at 10:17 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: But I am wondering why does Cassandra need to keep 3000+ commit log segment files open? Because you are writing faster than you can flush to disk. =Rob
Re: TTransportException (java.net.SocketException: Broken pipe)
Hi Bhaskar,
Can you check your limits using 'ulimit -a'? The default is 1024, which
needs to be increased if you have not done so already.
Here you will find a list of recommended production settings:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html
Mark
On Tue, Jul 8, 2014 at 5:30 AM, Bhaskar Singhal bhaskarsing...@yahoo.com
wrote:
Hi,
I am using Cassandra 2.0.7 (with default settings and 16GB heap on quad
core ubuntu server with 32gb ram) and trying to ingest 1MB values using
cassandra-stress. It works fine for a while(1600secs) but after ingesting
around 120GB data, I start getting the following error:
Operation [70668] retried 10 times - error inserting key 0070668
((TTransportException): java.net.SocketException: Broken pipe)
The cassandra server is still running but in the system.log I see the
below mentioned errors.
ERROR [COMMIT-LOG-ALLOCATOR] 2014-07-07 22:39:23,617 CassandraDaemon.java
(line 198) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.lang.NoClassDefFoundError:
org/apache/cassandra/db/commitlog/CommitLog$4
at
org.apache.cassandra.db.commitlog.CommitLog.handleCommitError(CommitLog.java:374)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:116)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException:
org.apache.cassandra.db.commitlog.CommitLog$4
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 4 more
Caused by: java.io.FileNotFoundException:
/path/2.0.7/cassandra/build/classes/main/org/apache/cassandra/db/commitlog/CommitLog$4.class
(Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.init(FileInputStream.java:146)
at
sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1086)
at sun.misc.Resource.cachedInputStream(Resource.java:77)
at sun.misc.Resource.getByteBuffer(Resource.java:160)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
... 10 more
ERROR [FlushWriter:7] 2014-07-07 22:39:24,924 CassandraDaemon.java (line
198) Exception in thread Thread[FlushWriter:7,5,main]
FSWriteError in
/cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
at
org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
at
org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
at
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:417)
at
org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:350)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException:
/cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db
(Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:221)
at java.io.FileOutputStream.init(FileOutputStream.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:466)
... 9 more
There are around 9685 open files by the Cassandra server process (using
lsof), 3938 commit log segments in /cassandra/commitlog and around 572
commit log segments deleted during the course of the test.
I am wondering what is causing Cassandra to open so many files, is the
flushing slow? or something else?
I tried increasing the flush writers, but that didn't help.
Regards,
Bhaskar
CREATE KEYSPACE Keyspace1 WITH replication = {
'class': 'SimpleStrategy',
'replication_factor': '1'
};
CREATE TABLE Standard1 (
key blob,
C0 blob,
PRIMARY KEY (key)
) WITH COMPACT STORAGE AND
bloom_filter_fp_chance
Re: TTransportException (java.net.SocketException: Broken pipe)
Thanks Mark. Yes the 1024 is the limit. I haven't changed it as per the recommended production settings. But I am wondering why does Cassandra need to keep 3000+ commit log segment files open? Regards, Bhaskar On Tuesday, 8 July 2014 1:50 PM, Mark Reddy mark.re...@boxever.com wrote: Hi Bhaskar, Can you check your limits using 'ulimit -a'? The default is 1024, which needs to be increased if you have not done so already. Here you will find a list of recommended production settings: http://www.datastax.com/documentation/cassandra/2.0/cassandra/install/installRecommendSettings.html Mark On Tue, Jul 8, 2014 at 5:30 AM, Bhaskar Singhal bhaskarsing...@yahoo.com wrote: Hi, I am using Cassandra 2.0.7 (with default settings and 16GB heap on quad core ubuntu server with 32gb ram) and trying to ingest 1MB values using cassandra-stress. It works fine for a while(1600secs) but after ingesting around 120GB data, I start getting the following error: Operation [70668] retried 10 times - error inserting key 0070668 ((TTransportException): java.net.SocketException: Broken pipe) The cassandra server is still running but in the system.log I see the below mentioned errors. ERROR [COMMIT-LOG-ALLOCATOR] 2014-07-07 22:39:23,617 CassandraDaemon.java (line 198) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.lang.NoClassDefFoundError: org/apache/cassandra/db/commitlog/CommitLog$4 at org.apache.cassandra.db.commitlog.CommitLog.handleCommitError(CommitLog.java:374) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:116) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.ClassNotFoundException: org.apache.cassandra.db.commitlog.CommitLog$4 at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 4 more Caused by: java.io.FileNotFoundException: /path/2.0.7/cassandra/build/classes/main/org/apache/cassandra/db/commitlog/CommitLog$4.class (Too many open files) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.init(FileInputStream.java:146) at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1086) at sun.misc.Resource.cachedInputStream(Resource.java:77) at sun.misc.Resource.getByteBuffer(Resource.java:160) at java.net.URLClassLoader.defineClass(URLClassLoader.java:436) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) ... 10 more ERROR [FlushWriter:7] 2014-07-07 22:39:24,924 CassandraDaemon.java (line 198) Exception in thread Thread[FlushWriter:7,5,main] FSWriteError in /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475) at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212) at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301) at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:417) at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:350) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.io.FileNotFoundException: /cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db (Too many open files) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.init(FileOutputStream.java:221) at java.io.FileOutputStream.init(FileOutputStream.java:110) at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:466) ... 9 more There are around 9685 open files by the Cassandra server process (using lsof), 3938 commit log segments in /cassandra/commitlog and around 572 commit log segments deleted during the course of the test. I am wondering what is causing Cassandra to open so many files, is the flushing slow? or something else? I tried increasing the flush writers, but that didn't help
TTransportException (java.net.SocketException: Broken pipe)
Hi,
I am using Cassandra 2.0.7 (with default settings and 16GB heap on quad core
ubuntu server with 32gb ram) and trying to ingest 1MB values using
cassandra-stress. It works fine for a while(1600secs) but after ingesting
around 120GB data, I start getting the following error:
Operation [70668] retried 10 times - error inserting key 0070668
((TTransportException): java.net.SocketException: Broken pipe)
The cassandra server is still running but in the system.log I see the below
mentioned errors.
ERROR [COMMIT-LOG-ALLOCATOR] 2014-07-07 22:39:23,617 CassandraDaemon.java (line
198) Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.lang.NoClassDefFoundError: org/apache/cassandra/db/commitlog/CommitLog$4
at
org.apache.cassandra.db.commitlog.CommitLog.handleCommitError(CommitLog.java:374)
at
org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:116)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException:
org.apache.cassandra.db.commitlog.CommitLog$4
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 4 more
Caused by: java.io.FileNotFoundException:
/path/2.0.7/cassandra/build/classes/main/org/apache/cassandra/db/commitlog/CommitLog$4.class
(Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.init(FileInputStream.java:146)
at
sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1086)
at sun.misc.Resource.cachedInputStream(Resource.java:77)
at sun.misc.Resource.getByteBuffer(Resource.java:160)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
... 10 more
ERROR [FlushWriter:7] 2014-07-07 22:39:24,924 CassandraDaemon.java (line 198)
Exception in thread Thread[FlushWriter:7,5,main]
FSWriteError in
/cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:475)
at
org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:212)
at
org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:301)
at
org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:417)
at
org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:350)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.io.FileNotFoundException:
/cassandra/data4/Keyspace1/Standard1/Keyspace1-Standard1-tmp-jb-593-Filter.db
(Too many open files)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:221)
at java.io.FileOutputStream.init(FileOutputStream.java:110)
at
org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:466)
... 9 more
There are around 9685 open files by the Cassandra server process (using lsof),
3938 commit log segments in /cassandra/commitlog and around 572 commit log
segments deleted during the course of the test.
I am wondering what is causing Cassandra to open so many files, is the flushing
slow? or something else?
I tried increasing the flush writers, but that didn't help.
Regards,
Bhaskar
CREATE KEYSPACE Keyspace1 WITH replication = {
'class': 'SimpleStrategy',
'replication_factor': '1'
};
CREATE TABLE Standard1 (
key blob,
C0 blob,
PRIMARY KEY (key)
) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='NONE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'SizeTieredCompactionStrategy'} AND
compression={};
