Re: StackOverflowError on high load
This sort of explain this, yes, but what solution can I use? I do see the OPP writes go faster than the RP, so this makes sense that when using the OPP there's higher chance that a host will fall behind with compaction and eventually crash. It's not a nice feature, but hopefully there are mitigations to this. So my question is - what are the mitigations? What should I tell my admin to do in order to prevent this? Telling him increase the directory size 2x isn't going to cut it as the directory just keeps growing and is not bound... I'm also no clear whether CASSANDRA-804 is going to be a real fix. Thanks On Sat, Feb 20, 2010 at 9:36 PM, Jonathan Ellis jbel...@gmail.com wrote: if OPP is configured w/ imbalanced ranges (or less balanced than RP) then that would explain it. OPP is actually slightly faster in terms of raw speed. On Sat, Feb 20, 2010 at 2:31 PM, Ran Tavory ran...@gmail.com wrote: interestingly, I ran the same load but this time with a random partitioner and, although from time to time test2 was a little behind with its compaction task, it did not crash and was able to eventually close the gaps that were opened. Does this make sense? Is there a reason why random partitioner is less likely to be faulty in this scenario? The scenario is of about 1300 writes/sec of small amounts of data to a single CF on a cluster with two nodes and no replication. With the order-preserving-partitioner after a few hours of load the compaction pool is behind on one of the hosts and eventually this host crashes, but with the random partitioner it doesn't crash. thanks On Sat, Feb 20, 2010 at 6:27 AM, Jonathan Ellis jbel...@gmail.com wrote: looks like test1 started gc storming, so test2 treats it as dead and starts doing hinted handoff for it, which increases test2's load, even though test1 is not completely dead yet. On Thu, Feb 18, 2010 at 1:16 AM, Ran Tavory ran...@gmail.com wrote: I found another interesting graph, attached. I looked at the write-count and write-latency of the CF I'm writing to and I see a few interesting things: 1. the host test2 crashed at 18:00 2. At 16:00, after a few hours of load both hosts dropped their write-count. test1 (which did not crash) started slowing down first and then test2 slowed. 3. At 16:00 I start seeing high write-latency on test2 only. This takes about 2h until finally at 18:00 it crashes. Does this help? On Thu, Feb 18, 2010 at 7:44 AM, Ran Tavory ran...@gmail.com wrote: I ran the process again and after a few hours the same node crashed the same way. Now I can tell for sure this is indeed what Jonathan proposed - the data directory needs to be 2x of what it is, but it looks like a design problem, how large to I need to tell my admin to set it then? Here's what I see when the server crashes: $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 46G 47G 50% /outbrain/cassandra/data The directory is 97G and when the host crashes it's at 50% use. I'm also monitoring various JMX counters and I see that COMPACTION-POOL PendingTasks grows for a while on this host (not on the other host, btw, which is fine, just this host) and then flats for 3 hours. After 3 hours of flat it crashes. I'm attaching the graph. When I restart cassandra on this host (not changed file allocation size, just restart) it does manage to compact the data files pretty fast, so after a minute I get 12% use, so I wonder what made it crash before that doesn't now? (could be the load that's not running now) $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data The question is what size does the data directory need to be? It's not 2x the size of the data I expect to have (I only have 11G of real data after compaction and the dir is 97G, so it should have been enough). If it's 2x of something dynamic that keeps growing and isn't bound then it'll just grow infinitely, right? What's the bound? Alternatively, what jmx counter thresholds are the best indicators for the crash that's about to happen? Thanks On Wed, Feb 17, 2010 at 9:00 PM, Tatu Saloranta tsalora...@gmail.com wrote: On Wed, Feb 17, 2010 at 6:40 AM, Ran Tavory ran...@gmail.com wrote: If it's the data directory, then I have a pretty big one. Maybe it's something else $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data Perhaps a temporary file? JVM defaults to /tmp, which may be on a smaller (root) partition? -+ Tatu +-
Re: StackOverflowError on high load
On Sun, Feb 21, 2010 at 10:01 AM, Ran Tavory ran...@gmail.com wrote: I'm also no clear whether CASSANDRA-804 is going to be a real fix. There's one way to find out. :P
Re: StackOverflowError on high load
Ran, There are bounds to how large your data directory will grow, relative to the actual data. Please read up on compaction: http://wiki.apache.org/cassandra/MemtableSSTable , and if you have a significant number of deletes occuring, also read http://wiki.apache.org/cassandra/DistributedDeletes The key mitigation is to ensure that minor compactions get a chance to occur regularly. This will happen automatically, but the faster you write data to your nodes, the more behind on compactions they can get. We consider this a bug, and CASSANDRA-685 will be exploring solutions so that your client automatically backs off as a node becomes overloaded. Thanks, Stu -Original Message- From: Ran Tavory ran...@gmail.com Sent: Sunday, February 21, 2010 9:01am To: cassandra-user@incubator.apache.org Subject: Re: StackOverflowError on high load This sort of explain this, yes, but what solution can I use? I do see the OPP writes go faster than the RP, so this makes sense that when using the OPP there's higher chance that a host will fall behind with compaction and eventually crash. It's not a nice feature, but hopefully there are mitigations to this. So my question is - what are the mitigations? What should I tell my admin to do in order to prevent this? Telling him increase the directory size 2x isn't going to cut it as the directory just keeps growing and is not bound... I'm also no clear whether CASSANDRA-804 is going to be a real fix. Thanks On Sat, Feb 20, 2010 at 9:36 PM, Jonathan Ellis jbel...@gmail.com wrote: if OPP is configured w/ imbalanced ranges (or less balanced than RP) then that would explain it. OPP is actually slightly faster in terms of raw speed. On Sat, Feb 20, 2010 at 2:31 PM, Ran Tavory ran...@gmail.com wrote: interestingly, I ran the same load but this time with a random partitioner and, although from time to time test2 was a little behind with its compaction task, it did not crash and was able to eventually close the gaps that were opened. Does this make sense? Is there a reason why random partitioner is less likely to be faulty in this scenario? The scenario is of about 1300 writes/sec of small amounts of data to a single CF on a cluster with two nodes and no replication. With the order-preserving-partitioner after a few hours of load the compaction pool is behind on one of the hosts and eventually this host crashes, but with the random partitioner it doesn't crash. thanks On Sat, Feb 20, 2010 at 6:27 AM, Jonathan Ellis jbel...@gmail.com wrote: looks like test1 started gc storming, so test2 treats it as dead and starts doing hinted handoff for it, which increases test2's load, even though test1 is not completely dead yet. On Thu, Feb 18, 2010 at 1:16 AM, Ran Tavory ran...@gmail.com wrote: I found another interesting graph, attached. I looked at the write-count and write-latency of the CF I'm writing to and I see a few interesting things: 1. the host test2 crashed at 18:00 2. At 16:00, after a few hours of load both hosts dropped their write-count. test1 (which did not crash) started slowing down first and then test2 slowed. 3. At 16:00 I start seeing high write-latency on test2 only. This takes about 2h until finally at 18:00 it crashes. Does this help? On Thu, Feb 18, 2010 at 7:44 AM, Ran Tavory ran...@gmail.com wrote: I ran the process again and after a few hours the same node crashed the same way. Now I can tell for sure this is indeed what Jonathan proposed - the data directory needs to be 2x of what it is, but it looks like a design problem, how large to I need to tell my admin to set it then? Here's what I see when the server crashes: $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 46G 47G 50% /outbrain/cassandra/data The directory is 97G and when the host crashes it's at 50% use. I'm also monitoring various JMX counters and I see that COMPACTION-POOL PendingTasks grows for a while on this host (not on the other host, btw, which is fine, just this host) and then flats for 3 hours. After 3 hours of flat it crashes. I'm attaching the graph. When I restart cassandra on this host (not changed file allocation size, just restart) it does manage to compact the data files pretty fast, so after a minute I get 12% use, so I wonder what made it crash before that doesn't now? (could be the load that's not running now) $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data The question is what size does the data directory need to be? It's not 2x the size of the data I expect to have (I only have 11G of real data after compaction and the dir is 97G, so
Re: StackOverflowError on high load
interestingly, I ran the same load but this time with a random partitioner and, although from time to time test2 was a little behind with its compaction task, it did not crash and was able to eventually close the gaps that were opened. Does this make sense? Is there a reason why random partitioner is less likely to be faulty in this scenario? The scenario is of about 1300 writes/sec of small amounts of data to a single CF on a cluster with two nodes and no replication. With the order-preserving-partitioner after a few hours of load the compaction pool is behind on one of the hosts and eventually this host crashes, but with the random partitioner it doesn't crash. thanks On Sat, Feb 20, 2010 at 6:27 AM, Jonathan Ellis jbel...@gmail.com wrote: looks like test1 started gc storming, so test2 treats it as dead and starts doing hinted handoff for it, which increases test2's load, even though test1 is not completely dead yet. On Thu, Feb 18, 2010 at 1:16 AM, Ran Tavory ran...@gmail.com wrote: I found another interesting graph, attached. I looked at the write-count and write-latency of the CF I'm writing to and I see a few interesting things: 1. the host test2 crashed at 18:00 2. At 16:00, after a few hours of load both hosts dropped their write-count. test1 (which did not crash) started slowing down first and then test2 slowed. 3. At 16:00 I start seeing high write-latency on test2 only. This takes about 2h until finally at 18:00 it crashes. Does this help? On Thu, Feb 18, 2010 at 7:44 AM, Ran Tavory ran...@gmail.com wrote: I ran the process again and after a few hours the same node crashed the same way. Now I can tell for sure this is indeed what Jonathan proposed - the data directory needs to be 2x of what it is, but it looks like a design problem, how large to I need to tell my admin to set it then? Here's what I see when the server crashes: $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 46G 47G 50% /outbrain/cassandra/data The directory is 97G and when the host crashes it's at 50% use. I'm also monitoring various JMX counters and I see that COMPACTION-POOL PendingTasks grows for a while on this host (not on the other host, btw, which is fine, just this host) and then flats for 3 hours. After 3 hours of flat it crashes. I'm attaching the graph. When I restart cassandra on this host (not changed file allocation size, just restart) it does manage to compact the data files pretty fast, so after a minute I get 12% use, so I wonder what made it crash before that doesn't now? (could be the load that's not running now) $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data The question is what size does the data directory need to be? It's not 2x the size of the data I expect to have (I only have 11G of real data after compaction and the dir is 97G, so it should have been enough). If it's 2x of something dynamic that keeps growing and isn't bound then it'll just grow infinitely, right? What's the bound? Alternatively, what jmx counter thresholds are the best indicators for the crash that's about to happen? Thanks On Wed, Feb 17, 2010 at 9:00 PM, Tatu Saloranta tsalora...@gmail.com wrote: On Wed, Feb 17, 2010 at 6:40 AM, Ran Tavory ran...@gmail.com wrote: If it's the data directory, then I have a pretty big one. Maybe it's something else $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data Perhaps a temporary file? JVM defaults to /tmp, which may be on a smaller (root) partition? -+ Tatu +-
Re: StackOverflowError on high load
https://issues.apache.org/jira/browse/CASSANDRA-804 should have fixed this in trunk / 0.6. or at least log more about what is going on so we can fix it better. :) On Thu, Feb 18, 2010 at 12:44 AM, Ran Tavory ran...@gmail.com wrote: I ran the process again and after a few hours the same node crashed the same way. Now I can tell for sure this is indeed what Jonathan proposed - the data directory needs to be 2x of what it is, but it looks like a design problem, how large to I need to tell my admin to set it then? Here's what I see when the server crashes: $ df -h /outbrain/cassandra/data/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 46G 47G 50% /outbrain/cassandra/data The directory is 97G and when the host crashes it's at 50% use. I'm also monitoring various JMX counters and I see that COMPACTION-POOL PendingTasks grows for a while on this host (not on the other host, btw, which is fine, just this host) and then flats for 3 hours. After 3 hours of flat it crashes. I'm attaching the graph. When I restart cassandra on this host (not changed file allocation size, just restart) it does manage to compact the data files pretty fast, so after a minute I get 12% use, so I wonder what made it crash before that doesn't now? (could be the load that's not running now) $ df -h /outbrain/cassandra/data/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data The question is what size does the data directory need to be? It's not 2x the size of the data I expect to have (I only have 11G of real data after compaction and the dir is 97G, so it should have been enough). If it's 2x of something dynamic that keeps growing and isn't bound then it'll just grow infinitely, right? What's the bound? Alternatively, what jmx counter thresholds are the best indicators for the crash that's about to happen? Thanks On Wed, Feb 17, 2010 at 9:00 PM, Tatu Saloranta tsalora...@gmail.com wrote: On Wed, Feb 17, 2010 at 6:40 AM, Ran Tavory ran...@gmail.com wrote: If it's the data directory, then I have a pretty big one. Maybe it's something else $ df -h /outbrain/cassandra/data/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data Perhaps a temporary file? JVM defaults to /tmp, which may be on a smaller (root) partition? -+ Tatu +-
StackOverflowError on high load
I'm running some high load writes on a pair of cassandra hosts using an OrderPresenrvingPartitioner and ran into the following error after which one of the hosts killed itself. Has anyone seen it and can advice? (cassandra v0.5.0) ERROR [HINTED-HANDOFF-POOL:1] 2010-02-17 04:50:09,602 CassandraDaemon.java (line 71) Fatal exception in thread Thread[HINTED-HANDOFF-POOL:1,5,main] java.lang.StackOverflowError at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:341) at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:447) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:544) at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:240) at java.lang.StringCoding.encode(StringCoding.java:272) at java.lang.String.getBytes(String.java:947) at java.io.UnixFileSystem.getSpace(Native Method) at java.io.File.getUsableSpace(File.java:1660) at org.apache.cassandra.config.DatabaseDescriptor.getDataFileLocationForTable(DatabaseDescriptor.java:891) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:876) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) ... at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) INFO [ROW-MUTATION-STAGE:28] 2010-02-17 04:50:53,230 ColumnFamilyStore.java (line 393) DocumentMapping has reached its threshold; switching in a fresh Memtable INFO [ROW-MUTATION-STAGE:28] 2010-02-17 04:50:53,230 ColumnFamilyStore.java (line 1035) Enqueuing flush of Memtable(DocumentMapping)@122980220 INFO [FLUSH-SORTER-POOL:1] 2010-02-17 04:50:53,230 Memtable.java (line 183) Sorting Memtable(DocumentMapping)@122980220 INFO [FLUSH-WRITER-POOL:1] 2010-02-17 04:50:53,386 Memtable.java (line 192) Writing Memtable(DocumentMapping)@122980220 ERROR [FLUSH-WRITER-POOL:1] 2010-02-17 04:50:54,010 DebuggableThreadPoolExecutor.java (line 162) Error in executor futuretask java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: No space left on device at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:154) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.RuntimeException: java.io.IOException: No space left on device at org.apache.cassandra.db.ColumnFamilyStore$3$1.run(ColumnFamilyStore.java:1060) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ... 2 more Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.write(Native Method) at java.io.DataOutputStream.writeInt(DataOutputStream.java:180) at org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilter.java:158) at org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilter.java:153) at org.apache.cassandra.io.SSTableWriter.closeAndOpenReader(SSTableWriter.java:123) at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:207) at org.apache.cassandra.db.ColumnFamilyStore$3$1.run(ColumnFamilyStore.java:1056) ... 6 more
Re: StackOverflowError on high load
I think that you have not enough room for your data. run df -h to see that one of your discs is full 2010/2/17 Ran Tavory ran...@gmail.com I'm running some high load writes on a pair of cassandra hosts using an OrderPresenrvingPartitioner and ran into the following error after which one of the hosts killed itself. Has anyone seen it and can advice? (cassandra v0.5.0) ERROR [HINTED-HANDOFF-POOL:1] 2010-02-17 04:50:09,602 CassandraDaemon.java (line 71) Fatal exception in thread Thread[HINTED-HANDOFF-POOL:1,5,main] java.lang.StackOverflowError at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:341) at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:447) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:544) at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:240) at java.lang.StringCoding.encode(StringCoding.java:272) at java.lang.String.getBytes(String.java:947) at java.io.UnixFileSystem.getSpace(Native Method) at java.io.File.getUsableSpace(File.java:1660) at org.apache.cassandra.config.DatabaseDescriptor.getDataFileLocationForTable(DatabaseDescriptor.java:891) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:876) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) ... at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) INFO [ROW-MUTATION-STAGE:28] 2010-02-17 04:50:53,230 ColumnFamilyStore.java (line 393) DocumentMapping has reached its threshold; switching in a fresh Memtable INFO [ROW-MUTATION-STAGE:28] 2010-02-17 04:50:53,230 ColumnFamilyStore.java (line 1035) Enqueuing flush of Memtable(DocumentMapping)@122980220 INFO [FLUSH-SORTER-POOL:1] 2010-02-17 04:50:53,230 Memtable.java (line 183) Sorting Memtable(DocumentMapping)@122980220 INFO [FLUSH-WRITER-POOL:1] 2010-02-17 04:50:53,386 Memtable.java (line 192) Writing Memtable(DocumentMapping)@122980220 ERROR [FLUSH-WRITER-POOL:1] 2010-02-17 04:50:54,010 DebuggableThreadPoolExecutor.java (line 162) Error in executor futuretask java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: No space left on device at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:154) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.RuntimeException: java.io.IOException: No space left on device at org.apache.cassandra.db.ColumnFamilyStore$3$1.run(ColumnFamilyStore.java:1060) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ... 2 more Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.write(Native Method) at java.io.DataOutputStream.writeInt(DataOutputStream.java:180) at org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilter.java:158) at org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilter.java:153) at org.apache.cassandra.io.SSTableWriter.closeAndOpenReader(SSTableWriter.java:123) at org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:207) at org.apache.cassandra.db.ColumnFamilyStore$3$1.run(ColumnFamilyStore.java:1056) ... 6 more
Re: StackOverflowError on high load
you temporarily need up to 2x your current space used to perform compactions. disk too full is almost certainly actually the problem. created https://issues.apache.org/jira/browse/CASSANDRA-804 to fix this. On Wed, Feb 17, 2010 at 5:59 AM, Ran Tavory ran...@gmail.com wrote: no, that's not it, disk isn't full. After restarting the server I can write again. Still, however, this error is troubling... On Wed, Feb 17, 2010 at 12:24 PM, ruslan usifov ruslan.usi...@gmail.com wrote: I think that you have not enough room for your data. run df -h to see that one of your discs is full 2010/2/17 Ran Tavory ran...@gmail.com I'm running some high load writes on a pair of cassandra hosts using an OrderPresenrvingPartitioner and ran into the following error after which one of the hosts killed itself. Has anyone seen it and can advice? (cassandra v0.5.0) ERROR [HINTED-HANDOFF-POOL:1] 2010-02-17 04:50:09,602 CassandraDaemon.java (line 71) Fatal exception in thread Thread[HINTED-HANDOFF-POOL:1,5,main] java.lang.StackOverflowError at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:341) at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:447) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:544) at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:240) at java.lang.StringCoding.encode(StringCoding.java:272) at java.lang.String.getBytes(String.java:947) at java.io.UnixFileSystem.getSpace(Native Method) at java.io.File.getUsableSpace(File.java:1660) at org.apache.cassandra.config.DatabaseDescriptor.getDataFileLocationForTable(DatabaseDescriptor.java:891) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:876) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) ... at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) INFO [ROW-MUTATION-STAGE:28] 2010-02-17 04:50:53,230 ColumnFamilyStore.java (line 393) DocumentMapping has reached its threshold; switching in a fresh Memtable INFO [ROW-MUTATION-STAGE:28] 2010-02-17 04:50:53,230 ColumnFamilyStore.java (line 1035) Enqueuing flush of Memtable(DocumentMapping)@122980220 INFO [FLUSH-SORTER-POOL:1] 2010-02-17 04:50:53,230 Memtable.java (line 183) Sorting Memtable(DocumentMapping)@122980220 INFO [FLUSH-WRITER-POOL:1] 2010-02-17 04:50:53,386 Memtable.java (line 192) Writing Memtable(DocumentMapping)@122980220 ERROR [FLUSH-WRITER-POOL:1] 2010-02-17 04:50:54,010 DebuggableThreadPoolExecutor.java (line 162) Error in executor futuretask java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: No space left on device at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:154) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.RuntimeException: java.io.IOException: No space left on device at org.apache.cassandra.db.ColumnFamilyStore$3$1.run(ColumnFamilyStore.java:1060) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ... 2 more Caused by: java.io.IOException: No space left on device at java.io.FileOutputStream.write(Native Method) at java.io.DataOutputStream.writeInt(DataOutputStream.java:180) at org.apache.cassandra.utils.BloomFilterSerializer.serialize(BloomFilter.java:158) at
Re: StackOverflowError on high load
Are we talking about the CommitLogDirectory that needs to be up 2x? So it needs to be 2x of what? Did I miss this in the config file somewhere? On Wed, Feb 17, 2010 at 3:52 PM, Jonathan Ellis jbel...@gmail.com wrote: you temporarily need up to 2x your current space used to perform compactions. disk too full is almost certainly actually the problem. created https://issues.apache.org/jira/browse/CASSANDRA-804 to fix this. On Wed, Feb 17, 2010 at 5:59 AM, Ran Tavory ran...@gmail.com wrote: no, that's not it, disk isn't full. After restarting the server I can write again. Still, however, this error is troubling... On Wed, Feb 17, 2010 at 12:24 PM, ruslan usifov ruslan.usi...@gmail.com wrote: I think that you have not enough room for your data. run df -h to see that one of your discs is full 2010/2/17 Ran Tavory ran...@gmail.com I'm running some high load writes on a pair of cassandra hosts using an OrderPresenrvingPartitioner and ran into the following error after which one of the hosts killed itself. Has anyone seen it and can advice? (cassandra v0.5.0) ERROR [HINTED-HANDOFF-POOL:1] 2010-02-17 04:50:09,602 CassandraDaemon.java (line 71) Fatal exception in thread Thread[HINTED-HANDOFF-POOL:1,5,main] java.lang.StackOverflowError at sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(UTF_8.java:341) at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:447) at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:544) at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:240) at java.lang.StringCoding.encode(StringCoding.java:272) at java.lang.String.getBytes(String.java:947) at java.io.UnixFileSystem.getSpace(Native Method) at java.io.File.getUsableSpace(File.java:1660) at org.apache.cassandra.config.DatabaseDescriptor.getDataFileLocationForTable(DatabaseDescriptor.java:891) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:876) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) ... at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) at org.apache.cassandra.db.ColumnFamilyStore.doFileCompaction(ColumnFamilyStore.java:884) INFO [ROW-MUTATION-STAGE:28] 2010-02-17 04:50:53,230 ColumnFamilyStore.java (line 393) DocumentMapping has reached its threshold; switching in a fresh Memtable INFO [ROW-MUTATION-STAGE:28] 2010-02-17 04:50:53,230 ColumnFamilyStore.java (line 1035) Enqueuing flush of Memtable(DocumentMapping)@122980220 INFO [FLUSH-SORTER-POOL:1] 2010-02-17 04:50:53,230 Memtable.java (line 183) Sorting Memtable(DocumentMapping)@122980220 INFO [FLUSH-WRITER-POOL:1] 2010-02-17 04:50:53,386 Memtable.java (line 192) Writing Memtable(DocumentMapping)@122980220 ERROR [FLUSH-WRITER-POOL:1] 2010-02-17 04:50:54,010 DebuggableThreadPoolExecutor.java (line 162) Error in executor futuretask java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: No space left on device at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) at java.util.concurrent.FutureTask.get(FutureTask.java:83) at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:154) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.RuntimeException: java.io.IOException: No space left on device at org.apache.cassandra.db.ColumnFamilyStore$3$1.run(ColumnFamilyStore.java:1060) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ... 2 more
Re: StackOverflowError on high load
On Wed, Feb 17, 2010 at 8:19 AM, Ran Tavory ran...@gmail.com wrote: Are we talking about the CommitLogDirectory that needs to be up 2x? no. data, not commitlog. So it needs to be 2x of what? Did I miss this in the config file somewhere? current space used, in worst case.
Re: StackOverflowError on high load
On Wed, Feb 17, 2010 at 6:40 AM, Ran Tavory ran...@gmail.com wrote: If it's the data directory, then I have a pretty big one. Maybe it's something else $ df -h /outbrain/cassandra/data/ Filesystem Size Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data Perhaps a temporary file? JVM defaults to /tmp, which may be on a smaller (root) partition? -+ Tatu +-
Re: StackOverflowError on high load
I ran the process again and after a few hours the same node crashed the same way. Now I can tell for sure this is indeed what Jonathan proposed - the data directory needs to be 2x of what it is, but it looks like a design problem, how large to I need to tell my admin to set it then? Here's what I see when the server crashes: $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 46G 47G 50% /outbrain/cassandra/data The directory is 97G and when the host crashes it's at 50% use. I'm also monitoring various JMX counters and I see that COMPACTION-POOL PendingTasks grows for a while on this host (not on the other host, btw, which is fine, just this host) and then flats for 3 hours. After 3 hours of flat it crashes. I'm attaching the graph. When I restart cassandra on this host (not changed file allocation size, just restart) it does manage to compact the data files pretty fast, so after a minute I get 12% use, so I wonder what made it crash before that doesn't now? (could be the load that's not running now) $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data The question is what size does the data directory need to be? It's not 2x the size of the data I expect to have (I only have 11G of real data after compaction and the dir is 97G, so it should have been enough). If it's 2x of something dynamic that keeps growing and isn't bound then it'll just grow infinitely, right? What's the bound? Alternatively, what jmx counter thresholds are the best indicators for the crash that's about to happen? Thanks On Wed, Feb 17, 2010 at 9:00 PM, Tatu Saloranta tsalora...@gmail.comwrote: On Wed, Feb 17, 2010 at 6:40 AM, Ran Tavory ran...@gmail.com wrote: If it's the data directory, then I have a pretty big one. Maybe it's something else $ df -h /outbrain/cassandra/data/ FilesystemSize Used Avail Use% Mounted on /dev/mapper/cassandra-data 97G 11G 82G 12% /outbrain/cassandra/data Perhaps a temporary file? JVM defaults to /tmp, which may be on a smaller (root) partition? -+ Tatu +- attachment: Zenoss_ test2.nydc1.outbrain.com.png
