I upgraded to 0.6.1 and was able to bring up all the nodes and make
queries.
After adding some new data, the java vm ran out of memory on three of
the nodes. Cassandra continues to run for about 20 minutes before it
exits completely:
DEBUG [ROW-MUTATION-STAGE:2] 2010-04-30 16:02:27,298
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='5f1c96b3', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7...@1272668502,])])
applied. Sending response to 749432@/10.1.29.20
DEBUG [ROW-MUTATION-STAGE:19] 2010-04-30 16:02:27,298
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='4dfdfc26', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272668513,])])
applied. Sending response to 1744234@/10.1.29.18
DEBUG [ROW-MUTATION-STAGE:91] 2010-04-30 16:02:27,449
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='5efd19a7', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272668511,])])
applied. Sending response to 969767@/10.1.29.22
ERROR [COMPACTION-POOL:1] 2010-04-30 16:02:27,295
DebuggableThreadPoolExecutor.java (line 94) Error in executor
futuretask
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
at java.util.concurrent.FutureTask$Sync.innerGet(Unknown
Source) at
java.util.concurrent.FutureTask.get(Unknown
Source) at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.jav
a:86)
at
org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.io.ByteArrayOutputStream.write(Unknown Source)
at java.io.DataOutputStream.write(Unknown Source)
at java.io.FilterOutputStream.write(Unknown Source)
at
org.apache.cassandra.db.ColumnSerializer.writeName(ColumnSerializer.java:39)
at
org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:61)
at
org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:29)
at
org.apache.cassandra.db.ColumnFamilySerializer.serializeForSSTable(ColumnFamilySerializer.java:87)
at
org.apache.cassandra.db.ColumnFamilySerializer.serializeWithIndexes(ColumnFamilySerializer.java:99)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:131)
at
org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41)
at
org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
at
org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at
org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at
org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:299)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:102)
at
org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:83)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
... 3 more
DEBUG [ROW-MUTATION-STAGE:64] 2010-04-30 16:02:27,247
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='298dff6c',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7...@1272668521,])])
DEBUG [ROW-MUTATION-STAGE:20] 2010-04-30 16:02:27,247
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='5ce38012',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272668520,])])
DEBUG [WRITE-/10.1.29.12] 2010-04-30 16:02:27,247
OutboundTcpConnection.java (line 142) attempting to connect to /10.1.29.12
.
.
.
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='29c4eeeb',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669788,])])
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,235
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='29c4eeeb', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669788,])])
applied. Sending response to 756622@/10.1.29.20
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='5f25be67',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669788,])])
DEBUG [ROW-MUTATION-STAGE:22] 2010-04-30 16:33:50,236
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='5f25be67', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669788,])])
applied. Sending response to 756627@/10.1.29.20
DEBUG [ROW-MUTATION-STAGE:34] 2010-04-30 16:24:01,230
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='51f2f208',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7...@1272669785,])])
DEBUG [ROW-MUTATION-STAGE:98] 2010-04-30 16:24:01,230
RowMutationVerbHandler.java (line 73) RowMutation(table='Tracking',
key='29ed2f93', modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:7...@1272669778,])])
applied. Sending response to 1017012@/10.1.29.16
DEBUG [ROW-MUTATION-STAGE:8] 2010-04-30 16:24:01,230
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='3d40ecfc',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669781,])])
DEBUG [ROW-MUTATION-STAGE:67] 2010-04-30 16:24:01,228
RowMutationVerbHandler.java (line 47) Applying
RowMutation(table='Tracking', key='4d1e2910',
modifications=[ColumnFamily(IpTag
[323031302d30342d33302c73696e6b686f6c655f636f6d62696e65642c646f776e61647570:false:8...@1272669780,])])
On 04/26/2010 06:48 PM, Jonathan Ellis wrote:
> 0.5 has a bug that allows it to OOM itself from replaying the log too
> fast. You should upgrade to 0.6.1.
>
>
