Jacek Furmankiewicz created CASSANDRA-7303:
----------------------------------------------
Summary: OutOfMemoryError during prolonged batch processing
Key: CASSANDRA-7303
URL: https://issues.apache.org/jira/browse/CASSANDRA-7303
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Server: RedHat 6, 64-bit, Oracle JDK 7, Cassandra 2.0.6
Client: Java 7, Astyanax
Reporter: Jacek Furmankiewicz
We have a prolonged batch processing job.
It writes a lot of records, every batch mutation creates probably on average
300-500 columns per row key (with many disparate row keys).
It works fine but within a few hours we get error like this:
ERROR [Thrift:15] 2014-05-24 14:16:20,192 CassandraDaemon.java (line |
|196) Except |
|ion in thread Thread[Thrift:15,5,main] |
|java.lang.OutOfMemoryError: Requested array size exceeds VM limit |
|at java.util.Arrays.copyOf(Arrays.java:2271) |
|at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)|
|at java.io.ByteArrayOutputStream.ensureCapacity |
|(ByteArrayOutputStream.ja |
|va:93) |
|at java.io.ByteArrayOutputStream.write |
|(ByteArrayOutputStream.java:140) |
|at org.apache.thrift.transport.TFramedTransport.write |
|(TFramedTransport.j |
|ava:146) |
|at org.apache.thrift.protocol.TBinaryProtocol.writeBinary |
|(TBinaryProtoco |
|l.java:183) |
|at org.apache.cassandra.thrift.Column$ColumnStandardScheme.write |
|(Column. |
|java:678) |
|at org.apache.cassandra.thrift.Column$ColumnStandardScheme.write |
|(Column. |
|java:611) |
|at org.apache.cassandra.thrift.Column.write(Column.java:538) |
|at org.apache.cassandra.thrift.ColumnOrSuperColumn |
|$ColumnOrSuperColumnSt |
|andardScheme.write(ColumnOrSuperColumn.java:673) |
|at org.apache.cassandra.thrift.ColumnOrSuperColumn |
|$ColumnOrSuperColumnSt |
|andardScheme.write(ColumnOrSuperColumn.java:607) |
|at org.apache.cassandra.thrift.ColumnOrSuperColumn.write |
|(ColumnOrSuperCo |
|lumn.java:517) |
|at org.apache.cassandra.thrift.Cassandra$get_slice_result |
|$get_slice_resu |
|ltStandardScheme.write(Cassandra.java:11682) |
|at org.apache.cassandra.thrift.Cassandra$get_slice_result |
|$get_slice_resu |
|ltStandardScheme.write(Cassandra.java:11603) |
|at org.apache.cassandra.thrift.Cassandra
The server already has 16 GB heap, which we hear is the max Cassandra can run
with. The writes are heavily multi-threaded from a single server.
The jist of the issue is that Cassandra should not crash with OOM when under
heavy load. It is OK to slow down, even maybe start throwing operation
timeout exceptions, etc.
But to just crash in the middle of the processing should not be allowed.
is there any internal monitoring of heap usage in Cassandra where it could
detect that it is getting close to the heap limit and start throttling the
incoming requests to avoid this type of error?
Thanks
--
This message was sent by Atlassian JIRA
(v6.2#6252)
