One big downside about major compaction is that (depending on your cassandra version) the bloom filters size is pre-calculated. Thus cassandra needs enough heap for your existing 33 k+ sstables and the new large compacted one. In the past this happened to us when the compaction thread got hung up, and the sstables grew. When this happens I delete the data directory and jump in a fresh node. The time to recover from that sstable build up can be huge even with multi-threaded compaction.
On Thu, Feb 27, 2014 at 2:09 PM, Nish garg <pipeli...@gmail.com> wrote: > I am having OOM during major compaction on one of the column family where > there are lot of SStables (33000) to be compacted. Is there any other way > for them to be compacted? Any help will be really appreciated. > > Here are the details > > /opt/cassandra/current/bin/nodetool -h us1emscsm-01 compact tomcat > sessions > Error occurred during compaction > java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java > heap space > at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > at java.util.concurrent.FutureTask.get(FutureTask.java:83) > at > org.apache.cassandra.db.compaction.CompactionManager.performMaximal(CompactionManager.java:334) > at > org.apache.cassandra.db.ColumnFamilyStore.forceMajorCompaction(ColumnFamilyStore.java:1691) > at > org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2168) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) > at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) > at > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) > at > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) > at > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) > at > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) > at > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) > at sun.rmi.transport.Transport$1.run(Transport.java:159) > at java.security.AccessController.doPrivileged(Native Method) > at sun.rmi.transport.Transport.serviceCall(Transport.java:155) > at > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.OutOfMemoryError: Java heap space > at > org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:77) > at > org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75) > at > org.apache.cassandra.io.compress.CompressedThrottledReader.<init>(CompressedThrottledReader.java:38) > at > org.apache.cassandra.io.compress.CompressedThrottledReader.open(CompressedThrottledReader.java:52) > at > org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1212) > at > org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:54) > at > org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1032) > at > org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableReader.java:1044) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:157) > at > org.apache.cassandra.db.compaction.AbstractCompactionStrategy.getScanners(AbstractCompactionStrategy.java:163) > at > org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:117) > at > org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) > at > org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:355) > at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > ... 3 more > > > Also > > > /opt/cassandra/current/bin/nodetool -h us1emscsm-01 > getcompactionthreshold tomcat sessions > Current compaction thresholds for tomcat/sessions: > min = 4, max = 32 > > > /opt/cassandra/current/bin/nodetool -h us1emscsm-01 > getcompactionthroughput tomcat sessions > Current compaction throughput: 16 MB/s > > > > /opt/cassandra/current/bin/nodetool -h us1emscsm-01 compactionstats > pending tasks: 167 > Active compaction remaining time : n/a > > >
