By default Cassandra is set to generate a heap dump on OOM. It can be a bit tricky to figure out what’s going on exactly but it’s the best evidence you can work with.
On Tue, Aug 7, 2018 at 6:30 AM Laszlo Szabo <laszlo.viktor.sz...@gmail.com> wrote: > Hi, > > Thanks for the fast response! > > We are not using any materialized views, but there are several indexes. I > don't have a recent heap dump, and it will be about 24 before I can > generate an interesting one, but most of the memory was allocated to byte > buffers, so not entirely helpful. > > nodetool cfstats is also below. > > I also see a lot of flushing happening, but it seems like there are too > many small allocations to be effective. Here are the messages I see, > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,459 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='gpsmessages') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,459 ColumnFamilyStore.java:915 >> - Enqueuing flush of gpsmessages: 0.000KiB (0%) on-heap, 0.014KiB (0%) >> off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,460 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='user_history') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,461 ColumnFamilyStore.java:915 >> - Enqueuing flush of user_history: 0.000KiB (0%) on-heap, 0.011KiB (0%) >> off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,465 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='tweets') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,465 ColumnFamilyStore.java:915 >> - Enqueuing flush of tweets: 0.000KiB (0%) on-heap, 0.188KiB (0%) off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,470 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='user_history') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,470 ColumnFamilyStore.java:915 >> - Enqueuing flush of user_history: 0.000KiB (0%) on-heap, 0.024KiB (0%) >> off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,470 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='tweets') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,470 ColumnFamilyStore.java:915 >> - Enqueuing flush of tweets: 0.000KiB (0%) on-heap, 0.188KiB (0%) off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,472 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='gpsmessages') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,472 ColumnFamilyStore.java:915 >> - Enqueuing flush of gpsmessages: 0.000KiB (0%) on-heap, 0.013KiB (0%) >> off-heap > > >> > > Stack traces from errors are below. > > >> java.io.IOException: Broken pipe > > at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >> ~[na:1.8.0_181] > > at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >> ~[na:1.8.0_181] > > at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) >> ~[na:1.8.0_181] > > at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[na:1.8.0_181] > > at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) >> ~[na:1.8.0_181] > > at >> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.doFlush(BufferedDataOutputStreamPlus.java:323) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.flush(BufferedDataOutputStreamPlus.java:331) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:409) >> [apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:380) >> [apache-cassandra-3.11.1.jar:3.11.1] > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] > > ERROR [MutationStage-226] 2018-08-06 07:16:08,236 >> JVMStabilityInspector.java:142 - JVM state determined to be unstable. >> Exiting forcefully due to: > > java.lang.OutOfMemoryError: Direct buffer memory > > at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_181] > > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) >> ~[na:1.8.0_181] > > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) >> ~[na:1.8.0_181] > > at >> org.apache.cassandra.utils.memory.SlabAllocator.getRegion(SlabAllocator.java:139) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.utils.memory.SlabAllocator.allocate(SlabAllocator.java:104) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.utils.memory.ContextAllocator.allocate(ContextAllocator.java:57) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.utils.memory.ContextAllocator.clone(ContextAllocator.java:47) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.utils.memory.MemtableBufferAllocator.clone(MemtableBufferAllocator.java:40) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at org.apache.cassandra.db.Memtable.put(Memtable.java:269) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1332) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:618) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at org.apache.cassandra.db.Keyspace.applyFuture(Keyspace.java:425) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:222) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:68) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> ~[na:1.8.0_181] > > at >> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) >> ~[apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) >> [apache-cassandra-3.11.1.jar:3.11.1] > > at >> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) >> [apache-cassandra-3.11.1.jar:3.11.1] > > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181] > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,459 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='gpsmessages') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,459 ColumnFamilyStore.java:915 >> - Enqueuing flush of gpsmessages: 0.000KiB (0%) on-heap, 0.014KiB (0%) >> off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,460 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='user_history') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,461 ColumnFamilyStore.java:915 >> - Enqueuing flush of user_history: 0.000KiB (0%) on-heap, 0.011KiB (0%) >> off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,465 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='tweets') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,465 ColumnFamilyStore.java:915 >> - Enqueuing flush of tweets: 0.000KiB (0%) on-heap, 0.188KiB (0%) off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,470 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='user_history') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,470 ColumnFamilyStore.java:915 >> - Enqueuing flush of user_history: 0.000KiB (0%) on-heap, 0.024KiB (0%) >> off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,470 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='tweets') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,470 ColumnFamilyStore.java:915 >> - Enqueuing flush of tweets: 0.000KiB (0%) on-heap, 0.188KiB (0%) off-heap > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,472 >> ColumnFamilyStore.java:1305 - Flushing largest CFS(Keyspace='userinfo', >> ColumnFamily='gpsmessages') to free up room. Used total: 0.54/0.05, live: >> 0.00/0.00, flushing: 0.40/0.04, this: 0.00/0.00 > > DEBUG [SlabPoolCleaner] 2018-08-06 07:16:08,472 ColumnFamilyStore.java:915 >> - Enqueuing flush of gpsmessages: 0.000KiB (0%) on-heap, 0.013KiB (0%) >> off-heap > > > Total number of tables: 40 > ---------------- > Keyspace : userinfo > Read Count: 143301 > Read Latency: 14.945587623254548 ms. > Write Count: 2754603904 > Write Latency: 0.020883145284324698 ms. > Pending Flushes: 0 > Table (index): > gpsmessages.gpsmessages_addresscount_idxgpsmessages.gpsmessages_addresscount_idx > SSTable count: 9 > Space used (live): 19043463189 > Space used (total): 19043463189 > Space used by snapshots (total): 0 > Off heap memory used (total): 6259448 > SSTable Compression Ratio: 0.3704785164266614 > Number of partitions (estimate): 1025 > Memtable cell count: 309066 > Memtable data size: 13602774 > Memtable off heap memory used: 0 > Memtable switch count: 0 > Local read count: 0 > Local read latency: NaN ms > Local write count: 46025778 > Local write latency: 0.034 ms > Pending flushes: 0 > Percent repaired: 0.0 > Bloom filter false positives: 0 > Bloom filter false ratio: 0.00000 > Bloom filter space used: 2504 > Bloom filter off heap memory used: 2432 > Index summary off heap memory used: 320 > Compression metadata off heap memory used: 6256696 > Compacted partition minimum bytes: 43 > Compacted partition maximum bytes: 44285675122 > Compacted partition mean bytes: 30405277 > Average live cells per slice (last five minutes): NaN > Maximum live cells per slice (last five minutes): 0 > Average tombstones per slice (last five minutes): NaN > Maximum tombstones per slice (last five minutes): 0 > Dropped Mutations: 0 > > Table (index): > gpsmessages.gpsmessages_addresses_idxgpsmessages.gpsmessages_addresses_idx > SSTable count: 18 > Space used (live): 409514565570 > Space used (total): 409514565570 > Space used by snapshots (total): 0 > Off heap memory used (total): 153405673 > SSTable Compression Ratio: 0.4447731157134059 > Number of partitions (estimate): 6013125 > Memtable cell count: 1110334 > Memtable data size: 67480140 > Memtable off heap memory used: 0 > Memtable switch count: 0 > Local read count: 0 > Local read latency: NaN ms > Local write count: 147639252 > Local write latency: 0.015 ms > Pending flushes: 0 > Percent repaired: 0.0 > Bloom filter false positives: 0 > Bloom filter false ratio: 0.00000 > Bloom filter space used: 34175400 > Bloom filter off heap memory used: 34175256 > Index summary off heap memory used: 7432177 > Compression metadata off heap memory used: 111798240 > Compacted partition minimum bytes: 61 > Compacted partition maximum bytes: 322381140 > Compacted partition mean bytes: 36692 > Average live cells per slice (last five minutes): NaN > Maximum live cells per slice (last five minutes): 0 > Average tombstones per slice (last five minutes): NaN > Maximum tombstones per slice (last five minutes): 0 > Dropped Mutations: 0 > > Table (index): > gpsmessages.addressreceivedtime_idxgpsmessages.addressreceivedtime_idx > SSTable count: 10 > Space used (live): 52738155477 > Space used (total): 52738155477 > Space used by snapshots (total): 0 > Off heap memory used (total): 1909362628 > SSTable Compression Ratio: 0.4106961621795128 > Number of partitions (estimate): 1338730016 > Memtable cell count: 308990 > Memtable data size: 13410867 > Memtable off heap memory used: 0 > Memtable switch count: 0 > Local read count: 0 > Local read latency: NaN ms > Local write count: 46012614 > Local write latency: 0.012 ms > Pending flushes: 0 > Percent repaired: 0.0 > Bloom filter false positives: 0 > Bloom filter false ratio: 0.00000 > Bloom filter space used: 1687550888 > Bloom filter off heap memory used: 1687550808 > Index summary off heap memory used: 213249180 > Compression metadata off heap memory used: 8562640 > Compacted partition minimum bytes: 36 > Compacted partition maximum bytes: 2759 > Compacted partition mean bytes: 54 > Average live cells per slice (last five minutes): NaN > Maximum live cells per slice (last five minutes): 0 > Average tombstones per slice (last five minutes): NaN > Maximum tombstones per slice (last five minutes): 0 > Dropped Mutations: 0 > > Table: gpsmessages > SSTable count: 13 > Space used (live): 337974446627 > Space used (total): 337974446627 > Space used by snapshots (total): 0 > Off heap memory used (total): 77833540 > SSTable Compression Ratio: 0.5300637241381126 > Number of partitions (estimate): 22034 > Memtable cell count: 308904 > Memtable data size: 72074512 > Memtable off heap memory used: 0 > Memtable switch count: 110 > Local read count: 0 > Local read latency: NaN ms > Local write count: 45996652 > Local write latency: 0.281 ms > Pending flushes: 0 > Percent repaired: 0.0 > Bloom filter false positives: 0 > Bloom filter false ratio: 0.00000 > Bloom filter space used: 67904 > Bloom filter off heap memory used: 67800 > Index summary off heap memory used: 11756 > Compression metadata off heap memory used: 77753984 > Compacted partition minimum bytes: 73 > Compacted partition maximum bytes: 1155149911 > Compacted partition mean bytes: 13158224 > Average live cells per slice (last five minutes): NaN > Maximum live cells per slice (last five minutes): 0 > Average tombstones per slice (last five minutes): NaN > Maximum tombstones per slice (last five minutes): 0 > Dropped Mutations: 13699 > > Table: user_history > SSTable count: 17 > Space used (live): 116361158882 > Space used (total): 116361158882 > Space used by snapshots (total): 0 > Off heap memory used (total): 29562319 > SSTable Compression Ratio: 0.5683114352331539 > Number of partitions (estimate): 1337206 > Memtable cell count: 773277 > Memtable data size: 40623368 > Memtable off heap memory used: 0 > Memtable switch count: 57 > Local read count: 209 > Local read latency: NaN ms > Local write count: 145853733 > Local write latency: 0.020 ms > Pending flushes: 0 > Percent repaired: 0.0 > Bloom filter false positives: 0 > Bloom filter false ratio: 0.00000 > Bloom filter space used: 3844416 > Bloom filter off heap memory used: 3844280 > Index summary off heap memory used: 800991 > Compression metadata off heap memory used: 24917048 > Compacted partition minimum bytes: 61 > Compacted partition maximum bytes: 464228842 > Compacted partition mean bytes: 72182 > Average live cells per slice (last five minutes): NaN > Maximum live cells per slice (last five minutes): 0 > Average tombstones per slice (last five minutes): NaN > Maximum tombstones per slice (last five minutes): 0 > Dropped Mutations: 66702 > > Table: users > SSTable count: 3 > Space used (live): 89945186 > Space used (total): 89945186 > Space used by snapshots (total): 0 > Off heap memory used (total): 2092053 > SSTable Compression Ratio: 0.5712127629253333 > Number of partitions (estimate): 1365645 > Memtable cell count: 3556 > Memtable data size: 150903 > Memtable off heap memory used: 0 > Memtable switch count: 42 > Local read count: 143087 > Local read latency: 6.094 ms > Local write count: 250971 > Local write latency: 0.024 ms > Pending flushes: 0 > Percent repaired: 0.0 > Bloom filter false positives: 0 > Bloom filter false ratio: 0.00000 > Bloom filter space used: 1709848 > Bloom filter off heap memory used: 1709824 > Index summary off heap memory used: 372125 > Compression metadata off heap memory used: 10104 > Compacted partition minimum bytes: 36 > Compacted partition maximum bytes: 310 > Compacted partition mean bytes: 66 > Average live cells per slice (last five minutes): 1.0 > Maximum live cells per slice (last five minutes): 1 > Average tombstones per slice (last five minutes): 1.0 > Maximum tombstones per slice (last five minutes): 1 > Dropped Mutations: 114 > > Table: tweets > SSTable count: 18 > Space used (live): 1809145656486 > Space used (total): 1809145656486 > Space used by snapshots (total): 0 > Off heap memory used (total): 435915908 > SSTable Compression Ratio: 0.5726200929451171 > Number of partitions (estimate): 26217889 > Memtable cell count: 710146 > Memtable data size: 31793929 > Memtable off heap memory used: 0 > Memtable switch count: 399 > Local read count: 5 > Local read latency: NaN ms > Local write count: 2322829524 > Local write latency: 0.019 ms > Pending flushes: 0 > Percent repaired: 0.0 > Bloom filter false positives: 0 > Bloom filter false ratio: 0.00000 > Bloom filter space used: 35019224 > Bloom filter off heap memory used: 35019080 > Index summary off heap memory used: 16454076 > Compression metadata off heap memory used: 384442752 > Compacted partition minimum bytes: 104 > Compacted partition maximum bytes: 3379391 > Compacted partition mean bytes: 124766 > Average live cells per slice (last five minutes): NaN > Maximum live cells per slice (last five minutes): 0 > Average tombstones per slice (last five minutes): NaN > Maximum tombstones per slice (last five minutes): 0 > Dropped Mutations: 697696 > > ---------------- > > > On Mon, Aug 6, 2018 at 8:57 PM, Jeff Jirsa <jji...@gmail.com> wrote: > >> >> >> Upgrading to 3.11.3 May fix it (there were some memory recycling bugs >> fixed recently), but analyzing the heap will be the best option >> >> If you can print out the heap histogram and stack trace or open a heap >> dump in your kit or visualvm or MAT and show us what’s at the top of the >> reclaimed objects, we may be able to figure out what’s going on >> >> -- >> Jeff Jirsa >> >> >> On Aug 6, 2018, at 5:42 PM, Jeff Jirsa <jji...@gmail.com> wrote: >> >> Are you using materialized views or secondary indices? >> >> -- >> Jeff Jirsa >> >> >> On Aug 6, 2018, at 3:49 PM, Laszlo Szabo <laszlo.viktor.sz...@gmail.com> >> wrote: >> >> Hello All, >> >> I'm having JVM unstable / OOM errors when attempting to auto bootstrap a >> 9th node to an existing 8 node cluster (256 tokens). Each machine has 24 >> cores 148GB RAM and 10TB (2TB used). Under normal operation the 8 nodes >> have JVM memory configured with Xms35G and Xmx35G, and handle 2-4 billion >> inserts per day. There are never updates, deletes, or sparsely populated >> rows. >> >> For the bootstrap node, I've tried memory values from 35GB to 135GB in >> 10GB increments. I've tried using both memtable_allocation_types >> (heap_buffers and offheap_buffers). I've not tried modifying the >> memtable_cleanup_threshold but instead have tried memtable_flush_writers >> from 2 to 8. I've tried memtable_(off)heap_space_in_mb from 20000 to >> 60000. I've tried both CMS and G1 garbage collection with various >> settings. >> >> Typically, after streaming about ~2TB of data, CPU load will hit a >> maximum, and the "nodetool info" heap memory will, over the course of an >> hour, approach the maximum. At that point, CPU load will drop to a single >> thread with minimal activity until the system becomes unstable and >> eventually the OOM error occurs. >> >> Excerpt of the system log is below, and what I consistently see is the >> MemtableFlushWriter and the MemtableReclaimMemory pending queues grow as >> the memory becomes depleted, but the number of completed seems to stop >> changing a few minutes after the CPU load spikes. >> >> One other data point is there seems to be a huge number of mutations that >> occur after most of the stream has occured. Concurrent_writes is set at >> 256 with the queue getting as high as 200K before dropping down. >> >> Any suggestions for yaml changes or jvm changes? JVM.options is >> currently the default with the memory set to the max, the current YAML file >> is below. >> >> Thanks! >> >> >> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,329 StatusLogger.java:51 - >>> MutationStage 1 2 191498052 0 >>> 0 >>> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,331 StatusLogger.java:51 - >>> ViewMutationStage 0 0 0 0 >>> 0 >>> INFO [Service Thread] 2018-08-06 17:49:26,338 StatusLogger.java:51 - >>> PerDiskMemtableFlushWriter_0 0 0 5865 0 >>> 0 >>> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,343 StatusLogger.java:51 - >>> ReadStage 0 0 0 0 >>> 0 >>> INFO [Service Thread] 2018-08-06 17:49:26,347 StatusLogger.java:51 - >>> ValidationExecutor 0 0 0 0 >>> 0 >>> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,360 StatusLogger.java:51 - >>> RequestResponseStage 0 0 8 0 >>> 0 >>> INFO [Service Thread] 2018-08-06 17:49:26,380 StatusLogger.java:51 - >>> Sampler 0 0 0 0 >>> 0 >>> INFO [Service Thread] 2018-08-06 17:49:26,382 StatusLogger.java:51 - >>> *MemtableFlushWriter >>> 8 74293 4716 0 * 0 >>> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,388 StatusLogger.java:51 - >>> ReadRepairStage 0 0 0 0 >>> 0 >>> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,389 StatusLogger.java:51 - >>> CounterMutationStage 0 0 0 0 >>> 0 >>> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,404 StatusLogger.java:51 - >>> MiscStage 0 0 0 0 >>> 0 >>> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,407 StatusLogger.java:51 - >>> CompactionExecutor 8 13 493 0 >>> 0 >>> INFO [Service Thread] 2018-08-06 17:49:26,410 StatusLogger.java:51 - >>> InternalResponseStage 0 0 16 0 >>> 0 >>> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,413 StatusLogger.java:51 - >>> *MemtableReclaimMemory >>> 1 6066 356 0 * 0 >>> INFO [Service Thread] 2018-08-06 17:49:26,421 StatusLogger.java:51 - >>> AntiEntropyStage 0 0 0 0 >>> 0 >>> INFO [Service Thread] 2018-08-06 17:49:26,430 StatusLogger.java:51 - >>> CacheCleanupExecutor 0 0 0 0 >>> 0 >>> INFO [ScheduledTasks:1] 2018-08-06 17:49:26,431 StatusLogger.java:51 - >>> PendingRangeCalculator 0 0 9 0 >>> 0 >>> INFO [Service Thread] 2018-08-06 17:49:26,436 StatusLogger.java:61 - >>> CompactionManager 8 19 >> >> >> >> >> Current Yaml >> >> num_tokens: 256 >> >> hinted_handoff_enabled: true >> >> hinted_handoff_throttle_in_kb: 10240 >> >> max_hints_delivery_threads: 8 >> >> hints_flush_period_in_ms: 10000 >> >> max_hints_file_size_in_mb: 128 >> >> batchlog_replay_throttle_in_kb: 10240 >> >> authenticator: AllowAllAuthenticator >> >> authorizer: AllowAllAuthorizer >> >> role_manager: CassandraRoleManager >> >> roles_validity_in_ms: 2000 >> >> permissions_validity_in_ms: 2000 >> >> credentials_validity_in_ms: 2000 >> >> partitioner: org.apache.cassandra.dht.Murmur3Partitioner >> >> data_file_directories: >> >> - /data/cassandra/data >> >> commitlog_directory: /data/cassandra/commitlog >> >> cdc_enabled: false >> >> disk_failure_policy: stop >> >> commit_failure_policy: stop >> >> prepared_statements_cache_size_mb: >> >> thrift_prepared_statements_cache_size_mb: >> >> key_cache_size_in_mb: >> >> key_cache_save_period: 14400 >> >> row_cache_size_in_mb: 0 >> >> row_cache_save_period: 0 >> >> counter_cache_size_in_mb: >> >> counter_cache_save_period: 7200 >> >> saved_caches_directory: /data/cassandra/saved_caches >> >> commitlog_sync: periodic >> >> commitlog_sync_period_in_ms: 10000 >> >> commitlog_segment_size_in_mb: 32 >> >> seed_provider: >> >> - class_name: org.apache.cassandra.locator.SimpleSeedProvider >> >> parameters: >> >> - seeds: "10.1.1.11,10.1.1.12,10.1.1.13" >> >> concurrent_reads: 128 >> >> concurrent_writes: 256 >> >> concurrent_counter_writes: 96 >> >> concurrent_materialized_view_writes: 32 >> >> disk_optimization_strategy: spinning >> >> memtable_heap_space_in_mb: 61440 >> >> memtable_offheap_space_in_mb: 61440 >> >> memtable_allocation_type: heap_buffers >> >> commitlog_total_space_in_mb: 81920 >> >> memtable_flush_writers: 8 >> >> >> > -- Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade
