Well, asking for 500MB of data at once for a server with such modest specs is asking for troubles. Here are my suggestions.
Disable the 1 GB row cache Consider allocating that memory for the java heap "Xms2500m Xmx2500m" Don't fetch all the columns at once -- page through them a slice at a time Increase the memtable to more than 64 MB if you want to write data to this cluster -Bryan On Wed, Nov 28, 2012 at 5:06 AM, Damien Lejeune <d.leje...@pepite.be> wrote: > Hi all, > > I'm currently experiencing a outOfMemory problem with Cassandra-1.1.6 on > Windows XP-Pro (32-bit). The server crashes when I try to query it with a > relatively small amount of data (around 100 rows with 50000 columns each: to > be precise, on my configuration, querying 75 or more rows makes the server > to crash). > I tried with different library (Hector, JDBC, Thrift) and with the Cassandra > stress tool. All lead to the same outOfMemory problem. > > My dataset is composed, for each row, of: 1 column in DateType, 49999 > columns in DoubleType. I ran a query to fetch the entire dataset (around > 330MB for the raw data + around 200MB for the metadata) and got the log at > the end of this message. > > I also checked the heap-dump with Mat which displays these top values: > Class Name > Objects Shallow Heap > java.nio.HeapByteBuffer 16,253,559 > 780,170,832 > bytes[] 16,254,013 > 330,207,640 <-- Data ? > java.util.TreeMap$Entry 8,126,711 > 260,054,752 > org.apache.cassandra.db.Column 8,116,589 > 194,798,136 <-- Metadata ? > > I tried to change the configuration in Cassandra for the values: > - row_cache_size_in_mb: tried different value between [0,1000] MB > - flush_largest_memtables_at: set to 0.1, but tried with 0.75 > - reduce_cache_sizes_at: tried 0.85, 0.6, 0.2 and 0.1 > - reduce_cache_capacity_to: tried 0.6 and 0.15 > - memtable_total_space_in_mb: 64 MB, but also tried to disable it (-> 1/3 of > the heap) > - Xms1G > - Xmx1500M > with no real observable improvements regarding my problem. > > My Cassandra server and client both run on the same machine. > > Here are the characteristics of my system configuration: > - Cassandra-1.1.6 > - java version "1.6.0_20" > Java(TM) SE Runtime Environment (build 1.6.0_20-b02) > Java HotSpot(TM) Client VM (build 16.3-b01, mixed mode, sharing) > - Windows XP-Pro 32 bits with service pack 3 > - CPU double-core, 32 bits @2.26GHz > - 3.48 of RAM > > I'm aware that my system configuration is not an optimized environment to > make Cassandra to run efficiently, but I wonder if you guys know a > workaround (or any idea on how) to fix this problem. Part of the answer is > probably that I do not have enough RAM to run the process, but I also wonder > if it is a 'normal' behaviour for Cassandra to handle this particular test > case that way. > > Cheers, > > Damien > > ---- Cassandra's LOG --- > > Starting Cassandra Server > INFO 09:10:27,171 Logging initialized > INFO 09:10:27,171 JVM vendor/version: Java HotSpot(TM) Client VM/1.6.0_18 > INFO 09:10:27,171 Heap size: 1072103424/1569521664 > INFO 09:10:27,171 Classpath: > E:\dl_benchmark\db\apache-cassandra-1.1.6\conf;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\antlr-3.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\apache-cassandra-1.1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\apache-cassandra-clientutil-1.1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\apache-cassandra-thrift-1.1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\avro-1.4.0-fixes.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\avro-1.4.0-sources-fixes.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\commons-cli-1.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\commons-codec-1.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\commons-lang-2.4.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\compress-lzf-0.8.4.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\concurrentlinkedhashmap-lru-1.3.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\guava-r08.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\high-scale-lib-1.1.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jackson-core-asl-1.9.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jackson-mapper-asl-1.9.2.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jamm-0.2.5.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jline-0.9.94.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\json-simple-1.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\libthrift-0.7.0.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\log4j-1.2.16.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\metrics-core-2.0.3.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\servlet-api-2.5-20081211.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\slf4j-api-1.6.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\slf4j-log4j12-1.6.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\snakeyaml-1.6.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\snappy-java-1.0.4.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\snaptree-0.1.jar;E:\dl_benchmark\db\apache-cassandra-1.1.6\build\classes\main;E:\dl_benchmark\db\apache-cassandra-1.1.6\build\classes\thrift;E:\dl_benchmark\db\apache-cassandra-1.1.6\lib\jamm-0.2.5.jar > INFO 09:10:27,171 JNA not found. Native methods will be disabled. > INFO 09:10:27,187 Loading settings from > file:/E:/dl_benchmark/db/apache-cassandra-1.1.6/conf/cassandra.yaml > INFO 09:10:27,296 32bit JVM detected. It is recommended to run Cassandra > on a 64bit JVM for better performance. > INFO 09:10:27,296 DiskAccessMode 'auto' determined to be standard, > indexAccessMode is standard > INFO 09:10:27,500 Global memtable threshold is enabled at 64MB > INFO 09:10:27,781 Initializing key cache with capacity of 51 MBs. > INFO 09:10:27,812 Scheduling key cache save to each 14400 seconds (going to > save all keys). > INFO 09:10:27,812 Initializing row cache with capacity of 1000 MBs and > provider org.apache.cassandra.cache.SerializingCacheProvider > INFO 09:10:27,812 Scheduling row cache save to each 0 seconds (going to > save all keys). > INFO 09:10:27,906 reading saved cache > \var\lib\cassandra\saved_caches\system-schema_columnfamilies-KeyCache > INFO 09:10:27,921 Opening > \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-hf-134 > (4938 bytes) > INFO 09:10:27,921 Opening > \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-hf-133 > (5122 bytes) > INFO 09:10:27,937 reading saved cache > \var\lib\cassandra\saved_caches\system-schema_columns-KeyCache > INFO 09:10:27,937 Opening > \var\lib\cassandra\data\system\schema_columns\system-schema_columns-hf-74 > (59 bytes) > INFO 09:10:27,937 Opening > \var\lib\cassandra\data\system\schema_columns\system-schema_columns-hf-73 > (13836053 bytes) > INFO 09:10:27,953 Opening > \var\lib\cassandra\data\system\Versions\system-Versions-hf-64 (247 bytes) > INFO 09:10:27,953 Opening > \var\lib\cassandra\data\system\schema_keyspaces\system-schema_keyspaces-hf-105 > (1244 bytes) > INFO 09:10:27,953 reading saved cache > \var\lib\cassandra\saved_caches\system-LocationInfo-KeyCache > INFO 09:10:27,953 Opening > \var\lib\cassandra\data\system\LocationInfo\system-LocationInfo-hf-134 (346 > bytes) > INFO 09:10:32,843 reading saved cache > \var\lib\cassandra\saved_caches\Keyspace1-Standard1-KeyCache > INFO 09:10:32,859 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-552 > (83124150 bytes) > INFO 09:10:32,859 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-44 > (5541610 bytes) > INFO 09:10:32,859 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-592 > (5541610 bytes) > INFO 09:10:32,859 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-591 > (5541610 bytes) > INFO 09:10:32,859 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-555 > (138540250 bytes) > INFO 09:10:32,859 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-565 > (60957710 bytes) > INFO 09:10:32,859 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-484 > (138540250 bytes) > INFO 09:10:32,859 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-586 > (49874490 bytes) > INFO 09:10:32,859 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-437 > (138540250 bytes) > INFO 09:10:32,875 Opening > \var\lib\cassandra\data\Keyspace1\Standard1\Keyspace1-Standard1-hf-590 > (60957710 bytes) > INFO 09:10:32,875 Opening > \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-133 (5514794 > bytes) > INFO 09:10:32,890 Opening > \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-130 (195755351 > bytes) > INFO 09:10:32,906 Opening > \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-131 (201295520 > bytes) > INFO 09:10:32,906 Opening > \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-132 (30327963 > bytes) > INFO 09:10:32,906 Opening > \var\lib\cassandra\data\TESTBENCHTWO\Row\TESTBENCHTWO-Row-hf-129 (336374894 > bytes) > INFO 09:10:32,953 Opening > \var\lib\cassandra\data\TESTBENCH\Row\TESTBENCH-Row-hf-198 (91008900 bytes) > INFO 09:10:32,968 Opening > \var\lib\cassandra\data\TESTBENCH\Row\TESTBENCH-Row-hf-202 (57927906 bytes) > INFO 09:10:32,968 Opening > \var\lib\cassandra\data\TESTBENCH\Row\TESTBENCH-Row-hf-188 (2917363977 > bytes) > INFO 09:10:33,156 Opening \var\lib\cassandra\data\DEMO\Row\DEMO-Row-hf-1 > (100 bytes) > INFO 09:10:33,171 Opening > \var\lib\cassandra\data\TESTBENCHTHREE\Row\TESTBENCHTHREE-Row-hf-60 > (96519635 bytes) > INFO 09:10:33,171 Opening > \var\lib\cassandra\data\TESTBENCHTHREE\Row\TESTBENCHTHREE-Row-hf-72 > (63929481 bytes) > INFO 09:10:43,890 completed pre-loading (10 keys) key cache. > INFO 09:10:44,000 Replaying > \var\lib\cassandra\commitlog\CommitLog-1354090021204.log, > \var\lib\cassandra\commitlog\CommitLog-1354090021205.log > INFO 09:10:44,015 Replaying > \var\lib\cassandra\commitlog\CommitLog-1354090021204.log > INFO 09:10:44,031 Finished reading > \var\lib\cassandra\commitlog\CommitLog-1354090021204.log > INFO 09:10:44,031 Replaying > \var\lib\cassandra\commitlog\CommitLog-1354090021205.log > INFO 09:10:44,031 Finished reading > \var\lib\cassandra\commitlog\CommitLog-1354090021205.log > INFO 09:10:44,046 Enqueuing flush of Memtable-Versions@6172767(83/103 > serialized/live bytes, 3 ops) > INFO 09:10:44,062 Writing Memtable-Versions@6172767(83/103 serialized/live > bytes, 3 ops) > INFO 09:10:44,296 Completed flushing > \var\lib\cassandra\data\system\Versions\system-Versions-hf-65-Data.db (247 > bytes) for commitlog position ReplayPosition(segmentId=1354090244001, > position=0) > INFO 09:10:44,312 Log replay complete, 3 replayed mutations > INFO 09:10:44,890 GC for ConcurrentMarkSweep: 210 ms for 1 collections, > 41139480 used; max is 1569521664 > INFO 09:10:49,906 Cassandra version: 1.1.6 > INFO 09:10:49,906 Thrift API version: 19.32.0 > INFO 09:10:49,906 CQL supported versions: 2.0.0,3.0.0-beta1 (default: > 2.0.0) > INFO 09:10:49,953 Loading persisted ring state > INFO 09:10:49,953 Starting up server gossip > INFO 09:10:49,953 Enqueuing flush of Memtable-LocationInfo@8678192(29/36 > serialized/live bytes, 1 ops) > INFO 09:10:49,968 Writing Memtable-LocationInfo@8678192(29/36 > serialized/live bytes, 1 ops) > INFO 09:10:49,968 Completed flushing > \var\lib\cassandra\data\system\LocationInfo\system-LocationInfo-hf-135-Data.db > (80 bytes) for commitlog position ReplayPosition(segmentId=1354090244001, > position=363) > INFO 09:10:55,890 Starting Messaging Service on port 7000 > INFO 09:10:55,890 Using saved token 6783448253301637730960311964445940653 > INFO 09:10:55,906 Enqueuing flush of Memtable-LocationInfo@24013134(53/66 > serialized/live bytes, 2 ops) > INFO 09:10:55,906 Writing Memtable-LocationInfo@24013134(53/66 > serialized/live bytes, 2 ops) > INFO 09:10:55,906 Completed flushing > \var\lib\cassandra\data\system\LocationInfo\system-LocationInfo-hf-136-Data.db > (163 bytes) for commitlog position ReplayPosition(segmentId=1354090244001, > position=544) > INFO 09:10:55,906 Node localhost/127.0.0.1 state jump to normal > INFO 09:10:55,921 Bootstrap/Replace/Move completed! Now serving reads. > INFO 09:10:55,953 Binding thrift service to localhost/127.0.0.1:9160 > INFO 09:10:55,953 Using TFastFramedTransport with a max frame size of > 33554432 bytes. > INFO 09:10:55,953 Using synchronous/threadpool thrift server on > localhost/127.0.0.1 : 9160 > INFO 09:10:55,953 Listening for thrift clients... > --- The Querying starts here --- > INFO 09:14:33,375 GC for ParNew: 742 ms for 1 collections, 1529127672 used; > max is 1569521664 > INFO 09:14:33,375 GC for ConcurrentMarkSweep: 7691 ms for 1 collections, > 1529135040 used; max is 1569521664 > INFO 09:14:33,421 Pool Name Active Pending Blocked > INFO 09:14:33,421 ReadStage 0 0 0 > INFO 09:14:33,421 RequestResponseStage 0 0 0 > INFO 09:14:33,421 ReadRepairStage 0 0 0 > INFO 09:14:33,421 MutationStage 0 0 0 > INFO 09:14:33,421 ReplicateOnWriteStage 0 0 0 > INFO 09:14:33,421 GossipStage 0 0 0 > INFO 09:14:33,421 AntiEntropyStage 0 0 0 > INFO 09:14:33,421 MigrationStage 0 0 0 > INFO 09:14:33,421 StreamStage 0 0 0 > INFO 09:14:33,421 MemtablePostFlusher 0 0 0 > INFO 09:14:33,421 FlushWriter 0 0 0 > INFO 09:14:33,421 MiscStage 0 0 0 > INFO 09:14:33,421 commitlog_archiver 0 0 0 > INFO 09:14:33,421 InternalResponseStage 0 0 0 > INFO 09:14:33,421 HintedHandoff 0 0 0 > INFO 09:14:33,421 CompactionManager 0 0 > INFO 09:14:33,468 MessagingService n/a 0,0 > INFO 09:14:33,468 Cache Type Size > Capacity KeysToSave > Provider > INFO 09:14:33,468 KeyCache 14 > 1114112 all > INFO 09:14:33,468 RowCache 0 > 1048576000 all > org.apache.cassandra.cache.SerializingCacheProvider > INFO 09:14:33,468 ColumnFamily Memtable ops,data > INFO 09:14:33,468 system.NodeIdInfo 0,0 > INFO 09:14:33,468 system.IndexInfo 0,0 > INFO 09:14:33,468 system.LocationInfo 0,0 > INFO 09:14:33,468 system.Versions 3,103 > INFO 09:14:33,468 system.schema_keyspaces 0,0 > INFO 09:14:33,468 system.Migrations 0,0 > INFO 09:14:33,468 system.schema_columnfamilies 0,0 > INFO 09:14:33,468 system.schema_columns 0,0 > INFO 09:14:33,468 system.HintsColumnFamily 0,0 > INFO 09:14:33,468 system.Schema 0,0 > INFO 09:14:33,468 Keyspace1.SuperCounter1 0,0 > INFO 09:14:33,468 Keyspace1.Standard1 0,0 > INFO 09:14:33,468 Keyspace1.Super1 0,0 > INFO 09:14:33,468 Keyspace1.Counter1 0,0 > INFO 09:14:33,468 TESTBENCHTWO.Row 0,0 > INFO 09:14:33,468 TESTBENCH.Row 0,0 > INFO 09:14:33,468 DEMO.Row 0,0 > INFO 09:14:33,468 TESTBENCHTHREE.Row 0,0 > WARN 09:14:33,468 Heap is 0.9742681958928348 full. You may need to reduce > memtable and/or cache sizes. Cassandra is now reducing cache sizes to free > up memory. Adjust reduce_cache_sizes_at threshold in cassandra.yaml if you > don't want Cassandra to do this automatically > WARN 09:14:33,468 Reducing RowCache capacity from 1048576000 to 0 to reduce > memory pressure > WARN 09:14:33,468 Reducing KeyCache capacity from 1114112 to 8 to reduce > memory pressure > WARN 09:14:33,468 Heap is 0.9742681958928348 full. You may need to reduce > memtable and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > WARN 09:14:33,468 Flushing CFS(Keyspace='system', ColumnFamily='Versions') > to relieve memory pressure > INFO 09:14:33,468 Enqueuing flush of Memtable-Versions@9422607(83/103 > serialized/live bytes, 3 ops) > INFO 09:14:33,468 Writing Memtable-Versions@9422607(83/103 serialized/live > bytes, 3 ops) > INFO 09:14:33,562 Completed flushing > \var\lib\cassandra\data\system\Versions\system-Versions-hf-66-Data.db (247 > bytes) for commitlog position ReplayPosition(segmentId=1354090244001, > position=544) > INFO 09:14:42,968 GC for ConcurrentMarkSweep: 8900 ms for 1 collections, > 1549381648 used; max is 1569521664 > INFO 09:14:42,968 Pool Name Active Pending Blocked > INFO 09:14:42,968 ReadStage 0 0 0 > INFO 09:14:42,968 RequestResponseStage 0 0 0 > INFO 09:14:42,984 ReadRepairStage 0 0 0 > INFO 09:14:42,984 MutationStage 0 0 0 > INFO 09:14:42,984 ReplicateOnWriteStage 0 0 0 > INFO 09:14:43,015 GossipStage 0 0 0 > INFO 09:14:43,015 AntiEntropyStage 0 0 0 > INFO 09:14:43,015 MigrationStage 0 0 0 > INFO 09:14:43,015 StreamStage 0 0 0 > INFO 09:14:43,015 MemtablePostFlusher 0 0 0 > INFO 09:14:43,015 FlushWriter 0 0 0 > INFO 09:14:43,015 MiscStage 0 0 0 > INFO 09:14:43,015 commitlog_archiver 0 0 0 > INFO 09:14:43,015 InternalResponseStage 0 0 0 > INFO 09:14:43,015 HintedHandoff 0 0 0 > INFO 09:14:43,015 CompactionManager 0 0 > INFO 09:14:43,015 MessagingService n/a 0,0 > INFO 09:14:43,015 Cache Type Size > Capacity KeysToSave > Provider > INFO 09:14:43,015 KeyCache 8 > 8 all > INFO 09:14:43,015 RowCache 0 > 0 all > org.apache.cassandra.cache.SerializingCacheProvider > INFO 09:14:43,015 ColumnFamily Memtable ops,data > INFO 09:14:43,015 system.NodeIdInfo 0,0 > INFO 09:14:43,015 system.IndexInfo 0,0 > INFO 09:14:43,015 system.LocationInfo 0,0 > INFO 09:14:43,015 system.Versions 0,0 > INFO 09:14:43,015 system.schema_keyspaces 0,0 > INFO 09:14:43,015 system.Migrations 0,0 > INFO 09:14:43,015 system.schema_columnfamilies 0,0 > INFO 09:14:43,015 system.schema_columns 0,0 > INFO 09:14:43,015 system.HintsColumnFamily 0,0 > INFO 09:14:43,015 system.Schema 0,0 > INFO 09:14:43,015 Keyspace1.SuperCounter1 0,0 > INFO 09:14:43,015 Keyspace1.Standard1 0,0 > INFO 09:14:43,015 Keyspace1.Super1 0,0 > INFO 09:14:43,015 Keyspace1.Counter1 0,0 > INFO 09:14:43,015 TESTBENCHTWO.Row 0,0 > INFO 09:14:43,015 TESTBENCH.Row 0,0 > INFO 09:14:43,015 DEMO.Row 0,0 > INFO 09:14:43,015 TESTBENCHTHREE.Row 0,0 > WARN 09:14:43,015 Heap is 0.9871680547889525 full. You may need to reduce > memtable and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO 09:14:43,015 Unable to reduce heap usage since there are no dirty > column families > INFO 09:14:50,046 GC for ConcurrentMarkSweep: 6530 ms for 1 collections, > 1561411240 used; max is 1569521664 > INFO 09:14:50,234 Pool Name Active Pending Blocked > INFO 09:14:50,234 ReadStage 0 0 0 > INFO 09:14:50,234 RequestResponseStage 0 0 0 > INFO 09:14:50,234 ReadRepairStage 0 0 0 > INFO 09:14:50,234 MutationStage 0 0 0 > INFO 09:14:50,234 ReplicateOnWriteStage 0 0 0 > INFO 09:14:50,234 GossipStage 0 0 0 > INFO 09:14:50,234 AntiEntropyStage 0 0 0 > INFO 09:14:50,234 MigrationStage 0 0 0 > INFO 09:14:50,234 StreamStage 0 0 0 > INFO 09:14:50,234 MemtablePostFlusher 0 0 0 > INFO 09:14:50,234 FlushWriter 0 0 0 > INFO 09:14:50,234 MiscStage 0 0 0 > INFO 09:14:50,234 commitlog_archiver 0 0 0 > INFO 09:14:50,234 InternalResponseStage 0 0 0 > INFO 09:14:50,234 HintedHandoff 0 0 0 > INFO 09:14:50,234 CompactionManager 0 0 > INFO 09:14:50,234 MessagingService n/a 0,0 > INFO 09:14:50,234 Cache Type Size > Capacity KeysToSave > Provider > INFO 09:14:50,234 KeyCache 8 > 8 all > INFO 09:14:50,234 RowCache 0 > 0 all > org.apache.cassandra.cache.SerializingCacheProvider > INFO 09:14:50,234 ColumnFamily Memtable ops,data > INFO 09:14:50,234 system.NodeIdInfo 0,0 > INFO 09:14:50,234 system.IndexInfo 0,0 > INFO 09:14:50,234 system.LocationInfo 0,0 > INFO 09:14:50,234 system.Versions 0,0 > INFO 09:14:50,234 system.schema_keyspaces 0,0 > INFO 09:14:50,234 system.Migrations 0,0 > INFO 09:14:50,234 system.schema_columnfamilies 0,0 > INFO 09:14:50,234 system.schema_columns 0,0 > INFO 09:14:50,265 system.HintsColumnFamily 0,0 > INFO 09:14:50,265 system.Schema 0,0 > INFO 09:14:50,265 Keyspace1.SuperCounter1 0,0 > INFO 09:14:50,265 Keyspace1.Standard1 0,0 > INFO 09:14:50,265 Keyspace1.Super1 0,0 > INFO 09:14:50,265 Keyspace1.Counter1 0,0 > INFO 09:14:50,265 TESTBENCHTWO.Row 0,0 > INFO 09:14:50,265 TESTBENCH.Row 0,0 > INFO 09:14:50,265 DEMO.Row 0,0 > INFO 09:14:50,265 TESTBENCHTHREE.Row 0,0 > WARN 09:14:50,265 Heap is 0.9948325504604185 full. You may need to reduce > memtable and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO 09:14:50,265 Unable to reduce heap usage since there are no dirty > column families > INFO 09:14:59,343 GC for ConcurrentMarkSweep: 8724 ms for 1 collections, > 1566184304 used; max is 1569521664 > INFO 09:14:59,343 Pool Name Active Pending Blocked > INFO 09:14:59,343 ReadStage 0 0 0 > INFO 09:14:59,343 RequestResponseStage 0 0 0 > INFO 09:14:59,343 ReadRepairStage 0 0 0 > INFO 09:14:59,343 MutationStage 0 0 0 > INFO 09:14:59,343 ReplicateOnWriteStage 0 0 0 > INFO 09:14:59,343 GossipStage 0 0 0 > INFO 09:14:59,343 AntiEntropyStage 0 0 0 > INFO 09:14:59,343 MigrationStage 0 0 0 > INFO 09:14:59,343 StreamStage 0 0 0 > INFO 09:14:59,343 MemtablePostFlusher 0 0 0 > INFO 09:14:59,343 FlushWriter 0 0 0 > INFO 09:14:59,343 MiscStage 0 0 0 > INFO 09:14:59,343 commitlog_archiver 0 0 0 > INFO 09:14:59,343 InternalResponseStage 0 0 0 > INFO 09:14:59,343 HintedHandoff 0 0 0 > INFO 09:14:59,343 CompactionManager 0 0 > INFO 09:14:59,343 MessagingService n/a 0,0 > INFO 09:14:59,343 Cache Type Size > Capacity KeysToSave > Provider > INFO 09:14:59,343 KeyCache 8 > 8 all > INFO 09:14:59,343 RowCache 0 > 0 all > org.apache.cassandra.cache.SerializingCacheProvider > INFO 09:14:59,343 ColumnFamily Memtable ops,data > INFO 09:14:59,343 system.NodeIdInfo 0,0 > INFO 09:14:59,343 system.IndexInfo 0,0 > INFO 09:14:59,343 system.LocationInfo 0,0 > INFO 09:14:59,343 system.Versions 0,0 > INFO 09:14:59,343 system.schema_keyspaces 0,0 > INFO 09:14:59,343 system.Migrations 0,0 > INFO 09:14:59,343 system.schema_columnfamilies 0,0 > INFO 09:14:59,343 system.schema_columns 0,0 > INFO 09:14:59,343 system.HintsColumnFamily 0,0 > INFO 09:14:59,343 system.Schema 0,0 > INFO 09:14:59,343 Keyspace1.SuperCounter1 0,0 > INFO 09:14:59,343 Keyspace1.Standard1 0,0 > INFO 09:14:59,343 Keyspace1.Super1 0,0 > INFO 09:14:59,343 Keyspace1.Counter1 0,0 > INFO 09:14:59,343 TESTBENCHTWO.Row 0,0 > INFO 09:14:59,343 TESTBENCH.Row 0,0 > INFO 09:14:59,343 DEMO.Row 0,0 > INFO 09:14:59,343 TESTBENCHTHREE.Row 0,0 > WARN 09:14:59,343 Heap is 0.997873645151546 full. You may need to reduce > memtable and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO 09:14:59,343 Unable to reduce heap usage since there are no dirty > column families > INFO 09:15:08,500 GC for ConcurrentMarkSweep: 8854 ms for 1 collections, > 1567778080 used; max is 1569521664 > INFO 09:15:08,500 Pool Name Active Pending Blocked > INFO 09:15:08,500 ReadStage 0 0 0 > INFO 09:15:08,500 RequestResponseStage 0 0 0 > INFO 09:15:08,500 ReadRepairStage 0 0 0 > INFO 09:15:08,500 MutationStage 0 0 0 > INFO 09:15:08,500 ReplicateOnWriteStage 0 0 0 > INFO 09:15:08,500 GossipStage 0 0 0 > INFO 09:15:08,515 AntiEntropyStage 0 0 0 > INFO 09:15:08,515 MigrationStage 0 0 0 > INFO 09:15:08,515 StreamStage 0 0 0 > INFO 09:15:08,515 MemtablePostFlusher 0 0 0 > INFO 09:15:08,515 FlushWriter 0 0 0 > INFO 09:15:08,515 MiscStage 0 0 0 > INFO 09:15:08,515 commitlog_archiver 0 0 0 > INFO 09:15:08,515 InternalResponseStage 0 0 0 > INFO 09:15:08,515 HintedHandoff 0 0 0 > INFO 09:15:08,515 CompactionManager 0 0 > INFO 09:15:08,515 MessagingService n/a 0,0 > INFO 09:15:08,515 Cache Type Size > Capacity KeysToSave > Provider > INFO 09:15:08,515 KeyCache 8 > 8 all > INFO 09:15:08,515 RowCache 0 > 0 all > org.apache.cassandra.cache.SerializingCacheProvider > INFO 09:15:08,515 ColumnFamily Memtable ops,data > INFO 09:15:08,515 system.NodeIdInfo 0,0 > INFO 09:15:08,515 system.IndexInfo 0,0 > INFO 09:15:08,515 system.LocationInfo 0,0 > INFO 09:15:08,515 system.Versions 0,0 > INFO 09:15:08,515 system.schema_keyspaces 0,0 > INFO 09:15:08,515 system.Migrations 0,0 > INFO 09:15:08,515 system.schema_columnfamilies 0,0 > INFO 09:15:08,515 system.schema_columns 0,0 > INFO 09:15:08,515 system.HintsColumnFamily 0,0 > INFO 09:15:08,515 system.Schema 0,0 > INFO 09:15:08,515 Keyspace1.SuperCounter1 0,0 > INFO 09:15:08,515 Keyspace1.Standard1 0,0 > INFO 09:15:08,515 Keyspace1.Super1 0,0 > INFO 09:15:08,515 Keyspace1.Counter1 0,0 > INFO 09:15:08,515 TESTBENCHTWO.Row 0,0 > INFO 09:15:08,515 TESTBENCH.Row 0,0 > INFO 09:15:08,515 DEMO.Row 0,0 > INFO 09:15:08,515 TESTBENCHTHREE.Row 0,0 > WARN 09:15:08,515 Heap is 0.9988890984814084 full. You may need to reduce > memtable and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO 09:15:08,515 Unable to reduce heap usage since there are no dirty > column families > INFO 09:15:13,734 GC for ConcurrentMarkSweep: 5129 ms for 1 collections, > 1568572088 used; max is 1569521664 > INFO 09:15:13,734 Pool Name Active Pending Blocked > INFO 09:15:13,734 ReadStage 0 0 0 > INFO 09:15:13,734 RequestResponseStage 0 0 0 > INFO 09:15:13,734 ReadRepairStage 0 0 0 > INFO 09:15:13,734 MutationStage 0 0 0 > INFO 09:15:13,734 ReplicateOnWriteStage 0 0 0 > INFO 09:15:13,734 GossipStage 0 0 0 > INFO 09:15:13,734 AntiEntropyStage 0 0 0 > INFO 09:15:13,734 MigrationStage 0 0 0 > INFO 09:15:13,734 StreamStage 0 0 0 > INFO 09:15:13,734 MemtablePostFlusher 0 0 0 > INFO 09:15:13,734 FlushWriter 0 0 0 > INFO 09:15:13,734 MiscStage 0 0 0 > INFO 09:15:13,765 commitlog_archiver 0 0 0 > INFO 09:15:13,765 InternalResponseStage 0 0 0 > INFO 09:15:13,765 HintedHandoff 0 0 0 > INFO 09:15:13,765 CompactionManager 0 0 > INFO 09:15:13,765 MessagingService n/a 0,0 > INFO 09:15:13,765 Cache Type Size > Capacity KeysToSave > Provider > INFO 09:15:13,765 KeyCache 8 > 8 all > INFO 09:15:13,765 RowCache 0 > 0 all > org.apache.cassandra.cache.SerializingCacheProvider > INFO 09:15:13,765 ColumnFamily Memtable ops,data > INFO 09:15:13,765 system.NodeIdInfo 0,0 > INFO 09:15:13,765 system.IndexInfo 0,0 > INFO 09:15:13,765 system.LocationInfo 0,0 > INFO 09:15:13,765 system.Versions 0,0 > INFO 09:15:13,765 system.schema_keyspaces 0,0 > INFO 09:15:13,765 system.Migrations 0,0 > INFO 09:15:13,765 system.schema_columnfamilies 0,0 > INFO 09:15:13,765 system.schema_columns 0,0 > INFO 09:15:13,765 system.HintsColumnFamily 0,0 > INFO 09:15:13,765 system.Schema 0,0 > INFO 09:15:13,765 Keyspace1.SuperCounter1 0,0 > INFO 09:15:13,765 Keyspace1.Standard1 0,0 > INFO 09:15:13,765 Keyspace1.Super1 0,0 > INFO 09:15:13,765 Keyspace1.Counter1 0,0 > INFO 09:15:13,765 TESTBENCHTWO.Row 0,0 > INFO 09:15:13,765 TESTBENCH.Row 0,0 > INFO 09:15:13,765 DEMO.Row 0,0 > INFO 09:15:13,765 TESTBENCHTHREE.Row 0,0 > WARN 09:15:13,765 Heap is 0.999394990192375 full. You may need to reduce > memtable and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO 09:15:13,765 Unable to reduce heap usage since there are no dirty > column families > INFO 09:15:21,484 GC for ConcurrentMarkSweep: 7718 ms for 1 collections, > 1568953672 used; max is 1569521664 > INFO 09:15:21,484 Pool Name Active Pending Blocked > INFO 09:15:21,484 ReadStage 0 0 0 > INFO 09:15:21,484 RequestResponseStage 0 0 0 > INFO 09:15:21,484 ReadRepairStage 0 0 0 > INFO 09:15:21,484 MutationStage 0 0 0 > INFO 09:15:21,484 ReplicateOnWriteStage 0 0 0 > INFO 09:15:21,484 GossipStage 0 0 0 > INFO 09:15:21,484 AntiEntropyStage 0 0 0 > INFO 09:15:21,500 MigrationStage 0 0 0 > INFO 09:15:21,500 StreamStage 0 0 0 > INFO 09:15:21,500 MemtablePostFlusher 0 0 0 > INFO 09:15:35,750 FlushWriter 0 0 0 > INFO 09:15:35,750 MiscStage 0 0 0 > INFO 09:15:35,750 commitlog_archiver 0 0 0 > INFO 09:15:35,750 InternalResponseStage 0 0 0 > INFO 09:15:35,750 HintedHandoff 0 0 0 > INFO 09:15:35,750 CompactionManager 0 0 > INFO 09:15:35,750 MessagingService n/a 0,0 > INFO 09:15:35,750 Cache Type Size > Capacity KeysToSave > Provider > INFO 09:16:26,046 KeyCache 8 > 8 all > INFO 09:16:31,140 RowCache 0 > 0 all > org.apache.cassandra.cache.SerializingCacheProvider > INFO 09:16:36,296 ColumnFamily Memtable ops,data > INFO 09:16:36,296 system.NodeIdInfo 0,0 > INFO 09:16:36,296 system.IndexInfo 0,0 > INFO 09:16:41,437 system.LocationInfo 0,0 > INFO 09:16:41,437 system.Versions 0,0 > java.lang.OutOfMemoryError: Java heap space > Dumping heap to java_pid5316.hprof ... > Heap dump file created [1755520972 bytes in 84.007 secs] > INFO 09:21:06,562 system.schema_keyspaces 0,0 > INFO 09:24:11,906 Stop listening to thrift clients > INFO 09:24:24,343 system.Migrations 0,0 > ERROR 09:24:24,343 Exception in thread Thread[COMMIT-LOG-WRITER,5,main] > java.lang.OutOfMemoryError: Java heap space > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1739) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1954) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395) > at > org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:46) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) > at java.lang.Thread.run(Thread.java:619) > ERROR 09:24:24,359 Exception in thread > Thread[PERIODIC-COMMIT-LOG-SYNCER,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR 09:24:24,359 Exception in thread Thread[OptionalTasks:1,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR 09:24:24,359 Exception in thread Thread[RMI TCP Connection(idle),5,RMI > Runtime] > java.lang.OutOfMemoryError: Java heap space > at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:59) > at java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:42) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:725) > at > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > ERROR 09:24:24,359 Exception in thread Thread[Thrift:1,5,main] > java.lang.OutOfMemoryError: Java heap space > INFO 09:24:24,343 system.schema_columnfamilies 0,0 > INFO 09:24:24,421 system.schema_columns 0,0 > INFO 09:24:24,421 system.HintsColumnFamily 0,0 > INFO 09:24:24,421 system.Schema 0,0 > INFO 09:24:24,421 Keyspace1.SuperCounter1 0,0 > INFO 09:24:24,421 Keyspace1.Standard1 0,0 > INFO 09:24:24,421 Keyspace1.Super1 0,0 > INFO 09:24:24,421 Keyspace1.Counter1 0,0 > INFO 09:24:24,421 TESTBENCHTWO.Row 0,0 > INFO 09:24:24,421 TESTBENCH.Row 0,0 > INFO 09:24:24,421 DEMO.Row 0,0 > INFO 09:24:24,421 TESTBENCHTHREE.Row 0,0 > WARN 09:24:24,421 Heap is 0.9996381113985057 full. You may need to reduce > memtable and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at threshold in > cassandra.yaml if you don't want Cassandra to do this automatically > INFO 09:24:24,421 Unable to reduce heap usage since there are no dirty > column families > INFO 09:24:24,484 Announcing shutdown > INFO 09:24:25,421 GC for ConcurrentMarkSweep: 463729 ms for 86 collections, > 22709920 used; max is 1569521664 > INFO 09:24:25,421 Pool Name Active Pending Blocked > INFO 09:24:25,421 ReadStage 0 0 0 > INFO 09:24:25,421 RequestResponseStage 0 0 0 > INFO 09:24:25,421 ReadRepairStage 0 0 0 > INFO 09:24:25,421 MutationStage 0 0 0 > INFO 09:24:25,421 ReplicateOnWriteStage 0 0 0 > INFO 09:24:25,421 GossipStage 0 0 0 > INFO 09:24:25,421 AntiEntropyStage 0 0 0 > INFO 09:24:25,421 MigrationStage 0 0 0 > INFO 09:24:25,437 StreamStage 0 0 0 > INFO 09:24:25,437 MemtablePostFlusher 0 0 0 > INFO 09:24:25,437 FlushWriter 0 0 0 > INFO 09:24:25,437 MiscStage 0 0 0 > INFO 09:24:25,437 commitlog_archiver 0 0 0 > INFO 09:24:25,437 InternalResponseStage 0 0 0 > INFO 09:24:25,437 HintedHandoff 0 0 0 > INFO 09:24:25,437 CompactionManager 0 0 > INFO 09:24:25,437 MessagingService n/a 0,0 > INFO 09:24:25,437 Cache Type Size > Capacity KeysToSave > Provider > INFO 09:24:25,437 KeyCache 8 > 8 all > INFO 09:24:25,437 RowCache 0 > 0 all > org.apache.cassandra.cache.SerializingCacheProvider > INFO 09:24:25,437 ColumnFamily Memtable ops,data > INFO 09:24:25,437 system.NodeIdInfo 0,0 > INFO 09:24:25,437 system.IndexInfo 0,0 > INFO 09:24:25,437 system.LocationInfo 0,0 > INFO 09:24:25,437 system.Versions 0,0 > INFO 09:24:25,437 system.schema_keyspaces 0,0 > INFO 09:24:25,437 system.Migrations 0,0 > INFO 09:24:25,437 system.schema_columnfamilies 0,0 > INFO 09:24:25,437 system.schema_columns 0,0 > INFO 09:24:25,437 system.HintsColumnFamily 0,0 > INFO 09:24:25,437 system.Schema 0,0 > INFO 09:24:25,437 Keyspace1.SuperCounter1 0,0 > INFO 09:24:25,437 Keyspace1.Standard1 0,0 > INFO 09:24:25,437 Keyspace1.Super1 0,0 > INFO 09:24:25,437 Keyspace1.Counter1 0,0 > INFO 09:24:25,437 TESTBENCHTWO.Row 0,0 > INFO 09:24:25,437 TESTBENCH.Row 0,0 > INFO 09:24:25,437 DEMO.Row 0,0 > INFO 09:24:25,437 TESTBENCHTHREE.Row 0,0 > INFO 09:24:25,484 Waiting for messaging service to quiesce > INFO 09:24:25,531 MessagingService shutting down server thread. > > ---- End of LOG -----
