Hi all,
We're trying to load a snapshot back into a cluster, but are running into
memory issues.
We've got about 190GB of data across 11 sstable-generations. Some of the
smaller ones load, but the larger ones aren't.
We've tried increasing the max-heap-size to 16G, but stil see this
exception:
sstableloader -d cass1
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19968-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19930-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19966-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19960-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19944-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-9639-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19964-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-18879-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19965-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19967-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19959-Data.db
to [<snip 4 IP addresses>]
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
exceeded
at
org.apache.cassandra.io.compress.CompressionMetadata.getChunksForSections(CompressionMetadata.java:257)
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage.<init>(OutgoingFileMessage.java:70)
at
org.apache.cassandra.streaming.StreamTransferTask.addTransferFile(StreamTransferTask.java:58)
at
org.apache.cassandra.streaming.StreamSession.addTransferFiles(StreamSession.java:378)
at
org.apache.cassandra.streaming.StreamCoordinator.transferFiles(StreamCoordinator.java:147)
at
org.apache.cassandra.streaming.StreamPlan.transferFiles(StreamPlan.java:144)
at
org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:185)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:106)
Has anyone run into this before? The next steps we're going to try are
running sstableloader on each generation individually (suspecting that it's
trying to open all 11 generations at the same time). If that doesn't work
we'll try sstablesplit, but aren't that confident that would do anything
since it probably uses the same code to read the sstables as sstableloader
and also run out of memory.
Thanks,
Nathan
