sstableloader out of memory

[Nathan Jackels]

Hi all,

We're trying to load a snapshot back into a cluster, but are running into
memory issues.
We've got about 190GB of data across 11 sstable-generations. Some of the
smaller ones load, but the larger ones aren't.
We've tried increasing the max-heap-size to 16G, but stil see this
exception:

sstableloader -d cass1
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372
Established connection to initial hosts
Opening sstables and calculating sections to stream
Streaming relevant part of
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19968-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19930-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19966-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19960-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19944-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-9639-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19964-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-18879-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19965-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19967-Data.db
/snapshot_data/keyspace1/cf1-2195c1a0bc1011e69b699bbcfdee6372/keyspace1-cf1-ka-19959-Data.db
to []
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit
exceeded
at
org.apache.cassandra.io.compress.CompressionMetadata.getChunksForSections(CompressionMetadata.java:257)
at
org.apache.cassandra.streaming.messages.OutgoingFileMessage.(OutgoingFileMessage.java:70)
at
org.apache.cassandra.streaming.StreamTransferTask.addTransferFile(StreamTransferTask.java:58)
at
org.apache.cassandra.streaming.StreamSession.addTransferFiles(StreamSession.java:378)
at
org.apache.cassandra.streaming.StreamCoordinator.transferFiles(StreamCoordinator.java:147)
at
org.apache.cassandra.streaming.StreamPlan.transferFiles(StreamPlan.java:144)
at
org.apache.cassandra.io.sstable.SSTableLoader.stream(SSTableLoader.java:185)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:106)

Has anyone run into this before? The next steps we're going to try are
running sstableloader on each generation individually (suspecting that it's
trying to open all 11 generations at the same time). If that doesn't work
we'll try sstablesplit, but aren't that confident that would do anything
since it probably uses the same code to read the sstables as sstableloader
and also run out of memory.

Thanks,
Nathan

Correct ways to use Nodetool JMX Classes in Seperate Process

[Nathan Jackels]

Hi all,

A project I'm working on right now requires that a daemon/service running
on the same host as Cassandra be able to connect via JMX for many of the
same functions as nodetool and sstablemetadata.
The classpath that nodetool uses includes all the jars in cassandra/lib, so
we are using the same list.

The issue that we're running into is that eventually some of the classes in
org.apache.cassandra.db.commitlog are loaded, threads are created and
started in the static blocks and the process opens a handle/fd on the
commitlog files.
It will also creates a new log file if the commitlog_directory is empty.

Right now we're getting around this by putting a jar with no-op
implementations of nearly everything in the db.commitlog package on the
classpath before the cassandra/lib jars.
I've created CASSANDRA-13605 because it seems possible that this could be
observed by a long-running nodetool repair or compaction command, but in
the meantime are there any more common practices than this kludge we've
implemented?

We want to call public methods from the Cassandra internals instead of
creating nodetool processes and parsing stdout.
Adding a jar to the Cassandra classpath as a proxy is also not preferred
because of the way our product dependencies are handled internally at the
company.

Thanks,
Nathan