[
https://issues.apache.org/jira/browse/CASSANDRA-5555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13666324#comment-13666324
]
Jonathan Ellis commented on CASSANDRA-5555:
-------------------------------------------
You're right, it's a bit messy. :)
Digging in a bit, I'm actually a bit confused by the premise here -- unless
you're generating sstables to load on a machine with vastly more disk space
than your cluster members, you should be able to open the sstables you're
generating, since that's what the nodes you're streaming to will have to do.
(Before looking at the code, I assumed we were opening some kind of scanner or
buffer per sstable a la CASSANDRA-4142 but it looks like it's just the
SSTableReader.)
bq. With the default heap size, sstableloader will OOM when there are roughly
25k files in the directory to load
This default?
{code}
$JAVA -ea -cp $CLASSPATH -Xmx256M
{code}
Honestly I think telling people "increase the heap size" is a perfectly
adequate solution if that's really what we're running with right now.
Increasing it to 1GB out of the box also seems reasonable.
> Allow sstableloader to handle a larger number of files
> ------------------------------------------------------
>
> Key: CASSANDRA-5555
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5555
> Project: Cassandra
> Issue Type: Improvement
> Components: Tools
> Reporter: Tyler Hobbs
> Assignee: Dave Brosius
> Fix For: 1.2.6
>
> Attachments: 5555-01.txt
>
>
> With the default heap size, sstableloader will OOM when there are roughly 25k
> files in the directory to load. It's easy to reach this number of files in a
> single LCS column family.
> By avoiding creating all SSTableReaders up front in SSTableLoader, we should
> be able to increase the number of files that sstableloader can handle
> considerably.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
