Re: running out of diskspace during maintenance tasks
Hi Brian, What compaction are you running? Have you tried using leveled compaction? AFAIK it should generally require less disk space during compaction. Cheers, Jens — Sent from Mailbox On Wed, Jun 18, 2014 at 6:02 PM, Brian Tarbox tar...@cabotresearch.com wrote: I'm running on AWS m2.2xlarge instances using the ~800 gig ephemeral/attached disk for my data directory. My data size per node is nearing 400 gig. Sometimes during maintenance operations (repairs mostly I think) I run out of disk space as my understanding is that some of these operations require double the space of one's data. Since I can't change the size of attached storage for my instance type my question is can I somehow get these maintenance operations to use other volumes? Failing that, what are my options? Thanks. Brian Tarbox
running out of diskspace during maintenance tasks
I'm running on AWS m2.2xlarge instances using the ~800 gig ephemeral/attached disk for my data directory. My data size per node is nearing 400 gig. Sometimes during maintenance operations (repairs mostly I think) I run out of disk space as my understanding is that some of these operations require double the space of one's data. Since I can't change the size of attached storage for my instance type my question is can I somehow get these maintenance operations to use other volumes? Failing that, what are my options? Thanks. Brian Tarbox
Re: running out of diskspace during maintenance tasks
One option is to add new nodes, and do a node repair/cleanup on everything. That will at least reduce your per-node data size. On Wed, Jun 18, 2014 at 11:01 AM, Brian Tarbox tar...@cabotresearch.com wrote: I'm running on AWS m2.2xlarge instances using the ~800 gig ephemeral/attached disk for my data directory. My data size per node is nearing 400 gig. Sometimes during maintenance operations (repairs mostly I think) I run out of disk space as my understanding is that some of these operations require double the space of one's data. Since I can't change the size of attached storage for my instance type my question is can I somehow get these maintenance operations to use other volumes? Failing that, what are my options? Thanks. Brian Tarbox
Re: running out of diskspace during maintenance tasks
We do a repair -pr on each node once a week on a rolling basis. Should we be running cleanup as well? My understanding that was only used after adding/removing nodes? We'd like to avoid adding nodes if possible (which might not be). Still curious if we can get C* to do the maintenance task on a separate volume. Thanks. On Wed, Jun 18, 2014 at 12:03 PM, Jeremy Jongsma jer...@barchart.com wrote: One option is to add new nodes, and do a node repair/cleanup on everything. That will at least reduce your per-node data size. On Wed, Jun 18, 2014 at 11:01 AM, Brian Tarbox tar...@cabotresearch.com wrote: I'm running on AWS m2.2xlarge instances using the ~800 gig ephemeral/attached disk for my data directory. My data size per node is nearing 400 gig. Sometimes during maintenance operations (repairs mostly I think) I run out of disk space as my understanding is that some of these operations require double the space of one's data. Since I can't change the size of attached storage for my instance type my question is can I somehow get these maintenance operations to use other volumes? Failing that, what are my options? Thanks. Brian Tarbox
Re: running out of diskspace during maintenance tasks
AFAIK, when you run a repair a snapshot is created. After the repair, I run nodetool clearsnapshot to save disk space. Not sure it's you case or not. []s 2014-06-18 13:10 GMT-03:00 Brian Tarbox tar...@cabotresearch.com: We do a repair -pr on each node once a week on a rolling basis. Should we be running cleanup as well? My understanding that was only used after adding/removing nodes? We'd like to avoid adding nodes if possible (which might not be). Still curious if we can get C* to do the maintenance task on a separate volume. Thanks. On Wed, Jun 18, 2014 at 12:03 PM, Jeremy Jongsma jer...@barchart.com wrote: One option is to add new nodes, and do a node repair/cleanup on everything. That will at least reduce your per-node data size. On Wed, Jun 18, 2014 at 11:01 AM, Brian Tarbox tar...@cabotresearch.com wrote: I'm running on AWS m2.2xlarge instances using the ~800 gig ephemeral/attached disk for my data directory. My data size per node is nearing 400 gig. Sometimes during maintenance operations (repairs mostly I think) I run out of disk space as my understanding is that some of these operations require double the space of one's data. Since I can't change the size of attached storage for my instance type my question is can I somehow get these maintenance operations to use other volumes? Failing that, what are my options? Thanks. Brian Tarbox
Re: running out of diskspace during maintenance tasks
repair only creates snapshots if you use the “-snapshot” option. On June 18, 2014 at 12:28:58 PM, Marcelo Elias Del Valle (marc...@s1mbi0se.com.br) wrote: AFAIK, when you run a repair a snapshot is created. After the repair, I run nodetool clearsnapshot to save disk space. Not sure it's you case or not. []s 2014-06-18 13:10 GMT-03:00 Brian Tarbox tar...@cabotresearch.com: We do a repair -pr on each node once a week on a rolling basis. Should we be running cleanup as well? My understanding that was only used after adding/removing nodes? We'd like to avoid adding nodes if possible (which might not be). Still curious if we can get C* to do the maintenance task on a separate volume. Thanks. On Wed, Jun 18, 2014 at 12:03 PM, Jeremy Jongsma jer...@barchart.com wrote: One option is to add new nodes, and do a node repair/cleanup on everything. That will at least reduce your per-node data size. On Wed, Jun 18, 2014 at 11:01 AM, Brian Tarbox tar...@cabotresearch.com wrote: I'm running on AWS m2.2xlarge instances using the ~800 gig ephemeral/attached disk for my data directory. My data size per node is nearing 400 gig. Sometimes during maintenance operations (repairs mostly I think) I run out of disk space as my understanding is that some of these operations require double the space of one's data. Since I can't change the size of attached storage for my instance type my question is can I somehow get these maintenance operations to use other volumes? Failing that, what are my options? Thanks. Brian Tarbox
Re: running out of diskspace during maintenance tasks
On Wed, Jun 18, 2014 at 9:10 AM, Brian Tarbox tar...@cabotresearch.com wrote: We do a repair -pr on each node once a week on a rolling basis. https://issues.apache.org/jira/browse/CASSANDRA-5850?focusedCommentId=14036057page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14036057 Should we be running cleanup as well? No. =Rob
