Re: Adding disk to operating C*

[James Shaw]

I think it's case by case, depended on chasing read performance or write
performance, or both.
Ours are used for application, read request 10 times larger than writing,
application wants read performance and doesn't care writing,  we use 4 SSD
each 380 GB for each node (total 1.5T a node),  read latency is 0.4 ms/op.
Of course, if cassandra used for reporting, that is different.

Thanks,

James

On Sat, Mar 10, 2018 at 7:55 AM, Rahul Singh 
wrote:

> My 1.5T bound is for high throughput for read and write with hundreds of
> nodes — specifically with needs for quick bootstrap / repairs when adding /
> replacing nodes.
>
> Lower the density the faster it is to add nodes.
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
>
> On Mar 9, 2018, 11:30 AM -0500, Jon Haddad , wrote:
>
> I agree with Jeff - I usually advise teams to cap their density around
> 3TB, especially with TWCS.  Read heavy workloads tend to use smaller
> datasets and ring size ends up being a function of performance tuning.
>
> Since 2.2 bootstrap can now be resumed, which helps quite a bit with the
> streaming problem, see CASSANDRA-8838.
>
> Jon
>
>
> On Mar 9, 2018, at 7:39 AM, Jeff Jirsa  wrote:
>
> 1.5 TB sounds very very conservative - 3-4T is where I set the limit at
> past jobs. Have heard of people doing twice that (6-8T).
>
> --
> Jeff Jirsa
>
>
> On Mar 8, 2018, at 11:09 PM, Niclas Hedhman  wrote:
>
> I am curious about the side comment; "Depending on your usecase you may not
> want to have a data density over 1.5 TB per node."
>
> Why is that? I am planning much bigger than that, and now you give me
> pause...
>
>
> Cheers
> Niclas
>
> On Wed, Mar 7, 2018 at 6:59 PM, Rahul Singh 
> wrote:
>
>> Are you putting both the commitlogs and the Sstables on the adds?
>> Consider moving your snapshots often if that’s also taking up space. Maybe
>> able to save some space before you add drives.
>>
>> You should be able to add these new drives and mount them without an
>> issue. Try to avoid different number of data dirs across nodes. It makes
>> automation of operational processes a little harder.
>>
>> As an aside, Depending on your usecase you may not want to have a data
>> density over 1.5 TB per node.
>>
>> --
>> Rahul Singh
>> rahul.si...@anant.us
>>
>> Anant Corporation
>>
>> On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , wrote:
>>
>> Hello,
>>
>> I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
>>
>> I'm trying to mount an additional disk(SSD 1TB) on each node because each
>> disk usage growth rate is higher than I expected. Then I will add the the
>> directory to data_file_directories in cassanra.yaml
>>
>> Can I get advice from who have experienced this situation?
>> If we go through the above steps one by one, will we be able to complete
>> the upgrade without losing data?
>> The replication strategy is SimpleStrategy, RF 2.
>>
>> Thank you in advance
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java
>
>
>

Re: Adding disk to operating C*

[Rahul Singh]

My 1.5T bound is for high throughput for read and write with hundreds of nodes 
— specifically with needs for quick bootstrap / repairs when adding / replacing 
nodes.

Lower the density the faster it is to add nodes.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Mar 9, 2018, 11:30 AM -0500, Jon Haddad , wrote:
> I agree with Jeff - I usually advise teams to cap their density around 3TB, 
> especially with TWCS.  Read heavy workloads tend to use smaller datasets and 
> ring size ends up being a function of performance tuning.
>
> Since 2.2 bootstrap can now be resumed, which helps quite a bit with the 
> streaming problem, see CASSANDRA-8838.
>
> Jon
>
>
> > On Mar 9, 2018, at 7:39 AM, Jeff Jirsa  wrote:
> >
> > 1.5 TB sounds very very conservative - 3-4T is where I set the limit at 
> > past jobs. Have heard of people doing twice that (6-8T).
> >
> > --
> > Jeff Jirsa
> >
> >
> > On Mar 8, 2018, at 11:09 PM, Niclas Hedhman  wrote:
> >
> > > I am curious about the side comment; "Depending on your usecase you may 
> > > not
> > > want to have a data density over 1.5 TB per node."
> > >
> > > Why is that? I am planning much bigger than that, and now you give me
> > > pause...
> > >
> > >
> > > Cheers
> > > Niclas
> > >
> > > > On Wed, Mar 7, 2018 at 6:59 PM, Rahul Singh 
> > > >  wrote:
> > > > > Are you putting both the commitlogs and the Sstables on the adds? 
> > > > > Consider moving your snapshots often if that’s also taking up space. 
> > > > > Maybe able to save some space before you add drives.
> > > > >
> > > > > You should be able to add these new drives and mount them without an 
> > > > > issue. Try to avoid different number of data dirs across nodes. It 
> > > > > makes automation of operational processes a little harder.
> > > > >
> > > > > As an aside, Depending on your usecase you may not want to have a 
> > > > > data density over 1.5 TB per node.
> > > > >
> > > > > --
> > > > > Rahul Singh
> > > > > rahul.si...@anant.us
> > > > >
> > > > > Anant Corporation
> > > > >
> > > > > On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , 
> > > > > wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
> > > > > >
> > > > > > I'm trying to mount an additional disk(SSD 1TB) on each node 
> > > > > > because each disk usage growth rate is higher than I expected. Then 
> > > > > > I will add the the directory to data_file_directories in 
> > > > > > cassanra.yaml
> > > > > >
> > > > > > Can I get advice from who have experienced this situation?
> > > > > > If we go through the above steps one by one, will we be able to 
> > > > > > complete the upgrade without losing data?
> > > > > > The replication strategy is SimpleStrategy, RF 2.
> > > > > >
> > > > > > Thank you in advance
> > > > > > -
> > > > > > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > > > > > For additional commands, e-mail: user-h...@cassandra.apache.org
> > > > > >
> > >
> > >
> > >
> > > --
> > > Niclas Hedhman, Software Developer
> > > http://zest.apache.org - New Energy for Java
>

Re: Adding disk to operating C*

[Jon Haddad]

I agree with Jeff - I usually advise teams to cap their density around 3TB, 
especially with TWCS.  Read heavy workloads tend to use smaller datasets and 
ring size ends up being a function of performance tuning.

Since 2.2 bootstrap can now be resumed, which helps quite a bit with the 
streaming problem, see CASSANDRA-8838.

Jon


> On Mar 9, 2018, at 7:39 AM, Jeff Jirsa  wrote:
> 
> 1.5 TB sounds very very conservative - 3-4T is where I set the limit at past 
> jobs. Have heard of people doing twice that (6-8T). 
> 
> -- 
> Jeff Jirsa
> 
> 
> On Mar 8, 2018, at 11:09 PM, Niclas Hedhman  > wrote:
> 
>> I am curious about the side comment; "Depending on your usecase you may not
>> want to have a data density over 1.5 TB per node."
>> 
>> Why is that? I am planning much bigger than that, and now you give me
>> pause...
>> 
>> 
>> Cheers
>> Niclas
>> 
>> On Wed, Mar 7, 2018 at 6:59 PM, Rahul Singh > > wrote:
>> Are you putting both the commitlogs and the Sstables on the adds? Consider 
>> moving your snapshots often if that’s also taking up space. Maybe able to 
>> save some space before you add drives.
>> 
>> You should be able to add these new drives and mount them without an issue. 
>> Try to avoid different number of data dirs across nodes. It makes automation 
>> of operational processes a little harder.
>> 
>> As an aside, Depending on your usecase you may not want to have a data 
>> density over 1.5 TB per node.
>> 
>> --
>> Rahul Singh
>> rahul.si...@anant.us 
>> 
>> Anant Corporation
>> 
>> On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim > >, wrote:
>>> Hello,
>>> 
>>> I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
>>> 
>>> I'm trying to mount an additional disk(SSD 1TB) on each node because each 
>>> disk usage growth rate is higher than I expected. Then I will add the the 
>>> directory to data_file_directories in cassanra.yaml
>>> 
>>> Can I get advice from who have experienced this situation?
>>> If we go through the above steps one by one, will we be able to complete 
>>> the upgrade without losing data?
>>> The replication strategy is SimpleStrategy, RF 2.
>>> 
>>> Thank you in advance
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org 
>>> 
>>> For additional commands, e-mail: user-h...@cassandra.apache.org 
>>> 
>>> 
>> 
>> 
>> 
>> -- 
>> Niclas Hedhman, Software Developer
>> http://zest.apache.org  - New Energy for Java

Re: Adding disk to operating C*

[Jeff Jirsa]

1.5 TB sounds very very conservative - 3-4T is where I set the limit at past 
jobs. Have heard of people doing twice that (6-8T). 

-- 
Jeff Jirsa


> On Mar 8, 2018, at 11:09 PM, Niclas Hedhman  wrote:
> 
> I am curious about the side comment; "Depending on your usecase you may not
> want to have a data density over 1.5 TB per node."
> 
> Why is that? I am planning much bigger than that, and now you give me
> pause...
> 
> 
> Cheers
> Niclas
> 
>> On Wed, Mar 7, 2018 at 6:59 PM, Rahul Singh  
>> wrote:
>> Are you putting both the commitlogs and the Sstables on the adds? Consider 
>> moving your snapshots often if that’s also taking up space. Maybe able to 
>> save some space before you add drives.
>> 
>> You should be able to add these new drives and mount them without an issue. 
>> Try to avoid different number of data dirs across nodes. It makes automation 
>> of operational processes a little harder.
>> 
>> As an aside, Depending on your usecase you may not want to have a data 
>> density over 1.5 TB per node.
>> 
>> --
>> Rahul Singh
>> rahul.si...@anant.us
>> 
>> Anant Corporation
>> 
>>> On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , wrote:
>>> Hello,
>>> 
>>> I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
>>> 
>>> I'm trying to mount an additional disk(SSD 1TB) on each node because each 
>>> disk usage growth rate is higher than I expected. Then I will add the the 
>>> directory to data_file_directories in cassanra.yaml
>>> 
>>> Can I get advice from who have experienced this situation?
>>> If we go through the above steps one by one, will we be able to complete 
>>> the upgrade without losing data?
>>> The replication strategy is SimpleStrategy, RF 2.
>>> 
>>> Thank you in advance
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> 
> 
> 
> 
> -- 
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java

Re: Adding disk to operating C*

[Rahul Singh]

Yep. Most of my arguments are the same from seeing it in production. Cass=
andra is used for fast writes and generally fast reads with redundancy an=
d failover for OLTP and OLAP. It=E2=80=99s not just a bunch of dumb disks=
. You can throw crap into S3 or HD=46S and analyze / report with Hive or =
Spark.

You can always have more data density for use adds that are not critical =
in a different cluster or DC.

Rahul

On Mar 9, 2018, 7:25 AM -0500, Kyrylo Lebediev <kyrylo_lebed...@epam.com>, 
wrote:
> Niclas,
> Here is Jeff's comment regarding this: https://stackoverflow.com/a/31690279
> From: Niclas Hedhman <nic...@apache.org>
> Sent: Friday, March 9, 2018 9:09:53 AM
> To: user@cassandra.apache.org; Rahul Singh
> Subject: Re: Adding disk to operating C*
>
> I am curious about the side comment; "Depending on your usecase you may not
> want to have a data density over 1.5 TB per node."
>
> Why is that? I am planning much bigger than that, and now you give me
> pause...
>
>
> Cheers
> Niclas
>
> On Wed, Mar 7, 2018 at 6:59 PM, Rahul Singh <rahul.xavier.si...@gmail.com> 
> wrote:
> > Are you putting both the commitlogs and the Sstables on the adds? Consider 
> > moving your snapshots often if that’s also taking up space. Maybe able to 
> > save some space before you add drives.
> >
> > You should be able to add these new drives and mount them without an issue. 
> > Try to avoid different number of data dirs across nodes. It makes 
> > automation of operational processes a little harder.
> >
> > As an aside, Depending on your usecase you may not want to have a data 
> > density over 1.5 TB per node.
> >
> > --
> > Rahul Singh
> > rahul.si...@anant.us
> >
> > Anant Corporation
> >
> > On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim <eunsu.bil...@gmail.com>, wrote:
> > > Hello,
> > >
> > > I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
> > >
> > > I'm trying to mount an additional disk(SSD 1TB) on each node because each 
> > > disk usage growth rate is higher than I expected. Then I will add the the 
> > > directory to data_file_directories in cassanra.yaml
> > >
> > > Can I get advice from who have experienced this situation?
> > > If we go through the above steps one by one, will we be able to complete 
> > > the upgrade without losing data?
> > > The replication strategy is SimpleStrategy, RF 2.
> > >
> > > Thank you in advance
> > > -
> > > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: user-h...@cassandra.apache.org
> > >
>
>
>
> --
> Niclas Hedhman, Software Developer
> http://zest.apache.org - New Energy for Java

Re: Adding disk to operating C*

[Kyrylo Lebediev]

Niclas,

Here is Jeff's comment regarding this: https://stackoverflow.com/a/31690279


From: Niclas Hedhman <nic...@apache.org>
Sent: Friday, March 9, 2018 9:09:53 AM
To: user@cassandra.apache.org; Rahul Singh
Subject: Re: Adding disk to operating C*

I am curious about the side comment; "Depending on your usecase you may not
want to have a data density over 1.5 TB per node."

Why is that? I am planning much bigger than that, and now you give me
pause...


Cheers
Niclas

On Wed, Mar 7, 2018 at 6:59 PM, Rahul Singh 
<rahul.xavier.si...@gmail.com<mailto:rahul.xavier.si...@gmail.com>> wrote:
Are you putting both the commitlogs and the Sstables on the adds? Consider 
moving your snapshots often if that’s also taking up space. Maybe able to save 
some space before you add drives.

You should be able to add these new drives and mount them without an issue. Try 
to avoid different number of data dirs across nodes. It makes automation of 
operational processes a little harder.

As an aside, Depending on your usecase you may not want to have a data density 
over 1.5 TB per node.

--
Rahul Singh
rahul.si...@anant.us<mailto:rahul.si...@anant.us>

Anant Corporation

On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim 
<eunsu.bil...@gmail.com<mailto:eunsu.bil...@gmail.com>>, wrote:
Hello,

I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)

I'm trying to mount an additional disk(SSD 1TB) on each node because each disk 
usage growth rate is higher than I expected. Then I will add the the directory 
to data_file_directories in cassanra.yaml

Can I get advice from who have experienced this situation?
If we go through the above steps one by one, will we be able to complete the 
upgrade without losing data?
The replication strategy is SimpleStrategy, RF 2.

Thank you in advance
-
To unsubscribe, e-mail: 
user-unsubscr...@cassandra.apache.org<mailto:user-unsubscr...@cassandra.apache.org>
For additional commands, e-mail: 
user-h...@cassandra.apache.org<mailto:user-h...@cassandra.apache.org>




--
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java

Re: Adding disk to operating C*

[Niclas Hedhman]

I am curious about the side comment; "Depending on your usecase you may not
want to have a data density over 1.5 TB per node."

Why is that? I am planning much bigger than that, and now you give me
pause...


Cheers
Niclas

On Wed, Mar 7, 2018 at 6:59 PM, Rahul Singh 
wrote:

> Are you putting both the commitlogs and the Sstables on the adds? Consider
> moving your snapshots often if that’s also taking up space. Maybe able to
> save some space before you add drives.
>
> You should be able to add these new drives and mount them without an
> issue. Try to avoid different number of data dirs across nodes. It makes
> automation of operational processes a little harder.
>
> As an aside, Depending on your usecase you may not want to have a data
> density over 1.5 TB per node.
>
> --
> Rahul Singh
> rahul.si...@anant.us
>
> Anant Corporation
>
> On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , wrote:
>
> Hello,
>
> I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
>
> I'm trying to mount an additional disk(SSD 1TB) on each node because each
> disk usage growth rate is higher than I expected. Then I will add the the
> directory to data_file_directories in cassanra.yaml
>
> Can I get advice from who have experienced this situation?
> If we go through the above steps one by one, will we be able to complete
> the upgrade without losing data?
> The replication strategy is SimpleStrategy, RF 2.
>
> Thank you in advance
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>


-- 
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java

Re: Adding disk to operating C*

[Eunsu Kim]

Thanks for the answer. I never forget to flush, drain before shutting down 
Cassandra.
It is a system that deals with lighter and faster data than accuracy. So rf = 2 
and cl = one.
Thank you again.


> On 9 Mar 2018, at 3:12 PM, Jeff Jirsa  wrote:
> 
> There is no shuffling as the servers go up and down. Cassandra doesn’t do 
> that. 
> 
> However, rf=2 is atypical and sometime problematic.
> 
> If you read or write with quorum / two / all, you’ll get unavailables during 
> the restart
> 
> If you read or write with cl one, you’ll potentially not see data previously 
> written (with or without the restart).
> 
> This is all just normal eventual consistency stuff, but be sure you 
> understand it - rf3 may be a better choice
> 
> On restart, be sure you shut down cleanly - nodetool flush and then 
> immediately nodetool drain.  Beyond that I’d expect you to be fine.
> 
> -- 
> Jeff Jirsa
> 
> 
>> On Mar 8, 2018, at 9:52 PM, Eunsu Kim  wrote:
>> 
>> There are currently 5 writes per second. I was worried that the server 
>> downtime would be quite long during disk mount operations.
>> If the data shuffling that occurs when the server goes down or up is working 
>> as expected, I seem to be an unnecessary concern.
>> 
>> 
>>> On 9 Mar 2018, at 2:19 PM, Jeff Jirsa  wrote:
>>> 
>>> I see no reason to believe you’d lose data doing this - why do you suspect 
>>> you may? 
>>> 
>>> -- 
>>> Jeff Jirsa
>>> 
>>> 
 On Mar 8, 2018, at 8:36 PM, Eunsu Kim  wrote:
 
 The auto_snapshot setting is disabled. And the directory architecture on 
 the five nodes will match exactly.
 
 (Cassandra/Server shutdown -> Mount disk -> Add directory to 
 data_file_directories -> Start Cassandra) * 5 rolling
 
 Is it possible to add disks without losing data by doing the above 
 procedure?
 
 
 
> On 7 Mar 2018, at 7:59 PM, Rahul Singh  
> wrote:
> 
> Are you putting both the commitlogs and the Sstables on the adds? 
> Consider moving your snapshots often if that’s also taking up space. 
> Maybe able to save some space before you add drives.
> 
> You should be able to add these new drives and mount them without an 
> issue. Try to avoid different number of data dirs across nodes. It makes 
> automation of operational processes a little harder.
> 
> As an aside, Depending on your usecase you may not want to have a data 
> density over 1.5 TB per node.
> 
> --
> Rahul Singh
> rahul.si...@anant.us
> 
> Anant Corporation
> 
>> On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , wrote:
>> Hello,
>> 
>> I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
>> 
>> I'm trying to mount an additional disk(SSD 1TB) on each node because 
>> each disk usage growth rate is higher than I expected. Then I will add 
>> the the directory to data_file_directories in cassanra.yaml
>> 
>> Can I get advice from who have experienced this situation?
>> If we go through the above steps one by one, will we be able to complete 
>> the upgrade without losing data?
>> The replication strategy is SimpleStrategy, RF 2.
>> 
>> Thank you in advance
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
 
 
 -
 To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: user-h...@cassandra.apache.org
 
>>> 
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Adding disk to operating C*

[Jeff Jirsa]

There is no shuffling as the servers go up and down. Cassandra doesn’t do that. 

However, rf=2 is atypical and sometime problematic.

If you read or write with quorum / two / all, you’ll get unavailables during 
the restart

If you read or write with cl one, you’ll potentially not see data previously 
written (with or without the restart).

This is all just normal eventual consistency stuff, but be sure you understand 
it - rf3 may be a better choice

On restart, be sure you shut down cleanly - nodetool flush and then immediately 
nodetool drain.  Beyond that I’d expect you to be fine.

-- 
Jeff Jirsa


> On Mar 8, 2018, at 9:52 PM, Eunsu Kim  wrote:
> 
> There are currently 5 writes per second. I was worried that the server 
> downtime would be quite long during disk mount operations.
> If the data shuffling that occurs when the server goes down or up is working 
> as expected, I seem to be an unnecessary concern.
> 
> 
>> On 9 Mar 2018, at 2:19 PM, Jeff Jirsa  wrote:
>> 
>> I see no reason to believe you’d lose data doing this - why do you suspect 
>> you may? 
>> 
>> -- 
>> Jeff Jirsa
>> 
>> 
>>> On Mar 8, 2018, at 8:36 PM, Eunsu Kim  wrote:
>>> 
>>> The auto_snapshot setting is disabled. And the directory architecture on 
>>> the five nodes will match exactly.
>>> 
>>> (Cassandra/Server shutdown -> Mount disk -> Add directory to 
>>> data_file_directories -> Start Cassandra) * 5 rolling
>>> 
>>> Is it possible to add disks without losing data by doing the above 
>>> procedure?
>>> 
>>> 
>>> 
 On 7 Mar 2018, at 7:59 PM, Rahul Singh  
 wrote:
 
 Are you putting both the commitlogs and the Sstables on the adds? Consider 
 moving your snapshots often if that’s also taking up space. Maybe able to 
 save some space before you add drives.
 
 You should be able to add these new drives and mount them without an 
 issue. Try to avoid different number of data dirs across nodes. It makes 
 automation of operational processes a little harder.
 
 As an aside, Depending on your usecase you may not want to have a data 
 density over 1.5 TB per node.
 
 --
 Rahul Singh
 rahul.si...@anant.us
 
 Anant Corporation
 
> On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , wrote:
> Hello,
> 
> I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
> 
> I'm trying to mount an additional disk(SSD 1TB) on each node because each 
> disk usage growth rate is higher than I expected. Then I will add the the 
> directory to data_file_directories in cassanra.yaml
> 
> Can I get advice from who have experienced this situation?
> If we go through the above steps one by one, will we be able to complete 
> the upgrade without losing data?
> The replication strategy is SimpleStrategy, RF 2.
> 
> Thank you in advance
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Adding disk to operating C*

[Eunsu Kim]

There are currently 5 writes per second. I was worried that the server 
downtime would be quite long during disk mount operations.
If the data shuffling that occurs when the server goes down or up is working as 
expected, I seem to be an unnecessary concern.


> On 9 Mar 2018, at 2:19 PM, Jeff Jirsa  wrote:
> 
> I see no reason to believe you’d lose data doing this - why do you suspect 
> you may? 
> 
> -- 
> Jeff Jirsa
> 
> 
>> On Mar 8, 2018, at 8:36 PM, Eunsu Kim  wrote:
>> 
>> The auto_snapshot setting is disabled. And the directory architecture on the 
>> five nodes will match exactly.
>> 
>> (Cassandra/Server shutdown -> Mount disk -> Add directory to 
>> data_file_directories -> Start Cassandra) * 5 rolling
>> 
>> Is it possible to add disks without losing data by doing the above procedure?
>> 
>> 
>> 
>>> On 7 Mar 2018, at 7:59 PM, Rahul Singh  wrote:
>>> 
>>> Are you putting both the commitlogs and the Sstables on the adds? Consider 
>>> moving your snapshots often if that’s also taking up space. Maybe able to 
>>> save some space before you add drives.
>>> 
>>> You should be able to add these new drives and mount them without an issue. 
>>> Try to avoid different number of data dirs across nodes. It makes 
>>> automation of operational processes a little harder.
>>> 
>>> As an aside, Depending on your usecase you may not want to have a data 
>>> density over 1.5 TB per node.
>>> 
>>> --
>>> Rahul Singh
>>> rahul.si...@anant.us
>>> 
>>> Anant Corporation
>>> 
 On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , wrote:
 Hello,
 
 I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
 
 I'm trying to mount an additional disk(SSD 1TB) on each node because each 
 disk usage growth rate is higher than I expected. Then I will add the the 
 directory to data_file_directories in cassanra.yaml
 
 Can I get advice from who have experienced this situation?
 If we go through the above steps one by one, will we be able to complete 
 the upgrade without losing data?
 The replication strategy is SimpleStrategy, RF 2.
 
 Thank you in advance
 -
 To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: user-h...@cassandra.apache.org
 
>> 
>> 
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Adding disk to operating C*

[Jeff Jirsa]

I see no reason to believe you’d lose data doing this - why do you suspect you 
may? 

-- 
Jeff Jirsa


> On Mar 8, 2018, at 8:36 PM, Eunsu Kim  wrote:
> 
> The auto_snapshot setting is disabled. And the directory architecture on the 
> five nodes will match exactly.
> 
> (Cassandra/Server shutdown -> Mount disk -> Add directory to 
> data_file_directories -> Start Cassandra) * 5 rolling
> 
> Is it possible to add disks without losing data by doing the above procedure?
> 
> 
> 
>> On 7 Mar 2018, at 7:59 PM, Rahul Singh  wrote:
>> 
>> Are you putting both the commitlogs and the Sstables on the adds? Consider 
>> moving your snapshots often if that’s also taking up space. Maybe able to 
>> save some space before you add drives.
>> 
>> You should be able to add these new drives and mount them without an issue. 
>> Try to avoid different number of data dirs across nodes. It makes automation 
>> of operational processes a little harder.
>> 
>> As an aside, Depending on your usecase you may not want to have a data 
>> density over 1.5 TB per node.
>> 
>> --
>> Rahul Singh
>> rahul.si...@anant.us
>> 
>> Anant Corporation
>> 
>>> On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , wrote:
>>> Hello,
>>> 
>>> I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
>>> 
>>> I'm trying to mount an additional disk(SSD 1TB) on each node because each 
>>> disk usage growth rate is higher than I expected. Then I will add the the 
>>> directory to data_file_directories in cassanra.yaml
>>> 
>>> Can I get advice from who have experienced this situation?
>>> If we go through the above steps one by one, will we be able to complete 
>>> the upgrade without losing data?
>>> The replication strategy is SimpleStrategy, RF 2.
>>> 
>>> Thank you in advance
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>> 
> 
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Adding disk to operating C*

[Eunsu Kim]

The auto_snapshot setting is disabled. And the directory architecture on the 
five nodes will match exactly.

 (Cassandra/Server shutdown -> Mount disk -> Add directory to 
data_file_directories -> Start Cassandra) * 5 rolling

Is it possible to add disks without losing data by doing the above procedure?



> On 7 Mar 2018, at 7:59 PM, Rahul Singh  wrote:
> 
> Are you putting both the commitlogs and the Sstables on the adds? Consider 
> moving your snapshots often if that’s also taking up space. Maybe able to 
> save some space before you add drives.
> 
> You should be able to add these new drives and mount them without an issue. 
> Try to avoid different number of data dirs across nodes. It makes automation 
> of operational processes a little harder.
> 
> As an aside, Depending on your usecase you may not want to have a data 
> density over 1.5 TB per node.
> 
> --
> Rahul Singh
> rahul.si...@anant.us
> 
> Anant Corporation
> 
> On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , wrote:
>> Hello,
>> 
>> I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
>> 
>> I'm trying to mount an additional disk(SSD 1TB) on each node because each 
>> disk usage growth rate is higher than I expected. Then I will add the the 
>> directory to data_file_directories in cassanra.yaml
>> 
>> Can I get advice from who have experienced this situation?
>> If we go through the above steps one by one, will we be able to complete the 
>> upgrade without losing data?
>> The replication strategy is SimpleStrategy, RF 2.
>> 
>> Thank you in advance
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Adding disk to operating C*

[Rahul Singh]

Are you putting both the commitlogs and the Sstables on the adds? Consider 
moving your snapshots often if that’s also taking up space. Maybe able to save 
some space before you add drives.

You should be able to add these new drives and mount them without an issue. Try 
to avoid different number of data dirs across nodes. It makes automation of 
operational processes a little harder.

As an aside, Depending on your usecase you may not want to have a data density 
over 1.5 TB per node.

--
Rahul Singh
rahul.si...@anant.us

Anant Corporation

On Mar 7, 2018, 1:26 AM -0500, Eunsu Kim , wrote:
> Hello,
>
> I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)
>
> I'm trying to mount an additional disk(SSD 1TB) on each node because each 
> disk usage growth rate is higher than I expected. Then I will add the the 
> directory to data_file_directories in cassanra.yaml
>
> Can I get advice from who have experienced this situation?
> If we go through the above steps one by one, will we be able to complete the 
> upgrade without losing data?
> The replication strategy is SimpleStrategy, RF 2.
>
> Thank you in advance
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>

Adding disk to operating C*

[Eunsu Kim]

Hello,

I use 5 nodes to create a cluster of Cassandra. (SSD 1TB)

I'm trying to mount an additional disk(SSD 1TB) on each node because each disk 
usage growth rate is higher than I expected. Then I will add the the directory 
to data_file_directories in cassanra.yaml

Can I get advice from who have experienced this situation?
If we go through the above steps one by one, will we be able to complete the 
upgrade without losing data?
The replication strategy is SimpleStrategy, RF 2.

Thank you in advance
-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org