Re: EC2 instance recommendations

[John Wong]

We run on both ephemeral and persistent on AWS. Ephemeral storage is the
local storage attached to the server host. We don't have extreme write &
read, so EBS is fine.

If youever shut down the EC2 instance, your data is guaranteed to be gone
because AWS moves your VM to another host after every shut down. Otherwise,
you can be certain with 99% of the time your data is safe with soft reboot
UNTIL you messed up your OS. In that case, your system is down, even though
the instance has a 100% uptime. Then you are pretty much out of luck
because now you can't access you ephemeral storage. (you have to contact
AWS support see if they can help you secure that data, but my bet is they
would hesitant to do that for you for security reason).

We run backup hourly by taking a snapshot, and use rsync to copy the files
over to a separate server. Since snapshots are hardlinks, we use this to
our benefit. So snapshot 1 and snapshot 2 may only have 1 file in
difference then we only transfer 1 file, while the other N files are shared
via hardlink.

As with every technology, whether it's EBS or local storage, data
corruption is an inevitable risk, but it's a very rare occasion for a
customer to find EBS corruption.

There are a number of tools available to do backup you can look it up.

On Tue, May 23, 2017 at 6:55 PM, Gopal, Dhruva 
wrote:

> By that do you mean it’s like bootstrapping a node if it fails or is
> shutdown and with a RF that is 2 or higher, data will get replicated when
> it’s brought up?
>
>
>
> *From: *Cogumelos Maravilha 
> *Date: *Tuesday, May 23, 2017 at 1:52 PM
> *To: *"user@cassandra.apache.org" 
>
> *Subject: *Re: EC2 instance recommendations
>
>
>
> Yes we can only reboot.
>
> But using rf=2 or higher it's only a node fresh restart.
>
> EBS is a network attached disk. Spinning disk or SSD is almost the same.
>
> It's better take the "risk" and use type i instances.
>
> Cheers.
>
>
>
> On 23-05-2017 21:39, sfesc...@gmail.com wrote:
>
> I think this is overstating it. If the instance ever stops you'll lose the
> data. That means if the server crashes for example, or if Amazon decides
> your instance requires maintenance.
>
>
>
> On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva 
> wrote:
>
> Thanks! So, I assume that as long we make sure we never explicitly
> “shutdown” the instance, we are good. Are you also saying we won’t be able
> to snapshot a directory with ephemeral storage and that is why EBS is
> better? We’re just finding that to get a reasonable amount of IOPS (gp2)
> out of EBS at a reasonable rate, it gets more expensive than an I3.
>
>
>
> *From: *Jonathan Haddad 
> *Date: *Tuesday, May 23, 2017 at 9:42 AM
> *To: *"Gopal, Dhruva"  ,
> Matija Gobec , Bhuvan Rawal 
> *Cc: *"user@cassandra.apache.org" 
>
>
> *Subject: *Re: EC2 instance recommendations
>
>
>
> > Oh, so all the data is lost if the instance is shutdown or restarted
> (for that instance)?
>
>
>
> When you restart the OS, you're technically not shutting down the
> instance.  As long as the instance isn't stopped / terminated, your data is
> fine.  I ran my databases on ephemeral storage for years without issue.  In
> general, ephemeral storage is going to give you lower latency since there's
> no network overhead.  EBS is generally cheaper than ephemeral, is
> persistent, and you can take snapshots easily.
>
>
>
> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva 
> wrote:
>
> Oh, so all the data is lost if the instance is shutdown or restarted (for
> that instance)? If we take a naïve approach to backing up the directory,
> and restoring it, if we ever have to bring down the instance and back up,
> will that work as a strategy? Data is only kept around for 2 days and is
> TTL’d after.
>
>
>
> *From: *Matija Gobec 
> *Date: *Tuesday, May 23, 2017 at 8:15 AM
> *To: *Bhuvan Rawal 
> *Cc: *"Gopal, Dhruva"  ,
> "user@cassandra.apache.org" 
> *Subject: *Re: EC2 instance recommendations
>
>
>
> We are running on I3s since they came out. NVMe SSDs are really fast and I
> managed to push them to 75k IOPs.
>
> As Bhuvan mentioned the i3 storage is ephemeral. If you can work around it
> and plan for failure recovery you are good to go.
>
>
>
> I ran Cassandra on m4s before and had no problems with EBS volumes (gp2)
> even in low latency use cases. With the cost of M4 instances and EBS
> volumes that make sense in IOPs, I would recommend going with more i3s and
> working around the ephemeral issue (if its an issue).
>
>
>
> Best,
>
> Matija
>
> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal  wrote:
>
> i3 instances will undoubtedly give you 

Re: memtable_allocation_type on Cassandra 2.1.x

[varun saluja]

Thanks Akhil for response.

I have set memtable_allocation_type as Off-heap. But cassandra 2.1.x does
not allow to set *memtable_heap_space_in_mb: 0.*

It mentions , we need to assign some positive value to heap space. In such
case, will memtable still use jvm heap space.

Can anyone suggest below parameters.

memtable_flush_writers:
memtable_cleanup_threshold:


PS : We have high write intensive workload . 5Node cluster (12 Core , 62GB
RAM and flash disk per node)


Regards,
Varun Saluja

On 23 May 2017 at 03:26, Akhil Mehra  wrote:

> I believe off-heap storage was reintroduced in 3.4 (
> https://issues.apache.org/jira/browse/CASSANDRA-9472). It was removed
> from 3.0 due to the refactoring of the storage engine.  Check out
> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1 to
> get an overview of the pros and cons of using off-heap storage.
>
> Regards,
> Akhil Mehra
>
>
> On Tue, May 23, 2017 at 12:32 AM, varun saluja  wrote:
>
>> Hi Experts,
>>
>> I have some concerns regarding memtable parameters for my current
>> version. 2.1.8.
>>
>> As per documentation , its mentioned to have Off-heap memtables in
>> Cassandra 2.1 . And in releases 3.2.0 and 3.2.1, the only option that works
>> is: heap-buffers.
>> Can you Please suggest what value should be use for below paramteres in
>> 2.1.x
>> memtable_allocation_type :
>>
>> Regards,
>> Varun Saluja
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

Re: Bottleneck for small inserts?

[Patrick McFadin]

When you are running a stress test, 1-1 match client to server won't
saturate a cluster. I would go closer to 3-5 clients per server, so 10-15
clients against your 3 node cluster.

Patrick

On Tue, May 23, 2017 at 4:18 PM, Jeff Jirsa  wrote:

>
> Are the 3 sending clients maxed out?
> Are you seeing JVM GC pauses?
>
>
> On 2017-05-22 14:02 (-0700), Eric Pederson  wrote:
> > Hi all:
> >
> > I'm new to Cassandra and I'm doing some performance testing.  One of
> things
> > that I'm testing is ingestion throughput.   My server setup is:
> >
> >- 3 node cluster
> >- SSD data (both commit log and sstables are on the same disk)
> >- 64 GB RAM per server
> >- 48 cores per server
> >- Cassandra 3.0.11
> >- 48 Gb heap using G1GC
> >- 1 Gbps NICs
> >
> > Since I'm using SSD I've tried tuning the following (one at a time) but
> > none seemed to make a lot of difference:
> >
> >- concurrent_writes=384
> >- memtable_flush_writers=8
> >- concurrent_compactors=8
> >
> > I am currently doing ingestion tests sending data from 3 clients on the
> > same subnet.  I am using cassandra-stress to do some ingestion testing.
> > The tests are using CL=ONE and RF=2.
> >
> > Using cassandra-stress (3.10) I am able to saturate the disk using a
> large
> > enough column size and the standard five column cassandra-stress schema.
> > For example, -col size=fixed(400) will saturate the disk and compactions
> > will start falling behind.
> >
> > One of our main tables has a row size that approximately 200 bytes,
> across
> > 64 columns.  When ingesting this table I don't see any resource
> > saturation.  Disk utilization is around 10-15% per iostat.  Incoming
> > network traffic on the servers is around 100-300 Mbps.  CPU utilization
> is
> > around 20-70%.  nodetool tpstats shows mostly zeros with occasional
> spikes
> > around 500 in MutationStage.
> >
> > The stress run does 10,000,000 inserts per client, each with a separate
> > range of partition IDs.  The run with 200 byte rows takes about 4
> minutes,
> > with mean Latency 4.5ms, Total GC time of 21 secs, Avg GC time 173 ms.
> >
> > The overall performance is good - around 120k rows/sec ingested.  But I'm
> > curious to know where the bottleneck is.  There's no resource saturation
> and
> > nodetool tpstats shows only occasional brief queueing.  Is the rest just
> > expected latency inside of Cassandra?
> >
> > Thanks,
> >
> > -- Eric
> >
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Bottleneck for small inserts?

[Jeff Jirsa]

Are the 3 sending clients maxed out?
Are you seeing JVM GC pauses?


On 2017-05-22 14:02 (-0700), Eric Pederson  wrote: 
> Hi all:
> 
> I'm new to Cassandra and I'm doing some performance testing.  One of things
> that I'm testing is ingestion throughput.   My server setup is:
> 
>- 3 node cluster
>- SSD data (both commit log and sstables are on the same disk)
>- 64 GB RAM per server
>- 48 cores per server
>- Cassandra 3.0.11
>- 48 Gb heap using G1GC
>- 1 Gbps NICs
> 
> Since I'm using SSD I've tried tuning the following (one at a time) but
> none seemed to make a lot of difference:
> 
>- concurrent_writes=384
>- memtable_flush_writers=8
>- concurrent_compactors=8
> 
> I am currently doing ingestion tests sending data from 3 clients on the
> same subnet.  I am using cassandra-stress to do some ingestion testing.
> The tests are using CL=ONE and RF=2.
> 
> Using cassandra-stress (3.10) I am able to saturate the disk using a large
> enough column size and the standard five column cassandra-stress schema.
> For example, -col size=fixed(400) will saturate the disk and compactions
> will start falling behind.
> 
> One of our main tables has a row size that approximately 200 bytes, across
> 64 columns.  When ingesting this table I don't see any resource
> saturation.  Disk utilization is around 10-15% per iostat.  Incoming
> network traffic on the servers is around 100-300 Mbps.  CPU utilization is
> around 20-70%.  nodetool tpstats shows mostly zeros with occasional spikes
> around 500 in MutationStage.
> 
> The stress run does 10,000,000 inserts per client, each with a separate
> range of partition IDs.  The run with 200 byte rows takes about 4 minutes,
> with mean Latency 4.5ms, Total GC time of 21 secs, Avg GC time 173 ms.
> 
> The overall performance is good - around 120k rows/sec ingested.  But I'm
> curious to know where the bottleneck is.  There's no resource saturation and
> nodetool tpstats shows only occasional brief queueing.  Is the rest just
> expected latency inside of Cassandra?
> 
> Thanks,
> 
> -- Eric
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: EC2 instance recommendations

[Gopal, Dhruva]

By that do you mean it’s like bootstrapping a node if it fails or is shutdown 
and with a RF that is 2 or higher, data will get replicated when it’s brought 
up?

From: Cogumelos Maravilha 
Date: Tuesday, May 23, 2017 at 1:52 PM
To: "user@cassandra.apache.org" 
Subject: Re: EC2 instance recommendations


Yes we can only reboot.

But using rf=2 or higher it's only a node fresh restart.

EBS is a network attached disk. Spinning disk or SSD is almost the same.

It's better take the "risk" and use type i instances.

Cheers.

On 23-05-2017 21:39, sfesc...@gmail.com wrote:
I think this is overstating it. If the instance ever stops you'll lose the 
data. That means if the server crashes for example, or if Amazon decides your 
instance requires maintenance.

On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva 
> wrote:
Thanks! So, I assume that as long we make sure we never explicitly “shutdown” 
the instance, we are good. Are you also saying we won’t be able to snapshot a 
directory with ephemeral storage and that is why EBS is better? We’re just 
finding that to get a reasonable amount of IOPS (gp2) out of EBS at a 
reasonable rate, it gets more expensive than an I3.

From: Jonathan Haddad >
Date: Tuesday, May 23, 2017 at 9:42 AM
To: "Gopal, Dhruva" , 
Matija Gobec >, Bhuvan Rawal 
>
Cc: "user@cassandra.apache.org" 
>

Subject: Re: EC2 instance recommendations

> Oh, so all the data is lost if the instance is shutdown or restarted (for 
> that instance)?

When you restart the OS, you're technically not shutting down the instance.  As 
long as the instance isn't stopped / terminated, your data is fine.  I ran my 
databases on ephemeral storage for years without issue.  In general, ephemeral 
storage is going to give you lower latency since there's no network overhead.  
EBS is generally cheaper than ephemeral, is persistent, and you can take 
snapshots easily.

On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva 
> wrote:
Oh, so all the data is lost if the instance is shutdown or restarted (for that 
instance)? If we take a naïve approach to backing up the directory, and 
restoring it, if we ever have to bring down the instance and back up, will that 
work as a strategy? Data is only kept around for 2 days and is TTL’d after.

From: Matija Gobec >
Date: Tuesday, May 23, 2017 at 8:15 AM
To: Bhuvan Rawal >
Cc: "Gopal, Dhruva" , 
"user@cassandra.apache.org" 
>
Subject: Re: EC2 instance recommendations

We are running on I3s since they came out. NVMe SSDs are really fast and I 
managed to push them to 75k IOPs.
As Bhuvan mentioned the i3 storage is ephemeral. If you can work around it and 
plan for failure recovery you are good to go.

I ran Cassandra on m4s before and had no problems with EBS volumes (gp2) even 
in low latency use cases. With the cost of M4 instances and EBS volumes that 
make sense in IOPs, I would recommend going with more i3s and working around 
the ephemeral issue (if its an issue).

Best,
Matija
On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal 
> wrote:
i3 instances will undoubtedly give you more meat for buck - easily 40K+ iops 
whereas on the other hand EBS maxes out at 20K PIOPS which is highly expensive 
(at times they can cost you significantly more than cost of instance).
But they have ephemeral local storage and data is lost once instance is 
stopped, you need to be prudent in case of i series, it is generally used for 
large persistent caches.

Regards,
Bhuvan
On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
> wrote:
Hi –
  We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and have 
been comparing this to I-3.2xlarge, which seems more cost effective when 
dealing with this amount of storage and from an IOPS perspective. Does anyone 
have any recommendations/ on the I-3s and how it performs overall, compared to 
the M4 equivalent? On the surface, without us having taken it through its paces 
performance-wise, it does seem to be pretty powerful. We just ran through an 
exercise with a RAIDed 200 TB volume (as opposed to a non RAIDed 3 TB volume) 
and were seeing a 20-30% improvement with the RAIDed setup, on a 6 node 
Cassandra ring. Just looking for any 

Re: EC2 instance recommendations

[Cogumelos Maravilha]

Yes we can only reboot.

But using rf=2 or higher it's only a node fresh restart.

EBS is a network attached disk. Spinning disk or SSD is almost the same.

It's better take the "risk" and use type i instances.

Cheers.


On 23-05-2017 21:39, sfesc...@gmail.com wrote:
> I think this is overstating it. If the instance ever stops you'll lose
> the data. That means if the server crashes for example, or if Amazon
> decides your instance requires maintenance.
>
> On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva
> > wrote:
>
> Thanks! So, I assume that as long we make sure we never explicitly
> “shutdown” the instance, we are good. Are you also saying we won’t
> be able to snapshot a directory with ephemeral storage and that is
> why EBS is better? We’re just finding that to get a reasonable
> amount of IOPS (gp2) out of EBS at a reasonable rate, it gets more
> expensive than an I3.
>
>  
>
> *From: *Jonathan Haddad >
> *Date: *Tuesday, May 23, 2017 at 9:42 AM
> *To: *"Gopal, Dhruva" , Matija Gobec
> >, Bhuvan Rawal
> >
> *Cc: *"user@cassandra.apache.org
> "  >
>
>
> *Subject: *Re: EC2 instance recommendations
>
>  
>
> > Oh, so all the data is lost if the instance is shutdown or
> restarted (for that instance)? 
>
>  
>
> When you restart the OS, you're technically not shutting down the
> instance.  As long as the instance isn't stopped / terminated,
> your data is fine.  I ran my databases on ephemeral storage for
> years without issue.  In general, ephemeral storage is going to
> give you lower latency since there's no network overhead.  EBS is
> generally cheaper than ephemeral, is persistent, and you can take
> snapshots easily.
>
>  
>
> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva
> > wrote:
>
> Oh, so all the data is lost if the instance is shutdown or
> restarted (for that instance)? If we take a naïve approach to
> backing up the directory, and restoring it, if we ever have to
> bring down the instance and back up, will that work as a
> strategy? Data is only kept around for 2 days and is TTL’d after.
>
>  
>
> *From: *Matija Gobec  >
> *Date: *Tuesday, May 23, 2017 at 8:15 AM
> *To: *Bhuvan Rawal  >
> *Cc: *"Gopal, Dhruva" ,
> "user@cassandra.apache.org "
> >
> *Subject: *Re: EC2 instance recommendations
>
>  
>
> We are running on I3s since they came out. NVMe SSDs are
> really fast and I managed to push them to 75k IOPs.
>
> As Bhuvan mentioned the i3 storage is ephemeral. If you can
> work around it and plan for failure recovery you are good to go.
>
>  
>
> I ran Cassandra on m4s before and had no problems with EBS
> volumes (gp2) even in low latency use cases. With the cost of
> M4 instances and EBS volumes that make sense in IOPs, I would
> recommend going with more i3s and working around the ephemeral
> issue (if its an issue).
>
>  
>
> Best,
>
> Matija
>
> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal
> > wrote:
>
> i3 instances will undoubtedly give you more meat for buck
> - easily 40K+ iops whereas on the other hand EBS maxes out
> at 20K PIOPS which is highly expensive (at times they can
> cost you significantly more than cost of instance). 
>
> But they have ephemeral local storage and data is lost
> once instance is stopped, you need to be prudent in case
> of i series, it is generally used for large persistent caches.
>
>  
>
> Regards,
>
> Bhuvan
>
> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva
> >
> wrote:
>
> Hi –
>
>   We’ve been running M4.2xlarge EC2 instances with 2-3
> TB of storage and have been comparing this to
> I-3.2xlarge, which seems more cost effective when
> dealing with this amount of storage and from an IOPS
> perspective. Does anyone have any recommendations/ 

Re: EC2 instance recommendations

[sfesc...@gmail.com]

I think this is overstating it. If the instance ever stops you'll lose the
data. That means if the server crashes for example, or if Amazon decides
your instance requires maintenance.

On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva 
wrote:

> Thanks! So, I assume that as long we make sure we never explicitly
> “shutdown” the instance, we are good. Are you also saying we won’t be able
> to snapshot a directory with ephemeral storage and that is why EBS is
> better? We’re just finding that to get a reasonable amount of IOPS (gp2)
> out of EBS at a reasonable rate, it gets more expensive than an I3.
>
>
>
> *From: *Jonathan Haddad 
> *Date: *Tuesday, May 23, 2017 at 9:42 AM
> *To: *"Gopal, Dhruva" , Matija Gobec <
> matija0...@gmail.com>, Bhuvan Rawal 
> *Cc: *"user@cassandra.apache.org" 
>
>
> *Subject: *Re: EC2 instance recommendations
>
>
>
> > Oh, so all the data is lost if the instance is shutdown or restarted
> (for that instance)?
>
>
>
> When you restart the OS, you're technically not shutting down the
> instance.  As long as the instance isn't stopped / terminated, your data is
> fine.  I ran my databases on ephemeral storage for years without issue.  In
> general, ephemeral storage is going to give you lower latency since there's
> no network overhead.  EBS is generally cheaper than ephemeral, is
> persistent, and you can take snapshots easily.
>
>
>
> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva 
> wrote:
>
> Oh, so all the data is lost if the instance is shutdown or restarted (for
> that instance)? If we take a naïve approach to backing up the directory,
> and restoring it, if we ever have to bring down the instance and back up,
> will that work as a strategy? Data is only kept around for 2 days and is
> TTL’d after.
>
>
>
> *From: *Matija Gobec 
> *Date: *Tuesday, May 23, 2017 at 8:15 AM
> *To: *Bhuvan Rawal 
> *Cc: *"Gopal, Dhruva" , "
> user@cassandra.apache.org" 
> *Subject: *Re: EC2 instance recommendations
>
>
>
> We are running on I3s since they came out. NVMe SSDs are really fast and I
> managed to push them to 75k IOPs.
>
> As Bhuvan mentioned the i3 storage is ephemeral. If you can work around it
> and plan for failure recovery you are good to go.
>
>
>
> I ran Cassandra on m4s before and had no problems with EBS volumes (gp2)
> even in low latency use cases. With the cost of M4 instances and EBS
> volumes that make sense in IOPs, I would recommend going with more i3s and
> working around the ephemeral issue (if its an issue).
>
>
>
> Best,
>
> Matija
>
> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal  wrote:
>
> i3 instances will undoubtedly give you more meat for buck - easily 40K+
> iops whereas on the other hand EBS maxes out at 20K PIOPS which is highly
> expensive (at times they can cost you significantly more than cost of
> instance).
>
> But they have ephemeral local storage and data is lost once instance is
> stopped, you need to be prudent in case of i series, it is generally used
> for large persistent caches.
>
>
>
> Regards,
>
> Bhuvan
>
> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
> wrote:
>
> Hi –
>
>   We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and
> have been comparing this to I-3.2xlarge, which seems more cost effective
> when dealing with this amount of storage and from an IOPS perspective. Does
> anyone have any recommendations/ on the I-3s and how it performs overall,
> compared to the M4 equivalent? On the surface, without us having taken it
> through its paces performance-wise, it does seem to be pretty powerful. We
> just ran through an exercise with a RAIDed 200 TB volume (as opposed to a
> non RAIDed 3 TB volume) and were seeing a 20-30% improvement with the
> RAIDed setup, on a 6 node Cassandra ring. Just looking for any
> feedback/experience folks may have had with the I-3s.
>
>
>
> Regards,
>
> *DHRUVA GOPAL*
>
> *sr. MANAGER, ENGINEERING*
>
> *REPORTING, ANALYTICS AND BIG DATA*
>
> *+1 408.325.2011 <+1%20408-325-2011>* *WORK*
>
> *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE*
>
> *UNITED STATES*
>
> *dhruva.go...@aspect.com  *
>
> *aspect.com *
>
> [image: cription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]
>
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may 

Re: InternalResponseStage low on some nodes

[Cogumelos Maravilha]

This is really atypical.

What about nodetool compactionstats?

crontab jobs in each node like nodetool repair, etc?

Also security these 2 nodes have the same ports open?

Same configuration, same JVM params?

nodetool ring normal?

Cheers.


On 23-05-2017 20:11, Andrew Jorgensen wrote:
> Hello,
>
> I will preface this and say that all of the nodes have been running
> for about the same amount of time and were not restarted before
> running nodetool tpstats.
>
> This is more for my understanding that anything else but I have a 20
> node cassandra cluster running cassandra 3.0.3. I have 0 read and 0
> writes going to the cluster and I see a strange banding of load
> average for some of the nodes.
>
> 18 out of 20 of the nodes sit around an 15 min LA of 0.3 while 2 nodes
> are at 0.08. Once I applied writes to the cluster all of the load
> averages increased by the same proportion, so 18 out of 20 nodes
> increased to ~0.4 and 2 nodes increased to 0.1
>
> When I look at what is different between these nodes, all of which
> have been running for the same amount of time, the only numerical
> difference in nodetool tpstats is the InternalResponseStage. 18 out of
> 20 of the nodes are in the range of 20,000 completed while 2 are only
> at 300. Interestingly it also looks like the 2 nodes that are on the
> low side are exhibiting symptoms
> of https://issues.apache.org/jira/browse/CASSANDRA-11090.
>
> I restarted one of the low nodes and it immediately jumped up to match
> the other 18 nodes in the cluster and settled around 0.4 15 minute LA.
>
> I am curious of two things, first why the InternalResponseStage is so
> low on two of the nodes in the cluster and what that means. Second is
> it atypical for the other 18 nodes in the cluster to have such a high
> number of InternalResponseStage completed tasks or do these numbers
> seems reasonable for a idle cluster?
>
> Thanks!
> Andrew Jorgensen
> @ajorgensen

InternalResponseStage low on some nodes

[Andrew Jorgensen]

Hello,

I will preface this and say that all of the nodes have been running for
about the same amount of time and were not restarted before running
nodetool tpstats.

This is more for my understanding that anything else but I have a 20 node
cassandra cluster running cassandra 3.0.3. I have 0 read and 0 writes going
to the cluster and I see a strange banding of load average for some of the
nodes.

18 out of 20 of the nodes sit around an 15 min LA of 0.3 while 2 nodes are
at 0.08. Once I applied writes to the cluster all of the load averages
increased by the same proportion, so 18 out of 20 nodes increased to ~0.4
and 2 nodes increased to 0.1

When I look at what is different between these nodes, all of which have
been running for the same amount of time, the only numerical difference in
nodetool tpstats is the InternalResponseStage. 18 out of 20 of the nodes
are in the range of 20,000 completed while 2 are only at 300. Interestingly
it also looks like the 2 nodes that are on the low side are exhibiting
symptoms of https://issues.apache.org/jira/browse/CASSANDRA-11090.

I restarted one of the low nodes and it immediately jumped up to match the
other 18 nodes in the cluster and settled around 0.4 15 minute LA.

I am curious of two things, first why the InternalResponseStage is so low
on two of the nodes in the cluster and what that means. Second is it
atypical for the other 18 nodes in the cluster to have such a high number
of InternalResponseStage completed tasks or do these numbers seems
reasonable for a idle cluster?

Thanks!
Andrew Jorgensen
@ajorgensen

Re: Slowness in C* cluster after implementing multiple network interface configuration.

[Cogumelos Maravilha]

Hi,

I never used version 2.0.x but I think port 7000 isn't enough.

Try enable:

7000 inter-node

7001 SSL inter-node

9042 CQL

9160 Thrift is enable in that version


And

**In Cassandra.yaml, add property “broadcast_address”.  = local ipv4

**In Cassandra.yaml, change “listen_address” to private IP. = local ipv4


As a starting point.


Cheers.


On 22-05-2017 12:36, Prakash Chauhan wrote:
>
> Hi All ,
>
>  
>
> Need Help !!!
>
>  
>
> *Setup Details:*
>
> Cassandra 2.0.14
>
> Geo Red setup
>
> · DC1 - 3 nodes
>
> · DC2 - 3 nodes
>
>  
>
>  
>
> We were trying to implement multiple network interfaces with Cassandra
> 2.0.14
>
> After doing all the steps mentioned in DataStax doc
> http://docs.datastax.com/en/archived/cassandra/2.0/cassandra/configuration/configMultiNetworks.html,
> we observed that nodes were not able to see each other (checked using
> nodetool status).
>
>  
>
> To resolve this issue, we followed the comment
> 
> mentioned in the JIRA : CASSANDRA-9748
> 
>
>  
>
> Exact steps that we followed are :
>
> 
>
> *1.   *Stop Cassandra
>
> *2.   *Add rule to “iptables” to forward all packets on the public
> interface to the private interface.
>
>  
>
> COMMAND: # iptables -t nat -A PREROUTING -p tcp -m tcp -d
>  --dport 7000 -j DNAT --to-destination :7000
>
>  
>
> *3.   *In Cassandra.yaml, add property “broadcast_address”.
>
> *4.   *In Cassandra.yaml, change “listen_address” to private IP.
>
> *5.   *Clear the data from directory “peers”.
>
> *6.   *Change Snitch to GossipingPropertyFileSnitch.
>
> *7.   *Append following property to the file
> “/etc/cassandra/conf/cassandra-env.sh” to purge gossip state.
>
> JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"
>
>  
>
> *8.   *Start Cassandra
>
> *9.   *After node has been started, remove following property from
> the file “/etc/cassandra/conf/cassandra-env.sh” (previously added in
> step 7)
>
> JVM_OPTS="$JVM_OPTS -Dcassandra.load_ring_state=false"
>
> *10.   *Delete file “/etc/cassandra/conf/cassandra-topology.properties”
>
>  
>
>  
>
> Now We have an observation that after multiple restarts of Cassandra
> on multiple nodes, slowness is observed in the cluster.
>
> The problem gets resolved when we revert the steps mentioned above.
>
>  
>
> *Do u think there is any step that can cause the problem ?*
>
> We are suspecting Step 2(iptable rule) but not very sure about it.
>
>  
>
>  
>
> Regards,
>
> Prakash Chauhan.
>

Re: EC2 instance recommendations

[Dor Laor]

Note that EBS durability isn't perfect, you cannot rely on them entirely:
https://aws.amazon.com/ebs/details/
"Amazon EBS volumes are designed for an annual failure rate (AFR) of
between 0.1% - 0.2%, where failure refers to a complete or partial loss of
the volume, depending on the size and performance of the volume. This makes
EBS volumes 20 times more reliable than typical commodity disk drives"
It's rare but once in awhile we get an email notification that one of our
EBS drives was corrupted.
In this case a combined write through cache and EBS is certainly better
than EBS w/o backup.



On Tue, May 23, 2017 at 10:40 AM, Jonathan Haddad  wrote:

> Another option that I like the idea of but never see used unfortunately is
> using ZFS, with EBS for storage and the SSD ephemeral drive as L2Arc.
> You'd get the performance of ephemeral storage with all the features of
> EBS.  Something to consider.
>
> On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva 
> wrote:
>
>> Thanks! So, I assume that as long we make sure we never explicitly
>> “shutdown” the instance, we are good. Are you also saying we won’t be able
>> to snapshot a directory with ephemeral storage and that is why EBS is
>> better? We’re just finding that to get a reasonable amount of IOPS (gp2)
>> out of EBS at a reasonable rate, it gets more expensive than an I3.
>>
>>
>>
>> *From: *Jonathan Haddad 
>>
>>
>> *Date: *Tuesday, May 23, 2017 at 9:42 AM
>> *To: *"Gopal, Dhruva" , Matija Gobec <
>> matija0...@gmail.com>, Bhuvan Rawal 
>>
>> *Cc: *"user@cassandra.apache.org" 
>>
>>
>> *Subject: *Re: EC2 instance recommendations
>>
>>
>>
>> > Oh, so all the data is lost if the instance is shutdown or restarted
>> (for that instance)?
>>
>>
>>
>> When you restart the OS, you're technically not shutting down the
>> instance.  As long as the instance isn't stopped / terminated, your data is
>> fine.  I ran my databases on ephemeral storage for years without issue.  In
>> general, ephemeral storage is going to give you lower latency since there's
>> no network overhead.  EBS is generally cheaper than ephemeral, is
>> persistent, and you can take snapshots easily.
>>
>>
>>
>> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva 
>> wrote:
>>
>> Oh, so all the data is lost if the instance is shutdown or restarted (for
>> that instance)? If we take a naïve approach to backing up the directory,
>> and restoring it, if we ever have to bring down the instance and back up,
>> will that work as a strategy? Data is only kept around for 2 days and is
>> TTL’d after.
>>
>>
>>
>> *From: *Matija Gobec 
>> *Date: *Tuesday, May 23, 2017 at 8:15 AM
>> *To: *Bhuvan Rawal 
>> *Cc: *"Gopal, Dhruva" , "
>> user@cassandra.apache.org" 
>> *Subject: *Re: EC2 instance recommendations
>>
>>
>>
>> We are running on I3s since they came out. NVMe SSDs are really fast and
>> I managed to push them to 75k IOPs.
>>
>> As Bhuvan mentioned the i3 storage is ephemeral. If you can work around
>> it and plan for failure recovery you are good to go.
>>
>>
>>
>> I ran Cassandra on m4s before and had no problems with EBS volumes (gp2)
>> even in low latency use cases. With the cost of M4 instances and EBS
>> volumes that make sense in IOPs, I would recommend going with more i3s and
>> working around the ephemeral issue (if its an issue).
>>
>>
>>
>> Best,
>>
>> Matija
>>
>> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal 
>> wrote:
>>
>> i3 instances will undoubtedly give you more meat for buck - easily 40K+
>> iops whereas on the other hand EBS maxes out at 20K PIOPS which is highly
>> expensive (at times they can cost you significantly more than cost of
>> instance).
>>
>> But they have ephemeral local storage and data is lost once instance is
>> stopped, you need to be prudent in case of i series, it is generally used
>> for large persistent caches.
>>
>>
>>
>> Regards,
>>
>> Bhuvan
>>
>> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
>> wrote:
>>
>> Hi –
>>
>>   We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and
>> have been comparing this to I-3.2xlarge, which seems more cost effective
>> when dealing with this amount of storage and from an IOPS perspective. Does
>> anyone have any recommendations/ on the I-3s and how it performs overall,
>> compared to the M4 equivalent? On the surface, without us having taken it
>> through its paces performance-wise, it does seem to be pretty powerful. We
>> just ran through an exercise with a RAIDed 200 TB volume (as opposed to a
>> non RAIDed 3 TB volume) and were seeing a 20-30% improvement with the
>> RAIDed setup, on a 6 node Cassandra ring. Just looking for any
>> feedback/experience folks may have had with the I-3s.
>>
>>
>>
>> Regards,
>>
>> 

Re: EC2 instance recommendations

[Jonathan Haddad]

Another option that I like the idea of but never see used unfortunately is
using ZFS, with EBS for storage and the SSD ephemeral drive as L2Arc.
You'd get the performance of ephemeral storage with all the features of
EBS.  Something to consider.

On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva 
wrote:

> Thanks! So, I assume that as long we make sure we never explicitly
> “shutdown” the instance, we are good. Are you also saying we won’t be able
> to snapshot a directory with ephemeral storage and that is why EBS is
> better? We’re just finding that to get a reasonable amount of IOPS (gp2)
> out of EBS at a reasonable rate, it gets more expensive than an I3.
>
>
>
> *From: *Jonathan Haddad 
>
>
> *Date: *Tuesday, May 23, 2017 at 9:42 AM
> *To: *"Gopal, Dhruva" , Matija Gobec <
> matija0...@gmail.com>, Bhuvan Rawal 
>
> *Cc: *"user@cassandra.apache.org" 
>
>
> *Subject: *Re: EC2 instance recommendations
>
>
>
> > Oh, so all the data is lost if the instance is shutdown or restarted
> (for that instance)?
>
>
>
> When you restart the OS, you're technically not shutting down the
> instance.  As long as the instance isn't stopped / terminated, your data is
> fine.  I ran my databases on ephemeral storage for years without issue.  In
> general, ephemeral storage is going to give you lower latency since there's
> no network overhead.  EBS is generally cheaper than ephemeral, is
> persistent, and you can take snapshots easily.
>
>
>
> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva 
> wrote:
>
> Oh, so all the data is lost if the instance is shutdown or restarted (for
> that instance)? If we take a naïve approach to backing up the directory,
> and restoring it, if we ever have to bring down the instance and back up,
> will that work as a strategy? Data is only kept around for 2 days and is
> TTL’d after.
>
>
>
> *From: *Matija Gobec 
> *Date: *Tuesday, May 23, 2017 at 8:15 AM
> *To: *Bhuvan Rawal 
> *Cc: *"Gopal, Dhruva" , "
> user@cassandra.apache.org" 
> *Subject: *Re: EC2 instance recommendations
>
>
>
> We are running on I3s since they came out. NVMe SSDs are really fast and I
> managed to push them to 75k IOPs.
>
> As Bhuvan mentioned the i3 storage is ephemeral. If you can work around it
> and plan for failure recovery you are good to go.
>
>
>
> I ran Cassandra on m4s before and had no problems with EBS volumes (gp2)
> even in low latency use cases. With the cost of M4 instances and EBS
> volumes that make sense in IOPs, I would recommend going with more i3s and
> working around the ephemeral issue (if its an issue).
>
>
>
> Best,
>
> Matija
>
> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal  wrote:
>
> i3 instances will undoubtedly give you more meat for buck - easily 40K+
> iops whereas on the other hand EBS maxes out at 20K PIOPS which is highly
> expensive (at times they can cost you significantly more than cost of
> instance).
>
> But they have ephemeral local storage and data is lost once instance is
> stopped, you need to be prudent in case of i series, it is generally used
> for large persistent caches.
>
>
>
> Regards,
>
> Bhuvan
>
> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
> wrote:
>
> Hi –
>
>   We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and
> have been comparing this to I-3.2xlarge, which seems more cost effective
> when dealing with this amount of storage and from an IOPS perspective. Does
> anyone have any recommendations/ on the I-3s and how it performs overall,
> compared to the M4 equivalent? On the surface, without us having taken it
> through its paces performance-wise, it does seem to be pretty powerful. We
> just ran through an exercise with a RAIDed 200 TB volume (as opposed to a
> non RAIDed 3 TB volume) and were seeing a 20-30% improvement with the
> RAIDed setup, on a 6 node Cassandra ring. Just looking for any
> feedback/experience folks may have had with the I-3s.
>
>
>
> Regards,
>
> *DHRUVA GOPAL*
>
> *sr. MANAGER, ENGINEERING*
>
> *REPORTING, ANALYTICS AND BIG DATA*
>
> *+1 408.325.2011 <+1%20408-325-2011>* *WORK*
>
> *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE*
>
> *UNITED STATES*
>
> *dhruva.go...@aspect.com  *
>
> *aspect.com *
>
> [image: cription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]
>
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> This email (including 

Re: EC2 instance recommendations

[Matija Gobec]

Exactly. You can easily build a solid backup/restore with snapshots and
automate it in case all hell breaks loose.
EBS volumes are expensive right now and with i3 you get much more IOPs and
a reasonable disk size for 1/2 or 1/3 the price.

Best,
Matija

On Tue, May 23, 2017 at 1:29 PM, Gopal, Dhruva 
wrote:

> Thanks! So, I assume that as long we make sure we never explicitly
> “shutdown” the instance, we are good. Are you also saying we won’t be able
> to snapshot a directory with ephemeral storage and that is why EBS is
> better? We’re just finding that to get a reasonable amount of IOPS (gp2)
> out of EBS at a reasonable rate, it gets more expensive than an I3.
>
>
>
> *From: *Jonathan Haddad 
> *Date: *Tuesday, May 23, 2017 at 9:42 AM
> *To: *"Gopal, Dhruva" , Matija Gobec <
> matija0...@gmail.com>, Bhuvan Rawal 
> *Cc: *"user@cassandra.apache.org" 
>
> *Subject: *Re: EC2 instance recommendations
>
>
>
> > Oh, so all the data is lost if the instance is shutdown or restarted
> (for that instance)?
>
>
>
> When you restart the OS, you're technically not shutting down the
> instance.  As long as the instance isn't stopped / terminated, your data is
> fine.  I ran my databases on ephemeral storage for years without issue.  In
> general, ephemeral storage is going to give you lower latency since there's
> no network overhead.  EBS is generally cheaper than ephemeral, is
> persistent, and you can take snapshots easily.
>
>
>
> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva 
> wrote:
>
> Oh, so all the data is lost if the instance is shutdown or restarted (for
> that instance)? If we take a naïve approach to backing up the directory,
> and restoring it, if we ever have to bring down the instance and back up,
> will that work as a strategy? Data is only kept around for 2 days and is
> TTL’d after.
>
>
>
> *From: *Matija Gobec 
> *Date: *Tuesday, May 23, 2017 at 8:15 AM
> *To: *Bhuvan Rawal 
> *Cc: *"Gopal, Dhruva" , "
> user@cassandra.apache.org" 
> *Subject: *Re: EC2 instance recommendations
>
>
>
> We are running on I3s since they came out. NVMe SSDs are really fast and I
> managed to push them to 75k IOPs.
>
> As Bhuvan mentioned the i3 storage is ephemeral. If you can work around it
> and plan for failure recovery you are good to go.
>
>
>
> I ran Cassandra on m4s before and had no problems with EBS volumes (gp2)
> even in low latency use cases. With the cost of M4 instances and EBS
> volumes that make sense in IOPs, I would recommend going with more i3s and
> working around the ephemeral issue (if its an issue).
>
>
>
> Best,
>
> Matija
>
> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal  wrote:
>
> i3 instances will undoubtedly give you more meat for buck - easily 40K+
> iops whereas on the other hand EBS maxes out at 20K PIOPS which is highly
> expensive (at times they can cost you significantly more than cost of
> instance).
>
> But they have ephemeral local storage and data is lost once instance is
> stopped, you need to be prudent in case of i series, it is generally used
> for large persistent caches.
>
>
>
> Regards,
>
> Bhuvan
>
> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
> wrote:
>
> Hi –
>
>   We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and
> have been comparing this to I-3.2xlarge, which seems more cost effective
> when dealing with this amount of storage and from an IOPS perspective. Does
> anyone have any recommendations/ on the I-3s and how it performs overall,
> compared to the M4 equivalent? On the surface, without us having taken it
> through its paces performance-wise, it does seem to be pretty powerful. We
> just ran through an exercise with a RAIDed 200 TB volume (as opposed to a
> non RAIDed 3 TB volume) and were seeing a 20-30% improvement with the
> RAIDed setup, on a 6 node Cassandra ring. Just looking for any
> feedback/experience folks may have had with the I-3s.
>
>
>
> Regards,
>
> *DHRUVA GOPAL*
>
> *sr. MANAGER, ENGINEERING*
>
> *REPORTING, ANALYTICS AND BIG DATA*
>
> *+1 408.325.2011 <+1%20408-325-2011>* *WORK*
>
> *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE*
>
> *UNITED STATES*
>
> *dhruva.go...@aspect.com  *
>
> *aspect.com *
>
> [image: cription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]
>
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> This email (including 

Re: EC2 instance recommendations

[Gopal, Dhruva]

Thanks! So, I assume that as long we make sure we never explicitly “shutdown” 
the instance, we are good. Are you also saying we won’t be able to snapshot a 
directory with ephemeral storage and that is why EBS is better? We’re just 
finding that to get a reasonable amount of IOPS (gp2) out of EBS at a 
reasonable rate, it gets more expensive than an I3.

From: Jonathan Haddad 
Date: Tuesday, May 23, 2017 at 9:42 AM
To: "Gopal, Dhruva" , Matija Gobec 
, Bhuvan Rawal 
Cc: "user@cassandra.apache.org" 
Subject: Re: EC2 instance recommendations

> Oh, so all the data is lost if the instance is shutdown or restarted (for 
> that instance)?

When you restart the OS, you're technically not shutting down the instance.  As 
long as the instance isn't stopped / terminated, your data is fine.  I ran my 
databases on ephemeral storage for years without issue.  In general, ephemeral 
storage is going to give you lower latency since there's no network overhead.  
EBS is generally cheaper than ephemeral, is persistent, and you can take 
snapshots easily.

On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva 
> wrote:
Oh, so all the data is lost if the instance is shutdown or restarted (for that 
instance)? If we take a naïve approach to backing up the directory, and 
restoring it, if we ever have to bring down the instance and back up, will that 
work as a strategy? Data is only kept around for 2 days and is TTL’d after.

From: Matija Gobec >
Date: Tuesday, May 23, 2017 at 8:15 AM
To: Bhuvan Rawal >
Cc: "Gopal, Dhruva" , 
"user@cassandra.apache.org" 
>
Subject: Re: EC2 instance recommendations

We are running on I3s since they came out. NVMe SSDs are really fast and I 
managed to push them to 75k IOPs.
As Bhuvan mentioned the i3 storage is ephemeral. If you can work around it and 
plan for failure recovery you are good to go.

I ran Cassandra on m4s before and had no problems with EBS volumes (gp2) even 
in low latency use cases. With the cost of M4 instances and EBS volumes that 
make sense in IOPs, I would recommend going with more i3s and working around 
the ephemeral issue (if its an issue).

Best,
Matija
On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal 
> wrote:
i3 instances will undoubtedly give you more meat for buck - easily 40K+ iops 
whereas on the other hand EBS maxes out at 20K PIOPS which is highly expensive 
(at times they can cost you significantly more than cost of instance).
But they have ephemeral local storage and data is lost once instance is 
stopped, you need to be prudent in case of i series, it is generally used for 
large persistent caches.

Regards,
Bhuvan
On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
> wrote:
Hi –
  We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and have 
been comparing this to I-3.2xlarge, which seems more cost effective when 
dealing with this amount of storage and from an IOPS perspective. Does anyone 
have any recommendations/ on the I-3s and how it performs overall, compared to 
the M4 equivalent? On the surface, without us having taken it through its paces 
performance-wise, it does seem to be pretty powerful. We just ran through an 
exercise with a RAIDed 200 TB volume (as opposed to a non RAIDed 3 TB volume) 
and were seeing a 20-30% improvement with the RAIDed setup, on a 6 node 
Cassandra ring. Just looking for any feedback/experience folks may have had 
with the I-3s.

Regards,
DHRUVA GOPAL
sr. MANAGER, ENGINEERING
REPORTING, ANALYTICS AND BIG DATA
+1 408.325.2011 WORK
+1 408.219.1094 MOBILE
UNITED STATES
dhruva.go...@aspect.com
aspect.com
[cription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.
This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.
This 

Re: EC2 instance recommendations

[sfesc...@gmail.com]

I'm experimenting with bcache to see about using the ephemeral storage as a
cache backed with EBS. Not sure if that makes sense in your use case though.

On Tue, May 23, 2017 at 9:43 AM Jonathan Haddad  wrote:

> > Oh, so all the data is lost if the instance is shutdown or restarted
> (for that instance)?
>
> When you restart the OS, you're technically not shutting down the
> instance.  As long as the instance isn't stopped / terminated, your data is
> fine.  I ran my databases on ephemeral storage for years without issue.  In
> general, ephemeral storage is going to give you lower latency since there's
> no network overhead.  EBS is generally cheaper than ephemeral, is
> persistent, and you can take snapshots easily.
>
> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva 
> wrote:
>
>> Oh, so all the data is lost if the instance is shutdown or restarted (for
>> that instance)? If we take a naïve approach to backing up the directory,
>> and restoring it, if we ever have to bring down the instance and back up,
>> will that work as a strategy? Data is only kept around for 2 days and is
>> TTL’d after.
>>
>>
>>
>> *From: *Matija Gobec 
>> *Date: *Tuesday, May 23, 2017 at 8:15 AM
>> *To: *Bhuvan Rawal 
>> *Cc: *"Gopal, Dhruva" , "
>> user@cassandra.apache.org" 
>> *Subject: *Re: EC2 instance recommendations
>>
>>
>>
>> We are running on I3s since they came out. NVMe SSDs are really fast and
>> I managed to push them to 75k IOPs.
>>
>> As Bhuvan mentioned the i3 storage is ephemeral. If you can work around
>> it and plan for failure recovery you are good to go.
>>
>>
>>
>> I ran Cassandra on m4s before and had no problems with EBS volumes (gp2)
>> even in low latency use cases. With the cost of M4 instances and EBS
>> volumes that make sense in IOPs, I would recommend going with more i3s and
>> working around the ephemeral issue (if its an issue).
>>
>>
>>
>> Best,
>>
>> Matija
>>
>> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal 
>> wrote:
>>
>> i3 instances will undoubtedly give you more meat for buck - easily 40K+
>> iops whereas on the other hand EBS maxes out at 20K PIOPS which is highly
>> expensive (at times they can cost you significantly more than cost of
>> instance).
>>
>> But they have ephemeral local storage and data is lost once instance is
>> stopped, you need to be prudent in case of i series, it is generally used
>> for large persistent caches.
>>
>>
>>
>> Regards,
>>
>> Bhuvan
>>
>> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
>> wrote:
>>
>> Hi –
>>
>>   We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and
>> have been comparing this to I-3.2xlarge, which seems more cost effective
>> when dealing with this amount of storage and from an IOPS perspective. Does
>> anyone have any recommendations/ on the I-3s and how it performs overall,
>> compared to the M4 equivalent? On the surface, without us having taken it
>> through its paces performance-wise, it does seem to be pretty powerful. We
>> just ran through an exercise with a RAIDed 200 TB volume (as opposed to a
>> non RAIDed 3 TB volume) and were seeing a 20-30% improvement with the
>> RAIDed setup, on a 6 node Cassandra ring. Just looking for any
>> feedback/experience folks may have had with the I-3s.
>>
>>
>>
>> Regards,
>>
>> *DHRUVA GOPAL*
>>
>> *sr. MANAGER, ENGINEERING*
>>
>> *REPORTING, ANALYTICS AND BIG DATA*
>>
>> *+1 408.325.2011 <+1%20408-325-2011>* *WORK*
>>
>> *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE*
>>
>> *UNITED STATES*
>>
>> *dhruva.go...@aspect.com  *
>>
>> *aspect.com *
>>
>> [image: scription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]
>>
>>
>>
>> This email (including any attachments) is proprietary to Aspect Software,
>> Inc. and may contain information that is confidential. If you have received
>> this message in error, please do not read, copy or forward this message.
>> Please notify the sender immediately, delete it from your system and
>> destroy any copies. You may not further disclose or distribute this email
>> or its attachments.
>>
>> This email (including any attachments) is proprietary to Aspect Software,
>> Inc. and may contain information that is confidential. If you have received
>> this message in error, please do not read, copy or forward this message.
>> Please notify the sender immediately, delete it from your system and
>> destroy any copies. You may not further disclose or distribute this email
>> or its attachments.
>>
>

Re: EC2 instance recommendations

[Jonathan Haddad]

> Oh, so all the data is lost if the instance is shutdown or restarted (for
that instance)?

When you restart the OS, you're technically not shutting down the
instance.  As long as the instance isn't stopped / terminated, your data is
fine.  I ran my databases on ephemeral storage for years without issue.  In
general, ephemeral storage is going to give you lower latency since there's
no network overhead.  EBS is generally cheaper than ephemeral, is
persistent, and you can take snapshots easily.

On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva 
wrote:

> Oh, so all the data is lost if the instance is shutdown or restarted (for
> that instance)? If we take a naïve approach to backing up the directory,
> and restoring it, if we ever have to bring down the instance and back up,
> will that work as a strategy? Data is only kept around for 2 days and is
> TTL’d after.
>
>
>
> *From: *Matija Gobec 
> *Date: *Tuesday, May 23, 2017 at 8:15 AM
> *To: *Bhuvan Rawal 
> *Cc: *"Gopal, Dhruva" , "
> user@cassandra.apache.org" 
> *Subject: *Re: EC2 instance recommendations
>
>
>
> We are running on I3s since they came out. NVMe SSDs are really fast and I
> managed to push them to 75k IOPs.
>
> As Bhuvan mentioned the i3 storage is ephemeral. If you can work around it
> and plan for failure recovery you are good to go.
>
>
>
> I ran Cassandra on m4s before and had no problems with EBS volumes (gp2)
> even in low latency use cases. With the cost of M4 instances and EBS
> volumes that make sense in IOPs, I would recommend going with more i3s and
> working around the ephemeral issue (if its an issue).
>
>
>
> Best,
>
> Matija
>
> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal  wrote:
>
> i3 instances will undoubtedly give you more meat for buck - easily 40K+
> iops whereas on the other hand EBS maxes out at 20K PIOPS which is highly
> expensive (at times they can cost you significantly more than cost of
> instance).
>
> But they have ephemeral local storage and data is lost once instance is
> stopped, you need to be prudent in case of i series, it is generally used
> for large persistent caches.
>
>
>
> Regards,
>
> Bhuvan
>
> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
> wrote:
>
> Hi –
>
>   We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and
> have been comparing this to I-3.2xlarge, which seems more cost effective
> when dealing with this amount of storage and from an IOPS perspective. Does
> anyone have any recommendations/ on the I-3s and how it performs overall,
> compared to the M4 equivalent? On the surface, without us having taken it
> through its paces performance-wise, it does seem to be pretty powerful. We
> just ran through an exercise with a RAIDed 200 TB volume (as opposed to a
> non RAIDed 3 TB volume) and were seeing a 20-30% improvement with the
> RAIDed setup, on a 6 node Cassandra ring. Just looking for any
> feedback/experience folks may have had with the I-3s.
>
>
>
> Regards,
>
> *DHRUVA GOPAL*
>
> *sr. MANAGER, ENGINEERING*
>
> *REPORTING, ANALYTICS AND BIG DATA*
>
> *+1 408.325.2011 <+1%20408-325-2011>* *WORK*
>
> *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE*
>
> *UNITED STATES*
>
> *dhruva.go...@aspect.com  *
>
> *aspect.com *
>
> [image: scription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]
>
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>

Re: EC2 instance recommendations

[Gopal, Dhruva]

Oh, so all the data is lost if the instance is shutdown or restarted (for that 
instance)? If we take a naïve approach to backing up the directory, and 
restoring it, if we ever have to bring down the instance and back up, will that 
work as a strategy? Data is only kept around for 2 days and is TTL’d after.

From: Matija Gobec 
Date: Tuesday, May 23, 2017 at 8:15 AM
To: Bhuvan Rawal 
Cc: "Gopal, Dhruva" , "user@cassandra.apache.org" 

Subject: Re: EC2 instance recommendations

We are running on I3s since they came out. NVMe SSDs are really fast and I 
managed to push them to 75k IOPs.
As Bhuvan mentioned the i3 storage is ephemeral. If you can work around it and 
plan for failure recovery you are good to go.

I ran Cassandra on m4s before and had no problems with EBS volumes (gp2) even 
in low latency use cases. With the cost of M4 instances and EBS volumes that 
make sense in IOPs, I would recommend going with more i3s and working around 
the ephemeral issue (if its an issue).

Best,
Matija

On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal 
> wrote:
i3 instances will undoubtedly give you more meat for buck - easily 40K+ iops 
whereas on the other hand EBS maxes out at 20K PIOPS which is highly expensive 
(at times they can cost you significantly more than cost of instance).
But they have ephemeral local storage and data is lost once instance is 
stopped, you need to be prudent in case of i series, it is generally used for 
large persistent caches.

Regards,
Bhuvan

On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
> wrote:
Hi –
  We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and have 
been comparing this to I-3.2xlarge, which seems more cost effective when 
dealing with this amount of storage and from an IOPS perspective. Does anyone 
have any recommendations/ on the I-3s and how it performs overall, compared to 
the M4 equivalent? On the surface, without us having taken it through its paces 
performance-wise, it does seem to be pretty powerful. We just ran through an 
exercise with a RAIDed 200 TB volume (as opposed to a non RAIDed 3 TB volume) 
and were seeing a 20-30% improvement with the RAIDed setup, on a 6 node 
Cassandra ring. Just looking for any feedback/experience folks may have had 
with the I-3s.

Regards,
DHRUVA GOPAL
sr. MANAGER, ENGINEERING
REPORTING, ANALYTICS AND BIG DATA
+1 408.325.2011 WORK
+1 408.219.1094 MOBILE
UNITED STATES
dhruva.go...@aspect.com
aspect.com
[scription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]

This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.


This email (including any attachments) is proprietary to Aspect Software, Inc. 
and may contain information that is confidential. If you have received this 
message in error, please do not read, copy or forward this message. Please 
notify the sender immediately, delete it from your system and destroy any 
copies. You may not further disclose or distribute this email or its 
attachments.

Re: EC2 instance recommendations

[Matija Gobec]

We are running on I3s since they came out. NVMe SSDs are really fast and I
managed to push them to 75k IOPs.
As Bhuvan mentioned the i3 storage is ephemeral. If you can work around it
and plan for failure recovery you are good to go.

I ran Cassandra on m4s before and had no problems with EBS volumes (gp2)
even in low latency use cases. With the cost of M4 instances and EBS
volumes that make sense in IOPs, I would recommend going with more i3s and
working around the ephemeral issue (if its an issue).

Best,
Matija

On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal  wrote:

> i3 instances will undoubtedly give you more meat for buck - easily 40K+
> iops whereas on the other hand EBS maxes out at 20K PIOPS which is highly
> expensive (at times they can cost you significantly more than cost of
> instance).
> But they have ephemeral local storage and data is lost once instance is
> stopped, you need to be prudent in case of i series, it is generally used
> for large persistent caches.
>
> Regards,
> Bhuvan
>
> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
> wrote:
>
>> Hi –
>>
>>   We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and
>> have been comparing this to I-3.2xlarge, which seems more cost effective
>> when dealing with this amount of storage and from an IOPS perspective. Does
>> anyone have any recommendations/ on the I-3s and how it performs overall,
>> compared to the M4 equivalent? On the surface, without us having taken it
>> through its paces performance-wise, it does seem to be pretty powerful. We
>> just ran through an exercise with a RAIDed 200 TB volume (as opposed to a
>> non RAIDed 3 TB volume) and were seeing a 20-30% improvement with the
>> RAIDed setup, on a 6 node Cassandra ring. Just looking for any
>> feedback/experience folks may have had with the I-3s.
>>
>>
>>
>> Regards,
>>
>> *DHRUVA GOPAL*
>>
>> *sr. MANAGER, ENGINEERING*
>>
>> *REPORTING, ANALYTICS AND BIG DATA*
>>
>> *+1 408.325.2011 <+1%20408-325-2011>* *WORK*
>>
>> *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE*
>>
>> *UNITED STATES*
>>
>> *dhruva.go...@aspect.com  *
>>
>> *aspect.com *
>>
>> [image: escription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]
>>
>>
>> This email (including any attachments) is proprietary to Aspect Software,
>> Inc. and may contain information that is confidential. If you have received
>> this message in error, please do not read, copy or forward this message.
>> Please notify the sender immediately, delete it from your system and
>> destroy any copies. You may not further disclose or distribute this email
>> or its attachments.
>>
>
>

Re: How to find dataSize at client side?

[Nicolas Guyomar]

Hi,

If you were to know the batch size on client side to make sure it does not
get above the 5kb limit, so that you can "limit the number of statements in
a batch", I would suspect you do not need batch at all right ? See
https://inoio.de/blog/2016/01/13/cassandra-to-batch-or-not-to-batch/

As for your question, you might get an answer on the java driver ML :
java-driver-u...@lists.datastax.com


On 23 May 2017 at 15:25, techpyaasa .  wrote:

>
> * WARN [SharedPool-Worker-1] 2017-05-22 20:28:46,204 BatchStatement.java
> (line 253) Batch of prepared statements for [site24x7.wm_rawstats_tb,
> site24x7.wm_rawstats] is of size 6122, exceeding specified threshold of
> 5120 by 1002*
> We are frequently getting this message in logs, so I wanted to restrict
> inserts at client side by calculating *dataSize* of insert/batch
> statements before sending it to c* servers.
>
> We are using datastax java drivers , how can I get dataSize here??
>
>
> Any ideas??
>
> Thanks in advance
> TechPyaasa
>

How to find dataSize at client side?

[techpyaasa .]

* WARN [SharedPool-Worker-1] 2017-05-22 20:28:46,204 BatchStatement.java
(line 253) Batch of prepared statements for [site24x7.wm_rawstats_tb,
site24x7.wm_rawstats] is of size 6122, exceeding specified threshold of
5120 by 1002*
We are frequently getting this message in logs, so I wanted to restrict
inserts at client side by calculating *dataSize* of insert/batch statements
before sending it to c* servers.

We are using datastax java drivers , how can I get dataSize here??


Any ideas??

Thanks in advance
TechPyaasa

Re: Bottleneck for small inserts?

[Cogumelos Maravilha]

Hi,

Change to *|durable_writes = false|*

And please post the results.

Thanks.

On 05/22/2017 10:08 PM, Jonathan Haddad wrote:
> How many CPUs are you using for interrupts?
>  
> http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux
>
> Have you tried making a flame graph to see where Cassandra is spending
> its
> time? http://www.brendangregg.com/blog/2014-06-12/java-flame-graphs.html
>
> Are you tracking GC pauses?
>
> Jon
>
> On Mon, May 22, 2017 at 2:03 PM Eric Pederson  > wrote:
>
> Hi all:
>
> I'm new to Cassandra and I'm doing some performance testing.  One
> of things that I'm testing is ingestion throughput.   My server
> setup is:
>
>   * 3 node cluster
>   * SSD data (both commit log and sstables are on the same disk)
>   * 64 GB RAM per server
>   * 48 cores per server
>   * Cassandra 3.0.11
>   * 48 Gb heap using G1GC
>   * 1 Gbps NICs
>
> Since I'm using SSD I've tried tuning the following (one at a
> time) but none seemed to make a lot of difference:
>
>   * concurrent_writes=384
>   * memtable_flush_writers=8
>   * concurrent_compactors=8
>
> I am currently doing ingestion tests sending data from 3 clients
> on the same subnet.  I am using cassandra-stress to do some
> ingestion testing.  The tests are using CL=ONE and RF=2.
>
> Using cassandra-stress (3.10) I am able to saturate the disk using
> a large enough column size and the standard five column
> cassandra-stress schema.  For example, -col size=fixed(400) will
> saturate the disk and compactions will start falling behind. 
>
> One of our main tables has a row size that approximately 200
> bytes, across 64 columns.  When ingesting this table I don't see
> any resource saturation.  Disk utilization is around 10-15% per
> iostat.  Incoming network traffic on the servers is around 100-300
> Mbps.  CPU utilization is around 20-70%.  nodetool tpstats shows
> mostly zeros with occasional spikes around 500 in MutationStage.  
>
> The stress run does 10,000,000 inserts per client, each with a
> separate range of partition IDs.  The run with 200 byte rows takes
> about 4 minutes, with mean Latency 4.5ms, Total GC time of 21
> secs, Avg GC time 173 ms.   
>
> The overall performance is good - around 120k rows/sec ingested. 
> But I'm curious to know where the bottleneck is.  There's no
> resource saturation andnodetool tpstats shows only occasional
> brief queueing.  Is the rest just expected latency inside of
> Cassandra?
>
> Thanks,
>
> -- Eric
>

Re: EC2 instance recommendations

[Bhuvan Rawal]

i3 instances will undoubtedly give you more meat for buck - easily 40K+
iops whereas on the other hand EBS maxes out at 20K PIOPS which is highly
expensive (at times they can cost you significantly more than cost of
instance).
But they have ephemeral local storage and data is lost once instance is
stopped, you need to be prudent in case of i series, it is generally used
for large persistent caches.

Regards,
Bhuvan

On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva 
wrote:

> Hi –
>
>   We’ve been running M4.2xlarge EC2 instances with 2-3 TB of storage and
> have been comparing this to I-3.2xlarge, which seems more cost effective
> when dealing with this amount of storage and from an IOPS perspective. Does
> anyone have any recommendations/ on the I-3s and how it performs overall,
> compared to the M4 equivalent? On the surface, without us having taken it
> through its paces performance-wise, it does seem to be pretty powerful. We
> just ran through an exercise with a RAIDed 200 TB volume (as opposed to a
> non RAIDed 3 TB volume) and were seeing a 20-30% improvement with the
> RAIDed setup, on a 6 node Cassandra ring. Just looking for any
> feedback/experience folks may have had with the I-3s.
>
>
>
> Regards,
>
> *DHRUVA GOPAL*
>
> *sr. MANAGER, ENGINEERING*
>
> *REPORTING, ANALYTICS AND BIG DATA*
>
> *+1 408.325.2011* *WORK*
>
> *+1 408.219.1094* *MOBILE*
>
> *UNITED STATES*
>
> *dhruva.go...@aspect.com  *
>
> *aspect.com *
>
> [image: escription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg]
>
>
> This email (including any attachments) is proprietary to Aspect Software,
> Inc. and may contain information that is confidential. If you have received
> this message in error, please do not read, copy or forward this message.
> Please notify the sender immediately, delete it from your system and
> destroy any copies. You may not further disclose or distribute this email
> or its attachments.
>