Re: How to avoid flush if the data can fit into memtable

[Akhil Mehra]

Hi Preetika,

After thinking about your scenario I believe your small SSTable size might
be due to data compression. By default, all tables enable SSTable
compression.

Let go through your scenario. Let's say you have allocated 4GB to your
Cassandra node. Your *memtable_heap_space_in_mb* and

*memtable_offheap_space_in_mb  *will roughly come to around 1GB. Since you
have memtable_cleanup_threshold to .50 table cleanup will be triggered when
total allocated memtable space exceeds 1/2GB. Note the cleanup threshold is
.50 of 1GB and not a combination of heap and off heap space. This memtable
allocation size is the total amount available for all tables on your node.
This includes all system related keyspaces. The cleanup process will write
the largest memtable to disk.

For your case, I am assuming that you are on a *single node with only one
table with insert activity*. I do not think the commit log will trigger a
flush in this circumstance as by default the commit log has 8192 MB of
space unless the commit log is placed on a very small disk.

I am assuming your table on disk is smaller than 500MB because of
compression. You can disable compression on your table and see if this
helps get the desired size.

I have written up a blog post explaining memtable flushing (
http://abiasforaction.net/apache-cassandra-memtable-flush/)

Let me know if you have any other question.

I hope this helps.

Regards,
Akhil Mehra


On Fri, May 26, 2017 at 6:58 AM, preetika tyagi 
wrote:

> I agree that for such a small data, Cassandra is obviously not needed.
> However, this is purely an experimental setup by using which I'm trying to
> understand how and exactly when memtable flush is triggered. As I mentioned
> in my post, I read the documentation and tweaked the parameters accordingly
> so that I never hit memtable flush but it is still doing that. As far the
> the setup is concerned, I'm just using 1 node and running Cassandra using
> "cassandra -R" option and then running some queries to insert some dummy
> data.
>
> I use the schema from CASSANDRA_HOME/tools/cqlstress-insanity-example.yaml
> and add "durable_writes=false" in the keyspace_definition.
>
> @Daemeon - The previous post lead to this post but since I was unaware of
> memtable flush and I assumed memtable flush wasn't happening, the previous
> post was about something else (throughput/latency etc.). This post is
> explicitly about exactly when memtable is being dumped to the disk. Didn't
> want to confuse two different goals that's why posted a new one.
>
> On Thu, May 25, 2017 at 10:38 AM, Avi Kivity  wrote:
>
>> It doesn't have to fit in memory. If your key distribution has strong
>> temporal locality, then a larger memtable that can coalesce overwrites
>> greatly reduces the disk I/O load for the memtable flush and subsequent
>> compactions. Of course, I have no idea if the is what the OP had in mind.
>>
>>
>> On 05/25/2017 07:14 PM, Jonathan Haddad wrote:
>>
>> Sorry for the confusion.  That was for the OP.  I wrote it quickly right
>> after waking up.
>>
>> What I'm asking is why does the OP want to keep his data in the memtable
>> exclusively?  If the goal is to "make reads fast", then just turn on row
>> caching.
>>
>> If there's so little data that it fits in memory (300MB), and there
>> aren't going to be any writes past the initial small dataset, why use
>> Cassandra?  It sounds like the wrong tool for this job.  Sounds like
>> something that could easily be stored in S3 and loaded in memory when the
>> app is fired up.
>>
>> On Thu, May 25, 2017 at 8:06 AM Avi Kivity  wrote:
>>
>>> Not sure whether you're asking me or the original poster, but the more
>>> times data gets overwritten in a memtable, the less it has to be compacted
>>> later on (and even without overwrites, larger memtables result in less
>>> compaction).
>>>
>>> On 05/25/2017 05:59 PM, Jonathan Haddad wrote:
>>>
>>> Why do you think keeping your data in the memtable is a what you need to
>>> do?
>>> On Thu, May 25, 2017 at 7:16 AM Avi Kivity  wrote:
>>>
 Then it doesn't have to (it still may, for other reasons).

 On 05/25/2017 05:11 PM, preetika tyagi wrote:

 What if the commit log is disabled?

 On May 25, 2017 4:31 AM, "Avi Kivity"  wrote:

> Cassandra has to flush the memtable occasionally, or the commit log
> grows without bounds.
>
> On 05/25/2017 03:42 AM, preetika tyagi wrote:
>
> Hi,
>
> I'm running Cassandra with a very small dataset so that the data can
> exist on memtable only. Below are my configurations:
>
> In jvm.options:
>
> -Xms4G
> -Xmx4G
>
> In cassandra.yaml,
>
> memtable_cleanup_threshold: 0.50
> memtable_allocation_type: heap_buffers
>
> As per the documentation in cassandra.yaml, the
> *memtable_heap_space_in_mb* and *memtable_heap_space_in_mb* 

Re: Restarting nodes and reported load

[Daniel Steuernol]

The cluster is running with RF=3, right now each node is storing about 3-4 TB of data. I'm using r4.2xlarge EC2 instances, these have 8 vCPU's, 61 GB of RAM, and the disks attached for the data drive are gp2 ssd ebs volumes with 10k iops. I guess this brings up the question of what's a good marker to decide on whether to increase disk space vs provisioning a new node?
  

On May 29 2017, at 9:35 am, tommaso barbugli  wrote:


  Hi Daniel,This is not normal. Possibly a capacity problem. Whats the RF, how much data do you store per node and what kind of servers do you use (core count, RAM, disk, ...)?Cheers,TommasoOn Mon, May 29, 2017 at 6:22 PM, Daniel Steuernol  wrote:I am running a 6 node cluster, and I have noticed that the reported load on each node rises throughout the week and grows way past the actual disk space used and available on each node. Also eventually latency for operations suffers and the nodes have to be restarted. A couple questions on this, is this normal? Also does cassandra need to be restarted every few days for best performance? Any insight on this behaviour would be helpful.Cheers,Daniel

-
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org




  

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Restarting nodes and reported load

[tommaso barbugli]

Hi Daniel,

This is not normal. Possibly a capacity problem. Whats the RF, how much
data do you store per node and what kind of servers do you use (core count,
RAM, disk, ...)?

Cheers,
Tommaso

On Mon, May 29, 2017 at 6:22 PM, Daniel Steuernol 
wrote:

>
> I am running a 6 node cluster, and I have noticed that the reported load
> on each node rises throughout the week and grows way past the actual disk
> space used and available on each node. Also eventually latency for
> operations suffers and the nodes have to be restarted. A couple questions
> on this, is this normal? Also does cassandra need to be restarted every few
> days for best performance? Any insight on this behaviour would be helpful.
>
> Cheers,
> Daniel
> - To
> unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional
> commands, e-mail: user-h...@cassandra.apache.org

Restarting nodes and reported load

[Daniel Steuernol]

I am running a 6 node cluster, and I have noticed that the reported load on each node rises throughout the week and grows way past the actual disk space used and available on each node. Also eventually latency for operations suffers and the nodes have to be restarted. A couple questions on this, is this normal? Also does cassandra need to be restarted every few days for best performance? Any insight on this behaviour would be helpful.Cheers,Daniel

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: How do you do automatic restacking of AWS instance for cassandra?

[John Wong]

My approach is the obvious taking a big outage window especially at work we
are using 1.2 and is using token range. I am generally a believer that (1)
patches should be applied, but (2) routinely we replace each host with a
new EC2, so (1) I know my infrastructure code (puppet/chef/ansible/salt
stack/whatever) doesn't just work on a 12-year old working EC2 (of course,
I am joking about EC2 been up for 12 years).

The other approach I have been meaning to try is simply taking one node out
at a time, and let the new node stream over (this has an issue with the IP
changed, but be careful).

But I think with the new versions of Cassandra, you can use vnode. I have
been meaning to also try just adding new vnode and gradually retiring the
older one. Does this approach sound like what people would do?

EBS is great, and generally performance is great (but please pick a
reasonable instance type appropriate for your workload, running on m3.large
is definitely not the way to go for production usually, just an example).

John

On Sun, May 28, 2017 at 1:59 PM, daemeon reiydelle 
wrote:

> This is in fact an interesting security practice that makes sense. It
> assumes the existing ami had security holes that WERE ALREADY exploited.
> See if you can negotiate moving the hdfs volumes to persistent storage. Fyi
> two major banks I have worked with did much the same, but as the storage
> was SAN (with VMware ) I was able to make adjustments to the ansible
> scripts (client was providing mobile banking solutions to the bank)
>
> I had another client using AWS, Chef, Terraform. I WAS NOT able to make
> this work in Chef. I can do it with Ansible, Terraform, AWS however.
>
> “All men dream, but not equally. Those who dream by night in the dusty
> recesses of their minds wake up in the day to find it was vanity, but the
> dreamers of the day are dangerous men, for they may act their dreams with
> open eyes, to make it possible.” — T.E. Lawrence
>
> sent from my mobile
> Daemeon Reiydelle
> skype daemeon.c.m.reiydelle
> USA 415.501.0198 <(415)%20501-0198>
>
> On May 28, 2017 1:25 AM, "Anthony Grasso" 
> wrote:
>
>> Hi Surbhi,
>>
>> Please see my comment inline below.
>>
>> On 28 May 2017 at 12:11, Jeff Jirsa  wrote:
>>
>>>
>>>
>>> On 2017-05-27 18:04 (-0700), Surbhi Gupta 
>>> wrote:
>>> > Thanks a lot for all of your reply.
>>> > Our requirement is :
>>> > Our company releases AMI almost every month where they have some or the
>>> > other security packages.
>>> > So as per our security team we need to move our cassandra cluster to
>>> the
>>> > new AMI .
>>> > As this process happens every month, we would like to automate the
>>> process .
>>> > Few points to consider here:
>>> >
>>> > 1. We are using ephemeral drives to store cassandra data
>>> > 2. We are on dse 4.8.x
>>> >
>>> > So currently to do the process, we pinup a new nodes with new DC name
>>> and
>>> > join that DC, alter the keyspace, do rebuild  and later alter the
>>> keyspace
>>> > again to remove the old DC .
>>> >
>>> > But all of this process is manually done as of now.
>>> >
>>> > So i wanted to understand , on AWS, how do you do above kind of task
>>> > automatically ?
>>>
>>>
>>> At a previous employer, they used M4 class instances with data on a
>>> dedicated EBS volumes, so we could swap AMIs / stop / start / adjust
>>> instances without having to deal with this. This worked reasonably well for
>>> their scale (which was petabytes of data).
>>>
>>
>> This is a really good option as it avoids streaming data to replace a
>> node which could potentially be quicker if dealing with large amounts of
>> data on each node.
>>
>>
>>>
>>> Other companies using ephemeral tend to be more willing to just
>>> terminate instances and replace them (-Dcassandra.replace_address). If you
>>> stop cassandra, then boot a replacement with 'replace_address' set, it'll
>>> take over for the stopped instance, including re-streaming all data (as
>>> best it can, subject to consistency level and repair status). This may be
>>> easier for you to script than switching your fleet to EBS, but it's not
>>> without risk.
>>>
>>
>> A quick note if you do decide to go down this path. If you are using
>> Cassandra version 2.x.x and above, the cassandra.replace_address_firs
>> t_boot can also be used. This option works once when Cassandra is first
>> started and the replacement node inserted into the cluster. After that, the
>> option is ignored for all subsequent restarts, where as
>> cassandra.replace_address needs to be removed from the *cassandra-env.sh*
>> file in order to restart the node. Restart behaviour aside, both options
>> operate in the same way to replace a node in the cluster.
>>
>>
>>>
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: 

How to know when repair repaired something?

[Jan Algermissen]

Hi,

is it possible to extract from repair logs the writetime of the writes 
that needed to be repaired?


I have some processes I would like to re-trigger from a time point if 
repair found problems.


Is that useful? Possible?

Jan

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org