Re: Memory usage more than heap memory

[Erick Ramirez]

>
> I observe long running nodes have high non heap memory footprint then a
> recently started node. That is the reason I am interested in find non heap
> memory usage by Cassandra node. What could be the reason of high non heap
> memory footprint in long running cluster?
>

Memory utilisation grows over time as SSTables get read. From Cassandra 2.2
onwards SSTables get mapped to memory -- i.e. the data files get copied to
memory to speed up reads. A newly restarted C* doesn't use as much memory
because the data hasn't been read yet so are not mapped to memory.

I would like to know why memory usage is high and is there way to control
> it.


Laxmikant already mentioned disk_access_mode previously. This was a hidden
property in C* but CASSANDRA-8464
 [1] changed the
behaviour in C* 2.2 so I wrote this KB article
 to explain
the memory issue in detail [2]. TL;DR -- set this in cassandra.yaml to only
mmap() the index files so there's less memory used:

disk_access_mode: mmap_index_only


I want to monitor memory pressure, I want to check that if heap+non heap
> usage goes above certain level then I can take certain action.


In any case as Michael Shuler explained, your approach is problematic so
try to follow his recommendations. Cheers!

[1] https://issues.apache.org/jira/browse/CASSANDRA-8464
[2] https://support.datastax.com/hc/en-us/articles/360027838911

Re: Memory usage more than heap memory

[HImanshu Sharma]

I observe long running nodes have high non heap memory footprint then a
recently started node. That is the reason I am interested in find non heap
memory usage by Cassandra node. What could be the reason of high non heap
memory footprint in long running cluster?

Regards
Himanshu


On Sat, Apr 11, 2020 at 11:56 PM Pekka Enberg  wrote:

> Hi Himanshu,
>
> On Sat, Apr 11, 2020 at 8:47 PM HImanshu Sharma
>  wrote:
> > I am not worried about page cache. I want to monitor memory pressure, I
> want to check that if heap+non heap usage goes above certain level then I
> can take certain action. But due to this page cache thing, I am not sure
> how to find actual memory usage( heap and off heap). Heap will not be more
> than 8G. Off heap increases with data on disk due to off heap storage of
> bloom filters and index files. I want to know the exact amount of memory
> occupied by off heap components. Nodetool info output does not look
> appropriate to me or I am missing something.
>
> There's "nodetool tablestats", which shows off-heap size for tables,
> but I don't know if that's sufficient for your purposes.
>
> Out of curiosity, why are you not interested in how much memory your
> process allocated in the page cache? As you said earlier, the resident
> set size (RSS) in Linux indeed includes memory allocated in the page
> cache:
>
> https://stackoverflow.com/a/21049737/153224
>
> But it is as much part of the "memory pressure" as anything else.
>
> - Pekka
>

Re: Memory usage more than heap memory

[Pekka Enberg]

Hi Himanshu,

On Sat, Apr 11, 2020 at 8:47 PM HImanshu Sharma
 wrote:
> I am not worried about page cache. I want to monitor memory pressure, I want 
> to check that if heap+non heap usage goes above certain level then I can take 
> certain action. But due to this page cache thing, I am not sure how to find 
> actual memory usage( heap and off heap). Heap will not be more than 8G. Off 
> heap increases with data on disk due to off heap storage of bloom filters and 
> index files. I want to know the exact amount of memory occupied by off heap 
> components. Nodetool info output does not look appropriate to me or I am 
> missing something.

There's "nodetool tablestats", which shows off-heap size for tables,
but I don't know if that's sufficient for your purposes.

Out of curiosity, why are you not interested in how much memory your
process allocated in the page cache? As you said earlier, the resident
set size (RSS) in Linux indeed includes memory allocated in the page
cache:

https://stackoverflow.com/a/21049737/153224

But it is as much part of the "memory pressure" as anything else.

- Pekka

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Memory usage more than heap memory

[Michael Shuler]

Nodetool isn't really meant for monitoring usage.

There are a number of existing tools to grab and track jvm heap usage. 
Enable JMX and pull the info into something custom, or using a grafana 
template, for instance, which usually uses JMX internally.


ie. this one graphs heap usage historically: 
https://grafana.com/grafana/dashboards/5408


Kind regars,
Michael

On 4/11/20 12:47 PM, HImanshu Sharma wrote:

Hi

I am not worried about page cache. I want to monitor memory pressure, I 
want to check that if heap+non heap usage goes above certain level then 
I can take certain action. But due to this page cache thing, I am not 
sure how to find actual memory usage( heap and off heap). Heap will not 
be more than 8G. Off heap increases with data on disk due to off heap 
storage of bloom filters and index files. I want to know the exact 
amount of memory occupied by off heap components. Nodetool info output 
does not look appropriate to me or I am missing something.


Regards
Himanshu

On Sat, Apr 11, 2020 at 10:13 PM Laxmikant Upadhyay 
mailto:laxmikant@gmail.com>> wrote:


Hi,

You can read section 'OS Page Cache Usage' on
http://cassandra.apache.org/doc/latest/troubleshooting/use_tools.html
Also, don't worry about memory usage (page cache) not decreasing
even if no traffic ...it will come down when required (for example:
a new application needs it when boot up).

On Sat, Apr 11, 2020 at 5:29 PM HImanshu Sharma
mailto:himanshusharma0...@gmail.com>>
wrote:

Hi

I am observing memory usage in top command, but there in RSS it
is showing 18G ( which I think is sum of used memo + page
cache). I want to know how to find how much used by Cassandra
process and how much of it is in page cache. I want this
information because I want to check memory usage for monitoring.

On Sat, Apr 11, 2020 at 9:42 PM Nitan Kainth
mailto:nitankai...@gmail.com>> wrote:

You can look at top command. There is column of memory


Regards,

Nitan

Cell: 510 449 9629 



On Apr 11, 2020, at 11:10 AM, HImanshu Sharma
mailto:himanshusharma0...@gmail.com>> wrote:


Hi

But I see memory not decreasing even if there is no
traffic on cluster. How can I find actual memory usage by
Cassandra process. If it is  OS page cache then how  to
find how much is page  cache and how much is used by process?

Thanks
Himanshu

On Sat, Apr 11, 2020 at 9:07 PM Laxmikant Upadhyay
mailto:laxmikant@gmail.com>>
wrote:

It is OS page cache used during read..your os will
leverage memory if not being used by any other
applications and it improves your read performance.

On Sat, Apr 11, 2020, 12:47 PM HImanshu Sharma
mailto:himanshusharma0...@gmail.com>> wrote:

Hi

I am very new to the use of cassandra. In a
cassandra cluster of 3 nodes, I am observing
memory usage of cassandra process going more than
the heap memor allocated. As I understand,
cassandra allocates off heap memory for bloom
filters, index summary etc.

When I run nodetool info, I see off heap memory
usage around 1.5G, my heap is 8G. So memory usage
should not go above  10G. But in running node it
goes as high as 18G with heavy loads. It comes
down with different loads but always more than 12G.

I would like to know why memory usage is high and
is there way to control it.

Thanks
Himanshu Sharma




-- 


regards,
Laxmikant Upadhyay



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Memory usage more than heap memory

[Michael Shuler]

For some simple and helpful explanations of the behavior you are 
observing, some ideas on what to look for in monitoring, as well as some 
interesting experiments on the "play" page (last link), have a look at 
https://www.linuxatemyram.com/ - this is general linux memory behavior 
and not unique to this software.


The Cassandra-specific configuration change suggestions may be more harm 
than help, without an understanding the basic kernel behavior. 
Performance benchmarking of your specific workload will be the best way 
to gauge whether configurations help or hurt. One ubiquitous Cassandra 
recommendation is that nodes should have no swap enabled. Paging out/in 
of memory will cause problems, so just don't.


Kind regards,
Michael

On 4/11/20 11:50 AM, Laxmikant Upadhyay wrote:
One more point, if you are worried about high memory usage then read 
about disk_access_mode configuration of Cassandra. By default it will 
cause high memory usage. Setting it to mmap_index_only can help.


On Sat, Apr 11, 2020 at 5:43 PM Laxmikant Upadhyay 
mailto:laxmikant@gmail.com>> wrote:


Hi,

You can read section 'OS Page Cache Usage' on
http://cassandra.apache.org/doc/latest/troubleshooting/use_tools.html
Also, don't worry about memory usage (page cache) not decreasing
even if no traffic ...it will come down when required (for example:
a new application needs it when boot up).

On Sat, Apr 11, 2020 at 5:29 PM HImanshu Sharma
mailto:himanshusharma0...@gmail.com>>
wrote:

Hi

I am observing memory usage in top command, but there in RSS it
is showing 18G ( which I think is sum of used memo + page
cache). I want to know how to find how much used by Cassandra
process and how much of it is in page cache. I want this
information because I want to check memory usage for monitoring.

On Sat, Apr 11, 2020 at 9:42 PM Nitan Kainth
mailto:nitankai...@gmail.com>> wrote:

You can look at top command. There is column of memory


Regards,

Nitan

Cell: 510 449 9629 



On Apr 11, 2020, at 11:10 AM, HImanshu Sharma
mailto:himanshusharma0...@gmail.com>> wrote:


Hi

But I see memory not decreasing even if there is no
traffic on cluster. How can I find actual memory usage by
Cassandra process. If it is  OS page cache then how  to
find how much is page  cache and how much is used by process?

Thanks
Himanshu

On Sat, Apr 11, 2020 at 9:07 PM Laxmikant Upadhyay
mailto:laxmikant@gmail.com>>
wrote:

It is OS page cache used during read..your os will
leverage memory if not being used by any other
applications and it improves your read performance.

On Sat, Apr 11, 2020, 12:47 PM HImanshu Sharma
mailto:himanshusharma0...@gmail.com>> wrote:

Hi

I am very new to the use of cassandra. In a
cassandra cluster of 3 nodes, I am observing
memory usage of cassandra process going more than
the heap memor allocated. As I understand,
cassandra allocates off heap memory for bloom
filters, index summary etc.

When I run nodetool info, I see off heap memory
usage around 1.5G, my heap is 8G. So memory usage
should not go above  10G. But in running node it
goes as high as 18G with heavy loads. It comes
down with different loads but always more than 12G.

I would like to know why memory usage is high and
is there way to control it.

Thanks
Himanshu Sharma




-- 


regards,
Laxmikant Upadhyay



--

regards,
Laxmikant Upadhyay



-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Memory usage more than heap memory

[HImanshu Sharma]

Hi

I am not worried about page cache. I want to monitor memory pressure, I
want to check that if heap+non heap usage goes above certain level then I
can take certain action. But due to this page cache thing, I am not sure
how to find actual memory usage( heap and off heap). Heap will not be more
than 8G. Off heap increases with data on disk due to off heap storage of
bloom filters and index files. I want to know the exact amount of memory
occupied by off heap components. Nodetool info output does not look
appropriate to me or I am missing something.

Regards
Himanshu

On Sat, Apr 11, 2020 at 10:13 PM Laxmikant Upadhyay 
wrote:

> Hi,
>
> You can read section 'OS Page Cache Usage' on
> http://cassandra.apache.org/doc/latest/troubleshooting/use_tools.html
> Also, don't worry about memory usage (page cache) not decreasing even if
> no traffic ...it will come down when required (for example: a new
> application needs it when boot up).
>
> On Sat, Apr 11, 2020 at 5:29 PM HImanshu Sharma <
> himanshusharma0...@gmail.com> wrote:
>
>> Hi
>>
>> I am observing memory usage in top command, but there in RSS it is
>> showing 18G ( which I think is sum of used memo + page cache). I want to
>> know how to find how much used by Cassandra process and how much of it is
>> in page cache. I want this information because I want to check memory usage
>> for monitoring.
>>
>> On Sat, Apr 11, 2020 at 9:42 PM Nitan Kainth 
>> wrote:
>>
>>> You can look at top command. There is column of memory
>>>
>>>
>>> Regards,
>>>
>>> Nitan
>>>
>>> Cell: 510 449 9629
>>>
>>> On Apr 11, 2020, at 11:10 AM, HImanshu Sharma <
>>> himanshusharma0...@gmail.com> wrote:
>>>
>>> 
>>> Hi
>>>
>>> But I see memory not decreasing even if there is no traffic on cluster.
>>> How can I find actual memory usage by Cassandra process. If it is  OS page
>>> cache then how  to find how much is page  cache and how much is used by
>>> process?
>>>
>>> Thanks
>>> Himanshu
>>>
>>> On Sat, Apr 11, 2020 at 9:07 PM Laxmikant Upadhyay <
>>> laxmikant@gmail.com> wrote:
>>>
 It is OS page cache used during read..your os will leverage memory if
 not being used by any other applications and it improves your read
 performance.

 On Sat, Apr 11, 2020, 12:47 PM HImanshu Sharma <
 himanshusharma0...@gmail.com> wrote:

> Hi
>
> I am very new to the use of cassandra. In a cassandra cluster of 3
> nodes, I am observing memory usage of cassandra process going more than 
> the
> heap memor allocated. As I understand, cassandra allocates off heap memory
> for bloom filters, index summary etc.
>
> When I run nodetool info, I see off heap memory usage around 1.5G, my
> heap is 8G. So memory usage should not go above  10G. But in running node
> it goes as high as 18G with heavy loads. It comes down with different 
> loads
> but always more than 12G.
>
> I would like to know why memory usage is high and is there way to
> control it.
>
> Thanks
> Himanshu Sharma
>

>
> --
>
> regards,
> Laxmikant Upadhyay
>
>

Re: Memory usage more than heap memory

[Laxmikant Upadhyay]

Hi,

You can read section 'OS Page Cache Usage' on
http://cassandra.apache.org/doc/latest/troubleshooting/use_tools.html
Also, don't worry about memory usage (page cache) not decreasing even if no
traffic ...it will come down when required (for example: a new application
needs it when boot up).

On Sat, Apr 11, 2020 at 5:29 PM HImanshu Sharma <
himanshusharma0...@gmail.com> wrote:

> Hi
>
> I am observing memory usage in top command, but there in RSS it is showing
> 18G ( which I think is sum of used memo + page cache). I want to know how
> to find how much used by Cassandra process and how much of it is in page
> cache. I want this information because I want to check memory usage for
> monitoring.
>
> On Sat, Apr 11, 2020 at 9:42 PM Nitan Kainth 
> wrote:
>
>> You can look at top command. There is column of memory
>>
>>
>> Regards,
>>
>> Nitan
>>
>> Cell: 510 449 9629
>>
>> On Apr 11, 2020, at 11:10 AM, HImanshu Sharma <
>> himanshusharma0...@gmail.com> wrote:
>>
>> 
>> Hi
>>
>> But I see memory not decreasing even if there is no traffic on cluster.
>> How can I find actual memory usage by Cassandra process. If it is  OS page
>> cache then how  to find how much is page  cache and how much is used by
>> process?
>>
>> Thanks
>> Himanshu
>>
>> On Sat, Apr 11, 2020 at 9:07 PM Laxmikant Upadhyay <
>> laxmikant@gmail.com> wrote:
>>
>>> It is OS page cache used during read..your os will leverage memory if
>>> not being used by any other applications and it improves your read
>>> performance.
>>>
>>> On Sat, Apr 11, 2020, 12:47 PM HImanshu Sharma <
>>> himanshusharma0...@gmail.com> wrote:
>>>
 Hi

 I am very new to the use of cassandra. In a cassandra cluster of 3
 nodes, I am observing memory usage of cassandra process going more than the
 heap memor allocated. As I understand, cassandra allocates off heap memory
 for bloom filters, index summary etc.

 When I run nodetool info, I see off heap memory usage around 1.5G, my
 heap is 8G. So memory usage should not go above  10G. But in running node
 it goes as high as 18G with heavy loads. It comes down with different loads
 but always more than 12G.

 I would like to know why memory usage is high and is there way to
 control it.

 Thanks
 Himanshu Sharma

>>>

-- 

regards,
Laxmikant Upadhyay

Re: Memory usage more than heap memory

[HImanshu Sharma]

Hi

I am observing memory usage in top command, but there in RSS it is showing
18G ( which I think is sum of used memo + page cache). I want to know how
to find how much used by Cassandra process and how much of it is in page
cache. I want this information because I want to check memory usage for
monitoring.

On Sat, Apr 11, 2020 at 9:42 PM Nitan Kainth  wrote:

> You can look at top command. There is column of memory
>
>
> Regards,
>
> Nitan
>
> Cell: 510 449 9629
>
> On Apr 11, 2020, at 11:10 AM, HImanshu Sharma <
> himanshusharma0...@gmail.com> wrote:
>
> 
> Hi
>
> But I see memory not decreasing even if there is no traffic on cluster.
> How can I find actual memory usage by Cassandra process. If it is  OS page
> cache then how  to find how much is page  cache and how much is used by
> process?
>
> Thanks
> Himanshu
>
> On Sat, Apr 11, 2020 at 9:07 PM Laxmikant Upadhyay <
> laxmikant@gmail.com> wrote:
>
>> It is OS page cache used during read..your os will leverage memory if not
>> being used by any other applications and it improves your read performance.
>>
>> On Sat, Apr 11, 2020, 12:47 PM HImanshu Sharma <
>> himanshusharma0...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> I am very new to the use of cassandra. In a cassandra cluster of 3
>>> nodes, I am observing memory usage of cassandra process going more than the
>>> heap memor allocated. As I understand, cassandra allocates off heap memory
>>> for bloom filters, index summary etc.
>>>
>>> When I run nodetool info, I see off heap memory usage around 1.5G, my
>>> heap is 8G. So memory usage should not go above  10G. But in running node
>>> it goes as high as 18G with heavy loads. It comes down with different loads
>>> but always more than 12G.
>>>
>>> I would like to know why memory usage is high and is there way to
>>> control it.
>>>
>>> Thanks
>>> Himanshu Sharma
>>>
>>

Re: Memory usage more than heap memory

[Nitan Kainth]

You can look at top command. There is column of memory


Regards,
Nitan
Cell: 510 449 9629

> On Apr 11, 2020, at 11:10 AM, HImanshu Sharma  
> wrote:
> 
> 
> Hi 
> 
> But I see memory not decreasing even if there is no traffic on cluster. How 
> can I find actual memory usage by Cassandra process. If it is  OS page cache 
> then how  to find how much is page  cache and how much is used by process?
> 
> Thanks
> Himanshu
> 
>> On Sat, Apr 11, 2020 at 9:07 PM Laxmikant Upadhyay  
>> wrote:
>> It is OS page cache used during read..your os will leverage memory if not 
>> being used by any other applications and it improves your read performance.
>> 
>>> On Sat, Apr 11, 2020, 12:47 PM HImanshu Sharma 
>>>  wrote:
>>> Hi 
>>> 
>>> I am very new to the use of cassandra. In a cassandra cluster of 3 nodes, I 
>>> am observing memory usage of cassandra process going more than the heap 
>>> memor allocated. As I understand, cassandra allocates off heap memory for 
>>> bloom filters, index summary etc. 
>>> 
>>> When I run nodetool info, I see off heap memory usage around 1.5G, my heap 
>>> is 8G. So memory usage should not go above  10G. But in running node it 
>>> goes as high as 18G with heavy loads. It comes down with different loads 
>>> but always more than 12G.
>>> 
>>> I would like to know why memory usage is high and is there way to control 
>>> it.
>>> 
>>> Thanks
>>> Himanshu Sharma

Re: Memory usage more than heap memory

[HImanshu Sharma]

Hi

But I see memory not decreasing even if there is no traffic on cluster. How
can I find actual memory usage by Cassandra process. If it is  OS page
cache then how  to find how much is page  cache and how much is used by
process?

Thanks
Himanshu

On Sat, Apr 11, 2020 at 9:07 PM Laxmikant Upadhyay 
wrote:

> It is OS page cache used during read..your os will leverage memory if not
> being used by any other applications and it improves your read performance.
>
> On Sat, Apr 11, 2020, 12:47 PM HImanshu Sharma <
> himanshusharma0...@gmail.com> wrote:
>
>> Hi
>>
>> I am very new to the use of cassandra. In a cassandra cluster of 3 nodes,
>> I am observing memory usage of cassandra process going more than the heap
>> memor allocated. As I understand, cassandra allocates off heap memory for
>> bloom filters, index summary etc.
>>
>> When I run nodetool info, I see off heap memory usage around 1.5G, my
>> heap is 8G. So memory usage should not go above  10G. But in running node
>> it goes as high as 18G with heavy loads. It comes down with different loads
>> but always more than 12G.
>>
>> I would like to know why memory usage is high and is there way to control
>> it.
>>
>> Thanks
>> Himanshu Sharma
>>
>

Re: Memory usage more than heap memory

[Laxmikant Upadhyay]

It is OS page cache used during read..your os will leverage memory if not
being used by any other applications and it improves your read performance.

On Sat, Apr 11, 2020, 12:47 PM HImanshu Sharma 
wrote:

> Hi
>
> I am very new to the use of cassandra. In a cassandra cluster of 3 nodes,
> I am observing memory usage of cassandra process going more than the heap
> memor allocated. As I understand, cassandra allocates off heap memory for
> bloom filters, index summary etc.
>
> When I run nodetool info, I see off heap memory usage around 1.5G, my heap
> is 8G. So memory usage should not go above  10G. But in running node it
> goes as high as 18G with heavy loads. It comes down with different loads
> but always more than 12G.
>
> I would like to know why memory usage is high and is there way to control
> it.
>
> Thanks
> Himanshu Sharma
>

Memory usage more than heap memory

[HImanshu Sharma]

Hi

I am very new to the use of cassandra. In a cassandra cluster of 3 nodes, I
am observing memory usage of cassandra process going more than the heap
memor allocated. As I understand, cassandra allocates off heap memory for
bloom filters, index summary etc.

When I run nodetool info, I see off heap memory usage around 1.5G, my heap
is 8G. So memory usage should not go above  10G. But in running node it
goes as high as 18G with heavy loads. It comes down with different loads
but always more than 12G.

I would like to know why memory usage is high and is there way to control
it.

Thanks
Himanshu Sharma