Re: High memory usage during nodetool repair

[manish khandelwal]

Missed the heap part, not sure why is that happening

On Tue, Aug 3, 2021 at 8:59 AM manish khandelwal <
manishkhandelwa...@gmail.com> wrote:

> mmap is used for faster reads and as you guessed right you might see read
> performance degradation. If you are seeing high memory usage after repairs
> due to mmaped files, the only way to reduce the memory usage is to trigger
> some other process which requires memory. *mmapped* files use
> buffer/cache memory which gets released as soon as some other process
> requests memory from the kernel. Kernel does not want to waste its effort
> until some request for resource comes in.
>
>
>
> On Tue, Aug 3, 2021 at 4:42 AM Amandeep Srivastava <
> amandeep.srivastava1...@gmail.com> wrote:
>
>> Can anyone please help with the above questions? To summarise:
>>
>> 1) What is the impact of using mmap only for indices besides a
>> degradation in read performance?
>> 2) Why does the off heap consumed during Cassandra full repair remains
>> occupied 12+ hours after the repair completion and is there a
>> manual/configuration driven way to clear that earlier?
>>
>> Thanks,
>> Aman
>>
>> On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, <
>> amandeep.srivastava1...@gmail.com> wrote:
>>
>>> Hi Erick,
>>>
>>> Limiting mmap to index only seems to have resolved the issue. The max
>>> ram usage remained at 60% this time. Could you please point me to the
>>> limitations for setting this param? - For starters, I can see read
>>> performance getting reduced up to 30% (CASSANDRA-8464
>>> )
>>>
>>> Also if you could please shed light on extended questions in my earlier
>>> email.
>>>
>>> Thanks a lot.
>>>
>>> Regards,
>>> Aman
>>>
>>> On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
>>> amandeep.srivastava1...@gmail.com> wrote:
>>>
 Thanks, Bowen, don't think that's an issue - but yes I can try
 upgrading to 3.11.5 and limit the merkle tree size to bring down the memory
 utilization.

 Thanks, Erick, let me try that.

 Can someone please share documentation relating to internal functioning
 of full repairs - if there exists one? Wanted to understand the role of the
 heap and off-heap memory separately during the process.

 Also, for my case, once the nodes reach the 95% memory usage, it stays
 there for almost 10-12 hours after the repair is complete, before falling
 back to 65%. Any pointers on what might be consuming off-heap for so long
 and can something be done to clear it earlier?

 Thanks,
 Aman



>>>
>>> --
>>> Regards,
>>> Aman
>>>
>>

Re: High memory usage during nodetool repair

[manish khandelwal]

mmap is used for faster reads and as you guessed right you might see read
performance degradation. If you are seeing high memory usage after repairs
due to mmaped files, the only way to reduce the memory usage is to trigger
some other process which requires memory. *mmapped* files use buffer/cache
memory which gets released as soon as some other process requests memory
from the kernel. Kernel does not want to waste its effort until some
request for resource comes in.



On Tue, Aug 3, 2021 at 4:42 AM Amandeep Srivastava <
amandeep.srivastava1...@gmail.com> wrote:

> Can anyone please help with the above questions? To summarise:
>
> 1) What is the impact of using mmap only for indices besides a degradation
> in read performance?
> 2) Why does the off heap consumed during Cassandra full repair remains
> occupied 12+ hours after the repair completion and is there a
> manual/configuration driven way to clear that earlier?
>
> Thanks,
> Aman
>
> On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, <
> amandeep.srivastava1...@gmail.com> wrote:
>
>> Hi Erick,
>>
>> Limiting mmap to index only seems to have resolved the issue. The max ram
>> usage remained at 60% this time. Could you please point me to the
>> limitations for setting this param? - For starters, I can see read
>> performance getting reduced up to 30% (CASSANDRA-8464
>> )
>>
>> Also if you could please shed light on extended questions in my earlier
>> email.
>>
>> Thanks a lot.
>>
>> Regards,
>> Aman
>>
>> On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
>> amandeep.srivastava1...@gmail.com> wrote:
>>
>>> Thanks, Bowen, don't think that's an issue - but yes I can try upgrading
>>> to 3.11.5 and limit the merkle tree size to bring down the memory
>>> utilization.
>>>
>>> Thanks, Erick, let me try that.
>>>
>>> Can someone please share documentation relating to internal functioning
>>> of full repairs - if there exists one? Wanted to understand the role of the
>>> heap and off-heap memory separately during the process.
>>>
>>> Also, for my case, once the nodes reach the 95% memory usage, it stays
>>> there for almost 10-12 hours after the repair is complete, before falling
>>> back to 65%. Any pointers on what might be consuming off-heap for so long
>>> and can something be done to clear it earlier?
>>>
>>> Thanks,
>>> Aman
>>>
>>>
>>>
>>
>> --
>> Regards,
>> Aman
>>
>

Re: High memory usage during nodetool repair

[Amandeep Srivastava]

Can anyone please help with the above questions? To summarise:

1) What is the impact of using mmap only for indices besides a degradation
in read performance?
2) Why does the off heap consumed during Cassandra full repair remains
occupied 12+ hours after the repair completion and is there a
manual/configuration driven way to clear that earlier?

Thanks,
Aman

On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, <
amandeep.srivastava1...@gmail.com> wrote:

> Hi Erick,
>
> Limiting mmap to index only seems to have resolved the issue. The max ram
> usage remained at 60% this time. Could you please point me to the
> limitations for setting this param? - For starters, I can see read
> performance getting reduced up to 30% (CASSANDRA-8464
> )
>
> Also if you could please shed light on extended questions in my earlier
> email.
>
> Thanks a lot.
>
> Regards,
> Aman
>
> On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava <
> amandeep.srivastava1...@gmail.com> wrote:
>
>> Thanks, Bowen, don't think that's an issue - but yes I can try upgrading
>> to 3.11.5 and limit the merkle tree size to bring down the memory
>> utilization.
>>
>> Thanks, Erick, let me try that.
>>
>> Can someone please share documentation relating to internal functioning
>> of full repairs - if there exists one? Wanted to understand the role of the
>> heap and off-heap memory separately during the process.
>>
>> Also, for my case, once the nodes reach the 95% memory usage, it stays
>> there for almost 10-12 hours after the repair is complete, before falling
>> back to 65%. Any pointers on what might be consuming off-heap for so long
>> and can something be done to clear it earlier?
>>
>> Thanks,
>> Aman
>>
>>
>>
>
> --
> Regards,
> Aman
>

Re: New Servers - Cassandra 4

[Joe Obernberger]

Thank you Max.  That is a solid choice.  You can even configure each 
blade with two 15TBytes SSDs (may not be wise), but that would yield 
~430TBytes of SSD across 14 nodes in 4u space for around $150k.


-Joe

On 8/2/2021 4:29 PM, Max C. wrote:
Have you considered a blade chassis?  Then you can get most of the 
redundancy of having lots of small nodes in few(er) rack units.


SuperMicro has a chassis that can accommodate 14 servers in 4U:

https://www.supermicro.com/en/products/superblade/enclosure#4U

- Max

On Aug 2, 2021, at 12:05 pm, Joe Obernberger 
 wrote:


Thank you Jeff.  Consider that if rack space is at a premium, what 
would make the most sense?


-Joe

On 8/2/2021 2:46 PM, Jeff Jirsa wrote:
IF you bought a server with that topology, you would definitely want 
to run lots of instances, perhaps 24, to effectively utilize that 
disk space.


You'd also need 24 IPs, and you'd need a NIC that could send/receive 
24x the normal bandwidth. And the cost of rebuilding such a node 
would be 24x higher than normal (so consider how many of those you'd 
have in a cluster, and how often they'd fail).




On Mon, Aug 2, 2021 at 11:06 AM Joe Obernberger 
 wrote:


We have a large amount of data to be stored in Cassandra, and if
we were
to purchase new hardware in limited space, what would make the
most sense?
Dell has machines with 24, 8TByte drives in a 2u configuration.
Given
Cassandra's limitations (?) to large nodes, would it make sense
to run
24 copies of Cassandra on that one node (one per drive)?
Thank you!

-Joe


 
	Virus-free. www.avg.com 
 

Re: New Servers - Cassandra 4

[Max C.]

Have you considered a blade chassis?  Then you can get most of the redundancy 
of having lots of small nodes in few(er) rack units.

SuperMicro has a chassis that can accommodate 14 servers in 4U:

https://www.supermicro.com/en/products/superblade/enclosure#4U

- Max

> On Aug 2, 2021, at 12:05 pm, Joe Obernberger  
> wrote:
> 
> Thank you Jeff.  Consider that if rack space is at a premium, what would make 
> the most sense?
> 
> -Joe
> 
> On 8/2/2021 2:46 PM, Jeff Jirsa wrote:
>> IF you bought a server with that topology, you would definitely want to run 
>> lots of instances, perhaps 24, to effectively utilize that disk space. 
>> 
>> You'd also need 24 IPs, and you'd need a NIC that could send/receive 24x the 
>> normal bandwidth. And the cost of rebuilding such a node would be 24x higher 
>> than normal (so consider how many of those you'd have in a cluster, and how 
>> often they'd fail).
>> 
>> 
>> 
>> On Mon, Aug 2, 2021 at 11:06 AM Joe Obernberger 
>> mailto:joseph.obernber...@gmail.com>> wrote:
>> We have a large amount of data to be stored in Cassandra, and if we were 
>> to purchase new hardware in limited space, what would make the most sense?
>> Dell has machines with 24, 8TByte drives in a 2u configuration. Given 
>> Cassandra's limitations (?) to large nodes, would it make sense to run 
>> 24 copies of Cassandra on that one node (one per drive)?
>> Thank you!
>> 
>> -Joe
>> 
>> 
>>  
>> 
>> Virus-free. www.avg.com 
>> 
>>  

Re: New Servers - Cassandra 4

[Joe Obernberger]

Thank you Jeff.  Consider that if rack space is at a premium, what would 
make the most sense?


-Joe

On 8/2/2021 2:46 PM, Jeff Jirsa wrote:
IF you bought a server with that topology, you would definitely want 
to run lots of instances, perhaps 24, to effectively utilize that disk 
space.


You'd also need 24 IPs, and you'd need a NIC that could send/receive 
24x the normal bandwidth. And the cost of rebuilding such a node 
would be 24x higher than normal (so consider how many of those you'd 
have in a cluster, and how often they'd fail).




On Mon, Aug 2, 2021 at 11:06 AM Joe Obernberger 
 wrote:


We have a large amount of data to be stored in Cassandra, and if
we were
to purchase new hardware in limited space, what would make the
most sense?
Dell has machines with 24, 8TByte drives in a 2u configuration. Given
Cassandra's limitations (?) to large nodes, would it make sense to
run
24 copies of Cassandra on that one node (one per drive)?
Thank you!

-Joe


 
	Virus-free. www.avg.com 
 



<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Re: New Servers - Cassandra 4

[Jeff Jirsa]

IF you bought a server with that topology, you would definitely want to run
lots of instances, perhaps 24, to effectively utilize that disk space.

You'd also need 24 IPs, and you'd need a NIC that could send/receive 24x
the normal bandwidth. And the cost of rebuilding such a node would be 24x
higher than normal (so consider how many of those you'd have in a cluster,
and how often they'd fail).



On Mon, Aug 2, 2021 at 11:06 AM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:

> We have a large amount of data to be stored in Cassandra, and if we were
> to purchase new hardware in limited space, what would make the most sense?
> Dell has machines with 24, 8TByte drives in a 2u configuration. Given
> Cassandra's limitations (?) to large nodes, would it make sense to run
> 24 copies of Cassandra on that one node (one per drive)?
> Thank you!
>
> -Joe
>
>

New Servers - Cassandra 4

[Joe Obernberger]

We have a large amount of data to be stored in Cassandra, and if we were 
to purchase new hardware in limited space, what would make the most sense?
Dell has machines with 24, 8TByte drives in a 2u configuration. Given 
Cassandra's limitations (?) to large nodes, would it make sense to run 
24 copies of Cassandra on that one node (one per drive)?

Thank you!

-Joe