Shouldn't cause GCs. You can usually think of heap memory separately from the rest. It's already allocated as far as the OS is concerned, and it doesn't know anything about GC going on inside of that allocation. You can set "-XX:+AlwaysPreTouch" to make sure it's physically allocated on startup. JVM OOMs when there's not enough memory in the heap, and a system OOM (invocation of oom-killer) happens when there's not enough memory outside of the heap. The kernel will generally pretty aggressively reclaim mmap'd memory before resorting to oom-killer.
The main con of disabling data mmap is exactly that - reduced read perf and increased I/O load. I think to some extent you're solving a non-problem by disabling mmap to reduce memory utilization. Unused memory is wasted memory, so there's not a lot of reasons to avoid using it as a file read cache. Especially if you're preallocating JVM memory and not running any other services on that host. You probably only want to disable data mmap if your data-to-RAM ratio is so high that it's just thrashing and not doing anything useful. On Tue, Aug 3, 2021 at 10:18 AM Amandeep Srivastava < amandeep.srivastava1...@gmail.com> wrote: > Thanks. I guess some earlier thread got truncated. > > I already applied Erick's recommendations and that seem to have worked in > reducing the ram consumption by around 50%. > > Regarding cheap memory and hardware, we are already running 96GB boxes and > getting multiple larger ones might be a little difficult at this point. > Hence I wanted to understand cons of disabling mmap use for data. > > Besides degraded read performance, wouldn't we be putting more pressure on > heap memory, when disabling mmap, which might cause frequent GCs and OOM > errors at some point? Since currently whatever was being served by mmap > would be loaded over heap now and processed/stored further. > > Also, we've disabled the swap usage on hosts as recommended to optimize > performance so cass won't be able to enter that too in case memory starts > to fill up. > > On Tue, 3 Aug, 2021, 6:33 pm Jim Shaw, <jxys...@gmail.com> wrote: > >> I think Erick posted https://community.datastax.com/questions/6947/. >> explained very clearly. >> >> We hit same issue only on a huge table when upgrade, and we changed back >> after done. >> My understanding, Which option to chose, shall depend on your user >> case. If chasing high performance on a big table, then go default one, and >> increase capacity in memory, nowadays hardware is cheaper. >> >> Thanks, >> Jim >> >> On Mon, Aug 2, 2021 at 7:12 PM Amandeep Srivastava < >> amandeep.srivastava1...@gmail.com> wrote: >> >>> Can anyone please help with the above questions? To summarise: >>> >>> 1) What is the impact of using mmap only for indices besides a >>> degradation in read performance? >>> 2) Why does the off heap consumed during Cassandra full repair remains >>> occupied 12+ hours after the repair completion and is there a >>> manual/configuration driven way to clear that earlier? >>> >>> Thanks, >>> Aman >>> >>> On Thu, 29 Jul, 2021, 6:47 pm Amandeep Srivastava, < >>> amandeep.srivastava1...@gmail.com> wrote: >>> >>>> Hi Erick, >>>> >>>> Limiting mmap to index only seems to have resolved the issue. The max >>>> ram usage remained at 60% this time. Could you please point me to the >>>> limitations for setting this param? - For starters, I can see read >>>> performance getting reduced up to 30% (CASSANDRA-8464 >>>> <https://issues.apache.org/jira/browse/CASSANDRA-8464>) >>>> >>>> Also if you could please shed light on extended questions in my earlier >>>> email. >>>> >>>> Thanks a lot. >>>> >>>> Regards, >>>> Aman >>>> >>>> On Thu, Jul 29, 2021 at 12:52 PM Amandeep Srivastava < >>>> amandeep.srivastava1...@gmail.com> wrote: >>>> >>>>> Thanks, Bowen, don't think that's an issue - but yes I can try >>>>> upgrading to 3.11.5 and limit the merkle tree size to bring down the >>>>> memory >>>>> utilization. >>>>> >>>>> Thanks, Erick, let me try that. >>>>> >>>>> Can someone please share documentation relating to internal >>>>> functioning of full repairs - if there exists one? Wanted to understand >>>>> the >>>>> role of the heap and off-heap memory separately during the process. >>>>> >>>>> Also, for my case, once the nodes reach the 95% memory usage, it stays >>>>> there for almost 10-12 hours after the repair is complete, before falling >>>>> back to 65%. Any pointers on what might be consuming off-heap for so long >>>>> and can something be done to clear it earlier? >>>>> >>>>> Thanks, >>>>> Aman >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Regards, >>>> Aman >>>> >>>
