Re: Upgrading to 3.11.8 Caused Map Failures

2020-12-12 Thread Shalom Sagges
You are right Yakir. How did I miss that?? It was a misconfiguration on my end. Thanks a lot! On Sat, Dec 12, 2020 at 9:28 PM Yakir Gibraltar wrote: > See also: > https://support.datastax.com/hc/en-us/articles/360027838911 > > > On Sat, Dec 12, 2020 at 9:11 PM Yakir Gibraltar wrote: > >> Hi

Re: Upgrading to 3.11.8 Caused Map Failures

2020-12-12 Thread Yakir Gibraltar
See also: https://support.datastax.com/hc/en-us/articles/360027838911 On Sat, Dec 12, 2020 at 9:11 PM Yakir Gibraltar wrote: > Hi Shalom, > See bug: https://issues.apache.org/jira/browse/CASSANDRA-14978 > Try to disable mmap: > disk_access_mode=standard > or > disk_access_mode=mmap_index_only

Re: Upgrading to 3.11.8 Caused Map Failures

2020-12-12 Thread Yakir Gibraltar
Hi Shalom, See bug: https://issues.apache.org/jira/browse/CASSANDRA-14978 Try to disable mmap: disk_access_mode=standard or disk_access_mode=mmap_index_only Yakir Gibraltar.

Re: Upgrading to 3.11.8 Caused Map Failures

2020-12-11 Thread Shalom Sagges
Forgot to mention that there were also LEAK DETECTED errors: ERROR [Reference-Reaper] 2020-12-11 03:25:42,172 Ref.java:229 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@451030de) to class

Upgrading to 3.11.8 Caused Map Failures

2020-12-11 Thread Shalom Sagges
Hi All, I upgraded Cassandra from v3.11.4 to v3.11.8. The upgrade went smoothly, however, after a few hours, a node crashed on OOM and a few hours later, another one crashed. Seems like they crashed from excessive GC behaviour (CMS). The logs show Map failures on CompactionExecutor: ERROR