Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

[Patrick Lee]

I don’t have a ticket.  What I saw in a scenario was a cluster that was upgraded from 3.11 to 4.0.X, we added another ring that was running Java 11.  Nodes on the ring with Java 8 saw this issue you described while the other ring running Java 11 did not.  Then if I updated from Java 8 to Java 11 I did not run into this issue again.  But the percentage of nodes/clusters that I experienced this issue was extremely low. We just happen to be in a phase of a lot of updates/upgrades of multiple things so it was just all baked into our process.  Completely get that updating the Java version is not a simple thing but wanted to throw that little Interesting bit of info that I had experienced.   Sent from my iPhone On Aug 16, 2023, at 1:28 PM, vaibhav khedkar <vkhedk...@gmail.com> wrote:  Thanks Patrick, We do have plans to upgrade to java 11 eventually but we will go through internal testing and would also need some time given the size of our infrastructure.  Is it safe to assume that the issue exists in the combination of upgrades from 3.11.x to 4.0.x and running on JAVA 8 ?  Did you happen to have created a ticket for this when you observed the issue ?  Thanks  vaibhav On Wed, Aug 16, 2023 at 11:15 AM Patrick Lee <patrickclee0...@gmail.com> wrote: I've actually noticed this as well on a few clusters I deal with but after upgrading Cassandra from 3.11 to 4 we also changed to use Java 11 shortly after the cluster upgrade.  After I moved to Java 11 I have not experienced a problem. On Wed, Aug 16, 2023 at 12:12 PM vaibhav khedkar <vkhedk...@gmail.com> wrote: Thank you Scott  We are seeing it for all the tables (Filter, Data ..etc )  <path>/nb-1-big-Statistics.db.tmp (deleted) <path>/nb-3-big-Statistics.db.tmp (deleted) <path>/nb-2-big-Data.db (deleted) <path>/nb-2-big-Statistics.db.tmp (deleted) <path>/nb-2-big-Index.db (deleted) <path>/nb-2-big-Statistics.db.tmp (deleted) <path>/nb-2-big-CompressionInfo.db (deleted) <path>/nb-2-big-Filter.db (deleted) <path>/nb-2-big-Summary.db (deleted) <path>/nb-2-big-Digest.crc32 (deleted) Also I believe the property `disk_access_mode` is only present in the DSE version of cassandra and we use Open source.  I will certainly open the ticket for this.  Thanks vaibhav On Wed, Aug 16, 2023 at 9:34 AM C. Scott Andreas <sc...@paradoxica.net> wrote: Vaibhav, thank you for reaching out and sharing this issue report. Could you run an `lsof` and share which SSTable files you see open (e.g., all SSTable components or a subset of them); and also share the value of the `disk_access_mode` property from your cassandra.yaml? Opening a Jira ticket for this for discussion / investigation is probably a good next step. Thanks, – Scott On Aug 16, 2023, at 9:28 AM, vaibhav khedkar <vkhedk...@gmail.com> wrote: Hi everyone, We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9 to 4.0.5. After the upgrade, we are seeing a unique issue where the compacted SSTables's file descriptors are still present and are never cleared. This is causing false disk alerts. We have to restart nodes very often in order to reclaim the disk space. We use the Apache version of C* along with CentOS Linux 7.9.2009 and Java 8. We are looking for help in resolving this issue. If you have any suggestions, please let us know. Thank you, vaibhav -- Best Regards, Vaibhav Khedkar Google Inc.  Mobile : (806)- 252 - 2912 Email: vkhedk...@gmail.com -- Best Regards, Vaibhav Khedkar Google Inc.  Mobile : (806)- 252 - 2912 Email: vkhedk...@gmail.com