Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread vaibhav khedkar
Thank you patrick. We have plans for upgrades anyway so keep this issue in mind and probably expedite it. I have updated and created a bug https://issues.apache.org/jira/browse/CASSANDRA-18770 in case you are interested. Thanks vaibhav On Wed, Aug 16, 2023 at 1:34 PM Patrick Lee wrote: > I

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread Patrick Lee
I don’t have a ticket.  What I saw in a scenario was a cluster that was upgraded from 3.11 to 4.0.X, we added another ring that was running Java 11.  Nodes on the ring with Java 8 saw this issue you described while the other ring running Java 11 did not.  Then if I updated from Java 8 to Java 11 I

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread vaibhav khedkar
Thanks Patrick, We do have plans to upgrade to *java 11* eventually but we will go through internal testing and would also need some time given the size of our infrastructure. Is it safe to assume that the issue exists in the combination of upgrades from 3.11.x to 4.0.x *and* running on JAVA 8

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread Patrick Lee
I've actually noticed this as well on a few clusters I deal with but after upgrading Cassandra from 3.11 to 4 we also changed to use Java 11 shortly after the cluster upgrade. After I moved to Java 11 I have not experienced a problem. On Wed, Aug 16, 2023 at 12:12 PM vaibhav khedkar wrote: >

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread vaibhav khedkar
Thank you Scott We are seeing it for all the tables (Filter, Data ..etc ) /nb-1-big-Statistics.db.tmp (deleted) /nb-3-big-Statistics.db.tmp (deleted) /nb-2-big-Data.db (deleted) /nb-2-big-Statistics.db.tmp (deleted) /nb-2-big-Index.db (deleted) /nb-2-big-Statistics.db.tmp (deleted)

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread C. Scott Andreas
Vaibhav, thank you for reaching out and sharing this issue report.Could you run an `lsof` and share which SSTable files you see open (e.g., all SSTable components or a subset of them); and also share the value of the `disk_access_mode` property from your cassandra.yaml?Opening a Jira ticket

Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread vaibhav khedkar
Hi everyone, We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9 to 4.0.5. After the upgrade, we are seeing a unique issue where the compacted SSTables's file descriptors are still present and are never cleared. This is causing false disk alerts. We have to restart nodes very