Thank you patrick. We have plans for upgrades anyway so keep this issue in mind and probably expedite it.
I have updated and created a bug https://issues.apache.org/jira/browse/CASSANDRA-18770 in case you are interested. Thanks vaibhav On Wed, Aug 16, 2023 at 1:34 PM Patrick Lee <patrickclee0...@gmail.com> wrote: > I don’t have a ticket. What I saw in a scenario was a cluster that was > upgraded from 3.11 to 4.0.X, we added another ring that was running Java > 11. Nodes on the ring with Java 8 saw this issue you described while the > other ring running Java 11 did not. Then if I updated from Java 8 to Java > 11 I did not run into this issue again. But the percentage of > nodes/clusters that I experienced this issue was extremely low. We just > happen to be in a phase of a lot of updates/upgrades of multiple things so > it was just all baked into our process. Completely get that updating the > Java version is not a simple thing but wanted to throw that little > Interesting bit of info that I had experienced. > > Sent from my iPhone > > On Aug 16, 2023, at 1:28 PM, vaibhav khedkar <vkhedk...@gmail.com> wrote: > > > Thanks Patrick, > > > We do have plans to upgrade to *java 11* eventually but we will go > through internal testing and would also need some time given the size of > our infrastructure. > > Is it safe to assume that the issue exists in the combination of upgrades > from 3.11.x to 4.0.x *and* running on JAVA 8 ? > > Did you happen to have created a ticket for this when you observed the > issue ? > > Thanks > vaibhav > > On Wed, Aug 16, 2023 at 11:15 AM Patrick Lee <patrickclee0...@gmail.com> > wrote: > >> I've actually noticed this as well on a few clusters I deal with but >> after upgrading Cassandra from 3.11 to 4 we also changed to use Java 11 >> shortly after the cluster upgrade. After I moved to Java 11 I have not >> experienced a problem. >> >> On Wed, Aug 16, 2023 at 12:12 PM vaibhav khedkar <vkhedk...@gmail.com> >> wrote: >> >>> Thank you Scott >>> >>> We are seeing it for all the tables (Filter, Data ..etc ) >>> >>> <path>/nb-1-big-Statistics.db.tmp (deleted) >>> <path>/nb-3-big-Statistics.db.tmp (deleted) >>> <path>/nb-2-big-Data.db (deleted) >>> <path>/nb-2-big-Statistics.db.tmp (deleted) >>> <path>/nb-2-big-Index.db (deleted) >>> <path>/nb-2-big-Statistics.db.tmp (deleted) >>> <path>/nb-2-big-CompressionInfo.db (deleted) >>> <path>/nb-2-big-Filter.db (deleted) >>> <path>/nb-2-big-Summary.db (deleted) >>> <path>/nb-2-big-Digest.crc32 (deleted) >>> >>> Also I believe the property `disk_access_mode` is only present in the >>> DSE version of cassandra and we use Open source. >>> >>> I will certainly open the ticket for this. >>> >>> Thanks >>> vaibhav >>> >>> On Wed, Aug 16, 2023 at 9:34 AM C. Scott Andreas <sc...@paradoxica.net> >>> wrote: >>> >>>> Vaibhav, thank you for reaching out and sharing this issue report. >>>> >>>> Could you run an `lsof` and share which SSTable files you see open >>>> (e.g., all SSTable components or a subset of them); and also share the >>>> value of the `disk_access_mode` property from your cassandra.yaml? >>>> >>>> Opening a Jira ticket for this for discussion / investigation is >>>> probably a good next step. >>>> >>>> Thanks, >>>> >>>> – Scott >>>> >>>> On Aug 16, 2023, at 9:28 AM, vaibhav khedkar <vkhedk...@gmail.com> >>>> wrote: >>>> >>>> >>>> Hi everyone, >>>> >>>> We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9 >>>> to 4.0.5. >>>> >>>> After the upgrade, we are seeing a unique issue where the compacted >>>> SSTables's file descriptors are still present and are never cleared. This >>>> is causing false disk alerts. We have to restart nodes very often in order >>>> to reclaim the disk space. >>>> >>>> We use the Apache version of C* along with CentOS Linux 7.9.2009 and >>>> Java 8. >>>> >>>> We are looking for help in resolving this issue. If you have any >>>> suggestions, please let us know. >>>> >>>> Thank you, >>>> vaibhav >>>> >>>> >>>> >>> >>> -- >>> Best Regards, >>> Vaibhav Khedkar >>> Google Inc. >>> Mobile : (806)- 252 - 2912 >>> >>> Email: vkhedk...@gmail.com >>> >> > > -- > Best Regards, > Vaibhav Khedkar > Google Inc. > Mobile : (806)- 252 - 2912 > > Email: vkhedk...@gmail.com > > -- Best Regards, Vaibhav Khedkar Google Inc. Mobile : (806)- 252 - 2912 Email: vkhedk...@gmail.com
