Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

[vaibhav khedkar]

Thank you patrick.

We have plans for upgrades anyway so keep this issue in mind and
probably expedite it.

I have updated and created a bug
https://issues.apache.org/jira/browse/CASSANDRA-18770  in case you are
interested.

Thanks
vaibhav

On Wed, Aug 16, 2023 at 1:34 PM Patrick Lee 
wrote:

> I don’t have a ticket.  What I saw in a scenario was a cluster that was
> upgraded from 3.11 to 4.0.X, we added another ring that was running Java
> 11.  Nodes on the ring with Java 8 saw this issue you described while the
> other ring running Java 11 did not.  Then if I updated from Java 8 to Java
> 11 I did not run into this issue again.  But the percentage of
> nodes/clusters that I experienced this issue was extremely low. We just
> happen to be in a phase of a lot of updates/upgrades of multiple things so
> it was just all baked into our process.  Completely get that updating the
> Java version is not a simple thing but wanted to throw that little
> Interesting bit of info that I had experienced.
>
> Sent from my iPhone
>
> On Aug 16, 2023, at 1:28 PM, vaibhav khedkar  wrote:
>
> 
> Thanks Patrick,
>
>
> We do have plans to upgrade to *java 11* eventually but we will go
> through internal testing and would also need some time given the size of
> our infrastructure.
>
> Is it safe to assume that the issue exists in the combination of upgrades
> from 3.11.x to 4.0.x *and* running on JAVA 8 ?
>
> Did you happen to have created a ticket for this when you observed the
> issue ?
>
> Thanks
> vaibhav
>
> On Wed, Aug 16, 2023 at 11:15 AM Patrick Lee 
> wrote:
>
>> I've actually noticed this as well on a few clusters I deal with but
>> after upgrading Cassandra from 3.11 to 4 we also changed to use Java 11
>> shortly after the cluster upgrade.  After I moved to Java 11 I have not
>> experienced a problem.
>>
>> On Wed, Aug 16, 2023 at 12:12 PM vaibhav khedkar 
>> wrote:
>>
>>> Thank you Scott
>>>
>>> We are seeing it for all the tables (Filter, Data ..etc )
>>>
>>> /nb-1-big-Statistics.db.tmp (deleted)
>>> /nb-3-big-Statistics.db.tmp (deleted)
>>> /nb-2-big-Data.db (deleted)
>>> /nb-2-big-Statistics.db.tmp (deleted)
>>> /nb-2-big-Index.db (deleted)
>>> /nb-2-big-Statistics.db.tmp (deleted)
>>> /nb-2-big-CompressionInfo.db (deleted)
>>> /nb-2-big-Filter.db (deleted)
>>> /nb-2-big-Summary.db (deleted)
>>> /nb-2-big-Digest.crc32 (deleted)
>>>
>>> Also I believe the property `disk_access_mode` is only present in the
>>> DSE version of cassandra and we use Open source.
>>>
>>> I will certainly open the ticket for this.
>>>
>>> Thanks
>>> vaibhav
>>>
>>> On Wed, Aug 16, 2023 at 9:34 AM C. Scott Andreas 
>>> wrote:
>>>
 Vaibhav, thank you for reaching out and sharing this issue report.

 Could you run an `lsof` and share which SSTable files you see open
 (e.g., all SSTable components or a subset of them); and also share the
 value of the `disk_access_mode` property from your cassandra.yaml?

 Opening a Jira ticket for this for discussion / investigation is
 probably a good next step.

 Thanks,

 – Scott

 On Aug 16, 2023, at 9:28 AM, vaibhav khedkar 
 wrote:


 Hi everyone,

 We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9
 to 4.0.5.

 After the upgrade, we are seeing a unique issue where the compacted
 SSTables's file descriptors are still present and are never cleared. This
 is causing false disk alerts. We have to restart nodes very often in order
 to reclaim the disk space.

 We use the Apache version of C* along with CentOS Linux 7.9.2009 and
 Java 8.

 We are looking for help in resolving this issue. If you have any
 suggestions, please let us know.

 Thank you,
 vaibhav



>>>
>>> --
>>> Best Regards,
>>> Vaibhav Khedkar
>>> Google Inc.
>>> Mobile : (806)- 252 - 2912
>>>
>>> Email: vkhedk...@gmail.com
>>>
>>
>
> --
> Best Regards,
> Vaibhav Khedkar
> Google Inc.
> Mobile : (806)- 252 - 2912
>
> Email: vkhedk...@gmail.com
>
>

-- 
Best Regards,
Vaibhav Khedkar
Google Inc.
Mobile : (806)- 252 - 2912

Email: vkhedk...@gmail.com

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

[Patrick Lee]

I don’t have a ticket.  What I saw in a scenario was a cluster that was upgraded from 3.11 to 4.0.X, we added another ring that was running Java 11.  Nodes on the ring with Java 8 saw this issue you described while the other ring running Java 11 did not.  Then if I updated from Java 8 to Java 11 I did not run into this issue again.  But the percentage of nodes/clusters that I experienced this issue was extremely low. We just happen to be in a phase of a lot of updates/upgrades of multiple things so it was just all baked into our process.  Completely get that updating the Java version is not a simple thing but wanted to throw that little Interesting bit of info that I had experienced.  Sent from my iPhoneOn Aug 16, 2023, at 1:28 PM, vaibhav khedkar  wrote:Thanks Patrick,We do have plans to upgrade to java 11 eventually but we will go through internal testing and would also need some time given the size of our infrastructure. Is it safe to assume that the issue exists in the combination of upgrades from 3.11.x to 4.0.x and running on JAVA 8 ? Did you happen to have created a ticket for this when you observed the issue ? Thanks vaibhavOn Wed, Aug 16, 2023 at 11:15 AM Patrick Lee  wrote:I've actually noticed this as well on a few clusters I deal with but after upgrading Cassandra from 3.11 to 4 we also changed to use Java 11 shortly after the cluster upgrade.  After I moved to Java 11 I have not experienced a problem.On Wed, Aug 16, 2023 at 12:12 PM vaibhav khedkar  wrote:Thank you Scott We are seeing it for all the tables (Filter, Data ..etc ) /nb-1-big-Statistics.db.tmp (deleted)/nb-3-big-Statistics.db.tmp (deleted)/nb-2-big-Data.db (deleted)/nb-2-big-Statistics.db.tmp (deleted)/nb-2-big-Index.db (deleted)/nb-2-big-Statistics.db.tmp (deleted)/nb-2-big-CompressionInfo.db (deleted)/nb-2-big-Filter.db (deleted)/nb-2-big-Summary.db (deleted)/nb-2-big-Digest.crc32 (deleted)Also I believe the property `disk_access_mode` is only present in the DSE version of cassandra and we use Open source. I will certainly open the ticket for this. ThanksvaibhavOn Wed, Aug 16, 2023 at 9:34 AM C. Scott Andreas  wrote:Vaibhav, thank you for reaching out and sharing this issue report.Could you run an `lsof` and share which SSTable files you see open (e.g., all SSTable components or a subset of them); and also share the value of the `disk_access_mode` property from your cassandra.yaml?Opening a Jira ticket for this for discussion / investigation is probably a good next step.Thanks,– ScottOn Aug 16, 2023, at 9:28 AM, vaibhav khedkar  wrote:Hi everyone,We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9 to 4.0.5.After the upgrade, we are seeing a unique issue where the compacted SSTables's file descriptors are still present and are never cleared. This is causing false disk alerts. We have to restart nodes very often in order to reclaim the disk space.We use the Apache version of C* along with CentOS Linux 7.9.2009 and Java 8.We are looking for help in resolving this issue. If you have any suggestions, please let us know.Thank you,vaibhav-- Best Regards,Vaibhav KhedkarGoogle Inc. Mobile : (806)- 252 - 2912Email: vkhedk...@gmail.com

-- Best Regards,Vaibhav KhedkarGoogle Inc. Mobile : (806)- 252 - 2912Email: vkhedk...@gmail.com

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

[vaibhav khedkar]

Thanks Patrick,


We do have plans to upgrade to *java 11* eventually but we will go through
internal testing and would also need some time given the size of our
infrastructure.

Is it safe to assume that the issue exists in the combination of upgrades
from 3.11.x to 4.0.x *and* running on JAVA 8 ?

Did you happen to have created a ticket for this when you observed the
issue ?

Thanks
vaibhav

On Wed, Aug 16, 2023 at 11:15 AM Patrick Lee 
wrote:

> I've actually noticed this as well on a few clusters I deal with but after
> upgrading Cassandra from 3.11 to 4 we also changed to use Java 11 shortly
> after the cluster upgrade.  After I moved to Java 11 I have not experienced
> a problem.
>
> On Wed, Aug 16, 2023 at 12:12 PM vaibhav khedkar 
> wrote:
>
>> Thank you Scott
>>
>> We are seeing it for all the tables (Filter, Data ..etc )
>>
>> /nb-1-big-Statistics.db.tmp (deleted)
>> /nb-3-big-Statistics.db.tmp (deleted)
>> /nb-2-big-Data.db (deleted)
>> /nb-2-big-Statistics.db.tmp (deleted)
>> /nb-2-big-Index.db (deleted)
>> /nb-2-big-Statistics.db.tmp (deleted)
>> /nb-2-big-CompressionInfo.db (deleted)
>> /nb-2-big-Filter.db (deleted)
>> /nb-2-big-Summary.db (deleted)
>> /nb-2-big-Digest.crc32 (deleted)
>>
>> Also I believe the property `disk_access_mode` is only present in the DSE
>> version of cassandra and we use Open source.
>>
>> I will certainly open the ticket for this.
>>
>> Thanks
>> vaibhav
>>
>> On Wed, Aug 16, 2023 at 9:34 AM C. Scott Andreas 
>> wrote:
>>
>>> Vaibhav, thank you for reaching out and sharing this issue report.
>>>
>>> Could you run an `lsof` and share which SSTable files you see open
>>> (e.g., all SSTable components or a subset of them); and also share the
>>> value of the `disk_access_mode` property from your cassandra.yaml?
>>>
>>> Opening a Jira ticket for this for discussion / investigation is
>>> probably a good next step.
>>>
>>> Thanks,
>>>
>>> – Scott
>>>
>>> On Aug 16, 2023, at 9:28 AM, vaibhav khedkar 
>>> wrote:
>>>
>>>
>>> Hi everyone,
>>>
>>> We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9
>>> to 4.0.5.
>>>
>>> After the upgrade, we are seeing a unique issue where the compacted
>>> SSTables's file descriptors are still present and are never cleared. This
>>> is causing false disk alerts. We have to restart nodes very often in order
>>> to reclaim the disk space.
>>>
>>> We use the Apache version of C* along with CentOS Linux 7.9.2009 and
>>> Java 8.
>>>
>>> We are looking for help in resolving this issue. If you have any
>>> suggestions, please let us know.
>>>
>>> Thank you,
>>> vaibhav
>>>
>>>
>>>
>>
>> --
>> Best Regards,
>> Vaibhav Khedkar
>> Google Inc.
>> Mobile : (806)- 252 - 2912
>>
>> Email: vkhedk...@gmail.com
>>
>

-- 
Best Regards,
Vaibhav Khedkar
Google Inc.
Mobile : (806)- 252 - 2912

Email: vkhedk...@gmail.com

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

[Patrick Lee]

I've actually noticed this as well on a few clusters I deal with but after
upgrading Cassandra from 3.11 to 4 we also changed to use Java 11 shortly
after the cluster upgrade.  After I moved to Java 11 I have not experienced
a problem.

On Wed, Aug 16, 2023 at 12:12 PM vaibhav khedkar 
wrote:

> Thank you Scott
>
> We are seeing it for all the tables (Filter, Data ..etc )
>
> /nb-1-big-Statistics.db.tmp (deleted)
> /nb-3-big-Statistics.db.tmp (deleted)
> /nb-2-big-Data.db (deleted)
> /nb-2-big-Statistics.db.tmp (deleted)
> /nb-2-big-Index.db (deleted)
> /nb-2-big-Statistics.db.tmp (deleted)
> /nb-2-big-CompressionInfo.db (deleted)
> /nb-2-big-Filter.db (deleted)
> /nb-2-big-Summary.db (deleted)
> /nb-2-big-Digest.crc32 (deleted)
>
> Also I believe the property `disk_access_mode` is only present in the DSE
> version of cassandra and we use Open source.
>
> I will certainly open the ticket for this.
>
> Thanks
> vaibhav
>
> On Wed, Aug 16, 2023 at 9:34 AM C. Scott Andreas 
> wrote:
>
>> Vaibhav, thank you for reaching out and sharing this issue report.
>>
>> Could you run an `lsof` and share which SSTable files you see open (e.g.,
>> all SSTable components or a subset of them); and also share the value of
>> the `disk_access_mode` property from your cassandra.yaml?
>>
>> Opening a Jira ticket for this for discussion / investigation is probably
>> a good next step.
>>
>> Thanks,
>>
>> – Scott
>>
>> On Aug 16, 2023, at 9:28 AM, vaibhav khedkar  wrote:
>>
>>
>> Hi everyone,
>>
>> We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9
>> to 4.0.5.
>>
>> After the upgrade, we are seeing a unique issue where the compacted
>> SSTables's file descriptors are still present and are never cleared. This
>> is causing false disk alerts. We have to restart nodes very often in order
>> to reclaim the disk space.
>>
>> We use the Apache version of C* along with CentOS Linux 7.9.2009 and Java
>> 8.
>>
>> We are looking for help in resolving this issue. If you have any
>> suggestions, please let us know.
>>
>> Thank you,
>> vaibhav
>>
>>
>>
>
> --
> Best Regards,
> Vaibhav Khedkar
> Google Inc.
> Mobile : (806)- 252 - 2912
>
> Email: vkhedk...@gmail.com
>

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

[vaibhav khedkar]

Thank you Scott

We are seeing it for all the tables (Filter, Data ..etc )

/nb-1-big-Statistics.db.tmp (deleted)
/nb-3-big-Statistics.db.tmp (deleted)
/nb-2-big-Data.db (deleted)
/nb-2-big-Statistics.db.tmp (deleted)
/nb-2-big-Index.db (deleted)
/nb-2-big-Statistics.db.tmp (deleted)
/nb-2-big-CompressionInfo.db (deleted)
/nb-2-big-Filter.db (deleted)
/nb-2-big-Summary.db (deleted)
/nb-2-big-Digest.crc32 (deleted)

Also I believe the property `disk_access_mode` is only present in the DSE
version of cassandra and we use Open source.

I will certainly open the ticket for this.

Thanks
vaibhav

On Wed, Aug 16, 2023 at 9:34 AM C. Scott Andreas 
wrote:

> Vaibhav, thank you for reaching out and sharing this issue report.
>
> Could you run an `lsof` and share which SSTable files you see open (e.g.,
> all SSTable components or a subset of them); and also share the value of
> the `disk_access_mode` property from your cassandra.yaml?
>
> Opening a Jira ticket for this for discussion / investigation is probably
> a good next step.
>
> Thanks,
>
> – Scott
>
> On Aug 16, 2023, at 9:28 AM, vaibhav khedkar  wrote:
>
>
> Hi everyone,
>
> We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9 to
> 4.0.5.
>
> After the upgrade, we are seeing a unique issue where the compacted
> SSTables's file descriptors are still present and are never cleared. This
> is causing false disk alerts. We have to restart nodes very often in order
> to reclaim the disk space.
>
> We use the Apache version of C* along with CentOS Linux 7.9.2009 and Java
> 8.
>
> We are looking for help in resolving this issue. If you have any
> suggestions, please let us know.
>
> Thank you,
> vaibhav
>
>
>

-- 
Best Regards,
Vaibhav Khedkar
Google Inc.
Mobile : (806)- 252 - 2912

Email: vkhedk...@gmail.com

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

[C. Scott Andreas]

Vaibhav, thank you for reaching out and sharing this issue report.Could you run an 
`lsof` and share which SSTable files you see open (e.g., all SSTable components or a 
subset of them); and also share the value of the `disk_access_mode` property from 
your cassandra.yaml?Opening a Jira ticket for this for discussion / investigation is 
probably a good next step.Thanks,– ScottOn Aug 16, 2023, at 9:28 AM, vaibhav khedkar 
 wrote:Hi everyone,We recently upgraded our fleet of ~2500 
Cassandra instances from 3.11.9 to 4.0.5.After the upgrade, we are seeing a unique 
issue where the compacted SSTables's file descriptors are still present and are never 
cleared. This is causing false disk alerts. We have to restart nodes very often in 
order to reclaim the disk space.We use the Apache version of C* along with CentOS 
Linux 7.9.2009 and Java 8.We are looking for help in resolving this issue. If you 
have any suggestions, please let us know.Thank you,vaibhav