Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread vaibhav khedkar
Thank you patrick. We have plans for upgrades anyway so keep this issue in mind and probably expedite it. I have updated and created a bug https://issues.apache.org/jira/browse/CASSANDRA-18770 in case you are interested. Thanks vaibhav On Wed, Aug 16, 2023 at 1:34 PM Patrick Lee wrote: > I

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread Patrick Lee
I don’t have a ticket.  What I saw in a scenario was a cluster that was upgraded from 3.11 to 4.0.X, we added another ring that was running Java 11.  Nodes on the ring with Java 8 saw this issue you described while the other ring running Java 11 did not.  Then if I updated from Java 8 to Java 11 I

Re: Unsubscribe

2023-08-16 Thread C. Scott Andreas
Hi Mark,You can unsubscribe from this mailing list by sending a blank email to "user-unsubscr...@cassandra.apache.org" from the address that is subscribed to the list. Other members of the list are not able to take this action on someone's behalf.Details on how to join and leave lists are

Unsubscribe

2023-08-16 Thread Mark Furlong
Please unsubscribe from this list Thanks Mark Furlong Sr. Database Administrator mfurl...@ancestry.com M: 801-859-7427 O: 801-705-7115 1300 W Traverse Pkwy Lehi, UT 84043 [http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif] We empower journeys of personal

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread vaibhav khedkar
Thanks Patrick, We do have plans to upgrade to *java 11* eventually but we will go through internal testing and would also need some time given the size of our infrastructure. Is it safe to assume that the issue exists in the combination of upgrades from 3.11.x to 4.0.x *and* running on JAVA 8

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread Patrick Lee
I've actually noticed this as well on a few clusters I deal with but after upgrading Cassandra from 3.11 to 4 we also changed to use Java 11 shortly after the cluster upgrade. After I moved to Java 11 I have not experienced a problem. On Wed, Aug 16, 2023 at 12:12 PM vaibhav khedkar wrote: >

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread vaibhav khedkar
Thank you Scott We are seeing it for all the tables (Filter, Data ..etc ) /nb-1-big-Statistics.db.tmp (deleted) /nb-3-big-Statistics.db.tmp (deleted) /nb-2-big-Data.db (deleted) /nb-2-big-Statistics.db.tmp (deleted) /nb-2-big-Index.db (deleted) /nb-2-big-Statistics.db.tmp (deleted)

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread C. Scott Andreas
Vaibhav, thank you for reaching out and sharing this issue report.Could you run an `lsof` and share which SSTable files you see open (e.g., all SSTable components or a subset of them); and also share the value of the `disk_access_mode` property from your cassandra.yaml?Opening a Jira ticket

Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread vaibhav khedkar
Hi everyone, We recently upgraded our fleet of ~2500 Cassandra instances from 3.11.9 to 4.0.5. After the upgrade, we are seeing a unique issue where the compacted SSTables's file descriptors are still present and are never cleared. This is causing false disk alerts. We have to restart nodes very

Re: Big Data Question

2023-08-16 Thread Jeff Jirsa
A lot of things depend on actual cluster config - compaction settings (LCS vs STCS vs TWCS) and token allocation (single token, vnodes, etc) matter a ton. With 4.0 and LCS, streaming for replacement is MUCH faster, so much so that most people should be fine with 4-8TB/node, because the rebuild

Big Data Question

2023-08-16 Thread Joe Obernberger
General question on how to configure Cassandra.  Say I have 1PByte of data to store.  The general rule of thumb is that each node (or at least instance of Cassandra) shouldn't handle more than 2TBytes of disk.  That means 500 instances of Cassandra. Assuming you have very fast persistent