Re: 99.999% uptime - Operations Best Practices?

2011-06-22 Thread C. Scott Andreas
food for thought. - Scott --- C. Scott Andreas Engineer, Urban Airship, Inc. http://www.urbanairship.com On Jun 22, 2011, at 4:16 PM, Les Hazlewood wrote: On Wed, Jun 22, 2011 at 4:11 PM, Peter Lin wool...@gmail.com wrote: you have to use multiple data centers to really deliver 4 or 5 9's

Flush / Snapshot Triggering Full GCs, Leaving Ring

2011-04-06 Thread C. Scott Andreas
+ promotion failures triggering full GCs during a snapshot. Has anyone seen this, or have suggestions on how to prevent full GCs from occurring during a flush / snapshot? Thanks, - Scott --- C. Scott Andreas Engineer, Urban Airship, Inc. http://www.urbanairship.com

Re: state of incremental repairs in cassandra 3.x

2021-09-16 Thread C. Scott Andreas
Hi James, thanks for reaching out.A large number of fixes have landed for Incremental Repair in the 3.x series, though it's possible some may have been committed to 4.0 without a backport. Incremental repair works well on Cassandra 4.0.1. I'd start here to ensure you're picking up all fixes

Re: gc throughput

2021-11-17 Thread C. Scott Andreas
Hi, I noticed that you mentioned your goal is to optimize write throughput, and that you're using Cassandra 3.11.2.Optimizing for write throughput is usually a proxy for optimizing for compaction, as the cost of writes are very cheap but compacting to keep up with it can be pretty

Re: Cassandra 3.0.14 transport completely blocked

2022-03-22 Thread C. Scott Andreas
Hi Jaydeep, thanks for reaching out.The most notable deadlock identified and resolved in the last few years is https://issues.apache.org/jira/browse/CASSANDRA-15367: Memtable memory allocations may deadlock (fixed in Apache Cassandra 3.0.21).Mentioning for completeness - since the release of

Re: Query timed out after PT2M

2022-02-03 Thread C. Scott Andreas
Hi Joe, it looks like "PT2M" may refer to a timeout value that could be set by your Spark job's initialization of the client. I don't see a string matching this in the Cassandra codebase itself, but I do see that this is parseable as a Duration.```jshell>

Re: Hanging repairs in Cassandra

2022-01-21 Thread C. Scott Andreas
Hi Manish, I understand this answer is non-specific and might not be the most helpful, but figured I’d mention — Cassandra 3.11.2 is nearly four years old and a large number of bugs in repair and other subsystems have been resolved in the time since. I’d recommend upgrading to the latest

Re: [HELP] Cassandra 4.1.1 Repeated Bootstrapping Failure

2023-09-11 Thread C. Scott Andreas
Bowen, thanks for reaching out.My mind immediately jumped to a ticket which has very similar pathology: "CASSANDRA-18110: Streaming progress virtual table lock contention can trigger TCP_USER_TIMEOUT and fail streaming" -- but I see this was fixed in 4.1.1.On Sep 11, 2023, at 2:09 PM, Bowen

Re: Upgrade from C* 3 to C* 4 per datacenter

2023-10-26 Thread C. Scott Andreas
The recommended approach to upgrading is to perform a replica-safe rolling restart of instances in each datacenter, one datacenter at a time. > In case of an upgrade failure, would it be possible to remove the data center from the cluster, restore the datacenter to C*3 SW and add it back to

Re: Big Data Question

2023-08-17 Thread C. Scott Andreas
A few thoughts on this:– 80TB per machine is pretty dense. Consider the amount of data you'd need to re-replicate in the event of a hardware failure that takes down all 80TB (DIMM failure requiring replacement, non-reduntant PSU failure, NIC, etc).– 24GB of heap is also pretty generous.

Re: Open File Descriptors not cleared post upgrade from 3.11.9 to 4.0.5.

2023-08-16 Thread C. Scott Andreas
Vaibhav, thank you for reaching out and sharing this issue report.Could you run an `lsof` and share which SSTable files you see open (e.g., all SSTable components or a subset of them); and also share the value of the `disk_access_mode` property from your cassandra.yaml?Opening a Jira ticket

Re: Unsubscribe

2023-08-16 Thread C. Scott Andreas
Hi Mark,You can unsubscribe from this mailing list by sending a blank email to "user-unsubscr...@cassandra.apache.org" from the address that is subscribed to the list. Other members of the list are not able to take this action on someone's behalf.Details on how to join and leave lists are

Re: Testing Cassandra connectivity at application startup

2023-08-25 Thread C. Scott Andreas
“select * from …” without a predicate from a user table would be very expensive, yes. A query from a small, node-local system table such as “select * from system.peers” would make a better health check.  - Scott > On Aug 25, 2023, at 10:58 AM, Raphael Mazelier wrote: > >  > Mind that a

Re: Datastax Java Driver Compatibility Matrix

2022-04-19 Thread C. Scott Andreas
The DataStax Java 3.x drivers work very well with Apache Cassandra 4.0. I'd recommend one of the more recent releases in the series, though (e.g., 3.6.x+).I'm not the author of this documentation, but it may refer to the fact that the 3.x Java Driver supports the CQL v4 wire protocol, but not

Re: Datastax Java Driver Compatibility Matrix

2022-04-19 Thread C. Scott Andreas
the documentation I see that 3.2 supports upto V5 version of protocol.Does this mean a) 3.2 driver with V3 protocol works for cassandra 4.0 or b) I have to change the protocol version to V4 or higher on 3.2 to be able to work with 4.0?On Tue, Apr 19, 2022 at 11:15 AM C. Scott Andreas wrote:The

Re: Cassandra 3.0 upgrade

2022-06-13 Thread C. Scott Andreas
Thank you for reaching out, and for planning the upgrade! Upgrading from 3.0.14 to 3.0.27 would be best, followed by upgrading to 4.0.4. 3.0.14 contains a number of serious bugs that are resolved in more recent 3.0.x releases (3.0.19+ are generally good/safe). Upgrading to 3.0.27 will put you

Re: Cassandra 3.0 upgrade

2022-06-13 Thread C. Scott Andreas
work fine? > > Jaydeep > >> On Mon, Jun 13, 2022 at 10:25 PM C. Scott Andreas >> wrote: >> Thank you for reaching out, and for planning the upgrade! >> >> Upgrading from 3.0.14 to 3.0.27 would be best, followed by upgrading to >> 4.0.4. >> &

Re: Gossip issues after upgrading to 4.0.4

2022-06-06 Thread C. Scott Andreas
Hi Gil, thanks for reaching out.Can you check Cassandra's logs to see if any uncaught exceptions are being thrown? What you described suggests the possibility of an uncaught exception being thrown in the Gossiper thread, preventing further tasks from making progress; however I'm not aware of

Re: Change the compression algorithm on a production table at runtime

2022-09-20 Thread C. Scott Andreas
Thanks for reaching out. Changing the compressor for a table is both safe and common. Future flushes / compactions will use the new codec as SSTables are written, and SSTables currently present on disk will remain readable with the previous codec. You may also want to take a look at the

Re: Understanding multi region read query and latency

2022-08-07 Thread C. Scott Andreas
> but still as I understand the documentation the read repair should not be in the blocking path of a query ?Read repair is in the blocking read path for the query, yep. At quorum consistency levels, the read repair must complete before returning a result to the client to ensure the data returned

Re: Cassandra 4.0 upgrade - Upgradesstables

2022-08-16 Thread C. Scott Andreas
No downside at all for 3.x -> 4.x (however, Cassandra 3.x reading 2.1 SSTables incurred a performance hit).Many users of Cassandra don't run upgradesstables after 3.x -> 4.x upgrades at all. It's not necessary to run until a hypothetical future time if/when support for reading Cassandra 3.x

Re: Compactions are stuck in 4.0.5 version

2023-01-13 Thread C. Scott Andreas
Hi Vaibhav, thanks for reaching out.Based on my understanding of this exception, this may be due to the index for this partition exceeding 2GiB (which is *extremely* large for a partition index component).Reducing the size of the column index below 2GiB may resolve this issue. You may be able

Re: Cassandra Summit update for 2023-01-24

2023-01-25 Thread C. Scott Andreas
Hugely excited to this – thanks to the Program Committee and to the Linux Foundation for organizing!It's been a long few years away from conferences and I can't wait to see all of you.Beyond learning about what everyone is doing with Apache Cassandra, I'm looking forward to the hallway chats

Re: Failed disks - correct procedure

2023-01-17 Thread C. Scott Andreas
Bumping this note from Andy downthread to make sure everyone has seen it and is aware:“Before you do that, you will want to make sure a cycle of repairs has run on the replicas of the down node to ensure they are consistent with each other.”When replacing an instance, it’s necessary to run repair

Re: Bootstrapping new node throwing error - Mutation too large

2023-03-01 Thread C. Scott Andreas
The performance implications would primarily be due to the challenge of handling mutations this large themselves rather than the commitlog segment size. These would occupy large, contiguous areas of heap and increase memory pressure in the process.Increasing  commit_log_segment_size_in_mb is

Re: Deletions getting omitted

2023-02-06 Thread C. Scott Andreas
Can you check the write timestamp of the data you're attempting to delete?https://docs.datastax.com/en/cql-oss/3.3/cql/cql_using/useWritetime.htmlIf the timestamp of the write is in the future (e.g., due to a time sync issue or an errant client-supplied timestamp at the time of that write),

Re: Materialized View inconsistency issue

2023-08-08 Thread C. Scott Andreas
That’s correct, yes. There is no current or upcoming version of Apache Cassandra in which materialized views are expected to be considered production-ready and maintain full consistency with their base table at this time.The feature is classified as “experimental” to indicate that this behavior is

Re: Question Regarding Cassandra-19336

2024-02-21 Thread C. Scott Andreas
The “Since Version” for the ticket is set to 3.0.19, presumably based on C-14096 as the predecessor for this ticket. C-14096 was merged up into 3.11.x in the 3.11.5 release, so 3.11.5 would be the equivalent “since version” for that release series. The patch addressing this ticket is included

Re: Upgrade from 3.11.x to 4.1.3

2024-01-06 Thread C. Scott Andreas
Upgrading from 3.11.x to 4.1.x is supported, yes. As the documentation you reference mentions, it is not possible to downgrade from 4.x to 3.x. Note that running repair during upgrades is not supported; please ensure it is disabled before beginning the upgrade and re-enable after. – Scott >

Re: Tests failing for ppc64le architecture.

2024-01-31 Thread C. Scott Andreas
Pardon me, that should read user-unsubscr...@cassandra.apache.org for this list. On Jan 31, 2024, at 10:07 AM, C. Scott Andreas wrote: Hi Matt, To unsubscribe from this list, send a blank email to dev-unsubscr...@cassandra.apache.org . All messages or replies to the list are distributed to all

Re: Tests failing for ppc64le architecture.

2024-01-31 Thread C. Scott Andreas
Hi Matt, To unsubscribe from this list, send a blank email to dev-unsubscr...@cassandra.apache.org . All messages or replies to the list are distributed to all subscribers of the list. As the project is volunteer-run, others are not able to take this action on behalf of subscribers. For more

Re: null values injected while drop compact storage was executed

2024-05-07 Thread C. Scott Andreas
If you don't have an explicit goal of dropping compact storage, it's not necessary to as a prerequisite to upgrading to 4.x+. Development community members recognized that introducing mandatory schema changes as a prerequisite to upgrading to 4.x would increase operator + user overhead and