Re: Survey about the parsing of the tooling's output

We parse the output of the following nodetool sub-commands in our custom scripts: * status * netstats * tpstats * ring We don't mind the output format change between major releases as long as all the following are true: 1. major releases are not too frequent e.g. no more frequent

Re: Impact of column names on storage

Actually, I was wrong. The column names are not stored in the *-Data.db files, but stored in the *-Statistics.db files. Cassandra only stores one copy of the column names per SSTable data file, therefore the disk space usage is negligible. On 12/06/2023 14:31, Bowen Song wrote: The SSTable

Re: Issue while node addition on cassandra 4.0.7

The expected value "-900387334" is the little endian decimal representation of the PROTOCOL_MAGIC value 0xCA552DFA defined in the net/Message.java file. The

Re: Issue while node addition on cassandra 4.0.7

29/06/2023 12:42, Bowen Song wrote: Did anyone connecting to the servers' storage port via telnet, nc (netcat) or something similar? 218762506 is 0x0D0A0D0A, which is two newlines. On 29/06/2023 11:49, MyWorld wrote: When checked in the source nodes, we got similar errors. Forgot to mention, w

Re: Issue while node addition on cassandra 4.0.7

essage$InvalidLegacyProtocolMagic: Read 218762506, Expected -900387334 On Thu, Jun 29, 2023 at 2:57 PM Bowen Song via user wrote: The expected value "-900387334" is the little endian decimal representation of the PROTOCOL_MAGIC value 0xCA552DFA defined in the net/Message.java <

Re: Is cleanup is required if cluster topology changes

Have you thought of using "-Dcassandra.replace_address_first_boot=..." (or "-Dcassandra.replace_address=..." if you are using an older version)? This will not result in a topology change, which means "nodetool cleanup" is not needed after the operation is completed. On 05/05/2023 05:24,

Re: Is cleanup is required if cluster topology changes

wn during the node replacement period, we will get availability drop because most of our use case is local_quorum with replication factor 3. On Fri, May 5, 2023 at 5:59 AM Bowen Song via user wrote: Have you thought of using "-Dcassandra.replace_address_first_boot=..." (or

Re: Is cleanup is required if cluster topology changes

d, and see how it goes. We will keep you updated on our progress. Thanks a lot once again! Jaydeep On Fri, May 5, 2023 at 8:55 AM Bowen Song via user wrote: Depending on the number of vnodes

Re: Over streaming in one node during repair.

Some common causes of over-streaming: * "repair_session_space" is too small (either manually specified, or heap size is small and data on disk is large) * Manually deleting SSTable files * Unexpected foreign (e.g. from a backup) SSTable files * Marking SSTable as repaired or unrepaired

Re: Switching to Incremental Repair

Not disabling auto-compaction may result in repaired SSTables getting compacted together with unrepaired SSTables before the repair state is set on them, which leads to mismatch in the repaired data between nodes, and potentially very expensive over-streaming in a future full repair. You

Re: Switching to Incremental Repair

Just one more thing. Make sure you run 'nodetool repair -full' instead of just 'nodetool repair'. That's because the command's default was changed in Cassandra 2.x. The default was full repair before that change, but the new default now is incremental repair. On 07/02/2024 10:28, Bowen Song

Re: Switching to Incremental Repair

, we have nodes where the disk usage is multiple TiBs so significant growth can be quite dangerous in our case. Would the only safe choice be to mark all SSTables as unrepaired before stopping regular incremental repair? Thanks, Kristijonas On Wed, Feb 7, 2024 at 4:33 PM Bowen Song via user

Re: Requesting Feedback for Cassandra as a backup solution.

Hi Gowtham, On the face of it, it sounds like you are planning to use Cassandra for a queue-like application, which is a well documented anti-pattern. If that's not the case, can you please show the table schema and some example queries? Cheers, Bowen On 17/02/2024 08:44, Gowtham S wrote:

Re: Requesting Feedback for Cassandra as a backup solution.

duce those to the respective topic when Kafka is live. Thanks and regards, Gowtham S On Sat, 17 Feb 2024 at 18:10, Bowen Song via user wrote: Hi Gowtham, On the face of it, it sounds like you are planning to use Cassandra for a queue-like application, which is a well documente

Re: Switching to Incremental Repair

Unfortunately repair doesn't compare each partition individually. Instead, it groups multiple partitions together and calculate a hash of them, stores the hash in a leaf of a merkle tree, and then compares the merkle trees between replicas during a repair session. If any one of the partitions

Re: Switching to Incremental Repair

Caution, using the method you described, the amount of data streamed at the end with the full repair is not the amount of data written between stopping the first node and the last node, but depends on the table size, the number of partitions written, their distribution in the ring and the

Re: Switching to Incremental Repair

schrieb Bowen Song via user : Unfortunately repair doesn't compare each partition individually. Instead, it groups multiple partitions together and calculate a hash of them, stores the hash in a leaf of a merkle tree, and then compares the merkle trees between replicas during a repair session

Re: Remove folders of deleted tables

nfirm (or invalidate) that please? Sébastien. Le mer. 6 déc. 2023 à 03:00, Bowen Song via user a écrit : The same table name with two different CF IDs is not just "temporary schema disagreements", it's much worse than that. This breaks the eventual consistency guarantee, and leads

Re: About Map column

Hi Sebastien, It's a bit more complicated than that. To begin with, the first-class citizen in Cassandra is partition, not row. All map fields in the same row are in the same partition, and all rows with the same partition key but different clustering keys are also in the same partition.

Re: Schema inconsistency in mixed-version cluster

I don't recognise those names: * channel_data_id * control_system_type * server_id * decimation_levels I assume these are column names of a non-system table. From the stack trace, this looks like an error from a node which was running 4.1.3, and this node was not the coordinator for this

Re: system_schema.tables id and table uuid on disk mismatch

It sounds like you have done some concurrent table creation/deletion in the past (e.g. CREATE TABLE IF NOT EXISTS from multiple clients), which resulted in this mismatch. After you restarted the node, Cassandra corrected it by discarding the old table ID and any data associated with it. This

Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch

data before nodetool import. Thanks again. *From:* Bowen Song via user *Sent:* Thursday, January 18, 2024 1:17:11 PM *To:* user@cassandra.apache.org *Cc:* Bowen Song *Subject:* COMMERCIAL:Re: system_schema.tables id

Re: COMMERCIAL:Re: COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch

. removing the incorrect directory afterwards *From:* Bowen Song via user *Sent:* Thursday, January 18, 2024 5:34:57 PM *To:* user@cassandra.apache.org *Cc:* Bowen Song *Subject:* COMMERCIAL:Re: COMMERCIAL:Re: COMMERCIAL:Re

Re: COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch

: It has same mismatch id in all nodes not just one node. *From:* Bowen Song via user *Sent:* Thursday, January 18, 2024 3:18:11 PM *To:* user@cassandra.apache.org *Cc:* Bowen Song *Subject:* COMMERCIAL:Re: COMMERCIAL:Re

Re: Remove folders of deleted tables

er has 65K subfolders, so I would say I have time to think of redesigning the data model ^^ Nevertheless, does it sound too much in terms of thombstones in the systems tables (with the default GC grace period of 10 days)? Sébastien. Le mar. 5 déc. 2023, 12:19, Bowen Song via user a écrit :

Re: Remove folders of deleted tables

Please rethink your use case. Create and delete tables concurrently often lead to schema disagreement. Even doing so on a single node sequentially will lead to a large number of tombstones in the system tables. On 04/12/2023 19:55, Sébastien Rebecchi wrote: Thank you Dipan. Do you know if

Re: Migrating to incremental repair in C* 4.x

Hi Jeff, Does subrange repair mark the SSTable as repaired? From my memory, it doesn't. Regards, Bowen On 27/11/2023 16:47, Jeff Jirsa wrote: I don’t work for datastax, thats not my blog, and I’m on a phone and potentially missing nuance, but I’d never try to convert a cluster to IR by

Re: Memory and caches

Hi Sebastien, What's your goal? Improving cache hit rate purely for the sake of having a higher hit rate is rarely a good goal, because higher cache hit rate doesn't always mean faster operations. Do you have specific issues with performance? If so, can you please tell us more about it?

Re: Migrating to incremental repair in C* 4.x

Hi Sebastian, It's better to walk down the path on which others have walked before you and had great success, than a path that nobody has ever walked. For the former, you know it's relatively safe and it works. The same can hardly be said for the later. You said it takes a week to run the

Re: Remove folders of deleted tables

1st use case, and then perform a small number of queries to merge pre-results client-side) and in that case TTL+TWCS would probably apply, it remains the same question as above. Thanks for your time :) Sébastien. Le mer. 6 déc. 2023 à 15:46, Bowen Song via user a écrit : There ar

Re: Switching to Incremental Repair

Hi Kristijonas, To answer your questions: 1. It's still necessary to run full repair on a cluster on which incremental repair is run periodically. The frequency of full repair is more of an art than science. Generally speaking, the less reliable the storage media, the more frequently full

Re: Tests failing for ppc64le architecture.

Hi Sunidhi, In case you haven't noticed, this is the Cassandra user mailing list, not the dev mailing list. Most people in this mailing list have never attempted to built Cassandra from the source code. IMHO you should try the Cassandra dev mailing list for this type of things. Cheers,

Re: Switching to Incremental Repair

condition? Thanks, Kristijonas On Fri, Feb 2, 2024 at 3:36 PM Bowen Song via user wrote: Hi Kristijonas, To answer your questions: 1. It's still necessary to run full repair on a cluster on which incremental repair is run periodically. The frequency of full repair is more

Re: Switching to Incremental Repair

Full repair running for an entire week sounds excessively long. Even if you've got 1 TB of data per node, 1 week means the repair speed is less than 2 MB/s, that's very slow. Perhaps you should focus on finding the bottleneck of the full repair speed and work on that instead. On 03/02/2024

Re: Cassandra 4.1 compaction thread no longer low priority (cpu nice)

Hi Pierre, Is there anything stopping you from using the compaction_throughput option in the cassandra.yaml file to manage the performance impact of compaction operations? With

Re: Cassandra 4.1 compaction thread no longer low priority (cpu nice)

commit/f382fb0bcef4c37dc049e9f6963e3baf204d815c). Regards, Dmitry On Thu, 22 Feb 2024 at 15:30, Bowen Song via user wrote: Hi Pierre, Is there anything stopping you from using the compaction_throughput <https://github.com/apache/cassandra/blob/f9e033f519c14596da4dc954875756a69aea4e78/conf/cassandra.

Re: Best Practices for Managing Concurrent Client Connections in Cassandra

They are suitable for production use for protecting your Cassandra server, not the clients. The clients likely will experience an error when the limit is reached, and it needs to handle that error appropriately. What you really want to do probably are: 1. change the client's behaviour, limit

Re: About Cassandra stable version having Java 17 support

Why Java 17? It makes no sense to choose an officially non-supported library version for a piece of software. That decision making process is the problem, not the software's library version compatibility. On 18/03/2024 09:44, Divyanshi Kaushik via user wrote: Hi All, As per my project

Re: [EXTERNAL] Re: About Cassandra stable version having Java 17 support

you please let us know when the team is planning to GA Cassandra 5.0 version which has Java 17 support? Regards, Divyanshi *From:* Bowen Song via user *Sent:* Monday, March 18, 2024 5:14 PM *To:* user@cassandra.apache.org

Re: Alternate apt repo for Debian installation?

You can try https://archive.apache.org/dist/cassandra/debian/ The deb files can be found here: https://archive.apache.org/dist/cassandra/debian/pool/main/c/cassandra/ On 20/03/2024 20:47, Grant Talarico wrote: Hi there. Hopefully this is the right place to ask this question. I'm trying to

Re: Trouble with using group commitlog_sync

If I set commitlog_sync_group_window to 1ms, the throughput increases to 13k / second, which is slightly less than batch commit mode. Is group commit mode supposed to have better performance than batch mode? On Tue, Apr 23, 2024 at 8:46 AM Bowen Song via user wrote: T

Re: Trouble with using group commitlog_sync

Why would you want to set commitlog_sync_batch_window to 1 second long when commitlog_sync is set to batch mode? The documentation on this says: /This window should be kept short because the writer threads

Re: Trouble with using group commitlog_sync

o the default 1000ms. On Tue, Apr 23, 2024 at 8:15 AM Bowen Song via user wrote: Why would you want to set commitlog_sync_batch_window to 1 second long when commitlog_sync is set to batch mode? The documentation <https://cassandra.apache.org/doc/stable/cassandra/architecture/stor

Re: Trouble with using group commitlog_sync

23, 2024 at 10:45 AM Bowen Song via user wrote: I suspect you are abusing batch statements. Batch statements should only be used where atomicity or isolation is needed. Using batch statements won't make inserting multiple partitions faster. In fact, it often will make that slower

Re: Trouble with using group commitlog_sync

, I do see 100% CPU utilization in every single one of these tests. On Tue, Apr 23, 2024 at 11:01 AM Bowen Song via user wrote: Have you checked the thread CPU utilisation of the client side? You likely will need more than one thread to do insertion in a loop to achieve tens

Re: Mixed Cluster 4.0 and 4.1

having a schema mismatch for this long time. Should I be concerned, or have others upgraded in a similar way? Thanks Paul On 24 Apr 2024, at 17:02, Bowen Song via user wrote: Hi Paul, You don't need to plan for or introduce an outage for a rolling upgrade, which is the preferred route

Re: Mixed Cluster 4.0 and 4.1

Hi Paul, You don't need to plan for or introduce an outage for a rolling upgrade, which is the preferred route. It isn't advisable to take down an entire DC to do upgrade. You should aim to complete upgrading the entire cluster and finish a full repair within the shortest gc_grace_seconds

Re: Trouble with using group commitlog_sync

, Apr 23, 2024 at 10:24 PM Bowen Song via user wrote: You might have run into the bottleneck of the driver's IO thread. Try increase the driver's connections-per-server limit to 2 or 3 if you've only got 1 server in the cluster. Or alternatively, run two client processes

Re: Trouble with using group commitlog_sync

:46 PM Bowen Song via user wrote: To achieve 10k loop iterations per second, each iteration must take 0.1 milliseconds or less. Considering that each iteration needs to lock and unlock the semaphore (two syscalls) and make network requests (more syscalls), that's a lots

Re: compaction trigger after every fix interval

There's many things that can trigger a compaction, knowing the type of compaction can help narrow it down. Have you looked at the nodetool compactionstats command output when it is happening? What is the compaction type? It can be "compaction", but can also be something else, such as

Re: Change num_tokens in a live cluster

You can also add a new DC with the desired number of nodes and num_tokens on each node with auto bootstrap disabled, then rebuild the new DC from the existing DC before decommission the existing DC. This method only needs to copy data once, and can copy from/to multiple nodes concurrently,

Re: Change num_tokens in a live cluster

to be moved? On 16/05/2024 15:54, Gábor Auth wrote: Hi, On Thu, 16 May 2024, 10:37 Bowen Song via user, wrote: You can also add a new DC with the desired number of nodes and num_tokens on each node with auto bootstrap disabled, then rebuild the new DC from the existing DC before

<    1   2   3