Re: About Map column
Hello Bowen, Arvydas, Thanks for your answers. May I clarify things about compaction. If I have a table with LCS with 160MB of sstable size target, if a partition has many rows so that the total size of the partition is greater than 160MB, then there will eventually be an sstable greater than 160MB, because the compaction will force the entire partition to reside in the same sstable, and this without hard limit? And this is the same whatever the table schema, with Map but even with clustering keys, all rows of the same partition will be forced to live in the same sstable? Sébastien. Le mar. 19 déc. 2023 à 06:57, Arvydas Jonusonis a écrit : > Sebastien, > > Another thing to keep in mind when writing/updating a map column is that > it is internally (in the memtable) backed by a synchronized data structure > - if the rate of writes/updates is sufficiently high, the resulting CPU > load will cripple the nodes (see CASSANDRA-15464 > <https://issues.apache.org/jira/browse/CASSANDRA-15464> - this mentions > sets, but same is true for maps as well). > > Arvydas > > > > On Mon, Dec 18, 2023 at 10:06 AM Bowen Song via user < > user@cassandra.apache.org> wrote: > >> Hi Sebastien, >> >> It's a bit more complicated than that. >> >> To begin with, the first-class citizen in Cassandra is partition, not >> row. All map fields in the same row are in the same partition, and all >> rows with the same partition key but different clustering keys are also >> in the same partition. During a compaction, Cassandra does its best not >> to split a partition into multiple SSTables, unless it must, e.g. when >> dealing with repaired vs unrepaired data. That means regardless it's a >> map field in a row or multiple rows within same partition, they get >> compacted into the same number of SSTables. >> >> A map type field's data may live in one column, but definitely not just >> one blob of data from the server's perspective, unless it's frozen. >> Reading such data is no cheaper than reading multiple columns and rows >> within the same partition, as each components of it, a key or a value, >> needs to be deserialised individually from the on-disk SSTable format, >> and then serialised again for the network protocol (often called the >> native protocol, NTP, or binary protocol) when it is read by a CQL client. >> >> There's no obvious performance benefit for reading key-value pairs from >> a map field in a row vs columns and rows in the same partition. However, >> each row can be read separately and selectively, but key-value pairs in >> a map cannot. All data in a map field must be fetched all at once. So if >> you ever need to selectively read the data, reading multiple columns and >> rows in the same partition filtered by clustering keys will actually >> perform better than reading all key-value pairs from a large map type >> field and then discarding the unwanted data. >> >> If you really want better server-side read performance and always read >> the whole thing, you should consider use a frozen map or frozen UDT >> instead. Of course, there's a cost to freeze them. A frozen data cannot >> be partially modified (e.g. add, remove or update a value in it), it can >> only be deleted or overwritten with new data at once. Which means it may >> not be suitable for your use case. >> >> I can see you also mentioned big partitions. Large partitions in >> Cassandra usually is a bad idea, regardless it's a single row with a few >> columns or many rows with many columns. There's some exceptions that may >> work well, but generally you should avoid creating large partitions if >> possible. The problem with large partitions is usually the JVM heap and >> GC pauses, rarely CPU or disk resources. >> >> Regards, >> Bowen >> >> >> On 18/12/2023 17:00, Sébastien Rebecchi wrote: >> > Hello >> > >> > If i have a colum of type Map, then with many insertions, the map >> > grows, but after compation, as the full map is 1 column of a table, >> > will it be contained fully in 1 SSTable? >> > I guess yes cause the map is contained in a single row. Am I right? >> > Versus if we use a clustering key + a standard column instead of a >> > map, insertions will create many rows, 1 per clustering key value, so >> > even after compaction the partition could be splitted in several >> SSTables. >> > Can you tell me if i understood correctly please? Because if it is >> > right then it means the pb of big partitions can be enhanced using Map >> > as it will induce much more CPU and disk resources to perform >> > compaction (on the other hand you will have lower read amplification >> > factor with map). >> > >> > Thanks, >> > >> > Sébastien >> >
About Map column
Hello If i have a colum of type Map, then with many insertions, the map grows, but after compation, as the full map is 1 column of a table, will it be contained fully in 1 SSTable? I guess yes cause the map is contained in a single row. Am I right? Versus if we use a clustering key + a standard column instead of a map, insertions will create many rows, 1 per clustering key value, so even after compaction the partition could be splitted in several SSTables. Can you tell me if i understood correctly please? Because if it is right then it means the pb of big partitions can be enhanced using Map as it will induce much more CPU and disk resources to perform compaction (on the other hand you will have lower read amplification factor with map). Thanks, Sébastien
Re: Remove folders of deleted tables
Thanks Bowen, I also thought about using TTL and TWCS, but in my past experience with Cassandra I have had a lot of issues with data models using TTL and creating many tombstones. I was probably not using the right compaction at that time, but this experiences has a great impact on me and I would say they made me very cautious about TTL, even today, several years after ^^ Anyway, in the current 2 cases I can not put a date as partition key alone. In the first one I have a pair of columns acting as partition key, one is a customer id and the other is a date (more precisely a second "block" of time to group all events of the same second in the same partition, to pre-group data). The customer id is mandatory in the partition key in order not to have too wide partitions, and I never have cases where I need to fetch cross-customer data. Is TWCS still suited? As I read from doc, you don't need to have a time window as partition key alone, as long as many partitions will die approx in the same time it could work fine cause sometimes entire SSTables will be deleted during compaction rather than rewritten in disk. As for my second use case that creates many tables on demand, I can not have the time window in the PK, cause I think it would lead to degraded read performance, I have to put a visitor id in the PK and a timestamp as CK in order to fetch all event of a visitor inside a time window in 1 query. I could probably put a time window as PK (e.g. a second block like the 1st use case, and then perform a small number of queries to merge pre-results client-side) and in that case TTL+TWCS would probably apply, it remains the same question as above. Thanks for your time :) Sébastien. Le mer. 6 déc. 2023 à 15:46, Bowen Song via user a écrit : > There are many different ways to avoid or minimise the chance of schema > disagreements, the easiest way is to always send DDL queries to the same > node in the cluster. This is very easy to implement and avoids schema > disagreements at the cost of creating a single point of failure for DDL > queries. More sophisticated methods also exist, such as locking and > centralised schema modification, and you should consider which one is more > suitable for your use case. Ignoring the schema disagreements problem is > not recommended, as this is not a tested state for the cluster, you are > likely to run into some known and unknown (and possibly severe) issues > later. > > The system_schema.columns table will almost certainly have more tombstones > created than the number of tables deleted, unless each deleted table had > only one column. I doubt creating and deleting 8 tables per day will be a > problem, but I would recommend you find a way to test it before doing that > on a production system, because I don't know anyone else is using Cassandra > in this way. > > From the surface, it does sound like TWCS with the date in in the > partition key may fit your use case better than creating and deleting > tables every day. > > > On 06/12/2023 08:26, Sébastien Rebecchi wrote: > > Hello Jeff, Bowen > > Thanks for your answer. > Now I understand that there is a bug in Cassandra that can not handle > concurrent schema modifications, I was not aware of that severity, I > thought that temporary schema mismatches were eventually resolved smartly, > by a kind of "merge" mechanism. > For my use cases, keyspaces and tables are created "on-demand", when > receiving exceptions for invalid KS or table on insert (then the KS and > table are created and the insert is retried). I can not afford to > centralize schema modifications in a bottleneck, but I can afford the data > inconsistencies, waiting for the fix in Cassandra. > I'm more worried about tombstones in system tables, I assume that 8 > tombstones per day (or even more, but in the order of no more than some > dozens) is reasonable, can you confirm (or invalidate) that please? > > Sébastien. > > Le mer. 6 déc. 2023 à 03:00, Bowen Song via user < > user@cassandra.apache.org> a écrit : > >> The same table name with two different CF IDs is not just "temporary >> schema disagreements", it's much worse than that. This breaks the eventual >> consistency guarantee, and leads to silent data corruption. It's silently >> happening in the background, and you don't realise it until you suddenly >> do, and then everything seems to blow up at the same time. You need to sort >> this out ASAP. >> >> >> On 05/12/2023 19:57, Sébastien Rebecchi wrote: >> >> Hi Bowen, >> >> Thanks for your answer. >> >> I was thinking of extreme use cases, but as far as I am concerned I can >> deal with creation and deletion of 2 tables every 6 hours for a keyspace. >&
Re: Remove folders of deleted tables
Hello Jeff, Bowen
Thanks for your answer.
Now I understand that there is a bug in Cassandra that can not handle
concurrent schema modifications, I was not aware of that severity, I
thought that temporary schema mismatches were eventually resolved smartly,
by a kind of "merge" mechanism.
For my use cases, keyspaces and tables are created "on-demand", when
receiving exceptions for invalid KS or table on insert (then the KS and
table are created and the insert is retried). I can not afford to
centralize schema modifications in a bottleneck, but I can afford the data
inconsistencies, waiting for the fix in Cassandra.
I'm more worried about tombstones in system tables, I assume that 8
tombstones per day (or even more, but in the order of no more than some
dozens) is reasonable, can you confirm (or invalidate) that please?
Sébastien.
Le mer. 6 déc. 2023 à 03:00, Bowen Song via user
a écrit :
> The same table name with two different CF IDs is not just "temporary
> schema disagreements", it's much worse than that. This breaks the eventual
> consistency guarantee, and leads to silent data corruption. It's silently
> happening in the background, and you don't realise it until you suddenly
> do, and then everything seems to blow up at the same time. You need to sort
> this out ASAP.
>
>
> On 05/12/2023 19:57, Sébastien Rebecchi wrote:
>
> Hi Bowen,
>
> Thanks for your answer.
>
> I was thinking of extreme use cases, but as far as I am concerned I can
> deal with creation and deletion of 2 tables every 6 hours for a keyspace.
> So it lets around 8 folders of deleted tables per day - sometimes more
> cause I can see sometimes 2 folders created for a same table name, with 2
> different ids, caused by temporary schema disagreements I guess.
> Basically it means 20 years before the KS folder has 65K subfolders, so I
> would say I have time to think of redesigning the data model ^^
> Nevertheless, does it sound too much in terms of thombstones in the
> systems tables (with the default GC grace period of 10 days)?
>
> Sébastien.
>
> Le mar. 5 déc. 2023, 12:19, Bowen Song via user
> a écrit :
>
>> Please rethink your use case. Create and delete tables concurrently often
>> lead to schema disagreement. Even doing so on a single node sequentially
>> will lead to a large number of tombstones in the system tables.
>> On 04/12/2023 19:55, Sébastien Rebecchi wrote:
>>
>> Thank you Dipan.
>>
>> Do you know if there is a good reason for Cassandra to let tables folder
>> even when there is no snapshot?
>>
>> I'm thinking of use cases where there is the need to create and delete
>> small tables at a high rate. You could quickly end with more than 65K
>> (limit of ext4) subdirectories in the KS directory, while 99.9.. % of them
>> are residual of deleted tables.
>>
>> That looks quite dirty from Cassandra to not clean its own "garbage" by
>> itself, and quite dangerous for the end user to have to do it alone, don't
>> you think so?
>>
>> Thanks,
>>
>> Sébastien.
>>
>> Le lun. 4 déc. 2023, 11:28, Dipan Shah a écrit :
>>
>>> Hello Sebastien,
>>>
>>> There are no inbuilt tools that will automatically remove folders of
>>> deleted tables.
>>>
>>> Thanks,
>>>
>>> Dipan Shah
>>> --
>>> *From:* Sébastien Rebecchi
>>> *Sent:* 04 December 2023 13:54
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Remove folders of deleted tables
>>>
>>> Hello,
>>>
>>> When we delete a table with Cassandra, it lets the folder of that table
>>> on file system, even if there is no snapshot (auto snapshots disabled).
>>> So we end with the empty folder {data folder}/{keyspace name}/{table
>>> name-table id} containing only 1 subfolder, backups, which is itself empty.
>>> Is there a way to automatically remove folders of deleted tables?
>>>
>>> Sébastien.
>>>
>>
Re: Remove folders of deleted tables
Hi Bowen,
Thanks for your answer.
I was thinking of extreme use cases, but as far as I am concerned I can
deal with creation and deletion of 2 tables every 6 hours for a keyspace.
So it lets around 8 folders of deleted tables per day - sometimes more
cause I can see sometimes 2 folders created for a same table name, with 2
different ids, caused by temporary schema disagreements I guess.
Basically it means 20 years before the KS folder has 65K subfolders, so I
would say I have time to think of redesigning the data model ^^
Nevertheless, does it sound too much in terms of thombstones in the systems
tables (with the default GC grace period of 10 days)?
Sébastien.
Le mar. 5 déc. 2023, 12:19, Bowen Song via user
a écrit :
> Please rethink your use case. Create and delete tables concurrently often
> lead to schema disagreement. Even doing so on a single node sequentially
> will lead to a large number of tombstones in the system tables.
> On 04/12/2023 19:55, Sébastien Rebecchi wrote:
>
> Thank you Dipan.
>
> Do you know if there is a good reason for Cassandra to let tables folder
> even when there is no snapshot?
>
> I'm thinking of use cases where there is the need to create and delete
> small tables at a high rate. You could quickly end with more than 65K
> (limit of ext4) subdirectories in the KS directory, while 99.9.. % of them
> are residual of deleted tables.
>
> That looks quite dirty from Cassandra to not clean its own "garbage" by
> itself, and quite dangerous for the end user to have to do it alone, don't
> you think so?
>
> Thanks,
>
> Sébastien.
>
> Le lun. 4 déc. 2023, 11:28, Dipan Shah a écrit :
>
>> Hello Sebastien,
>>
>> There are no inbuilt tools that will automatically remove folders of
>> deleted tables.
>>
>> Thanks,
>>
>> Dipan Shah
>> --
>> *From:* Sébastien Rebecchi
>> *Sent:* 04 December 2023 13:54
>> *To:* user@cassandra.apache.org
>> *Subject:* Remove folders of deleted tables
>>
>> Hello,
>>
>> When we delete a table with Cassandra, it lets the folder of that table
>> on file system, even if there is no snapshot (auto snapshots disabled).
>> So we end with the empty folder {data folder}/{keyspace name}/{table
>> name-table id} containing only 1 subfolder, backups, which is itself empty.
>> Is there a way to automatically remove folders of deleted tables?
>>
>> Sébastien.
>>
>
Re: Remove folders of deleted tables
Thank you Dipan.
Do you know if there is a good reason for Cassandra to let tables folder
even when there is no snapshot?
I'm thinking of use cases where there is the need to create and delete
small tables at a high rate. You could quickly end with more than 65K
(limit of ext4) subdirectories in the KS directory, while 99.9.. % of them
are residual of deleted tables.
That looks quite dirty from Cassandra to not clean its own "garbage" by
itself, and quite dangerous for the end user to have to do it alone, don't
you think so?
Thanks,
Sébastien.
Le lun. 4 déc. 2023, 11:28, Dipan Shah a écrit :
> Hello Sebastien,
>
> There are no inbuilt tools that will automatically remove folders of
> deleted tables.
>
> Thanks,
>
> Dipan Shah
> ------
> *From:* Sébastien Rebecchi
> *Sent:* 04 December 2023 13:54
> *To:* user@cassandra.apache.org
> *Subject:* Remove folders of deleted tables
>
> Hello,
>
> When we delete a table with Cassandra, it lets the folder of that table on
> file system, even if there is no snapshot (auto snapshots disabled).
> So we end with the empty folder {data folder}/{keyspace name}/{table
> name-table id} containing only 1 subfolder, backups, which is itself empty.
> Is there a way to automatically remove folders of deleted tables?
>
> Sébastien.
>
Remove folders of deleted tables
Hello,
When we delete a table with Cassandra, it lets the folder of that table on
file system, even if there is no snapshot (auto snapshots disabled).
So we end with the empty folder {data folder}/{keyspace name}/{table
name-table id} containing only 1 subfolder, backups, which is itself empty.
Is there a way to automatically remove folders of deleted tables?
Sébastien.
Re: Memory and caches
Hello Bowen, Jon, Thank you for your answers. I will continue investigating following your advice, currently there is no performance issue, and we have good HW etc, so probably no need to optimize caches. Have a good day, Sébastien. Le lun. 27 nov. 2023 à 20:00, Jon Haddad a écrit : > I haven't found chunk cache to be particularly useful. It's a fairly > small cache that could only help when you're dealing with a small hot > dataset. I wouldn't bother increasing memory for it. > > Key cache can be helpful, but it depends on the workload. I generally > recommend optimizing for your HW first for the case where you don't hit > cache. > > Generally, cache is used to make up for issues with bottlenecked I/O. If > you haven't already done so, I recommend taking a look at what you're > actually doing in terms of device I/O (bitehist), compare that to what's > being requested to your filesystem (ebpf probe + histo on vfs_read) and > looking at your page cache hit rate with cachestat. You're likely to find > you've got a ton of read amplification due to either misconfigured > compression or read ahead, both of which can saturate your disks and make > it appear like you need to give more memory to cache. I always recommend > optimizing things for the worst cache (all cache misses) then use cache to > improve things vs papering over an underlying perf issue. > > I wrote a bunch about this recently: > > https://rustyrazorblade.com/post/2023/2023-11-07-async-profiler/ > https://rustyrazorblade.com/post/2023/2023-11-14-bcc-tools/ > https://rustyrazorblade.com/post/2023/2023-11-21-bpftrace/ > > Jon > > On 2023/11/27 14:59:55 Sébastien Rebecchi wrote: > > Hello > > > > When I use nodetool info, it prints that relevant information > > > > Heap Memory (MB) : 14229.31 / 32688.00 > > Off Heap Memory (MB) : 5390.57 > > Key Cache : entries 670423, size 100 MiB, capacity 100 MiB, > > 13152259 hits, 47205855 requests, 0.279 recent hit rate, 14400 save > period > > in seconds > > Chunk Cache: entries 63488, size 992 MiB, capacity 992 MiB, > > 143250511 misses, 162302465 requests, 0.117 recent hit rate, 2497.557 > > microseconds miss latency > > > > Here I focus on lines relevant for that conversation. And the numbers are > > roughly the same for all nodes of the cluster. > > The key and chunk caches are full and the hit rate is low. At the same > time > > the heap memory is far from being used at full capacity. > > I would say that I can significantly increase the sizes of those caches > in > > order to increase hit rate and improve performance. > > In cassandra.yaml, key_cache_size_in_mb has a blank value, so 100 MiB by > > default, and file_cache_size_in_mb is set to 1024. > > I'm thinking about setting key_cache_size_in_mb to 1024 > > and file_cache_size_in_mb to 2048. What would you recommend? Is anyone > > having good experience with tuning those parameters? > > > > Thank you in advance. > > > > Sébastien. > > >
Memory and caches
Hello When I use nodetool info, it prints that relevant information Heap Memory (MB) : 14229.31 / 32688.00 Off Heap Memory (MB) : 5390.57 Key Cache : entries 670423, size 100 MiB, capacity 100 MiB, 13152259 hits, 47205855 requests, 0.279 recent hit rate, 14400 save period in seconds Chunk Cache: entries 63488, size 992 MiB, capacity 992 MiB, 143250511 misses, 162302465 requests, 0.117 recent hit rate, 2497.557 microseconds miss latency Here I focus on lines relevant for that conversation. And the numbers are roughly the same for all nodes of the cluster. The key and chunk caches are full and the hit rate is low. At the same time the heap memory is far from being used at full capacity. I would say that I can significantly increase the sizes of those caches in order to increase hit rate and improve performance. In cassandra.yaml, key_cache_size_in_mb has a blank value, so 100 MiB by default, and file_cache_size_in_mb is set to 1024. I'm thinking about setting key_cache_size_in_mb to 1024 and file_cache_size_in_mb to 2048. What would you recommend? Is anyone having good experience with tuning those parameters? Thank you in advance. Sébastien.
Re: Cassandra 4 production settings
Thank you for the answer Scott! Le lun. 20 nov. 2023 à 16:57, a écrit : > Hi Sébastien, > > Thanks for reaching out. > > This documentation is from the DataStax website rather than the Apache > Cassandra project. However, no major changes suggested between 3.x vs. 4.x > regarding what’s published here, aside from JDK version. > > Cassandra 4.0 supports JDK11 and adopting it is recommended given JDK8’s > age. > > – Scott > > On Nov 20, 2023, at 12:31 AM, Sébastien Rebecchi > wrote: > > Hello > > There is no such page for Cassandra 4: > https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/install/installRecommendSettings.html > > Do the recommended settings for Cassandra 3 still apply as such to > Cassandra 4. > > Thank you in advance. > > Sébastien. > > >
Cassandra 4 production settings
Hello There is no such page for Cassandra 4: https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/install/installRecommendSettings.html Do the recommended settings for Cassandra 3 still apply as such to Cassandra 4. Thank you in advance. Sébastien.
Delete too wide partitions
Hi everyone Is there a way to tell Cassandra to automatically delete a partition when its size increase a given threshold? Best regard Sébastien
Re: Codec not found for list
Please ignore that message, I found my mistake, it was clear in the error message, I forgot to collect the Java stream as List... :) Sorry for the disturbance Le lun. 19 sept. 2022 à 11:08, Sébastien Rebecchi < sebastien.rebec...@gmail.com> a écrit : > Hello, > > I have a table where I store data in a column of type list > I get that error when inserting data > > com.datastax.oss.driver.api.core.type.codec.CodecNotFoundException: Codec > not found for requested operation: [List(BLOB, not frozen) <-> > java.util.stream.ReferencePipeline$3] > > Do you know how I can work around this? Is there any way to store binary > data in lists in Cassandra? > > Thank you. > > Sébastien. > >
Codec not found for list
Hello, I have a table where I store data in a column of type list I get that error when inserting data com.datastax.oss.driver.api.core.type.codec.CodecNotFoundException: Codec not found for requested operation: [List(BLOB, not frozen) <-> java.util.stream.ReferencePipeline$3] Do you know how I can work around this? Is there any way to store binary data in lists in Cassandra? Thank you. Sébastien.
Re: UDF: adding custom jar to classpath
Hi everybody! Is there someone that could help me on that please? I even tried to add the jar in $CASSANDRA_HOME/lib/jsr223/java/, but it is not very clear from the official documentation. https://docs.datastax.com/en/cql-oss/3.3/cql/cql_using/useCreateUDF.html Is there any Cassandra developer having the answer to that issue, how to add jar in classpath so that its classes are taken into account in UDF? Thanks you! Sébastien. Le mer. 23 mars 2022 à 17:30, Sébastien Rebecchi a écrit : > Hello, > > I am trying to create a UDF based on custom methods. > So I set enable_user_defined_functions to true and added a jar in > "/usr/share/cassandra/lib/" folder on every node, restarted the nodes and I > can see from the command line that the jar is indeed used (in the classpath > with -cp). > > But when i create the UDF I got that error: > > CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON NULL > INPUT RETURNS text LANGUAGE java AS 'return > com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; > InvalidRequest: Error from server: code=2200 [Invalid query] message="Java > source compilation failed: > Line 1: com.kameleoon.visit.Visit cannot be resolved to a type > Line 1: com.kameleoon.visit.Visit cannot be resolved to a type > > Of course the class com.kameleoon.visit.Visit do exist in the jar and the > jar as read rights to every user (chmod 444). So I can not find the reason. > > versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | Native > protocol v5] > > Any help would be appreciated! > > Thanks! > > Sébastien. >
UDF: adding custom jar to classpath
Hello, I am trying to create a UDF based on custom methods. So I set enable_user_defined_functions to true and added a jar in "/usr/share/cassandra/lib/" folder on every node, restarted the nodes and I can see from the command line that the jar is indeed used (in the classpath with -cp). But when i create the UDF I got that error: CREATE OR REPLACE FUNCTION blobToJson (input blob) RETURNS NULL ON NULL INPUT RETURNS text LANGUAGE java AS 'return com.kameleoon.visit.Visit.writeToJson(com.kameleoon.visit.Visit.readFromByteBuffer(input));'; InvalidRequest: Error from server: code=2200 [Invalid query] message="Java source compilation failed: Line 1: com.kameleoon.visit.Visit cannot be resolved to a type Line 1: com.kameleoon.visit.Visit cannot be resolved to a type Of course the class com.kameleoon.visit.Visit do exist in the jar and the jar as read rights to every user (chmod 444). So I can not find the reason. versions are: [cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | Native protocol v5] Any help would be appreciated! Thanks! Sébastien.
Reload Cassandra clathpath
Hi! I want to define an UDF based on a method which is defined in an external jar that I want to add in classpath of every node of my Cassandra cluster. However this external method is going to change with time, then the jar also will change and occasionally replace the current one installed on Cassandra nodes. I am wondering, is it possible to reload classpath of nodes without restarting them? Thank you. Best regards. Sébastien.
OversizedMessageException in AntiEntropyStage
Hi, Digging in Cassandra logs, I saw this exception happening several times, see trace below. I was wondering, what can cause such big messages, and could this error make a repair session fail? I currently see that my repairs are very long and sometimes I even stop them cause it seems they are just hanging and will never return. For information, I run repair (node+table) by (node+table), 1 after 1 sequentially, and I stop after 5 days stucking on the same (node+table). I see the repair session is still alive, the pid is here, no repair failure message, just nothing happens and repair % is not increasing. Is there something I can do to avoid that error? If I could only fix by changing the configuration, what is the configuration setting to increase? I guess it is commitlog_segment_size_in_mb, but I am not able to find a definite confirmation on the web. Here some informations of the installation: - Apache Cassandra 4.0 GA release - output of cqlsh: cqlsh 6.0.0 | Cassandra 4.0.1 | CQL spec 3.4.5 | Native protocol v5 - OS: CentOS Linux release 8.4.2105 - Output of nodetool status (hiding address column which is sensitive). As you can see cluster remains imbalanced 2 weeks after I joined the 2 last nodes (the ones having the lesser load), before it was perfectly balanced with the same data model, if this information may help: -- LoadTokens Owns (effective) Host ID Rack UN 712.72 GiB 8 32.6% 00f8bb86-5283-4b01-9819-fe4d59337680 rack1 UN 830.91 GiB 8 35.5% b9490ee5-44ba-4898-add7-159a7eeb06d9 rack1 UN 763.69 GiB 8 34.7% 931ccba9-9aef-4d79-9fa0-53e4654554f5 rack1 UN720.29 GiB 8 33.0% 4b5445ad-fb3c-419e-98b9-80599014e2b4 rack1 UN 273.6 GiB 8 22.8% 34db97d3-6c6d-4544-b656-9d4ae2a82dca rack1 UN616.92 GiB 8 41.3% 631d7b77-de74-4fe3-86a2-8bd3beec191e rack1 Thank you for your help! Sébastien. -- /var/log/cassandra/system.log:ERROR [AntiEntropyStage:1] 2021-11-05 00:48:28,074 CassandraDaemon.java:579 - Exception in thread Thread[AntiEntropyStage:1,5,main] /var/log/cassandra/system.log-org.apache.cassandra.net.Message$OversizedMessageException: Message of size 140580605 bytes exceeds allowed maximum of 134217728 bytes /var/log/cassandra/system.log- at org.apache.cassandra.net.OutboundConnection.enqueue(OutboundConnection.java:328) /var/log/cassandra/system.log- at org.apache.cassandra.net.OutboundConnections.enqueue(OutboundConnections.java:84) /var/log/cassandra/system.log- at org.apache.cassandra.net.MessagingService.doSend(MessagingService.java:338) /var/log/cassandra/system.log- at org.apache.cassandra.net.OutboundSink.accept(OutboundSink.java:70) /var/log/cassandra/system.log- at org.apache.cassandra.net.MessagingService.send(MessagingService.java:327) /var/log/cassandra/system.log- at org.apache.cassandra.net.MessagingService.send(MessagingService.java:314) /var/log/cassandra/system.log- at org.apache.cassandra.repair.Validator.respond(Validator.java:269) /var/log/cassandra/system.log- at org.apache.cassandra.repair.Validator.run(Validator.java:257) /var/log/cassandra/system.log- at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) /var/log/cassandra/system.log- at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) /var/log/cassandra/system.log- at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) /var/log/cassandra/system.log- at java.lang.Thread.run(Thread.java:748)
Re: multiple clients making schema changes at once
Sometimes even waiting hours does not change. I have a cluster where I did like you, synchronization of create tables statement, then even I tried waiting for schema agreement, in loop until success, but sometimes the success never happens, i got that error in loop in the logs of a node, it seems we must restart nodes really often :( Sébastien ERROR [InternalResponseStage:1117] 2021-06-03 17:32:34,937 MigrationCoordinator.java:408 - Unable to merge schema from /135.181.222.100 org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found a991bb50-c475-11eb-83cb-df35fc5a9bea; expected 994bee02-c475-11eb-beff-6d70d473832f) at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:984) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:938) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.config.Schema.updateTable(Schema.java:687) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1478) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1434) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1403) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1380) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.MigrationCoordinator.mergeSchemaFrom(MigrationCoordinator.java:367) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:404) [apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:393) [apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) [apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) [apache-cassandra-3.11.10.jar:3.11.10] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_292] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.10.jar:3.11.10] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292] Le jeu. 3 juin 2021 à 17:23, Joe Obernberger a écrit : > How does this work? I have a program that runs a series of alter table > statements, and then does inserts. In some cases, the insert happens > immediately after the alter table statement and the insert fails because > the schema (apparently) has not had time to propagate. I get an Undefined > column name error. > > The alter statements run single threaded, but the inserts run in multiple > threads. The alter statement is run in a synchronized block (Java). > Should I put an artificial delay after the alter statement? > > -Joe > On 6/1/2021 2:59 PM, Max C. wrote: > > We use ZooKeeper + kazoo’s lock implementation. Kazoo is a Python client > library for ZooKeeper. > > - Max > > Yes this is quite annoying. How did you implement that "external lock"? I > also thought of doing an external service that would be dedicated to that. > Cassandra client apps would send create instruction to that service, that > would receive them and do the creates 1 by 1, and the client app would wait > the response from it before starting to insert. > > Best, > > Sébastien. > > Le mar. 1 juin 2021 à 05:21, Max C. a écrit : > >> In our case we have a shared dev cluster with (for example) a key space >> for each developer, a key space for each CI runner, etc. As part of >> initializing our test suite we setup the schema to match the code that is >> about to be tested. This can mean multiple CI runners each adding/dropping >> tables at the same time but for different key spaces. >> >> Our experience is even though the schema changes do not conflict, we >> still run into schema mismatch problems. Our solution to this was to have >> a lock (external to Cassandra) that ensures only a single schema change >> operation is being issued at a time. >> >> People assume schema changes in Cassandra work the same way as MySQL or >> multiple users editing files on disk — i.e. as long as you’re not editing >> the same file (or same MySQL table), then there’s no problem. *This is >> NOT the case.* Cassandra schema changes are more like “git push”ing a >> commit to the same branch — i.e. at most one change can be outstanding at a >> time (across all tables, all key spaces)…otherw
Re: multiple clients making schema changes at once
Hello, Yes this is quite annoying. How did you implement that "external lock"? I also thought of doing an external service that would be dedicated to that. Cassandra client apps would send create instruction to that service, that would receive them and do the creates 1 by 1, and the client app would wait the response from it before starting to insert. Best, Sébastien. Le mar. 1 juin 2021 à 05:21, Max C. a écrit : > In our case we have a shared dev cluster with (for example) a key space > for each developer, a key space for each CI runner, etc. As part of > initializing our test suite we setup the schema to match the code that is > about to be tested. This can mean multiple CI runners each adding/dropping > tables at the same time but for different key spaces. > > Our experience is even though the schema changes do not conflict, we still > run into schema mismatch problems. Our solution to this was to have a > lock (external to Cassandra) that ensures only a single schema change > operation is being issued at a time. > > People assume schema changes in Cassandra work the same way as MySQL or > multiple users editing files on disk — i.e. as long as you’re not editing > the same file (or same MySQL table), then there’s no problem. *This is > NOT the case.* Cassandra schema changes are more like “git push”ing a > commit to the same branch — i.e. at most one change can be outstanding at a > time (across all tables, all key spaces)…otherwise you will run into > trouble. > > Hope that helps. Best of luck. > > - Max > > Hello, >> >> I have a more general question about that, I cannot find clear answer. >> >> In my use case I have many tables (around 10k new tables created per >> months) and they are created from many clients and only dynamically, with >> several clients creating same tables simulteanously. >> >> What is the recommended way of creating tables dynamically? If I am doing >> "if not exists" queries + wait for schema aggreement before and after each >> create statement, will it work correctly for Cassandra? >> >> Sébastien. >> > >
Re: unable to repair
I was not aware of such limitations. Thank you for your answer. Sébastien Le dim. 30 mai 2021 à 18:52, Bowen Song a écrit : > This sounds like a really bad idea. > > In Cassandra 4.0 RC1, when you have more than 150 tables or 40 keyspaces (code > reference > <https://github.com/apache/cassandra/blob/3282f5ecf187ecbb56b8d73ab9a9110c010898b0/src/java/org/apache/cassandra/config/Config.java#L562>), > Cassandra will warn you about it: > > *Cluster already contains %d tables in %d keyspaces. Having a large number > of tables will significantly slow down schema dependent cluster operations.* > > The warning exists for a good reason. You really need to reconsider you > data model. > > > On 30/05/2021 10:12, Sébastien Rebecchi wrote: > > Hello, > > I have a more general question about that, I cannot find clear answer. > > In my use case I have many tables (around 10k new tables created per > months) and they are created from many clients and only dynamically, with > several clients creating same tables simulteanously. > > What is the recommended way of creating tables dynamically? If I am doing > "if not exists" queries + wait for schema aggreement before and after each > create statement, will it work correctly for Cassandra? > > Sébastien. > > Le ven. 28 mai 2021 à 20:45, Sébastien Rebecchi > a écrit : > >> Thank you for your answer. >> >> If I send all my create operations still from many clients but to 1 >> coordinator node, always the same, would it prevent schema mismatch? >> >> Sébastien. >> >> >> Le ven. 28 mai 2021 à 01:14, Kane Wilson a >> écrit : >> >>> Which client operations could trigger schema change at node level? Do >>>> you mean that for ex creating a new table trigger a schema change globally, >>>> not only at KS/table single level? >>>> >>> Yes, any DDL statement (creating tables, altering, dropping, etc) >>> triggers a schema change across the cluster (globally). All nodes need to >>> be told of this schema change. >>> >>> >>>> I don't have schema changes, except keyspaces and tables creations. But >>>> they are done from multiple sources indeed. With a "create if not exists" >>>> statement, on demand. Thanks you for your answer, I will try to see if I >>>> could precreate them then. >>>> >>> Yep, definitely do that. You don't want to be issuing simultaneous >>> create statements from different clients. IF NOT EXISTS won't necessarily >>> catch all cases. >>> >>> >>>> As for the schema mismatch, what is the best way of fixing that issue? >>>> Could Cassandra recover from that on its own or is there a nodetool command >>>> to force schema agreement? I have heard that we have to restart the nodes 1 >>>> by 1, but it seems a very heavy procedure for that. >>>> >>> A rolling restart is usually enough to fix the issue. You might want to >>> repair afterwards, and check that data didn't make it to different versions >>> of the table on different nodes (in which case some more intervention may >>> be necessary to save that data). >>> >>> -- >>> raft.so - Cassandra consulting, support, and managed services >>> >>
Re: unable to repair
Hello, I have a more general question about that, I cannot find clear answer. In my use case I have many tables (around 10k new tables created per months) and they are created from many clients and only dynamically, with several clients creating same tables simulteanously. What is the recommended way of creating tables dynamically? If I am doing "if not exists" queries + wait for schema aggreement before and after each create statement, will it work correctly for Cassandra? Sébastien. Le ven. 28 mai 2021 à 20:45, Sébastien Rebecchi a écrit : > Thank you for your answer. > > If I send all my create operations still from many clients but to 1 > coordinator node, always the same, would it prevent schema mismatch? > > Sébastien. > > > Le ven. 28 mai 2021 à 01:14, Kane Wilson a écrit : > >> Which client operations could trigger schema change at node level? Do you >>> mean that for ex creating a new table trigger a schema change globally, not >>> only at KS/table single level? >>> >> Yes, any DDL statement (creating tables, altering, dropping, etc) >> triggers a schema change across the cluster (globally). All nodes need to >> be told of this schema change. >> >> >>> I don't have schema changes, except keyspaces and tables creations. But >>> they are done from multiple sources indeed. With a "create if not exists" >>> statement, on demand. Thanks you for your answer, I will try to see if I >>> could precreate them then. >>> >> Yep, definitely do that. You don't want to be issuing simultaneous >> create statements from different clients. IF NOT EXISTS won't necessarily >> catch all cases. >> >> >>> As for the schema mismatch, what is the best way of fixing that issue? >>> Could Cassandra recover from that on its own or is there a nodetool command >>> to force schema agreement? I have heard that we have to restart the nodes 1 >>> by 1, but it seems a very heavy procedure for that. >>> >> A rolling restart is usually enough to fix the issue. You might want to >> repair afterwards, and check that data didn't make it to different versions >> of the table on different nodes (in which case some more intervention may >> be necessary to save that data). >> >> -- >> raft.so - Cassandra consulting, support, and managed services >> >
Re: unable to repair
Thank you for your answer. If I send all my create operations still from many clients but to 1 coordinator node, always the same, would it prevent schema mismatch? Sébastien. Le ven. 28 mai 2021 à 01:14, Kane Wilson a écrit : > Which client operations could trigger schema change at node level? Do you >> mean that for ex creating a new table trigger a schema change globally, not >> only at KS/table single level? >> > Yes, any DDL statement (creating tables, altering, dropping, etc) triggers > a schema change across the cluster (globally). All nodes need to be told of > this schema change. > > >> I don't have schema changes, except keyspaces and tables creations. But >> they are done from multiple sources indeed. With a "create if not exists" >> statement, on demand. Thanks you for your answer, I will try to see if I >> could precreate them then. >> > Yep, definitely do that. You don't want to be issuing simultaneous create > statements from different clients. IF NOT EXISTS won't necessarily catch > all cases. > > >> As for the schema mismatch, what is the best way of fixing that issue? >> Could Cassandra recover from that on its own or is there a nodetool command >> to force schema agreement? I have heard that we have to restart the nodes 1 >> by 1, but it seems a very heavy procedure for that. >> > A rolling restart is usually enough to fix the issue. You might want to > repair afterwards, and check that data didn't make it to different versions > of the table on different nodes (in which case some more intervention may > be necessary to save that data). > > -- > raft.so - Cassandra consulting, support, and managed services >
Re: unable to repair
OK I will check that, thank you! Sébastien Le jeu. 27 mai 2021 à 11:07, Bowen Song a écrit : > Hi Sébastien, > > > The error message you shared came from the repair coordinator node's > log, and it's the result of failures reported by 3 other nodes. If you > could have a look at the 3 nodes listed in the error message - > 135.181.222.100, 135.181.217.109 and 135.181.221.180, you should be able > to find that between the starting time of the repair and 2021-05-26 > 15:54:19 (the timestamp on the error message), each of the 3 nodes will > have at least one error message in their log indicating the exact reason > why they failed. There's many reasons a repair can fail, and schema > disagreement is only one of them. By looking at the logs on the other > nodes, you should be able to find out the root cause, and then come up > with a solution. > > > Regards, > > Bowen > > On 26/05/2021 15:05, Sébastien Rebecchi wrote: > > Hi, > > > > I have an issue with repairing my Casandra cluster, that was already > > the case with Cassandra 3 and the issue is not solved with Cassandra 4 > > RC1. > > > > I run in a for loop, one 1 by 1, the following command: > > > > nodetool -h THE_NODE -u jTHE_USER -pw THE_PASSWORD repair --full -pr > > > > and I always get the following error, see message and stack trace for > > Cassandra 4 RC1 at the bottom of the message (the same for C3). > > > > I don't know what to do with that. Are there some mistakes I could > > have made in my table design explaining that? I have heard for example > > that it was not recommended to have big partitions, so I changed my > > data model to remove clustering keys I add before and then split big > > partitions in many independent one, and now the partitions are max > > 500KB each (the vast majority of them are max 100KB). But it did not > > change anything. Also my partition key was a compound of 9 columns and > > I changed that to have only 1 column for partition key by generating > > ids by myself, the same, no improvement. > > > > Thank you for your help, > > > > Sébastien > > > > -- > > > > error: Repair job has failed with the error message: [2021-05-26 > > 15:54:19,981] Repair command #2 failed with error Got negative replies > > from endpoints [135.181.222.100, 135.181.217.109, 135.181.221.180] > > -- StackTrace -- > > java.lang.RuntimeException: Repair job has failed with the error > > message: [2021-05-26 15:54:19,981] Repair command #2 failed with error > > Got negative replies from endpoints [135.181.222.100, 135.181.217.109, > > 135.181.221.180] > > at > org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116) > > at > > > org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77) > > at > > > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583) > > at > > > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533) > > at > > > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452) > > at > > > com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108) >
Re: unable to repair
Sorry Kane, I am a little bit confused, we are talking about schema version at node level. Which client operations could trigger schema change at node level? Do you mean that for ex creating a new table trigger a schema change globally, not only at KS/table single level? Sébastien Le jeu. 27 mai 2021 à 08:33, Sébastien Rebecchi a écrit : > I don't have schema changes, except keyspaces and tables creations. But > they are done from multiple sources indeed. With a "create if not exists" > statement, on demand. Thanks you for your answer, I will try to see if I > could precreate them then. > > As for the schema mismatch, what is the best way of fixing that issue? > Could Cassandra recover from that on its own or is there a nodetool command > to force schema agreement? I have heard that we have to restart the nodes 1 > by 1, but it seems a very heavy procedure for that. > > Regards, > > Sébastien > > Le jeu. 27 mai 2021 à 02:45, Kane Wilson a écrit : > >> I have had that error sometimes when schema mismatch but also when all >>> schema match. So I think this is not the only cause. >>> >> Have you checked the logs for errors on 135.181.222.100, 135.181.217.109, >> and 135.181.221.180? They may give you some better information about why >> they are sending bad replies. >> >> By the way, what could cause such a shema mismatch. I would like to know >>> what should be or not be done in order to keep schema agreements between >>> nodes? Are there some mistakes to avoid in table design to prevent such >>> issue? >>> >> Schema changes aren't strongly consistent across the cluster, so they can >> occur through various problems. The main one would be multiple clients >> making schema changes simultaneously and separate nodes ending up with >> conflicting schema definitions. Best practice to avoid that is to only make >> schema changes from a single client. >> >> >> -- >> raft.so - Cassandra consulting, support, and managed services >> >
Re: unable to repair
I don't have schema changes, except keyspaces and tables creations. But they are done from multiple sources indeed. With a "create if not exists" statement, on demand. Thanks you for your answer, I will try to see if I could precreate them then. As for the schema mismatch, what is the best way of fixing that issue? Could Cassandra recover from that on its own or is there a nodetool command to force schema agreement? I have heard that we have to restart the nodes 1 by 1, but it seems a very heavy procedure for that. Regards, Sébastien Le jeu. 27 mai 2021 à 02:45, Kane Wilson a écrit : > I have had that error sometimes when schema mismatch but also when all >> schema match. So I think this is not the only cause. >> > Have you checked the logs for errors on 135.181.222.100, 135.181.217.109, > and 135.181.221.180? They may give you some better information about why > they are sending bad replies. > > By the way, what could cause such a shema mismatch. I would like to know >> what should be or not be done in order to keep schema agreements between >> nodes? Are there some mistakes to avoid in table design to prevent such >> issue? >> > Schema changes aren't strongly consistent across the cluster, so they can > occur through various problems. The main one would be multiple clients > making schema changes simultaneously and separate nodes ending up with > conflicting schema definitions. Best practice to avoid that is to only make > schema changes from a single client. > > > -- > raft.so - Cassandra consulting, support, and managed services >
Re: unable to repair
Thank you for your answer. I have had that error sometimes when schema mismatch but also when all schema match. So I think this is not the only cause. By the way, what could cause such a shema mismatch. I would like to know what should be or not be done in order to keep schema agreements between nodes? Are there some mistakes to avoid in table design to prevent such issue? Thank you in advance, Sébastien Le mer. 26 mai 2021 à 18:09, Dipan Shah a écrit : > Hello Sebastien, > > Not sure but have you checked the output of "nodetool describecluster"? A > schema mismatch or node unavailability might result in this. > > Thanks, > > Dipan Shah > > ------ > *From:* Sébastien Rebecchi > *Sent:* Wednesday, May 26, 2021 7:35 PM > *To:* user@cassandra.apache.org > *Subject:* unable to repair > > Hi, > > I have an issue with repairing my Casandra cluster, that was already the > case with Cassandra 3 and the issue is not solved with Cassandra 4 RC1. > > I run in a for loop, one 1 by 1, the following command: > > nodetool -h THE_NODE -u jTHE_USER -pw THE_PASSWORD repair --full -pr > > and I always get the following error, see message and stack trace for > Cassandra 4 RC1 at the bottom of the message (the same for C3). > > I don't know what to do with that. Are there some mistakes I could have > made in my table design explaining that? I have heard for example that it > was not recommended to have big partitions, so I changed my data model to > remove clustering keys I add before and then split big partitions in many > independent one, and now the partitions are max 500KB each (the vast > majority of them are max 100KB). But it did not change anything. Also my > partition key was a compound of 9 columns and I changed that to have only 1 > column for partition key by generating ids by myself, the same, no > improvement. > > Thank you for your help, > > Sébastien > > -- > > error: Repair job has failed with the error message: [2021-05-26 > 15:54:19,981] Repair command #2 failed with error Got negative replies from > endpoints [135.181.222.100, 135.181.217.109, 135.181.221.180] > -- StackTrace -- > java.lang.RuntimeException: Repair job has failed with the error message: > [2021-05-26 15:54:19,981] Repair command #2 failed with error Got negative > replies from endpoints [135.181.222.100, 135.181.217.109, 135.181.221.180] > at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116) > at > org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77) > at > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583) > at > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533) > at > com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452) > at > com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108) >
unable to repair
Hi, I have an issue with repairing my Casandra cluster, that was already the case with Cassandra 3 and the issue is not solved with Cassandra 4 RC1. I run in a for loop, one 1 by 1, the following command: nodetool -h THE_NODE -u jTHE_USER -pw THE_PASSWORD repair --full -pr and I always get the following error, see message and stack trace for Cassandra 4 RC1 at the bottom of the message (the same for C3). I don't know what to do with that. Are there some mistakes I could have made in my table design explaining that? I have heard for example that it was not recommended to have big partitions, so I changed my data model to remove clustering keys I add before and then split big partitions in many independent one, and now the partitions are max 500KB each (the vast majority of them are max 100KB). But it did not change anything. Also my partition key was a compound of 9 columns and I changed that to have only 1 column for partition key by generating ids by myself, the same, no improvement. Thank you for your help, Sébastien -- error: Repair job has failed with the error message: [2021-05-26 15:54:19,981] Repair command #2 failed with error Got negative replies from endpoints [135.181.222.100, 135.181.217.109, 135.181.221.180] -- StackTrace -- java.lang.RuntimeException: Repair job has failed with the error message: [2021-05-26 15:54:19,981] Repair command #2 failed with error Got negative replies from endpoints [135.181.222.100, 135.181.217.109, 135.181.221.180] at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:116) at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533) at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452) at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
