Hello everyone, The recovery was performed successfully some days ago. Finally, the problematic datacenter was removed and added back to the cluster.
BR MK From: Michalis Kotsiouros (EXT) via user <user@cassandra.apache.org> Sent: February 12, 2024 17:59 To: Sebastian Marsching <sebast...@marsching.com>; user@cassandra.apache.org Cc: Michalis Kotsiouros (EXT) <michalis.kotsiouros....@ericsson.com> Subject: RE: SStables stored in directory with different table ID than the one found in system_schema.tables Hello Sebastian and community, Thanks a lot for the post. It is really helpful. After some additional observations, I am more concerned about trying to rename/move the sstables directory. I have observed that my client processes complain about missing columns even though those columns appear on the describe schema output. My plan is to first try a restart of the Cassandra nodes and if that does not help to re-build the datacenter – remove it and then add it back to the cluster. BR MK From: Sebastian Marsching <sebast...@marsching.com <mailto:sebast...@marsching.com> > Sent: February 10, 2024 01:00 To: Bowen Song via user <user@cassandra.apache.org <mailto:user@cassandra.apache.org> > Cc: Michalis Kotsiouros (EXT) <michalis.kotsiouros....@ericsson.com <mailto:michalis.kotsiouros....@ericsson.com> > Subject: Re: SStables stored in directory with different table ID than the one found in system_schema.tables You might the following discussion from the mailing-list archive helpful: https://lists.apache.org/thread/6hnypp6vfxj1yc35ptp0xf15f11cx77d This thread discusses a similar situation gives a few pointers on when it might be save to simply move the SSTables around. Am 08.02.2024 um 13:06 schrieb Michalis Kotsiouros (EXT) via user <user@cassandra.apache.org <mailto:user@cassandra.apache.org> >: Hello everyone, I have found this post on-line and seems to be recent. <https://stackoverflow.com/questions/77837100/mismatch-between-cassandra-table-uuid-in-linux-file-directory-and-system-schema> Mismatch between Cassandra table uuid in linux file directory and system_schema.tables - Stack Overflow The description seems to be the same as my problem as well. In this post, the proposal is to copy the sstables to the dir with the ID found in system_schema.tables. I think it is equivalent with my assumption to rename the directories…. Have anyone seen this before? Do you consider those approaches safe? BR MK From: Michalis Kotsiouros (EXT) Sent: February 08, 2024 11:33 To: user@cassandra.apache.org <mailto:user@cassandra.apache.org> Subject: SStables stored in directory with different table ID than the one found in system_schema.tables Hello community, I have a Cassandra server on 3.11.13 on SLESS 12.5. I have noticed in the logs the following line: Datacenter A org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId d8c1bea0-82ed-11ee-8ac8-1513e17b60b1. If a table was just created, this is likely due to the schema not being fully propagated. Please wait for schema agreement on table creation. Datacenter B org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId 0fedabd0-11f7-11ea-9450-e3ff59b2496b. If a table was just created, this is likely due to the schema not being fully propagated. Please wait for schema agreement on table creation. This error results in failure of all streaming tasks. I have checked the sstables directories and I see that: In Datacenter A the sstables directory is: <table-name>-0fedabd0-11f7-11ea-9450-e3ff59b2496b In Datacenter B the sstables directory are: <table-name>-0fedabd011f711ea9450e3ff59b2496b <table-name>- d8c1bea082ed11ee8ac81513e17b60b1 In this datacenter although the <table-name>- d8c1bea082ed11ee8ac81513e17b60b1 dir is more recent it is empty and all sstables are stored under <table-name>-0fedabd011f711ea9450e3ff59b2496b I have also checked the system_schema.tables in all Cassandra nodes and I see that for the specific table the ID is consistent across all nodes and it is: d8c1bea0-82ed-11ee-8ac8-1513e17b60b1 So it seems that the schema is a bit mess in all my datacenters. I am not really interested to understand how it ended up in this status but more on how to recover. Both datacenters seem to have this inconsistency between the id stored system_schema.tables and the one used in the sstables directory. Do you have any proposal on how to recover? I have thought of renaming the dir from <table-name>-0fedabd011f711ea9450e3ff59b2496b to <table-name>- d8c1bea082ed11ee8ac81513e17b60b1 but it does not look safe and I would not want to risk my data since this is a production system. Thank you in advance. BR Michail Kotsiouros
smime.p7s
Description: S/MIME cryptographic signature
