Hi folks, We have a cluster with Cassandra 3.9, running on AWS in a single region and multi AZ setup. We use Priam do our backup to S3 and auto restore on node bootstrap. We created an index on a simple uuid field on one of our existing tables, after which the restores fail on that table (see log messages below) whenever a node is cycled.
This causes the Cassandra service to exit hard and the node fails to bootstrap. The restore used to worked fine before this and gives no errors on the other tables. Has anybody seen this issue before? Is this a bug with snapshotting? To get the node up and running we removed the table files and executed a full repair after. Our RF is 3. Do we also have to do a nodetool rebuild_index? Thanks, Melvin Jul 22 00:07:58 ip-123-12-123-123 su[22260]: INFO 00:07:58 Initializing path.redacted_table_name Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-181-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-204-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-185-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-177-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-101-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-192-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-212-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-216-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-208-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:58 ip-123-12-123-123 su[22260]: ERROR 00:07:58 Cannot open /path/path/redacted_table_name-88308780560011e7a7e7dba00a2b2866/mc-188-big; partitioner org.apache.cassandra.dht.LocalPartitioner does not match system partitioner org.apache.cassandra.dht.Murmur3Partitioner. Note that the default partitioner starting with Cassandra 1.2 is Murmur3Partitioner, so you will need to edit that to match your old partitioner if upgrading. Jul 22 00:07:59 ip-123-12-123-123 systemd[1]: cassandra.service: Main process exited, code=exited, status=1/FAILURE Jul 22 00:07:59 ip-123-12-123-123 systemd[1]: cassandra.service: Unit entered failed state. Jul 22 00:07:59 ip-123-12-123-123 systemd[1]: cassandra.service: Failed with result 'exit-code'. Jul 22 00:07:59 ip-123-12-123-123 systemd[1]: Stopping User Manager for UID 997...
