Alaykumar Barochia created CASSANDRA-19751:
----------------------------------------------
Summary: IllegalStateException when query on table having static
columns during the Cassandra cluster upgrade from 3.11.4 to 4.0.11
Key: CASSANDRA-19751
URL: https://issues.apache.org/jira/browse/CASSANDRA-19751
Project: Cassandra
Issue Type: Bug
Reporter: Alaykumar Barochia
We are upgrading Cassandra cluster from 3.11.4 to 4.0.11.
While performing upgrade on 1st DC, we observed below WARN/ERROR messages on C*
3 and C* 4 nodes.
*C*3 nodes:*
{noformat}
WARN [ReadStage-1] 2024-06-11 08:04:09,088
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-1,5,main]: {}
java.lang.IllegalStateException: [last_metadata_updt_ts, price_metadata] is not
a subset of [price_metadata]
WARN [ReadStage-1] 2024-06-19 05:10:31,226
AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
Thread[ReadStage-1,5,main]: {}
java.lang.IllegalStateException: [default_price_json, last_metadata_updt_ts,
price_metadata] is not a subset of [price_metadata]
{noformat}
*C*4 nodes:*
{noformat}
ERROR [ReadStage-1] 2024-06-19 05:48:47,388
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[ReadStage-1,5,main]
java.lang.IllegalStateException: [last_metadata_updt_ts, price_metadata] is not
a subset of [price_metadata]
{noformat}
Table definition for which above columns are associated is as below:
{noformat}
CREATE TABLE omni_price_ks_v2.location_price_mstr (
tcin text,
location_id bigint,
price_change_id text,
default_price_json text static,
end_ts bigint,
last_metadata_updt_ts bigint static,
last_update_ts bigint,
price_json text,
price_metadata text static,
price_type text,
start_ts bigint,
status text,
version text,
PRIMARY KEY (tcin, location_id, price_change_id)
) WITH CLUSTERING ORDER BY (location_id ASC, price_change_id ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = {'keys': 'ALL', 'rows_per_partition': '100'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
{noformat}
App team also observed below error in their application logs when try to read
from this table.
{noformat}
{ "code": "ERR_GETPRICE_0034", "message": "Cassandra failure during read query
at consistency LOCAL_QUORUM (2 responses were required but only 1 replica
responded, 1 failed)" }
{noformat}
Because of this error, the application is getting impacted during the upgrade.
Once the upgrade on all DCs is completed, this error stops.
I found below bug which matches our case.
https://issues.apache.org/jira/browse/CASSANDRA-17601
It seems like we are hitting some bug and hence raising this Jira.
Can you please have a look if this is still a bug and what would be the fix?
Let me know if you need any more details.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
