BHARATH KUMAR created CASSANDRA-19662:
-----------------------------------------
Summary: Data Corruption and OOM Issues During Schema Alterations
Key: CASSANDRA-19662
URL: https://issues.apache.org/jira/browse/CASSANDRA-19662
Project: Cassandra
Issue Type: Bug
Reporter: BHARATH KUMAR
h2. Description
*Overview:* The primary issue is data corruption occurring during schema
alterations (ADD/DROP column) on large tables(300+ columns and 6TB size ) in
the production cluster. This is accompanied by out-of-memory (OOM) errors and
other exceptions, specifically during batch reads. This problem has been
replicated on multiple clusters, running Apache Cassandra version 4.0.12.
*Details:*
*Main Issue:*
*
** *Data Corruption:* When dynamically adding a column to a table, the data
intended for the new column is shifted, causing misalignment in the data.
*
** *Symptoms:* The object implementing
{{com.datastax.oss.driver.api.core.cql.Row}} returns values shifted against the
column names returned by {{{}row.getColumnDefinitions(){}}}. The driver returns
a corrupted row, leading to incorrect data insertion.
*Additional Issues:*
*
** *Exceptions:*
*
**
*** {{java.nio.BufferUnderflowException}} during batch reads when ALTER TABLE
ADD/DROP column statements are issued.
*
**
*** {{java.lang.ArrayIndexOutOfBoundsException}} in some cases.
*
**
*** Buffer underflow exceptions with messages like "Invalid 32-bits integer
value, expecting 4 bytes but got 292".
*
**
*** OOM errors mostly occur during ADD column operations, while other
exceptions occur during DELETE column operations.
*
** *Method Specific:* Errors occur specifically with
{{{}row.getList(columnName, Float.class){}}}, returning incorrect values.
*Reproducibility:*
*
** The issue is reproducible on larger tables (300 columns, 6 TB size) but not
on smaller tables.
*
** SELECT * statements are used during reads
*
** *Method Specific:* Errors occur specifically with
{{{}row.getList(columnName, Float.class){}}}, returning incorrect values.
However, the code registers a driver exception when calling the method
{{{}row.getList(columnName, Float.class){}}}. We pass the exact column name
obtained from {{{}row.getColumnDefinition{}}}, but it returns the wrong value
for a column with this name. This suggests that the issue lies with the driver
returning an object with incorrect properties, rather than with the SQL query
itself.
*Debugging Efforts:*
*
** *Metadata Refresh:* Enabling metadata refresh did not resolve the issue.
** *Schema Agreement:* {{session.getCqlSession().checkSchemaAgreement()}} did
not detect inconsistencies during test execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
