[
https://issues.apache.org/jira/browse/CASSANDRA-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019084#comment-16019084
]
Andrei Zbikowski commented on CASSANDRA-13004:
----------------------------------------------
Hey all, I work with Stan and Jake at Discord and have a few updates for this
thread that may help drive us towards a solution.
First up, we've been able to produce a minimal reproduction with two sanitized
rows of our production data set. So far this appears to be a 100% consistent
reproduction, each time producing corrupted/unreadable data. Through the
process of building this reproduction we've noticed the following attributes to
this bug:
- From what we can tell, this is not reproducible with a single node. We've
tested both 1 node, 3 node and 6 node clusters, with only the 3/6 combinations
producing corruption.
- The bug does not seem to be related to client driver, we've tested both
Python and gocql
- Appears to exhibit itself on all versions (we've mostly been testing on
Cassandra 3.10)
I've uploaded the reproduction case to github over at
(https://github.com/b1naryth1ef/cassandra-13004). A few notes about this:
- Some stuff might have to be tweaked (e.g. the cluster ip, keyspace
replication datacenter) depending on your test env.
- Although we reproduced this in Python too, I've only included the Go version
(mostly because the Python one was more heavily intertwined with our internal
code). If it helps, I'm mor ethan happy to cleanup and provide a Python version
too.
To run the reproduction, you must spin up a Cassandra 3.10 cluster, and then
either run the script or the commands within 'test.sh'. This creates the
keyspace/types/table, and loads two rows in (there is a bug w/ recent versions
of cqlsh that break loading of multiple large rows, so the script loads one
road at a time). Finally, you can run the Go binary which will connect to the
cluster and start reading/writing data. At this point you can run alters (I've
been using `ALTER TABLE test13004.guilds ADD aaa bigint;` which seems to work
every time) and observe the output using cqlsh `SELECT * FROM test13004.guilds`.
Happy to provide any more details or information as it's needed.
> Corruption while adding a column to a table
> -------------------------------------------
>
> Key: CASSANDRA-13004
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13004
> Project: Cassandra
> Issue Type: Bug
> Reporter: Stanislav Vishnevskiy
>
> We had the following schema in production.
> {code:none}
> CREATE TYPE IF NOT EXISTS discord_channels.channel_recipient (
> nick text
> );
> CREATE TYPE IF NOT EXISTS discord_channels.channel_permission_overwrite (
> id bigint,
> type int,
> allow_ int,
> deny int
> );
> CREATE TABLE IF NOT EXISTS discord_channels.channels (
> id bigint,
> guild_id bigint,
> type tinyint,
> name text,
> topic text,
> position int,
> owner_id bigint,
> icon_hash text,
> recipients map<bigint, frozen<channel_recipient>>,
> permission_overwrites map<bigint, frozen<channel_permission_overwrite>>,
> bitrate int,
> user_limit int,
> last_pin_timestamp timestamp,
> last_message_id bigint,
> PRIMARY KEY (id)
> );
> {code}
> And then we executed the following alter.
> {code:none}
> ALTER TABLE discord_channels.channels ADD application_id bigint;
> {code}
> And one row (that we can tell) got corrupted at the same time and could no
> longer be read from the Python driver.
> {code:none}
> [E 161206 01:56:58 geventreactor:141] Error decoding response from Cassandra.
> ver(4); flags(0000); stream(27); op(8); offset(9); len(887); buffer:
> '\x84\x00\x00\x1b\x08\x00\x00\x03w\x00\x00\x00\x02\x00\x00\x00\x01\x00\x00\x00\x0f\x00\x10discord_channels\x00\x08channels\x00\x02id\x00\x02\x00\x0eapplication_id\x00\x02\x00\x07bitrate\x00\t\x00\x08guild_id\x00\x02\x00\ticon_hash\x00\r\x00\x0flast_message_id\x00\x02\x00\x12last_pin_timestamp\x00\x0b\x00\x04name\x00\r\x00\x08owner_id\x00\x02\x00\x15permission_overwrites\x00!\x00\x02\x000\x00\x10discord_channels\x00\x1cchannel_permission_overwrite\x00\x04\x00\x02id\x00\x02\x00\x04type\x00\t\x00\x06allow_\x00\t\x00\x04deny\x00\t\x00\x08position\x00\t\x00\nrecipients\x00!\x00\x02\x000\x00\x10discord_channels\x00\x11channel_recipient\x00\x01\x00\x04nick\x00\r\x00\x05topic\x00\r\x00\x04type\x00\x14\x00\nuser_limit\x00\t\x00\x00\x00\x01\x00\x00\x00\x08\x03\x8a\x19\x8e\xf8\x82\x00\x01\xff\xff\xff\xff\x00\x00\x00\x04\x00\x00\xfa\x00\x00\x00\x00\x08\x00\x00\xfa\x00\x00\xf8G\xc5\x00\x00\x00\x00\x00\x00\x00\x08\x03\x8b\xc0\xb5nB\x00\x02\x00\x00\x00\x08G\xc5\xffI\x98\xc4\xb4(\x00\x00\x00\x03\x8b\xc0\xa8\xff\xff\xff\xff\x00\x00\x01<\x00\x00\x00\x06\x00\x00\x00\x08\x03\x81L\xea\xfc\x82\x00\n\x00\x00\x00$\x00\x00\x00\x08\x03\x81L\xea\xfc\x82\x00\n\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x04\x00\x00\x08\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x08\x03\x8a\x1e\xe6\x8b\x80\x00\n\x00\x00\x00$\x00\x00\x00\x08\x03\x8a\x1e\xe6\x8b\x80\x00\n\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x040\x07\xf8Q\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x08\x03\x8a\x1f\x1b{\x82\x00\x00\x00\x00\x00$\x00\x00\x00\x08\x03\x8a\x1f\x1b{\x82\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00\x07\xf8Q\x00\x00\x00\x04\x10\x00\x00\x00\x00\x00\x00\x08\x03\x8a\x1fH6\x82\x00\x01\x00\x00\x00$\x00\x00\x00\x08\x03\x8a\x1fH6\x82\x00\x01\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00\x05\xe8A\x00\x00\x00\x04\x10\x02\x00\x00\x00\x00\x00\x08\x03\x8a+=\xca\xc0\x00\n\x00\x00\x00$\x00\x00\x00\x08\x03\x8a+=\xca\xc0\x00\n\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x08\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x08\x03\x8a\x8f\x979\x80\x00\n\x00\x00\x00$\x00\x00\x00\x08\x03\x8a\x8f\x979\x80\x00\n\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x04\x00
>
> \x08\x01\x00\x00\x00\x04\xc4\xb4(\x00\xff\xff\xff\xff\x00\x00\x00O[f\x80Q\x07general\x05\xf8G\xc5\xffI\x98\xc4\xb4(\x00\xf8O[f\x80Q\x00\x00\x00\x02\x04\xf8O[f\x80Q\x00\xf8G\xc5\xffI\x98\x01\x00\x00\xf8O[f\x80Q\x00\x00\x00\x00\xf8G\xc5\xffI\x97\xc4\xb4(\x06\x00\xf8O\x7fe\x1fm\x08\x03\x00\x00\x00\x01\x00\x00\x00\x00\x04\x00\x00\x00\x00'
> {code}
> And then in cqlsh when trying to read the row we got this.
> {code:none}
> /usr/bin/cqlsh.py:632: DateOverFlowWarning: Some timestamps are larger than
> Python datetime can represent. Timestamps are displayed in milliseconds from
> epoch.
> Traceback (most recent call last):
> File "/usr/bin/cqlsh.py", line 1301, in perform_simple_statement
> result = future.result()
> File
> "/usr/share/cassandra/lib/cassandra-driver-internal-only-3.5.0.post0-d8d0456.zip/cassandra-driver-3.5.0.post0-d8d0456/cassandra/cluster.py",
> line 3650, in result
> raise self._final_exception
> UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 2:
> invalid start byte
> {code}
> We tried to read the data and it would refuse to read the name column (the
> UTF8 error) and the last_pin_timestamp column had an absurdly large value.
> We ended up rewriting the whole row as we had the data in another place and
> it fixed the problem. However there is clearly a race condition in the schema
> change sub-system.
> Any ideas?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
