Cameron Zemek created CASSANDRA-18595:
-----------------------------------------
Summary: Dropped table causes node to get stuck in JOINING
Key: CASSANDRA-18595
URL: https://issues.apache.org/jira/browse/CASSANDRA-18595
Project: Cassandra
Issue Type: Improvement
Reporter: Cameron Zemek
Attachments: cass_stream.patch
To facilitate reproducibility of this issue I have created a patch.
[^cass_stream.patch]
It can also be reproduced without patch by dropping table while node is doing
stream for join.
{code:java}
cd ~/dev/cassandra
git checkout 3.11.14
git apply ~/downloads/cass_stream.patch
ant clean
ant
ccm create -n 3 test
ccm node1 start
ccm node2 start
ccm node1 cqlsh -e "CREATE KEYSPACE test WITH replication = {'class':
'NetworkTopologyStrategy', 'datacenter1': '3'};"
# receive warning since nodes < RF
ccm node1 cqlsh -e "CREATE TABLE test.streamtest(stream_id int PRIMARY KEY,
payload TEXT);"
ccm node1 cqlsh -e "INSERT INTO test.streamtest(stream_id, payload) VALUES(1,
'hello world')"
ccm flush
sed -i 's/auto_bootstrap: false/auto_bootstrap: true/'
~/.ccm/test/node3/node.conf
sed -i 's/auto_bootstrap: false/auto_bootstrap: true/'
~/.ccm/test/node3/conf/cassandra.yaml
sed -i 's/seeds: 127.0.0.1,127.0.0.2,127.0.0.3/seeds: 127.0.0.1,127.0.0.2/'
~/.ccm/test/node3/conf/cassandra.yaml
touch ~/.ccm/test/node3/logs/system.log
ccm node3 start
tail -f ~/.ccm/test/node3/logs/system.log{code}
The logs will display something like:
{noformat}
ERROR [STREAM-IN-/127.0.0.1:7000] 2023-06-14 11:08:37,399
StreamSession.java:609 - [Stream #fdd71710-0a4f-11ee-a768-c3d20ff8c2d1]
Streaming error occurred on session with peer 127.0.0.1
java.io.IOException: streamtest 38f79d70-0a4f-11ee-9849-a999e71c8739 CF was
dropped during compressed streaming
ERROR [main] 2023-06-14 11:08:37,405 StorageService.java:1630 - Error while
waiting on bootstrap to complete. Bootstrap will have to be restarted.{noformat}
The problem is in this use case the customer is rotating out tables every few
hours and so resuming the bootstrap keeps failing. In my opinion this is a bug
and Cassandra should be robust enough to continue streaming if a table is
dropped.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
