Amit Singh Chowdhery created CASSANDRA-9955:
-----------------------------------------------
Summary: In 3 node Cluster, when 1 node was forced down, data
failures are observed in other 2 nodes.
Key: CASSANDRA-9955
URL: https://issues.apache.org/jira/browse/CASSANDRA-9955
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Cassandra 2.0.14, Hector Client (1.0.1), Red Hat Linux OS,
Reporter: Amit Singh Chowdhery
Issue :
On 3 node cluster, inserts are happening normally but when 1 node was pulled
down, After few minutes application stops and then failure start coming on both
nodes. Hector keeps
Exception Logs hector:
ERROR m.p.c.c.ConcurrentHClientPool - Transport exception in re-opening client
in release on <ConcurrentCassandraClientPoolByHost>.
Cassandra Debug Logs :
DEBUG [OptionalTasks:1] 2015-07-31 11:57:37,698 ColumnFamilyStore.java (line
300) retryPolicy for local is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:38,969 ColumnFamilyStore.java (line
300) retryPolicy for encryptionKey is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:39,492 ColumnFamilyStore.java (line
300) retryPolicy for vouchers.c_per__batchIdIdx is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:39,504 ColumnFamilyStore.java (line
300) retryPolicy for vouchers.TX_STATEIdx is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:39,824 ColumnFamilyStore.java (line
300) retryPolicy for vouchers.c_per__serialNumberIdx is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:39,824 ColumnFamilyStore.java (line
300) retryPolicy for vouchers.c_per__subStateIdx is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:39,828 ColumnFamilyStore.java (line
300) retryPolicy for vouchers is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:40,011 ColumnFamilyStore.java (line
300) retryPolicy for voucherHistory is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:40,021 ColumnFamilyStore.java (line
300) retryPolicy for vshash is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:40,180 ColumnFamilyStore.java (line
300) retryPolicy for vouchersByPurgeDate is 0.99
DEBUG [OptionalTasks:1] 2015-07-31 11:57:40,395 ColumnFamilyStore.java (line
300) retryPolicy for serialNums is 0.99
DEBUG [Thrift:7] 2015-07-31 11:57:40,452 CassandraServer.java (line 311)
get_slice
DEBUG [Thrift:35] 2015-07-31 11:57:40,452 CassandraServer.java (line 943)
batch_mutate
DEBUG [MutationStage:56] 2015-07-31 11:57:40,453 StorageProxy.java (line 928)
Adding hint for /192.168.5.65
DEBUG [Thrift:35] 2015-07-31 11:57:40,453 Tracing.java (line 159) request
complete
DEBUG [Thrift:7] 2015-07-31 11:57:40,453 RowDigestResolver.java (line 62)
resolving 2 responses
DEBUG [Thrift:7] 2015-07-31 11:57:40,453 RowDigestResolver.java (line 94)
resolve: 0 ms.
DEBUG [Thrift:7] 2015-07-31 11:57:40,454 StorageProxy.java (line 1275) Read: 1
ms.
Steps to reproduce
Step 1 : In 3 node cluster, on any 2 node start inserting records.
Step 2 : Take down the node on which data insertion was not happening (init 0).
Step 3: Failures can be seen on other 2 nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
