[
https://issues.apache.org/jira/browse/CASSANDRA-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631347#comment-13631347
]
Ryan McGuire commented on CASSANDRA-5461:
-----------------------------------------
I haven't been able to reproduce this with just stress and cassandra, on 1.1.9,
nor 1.2. Here's what I've tried:
* Create a 4 node 1.1.9 cluster in a single datacenter:
{code}
$ ccm create -v 1.1.9 5641-1.1.9
$ ccm populate -n 4
$ ccm start
$ ccm node1 ring
Note: Ownership information does not include topology, please specify a
keyspace.
Address DC Rack Status State Load Owns
Token
127605887595351923798765477786913079296
127.0.0.1 datacenter1 rack1 Up Normal 11.15 KB 25.00%
0
127.0.0.2 datacenter1 rack1 Up Normal 6.73 KB 25.00%
42535295865117307932921825928971026432
127.0.0.3 datacenter1 rack1 Up Normal 13.52 KB 25.00%
85070591730234615865843651857942052864
127.0.0.4 datacenter1 rack1 Up Normal 9.09 KB 25.00%
127605887595351923798765477786913079296
{code}
* Write to the cluster with a network topology and consistency level of 4:
{code}
$ cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O
datacenter1:4 -n 100000 -e QUORUM
{code}
* Read from the cluster to make sure it's available:
{code}
$ cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O
datacenter1:4 -n 100000 -o read -e QUORUM
{code}
* Shut down nodes 2-4, breaking quorum and local_quorum consistency:
{code}
$ ccm node2 stop
$ ccm node3 stop
$ ccm node4 stop
{code}
This breaks quorum consistency:
{code}
$ ./cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O
datacenter1:4 -n 100000 -o read -e QUORUM
total,interval_op_rate,interval_key_rate,latency,95th,99th,elapsed_time
Operation [20] retried 10 times - error reading key 052627
((UnavailableException))
[...]
0,0,0,0.0,0.0,0.0,0
FAILURE
{code}
And it breaks local_quorum consistency:
{code}
$ cassandra-stress -R org.apache.cassandra.locator.NetworkTopologyStrategy -O
datacenter1:4 -n 100000 -o read -e LOCAL_QUORUM
total,interval_op_rate,interval_key_rate,latency,95th,99th,elapsed_time
Operation [12] retried 10 times - error reading key 059221
((UnavailableException))
[...]
0,0,0,0.0,0.0,0.0,0
FAILURE
{code}
I'm thinking there must be some other probken with hadoop going on here? Maybe
it's just hiding the error for some reason?
> LOCAL_QUORUM reads/writes don't throw UnavailableException if local quorum
> not present
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-5461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5461
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.9
> Environment: Cassandra 1.1.9.3, DSE 3.0.1, Ubuntu Linux 12.10
> Reporter: Piotr Kołaczkowski
> Assignee: Ryan McGuire
>
> {noformat}
> automaton@ip-10-171-69-238:~$ dsetool ring cfs
> Address DC Rack Workload Status State Load
> Effective-Ownership Token
>
> 127605887595351923798765477786913079296
> 10.171.69.238 Analytics rack1 Analytics(JT) Up Normal 151.19
> KB 100.00% 0
> 10.171.147.24 Analytics rack1 Analytics(TT) Down Down 103.79
> KB 100.00% 42535295865117307932921825928971026432
> 10.172.86.251 Analytics rack1 Analytics(TT) Down Down 145.03
> KB 100.00% 85070591730234615865843651857942052864
> 10.174.23.34 Analytics rack1 Analytics(TT) Down Down 158.88
> KB 100.00% 127605887595351923798765477786913079296
> {noformat}
> {noformat}
> Keyspace: cfs:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [analytics:4]
> {noformat}
> Reading from CFS with LOCAL_QUORUM:
> {noformat}
> automaton@ip-10-171-69-238:~$ dse hadoop fs -lsr /
> 13/04/12 16:18:23 INFO cfs.CassandraFileSystem: CassandraFileSystem.uri :
> cfs://ip-10-171-69-238.us-west-1.compute.internal/
> 13/04/12 16:18:23 INFO cfs.CassandraFileSystem: Default block size: 67108864
> 13/04/12 16:18:24 INFO cfs.CassandraFileSystemThriftStore: Consistency level
> for reads from cfs: LOCAL_QUORUM
> 13/04/12 16:18:24 INFO cfs.CassandraFileSystemThriftStore: Consistency level
> for writes into cfs: LOCAL_QUORUM
> 13/04/12 16:18:24 INFO cfs.CassandraFileSystemRules: Successfully loaded path
> rules for: cfs
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:42 /tf1
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:49 /tf10
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:43 /tf2
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:43 /tf3
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:43 /tf4
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:46 /tf5
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:46 /tf6
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:47 /tf7
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:47 /tf8
> -rwxrwxrwx 1 automaton automaton 10 2013-04-12 15:49 /tf9
> drwxrwxrwx - automaton automaton 0 2013-04-12 15:36 /tmp
> drwxrwxrwx - automaton automaton 0 2013-04-12 15:36
> /tmp/hadoop-automaton
> drwxrwxrwx - automaton automaton 0 2013-04-12 15:36
> /tmp/hadoop-automaton/mapred
> drwx------ - automaton automaton 0 2013-04-12 15:36
> /tmp/hadoop-automaton/mapred/system
> -rwxrwxrwx 1 automaton automaton 4 2013-04-12 15:36
> /tmp/hadoop-automaton/mapred/system/jobtracker.info
> {noformat}
> Now exactly the same with QUORUM:
> {noformat}
> automaton@ip-10-171-69-238:~$ dse hadoop fs -lsr /
> 13/04/12 16:19:21 INFO cfs.CassandraFileSystem: CassandraFileSystem.uri :
> cfs://ip-10-171-69-238.us-west-1.compute.internal/
> 13/04/12 16:19:21 INFO cfs.CassandraFileSystem: Default block size: 67108864
> 13/04/12 16:19:22 INFO cfs.CassandraFileSystemThriftStore: Consistency level
> for reads from cfs: QUORUM
> 13/04/12 16:19:22 INFO cfs.CassandraFileSystemThriftStore: Consistency level
> for writes into cfs: QUORUM
> 13/04/12 16:19:22 INFO cfs.CassandraFileSystemRules: Successfully loaded path
> rules for: cfs
> lsr: UnavailableException()
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
