[
https://issues.apache.org/jira/browse/CASSANDRA-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180069#comment-13180069
]
Brandon Williams commented on CASSANDRA-3696:
---------------------------------------------
I can reproduce this reliably, but only if read repair is on. With it off,
everything works. Here is failed read:
{noformat}
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 CassandraServer.java (line 323)
get_slice
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 StorageProxy.java (line 603)
Command/ConsistencyLevel is SliceFromReadCommand(table='Keyspace1',
key='303637', column_parent='QueryPath(columnFamilyName='Standard1',
superColumnName='null', columnName='null')', start='', finish='',
reversed=false, count=5)/LOCAL_QUORUM
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 PropertyFileSnitch.java (line
90) Could not find end point information for cassandra-1/10.179.65.102, will
use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 PropertyFileSnitch.java (line
90) Could not find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 ReadCallback.java (line 77)
Blockfor/repair is 1/true; setting up requests to /10.179.64.227,/10.179.111.137
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 PropertyFileSnitch.java (line
90) Could not find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,523 StorageProxy.java (line 624)
reading data from /10.179.64.227
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,524 StorageProxy.java (line 644)
reading digest from /10.179.111.137
DEBUG [RequestResponseStage:9] 2012-01-05 01:00:51,526 ResponseVerbHandler.java
(line 44) Processing response on a callback from 4145@/10.179.111.137
DEBUG [RequestResponseStage:10] 2012-01-05 01:00:51,526
ResponseVerbHandler.java (line 44) Processing response on a callback from
4144@/10.179.64.227
DEBUG [RequestResponseStage:9] 2012-01-05 01:00:51,526 AbstractRowResolver.java
(line 66) Preprocessed digest response
DEBUG [RequestResponseStage:10] 2012-01-05 01:00:51,526
AbstractRowResolver.java (line 66) Preprocessed data response
DEBUG [RequestResponseStage:9] 2012-01-05 01:00:51,527 PropertyFileSnitch.java
(line 90) Could not find end point information for /10.179.111.137, will use
default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,527 StorageProxy.java (line 672)
Read: 3 ms.
{noformat}
In this case, 10.179.65.102 and 10.179.111.137 are in the first datacenter, and
10.179.64.227 is in the second. Both DCs have an RF of one, .102 is always the
coordinator and the client is using a single connection.
For comparison, here is a successful local read where read repair did not fire
(set to 10%):
{noformat}
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,519 CassandraServer.java (line 323)
get_slice
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,519 StorageProxy.java (line 603)
Command/ConsistencyLevel is SliceFromReadCommand(table='Keyspace1',
key='303632', column_parent='QueryPath(columnFamilyName='Standard1',
superColumnName='null', columnName='null')', start='', finish='',
reversed=false, count=5)/LOCAL_QUORUM
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,519 PropertyFileSnitch.java (line
90) Could not find end point information for cassandra-1/10.179.65.102, will
use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,520 ReadCallback.java (line 77)
Blockfor/repair is 1/false; setting up requests to cassandra-1/10.179.65.102
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,520 PropertyFileSnitch.java (line
90) Could not find end point information for cassandra-1/10.179.65.102, will
use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,520 StorageProxy.java (line 619)
reading data locally
DEBUG [ReadStage:93] 2012-01-05 01:00:51,520 StorageProxy.java (line 763)
LocalReadRunnable reading SliceFromReadCommand(table='Keyspace1', key='303632',
column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null',
columnName='null')', start='', finish='', reversed=false, count=5)
DEBUG [ReadStage:93] 2012-01-05 01:00:51,520 CollationController.java (line
192) collectAllData
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123)
collecting 0 of 5: C0:false:34@1325724968371
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123)
collecting 1 of 5: C1:false:34@1325724968371
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123)
collecting 2 of 5: C2:false:34@1325724968371
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123)
collecting 3 of 5: C3:false:34@1325724968371
DEBUG [ReadStage:93] 2012-01-05 01:00:51,521 SliceQueryFilter.java (line 123)
collecting 4 of 5: C4:false:34@1325724968371
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,522 StorageProxy.java (line 672)
Read: 2 ms.
{noformat}
And one where it asked .137 for data (but did not leave the DC or repair):
{noformat}
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 CassandraServer.java (line 323)
get_slice
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 StorageProxy.java (line 603)
Command/ConsistencyLevel is SliceFromReadCommand(table='Keyspace1',
key='303731', column_parent='QueryPath(columnFamilyName='Standard1',
superColumnName='null', columnName='null')', start='', finish='',
reversed=false, count=5)/LOCAL_QUORUM
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 PropertyFileSnitch.java (line
90) Could not find end point information for cassandra-1/10.179.65.102, will
use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 PropertyFileSnitch.java (line
90) Could not find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,514 PropertyFileSnitch.java (line
90) Could not find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,515 ReadCallback.java (line 77)
Blockfor/repair is 1/false; setting up requests to /10.179.111.137
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,515 PropertyFileSnitch.java (line
90) Could not find end point information for /10.179.111.137, will use default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,515 StorageProxy.java (line 624)
reading data from /10.179.111.137
DEBUG [RequestResponseStage:12] 2012-01-05 01:00:51,517
ResponseVerbHandler.java (line 44) Processing response on a callback from
4143@/10.179.111.137
DEBUG [RequestResponseStage:12] 2012-01-05 01:00:51,517
AbstractRowResolver.java (line 66) Preprocessed data response
DEBUG [RequestResponseStage:12] 2012-01-05 01:00:51,517 PropertyFileSnitch.java
(line 90) Could not find end point information for /10.179.111.137, will use
default
DEBUG [pool-2-thread-7] 2012-01-05 01:00:51,517 StorageProxy.java (line 672)
Read: 2 ms.
{noformat}
> Adding another datacenter's node results in 0 rows returned on first
> datacenter
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-3696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3696
> Project: Cassandra
> Issue Type: Bug
> Reporter: Joaquin Casares
>
> On Cassandra-1.0.5:
> 1. Create a node in C* with a fresh installation and create a keyspace on
> that node with one column family -
> CREATE KEYSPACE test
> WITH placement_strategy = 'SimpleStrategy'
> and strategy_options={replication_factor:1};
> use test;
> create column family cf1;
> 2. Insert values into cf1 -
> set cf1[ascii('k')][ascii('c')] = ascii('v');
> get cf1[ascii('k')];
> => (column=63, value=v, timestamp=1325689630397000)
> Returned 1 results.
> 3. update the strategy options from simple to networktopology with
> {Cassandra:1, Backup:1}
> 4. read from cf1 to make sure the options change doesn't affect anything -
> consistencylevel as LOCAL_QUORUM;
> get cf1[ascii('k')];
> => (column=63, value=v, timestamp=1325689630397000)
> Returned 1 results.
> 5. start a second node in the Backup datacenter
> 6. read from cf1 again (on the first node) -
> consistencylevel as LOCAL_QUORUM;
> get cf1[ascii('k')];
> Returned 0 results.
> After about 60 seconds, "get cf1[ascii('k')]" started to return results
> again.
> Also, when running at a CL of ONE on 1.0's head, we were able to see issues
> as well.
> But, if more than one node was added to the second datacenter, then
> replication_strategy is changed, it seems okay.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
