So, I found a solution, but I think there is a (trivial) bug.
The broken keyspace:
create keyspace KeyspaceDC1 with
replication_factor = 1 and
placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
and
strategy_options = [{DC1:1, DC2:0}];
The fixed keyspace:
create keyspace KeyspaceDC1 with
replication_factor = 1 and
placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
and
strategy_options = [{DC1:1}];
I'm not sure if the issue is that there are no nodes in DC2 or that I'm saying
DC2 shouldn't get any replicas, or a combination of the two. If I get time
I'll try to poke around the code base.
-Thor
On Dec 29, 2010, at 12:38 PM, Thor Carpenter wrote:
In an attempt to replicate this error on a simpler system I have come across a
different error but seemingly related problem. Now I am getting the following
exception in system.log.
DEBUG [pool-1-thread-3] 2010-12-29 12:10:38,897 CassandraServer.java (line 362)
insert
ERROR [pool-1-thread-3] 2010-12-29 12:10:38,906 Cassandra.java (line 2960)
Internal error processing insert
java.lang.AssertionError
at
org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:392)
at
org.apache.cassandra.locator.TokenMetadata.ringIterator(TokenMetadata.java:417)
at
org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:95)
at
org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:99)
at
org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1411)
at
org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:1394)
at
org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:109)
at
org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:442)
at
org.apache.cassandra.thrift.CassandraServer.insert(CassandraServer.java:379)
at
org.apache.cassandra.thrift.Cassandra$Processor$insert.process(Cassandra.java:2952)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2555)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Looking at TokenMetadata.java:392, the assert that is triggered is:
assert ring.size() > 0;
Which led me to run nodetool ring which outputs a seemingly correct config:
-sh-3.2$ nodetool -h localhost -p 8080 ring
Address Status State Load Owns Token
10.5.64.26 Up Normal 47.16 KB 100.00% 1
To reproduce:
* I have upgraded to 0.7RC3 so install the latest riptano 0.7rc3 rpm on a
single node in "DC1".
* In cassandra.yaml set initial_token = 1
* cassandra-topology.properties:
10.5.64.26=DC1:R1
default=DC2:R1
* Schema loaded via cassandra-cli:
create keyspace KeyspaceDC1 with
replication_factor = 1 and
placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
and
strategy_options = [{DC1:1, DC2:0}];
use KeyspaceDC1;
create column family TestCF with
column_type = 'Standard' and
comparator = 'BytesType' and
keys_cached = 200000 and
rows_cached = 2000 and
gc_grace = 0 and
read_repair_chance = 0.0;
* In cassandra-cli execute the following:
[defa...@unknown] use KeyspaceDC1;
Authenticated to keyspace: KeyspaceDC1
[defa...@keyspacedc1] set TestCF['some key']['some col'] = 'some value';
Internal error processing insert
* Check system.log where you should find the above assertion error.
Can anybody find a flaw in this setup? Do others agree something looks buggy?
Thanks,
-Thor
On Dec 28, 2010, at 4:33 PM, Thor Carpenter wrote:
Third try for good luck. Maybe this time I'll get it all in one email. :-)
At time 15:55 I performed a write with CL = ONE, a subsequent read with CL =
ONE and received correct results. Similar results occur with any read CL other
than LOCAL_QUORUM.
At time 15:58 I performed a write with CL = ONE, a subsequent read with CL =
LOCAL_QUORUM and received a TimedOutException.
System.log and cassandra.log are from server dlc05-sea to corresponds with the
hector error message "Closing client CassandraClient<dlc05-sea:9160-50>". I
haven't found anything in the logs that indicates the problem but maybe others
with more discerning eyes can.
Please let me know if I can provide any additional information.
Thanks,
-Thor
<logs.zip><ATT00001..txt>
