[
https://issues.apache.org/jira/browse/CASSANDRA-781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833477#action_12833477
]
bjc commented on CASSANDRA-781:
-------------------------------
Thanks for fixing the patch, but I'm sorry...it still doesn't work right.. :( I
found a couple of weird things. First, sometimes when I ask for 10, it gives
more. Second, if I pass start="" it still doesn't start at the beginning.
However, this doesn't happen every time I start fresh. Maybe it's depedent on
the tokens. Here are the tokens for the case I show below:
INFO 23:38:21,989 Saved Token not found. Using 0njMRYmU9KiXE80d
INFO 23:38:20,112 Saved Token not found. Using I8LW8J6h9UCuz0dC
The first token belongs to the node I am attaching the client to.
This functionality seems surprisingly complicated. Maybe it would help to write
down pseudo-code for the way it should work? I have to admit that I cannot
piece it together by reading the comments in your patch.
Here is a transcript and test:
First run, looks ok:
ip-10-212-87-165$ python test_bug_simple3.py
insert 10b470f49a7c46bd938d784ca4096b63
insert 47f70aacb3e94e10acaf8e86edac7169
insert 9a3b7d3b921345bebc4f2bedc1db7c01
insert c1c9dab59abd4f4ca33ee79f71a179e9
insert cff81b145faf4648ac8ae001973c6c75
insert c752d33e5d344312908e5008e6cdae3e
insert 6e9e32e8b89845bb935d993a9c8bcb13
insert c286bf2711bc45c1ab033561112c2313
insert 2dad487ddfa94c81b52c8b4d35d3cb5c
insert 9c62c7dafdb94dfdbdf52b527bdd2b24
result 10b470f49a7c46bd938d784ca4096b63
result 2dad487ddfa94c81b52c8b4d35d3cb5c
result 47f70aacb3e94e10acaf8e86edac7169
result 6e9e32e8b89845bb935d993a9c8bcb13
result 9a3b7d3b921345bebc4f2bedc1db7c01
result 9c62c7dafdb94dfdbdf52b527bdd2b24
result c1c9dab59abd4f4ca33ee79f71a179e9
result c286bf2711bc45c1ab033561112c2313
result c752d33e5d344312908e5008e6cdae3e
result cff81b145faf4648ac8ae001973c6c75
total_keys 10
Second run, get 18 keys when I asked for 10:
ip-10-212-87-165$ python test_bug_simple3.py
insert f873d662dccf46c28080a01286e09ed8
insert 903776c2f45740389aa52675bf47c7ec
insert 0e80401a9052405a898d11e5ae874a13
insert 398d51ba174b4c9db8c25ca6cd2c9454
insert 50f1cd47dd284ee9b9573b4dfce39134
insert 20fa43d2365b4dfab9b05a93992315d0
insert e009d5b76e8840b784fe6b9b649ae1df
insert 63497f9d63c74b99a681fa2fc52751ac
insert 824bbcf997de48a99cad174e9e1f1eec
insert 01c5a6506f4247068660c20338a03bb3
result 01c5a6506f4247068660c20338a03bb3
result 0e80401a9052405a898d11e5ae874a13
result 10b470f49a7c46bd938d784ca4096b63
result 20fa43d2365b4dfab9b05a93992315d0
result 2dad487ddfa94c81b52c8b4d35d3cb5c
result 398d51ba174b4c9db8c25ca6cd2c9454
result 47f70aacb3e94e10acaf8e86edac7169
result 50f1cd47dd284ee9b9573b4dfce39134
result 63497f9d63c74b99a681fa2fc52751ac
result 6e9e32e8b89845bb935d993a9c8bcb13
result 824bbcf997de48a99cad174e9e1f1eec
result 903776c2f45740389aa52675bf47c7ec
result c1c9dab59abd4f4ca33ee79f71a179e9
result c286bf2711bc45c1ab033561112c2313
result c752d33e5d344312908e5008e6cdae3e
result cff81b145faf4648ac8ae001973c6c75
result e009d5b76e8840b784fe6b9b649ae1df
result f873d662dccf46c28080a01286e09ed8
total_keys 18
Third run, start at "ca.." even though I pass start="" and all the previous
keys remain:
ip-10-212-87-165$ python test_bug_simple3.py
insert ca97d7efb63448f8a62d6f7f73044236
insert 91363a713b714af88ac2191caeea5351
insert 7b6756d0ab8e450b826b1abc7210d524
insert e6c6765497af4078b93e1a1470bd3194
insert e3457f26754c4e7cb7ef606f98e7bb78
insert 99643eb237ea4ca8b50cac4bb4d58edd
insert ec3e1f81359b4ae08cfed73899934a93
insert ae2b990ceb044bf194a879059f823ecf
insert 2c1494f0ad3d48d2bf4feb33f40cf38e
insert 0cb1c2e906b64fee89f7729052e0810e
result ae2b990ceb044bf194a879059f823ecf
result c1c9dab59abd4f4ca33ee79f71a179e9
result c286bf2711bc45c1ab033561112c2313
result c752d33e5d344312908e5008e6cdae3e
result ca97d7efb63448f8a62d6f7f73044236
result cff81b145faf4648ac8ae001973c6c75
result e009d5b76e8840b784fe6b9b649ae1df
result e3457f26754c4e7cb7ef606f98e7bb78
result e6c6765497af4078b93e1a1470bd3194
result ec3e1f81359b4ae08cfed73899934a93
total_keys 10
ip-10-212-87-165$
Here is another run (different tokens):
INFO 00:04:17,105 Saved Token not found. Using Iw1khrAgM5sd6WnX
INFO 00:04:15,795 Saved Token not found. Using IgLbq912n2xEP99G
In this case I don't see the problem where I get back more keys than I asked
for, but I don't get the 10 lowest keys in the second request. 10 are returned,
but they are not ordered consistently with what I know is in the db.
First run, notice key "893.." is inserted:
ip-10-212-87-165$ python test_bug_simple3.py
insert 7be5d87bc45843cfaffd36fd654aee53
insert 8ef4727d83474570aa2111bee3929a5f
insert 9a9a91b6b662430092db0209d63a5c9e
insert e45afe1f0e364012acd0dead5b75ea13
insert 10171c87634842aea4f16d46d611c435
insert 10b6f92ac6a447088a82c4ec13056f1e
insert c1acbde9ae454ea2819322975322206b
insert 89352cf117dd4cb9ab935cbb5f230ba0
insert 0b5d924f04174459969594d6293b9aca
insert e2471db4d8f445f2b0c36f3b2a5bb650
result 0b5d924f04174459969594d6293b9aca
result 10171c87634842aea4f16d46d611c435
result 10b6f92ac6a447088a82c4ec13056f1e
result 7be5d87bc45843cfaffd36fd654aee53
result 89352cf117dd4cb9ab935cbb5f230ba0
result 8ef4727d83474570aa2111bee3929a5f
result 9a9a91b6b662430092db0209d63a5c9e
result c1acbde9ae454ea2819322975322206b
result e2471db4d8f445f2b0c36f3b2a5bb650
result e45afe1f0e364012acd0dead5b75ea13
total_keys 10
Second run, notice the results start with "0.." but "893.." is not returned
(though "b2.." is, and other higher keys):
ip-10-212-87-165$ python test_bug_simple3.py
insert 85d282dfa03a466eb51d03f4eb5dacd5
insert a61e7757eaed4ef79fc7bf35f47843f7
insert 9b9b6e3f22994827b0dddcc16105ff7d
insert 880b1644636845d8b1c92faf1f6d8484
insert 5d1d7d7b26ec4540a89c027bccc17e06
insert 4df05d38950f44b29df604c165e1148f
insert 0c9f818aa28a47fb832b6a0929b94280
insert 7e7b829c136046a88b120a4a373d9a6b
insert b2d497713f9342de85bb31b5c0e69af6
insert 80e250253f1942aaab2e9b49880918c0
result 0b5d924f04174459969594d6293b9aca
result 0c9f818aa28a47fb832b6a0929b94280
result 10171c87634842aea4f16d46d611c435
result 10b6f92ac6a447088a82c4ec13056f1e
result 4df05d38950f44b29df604c165e1148f
result a61e7757eaed4ef79fc7bf35f47843f7
result b2d497713f9342de85bb31b5c0e69af6
result c1acbde9ae454ea2819322975322206b
result e2471db4d8f445f2b0c36f3b2a5bb650
result e45afe1f0e364012acd0dead5b75ea13
total_keys 10
ip-10-212-87-165$
I found another issue with "nodetool ring" which might be related to the patch:
$ sudo bin/nodetool -h localhost ring
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at $Proxy0.getRangeToEndPointMap(Unknown Source)
at
org.apache.cassandra.tools.NodeProbe.getRangeToEndPointMap(NodeProbe.java:151)
at org.apache.cassandra.tools.NodeCmd.printRing(NodeCmd.java:74)
at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:403)
Caused by: java.rmi.UnmarshalException: error unmarshalling return; nested
exception is:
java.io.WriteAbortedException: writing aborted;
java.io.NotSerializableException:
org.apache.cassandra.dht.OrderPreservingPartitioner
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:173)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown
Source)
at
javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993)
at
javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288)
... 4 more
Caused by: java.io.WriteAbortedException: writing aborted;
java.io.NotSerializableException:
org.apache.cassandra.dht.OrderPreservingPartitioner
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1333)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1947)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1871)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at java.util.HashMap.readObject(HashMap.java:1029)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1849)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1753)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at sun.rmi.server.UnicastRef.unmarshalValue(UnicastRef.java:306)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:155)
... 8 more
Caused by: java.io.NotSerializableException:
org.apache.cassandra.dht.OrderPreservingPartitioner
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1156)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at java.util.HashMap.writeObject(HashMap.java:1000)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at
java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
$
Here is the test I ran above:
import uuid
from thrift import Thrift
from thrift.transport import TTransport
from thrift.transport import TSocket
from thrift.protocol.TBinaryProtocol import TBinaryProtocolAccelerated
import sys
sys.path.insert(0,'/usr/local/cassandra/interface/thrift/gen-py')
from cassandra import Cassandra
from cassandra.ttypes import *
socket = TSocket.TSocket("10.212.87.165", 9160)
transport = TTransport.TBufferedTransport(socket)
protocol = TBinaryProtocol.TBinaryProtocolAccelerated(transport)
client = Cassandra.Client(protocol)
transport.open()
ks = "Keyspace1"
cf = "Super1"
path = ColumnPath(cf, "foo", "is")
value = "cool"
for i in xrange(10):
key = uuid.uuid4().hex
print "insert", key
client.insert(ks, key, path, value, 0, ConsistencyLevel.ONE)
print
parent = ColumnParent(column_family=cf)
slice_range = SliceRange(start="key", finish="key")
predicate = SlicePredicate(slice_range=slice_range)
total_keys = 0
result = client.get_range_slice(ks, parent, predicate, "", "", 10,
ConsistencyLevel.ONE)
for row in result:
total_keys += 1
print "result", row.key
print "total_keys", total_keys
> in a cluster, get_range_slice() does not return all the keys it should
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-781
> URL: https://issues.apache.org/jira/browse/CASSANDRA-781
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.5
> Environment: Debian 5 lenny on EC2, Gentoo linux, Windows XP
> Reporter: bjc
> Assignee: Jonathan Ellis
> Fix For: 0.5, 0.6
>
> Attachments: 781.txt
>
>
> get_range_slice() does not return the same set of keys as get_key_range() in
> 0.5.0 final.
> I posted a program to reproduce the behavior:
> http://www.mail-archive.com/[email protected]/msg01474.html
> Apparently, you must have more than one node to get the behavior. Also, it
> may depend on the locations of the nodes on the ring.. I.e., if you don't
> generate enough keys randomly, then by chance they could all fall on the same
> host and you might not see the behavior, although I was able to get it to
> happen using only 2 nodes and 10 keys.
> Here are the other emails describing the issue:
> http://www.mail-archive.com/[email protected]/msg02423.html
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
