[jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018257#comment-14018257 ] Benedict edited comment on CASSANDRA-7245 at 6/4/14 10:08 PM: -- [~jasobrown] I got this one if you've got better stuff to do :-) [~tjake]: * I think the change you made doesn't actually totally eliminate the bug: since it extends DroppableRunnable, the finally block may never be run * I think it would be good in SP.mutate() you can't release the mutations until after we get the all clear from the replicas, as we may have to use the mutations to write local hints. It might be nice, however, to have the response handler release() the reference once we receive enough responses from our replicas, so that we don't keep all of the references if we're just waiting for one mutation * Why do we need a ThreadLocal in ClientState to store the sourceFrame. Can't we just store it directly in a field in QueryState? Nit: * WS in ClientState constructor; would also prefer to just create ThreadLocal in var initialiser since it's always init'd empty Also, this is off-topic, but I wonder if we shouldn't replace the NBHM in ServerConnection with a simple array. We know the range is 32K (.1K for older clients), and each index is accessed by a single thread at any given time, so we'd just need to be able to atomically swap in larger arrays if we wanted dynamic sizing. Otherwise LGTM was (Author: benedict): [~jasobrown] I got this one if you've got better stuff to do :-) [~tjke]: * I think the change you made doesn't actually totally eliminate the bug: since it extends DroppableRunnable, the finally block may never be run * I think it would be good in SP.mutate() you can't release the mutations until after we get the all clear from the replicas, as we may have to use the mutations to write local hints. It might be nice, however, to have the response handler release() the reference once we receive enough responses from our replicas, so that we don't keep all of the references if we're just waiting for one mutation * Why do we need a ThreadLocal in ClientState to store the sourceFrame. Can't we just store it directly in a field in QueryState? Nit: * WS in ClientState constructor; would also prefer to just create ThreadLocal in var initialiser since it's always init'd empty Also, this is off-topic, but I wonder if we shouldn't replace the NBHM in ServerConnection with a simple array. We know the range is 32K (.1K for older clients), and each index is accessed by a single thread at any given time, so we'd just need to be able to atomically swap in larger arrays if we wanted dynamic sizing. Otherwise LGTM Out-of-Order keys with stress + CQL3 Key: CASSANDRA-7245 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 Project: Cassandra Issue Type: Bug Components: Core Reporter: Pavel Yaskevich Assignee: T Jake Luciani Fix For: 2.1.0 Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, 7245v4.txt, netty-all-4.0.19.Final.jar We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables). We set up they keys to be 10 bytes in size (default) and population between 1 and 6. Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions: _There is a problem with sorting of normal looking keys:_ 30303039443538353645 30303039443745364242 java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) DecoratedKey(-1767746583617597213, *30303039443437454333*) 0a30303033343933 3734441388343933 java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *3734441388343933*) DecoratedKey(-7565486415339257200, *30303033344639443137*) 30303033354244363031 30303033354133423742 java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) DecoratedKey(-7838239767410066684, *30303033354145344534*) 30303034313442354137 3034313635363334 java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *3034313635363334*) DecoratedKey(-9106177395653818217, *3030303431444238*) 30303035373044373435 30303035373044334631 java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) DecoratedKey(-4296696226469000945, *30303035373132364138*) _And completely different ones:_ 30303041333745373543 7cd045c59a90d7587d8d java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) DecoratedKey(-5146766422778260690, *30303041333943303232*)
[jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14018257#comment-14018257 ] Benedict edited comment on CASSANDRA-7245 at 6/4/14 10:08 PM: -- [~jasobrown] I got this one if you've got better stuff to do :-) [~tjake]: * I think the change you made doesn't actually totally eliminate the bug: since it extends DroppableRunnable, the finally block may never be run * In SP.mutate() you can't release the mutations until after we get the all clear from the replicas, as we may have to use the mutations to write local hints. It might be nice, however, to have the response handler release() the reference once we receive enough responses from our replicas, so that we don't keep all of the references if we're just waiting for one mutation * Why do we need a ThreadLocal in ClientState to store the sourceFrame. Can't we just store it directly in a field in QueryState? Nit: * WS in ClientState constructor; would also prefer to just create ThreadLocal in var initialiser since it's always init'd empty Also, this is off-topic, but I wonder if we shouldn't replace the NBHM in ServerConnection with a simple array. We know the range is 32K (.1K for older clients), and each index is accessed by a single thread at any given time, so we'd just need to be able to atomically swap in larger arrays if we wanted dynamic sizing. Otherwise LGTM was (Author: benedict): [~jasobrown] I got this one if you've got better stuff to do :-) [~tjake]: * I think the change you made doesn't actually totally eliminate the bug: since it extends DroppableRunnable, the finally block may never be run * I think it would be good in SP.mutate() you can't release the mutations until after we get the all clear from the replicas, as we may have to use the mutations to write local hints. It might be nice, however, to have the response handler release() the reference once we receive enough responses from our replicas, so that we don't keep all of the references if we're just waiting for one mutation * Why do we need a ThreadLocal in ClientState to store the sourceFrame. Can't we just store it directly in a field in QueryState? Nit: * WS in ClientState constructor; would also prefer to just create ThreadLocal in var initialiser since it's always init'd empty Also, this is off-topic, but I wonder if we shouldn't replace the NBHM in ServerConnection with a simple array. We know the range is 32K (.1K for older clients), and each index is accessed by a single thread at any given time, so we'd just need to be able to atomically swap in larger arrays if we wanted dynamic sizing. Otherwise LGTM Out-of-Order keys with stress + CQL3 Key: CASSANDRA-7245 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 Project: Cassandra Issue Type: Bug Components: Core Reporter: Pavel Yaskevich Assignee: T Jake Luciani Fix For: 2.1.0 Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, 7245v4.txt, netty-all-4.0.19.Final.jar We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables). We set up they keys to be 10 bytes in size (default) and population between 1 and 6. Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions: _There is a problem with sorting of normal looking keys:_ 30303039443538353645 30303039443745364242 java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) DecoratedKey(-1767746583617597213, *30303039443437454333*) 0a30303033343933 3734441388343933 java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *3734441388343933*) DecoratedKey(-7565486415339257200, *30303033344639443137*) 30303033354244363031 30303033354133423742 java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) DecoratedKey(-7838239767410066684, *30303033354145344534*) 30303034313442354137 3034313635363334 java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *3034313635363334*) DecoratedKey(-9106177395653818217, *3030303431444238*) 30303035373044373435 30303035373044334631 java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) DecoratedKey(-4296696226469000945, *30303035373132364138*) _And completely different ones:_ 30303041333745373543 7cd045c59a90d7587d8d java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) DecoratedKey(-5146766422778260690, *30303041333943303232*) 3030303332314144
[jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14017249#comment-14017249 ] Jason Brown edited comment on CASSANDRA-7245 at 6/3/14 11:18 PM: - [~tjake] v4 looks good for my tests. [~benedict] looks like you've been playing the at-home game, as well. Do you want to handle the review? Else, I can do it (not a problem) was (Author: jasobrown): [~tjake] v4 looks good for my tests. [~benedict] looks like you've been playing the at home game, as well. Do you want to handle the review? Else, I can do it (not a problem) Out-of-Order keys with stress + CQL3 Key: CASSANDRA-7245 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 Project: Cassandra Issue Type: Bug Components: Core Reporter: Pavel Yaskevich Assignee: T Jake Luciani Fix For: 2.1.0 Attachments: 7245-v2.txt, 7245.txt, 7245v3-rebase.txt, 7245v3.txt, 7245v4.txt, netty-all-4.0.19.Final.jar We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables). We set up they keys to be 10 bytes in size (default) and population between 1 and 6. Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions: _There is a problem with sorting of normal looking keys:_ 30303039443538353645 30303039443745364242 java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) DecoratedKey(-1767746583617597213, *30303039443437454333*) 0a30303033343933 3734441388343933 java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *3734441388343933*) DecoratedKey(-7565486415339257200, *30303033344639443137*) 30303033354244363031 30303033354133423742 java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) DecoratedKey(-7838239767410066684, *30303033354145344534*) 30303034313442354137 3034313635363334 java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *3034313635363334*) DecoratedKey(-9106177395653818217, *3030303431444238*) 30303035373044373435 30303035373044334631 java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) DecoratedKey(-4296696226469000945, *30303035373132364138*) _And completely different ones:_ 30303041333745373543 7cd045c59a90d7587d8d java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) DecoratedKey(-5146766422778260690, *30303041333943303232*) 3030303332314144 30303033323346343932 java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, *30303033323346343932*) DecoratedKey(5233296131921119414, *53d83e0012287e03*) 30303034314531374431 3806734b256c27e41ec2 java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, *3806734b256c27e41ec2*) DecoratedKey(-8072288379146044663, *30303034314136413343*) _And sometimes there is no problem at all:_ 30303033353144463637 002a31b3b31a1c2f 5d616dd38211ebb5d6ec 444236451388 1388138844463744 30303033353143394343 It's worth to mention that we have got 22 timeout exceptions but number of out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011012#comment-14011012 ] Benedict edited comment on CASSANDRA-7245 at 5/28/14 11:00 AM: --- Well, 5663 depends on 6861, so it explicitly releases references (as 6861 did), so it's not a clean test. I reckon we fix the bug we know exists, and then we test thoroughly to confirm everything's okay once it's been stomped on - it'll soon come out of the woodwork if there's something else to blame as well. was (Author: benedict): Well, 5663 depends on 6861, so it explicitly releases references (as 6861 did), so it's not exactly a clean test Out-of-Order keys with stress + CQL3 Key: CASSANDRA-7245 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 Project: Cassandra Issue Type: Bug Components: Core Reporter: Pavel Yaskevich Assignee: T Jake Luciani Fix For: 2.1 rc1 We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables). We set up they keys to be 10 bytes in size (default) and population between 1 and 6. Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions: _There is a problem with sorting of normal looking keys:_ 30303039443538353645 30303039443745364242 java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) DecoratedKey(-1767746583617597213, *30303039443437454333*) 0a30303033343933 3734441388343933 java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *3734441388343933*) DecoratedKey(-7565486415339257200, *30303033344639443137*) 30303033354244363031 30303033354133423742 java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) DecoratedKey(-7838239767410066684, *30303033354145344534*) 30303034313442354137 3034313635363334 java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *3034313635363334*) DecoratedKey(-9106177395653818217, *3030303431444238*) 30303035373044373435 30303035373044334631 java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) DecoratedKey(-4296696226469000945, *30303035373132364138*) _And completely different ones:_ 30303041333745373543 7cd045c59a90d7587d8d java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) DecoratedKey(-5146766422778260690, *30303041333943303232*) 3030303332314144 30303033323346343932 java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, *30303033323346343932*) DecoratedKey(5233296131921119414, *53d83e0012287e03*) 30303034314531374431 3806734b256c27e41ec2 java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, *3806734b256c27e41ec2*) DecoratedKey(-8072288379146044663, *30303034314136413343*) _And sometimes there is no problem at all:_ 30303033353144463637 002a31b3b31a1c2f 5d616dd38211ebb5d6ec 444236451388 1388138844463744 30303033353143394343 It's worth to mention that we have got 22 timeout exceptions but number of out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011835#comment-14011835 ] Benedict edited comment on CASSANDRA-7245 at 5/28/14 11:45 PM: --- Except that we sort in the Memtable/CLSM (and assert order at flush) by Token which is computed much earlier, whereas we write the raw key only, which could have been corrupted by the time we copy it. was (Author: benedict): Except that we sort in the Memtable/CLSM (and assert order at flush) by Token which is computed much earlier, whereas we write the raw key only. Out-of-Order keys with stress + CQL3 Key: CASSANDRA-7245 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 Project: Cassandra Issue Type: Bug Components: Core Reporter: Pavel Yaskevich Assignee: T Jake Luciani Fix For: 2.1 rc1 Attachments: 7245-v2.txt, 7245.txt We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables). We set up they keys to be 10 bytes in size (default) and population between 1 and 6. Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions: _There is a problem with sorting of normal looking keys:_ 30303039443538353645 30303039443745364242 java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) DecoratedKey(-1767746583617597213, *30303039443437454333*) 0a30303033343933 3734441388343933 java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *3734441388343933*) DecoratedKey(-7565486415339257200, *30303033344639443137*) 30303033354244363031 30303033354133423742 java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) DecoratedKey(-7838239767410066684, *30303033354145344534*) 30303034313442354137 3034313635363334 java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *3034313635363334*) DecoratedKey(-9106177395653818217, *3030303431444238*) 30303035373044373435 30303035373044334631 java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) DecoratedKey(-4296696226469000945, *30303035373132364138*) _And completely different ones:_ 30303041333745373543 7cd045c59a90d7587d8d java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) DecoratedKey(-5146766422778260690, *30303041333943303232*) 3030303332314144 30303033323346343932 java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, *30303033323346343932*) DecoratedKey(5233296131921119414, *53d83e0012287e03*) 30303034314531374431 3806734b256c27e41ec2 java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, *3806734b256c27e41ec2*) DecoratedKey(-8072288379146044663, *30303034314136413343*) _And sometimes there is no problem at all:_ 30303033353144463637 002a31b3b31a1c2f 5d616dd38211ebb5d6ec 444236451388 1388138844463744 30303033353143394343 It's worth to mention that we have got 22 timeout exceptions but number of out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14011850#comment-14011850 ] Jason Brown edited comment on CASSANDRA-7245 at 5/28/14 11:58 PM: -- [~benedict] Just to clarify a few things: Is the sorting within Memtable.put(), due to adding the entry to the ConcurrentSkipListMap? bq. assert order at flush SSTableWriter.beforeAppend(), correct? bq. whereas we write the raw key only still SSTableWriter? was (Author: jasobrown): [~benedict] Just to clarify a few thing: Is the sorting within Memtable.put(), due to adding the entry to the ConcurrentSkipListMap? bq. assert order at flush SSTableWriter.beforeAppend(), correct? bq. whereas we write the raw key only still SSTableWriter? Out-of-Order keys with stress + CQL3 Key: CASSANDRA-7245 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 Project: Cassandra Issue Type: Bug Components: Core Reporter: Pavel Yaskevich Assignee: T Jake Luciani Fix For: 2.1 rc1 Attachments: 7245-v2.txt, 7245.txt We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables). We set up they keys to be 10 bytes in size (default) and population between 1 and 6. Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions: _There is a problem with sorting of normal looking keys:_ 30303039443538353645 30303039443745364242 java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) DecoratedKey(-1767746583617597213, *30303039443437454333*) 0a30303033343933 3734441388343933 java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *3734441388343933*) DecoratedKey(-7565486415339257200, *30303033344639443137*) 30303033354244363031 30303033354133423742 java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) DecoratedKey(-7838239767410066684, *30303033354145344534*) 30303034313442354137 3034313635363334 java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *3034313635363334*) DecoratedKey(-9106177395653818217, *3030303431444238*) 30303035373044373435 30303035373044334631 java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) DecoratedKey(-4296696226469000945, *30303035373132364138*) _And completely different ones:_ 30303041333745373543 7cd045c59a90d7587d8d java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) DecoratedKey(-5146766422778260690, *30303041333943303232*) 3030303332314144 30303033323346343932 java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, *30303033323346343932*) DecoratedKey(5233296131921119414, *53d83e0012287e03*) 30303034314531374431 3806734b256c27e41ec2 java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, *3806734b256c27e41ec2*) DecoratedKey(-8072288379146044663, *30303034314136413343*) _And sometimes there is no problem at all:_ 30303033353144463637 002a31b3b31a1c2f 5d616dd38211ebb5d6ec 444236451388 1388138844463744 30303033353143394343 It's worth to mention that we have got 22 timeout exceptions but number of out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010285#comment-14010285 ] Pavel Yaskevich edited comment on CASSANDRA-7245 at 5/27/14 9:18 PM: - That still leaves us with either 6861 or 5663, can you please generate with only 6861 reverted to rule out 5663? was (Author: xedin): It still leaves us with either 6861 or 5663, can you please generate with only 6861 reverted to rule out 5663? Out-of-Order keys with stress + CQL3 Key: CASSANDRA-7245 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 Project: Cassandra Issue Type: Bug Components: Core Reporter: Pavel Yaskevich Assignee: T Jake Luciani Fix For: 2.1 rc1 We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables). We set up they keys to be 10 bytes in size (default) and population between 1 and 6. Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions: _There is a problem with sorting of normal looking keys:_ 30303039443538353645 30303039443745364242 java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) DecoratedKey(-1767746583617597213, *30303039443437454333*) 0a30303033343933 3734441388343933 java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *3734441388343933*) DecoratedKey(-7565486415339257200, *30303033344639443137*) 30303033354244363031 30303033354133423742 java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) DecoratedKey(-7838239767410066684, *30303033354145344534*) 30303034313442354137 3034313635363334 java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *3034313635363334*) DecoratedKey(-9106177395653818217, *3030303431444238*) 30303035373044373435 30303035373044334631 java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) DecoratedKey(-4296696226469000945, *30303035373132364138*) _And completely different ones:_ 30303041333745373543 7cd045c59a90d7587d8d java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) DecoratedKey(-5146766422778260690, *30303041333943303232*) 3030303332314144 30303033323346343932 java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, *30303033323346343932*) DecoratedKey(5233296131921119414, *53d83e0012287e03*) 30303034314531374431 3806734b256c27e41ec2 java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, *3806734b256c27e41ec2*) DecoratedKey(-8072288379146044663, *30303034314136413343*) _And sometimes there is no problem at all:_ 30303033353144463637 002a31b3b31a1c2f 5d616dd38211ebb5d6ec 444236451388 1388138844463744 30303033353143394343 It's worth to mention that we have got 22 timeout exceptions but number of out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010389#comment-14010389 ] Benedict edited comment on CASSANDRA-7245 at 5/27/14 10:13 PM: --- To answer your question [~xedin], with RF1 it's possible for the surmised bug to be hit without timeouts - if the write stage finishes applying its mutation before the OTC has forwarded the data to the next node, and we're running with CL.ONE (stress default), then the other node can get the corrupted data. If the other node was in the middle of a flush/GC, the OTC could easily be backed up causing this bug to present very commonly. I believe Jason hit this bug much more easily with RF=2, so this would explain everything. FTR, I think it is unlikely 5663 has anything to do with this. was (Author: benedict): To answer your question [~xedin], with RF1 it's possible for the surmised bug to be hit without timeouts - if the write stage finishes applying its mutation before the OTC has forwarded the data to the next node, and we're running with CL.ONE (stress default), then the other node can get the corrupted data. If the other node was in the middle of a flush/GC, the OTC could easily be backed up causing this bug to present very commonly. I believe Jason hit this bug much more easily with RF=2, so this would explain everything. Out-of-Order keys with stress + CQL3 Key: CASSANDRA-7245 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 Project: Cassandra Issue Type: Bug Components: Core Reporter: Pavel Yaskevich Assignee: T Jake Luciani Fix For: 2.1 rc1 We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables). We set up they keys to be 10 bytes in size (default) and population between 1 and 6. Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions: _There is a problem with sorting of normal looking keys:_ 30303039443538353645 30303039443745364242 java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) DecoratedKey(-1767746583617597213, *30303039443437454333*) 0a30303033343933 3734441388343933 java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *3734441388343933*) DecoratedKey(-7565486415339257200, *30303033344639443137*) 30303033354244363031 30303033354133423742 java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) DecoratedKey(-7838239767410066684, *30303033354145344534*) 30303034313442354137 3034313635363334 java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *3034313635363334*) DecoratedKey(-9106177395653818217, *3030303431444238*) 30303035373044373435 30303035373044334631 java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) DecoratedKey(-4296696226469000945, *30303035373132364138*) _And completely different ones:_ 30303041333745373543 7cd045c59a90d7587d8d java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) DecoratedKey(-5146766422778260690, *30303041333943303232*) 3030303332314144 30303033323346343932 java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, *30303033323346343932*) DecoratedKey(5233296131921119414, *53d83e0012287e03*) 30303034314531374431 3806734b256c27e41ec2 java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, *3806734b256c27e41ec2*) DecoratedKey(-8072288379146044663, *30303034314136413343*) _And sometimes there is no problem at all:_ 30303033353144463637 002a31b3b31a1c2f 5d616dd38211ebb5d6ec 444236451388 1388138844463744 30303033353143394343 It's worth to mention that we have got 22 timeout exceptions but number of out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7245) Out-of-Order keys with stress + CQL3
[ https://issues.apache.org/jira/browse/CASSANDRA-7245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999517#comment-13999517 ] Pavel Yaskevich edited comment on CASSANDRA-7245 at 5/16/14 1:41 AM: - Jason told me that they idea was that it could only happen when there is an operation timeout but I'm not sure why almost every of 862 SSTables has such keys, when we only got total of 22 timeouts and there is no batching or anything, also it's not obvious why normal keys are getting sorted out of order too, also if the data is corrupted why the key is of correct size?... was (Author: xedin): Jason told me that they idea was that it could only happen when there is a operation timeout but I'm not sure what almost every of 862 SSTables has such keys when we only got 22 timeouts and there is no batching or something, also it's not obvious why normal keys are getting sorted out of order too, also if the data is corrupted why the key is of correct size?... Out-of-Order keys with stress + CQL3 Key: CASSANDRA-7245 URL: https://issues.apache.org/jira/browse/CASSANDRA-7245 Project: Cassandra Issue Type: Bug Components: Core Reporter: Pavel Yaskevich We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables). We set up they keys to be 10 bytes in size (default) and population between 1 and 6. Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions: _There is a problem with sorting of normal looking keys:_ 30303039443538353645 30303039443745364242 java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, *30303039443745364242*) DecoratedKey(-1767746583617597213, *30303039443437454333*) 0a30303033343933 3734441388343933 java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, *3734441388343933*) DecoratedKey(-7565486415339257200, *30303033344639443137*) 30303033354244363031 30303033354133423742 java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, *30303033354133423742*) DecoratedKey(-7838239767410066684, *30303033354145344534*) 30303034313442354137 3034313635363334 java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, *3034313635363334*) DecoratedKey(-9106177395653818217, *3030303431444238*) 30303035373044373435 30303035373044334631 java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, *30303035373044334631*) DecoratedKey(-4296696226469000945, *30303035373132364138*) _And completely different ones:_ 30303041333745373543 7cd045c59a90d7587d8d java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, *7cd045c59a90d7587d8d*) DecoratedKey(-5146766422778260690, *30303041333943303232*) 3030303332314144 30303033323346343932 java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, *30303033323346343932*) DecoratedKey(5233296131921119414, *53d83e0012287e03*) 30303034314531374431 3806734b256c27e41ec2 java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, *3806734b256c27e41ec2*) DecoratedKey(-8072288379146044663, *30303034314136413343*) _And sometimes there is no problem at all:_ 30303033353144463637 002a31b3b31a1c2f 5d616dd38211ebb5d6ec 444236451388 1388138844463744 30303033353143394343 It's worth to mention that we have got 22 timeout exceptions but number of out-of-order keys is much larger than that. -- This message was sent by Atlassian JIRA (v6.2#6252)
