[jira] [Updated] (CASSANDRA-5925) Race condition in update lightweight transaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hanna updated CASSANDRA-5925: Labels: LWT (was: ) > Race condition in update lightweight transaction > > > Key: CASSANDRA-5925 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5925 > Project: Cassandra > Issue Type: Bug > Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2. >Reporter: Phil Persad >Assignee: Sylvain Lebresne >Priority: Major > Labels: LWT > Fix For: 2.0.0 > > Attachments: 5925.txt, TokenConsumptionTest.java > > > I'm building some tests for a Cassandra PoC. One scenario I need to test is > consumption of 1 time tokens. These tokens must be consumed exactly once. > The cluster involved is a 3 node cluster. All queries are run with > ConsistencyLevel.QUORUM. I'm using the following queries: > CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' : > 'SimpleStrategy', 'replication_factor' : 3 }; > CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY > (tkn)); > INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30; > UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; > I use the '[applied]' column in the result set of the update statement to > determine whether the token has been successfully consumed or if the token is > being replayed. > My test involves concurrently executing many sets of 1 insert and 2 update > statements (using Session#execute on BoundStatemnts) then checking to make > sure that only one of the updates was applied. > When I run this test with relatively few iterations (~100) my results are > what I expect (exactly 1 update succeeds). At ~1000 iterations, I start > seeing both updates reporting success in 1-2% of cases. While my test is > running, I see corresponding error entries in the Cassandra log: > ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main] > java.lang.NullPointerException > ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main] > java.lang.NullPointerException > ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main] > java.lang.NullPointerException > ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main] > java.lang.NullPointerException > ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main] > java.lang.NullPointerException > Thanks. > Update: > I'm not sure what's going on with the logging the the dev release. I grabbed > the rc2 source and built that. The resultant log is a bit more informative: > ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main] > java.lang.NullPointerException > at > org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32) > at > org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26) > at > org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142) > at > org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131) > at > org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785) > at > org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118) > at > org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34) > at > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-5925) Race condition in update lightweight transaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-5925: Attachment: (was: 0001-Write-ballot-with-proposal.txt) Race condition in update lightweight transaction Key: CASSANDRA-5925 URL: https://issues.apache.org/jira/browse/CASSANDRA-5925 Project: Cassandra Issue Type: Bug Components: Core Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2. Reporter: Phil Persad Assignee: Sylvain Lebresne Fix For: 2.0.1 Attachments: TokenConsumptionTest.java I'm building some tests for a Cassandra PoC. One scenario I need to test is consumption of 1 time tokens. These tokens must be consumed exactly once. The cluster involved is a 3 node cluster. All queries are run with ConsistencyLevel.QUORUM. I'm using the following queries: CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }; CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn)); INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30; UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; I use the '[applied]' column in the result set of the update statement to determine whether the token has been successfully consumed or if the token is being replayed. My test involves concurrently executing many sets of 1 insert and 2 update statements (using Session#execute on BoundStatemnts) then checking to make sure that only one of the updates was applied. When I run this test with relatively few iterations (~100) my results are what I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both updates reporting success in 1-2% of cases. While my test is running, I see corresponding error entries in the Cassandra log: ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main] java.lang.NullPointerException Thanks. Update: I'm not sure what's going on with the logging the the dev release. I grabbed the rc2 source and built that. The resultant log is a bit more informative: ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main] java.lang.NullPointerException at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32) at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26) at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142) at org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131) at org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785) at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118) at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5925) Race condition in update lightweight transaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-5925: Attachment: 5925.txt Thanks for the test. There is indeed 2 problems: # the NPE while loading the paxos state. Contrarily to my first reading, the fact that savePaxosProposal doesn't save the inProgress ballot is not only a problem due to expiration. We may call that method if our own inProgress is older than the proposal, so given a node received a proposal without having seen the prepare first (and without having a previous state), we'd end up with a state where just the 'proposal' column is set. Note that technically, I don't think it breaks Paxos not to update the ballot when saving a proposal so we could just fix loadPaxosState to not NPE in that case, but it feels saner/simpler to me to write the proposal ballot when we save a proposal value. # the reason Phil test fails is different however. The problem is that when we were building the 'expected' ColumnFamily for the cas() call in CQL3, we were using the full parameters of the statement, including (which was the problem) the TTL. In that test, the TTL is 1 second, so it's possible (and even not that unlikely since we only have up to 1 second accuracy internally) that when we were comparing 'expected' to 'current' the expected column was considered deleted. So, if when the 2nd update to a row was processed the first one had expired (again, not unlikely given the 1 second ttl), the 2nd CAS update was comparing both a deleted 'expected' and a deleted 'current', thus succeeding. So anyway, the fix is to not use the TTL for the conditions in ModificationStatement. So attaching patch that fix both issue. Race condition in update lightweight transaction Key: CASSANDRA-5925 URL: https://issues.apache.org/jira/browse/CASSANDRA-5925 Project: Cassandra Issue Type: Bug Components: Core Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2. Reporter: Phil Persad Assignee: Sylvain Lebresne Fix For: 2.0.1 Attachments: 5925.txt, TokenConsumptionTest.java I'm building some tests for a Cassandra PoC. One scenario I need to test is consumption of 1 time tokens. These tokens must be consumed exactly once. The cluster involved is a 3 node cluster. All queries are run with ConsistencyLevel.QUORUM. I'm using the following queries: CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }; CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn)); INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30; UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; I use the '[applied]' column in the result set of the update statement to determine whether the token has been successfully consumed or if the token is being replayed. My test involves concurrently executing many sets of 1 insert and 2 update statements (using Session#execute on BoundStatemnts) then checking to make sure that only one of the updates was applied. When I run this test with relatively few iterations (~100) my results are what I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both updates reporting success in 1-2% of cases. While my test is running, I see corresponding error entries in the Cassandra log: ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main] java.lang.NullPointerException Thanks. Update: I'm not sure what's going on with the logging the the dev release. I grabbed the rc2 source and built that. The resultant log is a bit more informative: ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main] java.lang.NullPointerException at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32) at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26) at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142) at org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131) at org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785) at
[jira] [Updated] (CASSANDRA-5925) Race condition in update lightweight transaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne updated CASSANDRA-5925: Attachment: 0001-Write-ballot-with-proposal.txt Thanks for the full stack. Not completely sure what triggers that NPE however. I do see one scenario where the paxos state on-disk could not be empty but the there's no in_progress_ballot column (hence triggering that NPE): since savePaxosPromise only write the proposal and not the in_progress_ballot, and since we set TTL on inserts, it sounds possible to end up in a case where a paxos row only has the proposal column but everything else has expired. I do am attaching a simple patch to always write in_progress_ballot to avoid that, but given that paxos TTL is at least 3 hours, I doubt that's the scenario you are running into in your test. So not sure what's going on ([~jbellis], if you have a brilliant idea...). Phil, if you could check if the attached patch fixes it by any chance, that could be helpful. If it doesn't (likely), would you be able to provide a simple test script that reproduce this? Race condition in update lightweight transaction Key: CASSANDRA-5925 URL: https://issues.apache.org/jira/browse/CASSANDRA-5925 Project: Cassandra Issue Type: Bug Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2. Reporter: Phil Persad Attachments: 0001-Write-ballot-with-proposal.txt I'm building some tests for a Cassandra PoC. One scenario I need to test is consumption of 1 time tokens. These tokens must be consumed exactly once. The cluster involved is a 3 node cluster. All queries are run with ConsistencyLevel.QUORUM. I'm using the following queries: CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }; CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn)); INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30; UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; I use the '[applied]' column in the result set of the update statement to determine whether the token has been successfully consumed or if the token is being replayed. My test involves concurrently executing many sets of 1 insert and 2 update statements (using Session#execute on BoundStatemnts) then checking to make sure that only one of the updates was applied. When I run this test with relatively few iterations (~100) my results are what I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both updates reporting success in 1-2% of cases. While my test is running, I see corresponding error entries in the Cassandra log: ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main] java.lang.NullPointerException Thanks. Update: I'm not sure what's going on with the logging the the dev release. I grabbed the rc2 source and built that. The resultant log is a bit more informative: ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main] java.lang.NullPointerException at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32) at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26) at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142) at org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131) at org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785) at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118) at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5925) Race condition in update lightweight transaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Persad updated CASSANDRA-5925: --- Attachment: TokenConsumptionTest.java JUnit test to demonstrate the issue. Race condition in update lightweight transaction Key: CASSANDRA-5925 URL: https://issues.apache.org/jira/browse/CASSANDRA-5925 Project: Cassandra Issue Type: Bug Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2. Reporter: Phil Persad Attachments: 0001-Write-ballot-with-proposal.txt, TokenConsumptionTest.java I'm building some tests for a Cassandra PoC. One scenario I need to test is consumption of 1 time tokens. These tokens must be consumed exactly once. The cluster involved is a 3 node cluster. All queries are run with ConsistencyLevel.QUORUM. I'm using the following queries: CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }; CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn)); INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30; UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; I use the '[applied]' column in the result set of the update statement to determine whether the token has been successfully consumed or if the token is being replayed. My test involves concurrently executing many sets of 1 insert and 2 update statements (using Session#execute on BoundStatemnts) then checking to make sure that only one of the updates was applied. When I run this test with relatively few iterations (~100) my results are what I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both updates reporting success in 1-2% of cases. While my test is running, I see corresponding error entries in the Cassandra log: ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main] java.lang.NullPointerException Thanks. Update: I'm not sure what's going on with the logging the the dev release. I grabbed the rc2 source and built that. The resultant log is a bit more informative: ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main] java.lang.NullPointerException at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32) at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26) at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142) at org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131) at org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785) at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118) at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5925) Race condition in update lightweight transaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-5925: -- Reviewer: jbellis Component/s: Core Reproduced In: 2.0 rc2 Fix Version/s: 2.0.1 Assignee: Sylvain Lebresne Race condition in update lightweight transaction Key: CASSANDRA-5925 URL: https://issues.apache.org/jira/browse/CASSANDRA-5925 Project: Cassandra Issue Type: Bug Components: Core Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2. Reporter: Phil Persad Assignee: Sylvain Lebresne Fix For: 2.0.1 Attachments: 0001-Write-ballot-with-proposal.txt, TokenConsumptionTest.java I'm building some tests for a Cassandra PoC. One scenario I need to test is consumption of 1 time tokens. These tokens must be consumed exactly once. The cluster involved is a 3 node cluster. All queries are run with ConsistencyLevel.QUORUM. I'm using the following queries: CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }; CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn)); INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30; UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; I use the '[applied]' column in the result set of the update statement to determine whether the token has been successfully consumed or if the token is being replayed. My test involves concurrently executing many sets of 1 insert and 2 update statements (using Session#execute on BoundStatemnts) then checking to make sure that only one of the updates was applied. When I run this test with relatively few iterations (~100) my results are what I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both updates reporting success in 1-2% of cases. While my test is running, I see corresponding error entries in the Cassandra log: ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main] java.lang.NullPointerException Thanks. Update: I'm not sure what's going on with the logging the the dev release. I grabbed the rc2 source and built that. The resultant log is a bit more informative: ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main] java.lang.NullPointerException at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32) at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26) at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142) at org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131) at org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785) at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118) at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5925) Race condition in update lightweight transaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Persad updated CASSANDRA-5925: --- Description: I'm building some tests for a Cassandra PoC. One scenario I need to test is consumption of 1 time tokens. These tokens must be consumed exactly once. The cluster involved is a 3 node cluster. All queries are run with ConsistencyLevel.QUORUM. I'm using the following queries: CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }; CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn)); INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30; UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; I use the '[applied]' column in the result set of the update statement to determine whether the token has been successfully consumed or if the token is being replayed. My test involves concurrently executing many sets of 1 insert and 2 update statements (using Session#execute on BoundStatemnts) then checking to make sure that only one of the updates was applied. When I run this test with relatively few iterations (~100) my results are what I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both updates reporting success in 1-2% of cases. While my test is running, I see corresponding error entries in the Cassandra log: ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main] java.lang.NullPointerException Thanks. Update: I'm not sure what's going on with the logging the the dev release. I grabbed the rc2 source and built that. The resultant log is a bit more informative: ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main] java.lang.NullPointerException at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32) at org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26) at org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142) at org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131) at org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785) at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118) at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) was: I'm building some tests for a Cassandra PoC. One scenario I need to test is consumption of 1 time tokens. These tokens must be consumed exactly once. The cluster involved is a 3 node cluster. All queries are run with ConsistencyLevel.QUORUM. I'm using the following queries: CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 3 }; CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn)); INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30; UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; I use the '[applied]' column in the result set of the update statement to determine whether the token has been successfully consumed or if the token is being replayed. My test involves concurrently executing many sets of 1 insert and 2 update statements (using Session#execute on BoundStatemnts) then checking to make sure that only one of the updates was applied. When I run this test with relatively few iterations (~100) my results are what I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both updates reporting success in 1-2% of cases. While my test is running, I see corresponding error entries in the Cassandra log: ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main] java.lang.NullPointerException ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main] java.lang.NullPointerException ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main] java.lang.NullPointerException ERROR