[jira] [Commented] (CASSANDRA-5925) Race condition in update lightweight transaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13751750#comment-13751750
]
Phil Persad commented on CASSANDRA-5925:
I've updated my test to ensure uniqueness of the tokens and am still
experiencing failures. I've created CASSANDRA-5945 to track the issue so that
the fixes here can be committed.
> Race condition in update lightweight transaction
>
>
> Key: CASSANDRA-5925
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5925
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2.
>Reporter: Phil Persad
>Assignee: Sylvain Lebresne
> Fix For: 2.0
>
> Attachments: 5925.txt, TokenConsumptionTest.java
>
>
> I'm building some tests for a Cassandra PoC. One scenario I need to test is
> consumption of 1 time tokens. These tokens must be consumed exactly once.
> The cluster involved is a 3 node cluster. All queries are run with
> ConsistencyLevel.QUORUM. I'm using the following queries:
> CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 3 };
> CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY
> (tkn));
> INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
> UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
> I use the '[applied]' column in the result set of the update statement to
> determine whether the token has been successfully consumed or if the token is
> being replayed.
> My test involves concurrently executing many sets of 1 insert and 2 update
> statements (using Session#execute on BoundStatemnts) then checking to make
> sure that only one of the updates was applied.
> When I run this test with relatively few iterations (~100) my results are
> what I expect (exactly 1 update succeeds). At ~1000 iterations, I start
> seeing both updates reporting success in 1-2% of cases. While my test is
> running, I see corresponding error entries in the Cassandra log:
> ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main]
> java.lang.NullPointerException
> Thanks.
> Update:
> I'm not sure what's going on with the logging the the dev release. I grabbed
> the rc2 source and built that. The resultant log is a bit more informative:
> ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32)
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26)
> at
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142)
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131)
> at
> org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785)
> at
> org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118)
> at
> org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5945) CAS transactions permitting multiple updates
Phil Persad created CASSANDRA-5945: -- Summary: CAS transactions permitting multiple updates Key: CASSANDRA-5945 URL: https://issues.apache.org/jira/browse/CASSANDRA-5945 Project: Cassandra Issue Type: Bug Components: Core Environment: 3 node Cassandra 2.0.0-rc2 cluster Java driver 1.0.2 Replication factor 3 Quorum consistency Reporter: Phil Persad Attachments: TokenConsumptionTest.java This bug is spawned off CASSANDRA-5925 to track an underlying issue not related to TTLs. To reproduce: Step 1: CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn)); Step 2: INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE); Step 3: UPDATE tkns SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; Step 4: UPDATE tkns SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; Repeat steps 2-4 about 100,000 times. Expectation: For the '[applied]' column in the result sets for steps 3 and 4, exactly one should be true and one should be false. Bug: In a small number of cases (varying from 0.002% to 1%) both updates will report success. See attached unit test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5945) CAS transactions permitting multiple updates
[ https://issues.apache.org/jira/browse/CASSANDRA-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Persad updated CASSANDRA-5945: --- Attachment: TokenConsumptionTest.java > CAS transactions permitting multiple updates > > > Key: CASSANDRA-5945 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5945 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 3 node Cassandra 2.0.0-rc2 cluster > Java driver 1.0.2 > Replication factor 3 > Quorum consistency >Reporter: Phil Persad > Attachments: TokenConsumptionTest.java > > > This bug is spawned off CASSANDRA-5925 to track an underlying issue not > related to TTLs. To reproduce: > Step 1: > CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY > (tkn)); > Step 2: > INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE); > Step 3: > UPDATE tkns SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; > Step 4: > UPDATE tkns SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE; > Repeat steps 2-4 about 100,000 times. > Expectation: > For the '[applied]' column in the result sets for steps 3 and 4, exactly one > should be true and one should be false. > Bug: > In a small number of cases (varying from 0.002% to 1%) both updates will > report success. See attached unit test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5925) Race condition in update lightweight transaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13751653#comment-13751653
]
Phil Persad commented on CASSANDRA-5925:
I take your point and I'll tweak my test to check for duplicate keys. That
being said, even without the use of SecureRandom, I seriously doubt that I'm
getting collisions on a 64 Byte key.
I'll create a new ticket once I've had time to update my test.
> Race condition in update lightweight transaction
>
>
> Key: CASSANDRA-5925
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5925
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2.
>Reporter: Phil Persad
>Assignee: Sylvain Lebresne
> Fix For: 2.0
>
> Attachments: 5925.txt, TokenConsumptionTest.java
>
>
> I'm building some tests for a Cassandra PoC. One scenario I need to test is
> consumption of 1 time tokens. These tokens must be consumed exactly once.
> The cluster involved is a 3 node cluster. All queries are run with
> ConsistencyLevel.QUORUM. I'm using the following queries:
> CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 3 };
> CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY
> (tkn));
> INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
> UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
> I use the '[applied]' column in the result set of the update statement to
> determine whether the token has been successfully consumed or if the token is
> being replayed.
> My test involves concurrently executing many sets of 1 insert and 2 update
> statements (using Session#execute on BoundStatemnts) then checking to make
> sure that only one of the updates was applied.
> When I run this test with relatively few iterations (~100) my results are
> what I expect (exactly 1 update succeeds). At ~1000 iterations, I start
> seeing both updates reporting success in 1-2% of cases. While my test is
> running, I see corresponding error entries in the Cassandra log:
> ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main]
> java.lang.NullPointerException
> Thanks.
> Update:
> I'm not sure what's going on with the logging the the dev release. I grabbed
> the rc2 source and built that. The resultant log is a bit more informative:
> ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32)
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26)
> at
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142)
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131)
> at
> org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785)
> at
> org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118)
> at
> org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5925) Race condition in update lightweight transaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13751553#comment-13751553
]
Phil Persad commented on CASSANDRA-5925:
I applied the patch and it does seem to improve the situation. Unfortunately,
I'm still seeing double consumptions. However, the occurrence has dropped from
1-2% to 0.002%-0.04%. That rate is low enough that it may not show up with the
sample size of 40,000 in the test I posted. Increasing the iterations by 10-20
times should serve to demonstrate the issue.
For the sake of exploration, after I patched I tried running the test both with
and without the TTL in the update statement an saw no appreciable difference in
the number of failures. It looks like there may be yet a third problem.
> Race condition in update lightweight transaction
>
>
> Key: CASSANDRA-5925
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5925
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2.
>Reporter: Phil Persad
>Assignee: Sylvain Lebresne
> Fix For: 2.0
>
> Attachments: 5925.txt, TokenConsumptionTest.java
>
>
> I'm building some tests for a Cassandra PoC. One scenario I need to test is
> consumption of 1 time tokens. These tokens must be consumed exactly once.
> The cluster involved is a 3 node cluster. All queries are run with
> ConsistencyLevel.QUORUM. I'm using the following queries:
> CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 3 };
> CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY
> (tkn));
> INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
> UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
> I use the '[applied]' column in the result set of the update statement to
> determine whether the token has been successfully consumed or if the token is
> being replayed.
> My test involves concurrently executing many sets of 1 insert and 2 update
> statements (using Session#execute on BoundStatemnts) then checking to make
> sure that only one of the updates was applied.
> When I run this test with relatively few iterations (~100) my results are
> what I expect (exactly 1 update succeeds). At ~1000 iterations, I start
> seeing both updates reporting success in 1-2% of cases. While my test is
> running, I see corresponding error entries in the Cassandra log:
> ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main]
> java.lang.NullPointerException
> Thanks.
> Update:
> I'm not sure what's going on with the logging the the dev release. I grabbed
> the rc2 source and built that. The resultant log is a bit more informative:
> ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32)
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26)
> at
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142)
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131)
> at
> org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785)
> at
> org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118)
> at
> org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5925) Race condition in update lightweight transaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13750340#comment-13750340
]
Phil Persad commented on CASSANDRA-5925:
I applied the patch and it did indeed make the NPEs go away. However the
double-consumption of tokens persists. See the attached JUnit test class (the
CassandraClient mentioned in the test is just a simple wrapper that creates a
Cluster and Session and sets a default consistency of QUORUM on every query).
> Race condition in update lightweight transaction
>
>
> Key: CASSANDRA-5925
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5925
> Project: Cassandra
> Issue Type: Bug
> Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2.
>Reporter: Phil Persad
> Attachments: 0001-Write-ballot-with-proposal.txt,
> TokenConsumptionTest.java
>
>
> I'm building some tests for a Cassandra PoC. One scenario I need to test is
> consumption of 1 time tokens. These tokens must be consumed exactly once.
> The cluster involved is a 3 node cluster. All queries are run with
> ConsistencyLevel.QUORUM. I'm using the following queries:
> CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 3 };
> CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY
> (tkn));
> INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
> UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
> I use the '[applied]' column in the result set of the update statement to
> determine whether the token has been successfully consumed or if the token is
> being replayed.
> My test involves concurrently executing many sets of 1 insert and 2 update
> statements (using Session#execute on BoundStatemnts) then checking to make
> sure that only one of the updates was applied.
> When I run this test with relatively few iterations (~100) my results are
> what I expect (exactly 1 update succeeds). At ~1000 iterations, I start
> seeing both updates reporting success in 1-2% of cases. While my test is
> running, I see corresponding error entries in the Cassandra log:
> ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main]
> java.lang.NullPointerException
> Thanks.
> Update:
> I'm not sure what's going on with the logging the the dev release. I grabbed
> the rc2 source and built that. The resultant log is a bit more informative:
> ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32)
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26)
> at
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142)
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131)
> at
> org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785)
> at
> org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118)
> at
> org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5925) Race condition in update lightweight transaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Persad updated CASSANDRA-5925:
---
Attachment: TokenConsumptionTest.java
JUnit test to demonstrate the issue.
> Race condition in update lightweight transaction
>
>
> Key: CASSANDRA-5925
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5925
> Project: Cassandra
> Issue Type: Bug
> Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2.
>Reporter: Phil Persad
> Attachments: 0001-Write-ballot-with-proposal.txt,
> TokenConsumptionTest.java
>
>
> I'm building some tests for a Cassandra PoC. One scenario I need to test is
> consumption of 1 time tokens. These tokens must be consumed exactly once.
> The cluster involved is a 3 node cluster. All queries are run with
> ConsistencyLevel.QUORUM. I'm using the following queries:
> CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 3 };
> CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY
> (tkn));
> INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
> UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
> I use the '[applied]' column in the result set of the update statement to
> determine whether the token has been successfully consumed or if the token is
> being replayed.
> My test involves concurrently executing many sets of 1 insert and 2 update
> statements (using Session#execute on BoundStatemnts) then checking to make
> sure that only one of the updates was applied.
> When I run this test with relatively few iterations (~100) my results are
> what I expect (exactly 1 update succeeds). At ~1000 iterations, I start
> seeing both updates reporting success in 1-2% of cases. While my test is
> running, I see corresponding error entries in the Cassandra log:
> ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main]
> java.lang.NullPointerException
> Thanks.
> Update:
> I'm not sure what's going on with the logging the the dev release. I grabbed
> the rc2 source and built that. The resultant log is a bit more informative:
> ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32)
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26)
> at
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142)
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131)
> at
> org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785)
> at
> org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118)
> at
> org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-5925) Race condition in update lightweight transaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748869#comment-13748869
]
Phil Persad commented on CASSANDRA-5925:
I've updated the ticket with a more complete stack trace.
> Race condition in update lightweight transaction
>
>
> Key: CASSANDRA-5925
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5925
> Project: Cassandra
> Issue Type: Bug
> Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2.
>Reporter: Phil Persad
>
> I'm building some tests for a Cassandra PoC. One scenario I need to test is
> consumption of 1 time tokens. These tokens must be consumed exactly once.
> The cluster involved is a 3 node cluster. All queries are run with
> ConsistencyLevel.QUORUM. I'm using the following queries:
> CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 3 };
> CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY
> (tkn));
> INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
> UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
> I use the '[applied]' column in the result set of the update statement to
> determine whether the token has been successfully consumed or if the token is
> being replayed.
> My test involves concurrently executing many sets of 1 insert and 2 update
> statements (using Session#execute on BoundStatemnts) then checking to make
> sure that only one of the updates was applied.
> When I run this test with relatively few iterations (~100) my results are
> what I expect (exactly 1 update succeeds). At ~1000 iterations, I start
> seeing both updates reporting success in 1-2% of cases. While my test is
> running, I see corresponding error entries in the Cassandra log:
> ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main]
> java.lang.NullPointerException
> Thanks.
> Update:
> I'm not sure what's going on with the logging the the dev release. I grabbed
> the rc2 source and built that. The resultant log is a bit more informative:
> ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main]
> java.lang.NullPointerException
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32)
> at
> org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26)
> at
> org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142)
> at
> org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131)
> at
> org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785)
> at
> org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118)
> at
> org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-5925) Race condition in update lightweight transaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Persad updated CASSANDRA-5925:
---
Description:
I'm building some tests for a Cassandra PoC. One scenario I need to test is
consumption of 1 time tokens. These tokens must be consumed exactly once. The
cluster involved is a 3 node cluster. All queries are run with
ConsistencyLevel.QUORUM. I'm using the following queries:
CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
'SimpleStrategy', 'replication_factor' : 3 };
CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn));
INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
I use the '[applied]' column in the result set of the update statement to
determine whether the token has been successfully consumed or if the token is
being replayed.
My test involves concurrently executing many sets of 1 insert and 2 update
statements (using Session#execute on BoundStatemnts) then checking to make sure
that only one of the updates was applied.
When I run this test with relatively few iterations (~100) my results are what
I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both
updates reporting success in 1-2% of cases. While my test is running, I see
corresponding error entries in the Cassandra log:
ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
java.lang.NullPointerException
ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
java.lang.NullPointerException
ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
java.lang.NullPointerException
ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
java.lang.NullPointerException
ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main]
java.lang.NullPointerException
Thanks.
Update:
I'm not sure what's going on with the logging the the dev release. I grabbed
the rc2 source and built that. The resultant log is a bit more informative:
ERROR 11:53:38,967 Exception in thread Thread[MutationStage:114,5,main]
java.lang.NullPointerException
at
org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:32)
at
org.apache.cassandra.serializers.UUIDSerializer.deserialize(UUIDSerializer.java:26)
at
org.apache.cassandra.db.marshal.AbstractType.compose(AbstractType.java:142)
at
org.apache.cassandra.cql3.UntypedResultSet$Row.getUUID(UntypedResultSet.java:131)
at
org.apache.cassandra.db.SystemKeyspace.loadPaxosState(SystemKeyspace.java:785)
at
org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:118)
at
org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
was:
I'm building some tests for a Cassandra PoC. One scenario I need to test is
consumption of 1 time tokens. These tokens must be consumed exactly once. The
cluster involved is a 3 node cluster. All queries are run with
ConsistencyLevel.QUORUM. I'm using the following queries:
CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
'SimpleStrategy', 'replication_factor' : 3 };
CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn));
INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
I use the '[applied]' column in the result set of the update statement to
determine whether the token has been successfully consumed or if the token is
being replayed.
My test involves concurrently executing many sets of 1 insert and 2 update
statements (using Session#execute on BoundStatemnts) then checking to make sure
that only one of the updates was applied.
When I run this test with relatively few iterations (~100) my results are what
I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both
updates reporting success in 1-2% of cases. While my test is running, I see
corresponding error entries in the Cassandra log:
ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
java.lang.NullPointerException
ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
java.lang.NullPointerException
ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
java.lang.NullPointerException
ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
java.lang.NullPointerException
ERROR 15:34:53,
[jira] [Commented] (CASSANDRA-5925) Race condition in update lightweight transaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13748698#comment-13748698
]
Phil Persad commented on CASSANDRA-5925:
I'm a little puzzled by that myself. What I've posted is exactly what I see in
the log output. The exception seems to be being logged by the
UncaughtExceptionHandler set at CassandraDaemon:129. That's the only place in
the code base where I can find a matching exception message. That call looks
like it should have a full stack trace, but I'm not seeing it in the log.
> Race condition in update lightweight transaction
>
>
> Key: CASSANDRA-5925
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5925
> Project: Cassandra
> Issue Type: Bug
> Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2.
>Reporter: Phil Persad
>
> I'm building some tests for a Cassandra PoC. One scenario I need to test is
> consumption of 1 time tokens. These tokens must be consumed exactly once.
> The cluster involved is a 3 node cluster. All queries are run with
> ConsistencyLevel.QUORUM. I'm using the following queries:
> CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
> 'SimpleStrategy', 'replication_factor' : 3 };
> CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY
> (tkn));
> INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
> UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
> I use the '[applied]' column in the result set of the update statement to
> determine whether the token has been successfully consumed or if the token is
> being replayed.
> My test involves concurrently executing many sets of 1 insert and 2 update
> statements (using Session#execute on BoundStatemnts) then checking to make
> sure that only one of the updates was applied.
> When I run this test with relatively few iterations (~100) my results are
> what I expect (exactly 1 update succeeds). At ~1000 iterations, I start
> seeing both updates reporting success in 1-2% of cases. While my test is
> running, I see corresponding error entries in the Cassandra log:
> ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
> java.lang.NullPointerException
> ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main]
> java.lang.NullPointerException
> Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (CASSANDRA-5925) Race condition in update lightweight transaction
Phil Persad created CASSANDRA-5925:
--
Summary: Race condition in update lightweight transaction
Key: CASSANDRA-5925
URL: https://issues.apache.org/jira/browse/CASSANDRA-5925
Project: Cassandra
Issue Type: Bug
Environment: 3 node Cassandra 2.0.0-rc2 cluster. Java driver 1.0.2.
Reporter: Phil Persad
I'm building some tests for a Cassandra PoC. One scenario I need to test is
consumption of 1 time tokens. These tokens must be consumed exactly once. The
cluster involved is a 3 node cluster. All queries are run with
ConsistencyLevel.QUORUM. I'm using the following queries:
CREATE KEYSPACE IF NOT EXISTS test WITH replication = { 'class' :
'SimpleStrategy', 'replication_factor' : 3 };
CREATE TABLE IF NOT EXISTS tkns (tkn blob, consumed boolean, PRIMARY KEY (tkn));
INSERT INTO tkns (tkn, consumed) VALUES (?,FALSE) USING TTL 30;
UPDATE tkns USING TTL 1 SET consumed = TRUE WHERE tkn = ? IF consumed = FALSE;
I use the '[applied]' column in the result set of the update statement to
determine whether the token has been successfully consumed or if the token is
being replayed.
My test involves concurrently executing many sets of 1 insert and 2 update
statements (using Session#execute on BoundStatemnts) then checking to make sure
that only one of the updates was applied.
When I run this test with relatively few iterations (~100) my results are what
I expect (exactly 1 update succeeds). At ~1000 iterations, I start seeing both
updates reporting success in 1-2% of cases. While my test is running, I see
corresponding error entries in the Cassandra log:
ERROR 15:34:53,583 Exception in thread Thread[MutationStage:522,5,main]
java.lang.NullPointerException
ERROR 15:34:53,584 Exception in thread Thread[MutationStage:474,5,main]
java.lang.NullPointerException
ERROR 15:34:53,584 Exception in thread Thread[MutationStage:536,5,main]
java.lang.NullPointerException
ERROR 15:34:53,729 Exception in thread Thread[MutationStage:480,5,main]
java.lang.NullPointerException
ERROR 15:34:53,729 Exception in thread Thread[MutationStage:534,5,main]
java.lang.NullPointerException
Thanks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
