[
https://issues.apache.org/jira/browse/CASSANDRA-16453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-16453:
------------------------------------
Description:
There’s an NPE in Slice#make on RT + partition deletion reconciliation.
Minimal repro:
{code:java}
try (Cluster cluster = init(builder().withNodes(3).start()))
{
cluster.schemaChange(withKeyspace("CREATE TABLE
distributed_test_keyspace.table_0 (pk0 bigint,ck0 bigint,regular0
bigint,regular1 bigint,regular2 bigint, PRIMARY KEY (pk0, ck0)) WITH
CLUSTERING ORDER BY (ck0 ASC);"));
long pk = 0L;
cluster.coordinator(1).execute("DELETE FROM
distributed_test_keyspace.table_0 USING TIMESTAMP 100230 WHERE pk0=? AND
ck0>?;", ConsistencyLevel.ALL, pk, 2L);
cluster.get(3).executeInternal("DELETE FROM
distributed_test_keyspace.table_0 USING TIMESTAMP 100230 WHERE pk0=?;", pk);
cluster.coordinator(1).execute("SELECT * FROM
distributed_test_keyspace.table_0 WHERE pk0=? AND ck0>=? AND ck0<?;",
ConsistencyLevel.ALL, pk,
1L, 3L);
}
{code}
Details:
{code:java}
java.lang.AssertionError: Error merging RTs on
distributed_test_keyspace.table_0: merged=null, versions=[Marker
EXCL_START_BOUND(2)@100230/1613500432, Marker
EXCL_START_BOUND(2)@100230/1613500432, null],
sources={[Full(/127.0.0.1:7012,(-3074457345618258603,3074457345618258601]),
Full(/127.0.0.2:7012,(-3074457345618258603,3074457345618258601]),
Full(/127.0.0.3:7012,(-3074457345618258603,3074457345618258601])]}, debug info:
/127.0.0.1:7012 => [distributed_test_keyspace.table_0] key=0
partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
columns=[[] | [regular0 regular1 regular2]] repaired_digest=
repaired_digest_conclusive==true
Marker EXCL_START_BOUND(2)@100230/1613500432
Marker EXCL_END_BOUND(3)@100230/1613500432,
/127.0.0.2:7012 => [distributed_test_keyspace.table_0] key=0
partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
columns=[[] | [regular0 regular1 regular2]] repaired_digest=
repaired_digest_conclusive==true
Marker EXCL_START_BOUND(2)@100230/1613500432
Marker EXCL_END_BOUND(3)@100230/1613500432,
/127.0.0.3:7012 => [distributed_test_keyspace.table_0] key=0
partition_deletion=deletedAt=100230, localDeletion=1613500432 columns=[[] |
[regular0 regular1 regular2]] repaired_digest= repaired_digest_conclusive==true
{code}
Exception:
{code:java}
java.lang.NullPointerException
at org.apache.cassandra.db.Slice.make(Slice.java:74)
at
org.apache.cassandra.service.reads.repair.RowIteratorMergeListener.closeOpenMarker(RowIteratorMergeListener.java:351)
at
org.apache.cassandra.service.reads.repair.RowIteratorMergeListener.onMergedRangeTombstoneMarkers(RowIteratorMergeListener.java:315)
at
org.apache.cassandra.service.reads.DataResolver$2$1.onMergedRangeTombstoneMarkers(DataResolver.java:378)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:592)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:541)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:219)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:523)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:391)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
at
org.apache.cassandra.db.transform.FilteredRows.isEmpty(FilteredRows.java:50)
at
org.apache.cassandra.db.transform.EmptyPartitionsDiscarder.applyToPartition(EmptyPartitionsDiscarder.java:27)
at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:97)
at
org.apache.cassandra.service.StorageProxy$6.hasNext(StorageProxy.java:1908)
at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:93)
at
org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:777)
at
org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:425)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:402)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:250)
at
org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithPagingWithResult$2(Coordinator.java:162)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
{code}
This behaviour is new to 4.0 and was introduced in [CASSANDRA-15369]. The
difference with 3.0 is that in 3.0 {{RangeTombstoneMarker merged}} would be
{{null}}, so we would never be hitting the code path where some of the sources
is opening/closing marker, and instead will fall through to opening/closing of
deletions below. I've checked code in 15369 and it looks like this condition is
a new edge case in otherwise correct code. Since the goal was to avoid
generating range tombstones on ties with partition deletion, fix for this issue
is also consistent with that goal.
was:
There’s an NPE in Slice#make on RT + partition deletion reconciliation.
Minimal repro:
{code:java}
try (Cluster cluster = init(builder().withNodes(3).start()))
{
cluster.schemaChange(withKeyspace("CREATE TABLE
distributed_test_keyspace.table_0 (pk0 bigint,ck0 bigint,regular0
bigint,regular1 bigint,regular2 bigint, PRIMARY KEY (pk0, ck0)) WITH
CLUSTERING ORDER BY (ck0 ASC);"));
long pk = 0L;
cluster.coordinator(1).execute("DELETE FROM
distributed_test_keyspace.table_0 USING TIMESTAMP 100230 WHERE pk0=? AND
ck0>?;", ConsistencyLevel.ALL, pk, 2L);
cluster.get(3).executeInternal("DELETE FROM
distributed_test_keyspace.table_0 USING TIMESTAMP 100230 WHERE pk0=?;", pk);
cluster.coordinator(1).execute("SELECT * FROM
distributed_test_keyspace.table_0 WHERE pk0=? AND ck0>=? AND ck0<?;",
ConsistencyLevel.ALL, pk,
1L, 3L);
}
{code}
Details:
{code:java}
java.lang.AssertionError: Error merging RTs on
distributed_test_keyspace.table_0: merged=null, versions=[Marker
EXCL_START_BOUND(2)@100230/1613500432, Marker
EXCL_START_BOUND(2)@100230/1613500432, null],
sources={[Full(/127.0.0.1:7012,(-3074457345618258603,3074457345618258601]),
Full(/127.0.0.2:7012,(-3074457345618258603,3074457345618258601]),
Full(/127.0.0.3:7012,(-3074457345618258603,3074457345618258601])]}, debug info:
/127.0.0.1:7012 => [distributed_test_keyspace.table_0] key=0
partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
columns=[[] | [regular0 regular1 regular2]] repaired_digest=
repaired_digest_conclusive==true
Marker EXCL_START_BOUND(2)@100230/1613500432
Marker EXCL_END_BOUND(3)@100230/1613500432,
/127.0.0.2:7012 => [distributed_test_keyspace.table_0] key=0
partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
columns=[[] | [regular0 regular1 regular2]] repaired_digest=
repaired_digest_conclusive==true
Marker EXCL_START_BOUND(2)@100230/1613500432
Marker EXCL_END_BOUND(3)@100230/1613500432,
/127.0.0.3:7012 => [distributed_test_keyspace.table_0] key=0
partition_deletion=deletedAt=100230, localDeletion=1613500432 columns=[[] |
[regular0 regular1 regular2]] repaired_digest= repaired_digest_conclusive==true
{code}
Exception:
{code:java}
java.lang.NullPointerException
at org.apache.cassandra.db.Slice.make(Slice.java:74)
at
org.apache.cassandra.service.reads.repair.RowIteratorMergeListener.closeOpenMarker(RowIteratorMergeListener.java:351)
at
org.apache.cassandra.service.reads.repair.RowIteratorMergeListener.onMergedRangeTombstoneMarkers(RowIteratorMergeListener.java:315)
at
org.apache.cassandra.service.reads.DataResolver$2$1.onMergedRangeTombstoneMarkers(DataResolver.java:378)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:592)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:541)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:219)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:523)
at
org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:391)
at
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
at
org.apache.cassandra.db.transform.FilteredRows.isEmpty(FilteredRows.java:50)
at
org.apache.cassandra.db.transform.EmptyPartitionsDiscarder.applyToPartition(EmptyPartitionsDiscarder.java:27)
at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:97)
at
org.apache.cassandra.service.StorageProxy$6.hasNext(StorageProxy.java:1908)
at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:93)
at
org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:777)
at
org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:425)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:402)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:250)
at
org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithPagingWithResult$2(Coordinator.java:162)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
{code}
> NPE in Slice#make on RT + partition deletion reconciliation on timestamp tie
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-16453
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16453
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination
> Reporter: Alex Petrov
> Assignee: Alex Petrov
> Priority: Normal
> Fix For: 4.0-rc
>
>
> There’s an NPE in Slice#make on RT + partition deletion reconciliation.
> Minimal repro:
> {code:java}
> try (Cluster cluster = init(builder().withNodes(3).start()))
> {
> cluster.schemaChange(withKeyspace("CREATE TABLE
> distributed_test_keyspace.table_0 (pk0 bigint,ck0 bigint,regular0
> bigint,regular1 bigint,regular2 bigint, PRIMARY KEY (pk0, ck0)) WITH
> CLUSTERING ORDER BY (ck0 ASC);"));
> long pk = 0L;
> cluster.coordinator(1).execute("DELETE FROM
> distributed_test_keyspace.table_0 USING TIMESTAMP 100230 WHERE pk0=? AND
> ck0>?;", ConsistencyLevel.ALL, pk, 2L);
> cluster.get(3).executeInternal("DELETE FROM
> distributed_test_keyspace.table_0 USING TIMESTAMP 100230 WHERE pk0=?;", pk);
> cluster.coordinator(1).execute("SELECT * FROM
> distributed_test_keyspace.table_0 WHERE pk0=? AND ck0>=? AND ck0<?;",
> ConsistencyLevel.ALL,
> pk, 1L, 3L);
> }
> {code}
> Details:
> {code:java}
> java.lang.AssertionError: Error merging RTs on
> distributed_test_keyspace.table_0: merged=null, versions=[Marker
> EXCL_START_BOUND(2)@100230/1613500432, Marker
> EXCL_START_BOUND(2)@100230/1613500432, null],
> sources={[Full(/127.0.0.1:7012,(-3074457345618258603,3074457345618258601]),
> Full(/127.0.0.2:7012,(-3074457345618258603,3074457345618258601]),
> Full(/127.0.0.3:7012,(-3074457345618258603,3074457345618258601])]}, debug
> info:
> /127.0.0.1:7012 => [distributed_test_keyspace.table_0] key=0
> partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
> columns=[[] | [regular0 regular1 regular2]] repaired_digest=
> repaired_digest_conclusive==true
> Marker EXCL_START_BOUND(2)@100230/1613500432
> Marker EXCL_END_BOUND(3)@100230/1613500432,
> /127.0.0.2:7012 => [distributed_test_keyspace.table_0] key=0
> partition_deletion=deletedAt=-9223372036854775808, localDeletion=2147483647
> columns=[[] | [regular0 regular1 regular2]] repaired_digest=
> repaired_digest_conclusive==true
> Marker EXCL_START_BOUND(2)@100230/1613500432
> Marker EXCL_END_BOUND(3)@100230/1613500432,
> /127.0.0.3:7012 => [distributed_test_keyspace.table_0] key=0
> partition_deletion=deletedAt=100230, localDeletion=1613500432 columns=[[] |
> [regular0 regular1 regular2]] repaired_digest=
> repaired_digest_conclusive==true
> {code}
> Exception:
> {code:java}
> java.lang.NullPointerException
> at org.apache.cassandra.db.Slice.make(Slice.java:74)
> at
> org.apache.cassandra.service.reads.repair.RowIteratorMergeListener.closeOpenMarker(RowIteratorMergeListener.java:351)
> at
> org.apache.cassandra.service.reads.repair.RowIteratorMergeListener.onMergedRangeTombstoneMarkers(RowIteratorMergeListener.java:315)
> at
> org.apache.cassandra.service.reads.DataResolver$2$1.onMergedRangeTombstoneMarkers(DataResolver.java:378)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:592)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator$MergeReducer.getReduced(UnfilteredRowIterators.java:541)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:219)
> at
> org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:523)
> at
> org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:391)
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:133)
> at
> org.apache.cassandra.db.transform.FilteredRows.isEmpty(FilteredRows.java:50)
> at
> org.apache.cassandra.db.transform.EmptyPartitionsDiscarder.applyToPartition(EmptyPartitionsDiscarder.java:27)
> at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:97)
> at
> org.apache.cassandra.service.StorageProxy$6.hasNext(StorageProxy.java:1908)
> at
> org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:93)
> at
> org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:777)
> at
> org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:425)
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:402)
> at
> org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:250)
> at
> org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithPagingWithResult$2(Coordinator.java:162)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> This behaviour is new to 4.0 and was introduced in [CASSANDRA-15369]. The
> difference with 3.0 is that in 3.0 {{RangeTombstoneMarker merged}} would be
> {{null}}, so we would never be hitting the code path where some of the
> sources is opening/closing marker, and instead will fall through to
> opening/closing of deletions below. I've checked code in 15369 and it looks
> like this condition is a new edge case in otherwise correct code. Since the
> goal was to avoid generating range tombstones on ties with partition
> deletion, fix for this issue is also consistent with that goal.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
