[
https://issues.apache.org/jira/browse/CASSANDRA-21710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118428#comment-18118428
]
Dmitry Konstantinov edited comment on CASSANDRA-21710 at 9/23/26 5:22 PM:
--------------------------------------------------------------------------
Error handling logic from BaseIterator.close (captured it just for a better
context):
{code}
"node1_isolatedExecutor:1@14827" daemon prio=5 tid=0x69 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:118)
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.onPartitionClose(RTBoundValidator.java:113)
at
org.apache.cassandra.db.transform.BaseRows.runOnClose(BaseRows.java:95)
at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:96)
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:347)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:321)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:294)
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:108)
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:461)
at
org.apache.cassandra.db.ReadCommand.createLocalObjectResponse(ReadCommand.java:466)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2762)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:3166)
at
org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:168)
at
org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:218)
at
org.apache.cassandra.concurrent.Stage.maybeExecuteImmediately(Stage.java:130)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:168)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.makeFullDataRequests(AbstractReadExecutor.java:123)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.executeAsync(AbstractReadExecutor.java:185)
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:2679)
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:2562)
at
org.apache.cassandra.service.StorageProxy.dispatchReadWithRetryOnDifferentSystem(StorageProxy.java:2465)
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:2194)
at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1439)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:525)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:430)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:160)
at
org.apache.cassandra.distributed.impl.CoordinatorHelper.unsafeExecuteInternal(CoordinatorHelper.java:70)
at
org.apache.cassandra.distributed.impl.CoordinatorHelper.unsafeExecuteInternal(CoordinatorHelper.java:48)
at
org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:127)
at
org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:64)
at
org.apache.cassandra.distributed.impl.Coordinator$$Lambda$4523.1485733378.call(Unknown
Source:-1)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:829)
at
org.apache.cassandra.concurrent.CassandraThread.run(CassandraThread.java:169)
{code}
and QueryCancelledException with suppressed IllegalStateException:
{code}
g.apache.cassandra.exceptions.QueryCancelledException: Query cancelled for
taking too long: SELECT * FROM distributed_test_keyspace.tbl WHERE id = 1 AND
ck1 = 77 ALLOW FILTERING
at
org.apache.cassandra.db.ReadCommand$QueryCancellationChecker.maybeCancel(ReadCommand.java:905)
at
org.apache.cassandra.db.ReadCommand$QueryCancellationChecker.applyToRow(ReadCommand.java:887)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:145)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:137)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serializeWithoutKey(UnfilteredRowIteratorSerializer.java:167)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:127)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:108)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:96)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:349)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:321)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:294)
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:108)
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:461)
at
org.apache.cassandra.db.ReadCommand.createLocalObjectResponse(ReadCommand.java:466)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2762)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:3166)
at
org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:168)
at
org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:218)
at
org.apache.cassandra.concurrent.Stage.maybeExecuteImmediately(Stage.java:130)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:168)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.makeFullDataRequests(AbstractReadExecutor.java:123)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.executeAsync(AbstractReadExecutor.java:185)
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:2679)
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:2562)
at
org.apache.cassandra.service.StorageProxy.dispatchReadWithRetryOnDifferentSystem(StorageProxy.java:2465)
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:2194)
at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1439)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:525)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:430)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:160)
at
org.apache.cassandra.distributed.impl.CoordinatorHelper.unsafeExecuteInternal(CoordinatorHelper.java:70)
at
org.apache.cassandra.distributed.impl.CoordinatorHelper.unsafeExecuteInternal(CoordinatorHelper.java:48)
at
org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:127)
at
org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:64)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
at
org.apache.cassandra.concurrent.CassandraThread.run(CassandraThread.java:169)
Suppressed: java.lang.IllegalStateException: PROCESSED
UnfilteredRowIterator for distributed_test_keyspace.tbl (key: 1 omdt:
[deletedAt=5, localDeletion=1790180956]) has an illegal Range Tombstone bounds
sequence: expected all Range Tombstones to be closed, but the last one is open
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:118)
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.onPartitionClose(RTBoundValidator.java:113)
at
org.apache.cassandra.db.transform.BaseRows.runOnClose(BaseRows.java:95)
at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:96)
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:347)
... 32 more
{code}
was (Author: dnk):
Original error handling logic was in BaseIterator.close:
{code}
"node1_isolatedExecutor:1@14827" daemon prio=5 tid=0x69 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:118)
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.onPartitionClose(RTBoundValidator.java:113)
at
org.apache.cassandra.db.transform.BaseRows.runOnClose(BaseRows.java:95)
at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:96)
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:347)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:321)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:294)
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:108)
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:461)
at
org.apache.cassandra.db.ReadCommand.createLocalObjectResponse(ReadCommand.java:466)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2762)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:3166)
at
org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:168)
at
org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:218)
at
org.apache.cassandra.concurrent.Stage.maybeExecuteImmediately(Stage.java:130)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:168)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.makeFullDataRequests(AbstractReadExecutor.java:123)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.executeAsync(AbstractReadExecutor.java:185)
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:2679)
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:2562)
at
org.apache.cassandra.service.StorageProxy.dispatchReadWithRetryOnDifferentSystem(StorageProxy.java:2465)
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:2194)
at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1439)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:525)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:430)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:160)
at
org.apache.cassandra.distributed.impl.CoordinatorHelper.unsafeExecuteInternal(CoordinatorHelper.java:70)
at
org.apache.cassandra.distributed.impl.CoordinatorHelper.unsafeExecuteInternal(CoordinatorHelper.java:48)
at
org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:127)
at
org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:64)
at
org.apache.cassandra.distributed.impl.Coordinator$$Lambda$4523.1485733378.call(Unknown
Source:-1)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:829)
at
org.apache.cassandra.concurrent.CassandraThread.run(CassandraThread.java:169)
{code}
and QueryCancelledException with suppressed IllegalStateException:
{code}
g.apache.cassandra.exceptions.QueryCancelledException: Query cancelled for
taking too long: SELECT * FROM distributed_test_keyspace.tbl WHERE id = 1 AND
ck1 = 77 ALLOW FILTERING
at
org.apache.cassandra.db.ReadCommand$QueryCancellationChecker.maybeCancel(ReadCommand.java:905)
at
org.apache.cassandra.db.ReadCommand$QueryCancellationChecker.applyToRow(ReadCommand.java:887)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:145)
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:137)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serializeWithoutKey(UnfilteredRowIteratorSerializer.java:167)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:127)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:108)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:96)
at
org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:89)
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:349)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:321)
at
org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:294)
at
org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:108)
at
org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:461)
at
org.apache.cassandra.db.ReadCommand.createLocalObjectResponse(ReadCommand.java:466)
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2762)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:3166)
at
org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:168)
at
org.apache.cassandra.concurrent.SEPExecutor.maybeExecuteImmediately(SEPExecutor.java:218)
at
org.apache.cassandra.concurrent.Stage.maybeExecuteImmediately(Stage.java:130)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.makeRequests(AbstractReadExecutor.java:168)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.makeFullDataRequests(AbstractReadExecutor.java:123)
at
org.apache.cassandra.service.reads.AbstractReadExecutor.executeAsync(AbstractReadExecutor.java:185)
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:2679)
at
org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:2562)
at
org.apache.cassandra.service.StorageProxy.dispatchReadWithRetryOnDifferentSystem(StorageProxy.java:2465)
at
org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:2194)
at
org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1439)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:525)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:430)
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:160)
at
org.apache.cassandra.distributed.impl.CoordinatorHelper.unsafeExecuteInternal(CoordinatorHelper.java:70)
at
org.apache.cassandra.distributed.impl.CoordinatorHelper.unsafeExecuteInternal(CoordinatorHelper.java:48)
at
org.apache.cassandra.distributed.impl.Coordinator.unsafeExecuteInternal(Coordinator.java:127)
at
org.apache.cassandra.distributed.impl.Coordinator.lambda$executeWithResult$0(Coordinator.java:64)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:829)
at
org.apache.cassandra.concurrent.CassandraThread.run(CassandraThread.java:169)
Suppressed: java.lang.IllegalStateException: PROCESSED
UnfilteredRowIterator for distributed_test_keyspace.tbl (key: 1 omdt:
[deletedAt=5, localDeletion=1790180956]) has an illegal Range Tombstone bounds
sequence: expected all Range Tombstones to be closed, but the last one is open
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:118)
at
org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.onPartitionClose(RTBoundValidator.java:113)
at
org.apache.cassandra.db.transform.BaseRows.runOnClose(BaseRows.java:95)
at
org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:96)
at
org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:347)
... 32 more
{code}
> Preserve the original exception when an in-memory local response fails to
> close
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-21710
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21710
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Other
> Reporter: Francisco Guerrero
> Assignee: Francisco Guerrero
> Priority: Normal
> Fix For: 6.0-alpha3, 7.0
>
>
> {{InMemoryDataResponse.build}} closes the row iterator in a finally block, so
> a failure from {{iter.close()}} will shadow the already in-flight exception.
> In the test where I noticed this issue, a read aborted by
> {{ReadCommand.QueryCancellationChecker}} leaves a partition that can end on
> an open range tombstone bound, which the {{PROCESSED RTBoundValidator}}
> rejects on close. The client is seeing a {{ReadFailureException}} instead of
> a {{ReadTimeoutException}}. We should instead accumulate the failures into
> the current exception, mirroring what the serialized path already does by
> closing through try-with-resources.
> This behavior change was introduced in CASSANDRA-21354
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
