[
https://issues.apache.org/jira/browse/CASSANDRA-19427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821546#comment-17821546
]
Stefan Miklosovic commented on CASSANDRA-19427:
-----------------------------------------------
[CASSANDRA-19427-4.1|https://github.com/instaclustr/cassandra/tree/CASSANDRA-19427-4.1]
{noformat}
java11_pre-commit_tests
java11_separate_tests
java8_pre-commit_tests
✓ j8_build 8m 18s
✓ j8_cqlsh_dtests_py3 7m 13s
✓ j8_cqlsh_dtests_py311 8m 45s
✓ j8_cqlsh_dtests_py311_vnode 8m 8s
✓ j8_cqlsh_dtests_py38 6m 41s
✓ j8_cqlsh_dtests_py38_vnode 6m 1s
✓ j8_cqlsh_dtests_py3_vnode 6m 59s
✓ j8_cqlshlib_cython_tests 13m 37s
✓ j8_cqlshlib_tests 8m 39s
✓ j8_dtests 34m 1s
✓ j8_dtests_vnode 35m 20s
✓ j8_jvm_dtests 18m 29s
✓ j8_jvm_dtests_vnode 16m 11s
✓ j8_simulator_dtests 1m 33s
✓ j11_jvm_dtests_vnode 12m 35s
✓ j11_jvm_dtests 15m 32s
✓ j11_dtests_vnode 37m 3s
✓ j11_dtests 33m 48s
✓ j11_cqlshlib_tests 6m 41s
✓ j11_cqlshlib_cython_tests 7m 14s
✓ j11_cqlsh_dtests_py3_vnode 5m 37s
✓ j11_cqlsh_dtests_py38_vnode 6m 7s
✓ j11_cqlsh_dtests_py38 5m 42s
✓ j11_cqlsh_dtests_py311_vnode 5m 48s
✓ j11_cqlsh_dtests_py311 5m 43s
✓ j11_cqlsh_dtests_py3 5m 48s
✕ j8_unit_tests 10m 29s
org.apache.cassandra.cql3.MemtableSizeTest testSize[skiplist]
✕ j8_utests_system_keyspace_directory 11m 13s
org.apache.cassandra.cql3.MemtableSizeTest testSize[skiplist]
✕ j11_unit_tests 11m 23s
org.apache.cassandra.cql3.MemtableSizeTest testSize[skiplist]
java8_separate_tests
{noformat}
[java11_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3933/workflows/92e861db-4f2c-4bdf-83ed-1966aca7a3f7]
[java11_separate_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3933/workflows/b12c07ea-62f1-492d-8959-4c2737ce53c6]
[java8_pre-commit_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3933/workflows/65dba7d5-40d9-46a1-8322-388a70e1e2aa]
[java8_separate_tests|https://app.circleci.com/pipelines/github/instaclustr/cassandra/3933/workflows/4b5b14bf-5d78-4c0c-9de9-322ca6a9aab5]
> Fix concurrent access of ClientWarn causing AIOBE for SELECT WHERE IN queries
> with multiple coordinator-local partitions
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-19427
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19427
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Coordination, Legacy/Local Write-Read Paths
> Reporter: Abe Ratnofsky
> Assignee: Abe Ratnofsky
> Priority: Normal
> Fix For: 3.11.x, 4.0.x, 4.1.x, 5.0.x, 5.x
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> On one of our clusters, we noticed rare but periodic
> ArrayIndexOutOfBoundsExceptions:
>
> {code:java}
> message="Uncaught exception on thread Thread[ReadStage-3,5,main]"
> exception="java.lang.RuntimeException:
> java.lang.ArrayIndexOutOfBoundsException
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2579)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.ArrayIndexOutOfBoundsException"{code}
>
>
> The error was in a Runnable, so the stacktrace didn't directly indicate where
> the error was coming from. We enabled JFR to log the underlying exception
> that was thrown:
>
> {code:java}
> message="Uncaught exception on thread Thread[ReadStage-2,5,main]"
> exception="java.lang.RuntimeException:
> java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 0
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2579)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds
> for length 0
> at java.base/java.util.ArrayList.add(ArrayList.java:487)
> at java.base/java.util.ArrayList.add(ArrayList.java:499)
> at org.apache.cassandra.service.ClientWarn$State.add(ClientWarn.java:84)
> at
> org.apache.cassandra.service.ClientWarn$State.access$000(ClientWarn.java:77)
> at org.apache.cassandra.service.ClientWarn.warn(ClientWarn.java:51)
> at
> org.apache.cassandra.db.ReadCommand$1MetricRecording.onClose(ReadCommand.java:596)
> at
> org.apache.cassandra.db.transform.BasePartitions.runOnClose(BasePartitions.java:70)
> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:95)
> at
> org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:2260)
> at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2575)
> ... 6 more"{code}
>
>
> An AIOBE on ArrayList.add(E) should only be possible when multiple threads
> attempt to call the method at the same time.
>
> This was seen while executing a SELECT WHERE IN query with multiple partition
> keys. This exception could happen when multiple local reads are dispatched by
> the coordinator in
> org.apache.cassandra.service.reads.AbstractReadExecutor#makeRequests. In this
> case, multiple local reads exceed the tombstone warning threshold, so
> multiple tombstone warnings are added to the same ClientWarn.State reference.
> Currently, org.apache.cassandra.service.ClientWarn.State#warnings is an
> ArrayList, which isn't safe for concurrent modification, causing the AIOBE to
> be thrown.
>
> I have a patch available for this, and I'm preparing it now. The patch is
> simple - it just changes
> org.apache.cassandra.service.ClientWarn.State#warnings to a thread-safe
> CopyOnWriteArrayList. I also have a jvm-dtest that demonstrates the issue but
> doesn't need to be merged - it shows how a SELECT WHERE IN query with local
> reads that add client warnings can add to the same ClientWarn.State from
> different threads. I'll push that in a separate branch just for demonstration
> purposes.
>
> Demonstration branch:
> [https://github.com/apache/cassandra/compare/trunk...aratno:cassandra:CASSANDRA-19427-aiobe-clientwarn-demo]
> Fix branch:
> [https://github.com/apache/cassandra/compare/trunk...aratno:cassandra:CASSANDRA-19427-aiobe-clientwarn-fix]
> (PR linked below)
>
> This appears to have been an issue since at least 3.11, that was the earliest
> release I checked.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
