[
https://issues.apache.org/jira/browse/CASSANDRA-15526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049522#comment-17049522
]
Gianluca Righetto edited comment on CASSANDRA-15526 at 3/6/20, 7:39 AM:
------------------------------------------------------------------------
I'm able to consistently reproduce this test failure in my dev environment on
OSX with JDK 11 using breakpoints. If I set a breakpoint at the following line
and keep stepping over it in the IDE, it will eventually throw the assertion
error in RangeIterator.
[https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob;f=test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java;h=0be05b901a3425adbb1bcd88d6d96a9f1efe5652;hb=HEAD#l1087]
Note, you'll have to increase the test.timeout and range_request_timeout_in_ms
to be able to debug it properly.
The implementation of {{ConcurrentSkipListMap}} changed significantly from Java
v8 to v11. In v8, {{ConcurrentSkipListMap#size}} iterates over the elements in
the map and counts them as it goes. But in v11, {{ConcurrentSkipListMap}} holds
an internal {{LongAdder}} instance, which is incremented as new elements are
added.
In both versions, {{isEmpty}} will return false if it's able to find the head
node.
The issue is that in v11 there's a potential race condition in which the head
node may have been initialized, but the {{LongAdder}} hasn't been incremented
yet, which leaves it briefly in an inconsistent state.
I confirmed that's the case by logging the value of {{keys.size()}} in
{{TrieMemIndex#search}} right before and after the {{keys.isEmpty()}} check.
The size is 0 at runtime, but when the IDE debugger inspects the value, it has
changed to 1 already.
I'm attaching a patch that fixes this issue by replacing the
{{!keys.isEmpty()}} check by {{keys.size() > 0}} in {{TrieMemIndex}}.
{{ConcurrentSkipListMap#size}} also checks the head node first, if it's unset,
it returns 0 right away. Otherwise, it will compute the sum of the
{{LongAdder}}, which also already happens in the constructor of
{{KeyRangeIterator}} anyway, so it shouldn't be too big of a performance
penalty, asymptotically speaking.
was (Author: gianluca):
I'm able to consistently reproduce this test failure in my dev environment on
OSX with JDK 11 using breakpoints. If I set a breakpoint at the following line
and keep stepping over it in the IDE, it will eventually throw the assertion
error in RangeIterator.
https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob;f=test/unit/org/apache/cassandra/index/sasi/SASIIndexTest.java;h=0be05b901a3425adbb1bcd88d6d96a9f1efe5652;hb=HEAD#l1087
Note, you'll have to increase the test.timeout and range_request_timeout_in_ms
to be able to debug it properly.
The implementation of {{ConcurrentSkipListMap}} changed significantly from Java
v8 to v11. In v8, {{ConcurrentSkipListMap#size}} iterates over the elements in
the map and counts them as it goes. But in v11, {{ConcurrentSkipListMap}} holds
an internal {{LongAdder}} instance, which is incremented as new elements are
added.
In both versions, {{isEmpty}} will return false if it's able to find the head
node.
The issue is that in v11 there's a potential race condition in which the head
node may have been initialized, but the {{LongAdder}} hasn't been incremented
yet, which leaves it briefly in an inconsistent state.
I confirmed that's the case by logging the value of {{keys.size()}} in
{{TrieMemIndex#search}} right before and after the {{keys.isEmpty()}} check.
The size is 0 at runtime, but when the IDE debugger inspects the value, it has
changed to 1 already.
I'm attaching a patch that fixes this issue by replacing the
{{!keys.isEmpty()}} check by {{keys.size() > 0}} in {{TrieMemIndex}}.
{{ConcurrentSkipListMap#size}} also checks the head node first, if it's unset,
it returns 0 right away. Otherwise, it will compute the sum of the
{{LongAdder}}, which also already happens in the constructor of
{{KeyRangeIterator }}anyway, so it shouldn't be too big of a performance
penalty, asymptotically speaking.
> Fix flakey test - org.apache.cassandra.index.sasi.SASIIndexTest
> testConcurrentMemtableReadsAndWrites
> ----------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-15526
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15526
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
> Reporter: David Capwell
> Assignee: Gianluca Righetto
> Priority: Normal
> Fix For: 4.0-alpha
>
> Attachments: 15526-trunk-4.0.txt
>
>
> {code}
> junit.framework.AssertionFailedError
> at
> org.apache.cassandra.index.sasi.utils.RangeIterator.<init>(RangeIterator.java:46)
> at
> org.apache.cassandra.index.sasi.memory.KeyRangeIterator.<init>(KeyRangeIterator.java:42)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex$ConcurrentTrie.search(TrieMemIndex.java:150)
> at
> org.apache.cassandra.index.sasi.memory.TrieMemIndex.search(TrieMemIndex.java:102)
> at
> org.apache.cassandra.index.sasi.memory.IndexMemtable.search(IndexMemtable.java:70)
> at
> org.apache.cassandra.index.sasi.conf.ColumnIndex.searchMemtable(ColumnIndex.java:138)
> at org.apache.cassandra.index.sasi.TermIterator.build(TermIterator.java:91)
> at
> org.apache.cassandra.index.sasi.plan.QueryController.getIndexes(QueryController.java:145)
> at
> org.apache.cassandra.index.sasi.plan.Operation$Builder.complete(Operation.java:434)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.analyze(QueryPlan.java:57)
> at org.apache.cassandra.index.sasi.plan.QueryPlan.execute(QueryPlan.java:68)
> at
> org.apache.cassandra.index.sasi.SASIIndex.lambda$searcherFor$2(SASIIndex.java:301)
> at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:455)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getIndexed(SASIIndexTest.java:2576)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.getPaged(SASIIndexTest.java:2537)
> at
> org.apache.cassandra.index.sasi.SASIIndexTest.testConcurrentMemtableReadsAndWrites(SASIIndexTest.java:1108)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
