[
https://issues.apache.org/jira/browse/OAK-9502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434681#comment-17434681
]
Amrit Verma edited comment on OAK-9502 at 10/27/21, 6:26 AM:
-------------------------------------------------------------
I looked into this and there is no issue with the test. We just use randomly
generated data and do similarity search on it and expect atleast 60% match of
the top 10 similarity search results with the results of brute force. However
this match is coming 50% sometimes.
In the oak trunk [build
history|https://ci-builds.apache.org/job/Jackrabbit/job/jackrabbit-oak-trunk/org.apache.jackrabbit$oak-search-elastic/381/testReport/org.apache.jackrabbit.oak.plugins.index.elastic/ElasticSimilarQueryTest/vectorSimilarityLargeData/history/]
, this test failed 5 out of 19 times and each of those times the match was 50%.
There is not much we can do right now about this apart from reducing
expectation or letting it fail. If we need better accuracy, we may need to fine
tune the algorithm further (or use a different one).
was (Author: amrverma):
I looked into this and there is no issue with the test. We just use randomly
generated data and do similarity search on it and expect atleast 60% match of
the top 10 similarity search results with the results of brute force. However
this match is coming 50% sometimes.
In the oak trunk [build
history|https://ci-builds.apache.org/job/Jackrabbit/job/jackrabbit-oak-trunk/org.apache.jackrabbit$oak-search-elastic/381/testReport/org.apache.jackrabbit.oak.plugins.index.elastic/ElasticSimilarQueryTest/vectorSimilarityLargeData/history/]
, this test failed 5 out of 19 times and each of those times the match was 50%.
There is not much we can do right now about this apart from reducing
expectation to 30% or letting it fail. If we need better accuracy, we may need
to fine tune the algorithm further (or use a different one).
> ElasticSimilarQueryTest.vectorSimilarityLargeData fails intermittently on ASF
> Jenkins
> -------------------------------------------------------------------------------------
>
> Key: OAK-9502
> URL: https://issues.apache.org/jira/browse/OAK-9502
> Project: Jackrabbit Oak
> Issue Type: Bug
> Components: elastic-search
> Reporter: Konrad Windszus
> Assignee: Amrit Verma
> Priority: Major
>
> E.g. in
> [https://ci-builds.apache.org/job/Jackrabbit/job/oak-trunk-pr/job/OAK-8910/1/testReport/.|https://ci-builds.apache.org/job/Jackrabbit/job/oak-trunk-pr/job/OAK-8910/1/testReport/]
> The log just exposes
> {code}
> java.lang.AssertionError: Condition not satisfied after 15.36 seconds and 13
> attempts
> at
> org.apache.jackrabbit.oak.plugins.index.elastic.ElasticTestUtils.assertEventually(ElasticTestUtils.java:45)
> at
> org.apache.jackrabbit.oak.plugins.index.elastic.ElasticAbstractQueryTest.assertEventually(ElasticAbstractQueryTest.java:179)
> at
> org.apache.jackrabbit.oak.plugins.index.elastic.ElasticSimilarQueryTest.verifyLSHResults(ElasticSimilarQueryTest.java:311)
> at
> org.apache.jackrabbit.oak.plugins.index.elastic.ElasticSimilarQueryTest.vectorSimilarityLargeData(ElasticSimilarQueryTest.java:386)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at
> org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:30)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: java.lang.AssertionError: expected: [img1352, img1869, img21,
> img747, img817, img1101, img177, img1600, img825, img682] got: [img1352,
> img817, img814, img305, img1600, img1101, img825, img1901, img889, img864]
> expected:<80.0> but was:<50.0>
> at org.junit.Assert.fail(Assert.java:89)
> at org.junit.Assert.failNotEquals(Assert.java:835)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.jackrabbit.oak.plugins.index.elastic.ElasticSimilarQueryTest.lambda$verifyLSHResults$10(ElasticSimilarQueryTest.java:324)
> at
> org.apache.jackrabbit.oak.plugins.index.elastic.ElasticTestUtils.assertEventually(ElasticTestUtils.java:37)
> ... 36 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
