[
https://issues.apache.org/jira/browse/LUCENE-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hoss Man updated LUCENE-7132:
-----------------------------
Attachment: LUCENE-7132.patch
I was reminded of this issue today and pinged [~mikemccand] about it on IRC.
In reviewing the issue myself, to try and describe it to him succinctly, I
realized the key bullet points of this bug is:
* It is possible to build an index, _with *no* deleted documents_ such that a
particular BooleanQuery returns drastically different scores for some documents
depending on if/how the segments in the index have been merged.
* The Explanations for every document matching this BooleanQuery do *not*
change based on if/how the segments in the index have been merged -- such that
the Explanations can be drastically different then the scores
Couched that way, and with a request from mike to try and confirm if the bug
was dependent on using LuceneTestCase.newIndexWriterConfig (i gather he
suspected it might be a test only bug) I revamped the previous test class to
attempt to more straight forwardly demonstrate the crux of the matter.
There are now two test methods: testScoresWithDefaultIWC &
testScoresWithRandomIWC -- both delegating to the same helpe method for the
meat of the test, just using different IndexWriterConfigs.
"checkScores" is the meat of both test methods. It builds up an index using
the same "RajeshData.txt" and then executes a fixed query against the index,
recording the scores of every mathcing document. It then does a
{{forceMerge(1,true)}} on the IndexWriter, and re-executes the query comparing
the scores of every matching document against the scores from the previous
query execution.
If the tests make it this far (and they rarely do) then does the query again,
this time checking the "Explanation" for every matching document against the
score.
----
With this seed, both tests demonstrate an identical inconsistency between the
scores of a document containing body="Microsoft Office 365" between the
pre-merge and single-segment indexes....
{noformat}
[junit4] <JUnit4> says ahoj! Master seed: B6DEF72C383813DE
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(29777@localhost).
[junit4] Suite: org.apache.lucene.search.TestBooleanScoreConsistency
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithRandomIWC
-Dtests.seed=B6DEF72C383813DE -Dtests.slow=true -Dtests.locale=ar-IQ
-Dtests.timezone=Asia/Jayapura -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 2.80s | TestBooleanScoreConsistency.testScoresWithRandomIWC
<<<
[junit4] > Throwable #1: java.lang.AssertionError: 7614: score doesn't
match (previously returned) expected score for Microsoft Office 365
expected:<1.2357021570205688> but was:<0.41190072894096375>
[junit4] > at
__randomizedtesting.SeedInfo.seed([B6DEF72C383813DE:2C10E99E7CDA36F8]:0)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExpected(TestBooleanScoreConsistency.java:178)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:137)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithRandomIWC(TestBooleanScoreConsistency.java:85)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithDefaultIWC
-Dtests.seed=B6DEF72C383813DE -Dtests.slow=true -Dtests.locale=ar-IQ
-Dtests.timezone=Asia/Jayapura -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 1.63s |
TestBooleanScoreConsistency.testScoresWithDefaultIWC <<<
[junit4] > Throwable #1: java.lang.AssertionError: 7614: score doesn't
match (previously returned) expected score for Microsoft Office 365
expected:<1.2357021570205688> but was:<0.41190072894096375>
[junit4] > at
__randomizedtesting.SeedInfo.seed([B6DEF72C383813DE:1CF87A1A7D4B3415]:0)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExpected(TestBooleanScoreConsistency.java:178)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:137)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC(TestBooleanScoreConsistency.java:79)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62):
{id=PostingsFormat(name=Direct), body=Lucene50(blocksize=128)}, docValues:{},
maxPointsInLeafNode=825, maxMBSortInHeap=5.5977062021908015,
sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=ar-IQ,
timezone=Asia/Jayapura
[junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation
1.8.0_74 (64-bit)/cpus=4,threads=1,free=266511232,total=336068608
[junit4] 2> NOTE: All tests run in this JVM: [TestBooleanScoreConsistency]
[junit4] Completed [1/1 (1!)] in 4.72s, 2 tests, 2 failures <<< FAILURES!
[junit4]
[junit4]
[junit4] Tests with failures [seed: B6DEF72C383813DE]:
[junit4] -
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithRandomIWC
[junit4] -
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC
[junit4]
[junit4]
[junit4] JVM J0: 0.40 .. 5.70 = 5.30s
[junit4] Execution time total: 5.77 sec.
[junit4] Tests summary: 1 suite, 2 tests, 2 failures
{noformat}
With this seed, the pre-merge and single-segment indexes in both tests return
identical scores for all matching docs, but the Explanations (which seem to be
correct AFAICT) don't match the scores...
{noformat}
[junit4] <JUnit4> says salut! Master seed: AEB2F91E0E8938CC
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(29716@localhost).
[junit4] Suite: org.apache.lucene.search.TestBooleanScoreConsistency
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithDefaultIWC
-Dtests.seed=AEB2F91E0E8938CC -Dtests.slow=true -Dtests.locale=zh
-Dtests.timezone=Australia/Sydney -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 4.08s |
TestBooleanScoreConsistency.testScoresWithDefaultIWC <<<
[junit4] > Throwable #1: java.lang.AssertionError: 15581: score() and
explain() return different values for: Office 365
[junit4] > 0.7475659 = product of:
[junit4] > 2.2426977 = sum of:
[junit4] > 2.2426977 = weight(body:365 in 15581) [ClassicSimilarity],
result of:
[junit4] > 2.2426977 = score(doc=15581,freq=1.0), product of:
[junit4] > 0.5300819 = queryWeight, product of:
[junit4] > 6.769362 = idf(docFreq=53, docCount=17297)
[junit4] > 0.07830604 = queryNorm
[junit4] > 4.230851 = fieldWeight in 15581, product of:
[junit4] > 1.0 = tf(freq=1.0), with freq of:
[junit4] > 1.0 = termFreq=1.0
[junit4] > 6.769362 = idf(docFreq=53, docCount=17297)
[junit4] > 0.625 = fieldNorm(doc=15581)
[junit4] > 0.33333334 = coord(1/3)
[junit4] > expected:<0.2491886466741562> but was:<0.7475659251213074>
[junit4] > at
__randomizedtesting.SeedInfo.seed([AEB2F91E0E8938CC:49474284BFA1F07]:0)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExplanations(TestBooleanScoreConsistency.java:214)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:138)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC(TestBooleanScoreConsistency.java:79)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithRandomIWC
-Dtests.seed=AEB2F91E0E8938CC -Dtests.slow=true -Dtests.locale=zh
-Dtests.timezone=Australia/Sydney -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 1.90s | TestBooleanScoreConsistency.testScoresWithRandomIWC
<<<
[junit4] > Throwable #1: java.lang.AssertionError: 7614: score() and
explain() return different values for: Microsoft Office 365
[junit4] > 1.2357022 = product of:
[junit4] > 1.8535532 = sum of:
[junit4] > 0.05939492 = weight(body:microsoft in 7614)
[ClassicSimilarity], result of:
[junit4] > 0.05939492 = score(doc=7614,freq=1.0), product of:
[junit4] > 0.09644668 = queryWeight, product of:
[junit4] > 1.2316633 = idf(docFreq=13720, docCount=17297)
[junit4] > 0.07830604 = queryNorm
[junit4] > 0.6158317 = fieldWeight in 7614, product of:
[junit4] > 1.0 = tf(freq=1.0), with freq of:
[junit4] > 1.0 = termFreq=1.0
[junit4] > 1.2316633 = idf(docFreq=13720, docCount=17297)
[junit4] > 0.5 = fieldNorm(doc=7614)
[junit4] > 1.7941582 = weight(body:365 in 7614) [ClassicSimilarity],
result of:
[junit4] > 1.7941582 = score(doc=7614,freq=1.0), product of:
[junit4] > 0.5300819 = queryWeight, product of:
[junit4] > 6.769362 = idf(docFreq=53, docCount=17297)
[junit4] > 0.07830604 = queryNorm
[junit4] > 3.384681 = fieldWeight in 7614, product of:
[junit4] > 1.0 = tf(freq=1.0), with freq of:
[junit4] > 1.0 = termFreq=1.0
[junit4] > 6.769362 = idf(docFreq=53, docCount=17297)
[junit4] > 0.5 = fieldNorm(doc=7614)
[junit4] > 0.6666667 = coord(2/3)
[junit4] > expected:<0.41190072894096375> but was:<1.2357021570205688>
[junit4] > at
__randomizedtesting.SeedInfo.seed([AEB2F91E0E8938CC:347CE7AC4A6B1DEA]:0)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExplanations(TestBooleanScoreConsistency.java:214)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:138)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithRandomIWC(TestBooleanScoreConsistency.java:85)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62):
{id=PostingsFormat(name=MockRandom), body=PostingsFormat(name=LuceneFixedGap)},
docValues:{}, maxPointsInLeafNode=127, maxMBSortInHeap=5.70855157410318,
sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=zh,
timezone=Australia/Sydney
[junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation
1.8.0_74 (64-bit)/cpus=4,threads=1,free=181104832,total=342360064
[junit4] 2> NOTE: All tests run in this JVM: [TestBooleanScoreConsistency]
[junit4] Completed [1/1 (1!)] in 6.29s, 2 tests, 2 failures <<< FAILURES!
[junit4]
[junit4]
[junit4] Tests with failures [seed: AEB2F91E0E8938CC]:
[junit4] -
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC
[junit4] -
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithRandomIWC
[junit4]
[junit4]
[junit4] JVM J0: 0.41 .. 7.13 = 6.73s
[junit4] Execution time total: 7.18 sec.
[junit4] Tests summary: 1 suite, 2 tests, 2 failures
{noformat}
With this perplexing seed (found using tests.iters) the Randomized
IndexWriterConfig manages to pass all checks in the test -- getting scores
consistent with the Explanation both before and after the forceMerge -- but the
default IndexWriterConfig still demonstrates the problem...
{noformat}
[junit4] <JUnit4> says hallo! Master seed: 29D39C39B6047864:5BA9313A359A1831
[junit4] Executing 1 suite with 1 JVM.
[junit4]
[junit4] Started J0 PID(29867@localhost).
[junit4] Suite: org.apache.lucene.search.TestBooleanScoreConsistency
[junit4] OK 6.16s | TestBooleanScoreConsistency.testScoresWithRandomIWC
[junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestBooleanScoreConsistency -Dtests.method=testScoresWithDefaultIWC
-Dtests.seed=29D39C39B6047864 -Dtests.slow=true -Dtests.locale=uk-UA
-Dtests.timezone=Indian/Comoro -Dtests.asserts=true
-Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 1.26s |
TestBooleanScoreConsistency.testScoresWithDefaultIWC <<<
[junit4] > Throwable #1: java.lang.AssertionError: 7614: score doesn't
match (previously returned) expected score for Microsoft Office 365
expected:<1.2357021570205688> but was:<0.41190072894096375>
[junit4] > at
__randomizedtesting.SeedInfo.seed([29D39C39B6047864:5BA9313A359A1831]:0)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScoresAgainstExpected(TestBooleanScoreConsistency.java:178)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.checkScores(TestBooleanScoreConsistency.java:137)
[junit4] > at
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC(TestBooleanScoreConsistency.java:79)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene62),
sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=uk-UA,
timezone=Indian/Comoro
[junit4] 2> NOTE: Linux 3.19.0-51-generic amd64/Oracle Corporation
1.8.0_74 (64-bit)/cpus=4,threads=1,free=285815104,total=350748672
[junit4] 2> NOTE: All tests run in this JVM: [TestBooleanScoreConsistency]
[junit4] Completed [1/1 (1!)] in 7.73s, 2 tests, 1 failure <<< FAILURES!
[junit4]
[junit4]
[junit4] Tests with failures [seed: 29D39C39B6047864:5BA9313A359A1831]:
[junit4] -
org.apache.lucene.search.TestBooleanScoreConsistency.testScoresWithDefaultIWC
[junit4]
[junit4]
[junit4] JVM J0: 0.37 .. 8.58 = 8.21s
[junit4] Execution time total: 8.62 sec.
[junit4] Tests summary: 1 suite, 2 tests, 1 failure
{noformat}
> BooleanQuery scores can be diff for same docs+sim when using coord (disagree
> with Explanation which doesn't change)
> -------------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-7132
> URL: https://issues.apache.org/jira/browse/LUCENE-7132
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/search
> Affects Versions: 5.5
> Reporter: Ahmet Arslan
> Assignee: Steve Rowe
> Attachments: LUCENE-7132.patch, LUCENE-7132.patch, LUCENE-7132.patch,
> SOLR-8884.patch, SOLR-8884.patch, debug.xml
>
>
> Some of the folks
> [reported|http://find.searchhub.org/document/80666f5c3b86ddda] that sometimes
> explain's score can be different than the score requested by fields
> parameter. Interestingly, Explain's scores would create a different ranking
> than the original result list. This is something users experience, but it
> cannot be re-produced deterministically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
