[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877233#comment-16877233 ] ASF subversion and git services commented on LUCENE-8857: - Commit f686167e710e2dd1d8b37f938ee0fcda9cebbe1a in lucene-solr's branch refs/heads/master from Atri Sharma [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=f686167 ] LUCENE-8857: Introduce Custom Tiebreakers in TopDocs#merge (#757) This commit introduces custom tiebreakers which allows users to specify custom tiebreakers when ordering hits to return. A default tiebreaker is introduced for tie breaking on shard index first and then docID. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4h 40m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877232#comment-16877232 ] Atri Sharma commented on LUCENE-8857: - [~jpountz] Yes, I ran the Solr suite twice. The first time, failures with tracer not able to close were seen. The second time, the entire suite came in clean. I also ran ant precommit – came in clean. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877157#comment-16877157 ] Adrien Grand commented on LUCENE-8857: -- Double checking, have you run all Solr tests or only TestDistributedGrouping? > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877156#comment-16877156 ] Atri Sharma commented on LUCENE-8857: - [~jpountz] Thanks for confirming. I wanted to ensure that no unsuspecting user gets bitten :) > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877155#comment-16877155 ] Adrien Grand commented on LUCENE-8857: -- Thanks [~atris] I'll look into merging now. MIGRATE is considered quite loud already, plus the fact that it is there makes it pretty likely to be included in the release notes. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876965#comment-16876965 ] Atri Sharma commented on LUCENE-8857: - Since this is a breaking API change, is there a way we can highlight this to existing users in a "louder" manner, or is MIGRATE.txt entry enough? > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876773#comment-16876773 ] Atri Sharma commented on LUCENE-8857: - JFYI The latest iteration on PR also fixes the compilation failure in Solr, introduced in SOLR-13404 > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[
https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876759#comment-16876759
]
Atri Sharma commented on LUCENE-8857:
-
[~jpountz] I have pushed the latest iteration to the new PR. It passes ant test:
{code:java}
[junit4:tophints] 59.58s |
org.apache.lucene.search.suggest.document.TestSuggestField
[junit4:tophints] 17.10s |
org.apache.lucene.search.suggest.DocumentDictionaryTest
[junit4:tophints] 14.56s |
org.apache.lucene.search.suggest.fst.FSTCompletionTest
[junit4:tophints] 14.21s |
org.apache.lucene.search.suggest.analyzing.FuzzySuggesterTest
-check-totals:
common.test:
-check-totals:
test:
BUILD SUCCESSFUL
Total time: 74 minutes 29 seconds
f01898a404cf:lucene atris$
{code}
It also passes the offending Solr test:
ant test -Dtestcase=TestDistributedGrouping -Dtests.method=test
-Dtests.seed=B5D95BEAE23E9468 -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=nl-AW -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
{code:java}
27429 INFO (closeThreadPool-74-thread-4) [ ] o.e.j.s.AbstractConnector
Stopped ServerConnector@3e6caf50{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:0}
27430 INFO (closeThreadPool-74-thread-4) [ ] o.e.j.s.h.ContextHandler
Stopped o.e.j.s.ServletContextHandler@169e0265{/,null,UNAVAILABLE}
27430 INFO (closeThreadPool-74-thread-4) [ ] o.e.j.s.session node0 Stopped
scavenging
27431 INFO (closeThreadPool-74-thread-1) [ ] o.e.j.s.AbstractConnector
Stopped ServerConnector@1be02e89{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:0}
27431 INFO (closeThreadPool-74-thread-1) [ ] o.e.j.s.h.ContextHandler
Stopped o.e.j.s.ServletContextHandler@6b6f3dda{/,null,UNAVAILABLE}
27432 INFO (closeThreadPool-74-thread-1) [ ] o.e.j.s.session node0 Stopped
scavenging
27432 INFO (closeThreadPool-74-thread-5) [ ] o.e.j.s.AbstractConnector
Stopped ServerConnector@4052b482{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:0}
27432 INFO (closeThreadPool-74-thread-5) [ ] o.e.j.s.h.ContextHandler
Stopped o.e.j.s.ServletContextHandler@7063254f{/,null,UNAVAILABLE}
27432 INFO (closeThreadPool-74-thread-5) [ ] o.e.j.s.session node0 Stopped
scavenging
27436 INFO (SUITE-TestDistributedGrouping-seed#[C817F4DEFFC8F2A7]-worker) [
] o.a.s.SolrTestCaseJ4 ---
Done waiting for tracked resources to be released{code}
> Refactor TopDocs#Merge To Take In Custom Tie Breakers
> -
>
> Key: LUCENE-8857
> URL: https://issues.apache.org/jira/browse/LUCENE-8857
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch,
> LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> In LUCENE-8829, the idea of having lambdas passed in to the API to allow
> finer control over the process was discussed.
> This JIRA tracks adding a parameter to the API which allows passing in
> lambdas to define custom tie breakers, thus allowing users to do custom
> algorithms when required.
> CC: [~jpountz] [~simonw]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876711#comment-16876711 ] Atri Sharma commented on LUCENE-8857: - [~munendrasn] Thanks for the compilation fix. Yes, the test will fail. I fixed that test failure – will update the PR once my local test suite run completes > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876707#comment-16876707 ] Munendra S N commented on LUCENE-8857: -- [^LUCENE-8857-compile-fix.patch] [~atris] I made some changes as part of SOLR-13404. That along with your patch will cause compilation error. I'm attaching the fix for that. Still test fails for which you have raised the issue > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876708#comment-16876708 ] Atri Sharma commented on LUCENE-8857: - Ok, updating the PR now. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876705#comment-16876705 ] Adrien Grand commented on LUCENE-8857: -- We need to have all changes in the same pull request, otherwise there will be a window of time during which we will get test failures when testing Solr, which could break a lot of people. As you noticed, it didn't take long to Munendra to notice something had broken. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857-compile-fix.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876699#comment-16876699 ] Atri Sharma commented on LUCENE-8857: - [~jpountz] Yes, we will. I did not want to add the fix for Solr in this PR since that kind of muddles up (going across two modules). I can raise a separate PR just for the Solr fixes, though, if that works. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876698#comment-16876698 ] Adrien Grand commented on LUCENE-8857: -- [~atris] Thanks for looking into the grouping failure. I'm not seeing changes to Solr, so I'm assuming we would still get the failure that [~munendrasn] shared if we pushed? > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876692#comment-16876692 ] Atri Sharma commented on LUCENE-8857: - I have opened https://issues.apache.org/jira/browse/SOLR-13597 to track fixes to Solr to use the new API (that is what is causing the Solr test to fail). I will raise a PR for that Jira post the merging of this PR. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4.5h > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[
https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876378#comment-16876378
]
Atri Sharma commented on LUCENE-8857:
-
[~jpountz] Ran ant test 5 times again: all came in clean:
I have raised a new PR with testGrouping fixes:
[https://github.com/apache/lucene-solr/pull/757]
Can we merge it, if it looks fine?
{code:java}
junit4:tophints] 54.39s |
org.apache.lucene.search.suggest.document.TestSuggestField
[junit4:tophints] 16.93s |
org.apache.lucene.search.suggest.DocumentDictionaryTest
[junit4:tophints] 16.63s |
org.apache.lucene.search.suggest.analyzing.FuzzySuggesterTest
[junit4:tophints] 16.42s |
org.apache.lucene.search.suggest.fst.FSTCompletionTest
-check-totals:
common.test:
-check-totals:
test:
BUILD SUCCESSFUL
Total time: 45 minutes 8 seconds
f01898a404cf:lucene atris$ {code}
> Refactor TopDocs#Merge To Take In Custom Tie Breakers
> -
>
> Key: LUCENE-8857
> URL: https://issues.apache.org/jira/browse/LUCENE-8857
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch,
> LUCENE-8857.patch, LUCENE-8857.patch
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> In LUCENE-8829, the idea of having lambdas passed in to the API to allow
> finer control over the process was discussed.
> This JIRA tracks adding a parameter to the API which allows passing in
> lambdas to define custom tie breakers, thus allowing users to do custom
> algorithms when required.
> CC: [~jpountz] [~simonw]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[
https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876335#comment-16876335
]
Atri Sharma commented on LUCENE-8857:
-
[~jpountz] I investigated this and it turned out to be a test limitation
(testGrouping assumed that TopDocs.merge was setting the shard indices). It
took a while to reproduce since it was the random test which was failing
(thanks for providing the seed!) I have fixed the test and ran ant test a
couple of times – it came in clean:
Can we push this in now?
{code:java}
[junit4:tophints] 49.54s |
org.apache.lucene.search.suggest.document.TestSuggestField
[junit4:tophints] 21.55s |
org.apache.lucene.search.suggest.analyzing.FuzzySuggesterTest
[junit4:tophints] 21.51s |
org.apache.lucene.search.suggest.DocumentDictionaryTest
[junit4:tophints] 15.45s | org.apache.lucene.search.spell.TestSpellChecker
-check-totals:
common.test:
-check-totals:
test:
BUILD SUCCESSFUL
Total time: 49 minutes 49 seconds
f01898a404cf:lucene atris$ {code}
[~munendrasn] I am not too aware of Solr's internals, but looking at the error
you pointed to, looks like that the test is not setting shard indices or hit
indices. This points to an assumption in the test – that TopDocs.merge is
setting the shard indices. Can you check
{code:java}
search/grouping/distributed/responseprocessor/TopGroupsShardResponseProcessor.java{code}
where the TopDocs.merge call is done? We can set shard indices for all TopHits
based on the QueryCommandResult they come from.
> Refactor TopDocs#Merge To Take In Custom Tie Breakers
> -
>
> Key: LUCENE-8857
> URL: https://issues.apache.org/jira/browse/LUCENE-8857
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch,
> LUCENE-8857.patch, LUCENE-8857.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> In LUCENE-8829, the idea of having lambdas passed in to the API to allow
> finer control over the process was discussed.
> This JIRA tracks adding a parameter to the API which allows passing in
> lambdas to define custom tie breakers, thus allowing users to do custom
> algorithms when required.
> CC: [~jpountz] [~simonw]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876253#comment-16876253 ] Atri Sharma commented on LUCENE-8857: - I did – I was not able to see any failures (probably due to seeds?). I will try with the seed in your command now. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876248#comment-16876248 ] Adrien Grand commented on LUCENE-8857: -- [~atris] Can you look into those failures? I had understood from your comment on the PR that you had run tests? FYI I'm seeing issues on the Lucene end as well, e.g. ant test -Dtestcase=TestGrouping -Dtests.method=testRandom -Dtests.seed=1039BE5B957F7FDD -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=be -Dtests.timezone=Europe/Rome -Dtests.asserts=true -Dtests.file.encoding=UTF-8. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876244#comment-16876244 ] ASF subversion and git services commented on LUCENE-8857: - Commit 3f0ecfa9c467c336ed8583ba8a2f2d920beb8763 in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=3f0ecfa ] Revert "LUCENE-8857: Introduce Custom Tiebreakers in TopDocs#merge (#734)" This reverts commit e70b43c39a2070a72a50a3b47ab3b9d485cdd044. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876242#comment-16876242 ] Adrien Grand commented on LUCENE-8857: -- Thanks [~munendrasn] I'm reverting now. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4h 20m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[
https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876227#comment-16876227
]
Munendra S N commented on LUCENE-8857:
--
I'm working on SOLR-13404, after this merge som tests started failing. Just to
be sure I tested in the latest master(without my changes), I was able to
reproduce the failure
{code:java}
ant test -Dtestcase=TestDistributedGrouping -Dtests.method=test
-Dtests.seed=B5D95BEAE23E9468 -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=nl-AW -Dtests.timezone=Asia/Jayapura -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
{code}
This is where it fails due to assertion Error
{code:java}
https://github.com/apache/lucene-solr/blob/e70b43c39a2070a72a50a3b47ab3b9d485cdd044/solr/core/src/test/org/apache/solr/TestDistributedGrouping.java#L329
{code}
{code:java}
https://github.com/apache/lucene-solr/blob/e70b43c39a2070a72a50a3b47ab3b9d485cdd044/lucene/core/src/java/org/apache/lucene/search/TopDocs.java#L79
{code}
> Refactor TopDocs#Merge To Take In Custom Tie Breakers
> -
>
> Key: LUCENE-8857
> URL: https://issues.apache.org/jira/browse/LUCENE-8857
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch,
> LUCENE-8857.patch, LUCENE-8857.patch
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> In LUCENE-8829, the idea of having lambdas passed in to the API to allow
> finer control over the process was discussed.
> This JIRA tracks adding a parameter to the API which allows passing in
> lambdas to define custom tie breakers, thus allowing users to do custom
> algorithms when required.
> CC: [~jpountz] [~simonw]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876218#comment-16876218 ] Atri Sharma commented on LUCENE-8857: - [~jpountz] Thanks for committing and reviewing, [~simonw] Thanks for your constructive inputs! > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876200#comment-16876200 ] ASF subversion and git services commented on LUCENE-8857: - Commit e70b43c39a2070a72a50a3b47ab3b9d485cdd044 in lucene-solr's branch refs/heads/master from Atri Sharma [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e70b43c ] LUCENE-8857: Introduce Custom Tiebreakers in TopDocs#merge (#734) This commit introduces custom tiebreakers which allows users to specify custom tiebreakers when ordering hits to return. A default tiebreaker is introduced for tie breaking on shard index first and then docID. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 4h 10m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876026#comment-16876026 ] Atri Sharma commented on LUCENE-8857: - Should we push the latest iteration on the PR, if it looks fine? > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 2h 50m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874125#comment-16874125 ] Atri Sharma commented on LUCENE-8857: - Updated the PR with latest comments, removing merge functionality as well. Happy to iterate further > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 1h 50m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868909#comment-16868909 ] Atri Sharma commented on LUCENE-8857: - [~simonw] I have added the default tie breaker which tie breaks by shard indices first and then docIDs, as suggested. The new PR has the latest iteration, please let me know if it seems fine. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > Time Spent: 10m > Remaining Estimate: 0h > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[
https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868717#comment-16868717
]
Atri Sharma commented on LUCENE-8857:
-
{quote}Any chance we can select the tie-breaker based on if one of the TopDocs
has a shardIndex != -1 and assert that all of them have it or not? Another
option would be to have only one comparator and first tie-break on shardIndex
and then on doc since we don't set the shard index it should be fine since they
are all -1?
{quote}
Would that not defeat the purpose of passing in the custom tie breaker? I
thought the reason we added passing in the Comparator was to allow users to
specify custom tie breaking algorithms, and define a custom one. Am I missing
something?
> Refactor TopDocs#Merge To Take In Custom Tie Breakers
> -
>
> Key: LUCENE-8857
> URL: https://issues.apache.org/jira/browse/LUCENE-8857
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
> Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch,
> LUCENE-8857.patch, LUCENE-8857.patch
>
>
> In LUCENE-8829, the idea of having lambdas passed in to the API to allow
> finer control over the process was discussed.
> This JIRA tracks adding a parameter to the API which allows passing in
> lambdas to define custom tie breakers, thus allowing users to do custom
> algorithms when required.
> CC: [~jpountz] [~simonw]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868555#comment-16868555 ] Simon Willnauer commented on LUCENE-8857: - A couple of comments: * can you open a PR and associate it with this issue. Patches are so hard to review without context and the ability to comment * for the second case in IndexsSearcher should we also tie-break by doc? * Can we replace the verbose comparators with _Comparator.comparingInt(d -> d.shardIndex);_ and _Comparator.comparingInt(d -> d.doc);_ respectively? * Any chance we can select the tie-breaker based on if one of the TopDocs has a shardIndex != -1 and assert that all of them have it or not? Another option would be to have only one comparator and first tie-break on shardIndex and then on doc since we don't set the shard index it should be fine since they are all -1? WDYT? > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868479#comment-16868479 ] Atri Sharma commented on LUCENE-8857: - Does this iteration look fine? Happy to iterate further if needed. > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867411#comment-16867411 ] Atri Sharma commented on LUCENE-8857: - Updated patch with improved javadocs and removal of now redundant methods [^LUCENE-8857.patch] > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch, LUCENE-8857.patch > > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866608#comment-16866608 ] Atri Sharma commented on LUCENE-8857: - [~simonw] The reason I did not add a default comparator for shard indices is because we would need the logic for tie breaking within a shard. The idea there was that users passing in custom tie breakers would be responsible for ensuring the sanity of their algorithms. I have changed the patch now to always treat hitIndex as the second level tie breaker i.e. when the passed in comparator returns 0, use hit index as the tie breaker, irrespective of what the tiebreaker used as the primary tie breaking algorithm. [^LUCENE-8857.patch] > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch, > LUCENE-8857.patch > > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[
https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866575#comment-16866575
]
Simon Willnauer commented on LUCENE-8857:
-
Why don't we just use the comparator and have a default and a doc one? like
this:
{code}
Comparator defaultComparator = Comparator.comparingInt(d ->
d.shardIndex);
Comparator docComparator = Comparator.comparingInt(d -> d.doc);
{code}
> Refactor TopDocs#Merge To Take In Custom Tie Breakers
> -
>
> Key: LUCENE-8857
> URL: https://issues.apache.org/jira/browse/LUCENE-8857
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
> Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch
>
>
> In LUCENE-8829, the idea of having lambdas passed in to the API to allow
> finer control over the process was discussed.
> This JIRA tracks adding a parameter to the API which allows passing in
> lambdas to define custom tie breakers, thus allowing users to do custom
> algorithms when required.
> CC: [~jpountz] [~simonw]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866564#comment-16866564 ] Atri Sharma commented on LUCENE-8857: - I havent added a line in MIGRATE.txt highlighting that TopDocs#merge shall not support setting shard indices now. Is that the right place for it? > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866328#comment-16866328 ] Atri Sharma commented on LUCENE-8857: - Attached is an updated patch. Does that look inline? [^LUCENE-8857.patch] > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch, LUCENE-8857.patch > > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866253#comment-16866253 ] Simon Willnauer commented on LUCENE-8857: - >From my perspective we should simplify this even more and remove >_TieBreakingParameters_. TopDocs can use _Comparator_ and default >to the shard index if it's not supplied. That should be sufficient? > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch > > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16864258#comment-16864258 ] Atri Sharma commented on LUCENE-8857: - Removed setShardIndex and the ability to set shard index in TopDocs#merge. [^LUCENE-8857.patch] > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch, LUCENE-8857.patch > > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers
[ https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863821#comment-16863821 ] Atri Sharma commented on LUCENE-8857: - Attached is a patch implementing this technique. It adds default implementations for the predicates handling shard index and docID tie breakers. Note that setShardIndex is still present, but is only used for controlling if shard indices should be set or not. It is totally doable to ask TopDocs#merge to set shard indices but still tie break on docIDs. If this approach seems fine, I will add more javadocs and tests. [^LUCENE-8857.patch] > Refactor TopDocs#Merge To Take In Custom Tie Breakers > - > > Key: LUCENE-8857 > URL: https://issues.apache.org/jira/browse/LUCENE-8857 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Attachments: LUCENE-8857.patch > > > In LUCENE-8829, the idea of having lambdas passed in to the API to allow > finer control over the process was discussed. > This JIRA tracks adding a parameter to the API which allows passing in > lambdas to define custom tie breakers, thus allowing users to do custom > algorithms when required. > CC: [~jpountz] [~simonw] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
