[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876069#comment-15876069 ] ASF subversion and git services commented on LUCENE-7628: - Commit cbe7e87d82a5a64fb8b019b215b2c59814ef5462 in lucene-solr's branch refs/heads/branch_6x from [~romseygeek] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=cbe7e87 ] LUCENE-7628: Scorer.getChildren() returns only matching child scorers > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.5 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876070#comment-15876070 ] ASF subversion and git services commented on LUCENE-7628: - Commit ac38872a7916b8df0e218303b439aa4434c1dc52 in lucene-solr's branch refs/heads/master from [~romseygeek] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ac38872 ] LUCENE-7628: Scorer.getChildren() returns only matching child scorers > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.5 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872061#comment-15872061 ] Alan Woodward commented on LUCENE-7628: --- Now that ToParentBlockJoinCollector is gone, I think I can re-apply this patch for 6.5? Running tests now. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.5 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834308#comment-15834308 ] Alan Woodward commented on LUCENE-7628: --- The facet fix is simple enough (it's just in a test). ToParentBlockJoinCollector relies on being able to call this method when the Scorer is unpositioned, though, so I think the change isn't going to work. Back to the drawing board. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.5 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834300#comment-15834300 ] ASF subversion and git services commented on LUCENE-7628: - Commit 94e3460305ae652531fbe55a27158490c55c8f0e in lucene-solr's branch refs/heads/master from [~romseygeek] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=94e3460 ] Revert "LUCENE-7628: Scorer.getChildren() returns only matching Scorers" This reverts commit 5bdc492c9ca8f866d9827d83a05fbab4b95f5ce9. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.5 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834299#comment-15834299 ] ASF subversion and git services commented on LUCENE-7628: - Commit 6818de199e0fe54636e13c3148471780a9a63b0f in lucene-solr's branch refs/heads/branch_6x from [~romseygeek] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6818de1 ] Revert "LUCENE-7628: Scorer.getChildren() returns only matching Scorers" This reverts commit 3c12fadb58574c42efcfa0e44e7603eaa20fc2d4. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.5 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834131#comment-15834131 ] ASF subversion and git services commented on LUCENE-7628: - Commit 3c12fadb58574c42efcfa0e44e7603eaa20fc2d4 in lucene-solr's branch refs/heads/branch_6x from [~romseygeek] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3c12fad ] LUCENE-7628: Scorer.getChildren() returns only matching Scorers > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.4 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15834132#comment-15834132 ] ASF subversion and git services commented on LUCENE-7628: - Commit 5bdc492c9ca8f866d9827d83a05fbab4b95f5ce9 in lucene-solr's branch refs/heads/master from [~romseygeek] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5bdc492 ] LUCENE-7628: Scorer.getChildren() returns only matching Scorers > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.4 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830332#comment-15830332 ] Adrien Grand commented on LUCENE-7628: -- I am still not fond of the change but I guess this is due to the fact that I don't like the current getChildrien() we have in master either. It does not make things worse though. Maybe docs should state that this method is only valid when the scorer is positioned, and AssertingScorer should check it? > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.4 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830268#comment-15830268 ] Alan Woodward commented on LUCENE-7628: --- [~jpountz] are you happy with the patch as it is? We can look at moving things to Weight in another issue. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.4 > > Attachments: LUCENE-7628.patch, LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824036#comment-15824036 ] Alan Woodward commented on LUCENE-7628: --- Ah, I see what you mean. That would still work, I think, although it would probably slow down highlighting batches of documents, as we'd have to create a new Scorer tree for every matching doc in the batch whereas now we can reuse them. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.4 > > Attachments: LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[
https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15824014#comment-15824014
]
Adrien Grand commented on LUCENE-7628:
--
I meant something similar to explain, which takes a docID, eg.
{{Collection Weight.getMatchingChildren(int docID)}}.
> Add a getMatchingChildren() method to DisjunctionScorer
> ---
>
> Key: LUCENE-7628
> URL: https://issues.apache.org/jira/browse/LUCENE-7628
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Minor
> Fix For: 6.4
>
> Attachments: LUCENE-7628.patch
>
>
> This one is a bit convoluted, so bear with me...
> The luwak highlighter works by rewriting queries into their Span-equivalents,
> and then running them with a special Collector. At each matching doc, the
> highlighter gathers all the Spans objects positioned on the current doc and
> collects their positions using the SpanCollection API.
> Some queries can't be translated into Spans. For those queries that generate
> Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on
> the Scorer and see if any of them are SpanScorers, and for those that aren't
> we can call .getChildren() again and recurse down. For each child scorer, we
> check that it's positioned on the current document, so non-matching
> subscorers can be skipped.
> This all works correctly *except* in the case of a DisjunctionScorer where
> one of the children is a two-phase iterator that has matched its
> approximation, but not its refinement query. A SpanScorer in this situation
> will be correctly positioned on the current document, but its Spans will be
> in an undefined state, meaning the highlighter will either collect incorrect
> hits, or it will throw an Exception and prevent hits being collected from
> other subspans.
> We've tried various ways around this (including forking SpanNearQuery and
> adding a bunch of slow position checks to it that are used only by the
> highlighting code), but it turns out that the simplest fix is to add a new
> method to DisjunctionScorer that only returns the currently matching child
> Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's
> a fairly small and contained hack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823821#comment-15823821 ] Alan Woodward commented on LUCENE-7628: --- I'm not sure how moving the introspection API to Weight would work, though, as we need to check subscorers when the parent scorer is positioned on a specific document. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.4 > > Attachments: LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[
https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823736#comment-15823736
]
Adrien Grand commented on LUCENE-7628:
--
I think that would work, but Collector is another API where I'd like to be
careful about adding new methods. I think {{needsScores}} was very compelling
because it enabled significant optimizations as well as merging queries and
filters. I think I would need more compelling use-cases to be convinced about
adding such a new API on Collector. Out of curiosity, would it work for your
use-case if the introspection API was on Weight rather than Scorer? That would
work better for me since Weight is not exposed in Collector like Scorer and
does not have the same performance requirements.
> Add a getMatchingChildren() method to DisjunctionScorer
> ---
>
> Key: LUCENE-7628
> URL: https://issues.apache.org/jira/browse/LUCENE-7628
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Minor
> Fix For: 6.4
>
> Attachments: LUCENE-7628.patch
>
>
> This one is a bit convoluted, so bear with me...
> The luwak highlighter works by rewriting queries into their Span-equivalents,
> and then running them with a special Collector. At each matching doc, the
> highlighter gathers all the Spans objects positioned on the current doc and
> collects their positions using the SpanCollection API.
> Some queries can't be translated into Spans. For those queries that generate
> Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on
> the Scorer and see if any of them are SpanScorers, and for those that aren't
> we can call .getChildren() again and recurse down. For each child scorer, we
> check that it's positioned on the current document, so non-matching
> subscorers can be skipped.
> This all works correctly *except* in the case of a DisjunctionScorer where
> one of the children is a two-phase iterator that has matched its
> approximation, but not its refinement query. A SpanScorer in this situation
> will be correctly positioned on the current document, but its Spans will be
> in an undefined state, meaning the highlighter will either collect incorrect
> hits, or it will throw an Exception and prevent hits being collected from
> other subspans.
> We've tried various ways around this (including forking SpanNearQuery and
> adding a bunch of slow position checks to it that are used only by the
> highlighting code), but it turns out that the simplest fix is to add a new
> method to DisjunctionScorer that only returns the currently matching child
> Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's
> a fairly small and contained hack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823096#comment-15823096 ] Alan Woodward commented on LUCENE-7628: --- I've reverted the change. To keep the API the same size, I can try merging the functionality of getChildren() and getMatchingChildren() (my second suggestion here: https://issues.apache.org/jira/browse/LUCENE-7628?focusedCommentId=15818614=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15818614). On the issue of bulk scoring, maybe we should add a visitsSubScorers() method to Collector, analogous to needsScores(). Then we can enable bulk-scoring or not depending on the needs of the Collector implementation. This would be another way to deal with LUCENE-7365. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Fix For: 6.4 > > Attachments: LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[
https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822947#comment-15822947
]
Adrien Grand commented on LUCENE-7628:
--
I'm not sure about this change since it puts more pressure on the Scorer API.
For instance if scores are not needed, the Scorer does not need to know about
the matching sub clauses, so there could be optimizations based on that, but
that change requires that any Scorer be able to return the list of matching sub
scorers.
By the way, the MinShouldMatchSumScorer impl is buggy since this Scorer
advances the sub scorers lazily: for instance if {{minShouldMatch}} is 2, it
will stop advancing sub clauses as soon as 2 matching scorers are found. The
only scorers will only be advanced if {{score()}} is called. I think that could
be called by calling {{updateFreq()}} before iterating the matching scorers,
like {{score()}} does.
I know you marked this new method as experimental, so we could remove this
method if that ever becomes a problem. However, I have the feeling that
{{getChildren()}} exists for the exact same reason that you are now adding
{{getMatchingChildren()}} so could we remove {{getChildren()}} now? Sorry for
being annoying but I think it is important to keep the Scorer API small.
Could you also add javadocs that this method may only be called from scorers
created though {{Weight.scorer}} and not eg. from collectors? Otherwise there
will be issues if users try to call this API when bulk scorers are used that
pass fake scorers to collectors. We have {{ToParentBlockJoinCollector}} that
uses {{getChildren}}, and it is an issue since it means it cannot work with
BS1, which is one of the most commonly used scorers.
Could we revert this change on the 6.4 branch so that we have time to clean
this up a bit before exposing it to users?
> Add a getMatchingChildren() method to DisjunctionScorer
> ---
>
> Key: LUCENE-7628
> URL: https://issues.apache.org/jira/browse/LUCENE-7628
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Minor
> Fix For: 6.4
>
> Attachments: LUCENE-7628.patch
>
>
> This one is a bit convoluted, so bear with me...
> The luwak highlighter works by rewriting queries into their Span-equivalents,
> and then running them with a special Collector. At each matching doc, the
> highlighter gathers all the Spans objects positioned on the current doc and
> collects their positions using the SpanCollection API.
> Some queries can't be translated into Spans. For those queries that generate
> Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on
> the Scorer and see if any of them are SpanScorers, and for those that aren't
> we can call .getChildren() again and recurse down. For each child scorer, we
> check that it's positioned on the current document, so non-matching
> subscorers can be skipped.
> This all works correctly *except* in the case of a DisjunctionScorer where
> one of the children is a two-phase iterator that has matched its
> approximation, but not its refinement query. A SpanScorer in this situation
> will be correctly positioned on the current document, but its Spans will be
> in an undefined state, meaning the highlighter will either collect incorrect
> hits, or it will throw an Exception and prevent hits being collected from
> other subspans.
> We've tried various ways around this (including forking SpanNearQuery and
> adding a bunch of slow position checks to it that are used only by the
> highlighting code), but it turns out that the simplest fix is to add a new
> method to DisjunctionScorer that only returns the currently matching child
> Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's
> a fairly small and contained hack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822766#comment-15822766 ] Alan Woodward commented on LUCENE-7628: --- That sounds like an interesting idea, [~paul.elsc...@xs4all.nl], but probably for another issue :) > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822372#comment-15822372 ] Paul Elschot commented on LUCENE-7628: -- Hopefully this is not getting too far off topic. In case there is interest in taking DocIdSetIterator out of Spans, please let me know, I have version that passes the lucene tests. I don't know whether it speeds up Spans. The split makes a lot of the Spans code more readable. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15822361#comment-15822361 ] Paul Elschot commented on LUCENE-7628: -- To continue about using Spans directly for this (earlier posted on github, see https://github.com/flaxsearch/luwak/commit/36c91e8bdd3ab0d07578b76359d1f2a87eb53797) Other than AND and OR in the same field, what is also still needed is dealing with multiple fields. For this we need a Spans that can share its DocIdSetIterator with another Spans. Iirc that is what LUCENE-2878 is about, so I'm finally beginning to understand the real point of that issue, and why it is still open. Meanwhile we had DocIdSetIterator split off from Searcher (for speed). How about doing something similar for Spans? I think that would leave Spans pretty close to the Positions of LUCENE-2787. The only change in semantics for Spans would be that at least one of the Spans that share a DocIdSetIterator should provide a real position in a document. Maybe we could have sth like MultiFieldSpans for that. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > Attachments: LUCENE-7628.patch > > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819228#comment-15819228 ] Paul Elschot commented on LUCENE-7628: -- I'll answer myself. With AND and OR available, i.e. the Spans parallels of ConjunctionScorer and DisjunctionScorer, what would still be needed is the Spans parallel of ReqExclScorer, for NOT at document level. Something like ReqExclSpans for a SpanBooleanNotQuery. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819158#comment-15819158 ] Paul Elschot commented on LUCENE-7628: -- And if there was also something like SpanAndMergeQuery that merges the Spans positions when all of them are present in a document? > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819109#comment-15819109 ] Alan Woodward commented on LUCENE-7628: --- You can see the relevant code here: https://github.com/flaxsearch/luwak/blob/master/luwak/src/main/java/uk/co/flax/luwak/util/SpanExtractor.java The problem isn't calling collect() before nextStartPosition(), it's calling nextStartPosition() on a Spans that may not be correctly positioned on a document (even though it's docId() suggests that it is). We can't use SpanOrQuery, unfortunately, because we're pulling Spans from what may be very complicated and deeply-nested booleans. > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[
https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819020#comment-15819020
]
Paul Elschot commented on LUCENE-7628:
--
bq. This all works correctly except in the case of a DisjunctionScorer where
one of the children is a two-phase iterator that has matched its approximation,
but not its refinement query. A SpanScorer in this situation will be correctly
positioned on the current document, but its Spans will be in an undefined
state, meaning the highlighter will either collect incorrect hits, or it will
throw an Exception and prevent hits being collected from other subspans.
Does the highlight code call collect() before nextStartPosition() ?
That should be avoided, see the javadocs of Spans.
For LUCENE-7580 I had a very similar issue, that one computes scores per
matching (i.e. to be highlighted) term occurrence.
The solution there is to split off DisjunctionSpans from SpanOrQuery and to add
these methods:
{code}
public List subSpans() {
return subSpans;
}
public void extractSubSpansAtCurrentDoc(List spansList) {
byPositionQueue.extractSpansList(spansList);
}
public Spans getCurrentPositionSpans() {
return byPositionQueue.top();
}
{code}
With that in place a highlighter could use SpanOrQuery instead of a
BooleanQuery OR, and then the highlighter should be able to do its work.
(Aside: getCurrentPositionSpans() is wrongly named getFirstPositionSpans() at
LUCENE-7580, I'll fix that later).
> Add a getMatchingChildren() method to DisjunctionScorer
> ---
>
> Key: LUCENE-7628
> URL: https://issues.apache.org/jira/browse/LUCENE-7628
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Minor
>
> This one is a bit convoluted, so bear with me...
> The luwak highlighter works by rewriting queries into their Span-equivalents,
> and then running them with a special Collector. At each matching doc, the
> highlighter gathers all the Spans objects positioned on the current doc and
> collects their positions using the SpanCollection API.
> Some queries can't be translated into Spans. For those queries that generate
> Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on
> the Scorer and see if any of them are SpanScorers, and for those that aren't
> we can call .getChildren() again and recurse down. For each child scorer, we
> check that it's positioned on the current document, so non-matching
> subscorers can be skipped.
> This all works correctly *except* in the case of a DisjunctionScorer where
> one of the children is a two-phase iterator that has matched its
> approximation, but not its refinement query. A SpanScorer in this situation
> will be correctly positioned on the current document, but its Spans will be
> in an undefined state, meaning the highlighter will either collect incorrect
> hits, or it will throw an Exception and prevent hits being collected from
> other subspans.
> We've tried various ways around this (including forking SpanNearQuery and
> adding a bunch of slow position checks to it that are used only by the
> highlighting code), but it turns out that the simplest fix is to add a new
> method to DisjunctionScorer that only returns the currently matching child
> Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's
> a fairly small and contained hack.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818614#comment-15818614 ] Alan Woodward commented on LUCENE-7628: --- Turns out it's not as simple as just adding the method, because DisjunctionScorer is package-private and so it can't be accessed from the highlighter code. There are a couple of options I see: * add getMatchingChildren() to Scorer itself - a fairly minimal change (default implementation just forwards to getChildren()), but increases the Scorer API surface area * make DisjunctionScorer.getChildren() only return matching children - this is a bigger change, altering current behaviour and adding IOException to the getChildren() signature, although it's still pretty small in terms of the number of changed lines in the codebase. Any opinions? > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818467#comment-15818467 ] Alan Woodward commented on LUCENE-7628: --- Well, LUCENE-2878 is still open :) Some of the luwak highlighter will probably make it back into core at some point - I think [~dsmiley] is planning on using at least some of it in the UnifiedHighlighter in the future. In the meantime, it's all open source - help yourself! https://github.com/flaxsearch/luwak/blob/master/luwak/src/main/java/uk/co/flax/luwak/matchers/HighlightingMatcher.java > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818456#comment-15818456 ] Dawid Weiss commented on LUCENE-7628: - bq. The luwak highlighter works by rewriting queries into their Span-equivalents, and then running them with a special Collector. Seems like everyone has a highlighter that reinvents the wheel. :) What I just posted to the dev list actually asks for pretty much what you described above. Clearly there is a need for some common infrastructure that would "just" return highlight regions (which can be then turned into larger passages or otherwise manipulated)? > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7628) Add a getMatchingChildren() method to DisjunctionScorer
[ https://issues.apache.org/jira/browse/LUCENE-7628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15818451#comment-15818451 ] Alan Woodward commented on LUCENE-7628: --- See also https://github.com/flaxsearch/luwak/issues/120, and https://github.com/flaxsearch/luwak/commit/36c91e8bdd3ab0d07578b76359d1f2a87eb53797 > Add a getMatchingChildren() method to DisjunctionScorer > --- > > Key: LUCENE-7628 > URL: https://issues.apache.org/jira/browse/LUCENE-7628 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Alan Woodward >Assignee: Alan Woodward >Priority: Minor > > This one is a bit convoluted, so bear with me... > The luwak highlighter works by rewriting queries into their Span-equivalents, > and then running them with a special Collector. At each matching doc, the > highlighter gathers all the Spans objects positioned on the current doc and > collects their positions using the SpanCollection API. > Some queries can't be translated into Spans. For those queries that generate > Scorers with ChildScorers, like BooleanQuery, we can call .getChildren() on > the Scorer and see if any of them are SpanScorers, and for those that aren't > we can call .getChildren() again and recurse down. For each child scorer, we > check that it's positioned on the current document, so non-matching > subscorers can be skipped. > This all works correctly *except* in the case of a DisjunctionScorer where > one of the children is a two-phase iterator that has matched its > approximation, but not its refinement query. A SpanScorer in this situation > will be correctly positioned on the current document, but its Spans will be > in an undefined state, meaning the highlighter will either collect incorrect > hits, or it will throw an Exception and prevent hits being collected from > other subspans. > We've tried various ways around this (including forking SpanNearQuery and > adding a bunch of slow position checks to it that are used only by the > highlighting code), but it turns out that the simplest fix is to add a new > method to DisjunctionScorer that only returns the currently matching child > Scorers. It's a bit of a hack, and it won't be used anywhere else, but it's > a fairly small and contained hack. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
