[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014543#comment-17014543
]
ASF subversion and git services commented on SOLR-13890:
Commit 6e4756fd48076abcecc94446ce3defeb3f907cc1 in lucene-solr's branch
refs/heads/gradle-master from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=6e4756f ]
SOLR-13890: Add "top-level" DV "terms" implementation (#1151)
{!terms} queries have a docValues-based implementation that uses per-segment DV
structures. This does well with a small to moderate (a few hundred) number of
query terms, but doesn't well scale beyond that due to repetitive seeks done on
each segment.
This commit introduces an implementation that uses a "top-level" docValues
structure, which scales much better to very large {!terms} queries (many
hundreds, thousands of terms).
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch, Screen Shot 2020-01-02
> at 2.25.12 PM.png, post_optimize_performance.png,
> toplevel-tpi-perf-comparison.png
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014254#comment-17014254
]
ASF subversion and git services commented on SOLR-13890:
Commit 2393b1ed587ccbf59a5907005b68f389ddcd8bcc in lucene-solr's branch
refs/heads/branch_8x from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2393b1e ]
SOLR-13890: Add "top-level" DV "terms" implementation (#1151)
{!terms} queries have a docValues-based implementation that uses per-segment DV
structures. This does well with a small to moderate (a few hundred) number of
query terms, but doesn't well scale beyond that due to repetitive seeks done on
each segment.
This commit introduces an implementation that uses a "top-level" docValues
structure, which scales much better to very large {!terms} queries (many
hundreds, thousands of terms).
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch, Screen Shot 2020-01-02
> at 2.25.12 PM.png, post_optimize_performance.png,
> toplevel-tpi-perf-comparison.png
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014242#comment-17014242
]
ASF subversion and git services commented on SOLR-13890:
Commit 6e4756fd48076abcecc94446ce3defeb3f907cc1 in lucene-solr's branch
refs/heads/master from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=6e4756f ]
SOLR-13890: Add "top-level" DV "terms" implementation (#1151)
{!terms} queries have a docValues-based implementation that uses per-segment DV
structures. This does well with a small to moderate (a few hundred) number of
query terms, but doesn't well scale beyond that due to repetitive seeks done on
each segment.
This commit introduces an implementation that uses a "top-level" docValues
structure, which scales much better to very large {!terms} queries (many
hundreds, thousands of terms).
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch, Screen Shot 2020-01-02
> at 2.25.12 PM.png, post_optimize_performance.png,
> toplevel-tpi-perf-comparison.png
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009789#comment-17009789
]
Mikhail Khludnev commented on SOLR-13890:
-
bq. Backwards compatible? Does that apply here
Fair. It's private. Haven't noticed it before.
bq. avoid per-segment iteration for performance reasons.
Here in _per-segment iteration_ you mean _reading per-segment ordinals_, since
iteration, scorer, TPI are always per-segment. The patch avoids _reading
per-segment ordinals_ via top-level wrapper SlowAtomicReader, which is a little
bit outdated concept. What code can do: obtain per-segment docvalues, get
segement ordinal per doc, than map it to global ordinal (that's done in
MultiDocValues), than global ordinal might be checked in bitset (if I remember
concept right) .
Then, cast occurrence in file doesn't mean a query doesn't have a condition to
accept Lucene's searcher. As a test case you can use this query in
deleteByQuery.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch, Screen Shot 2020-01-02
> at 2.25.12 PM.png, post_optimize_performance.png,
> toplevel-tpi-perf-comparison.png
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009681#comment-17009681
]
Jason Gerlowski commented on SOLR-13890:
Per a request offline by [~dsmiley], I've created a PR for all subsequent
development on this issue (as it makes in-line review easier). Any patches
attached here predate this PR: https://github.com/apache/lucene-solr/pull/1151
I'll reply to Mikhail's comments here, but maybe further review should be done
on the PR itself:
bq. adding argument to method QueryMethod.makeFilter(String fname, BytesRef[]
bytesRefs, SolrParams localParams) is not something which is backward
compatible, and might frustrate other devs.
Backwards compatible? Does that apply here? We aim to keep backcompat for our
public interfaces, plugins, and SolrJ, but this is neither of those. It's a
private nested class not visible outside this one file. Is there some reason
I'm missing why we should care about backcompat here?
bq. TopLevelDocValuesTermsQuery uses OrdinalMap via getSlowAtomicReader(). It
might be clearer to iterate persegment
Maybe I'm misreading your suggestion, but the whole purpose of this issue is
that we're trying to avoid per-segment iteration for performance reasons. I'd
be happy to change gears if you have an alternative that has comparable
performance to what we're seeing with the global iteration, but our perf tests
have borne out global-iteration as the more efficient approach at large numbers
of query terms.
bq. Also, this query relies on SolrIndexSearcher, but iirc even in Solr queries
sometimes invoked with Lucene's Searcher. There's some issues with such cast
I'm still reading through SOLR-6357 to understand the exact context here. But
the cast to SolrIndexSearcher in QParserPlugins and query implementations is
very common in our codebase (see below). If you think it'd be safer, I can add
an {{instanceof}} check there, and try to fall back to the per-segment approach
if we ever get a non-SolrIndexSearcher. But from how frequently this is done
in our query implementations, I'm not sure the danger is still there?
{code}
➜ lucene-solr git:(SOLR_13890) ✗ grep -rIl "(SolrIndexSearcher)[ ]\?searcher" .
./solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java
./solr/core/src/java/org/apache/solr/search/TextLogisticRegressionQParserPlugin.java
./solr/core/src/java/org/apache/solr/search/SignificantTermsQParserPlugin.java
./solr/core/src/java/org/apache/solr/search/IGainTermsQParserPlugin.java
./solr/core/src/java/org/apache/solr/search/GraphTermsQParserPlugin.java
./solr/core/src/java/org/apache/solr/search/join/GraphQuery.java
./solr/core/src/java/org/apache/solr/search/join/HashRangeQuery.java
./solr/core/src/java/org/apache/solr/search/join/XCJFQuery.java
./solr/core/src/java/org/apache/solr/search/QueryContext.java
./solr/core/src/java/org/apache/solr/search/ReRankCollector.java
./solr/core/src/java/org/apache/solr/search/HashQParserPlugin.java
./solr/core/src/java/org/apache/solr/search/JoinQParserPlugin.java
./solr/core/src/java/org/apache/solr/search/TermsQParserPlugin.java
./solr/core/src/java/org/apache/solr/query/SolrRangeQuery.java
./solr/core/src/java/org/apache/solr/query/FilterQuery.java
{code}
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch, Screen Shot 2020-01-02
> at 2.25.12 PM.png, post_optimize_performance.png,
> toplevel-tpi-perf-comparison.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009194#comment-17009194
]
Mikhail Khludnev commented on SOLR-13890:
-
regarding {{PerSegmentViewDocIdSetIterator}}: I don't follow. Lucene's
{{DocIdSetIterator}} is strictly per-segment, using it for top-level iteration
is something that never happen. fwiw, usually toplevel Solr docsets converted
to Lucene's DocIdSets via {{DocSet.getTopFilter()}}.
adding argument to method {{QueryMethod.makeFilter(String fname, BytesRef[]
bytesRefs, SolrParams localParams)}} is not something which is backward
compatible, and might frustrate other devs.
Note: {{TopLevelDocValuesTermsQuery}} uses {{OrdinalMap}} via
{{getSlowAtomicReader()}}. It might be clearer to iterate persegment, and then
access global ordinals via MultiSortedDocValues.mapping.getGlobalOrds()
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch, Screen Shot 2020-01-02
> at 2.25.12 PM.png, post_optimize_performance.png,
> toplevel-tpi-perf-comparison.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009170#comment-17009170
]
Jason Gerlowski commented on SOLR-13890:
Latest patch ties up some of the loose ends I mentioned in my last comment.
Pending review from you guys, I'm pretty happy pulling the trigger on what
we've got right now. We get the good performance I was after without
introducing another postfilter.
Pending more feedback I'll aim to merge this on Wednesday.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png, toplevel-tpi-perf-comparison.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17008415#comment-17008415
]
David Smiley commented on SOLR-13890:
-
Linking SOLR-14166 which I spent some time on this weekend concerning how to
get Solr to use TwoPhaseIterator in a scenario where it's not really today --
multiple non-cached, non-PostFilter FQs.
I don't think we can actually remove PostFilter, nor should we, because it's
necessary for queries like Collapse & some others that must have "lookAhead"
into the incoming documents before it can then decide what docs to make
visible. Still; I don't think this abstraction should be used where a normal
Query can easily be used.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007798#comment-17007798
]
Joel Bernstein commented on SOLR-13890:
---
[~mkhl], I'd like to get this segment level cache in use for all filter queries.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007754#comment-17007754
]
Mikhail Khludnev commented on SOLR-13890:
-
Note, I hardly follow whole discussion, but regarding segmented filter cache I
can comment that there's one already. Please find here
https://lucene.apache.org/solr/guide/8_2/other-parsers.html#all-parents-syntax
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007737#comment-17007737
]
Joel Bernstein commented on SOLR-13890:
---
If we auto-default cache=false then I'm fine with moving the top level
docvalues / TPI approach. Let's just give the users a terms query thats fast
and works well with a *large number of terms*, *larger indexes, large number of
filters* in a *frequently indexing environment.* These goals cannot be achieved
with the current filter cache. So let's turn it off by default for this
implementation.
Then let's get a segment level filter cache in place.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007714#comment-17007714
]
David Smiley commented on SOLR-13890:
-
I could pick apart lots of statements made but I choose these:
bq. I'm still thinking about the postfilter vs TPI question. The downside of
continuing with postfilter is low, since Solr has a handful of others already
and no one has shown interest in removing them.
I have an interest in removing them, albeit slowly. I've already removed a
spatial one on a feature branch related to spatial deprecations generally
(there's an existing issue); targeting 9.0. FuntionRangeQuery is one on my
mind, and I have a patch in progress.
bq. And there's a bit of an advantage to doing postfilter here too: in that it
lets users pick between top-level and per-segment logic as they'd like without
requiring any additional params.
What the parameters and their defaults is entirely up to us right here.
bq. Eliminating the postfilter means, in my opinion overloading the behavior of
traditional filters to have to deal intelligently with caches, or asking the
user to dive into the details of caching.
What do you mean by "traditional filters"? If you mean "fq", then to be clear
I'm totally accepting of having the queries auto-default cache=false depending
on whatever. There are trade-offs but I'm good with having defaults that are
more optimal for more users, even if that may make it harder to reason about
what that default is. This is orthogonal to TPI vs PostFilter or per-segment
vs top-level.
In a nutshell, PostFilter was once the only way to "post filter" but has become
entirely redundant (correct me if I'm wrong on that). Even if someone likes
this interface and likes its documentation, it nonetheless exists in a large
search stack where there is already a solution, and is thus extra cognitive
overhead (complexity) to think about and to maintain in Solr itself
(SolrIndexSearcher has to know about it).
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007636#comment-17007636
]
Joel Bernstein commented on SOLR-13890:
---
The problem that I have with the scorer based filter is only about the use of
the filter cache. Traditionally with filters the filter cache is used and with
postfilters the filter cache is not used. This actually a nice split, it's
clean, easy to implement and easy to document.
For these types of large filters, most of the time, the filter cache is not fit
for purpose. But its there and it traps people, I've seen it over and over
again.
The issue is that as the indexes get larger the top level filter cache becomes
untenable because the filter is built against the entire index. Then when one
document is indexed the entire cache is dropped and needs to be warmed. This is
a huge problem for use cases like access control where there are large filters
that need to get rebuilt for thousands of users. This drains memory and cpu and
cause GC issues and performance problems.
So, I have traditionally turned to high performant postfilters, that are fast
enough that caching is not needed for these types of problems.
Eliminating the postfilter means, in my opinion overloading the behavior of
traditional filters to have to deal intelligently with caches, or asking the
user to dive into the details of caching.
The other option is add the segment level filter cache that is actually fit for
purpose and then standardize on the filter based approach. Until we do this,
though the postfilters provide a simple approach for getting the behavior that
is needed for these types of large filters.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007627#comment-17007627
]
Jason Gerlowski commented on SOLR-13890:
bq. You characterize the two above as "Existing Impl" vs "Postfilter
Impl"[but] the differentiator is per-segment algorithm vs top-level
algorithm
In hindsight, you're right. I chose the labels I did because I set the graph
up before looking at the experiment results. My mistake.
bq. keep method=docValuesTermsFilter but have it choose between these two
implementations based on the number of terms; 700 being the pivot
It'd be really cool to have {{terms}} be smart like this, but I've got very
little trust in 700 as a general pivot. In work with customers Joel and I have
seen the pivot point happen both earlier and later depending on load, IO speed,
index size and cardinality, numDocs matched by already processed query clauses,
etc. With more benchmarking I think we could choose a more informed pivot
value, but it'd take more time than I can spend right now. But maybe not, I'll
think about it.
I'm still thinking about the postfilter vs TPI question. The downside of
continuing with postfilter is low, since Solr has a handful of others already
and no one has shown interest in removing them. And there's a bit of an
advantage to doing postfilter here too: in that it lets users pick between
top-level and per-segment logic as they'd like without requiring any additional
params. But of course there's downsides too...
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007566#comment-17007566
]
David Smiley commented on SOLR-13890:
-
Thanks for the benchmarks and postings the charts. This is really helpful!
You characterize the two above as "Existing Impl" vs "Postfilter Impl". As
I've been saying (and validated by your benchmark) the "Postfilter" aspect is
not the interesting part as it's roughly equivalent to TwoPhaseIterator (which
is in the existing impl). The differentiator is per-segment algorithm vs
top-level algorithm. And you could do that via a TwoPhaseIterator and avoid
Solr's old PostFilter. I consider PostFilter to be a relic to be avoided if
you can do so. There's special code in SolrIndexSearcher about PostFilter to
maintain and perhaps one day we can make it go away to reduce complexity.
Continuing to use PostFilter needlessly will make that harder. I'm not going
to veto use of PostFilter if you want to stick with what you have already
coded, but I don't recommend it.
I suggest you:
* retain the existing implementation
* add your new implementation (_potentially_ redeveloped to not use PostFilter)
* keep method=docValuesTermsFilter but have it choose between these two
implementations based on the number of terms; 700 being the pivot. Maybe make
the pivot configurable and/or add a mechanism to explicitly choose one or the
other. FWIW TermInSetQuery uses a non-configurable heuristic so don't feel you
_have_ to make the threshold here configurable.
* manually verify that an explicit cache=true/false has the intended effect for
both impls
I don't know wether to change the _default_ cache local-param to false. Seems
like kind of a larger issue... like if we wanted this then it ought to apply to
any query that does not use an index, not just some queries this parser
produces. On the other hand, it's nice to have a simple understood rule that
all filter queries are cached unless you say otherwise. Shrug.
Ideally this QParser would be a bit smarter about choosing an optimal "method".
If there is no terms index but there is docValues, then you should get a
docValues impl instead of what appears to be a no-results query (ouch!). If
there's both data structures, and the user sets cache=false and the number of
terms is large, then the method should use docValues. Of course this is out of
scope.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17007449#comment-17007449
]
Jason Gerlowski commented on SOLR-13890:
bq. If other lower cost queries are in play then TPI matches() won't be called
if the document can be excluded by them.
Did some quick testing on this, and it looks like you're right (and my initial
reading was wrong). At least, with a caveat. The behavior is as you described
when run with {{cache=false}}. When the query is run with {{cache=true}} (or
unspecified) and the query isn't cached yet, {{matches}} _is_ called for every
doc in the index so that the filter can be cached.
Now to the interesting bit. I wrote a JUnit driver to perf-compare DVTQ's
existing TPI implementation and the proposed postfilter implementation. For
details on how the perf test was set up, see the latest patch. The driver
indexes data and submits increasingly larger {{terms}} queries, measuring the
performance with both approaches.
The graphs below show the results of a few of these runs, with numTerms
increasing left to right and QTime measured on the Y axis. DVTQ's per-segment
TPI implementation is shown in blue, postfilter performance is in red.
In a "normal" run the TPI implementation starts out more performant for small
"numTerms" values, but its QTime increases linearly as the size of the "terms"
query increases. At around 700 terms the postfilter implementation takes and
keeps the lead.
!Screen Shot 2020-01-02 at 2.25.12 PM.png!
But after an optimize, with the whole index in one segment, this difference
disappears, confirming what David suspected. So if we wanted to replace the
post-filter implementation with a top-level TPI implementation, we'd see the
same performance.
!post_optimize_performance.png!
The question remains though whether we should stick with postfilter or switch
to TPI here though. Both approaches really suffer performance-wise when the
query matches a large number of documents. That makes me wonder whether we
might be better off with "postfilter", which has a very explicit switch that
users can control (unlike DVTQ which seems to always operate using TPI). But
that's just a thought, I was a little surprised to see the post-optimize perf
is exactly the same, so I'm a bit flatfooted how to proceed.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch,
> SOLR-13890.patch, Screen Shot 2020-01-02 at 2.25.12 PM.png,
> post_optimize_performance.png
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002960#comment-17002960
]
David Smiley commented on SOLR-13890:
-
bq. The TPI "approximation" for DocValuesTermsQuery is the unfiltered
doc-values structure for the field. As a result TPI matches() is going to be
called on all documents that have any value at all for the field in question.
Under a post-filter implementation, the bitset lookup is (potentially) called
much less frequently, as we only lookup values for docs that have matched all
the other (non-postfilter) query clauses. Does that make sense, or am I
off-base David Smiley?
I don't think this characterization is accurate. If other lower cost queries
are in play then TPI matches() won't be called if the document can be excluded
by them.
bq. So, the postfilters behavior (not cached in filter cache) provides the best
solution for certain situations where the filter cache is problematic.
We can make an estimation that it's best to not cache; the Query could
implement ExtendedQuery to return a default getCache() of false sometimes.
Perhaps always default to false... maybe all O(docs) queries should default
this way. Perhaps a better heuristic, is how long the IndexSearcher has been
open for. Regardless the user can & should retain the ability to be explicit
if he/she chooses.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002424#comment-17002424
]
Joel Bernstein commented on SOLR-13890:
---
The other really big aspect of this is caching.
Even though scorer based filter can be fast if it's applied with the main
query, in Solr that's not going to happen.
The reason is the filter cache. Which will apply the filter against the entire
index and create a DocSet to cache.
Our filter cache is top level so it gets dumped after a single document is
loaded. So scenarios where there is lot's of indexing going the filter cache
becomes problematic.
There are ways around this issue, like turning off caching using local params,
or not using filter queries. But these approaches are not what users typically
do with a filter.
So, the postfilters behavior provides the best solution for certain situations
where the filter cache is problematic.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002420#comment-17002420
]
Joel Bernstein commented on SOLR-13890:
---
We have code somewhat similar to this patch deployed with a cross-core join
that provides sub-second performance with 50,000 join terms. We will not
achieve that with the terms query because 50,000 terms is too large to pass in
efficiently, but the term lookups are scalable with the top level ordinal
approach.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002416#comment-17002416
]
Joel Bernstein commented on SOLR-13890:
---
I dug into this pretty deeply and I believe there is large advantage to top
level doc values approach when there is a large number of terms. The reason is
that *MultiSortedSetDocValues.lookupOrd* (in MultiDocValues) is really clever,
so the overhead of doing the top level term lookup is much less than doing the
segment by segment term lookups. Using the top level ordinals inside of the
scorer would be possible also but seemed kind of awkward. But, in theory using
top level ordinals in the scorer would get as similar performance as this patch.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17002388#comment-17002388
]
Jason Gerlowski commented on SOLR-13890:
bq. I highly doubt the PostFilter abstraction somehow offers a perf benefit in
your benchmark that cannot be achieved with TwoPhaseIterator
I'm leaning on your correction a bit here as you're more familiar with the
Lucene code than I am. But as I read the TPI implementation for
DocValuesTermsQuery, I see one reason why a postfilter impl might be faster
(other than segment-level vs top-level)
The TPI "approximation" for DocValuesTermsQuery is the unfiltered doc-values
structure for the field. As a result TPI {{matches()}} is going to be called
on all documents that have any value at all for the field in question. Under a
post-filter implementation, the bitset lookup is (potentially) called much less
frequently, as we only lookup values for docs that have matched all the other
(non-postfilter) query clauses. Does that make sense, or am I off-base
[~dsmiley]?
In either case, this is hypothetical. The real proof is in a perf experiment.
I'm putting one together now to share soon.
bq. Though I don't know whether the details of my test would have tripped
whatever heuristics Lucene uses to turn TPI on/off.
As best as I can tell from the
[code|https://github.com/apache/lucene-solr/blob/174cc63bad411eace196a6c7028bdd24864fefed/lucene/sandbox/src/java/org/apache/lucene/search/DocValuesTermsQuery.java#L218],
it looks like DVTQ always uses TPI processing. So there's no particular
concern about ensuring that logic is triggered when I perf test.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988831#comment-16988831
]
David Smiley commented on SOLR-13890:
-
I think PostFilterDocValuesTermsQuery needs to justify it's existence in a
javadoc. Such a javadoc should acknowledge that DVTQ has a TwoPhaseIterator
and acknowledge it's similarity to a PostFilter (if you continue with your
PostFilter approach, otherwise it's irrelevant).
The likely explanation for a performance improvement is top-level vs
per-segment algorithm. I _highly doubt_ the PostFilter abstraction somehow
offers a perf benefit in your benchmark that cannot be achieved with
TwoPhaseIterator. You could benchmark/test this easily by simply doing a
segment optimize (forceMerge) and seeing if the performance suddenly becomes
the same. This is because a single segment scenario makes the two algorithmic
choices equivalent. To do such a top-level algorithm with TwoPhaseIterator,
you'd override createWeight and resolve the top level ordinals up front there.
Lucene has no query logs. It'd be nice if query explain emitted wether it uses
TPI but it does not. I'm not arguing against users having the ability to
_force_ TPI if they choose; that's quite doable.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988801#comment-16988801
]
Jason Gerlowski commented on SOLR-13890:
bq. I haven't investigated how feasible it is but I wonder if Solr even needs
PostFilter given TwoPhaseIterator exists. For another day.
It's a good question. I'd imagine that two things are necessary if we wanted
to replace Solr's postfilter:
# We'd need to make sure that TPI implementations provide the same performance
gains as postfilter ones. I wouldn't have considered this previously. But
knowing that DVTQ already has TPI, and recalling the gains we saw with our
postfilter in (unpublished) perf tests is enough to plant a seed of doubt for
me.
# We'd need to see whether users are fine ceding control of when this special
execution mode is triggered. TPI being heuristic-triggered seems double-edged
to me if a user finds themselves fighting those heuristics on a query they know
would benefit from TPI. Though maybe there's an override flag I'm just not
aware of that forces TPI to be used/ignored.
That said, "for another day" for sure.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988786#comment-16988786
]
Jason Gerlowski commented on SOLR-13890:
I did profile against "standard" DVTQ method, and found the postfilter
implementation to be significantly more performant at very large numbers of
terms. Though I don't know whether the details of my test would have tripped
whatever heuristics Lucene uses to turn TPI on/off. Is there any logging
you're aware of that indicates whether Lucene ended up using TwoPhaseIterator
for a particular query?
I'll hold off on committing until I have time to re-run this perf experiment
and can share the results for inspection.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988491#comment-16988491
]
David Smiley commented on SOLR-13890:
-
DocValuesTermsQuery already internally uses TwoPhaseIterator which is Lucene's
answer to Solr's PostFilter. TwoPhaseIterator has a cool advantage in its
ability to automatically be employed by the Query that uses it conditionally
based on stats/weights. Compared this to PostFilter which the user must know
to even use and then it's not always the ideal choice to use it. Did you test
with the existing implementation on an fq with terms and cache=false to thus
mix in the query with the main query? I think it should perform decently. I
haven't investigated how feasible it is but I wonder if Solr even needs
PostFilter given TwoPhaseIterator exists. For another day.
Another difference I see is that you're doing a top level reader approach
instead of per-segment. Both are "valid"; it's a trade-off between real-time
vs static index.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988409#comment-16988409
]
Jason Gerlowski commented on SOLR-13890:
Latest patch updates the docs and does some small cleanup. Tests pass and I'm
ready to commit this. Targeting this weekend or early next week.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974534#comment-16974534
]
Jason Gerlowski commented on SOLR-13890:
Thanks Mikhail, I'll update my pass at the docs soon!
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
[jira] [Commented] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16973952#comment-16973952
]
Mikhail Khludnev commented on SOLR-13890:
-
FYI, Automaton uses inverted index, it works per segment. If number of terms is
small it builds per segment conjunction, thus if it bypasses filtercache it
will be even lazy, otherwise it eagerly builds per-segment docset.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
