[jira] [Comment Edited] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572569#comment-17572569
]
Deepika Sharma edited comment on LUCENE-10544 at 7/28/22 5:16 PM:
--
Since query timeout feature is implemented through {{BulkScorer}} class, should
there be changes made such that {{ExitableTermsEnum}} wraps
{{postings/impacts}} along with timeout checks in it?
was (Author: JIRAUSER288832):
Since query timeout feature is implemented through `BulkScorer` class, should
there be changes made such that `ExitableTermsEnum` wraps `postings/impacts`
along with timeout checks in it?
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572569#comment-17572569
]
Deepika Sharma edited comment on LUCENE-10544 at 7/28/22 5:15 PM:
--
Since query timeout feature is implemented through `BulkScorer` class, should
there be changes made such that `ExitableTermsEnum` wraps `postings/impacts`
along with timeout checks in it?
was (Author: JIRAUSER288832):
Since query timeout feature is implemented through BulkScorer class, should
there be changes made such that ExitableTermsEnum wraps postings/impacts
along with timeout checks in it?
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17572569#comment-17572569
]
Deepika Sharma commented on LUCENE-10544:
-
Since query timeout feature is implemented through BulkScorer class, should
there be changes made such that ExitableTermsEnum wraps postings/impacts
along with timeout checks in it?
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10151) Add timeout support to IndexSearcher
[
https://issues.apache.org/jira/browse/LUCENE-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543885#comment-17543885
]
Deepika Sharma edited comment on LUCENE-10151 at 5/30/22 11:08 AM:
---
I have raised a PR for the {{BulkScorer}} approach. Please take a look and let
me know your thoughts.
was (Author: JIRAUSER288832):
I have raised a PR for the `BulkScorer` approach. Please take a look and let me
know your thoughts.
> Add timeout support to IndexSearcher
>
>
> Key: LUCENE-10151
> URL: https://issues.apache.org/jira/browse/LUCENE-10151
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
>Reporter: Greg Miller
>Priority: Minor
>
> I'd like to explore adding optional "timeout" capabilities to
> {{IndexSearcher}}. This would enable users to (optionally) specify a maximum
> time budget for search execution. If the search "times out", partial results
> would be available.
> This idea originated on the dev list (thanks [~jpountz] for the suggestion).
> Thread for reference:
> [http://mail-archives.apache.org/mod_mbox/lucene-dev/202110.mbox/%3CCAL8PwkZdNGmYJopPjeXYK%3DF7rvLkWon91UEXVxMM4MeeJ3UHxQ%40mail.gmail.com%3E]
>
> A couple things to watch out for with this change:
> # We want to make sure it's robust to a two-phase query evaluation scenario
> where the "approximate" step matches a large number of candidates but the
> "confirmation" step matches very few (or none). This is a particularly tricky
> case.
> # We want to make sure the {{TotalHits#Relation}} reported by {{TopDocs}} is
> {{GREATER_THAN_OR_EQUAL_TO}} if the query times out
> # We want to make sure it plays nice with the {{LRUCache}} since it iterates
> the query to pre-populate a {{BitSet}} when caching. That step shouldn't be
> allowed to overrun the timeout. The proper way to handle this probably needs
> some thought.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10151) Add timeout support to IndexSearcher
[
https://issues.apache.org/jira/browse/LUCENE-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17543885#comment-17543885
]
Deepika Sharma commented on LUCENE-10151:
-
I have raised a PR for the `BulkScorer` approach. Please take a look and let me
know your thoughts.
> Add timeout support to IndexSearcher
>
>
> Key: LUCENE-10151
> URL: https://issues.apache.org/jira/browse/LUCENE-10151
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
>Reporter: Greg Miller
>Priority: Minor
>
> I'd like to explore adding optional "timeout" capabilities to
> {{IndexSearcher}}. This would enable users to (optionally) specify a maximum
> time budget for search execution. If the search "times out", partial results
> would be available.
> This idea originated on the dev list (thanks [~jpountz] for the suggestion).
> Thread for reference:
> [http://mail-archives.apache.org/mod_mbox/lucene-dev/202110.mbox/%3CCAL8PwkZdNGmYJopPjeXYK%3DF7rvLkWon91UEXVxMM4MeeJ3UHxQ%40mail.gmail.com%3E]
>
> A couple things to watch out for with this change:
> # We want to make sure it's robust to a two-phase query evaluation scenario
> where the "approximate" step matches a large number of candidates but the
> "confirmation" step matches very few (or none). This is a particularly tricky
> case.
> # We want to make sure the {{TotalHits#Relation}} reported by {{TopDocs}} is
> {{GREATER_THAN_OR_EQUAL_TO}} if the query times out
> # We want to make sure it plays nice with the {{LRUCache}} since it iterates
> the query to pre-populate a {{BitSet}} when caching. That step shouldn't be
> allowed to overrun the timeout. The proper way to handle this probably needs
> some thought.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17540056#comment-17540056
]
Deepika Sharma commented on LUCENE-10544:
-
For adding timeout feature In {{IndexSearcher}} through {{BulkScorer}}
approach, I think there are two ways we can implement this:
* One way could be we can make another class which will wrap {{BulkScorer}}
along with timeout and other methods as per required (similar to existing
{{TimeLimitingCollector}} class).
* Another way could be making a helper function inside {{BulkScorer}} class
which will wrap score method of {{BulkScorer}} along with timeout and it will
invoke whenever timeout is enabled.
So, I am looking for the suggestion which approach would be better to go ahead
or is there any other way I should consider?
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538981#comment-17538981
]
Deepika Sharma edited comment on LUCENE-10544 at 5/18/22 5:36 PM:
--
Thanks [~jpountz] for sharing this approach. I also feel this approach seems to
me more generic in terms of handling all types of query.
So what I currently understand is to have basically have some sort of a wrapper
class around a {{BulkScorer}} which does the timeout checks inside the
{{score}} method? Is this method similar to what is being done for all those
{{*Enum}} classes, where we have a wrapper which takes an instance and does
timeout checks and then calls the wrapper object's methods?
was (Author: JIRAUSER288832):
Thanks [~jpountz] for sharing this approach. I also feel this approach seems to
me more generic in terms of handling all type of query. So what I currently
understand is to have basically have some sort of a wrapper class around a
{{BulkScorer}} which does the timeout checks inside the {{score}} method? Is
this method somewhat similar to what is being done for all those {{*Enum}}
classes, where you have a wrapper which takes an instance, does something extra
(timeout checks in this case) and then calls the wrapper object's methods?
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538981#comment-17538981
]
Deepika Sharma commented on LUCENE-10544:
-
Thanks [~jpountz] for sharing this approach. I also feel this approach seems to
me more generic in terms of handling all type of query. So what I currently
understand is to have basically have some sort of a wrapper class around a
{{BulkScorer}} which does the timeout checks inside the {{score}} method? Is
this method somewhat similar to what is being done for all those {{*Enum}}
classes, where you have a wrapper which takes an instance, does something extra
(timeout checks in this case) and then calls the wrapper object's methods?
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529965#comment-17529965
]
Deepika Sharma edited comment on LUCENE-10544 at 5/18/22 5:34 PM:
--
Yeah, I think you’re right [~jpountz] about the BulkScorer#score. One edge case
though would probably be if a user passes their own BulkScorer, in which case
this approach might not work properly. I guess what we could do is to allow a
user to use a custom BulkScorer, when timeout is enabled, but this might not be
a desirable restriction.
was (Author: JIRAUSER288832):
Yeah, I think you’re right [~jpountz] about the BulkScorer#score. One edge case
though would probably be if a user passes their own BulkScorer, in which case
this approach might not work properly. I guess what we could do is to allow a
user to use a custom BulkScorer, when timeout is enabled, but this might not be
a desirable restriction.
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537457#comment-17537457
]
Deepika Sharma commented on LUCENE-10544:
-
For unit test I am thinking that since in the existing implementation of
{{{}ExitableDirectoryReader{}}}, timeout is checked only at the time of
instantiating {{BulkScorer}} so either we receive all the required number of
results or 0 result based on the fact that whether timeout exceeded while
instantiating {{BulkScorer}} or not. If once we escaped timeout exceeding while
instantiating {{BulkScorer}} then we receive all the results. Otherwise we
receive 0 result because we don’t enter {{#scoreAll}} to iterate with
{{{}DocIdSetIterator{}}}.
Considering the above fact, it can be easily observed that it is not possible
to get partial results in case of single leaf until timeout is checked inside
{{#scoreAll}} while iterating with {{DocIdSetIterator}} .
So, if in the test we are able to get partial result then it will ensure that
timeout check is now being done inside {{#scoreAll}} also.
Is there any other way around that I should consider for testing?
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17536624#comment-17536624
]
Deepika Sharma commented on LUCENE-10544:
-
I am currently working on wrapping postings/impacts with {{ExitableTermsEnum}}
in {{{}ExitableDirectoryReader{}}}. I am looking suggestion for unit test
regarding this.
More context regarding the approach I have done is:
To wrap postings/impacts with {{{}ExitableTermsEnum{}}}, I tried overriding
other methods of {{FilterTermsEnum}} with timeout check in
{{ExitableTermsEnum}} along with already overridden {{next()}} method. Also,
{{PostingsEnum}} and {{ImpactsEnum}} needs to be wrapped. So, for this I
created {{ExitableImpactsEnum }}extended from {{ImpactsEnum}} and similarly
{{ExitablePostingsEnum}} extended from {{PostingsEnum}} in
{{ExitableDirectoryReader}} class where all the methods of the respective
classes are overridden along with timeout checks.
Also, modified {{ExitableTermsEnum}} such that when {{#impacts(...)}} is
invoked then we return instance of {{{}ExitableImpactsEnum{}}}. Similarly, when
{{#postings(...)}} is invoked then we return instance of
{{{}ExitablePostingsEnum{}}}.
It was observed that for {{ExitableImpactsEnum}} and {{ExitablePostingsEnum}}
are getting invoked through {{#scoreAll}} for {{TermQuery}} and {{PrefixQuery}}
respectively.
One problem that I have observed is, for any particular timeout value the no.
of results are not fixed. For example, I tried running the test with timeout
value 100 and I got all the results, but at some subsequent running of test
with same timeout value I received 0, 3, and 5 (different number) results
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-10151) Add timeout support to IndexSearcher
[
https://issues.apache.org/jira/browse/LUCENE-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532223#comment-17532223
]
Deepika Sharma edited comment on LUCENE-10151 at 5/5/22 12:16 PM:
--
I am exploring adding timeout support to the {{IndexSearcher}} by using
{{ExitableDirectoryReader.}} However, one issue with
{{ExitableDirectoryReader}} is that it enforces timeout checking at the time of
instantiating {{BulkScorer}} and doesn't actually enforce it once you start
iterating postings/impacts. This is being discussed in LUCENE-10544
I want to ask if there are any suggestions on alternative ways to approach this
problem that I should consider?
was (Author: JIRAUSER288832):
I am exploring adding timeout support to the {{IndexSearcher}} by using
{{ExitableDirectoryReader.}} However, one issue with
{{ExitableDirectoryReader}} is that it enforces timeout checking at the time of
instantiating {{BulkScorer}} and doesn't actually enforce it once you start
iterating postings/impacts. This is being discussed in
[LUCENE-10544|https://issues.apache.org/jira/browse/LUCENE-10544]
I want to ask if there are any suggestions on alternative ways to approach this
problem that I should consider?{{{}{}}}
> Add timeout support to IndexSearcher
>
>
> Key: LUCENE-10151
> URL: https://issues.apache.org/jira/browse/LUCENE-10151
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
>Reporter: Greg Miller
>Priority: Minor
>
> I'd like to explore adding optional "timeout" capabilities to
> {{IndexSearcher}}. This would enable users to (optionally) specify a maximum
> time budget for search execution. If the search "times out", partial results
> would be available.
> This idea originated on the dev list (thanks [~jpountz] for the suggestion).
> Thread for reference:
> [http://mail-archives.apache.org/mod_mbox/lucene-dev/202110.mbox/%3CCAL8PwkZdNGmYJopPjeXYK%3DF7rvLkWon91UEXVxMM4MeeJ3UHxQ%40mail.gmail.com%3E]
>
> A couple things to watch out for with this change:
> # We want to make sure it's robust to a two-phase query evaluation scenario
> where the "approximate" step matches a large number of candidates but the
> "confirmation" step matches very few (or none). This is a particularly tricky
> case.
> # We want to make sure the {{TotalHits#Relation}} reported by {{TopDocs}} is
> {{GREATER_THAN_OR_EQUAL_TO}} if the query times out
> # We want to make sure it plays nice with the {{LRUCache}} since it iterates
> the query to pre-populate a {{BitSet}} when caching. That step shouldn't be
> allowed to overrun the timeout. The proper way to handle this probably needs
> some thought.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10151) Add timeout support to IndexSearcher
[
https://issues.apache.org/jira/browse/LUCENE-10151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17532223#comment-17532223
]
Deepika Sharma commented on LUCENE-10151:
-
I am exploring adding timeout support to the {{IndexSearcher}} by using
{{ExitableDirectoryReader.}} However, one issue with
{{ExitableDirectoryReader}} is that it enforces timeout checking at the time of
instantiating {{BulkScorer}} and doesn't actually enforce it once you start
iterating postings/impacts. This is being discussed in
[LUCENE-10544|https://issues.apache.org/jira/browse/LUCENE-10544]
I want to ask if there are any suggestions on alternative ways to approach this
problem that I should consider?{{{}{}}}
> Add timeout support to IndexSearcher
>
>
> Key: LUCENE-10151
> URL: https://issues.apache.org/jira/browse/LUCENE-10151
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
>Reporter: Greg Miller
>Priority: Minor
>
> I'd like to explore adding optional "timeout" capabilities to
> {{IndexSearcher}}. This would enable users to (optionally) specify a maximum
> time budget for search execution. If the search "times out", partial results
> would be available.
> This idea originated on the dev list (thanks [~jpountz] for the suggestion).
> Thread for reference:
> [http://mail-archives.apache.org/mod_mbox/lucene-dev/202110.mbox/%3CCAL8PwkZdNGmYJopPjeXYK%3DF7rvLkWon91UEXVxMM4MeeJ3UHxQ%40mail.gmail.com%3E]
>
> A couple things to watch out for with this change:
> # We want to make sure it's robust to a two-phase query evaluation scenario
> where the "approximate" step matches a large number of candidates but the
> "confirmation" step matches very few (or none). This is a particularly tricky
> case.
> # We want to make sure the {{TotalHits#Relation}} reported by {{TopDocs}} is
> {{GREATER_THAN_OR_EQUAL_TO}} if the query times out
> # We want to make sure it plays nice with the {{LRUCache}} since it iterates
> the query to pre-populate a {{BitSet}} when caching. That step shouldn't be
> allowed to overrun the timeout. The proper way to handle this probably needs
> some thought.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-10544) Should ExitableTermsEnum wrap postings and impacts?
[
https://issues.apache.org/jira/browse/LUCENE-10544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529965#comment-17529965
]
Deepika Sharma commented on LUCENE-10544:
-
Yeah, I think you’re right [~jpountz] about the BulkScorer#score. One edge case
though would probably be if a user passes their own BulkScorer, in which case
this approach might not work properly. I guess what we could do is to allow a
user to use a custom BulkScorer, when timeout is enabled, but this might not be
a desirable restriction.
> Should ExitableTermsEnum wrap postings and impacts?
> ---
>
> Key: LUCENE-10544
> URL: https://issues.apache.org/jira/browse/LUCENE-10544
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/index
>Reporter: Greg Miller
>Priority: Major
>
> While looking into options for LUCENE-10151, I noticed that
> {{ExitableDirectoryReader}} doesn't actually do any timeout checking once you
> start iterating postings/impacts. It *does* create a {{ExitableTermsEnum}}
> wrapper when loading a {{{}TermsEnum{}}}, but that wrapper doesn't do
> anything to wrap postings or impacts. So timeouts will be enforced when
> moving to the "next" term, but not when iterating the postings/impacts
> associated with a term.
> I think we ought to wrap the postings/impacts as well with some form of
> timeout checking so timeouts can be enforced on long-running queries. I'm not
> sure why this wasn't done originally (back in 2014), but it was questioned
> back in 2020 on the original Jira SOLR-5986. Does anyone know of a good
> reason why we shouldn't enforce timeouts in this way?
> Related, we may also want to wrap things like {{seekExact}} and {{seekCeil}}
> given that only {{next}} is being wrapped currently.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
