[jira] [Assigned] (LUCENE-8978) "Max Bottom" Based Early Termination For Concurrent Search
[ https://issues.apache.org/jira/browse/LUCENE-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned LUCENE-8978: --- Assignee: Atri Sharma > "Max Bottom" Based Early Termination For Concurrent Search > -- > > Key: LUCENE-8978 > URL: https://issues.apache.org/jira/browse/LUCENE-8978 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 5h 50m > Remaining Estimate: 0h > > When running a search concurrently, collectors which have collected the > number of hits requested locally i.e. their local priority queue is full can > then globally publish their bottom hit's score, and other collectors can then > use that score as the filter. If multiple collectors have full priority > queues, the maximum of all bottom scores will be considered as the global > bottom score. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7798) add equals/hashCode to ToParentBlockJoinSortField
[
https://issues.apache.org/jira/browse/LUCENE-7798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Atri Sharma updated LUCENE-7798:
Status: Resolved (was: Closed)
> add equals/hashCode to ToParentBlockJoinSortField
> -
>
> Key: LUCENE-7798
> URL: https://issues.apache.org/jira/browse/LUCENE-7798
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/join
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: 6.6, 7.0
>
> Attachments: LUCENE-7798.patch
>
>
> Since SOLR-10521 {{ToParentBlockJoinSortField}} is going to be used as query
> result key, therefore it's worth to implemend proper equality methods.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved LUCENE-8213. - Resolution: Fixed > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Time Spent: 9h 40m > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8989) IndexSearcher Should Handle Rejection of Concurrent Task
[ https://issues.apache.org/jira/browse/LUCENE-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940005#comment-16940005 ] Atri Sharma commented on LUCENE-8989: - If the caller has specified a different policy -- we should not be even getting here. IndexSearcher#search API does not advertise what happens if the underlying Executor does not have enough capacity, nor does it consider the size of the executor during allocation of leaves. Both of the above facts lead to an implicit API contract where we are expected to gracefully handle the case of unavailable capacity. In case the user wishes to handle the exception themselves, they are free to override the default methods -- similar to what we advice for other methods in IndexSearcher. > IndexSearcher Should Handle Rejection of Concurrent Task > > > Key: LUCENE-8989 > URL: https://issues.apache.org/jira/browse/LUCENE-8989 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > As discussed in [https://github.com/apache/lucene-solr/pull/815,] > IndexSearcher should handle the case when the executor rejects the execution > of a task (unavailability of threads?). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941772#comment-16941772 ] Atri Sharma commented on LUCENE-8213: - Opened [https://github.com/apache/lucene-solr/pull/912] Curious – looks like the query being cached is insanely heavy – it takes more than 200ms to cache! > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Time Spent: 9h 40m > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942061#comment-16942061 ] Atri Sharma commented on LUCENE-8213: - [~ivera], I am not referring to the LatLonShape failure – I agree with you that the test should be fine even with async caching since it directly checks the result count. However, for testMinSegmentSizePredicate, the failing assertion is on the cache's cache count – which is directly dependent on the race between the call and the completion of the async load (as the count is incremented only when the async load completes). Hence, for testMinSegmentSizePredicate, the correct fix would be to retry the check a few times before marking a test failure. WDYT? > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Time Spent: 9h 40m > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16941913#comment-16941913 ] Atri Sharma commented on LUCENE-8213: - [~ivera] , Regarding the other failure in TestLRUQueryCache, I investigated and that is a definite case of async caching in action – the cache's cache count will not be updated until the async loading completes, before which the test cannot check the value in a deterministic manner. The solution is to retry in case the cache count does not match the target. Making that change lead to a clean beast of TestLRUQueryCache 50 times. I will push the change shortly. > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Time Spent: 9h 40m > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8727) IndexSearcher#search(Query,int) should operate on a shared priority queue when configured with an executor
[ https://issues.apache.org/jira/browse/LUCENE-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942080#comment-16942080 ] Atri Sharma commented on LUCENE-8727: - [~jim.ferenczi] I am not sure if we should close this one – we could still have a shared PQ implementation that would collect to the point hits (with a significantly larger contention). Maybe we should still consider pursuing that as a separate effort (bake my in flight PR that I raised a while back?) > IndexSearcher#search(Query,int) should operate on a shared priority queue > when configured with an executor > -- > > Key: LUCENE-8727 > URL: https://issues.apache.org/jira/browse/LUCENE-8727 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > > If IndexSearcher is configured with an executor, then the top docs for each > slice are computed separately before being merged once the top docs for all > slices are computed. With block-max WAND this is a bit of a waste of > resources: it would be better if an increase of the min competitive score > could help skip non-competitive hits on every slice and not just the current > one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8990) IndexOrDocValuesQuery can take a bad decision for range queries if field has many values per document
[ https://issues.apache.org/jira/browse/LUCENE-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938750#comment-16938750 ] Atri Sharma commented on LUCENE-8990: - +1 I am happy to take a crack at this if you are not planning to do so. > IndexOrDocValuesQuery can take a bad decision for range queries if field has > many values per document > - > > Key: LUCENE-8990 > URL: https://issues.apache.org/jira/browse/LUCENE-8990 > Project: Lucene - Core > Issue Type: Bug >Reporter: Ignacio Vera >Priority: Major > > Heuristics of IndexOrDocValuesQuery are somewhat inconsistent for range > queries . The leadCost that is provided is based on number of documents, > meanwhile the cost() of a range query is based on the number of points that > potentially match the query. > Therefore it might happen that a BKD tree has millions of points but this > points correspond to just a few documents. Therefore we can take the decision > of executing the query using docValues and in fact we are almost scanning all > the points. > Maybe the cost() function for range queries need to take into account the > average number of points per document in the tree and adjust the value > accordingly. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8990) IndexOrDocValuesQuery can take a bad decision for range queries if field has many values per document
[ https://issues.apache.org/jira/browse/LUCENE-8990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16938700#comment-16938700 ] Atri Sharma commented on LUCENE-8990: - +1, I think that is a good heuristic – strangely enough, I was thinking of this limitation for a similar problem. Would it suffice if we just made PointRangeQuery also consider the BKDReader's docCount, in addition to pointCount? e.g. (cost = values.estimatePointCount() / values.estimateDocCount())? > IndexOrDocValuesQuery can take a bad decision for range queries if field has > many values per document > - > > Key: LUCENE-8990 > URL: https://issues.apache.org/jira/browse/LUCENE-8990 > Project: Lucene - Core > Issue Type: Bug >Reporter: Ignacio Vera >Priority: Major > > Heuristics of IndexOrDocValuesQuery are somewhat inconsistent for range > queries . The leadCost that is provided is based on number of documents, > meanwhile the cost() of a range query is based on the number of points that > potentially match the query. > Therefore it might happen that a BKD tree has millions of points but this > points correspond to just a few documents. Therefore we can take the decision > of executing the query using docValues and in fact we are almost scanning all > the points. > Maybe the cost() function for range queries need to take into account the > average number of points per document in the tree and adjust the value > accordingly. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8978) "Max Bottom" Based Early Termination For Concurrent Search
[
https://issues.apache.org/jira/browse/LUCENE-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936554#comment-16936554
]
Atri Sharma commented on LUCENE-8978:
-
|| ||Task ('HighSpanNear', None)||P50 Base 8.692413||P50 Cmp 8.41859||Pct Diff
-3.15013794213||P90 Base 9.420011||P90 Cmp 9.484431||Pct Diff
0.683863320329||P99 Base 48.296534||P99 Cmp 10.859718||Pct Diff
-77.5144982454||P999 Base 48.296534||P999 Cmp 10.859718||Pct Diff
-77.5144982454||P100 Base 212.381771||P100 Cmp 192.26542||Pct Diff
-9.47178795303||
||Task ('BrowseDayOfYearSSDVFacets', None)||P50 Base 25.499444||P50 Cmp
25.523238||Pct Diff 0.0933118384856||P90 Base 41.864402||P90 Cmp 42.868151||Pct
Diff 2.39761934256||P99 Base 46.794425||P99 Cmp 48.499385||Pct Diff
3.64351095243||P999 Base 46.794425||P999 Cmp 48.499385||Pct Diff
3.64351095243||P100 Base 138.543202||P100 Cmp 168.18427||Pct Diff
21.3948195019||
||Task ('HighTermDayOfYearSort', 'DayOfYear')||P50 Base 12.722382||P50 Cmp
14.518649||Pct Diff 14.1189519384||P90 Base 42.669534||P90 Cmp 54.136175||Pct
Diff 26.8731338852||P99 Base 60.871539||P99 Cmp 54.333635||Pct Diff
-10.7404940099||P999 Base 60.871539||P999 Cmp 54.333635||Pct Diff
-10.7404940099||P100 Base 113.663795||P100 Cmp 164.736771||Pct Diff
44.9333721437||
||Task ('HighTerm', None)||P50 Base 2.158854||P50 Cmp 0.82676||Pct Diff
-61.703755789||P90 Base 2.543884||P90 Cmp 1.219797||Pct Diff
-52.0498183093||P99 Base 5.612049||P99 Cmp 2.023208||Pct Diff
-63.9488536183||P999 Base 5.612049||P999 Cmp 2.023208||Pct Diff
-63.9488536183||P100 Base 124.091024||P100 Cmp 142.900739||Pct Diff
15.1579980515||
||Task ('LowTerm', None)||P50 Base 1.05004||P50 Cmp 0.942199||Pct Diff
-10.2701801836||P90 Base 10.419503||P90 Cmp 4.762621||Pct Diff
-54.2912843348||P99 Base 41.745537||P99 Cmp 5.210985||Pct Diff
-87.5172644204||P999 Base 41.745537||P999 Cmp 5.210985||Pct Diff
-87.5172644204||P100 Base 82.929583||P100 Cmp 7.583367||Pct Diff
-90.8556552129||
||Task ('AndHighLow', None)||P50 Base 1.075444||P50 Cmp 1.043672||Pct Diff
-2.95431468305||P90 Base 2.378816||P90 Cmp 2.075887||Pct Diff
-12.733622||P99 Base 3.00708||P99 Cmp 3.089471||Pct Diff
2.73990050148||P999 Base 3.00708||P999 Cmp 3.089471||Pct Diff
2.73990050148||P100 Base 21.82372||P100 Cmp 7.713749||Pct Diff -64.654289003||
||Task ('MedTerm', None)||P50 Base 1.926493||P50 Cmp 1.441123||Pct Diff
-25.1944855237||P90 Base 3.428152||P90 Cmp 6.095949||Pct Diff
77.8202658459||P99 Base 6.520204||P99 Cmp 9.745692||Pct Diff
49.4691270396||P999 Base 6.520204||P999 Cmp 9.745692||Pct Diff
49.4691270396||P100 Base 6.535761||P100 Cmp 24.348327||Pct Diff 272.540045451||
||Task ('AndHighHigh', None)||P50 Base 3.024923||P50 Cmp 2.943217||Pct Diff
-2.7010935485||P90 Base 8.739171||P90 Cmp 6.218642||Pct Diff -28.841740252||P99
Base 26.601864||P99 Cmp 8.478087||Pct Diff -68.1297257967||P999 Base
26.601864||P999 Cmp 8.478087||Pct Diff -68.1297257967||P100 Base
34.451895||P100 Cmp 30.443477||Pct Diff -11.6348258927||
||Task ('LowSloppyPhrase', None)||P50 Base 1.8272||P50 Cmp 1.788978||Pct Diff
-2.09183450088||P90 Base 21.803314||P90 Cmp 2.874724||Pct Diff
-86.8151969925||P99 Base 25.437194||P99 Cmp 8.389269||Pct Diff
-67.019675991||P999 Base 25.437194||P999 Cmp 8.389269||Pct Diff
-67.019675991||P100 Base 36.697506||P100 Cmp 34.158574||Pct Diff
-6.91854100385||
||Task ('Wildcard', None)||P50 Base 1.656227||P50 Cmp 1.548265||Pct Diff
-6.51855089912||P90 Base 14.639417||P90 Cmp 11.039393||Pct Diff
-24.5913071538||P99 Base 78.00446||P99 Cmp 83.830524||Pct Diff
7.46888575346||P999 Base 78.00446||P999 Cmp 83.830524||Pct Diff
7.46888575346||P100 Base 205.441721||P100 Cmp 207.354916||Pct Diff
0.931259235314||
||Task ('HighSloppyPhrase', None)||P50 Base 17.271592||P50 Cmp 15.689429||Pct
Diff -9.1604931381||P90 Base 24.643966||P90 Cmp 22.591891||Pct Diff
-8.32688618382||P99 Base 24.794132||P99 Cmp 26.49345||Pct Diff
6.85371038599||P999 Base 24.794132||P999 Cmp 26.49345||Pct Diff
6.85371038599||P100 Base 140.911694||P100 Cmp 122.87015||Pct Diff
-12.8034398621||
||Task ('HighIntervalsOrdered', None)||P50 Base 7.54805||P50 Cmp 7.610592||Pct
Diff 0.82858486629||P90 Base 9.283201||P90 Cmp 8.87224||Pct Diff
-4.42693204639||P99 Base 11.687616||P99 Cmp 8.97722||Pct Diff
-23.1903238436||P999 Base 11.687616||P999 Cmp 8.97722||Pct Diff
-23.1903238436||P100 Base 65.332843||P100 Cmp 72.605771||Pct Diff
11.1321162007||
||Task ('IntNRQ', None)||P50 Base 4.169943||P50 Cmp 4.055196||Pct Diff
-2.75176423275||P90 Base 81.357868||P90 Cmp 16.793662||Pct Diff
-79.3582816108||P99 Base 93.674773||P99 Cmp 77.915834||Pct Diff
-16.8230340948||P999 Base 93.674773||P999 Cmp 77.915834||Pct Diff
-16.8230340948||P100 Base 107.961319||P100 Cmp 102.590001||Pct Diff
-4.9752245061||
||Task ('BrowseDayOfYearTaxoFacets', None)||P50 Base 0.136203||P50 Cmp
0.122544||Pct Diff -10.0284134711||P90 Base 0.172747||P90 Cmp
[jira] [Commented] (LUCENE-8978) "Max Bottom" Based Early Termination For Concurrent Search
[
https://issues.apache.org/jira/browse/LUCENE-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936673#comment-16936673
]
Atri Sharma commented on LUCENE-8978:
-
Run with latest iteration for wikimedium10m
||Task ('HighSpanNear', None)||P50 Base 26.598803||P50 Cmp 23.506373||Pct Diff
-11.6261998707||P90 Base 31.522045||P90 Cmp 31.316406||Pct Diff
-0.652365669803||P99 Base 33.219318||P99 Cmp 31.318738||Pct Diff
-5.72130951033||P999 Base 33.219318||P999 Cmp 31.318738||Pct Diff
-5.72130951033||P100 Base 495.679543||P100 Cmp 372.126976||Pct Diff
-24.925895923||
||Task ('BrowseDayOfYearSSDVFacets', None)||P50 Base 82.520762||P50 Cmp
81.957502||Pct Diff -0.682567618559||P90 Base 90.89795||P90 Cmp 97.189187||Pct
Diff 6.92120889415||P99 Base 102.318836||P99 Cmp 103.060982||Pct Diff
0.725326859661||P999 Base 102.318836||P999 Cmp 103.060982||Pct Diff
0.725326859661||P100 Base 214.599317||P100 Cmp 214.460731||Pct Diff
-0.0645789566982||
||Task ('HighTermDayOfYearSort', 'DayOfYear')||P50 Base 22.968721||P50 Cmp
22.605351||Pct Diff -1.58202104506||P90 Base 107.394282||P90 Cmp
111.227532||Pct Diff 3.56932410983||P99 Base 116.001988||P99 Cmp
121.010238||Pct Diff 4.31738290554||P999 Base 116.001988||P999 Cmp
121.010238||Pct Diff 4.31738290554||P100 Base 301.747617||P100 Cmp
343.131498||Pct Diff 13.7147333296||
||Task ('OrHighMed', None)||P50 Base 4.434176||P50 Cmp 3.536176||Pct Diff
-20.251789735||P90 Base 7.196299||P90 Cmp 4.998032||Pct Diff
-30.5471882144||P99 Base 12.916992||P99 Cmp 6.256034||Pct Diff
-51.5674082635||P999 Base 12.916992||P999 Cmp 6.256034||Pct Diff
-51.5674082635||P100 Base 209.769114||P100 Cmp 191.717912||Pct Diff
-8.60527160352||
||Task ('LowTerm', None)||P50 Base 1.47407||P50 Cmp 1.238717||Pct Diff
-15.9662024192||P90 Base 2.206357||P90 Cmp 1.89022||Pct Diff
-14.3284608973||P99 Base 2.338337||P99 Cmp 2.593077||Pct Diff
10.8940670228||P999 Base 2.338337||P999 Cmp 2.593077||Pct Diff
10.8940670228||P100 Base 18.287071||P100 Cmp 5.760773||Pct Diff -68.4981099488||
||Task ('OrNotHighHigh', None)||P50 Base 3.064847||P50 Cmp 1.720367||Pct Diff
-43.8677689294||P90 Base 13.529394||P90 Cmp 10.457226||Pct Diff
-22.7073585114||P99 Base 13.855749||P99 Cmp 10.620415||Pct Diff
-23.3501198672||P999 Base 13.855749||P999 Cmp 10.620415||Pct Diff
-23.3501198672||P100 Base 211.859682||P100 Cmp 210.245444||Pct Diff
-0.761937327934||
||Task ('AndHighLow', None)||P50 Base 1.214218||P50 Cmp 1.206011||Pct Diff
-0.675908280062||P90 Base 6.871352||P90 Cmp 2.151958||Pct Diff
-68.6821749199||P99 Base 10.570057||P99 Cmp 7.800031||Pct Diff
-26.2063487453||P999 Base 10.570057||P999 Cmp 7.800031||Pct Diff
-26.2063487453||P100 Base 10.768668||P100 Cmp 13.612023||Pct Diff
26.4039619385||
||Task ('MedTerm', None)||P50 Base 2.174201||P50 Cmp 1.462321||Pct Diff
-32.7421429757||P90 Base 2.855373||P90 Cmp 2.293064||Pct Diff
-19.6930138374||P99 Base 3.017827||P99 Cmp 2.752252||Pct Diff
-8.80020624111||P999 Base 3.017827||P999 Cmp 2.752252||Pct Diff
-8.80020624111||P100 Base 6.437813||P100 Cmp 3.417385||Pct Diff -46.9169887351||
||Task ('AndHighHigh', None)||P50 Base 5.272528||P50 Cmp 5.172284||Pct Diff
-1.90125116453||P90 Base 6.778963||P90 Cmp 8.492454||Pct Diff 25.276594665||P99
Base 23.195234||P99 Cmp 22.413482||Pct Diff -3.37031305655||P999 Base
23.195234||P999 Cmp 22.413482||Pct Diff -3.37031305655||P100 Base
43.983841||P100 Cmp 40.59584||Pct Diff -7.70283113746||
||Task ('OrHighNotMed', None)||P50 Base 3.606809||P50 Cmp 2.200318||Pct Diff
-38.9954389046||P90 Base 5.477783||P90 Cmp 2.811948||Pct Diff
-48.6663126305||P99 Base 12.656732||P99 Cmp 2.848727||Pct Diff
-77.4923969315||P999 Base 12.656732||P999 Cmp 2.848727||Pct Diff
-77.4923969315||P100 Base 21.971503||P100 Cmp 3.707231||Pct Diff
-83.1270942184||
||Task ('LowSloppyPhrase', None)||P50 Base 32.716729||P50 Cmp 33.41864||Pct
Diff 2.14541924408||P90 Base 46.672458||P90 Cmp 46.895997||Pct Diff
0.478952704826||P99 Base 54.76534||P99 Cmp 47.03021||Pct Diff
-14.1241339869||P999 Base 54.76534||P999 Cmp 47.03021||Pct Diff
-14.1241339869||P100 Base 94.839019||P100 Cmp 85.795202||Pct Diff
-9.53596641484||
||Task ('Wildcard', None)||P50 Base 4.968175||P50 Cmp 4.007837||Pct Diff
-19.3297941397||P90 Base 15.318453||P90 Cmp 18.425081||Pct Diff
20.2802985393||P99 Base 343.578467||P99 Cmp 474.913619||Pct Diff
38.2256644739||P999 Base 343.578467||P999 Cmp 474.913619||Pct Diff
38.2256644739||P100 Base 489.133244||P100 Cmp 488.166182||Pct Diff
-0.197709317832||
||Task ('HighSloppyPhrase', None)||P50 Base 174.74911||P50 Cmp 169.334563||Pct
Diff -3.09846899936||P90 Base 213.368561||P90 Cmp 212.781576||Pct Diff
-0.275103790947||P99 Base 214.896944||P99 Cmp 215.594741||Pct Diff
0.324712388651||P999 Base 214.896944||P999 Cmp 215.594741||Pct Diff
0.324712388651||P100 Base 304.019537||P100 Cmp 339.145455||Pct Diff
11.5538357655||
||Task ('HighIntervalsOrdered',
[jira] [Commented] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942788#comment-16942788 ] Atri Sharma commented on LUCENE-8213: - [~jpountz] Just noticed your latest comment – I raised a PR ([https://github.com/apache/lucene-solr/pull/916)] for this change – it includes the logic to use CacheKey + Query as the map's key as well (sorry about that). Please let me know if this logic looks fine – I will raise another PR to use that combination as the key for the serial case. > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Attachments: 0001-Reproduce-across-segment-caching-of-same-query.patch > > Time Spent: 10h 20m > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942629#comment-16942629 ] Atri Sharma commented on LUCENE-8213: - [~jpountz] I am not sure I follow – inFlightAsyncQueries map will only be involved in the concurrent load case? I am not sure I get your thoughts behind fixing the serial case. > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Attachments: 0001-Reproduce-across-segment-caching-of-same-query.patch > > Time Spent: 10h > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942551#comment-16942551 ] Atri Sharma commented on LUCENE-8213: - Hi [~jpountz] bq. It would probably make things easier indeed, this way we can take the time to think about how to address the various issues. Sure, happy to revert. I managed to create a standalone unit test which reproduces this case, though, and posted a patch last night. I am in process of raising a PR with the fix and the test -- would you want to take a look at that, or should I revert right away? bq. I don't think the ord is enough as the query cache can be shared across multiple readers One way I hacked around this was to use the LeafReaderContext's ID field + the query as the unique identifier for the cache. This was inspired by the way CompositeReaderContext uses its ID field. Does this sounds a viable approach? > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Attachments: 0001-Reproduce-across-segment-caching-of-same-query.patch > > Time Spent: 9h 40m > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8978) "Max Bottom" Based Early Termination For Concurrent Search
[
https://issues.apache.org/jira/browse/LUCENE-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934206#comment-16934206
]
Atri Sharma commented on LUCENE-8978:
-
I had some issues with my earlier test runs. Fixed them now and here are the
updated results:
Wikimedium10m with concurrent search:
||Task ('HighSpanNear', None)||P50 Base 18.971856||P50 Cmp 19.115405||Pct Diff
0.75664183831||P90 Base 25.394956||P90 Cmp 22.396552||Pct Diff
-11.80708484||P99 Base 25.585029||P99 Cmp 22.825023||Pct Diff
-10.7875820661||P999 Base 25.585029||P999 Cmp 22.825023||Pct Diff
-10.7875820661||P100 Base 29.948754||P100 Cmp 22.990658||Pct Diff
-23.2333405256||
||Task ('BrowseDayOfYearSSDVFacets', None)||P50 Base 100.813812||P50 Cmp
101.237299||Pct Diff 0.420068432687||P90 Base 112.729388||P90 Cmp
120.730533||Pct Diff 7.09765673526||P99 Base 122.198435||P99 Cmp
132.415702||Pct Diff 8.36120937228||P999 Base 122.198435||P999 Cmp
132.415702||Pct Diff 8.36120937228||P100 Base 294.000735||P100 Cmp
276.961033||Pct Diff -5.7958025173||
||Task ('HighTermDayOfYearSort', 'DayOfYear')||P50 Base 29.38945||P50 Cmp
30.619796||Pct Diff 4.18635258571||P90 Base 71.768952||P90 Cmp 80.413425||Pct
Diff 12.0448644701||P99 Base 85.238118||P99 Cmp 98.349615||Pct Diff
15.3821990767||P999 Base 85.238118||P999 Cmp 98.349615||Pct Diff
15.3821990767||P100 Base 100.262364||P100 Cmp 142.551521||Pct Diff
42.1784958112||
||Task ('HighTerm', None)||P50 Base 2.238097||P50 Cmp 1.268568||Pct Diff
-43.3193467486||P90 Base 3.279594||P90 Cmp 2.217695||Pct Diff
-32.3789773978||P99 Base 3.833247||P99 Cmp 7.549833||Pct Diff
96.9566010226||P999 Base 3.833247||P999 Cmp 7.549833||Pct Diff
96.9566010226||P100 Base 8.530505||P100 Cmp 37.05043||Pct Diff 334.328682768||
||Task ('LowTerm', None)||P50 Base 1.543205||P50 Cmp 1.231248||Pct Diff
-20.214877479||P90 Base 7.173272||P90 Cmp 2.116812||Pct Diff
-70.4902867199||P99 Base 11.802009||P99 Cmp 6.056692||Pct Diff
-48.6808389995||P999 Base 11.802009||P999 Cmp 6.056692||Pct Diff
-48.6808389995||P100 Base 21.263545||P100 Cmp 29.650059||Pct Diff
39.4408082001||
||Task ('OrNotHighHigh', None)||P50 Base 2.995964||P50 Cmp 1.803324||Pct Diff
-39.8082219947||P90 Base 3.829869||P90 Cmp 2.761644||Pct Diff
-27.8919461736||P99 Base 3.835793||P99 Cmp 2.90684||Pct Diff
-24.2180169785||P999 Base 3.835793||P999 Cmp 2.90684||Pct Diff
-24.2180169785||P100 Base 41.140203||P100 Cmp 38.965549||Pct Diff
-5.28595836049||
||Task ('MedPhrase', None)||P50 Base 2.679309||P50 Cmp 2.680824||Pct Diff
0.0565444299258||P90 Base 3.448301||P90 Cmp 4.514041||Pct Diff
30.9062346935||P99 Base 95.803532||P99 Cmp 38.188159||Pct Diff
-60.1390906966||P999 Base 95.803532||P999 Cmp 38.188159||Pct Diff
-60.1390906966||P100 Base 101.641901||P100 Cmp 73.856804||Pct Diff
-27.3362626305||
||Task ('AndHighLow', None)||P50 Base 1.409244||P50 Cmp 1.347221||Pct Diff
-4.40115409397||P90 Base 4.495005||P90 Cmp 3.099544||Pct Diff
-31.044704066||P99 Base 6.043321||P99 Cmp 5.084951||Pct Diff
-15.8583335222||P999 Base 6.043321||P999 Cmp 5.084951||Pct Diff
-15.8583335222||P100 Base 8.454945||P100 Cmp 7.701492||Pct Diff -8.91138854244||
||Task ('MedTerm', None)||P50 Base 1.682272||P50 Cmp 1.147723||Pct Diff
-31.7754203839||P90 Base 15.24221||P90 Cmp 4.303477||Pct Diff
-71.7660562346||P99 Base 24.52032||P99 Cmp 5.817554||Pct Diff
-76.2745592227||P999 Base 24.52032||P999 Cmp 5.817554||Pct Diff
-76.2745592227||P100 Base 45.496262||P100 Cmp 20.414843||Pct Diff
-55.1285268227||
||Task ('Prefix3', None)||P50 Base 1.904465||P50 Cmp 1.8418||Pct Diff
-3.29042539506||P90 Base 6.631238||P90 Cmp 21.78927||Pct Diff
228.585250597||P99 Base 20.862886||P99 Cmp 164.172423||Pct Diff
686.911374582||P999 Base 20.862886||P999 Cmp 164.172423||Pct Diff
686.911374582||P100 Base 235.494494||P100 Cmp 311.228238||Pct Diff
32.1594542249||
||Task ('OrHighNotMed', None)||P50 Base 2.592121||P50 Cmp 1.741687||Pct Diff
-32.8084221377||P90 Base 3.21777||P90 Cmp 2.281574||Pct Diff
-29.0945592755||P99 Base 4.108258||P99 Cmp 2.356369||Pct Diff
-42.6431105349||P999 Base 4.108258||P999 Cmp 2.356369||Pct Diff
-42.6431105349||P100 Base 36.452029||P100 Cmp 24.359319||Pct Diff
-33.1743124642||
||Task ('LowSloppyPhrase', None)||P50 Base 7.17009||P50 Cmp 6.775814||Pct Diff
-5.49889889806||P90 Base 10.237824||P90 Cmp 11.639566||Pct Diff
13.6917962254||P99 Base 15.983773||P99 Cmp 11.864554||Pct Diff
-25.7712556353||P999 Base 15.983773||P999 Cmp 11.864554||Pct Diff
-25.7712556353||P100 Base 27.466558||P100 Cmp 15.832752||Pct Diff
-42.3562573803||
||Task ('Wildcard', None)||P50 Base 3.182749||P50 Cmp 2.826948||Pct Diff
-11.1790467926||P90 Base 10.75107||P90 Cmp 9.240186||Pct Diff
-14.0533360865||P99 Base 143.622905||P99 Cmp 9.604664||Pct Diff
-93.3125820008||P999 Base 143.622905||P999 Cmp 9.604664||Pct Diff
-93.3125820008||P100 Base 356.310669||P100 Cmp 165.943325||Pct Diff
-53.4273488173||
||Task
[jira] [Commented] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940673#comment-16940673 ] Atri Sharma commented on LUCENE-8213: - Taking a look – although I am unable to reproduce with a simple ant test. Can you point me to the CI link so that I can dive deeper into the error output? > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Time Spent: 9h 40m > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16940798#comment-16940798 ] Atri Sharma commented on LUCENE-8213: - Interesting – I did not realise that testLRUEviction could also cause LRUQueryCache to cache asynchronously, hence did not update it to handle the same (in the manner testLRUConcurrentLoadAndEviction does). I have pushed a test fix now – beasted the test 50 times with the seed you provided, and also beasted the entire TestLRUQueryCache suite 20 times with the seed. Ran the entire Lucene test suite – came in clean. It is curious to note that I could not reproduce the test failure without the seed even after running multiple times – kudos to the CI! > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Time Spent: 9h 40m > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16942285#comment-16942285 ] Atri Sharma commented on LUCENE-8213: - Attached is a patch which reproduces the issue [~jpountz] mentioned in a unit test. Using a Pair of Query + Segment's ID object as the Set (inFlightAsyncLoadQueries)'s key makes the test pass. [^0001-Reproduce-across-segment-caching-of-same-query.patch] > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Attachments: 0001-Reproduce-across-segment-caching-of-same-query.patch > > Time Spent: 9h 40m > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9066) Modal Strategy In Concurrent Query Execution
Atri Sharma created LUCENE-9066: --- Summary: Modal Strategy In Concurrent Query Execution Key: LUCENE-9066 URL: https://issues.apache.org/jira/browse/LUCENE-9066 Project: Lucene - Core Issue Type: Improvement Reporter: Atri Sharma When executing a query concurrently today, we do not take any sort of system statistics into account. For e.g. if the node is under high pressure, it is not advisable to spawn a large number of threads for a query since they are most likely to be blocked waiting for CPU to be available. However, the converse is that for a lightly loaded cluster, the query can consume as many threads as required. This Jira tracks high level efforts in this direction. The first idea is to account the Executor's wait queue's size as a factor when allocating slices to a query's segments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9074) Account for Executor's Queue Length When Planning Slices in IndexSearhcer
Atri Sharma created LUCENE-9074: --- Summary: Account for Executor's Queue Length When Planning Slices in IndexSearhcer Key: LUCENE-9074 URL: https://issues.apache.org/jira/browse/LUCENE-9074 Project: Lucene - Core Issue Type: Improvement Reporter: Atri Sharma -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9066) Modal Strategy In Concurrent Query Execution
[ https://issues.apache.org/jira/browse/LUCENE-9066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16985954#comment-16985954 ] Atri Sharma commented on LUCENE-9066: - [~jpountz] Agreed that we might not need the actual system statistics – queue length should be fine. I opened LUCENE-9074 for adding queue length as a factor when allocating slices. I will open another Jira to check for the current queue length before creating a new thread for a given slice. This will ensure more dynamic behaviour for a system that is intermittently in and out of system stress. > Modal Strategy In Concurrent Query Execution > > > Key: LUCENE-9066 > URL: https://issues.apache.org/jira/browse/LUCENE-9066 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > When executing a query concurrently today, we do not take any sort of system > statistics into account. For e.g. if the node is under high pressure, it is > not advisable to spawn a large number of threads for a query since they are > most likely to be blocked waiting for CPU to be available. However, the > converse is that for a lightly loaded cluster, the query can consume as many > threads as required. > > This Jira tracks high level efforts in this direction. The first idea is to > account the Executor's wait queue's size as a factor when allocating slices > to a query's segments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9074) Account for Executor's Queue Length When Planning Slices in IndexSearcher
[ https://issues.apache.org/jira/browse/LUCENE-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned LUCENE-9074: --- Assignee: Atri Sharma > Account for Executor's Queue Length When Planning Slices in IndexSearcher > - > > Key: LUCENE-9074 > URL: https://issues.apache.org/jira/browse/LUCENE-9074 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9074) Account for Executor's Queue Length When Planning Slices in IndexSearcher
[ https://issues.apache.org/jira/browse/LUCENE-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma updated LUCENE-9074: Summary: Account for Executor's Queue Length When Planning Slices in IndexSearcher (was: Account for Executor's Queue Length When Planning Slices in IndexSearhcer) > Account for Executor's Queue Length When Planning Slices in IndexSearcher > - > > Key: LUCENE-9074 > URL: https://issues.apache.org/jira/browse/LUCENE-9074 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9075) Check Current System Pressure Before Allocating a New Thread In Concurrent Search
Atri Sharma created LUCENE-9075: --- Summary: Check Current System Pressure Before Allocating a New Thread In Concurrent Search Key: LUCENE-9075 URL: https://issues.apache.org/jira/browse/LUCENE-9075 Project: Lucene - Core Issue Type: Improvement Reporter: Atri Sharma When executing slices for a concurrent query, we should check the current status of the system before allocating since that can contribute to the pressure on the system and further degrade the situation -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-8749) Proposal: Pluggable Interface for Slice Allocation Algorithm
[ https://issues.apache.org/jira/browse/LUCENE-8749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved LUCENE-8749. - Resolution: Won't Fix Not needed post the smarter slice allocation algorithm > Proposal: Pluggable Interface for Slice Allocation Algorithm > > > Key: LUCENE-8749 > URL: https://issues.apache.org/jira/browse/LUCENE-8749 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Reporter: Atri Sharma >Priority: Major > > The slice allocation method allocates one thread per segment today. If a user > wishes to use a different slice allocation algorithm, there is no way except > to make a change in IndexSearcher. This Jira proposes an interface to > decouple the slice allocation mechanism from IndexSearcher and allow plugging > in the method from an external factory (like Collectors). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9070) TestLRUCache failures
[
https://issues.apache.org/jira/browse/LUCENE-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984136#comment-16984136
]
Atri Sharma commented on LUCENE-9070:
-
[~jpountz] Investigated. I could not reproduce the first one and the second one
was an over zealous assert in the test which was not actually related to the
test itself (checking if number of segments generated in current random run is
more than 1). That should not be an assert since it is not a deterministic
scenario and also does not affect the functionality that the test is checking.
I pushed a fix to remove the same and beasted the test 20 times with the seed
that triggered the failure – no run failed
> TestLRUCache failures
> -
>
> Key: LUCENE-9070
> URL: https://issues.apache.org/jira/browse/LUCENE-9070
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Priority: Major
>
> {noformat}
> 02:11:11[junit4] Suite: org.apache.lucene.search.TestLRUQueryCache
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedAgreesWithRamUsageTester
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedConstantEntryOverhead
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testDetectMutatedQueries
> 02:11:11[junit4]> Assumption #1: LUCENE-7604: For some unknown reason
> the non-constant BadQuery#hashCode() does not trigger
> ConcurrentModificationException on Java 9 b150
> 02:11:11[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLRUQueryCache -Dtests.method=testLRUConcurrentLoadsOfSameQuery
> -Dtests.seed=205F9EFCF29BA4D6 -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=so-ET -Dtests.timezone=Europe/Ljubljana -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> 02:11:11[junit4] ERROR 0.02s J3 |
> TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery <<<
> 02:11:11[junit4]> Throwable #1: java.lang.RuntimeException:
> MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1}
> 02:11:11[junit4]> at
> __randomizedtesting.SeedInfo.seed([205F9EFCF29BA4D6:5E7DB1C91CF54294]:0)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:807)
> 02:11:11[junit4]> at
> org.apache.lucene.search.TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery(TestLRUQueryCache.java:465)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 02:11:11[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 02:11:11[junit4]> at
> java.base/java.lang.Thread.run(Thread.java:834)
> 02:11:11[junit4]> Caused by: java.lang.RuntimeException: unclosed
> IndexInput: _0.cfs
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:725)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:768)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:79)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:101)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentReader.(SegmentReader.java:84)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219)
> 02:11:11[junit4]> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
> 02:11:11[junit4]> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:442)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:376)
> 02:11:11[junit4]> at
>
[jira] [Commented] (LUCENE-9070) TestLRUCache failures
[
https://issues.apache.org/jira/browse/LUCENE-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984120#comment-16984120
]
Atri Sharma commented on LUCENE-9070:
-
Looking
> TestLRUCache failures
> -
>
> Key: LUCENE-9070
> URL: https://issues.apache.org/jira/browse/LUCENE-9070
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Priority: Major
>
> {noformat}
> 02:11:11[junit4] Suite: org.apache.lucene.search.TestLRUQueryCache
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedAgreesWithRamUsageTester
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedConstantEntryOverhead
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testDetectMutatedQueries
> 02:11:11[junit4]> Assumption #1: LUCENE-7604: For some unknown reason
> the non-constant BadQuery#hashCode() does not trigger
> ConcurrentModificationException on Java 9 b150
> 02:11:11[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLRUQueryCache -Dtests.method=testLRUConcurrentLoadsOfSameQuery
> -Dtests.seed=205F9EFCF29BA4D6 -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=so-ET -Dtests.timezone=Europe/Ljubljana -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> 02:11:11[junit4] ERROR 0.02s J3 |
> TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery <<<
> 02:11:11[junit4]> Throwable #1: java.lang.RuntimeException:
> MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1}
> 02:11:11[junit4]> at
> __randomizedtesting.SeedInfo.seed([205F9EFCF29BA4D6:5E7DB1C91CF54294]:0)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:807)
> 02:11:11[junit4]> at
> org.apache.lucene.search.TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery(TestLRUQueryCache.java:465)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 02:11:11[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 02:11:11[junit4]> at
> java.base/java.lang.Thread.run(Thread.java:834)
> 02:11:11[junit4]> Caused by: java.lang.RuntimeException: unclosed
> IndexInput: _0.cfs
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:725)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:768)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:79)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:101)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentReader.(SegmentReader.java:84)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219)
> 02:11:11[junit4]> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
> 02:11:11[junit4]> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:442)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:376)
> 02:11:11[junit4]> at
> org.apache.lucene.search.TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery(TestLRUQueryCache.java:418)
> 02:11:11[junit4]> ... 35 more
> 02:11:11[junit4] 2> NOTE: leaving temporary files on disk at:
> /var/lib/jenkins/workspace/apache+lucene-solr+master/lucene/build/core/test/J3/temp/lucene.search.TestLRUQueryCache_205F9EFCF29BA4D6-001
> 02:11:11[junit4] 2> Nov 27, 2019 12:10:48 AM
> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
> 02:11:11[junit4] 2>
[jira] [Commented] (LUCENE-9066) Modal Strategy In Concurrent Query Execution
[ https://issues.apache.org/jira/browse/LUCENE-9066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984827#comment-16984827 ] Atri Sharma commented on LUCENE-9066: - Here is the high level plan of attack that I am considering: 1) Introduce a heuristic in IndexSearcher#slices which accounts for the current work queue length of the passed in Executor. A simple idea could be to only create a new slice iff all of current conditions are met AND the executor's work queue length is less than the number of cores. 2) Introduce an interface which allows giving real time statistics about the current state of the environment which can be used by IndexSearcher to decide whether to execute a slice on a new thread or to execute it on a caller thread. Potentially, the interface can be as simple as the same heuristic as above where IndexSearcher executes a slice on a new thread only when the queue of the executor is smaller than number of cores * 1.5 (similar to what Elasticsearch does, AFAIK). The interface can be used to provide a custom implementation to IndexSearcher thus paving the path for external modules to provide statistics into the system and allowing Lucene to be a friendly neighbor in multi tenant systems. Thoughts? CC [~jpountz] > Modal Strategy In Concurrent Query Execution > > > Key: LUCENE-9066 > URL: https://issues.apache.org/jira/browse/LUCENE-9066 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > When executing a query concurrently today, we do not take any sort of system > statistics into account. For e.g. if the node is under high pressure, it is > not advisable to spawn a large number of threads for a query since they are > most likely to be blocked waiting for CPU to be available. However, the > converse is that for a lightly loaded cluster, the query can consume as many > threads as required. > > This Jira tracks high level efforts in this direction. The first idea is to > account the Executor's wait queue's size as a factor when allocating slices > to a query's segments. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9070) TestLRUCache failures
[
https://issues.apache.org/jira/browse/LUCENE-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984433#comment-16984433
]
Atri Sharma commented on LUCENE-9070:
-
[~jpountz] I investigated the two that you pointed a few hours back -- could
not reproduce them
Looking at the latest one now
> TestLRUCache failures
> -
>
> Key: LUCENE-9070
> URL: https://issues.apache.org/jira/browse/LUCENE-9070
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Priority: Major
>
> {noformat}
> 02:11:11[junit4] Suite: org.apache.lucene.search.TestLRUQueryCache
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedAgreesWithRamUsageTester
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedConstantEntryOverhead
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testDetectMutatedQueries
> 02:11:11[junit4]> Assumption #1: LUCENE-7604: For some unknown reason
> the non-constant BadQuery#hashCode() does not trigger
> ConcurrentModificationException on Java 9 b150
> 02:11:11[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLRUQueryCache -Dtests.method=testLRUConcurrentLoadsOfSameQuery
> -Dtests.seed=205F9EFCF29BA4D6 -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=so-ET -Dtests.timezone=Europe/Ljubljana -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> 02:11:11[junit4] ERROR 0.02s J3 |
> TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery <<<
> 02:11:11[junit4]> Throwable #1: java.lang.RuntimeException:
> MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1}
> 02:11:11[junit4]> at
> __randomizedtesting.SeedInfo.seed([205F9EFCF29BA4D6:5E7DB1C91CF54294]:0)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:807)
> 02:11:11[junit4]> at
> org.apache.lucene.search.TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery(TestLRUQueryCache.java:465)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 02:11:11[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 02:11:11[junit4]> at
> java.base/java.lang.Thread.run(Thread.java:834)
> 02:11:11[junit4]> Caused by: java.lang.RuntimeException: unclosed
> IndexInput: _0.cfs
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:725)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:768)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:79)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:101)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentReader.(SegmentReader.java:84)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219)
> 02:11:11[junit4]> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
> 02:11:11[junit4]> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:442)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:376)
> 02:11:11[junit4]> at
> org.apache.lucene.search.TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery(TestLRUQueryCache.java:418)
> 02:11:11[junit4]> ... 35 more
> 02:11:11[junit4] 2> NOTE: leaving temporary files on disk at:
> /var/lib/jenkins/workspace/apache+lucene-solr+master/lucene/build/core/test/J3/temp/lucene.search.TestLRUQueryCache_205F9EFCF29BA4D6-001
> 02:11:11[junit4] 2> Nov
[jira] [Commented] (LUCENE-9070) TestLRUCache failures
[
https://issues.apache.org/jira/browse/LUCENE-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984462#comment-16984462
]
Atri Sharma commented on LUCENE-9070:
-
I am +1 for reverting, though trying to identify if it is more of a test issue
rather than a functionality issue. I beasted the test suite using the seeds
that were mentioned in the stack trace of the failing tests but to no avail.
The last failure seems more of an edge case of how a List is printed in Java (I
believe there is no guarantee to the order in which elements would be printed
when printing a list, and the test seems to be taking a dependency on that).
> TestLRUCache failures
> -
>
> Key: LUCENE-9070
> URL: https://issues.apache.org/jira/browse/LUCENE-9070
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Priority: Major
>
> {noformat}
> 02:11:11[junit4] Suite: org.apache.lucene.search.TestLRUQueryCache
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedAgreesWithRamUsageTester
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedConstantEntryOverhead
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testDetectMutatedQueries
> 02:11:11[junit4]> Assumption #1: LUCENE-7604: For some unknown reason
> the non-constant BadQuery#hashCode() does not trigger
> ConcurrentModificationException on Java 9 b150
> 02:11:11[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLRUQueryCache -Dtests.method=testLRUConcurrentLoadsOfSameQuery
> -Dtests.seed=205F9EFCF29BA4D6 -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=so-ET -Dtests.timezone=Europe/Ljubljana -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> 02:11:11[junit4] ERROR 0.02s J3 |
> TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery <<<
> 02:11:11[junit4]> Throwable #1: java.lang.RuntimeException:
> MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1}
> 02:11:11[junit4]> at
> __randomizedtesting.SeedInfo.seed([205F9EFCF29BA4D6:5E7DB1C91CF54294]:0)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:807)
> 02:11:11[junit4]> at
> org.apache.lucene.search.TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery(TestLRUQueryCache.java:465)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 02:11:11[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 02:11:11[junit4]> at
> java.base/java.lang.Thread.run(Thread.java:834)
> 02:11:11[junit4]> Caused by: java.lang.RuntimeException: unclosed
> IndexInput: _0.cfs
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:725)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:768)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:79)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:101)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentReader.(SegmentReader.java:84)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219)
> 02:11:11[junit4]> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
> 02:11:11[junit4]> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:442)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:376)
> 02:11:11[junit4]> at
>
[jira] [Commented] (LUCENE-9070) TestLRUCache failures
[
https://issues.apache.org/jira/browse/LUCENE-9070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984474#comment-16984474
]
Atri Sharma commented on LUCENE-9070:
-
[~jpountz] Done
> TestLRUCache failures
> -
>
> Key: LUCENE-9070
> URL: https://issues.apache.org/jira/browse/LUCENE-9070
> Project: Lucene - Core
> Issue Type: Bug
>Reporter: Adrien Grand
>Priority: Major
>
> {noformat}
> 02:11:11[junit4] Suite: org.apache.lucene.search.TestLRUQueryCache
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedAgreesWithRamUsageTester
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testRamBytesUsedConstantEntryOverhead
> 02:11:11[junit4]> Assumption #1: LUCENE-7595: RamUsageTester does not
> work exact in Java 9 (estimations for maps and lists)
> 02:11:11[junit4] IGNOR/A 0.00s J3 |
> TestLRUQueryCache.testDetectMutatedQueries
> 02:11:11[junit4]> Assumption #1: LUCENE-7604: For some unknown reason
> the non-constant BadQuery#hashCode() does not trigger
> ConcurrentModificationException on Java 9 b150
> 02:11:11[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TestLRUQueryCache -Dtests.method=testLRUConcurrentLoadsOfSameQuery
> -Dtests.seed=205F9EFCF29BA4D6 -Dtests.slow=true -Dtests.badapples=true
> -Dtests.locale=so-ET -Dtests.timezone=Europe/Ljubljana -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
> 02:11:11[junit4] ERROR 0.02s J3 |
> TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery <<<
> 02:11:11[junit4]> Throwable #1: java.lang.RuntimeException:
> MockDirectoryWrapper: cannot close: there are still 1 open files: {_0.cfs=1}
> 02:11:11[junit4]> at
> __randomizedtesting.SeedInfo.seed([205F9EFCF29BA4D6:5E7DB1C91CF54294]:0)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.close(MockDirectoryWrapper.java:807)
> 02:11:11[junit4]> at
> org.apache.lucene.search.TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery(TestLRUQueryCache.java:465)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 02:11:11[junit4]> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 02:11:11[junit4]> at
> java.base/java.lang.reflect.Method.invoke(Method.java:566)
> 02:11:11[junit4]> at
> java.base/java.lang.Thread.run(Thread.java:834)
> 02:11:11[junit4]> Caused by: java.lang.RuntimeException: unclosed
> IndexInput: _0.cfs
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.addFileHandle(MockDirectoryWrapper.java:725)
> 02:11:11[junit4]> at
> org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:768)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.(Lucene50CompoundReader.java:79)
> 02:11:11[junit4]> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:101)
> 02:11:11[junit4]> at
> org.apache.lucene.index.SegmentReader.(SegmentReader.java:84)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177)
> 02:11:11[junit4]> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219)
> 02:11:11[junit4]> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109)
> 02:11:11[junit4]> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:442)
> 02:11:11[junit4]> at
> org.apache.lucene.index.RandomIndexWriter.getReader(RandomIndexWriter.java:376)
> 02:11:11[junit4]> at
> org.apache.lucene.search.TestLRUQueryCache.testLRUConcurrentLoadsOfSameQuery(TestLRUQueryCache.java:418)
> 02:11:11[junit4]> ... 35 more
> 02:11:11[junit4] 2> NOTE: leaving temporary files on disk at:
> /var/lib/jenkins/workspace/apache+lucene-solr+master/lucene/build/core/test/J3/temp/lucene.search.TestLRUQueryCache_205F9EFCF29BA4D6-001
> 02:11:11[junit4] 2> Nov 27, 2019 12:10:48 AM
> com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
> 02:11:11
[jira] [Resolved] (LUCENE-8978) "Max Bottom" Based Early Termination For Concurrent Search
[ https://issues.apache.org/jira/browse/LUCENE-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved LUCENE-8978. - Resolution: Fixed > "Max Bottom" Based Early Termination For Concurrent Search > -- > > Key: LUCENE-8978 > URL: https://issues.apache.org/jira/browse/LUCENE-8978 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 8.5h > Remaining Estimate: 0h > > When running a search concurrently, collectors which have collected the > number of hits requested locally i.e. their local priority queue is full can > then globally publish their bottom hit's score, and other collectors can then > use that score as the filter. If multiple collectors have full priority > queues, the maximum of all bottom scores will be considered as the global > bottom score. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-8988) Maximal -- Minimum Based Early Termination For TopFieldCollector
[ https://issues.apache.org/jira/browse/LUCENE-8988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned LUCENE-8988: --- Assignee: Atri Sharma > Maximal -- Minimum Based Early Termination For TopFieldCollector > > > Key: LUCENE-8988 > URL: https://issues.apache.org/jira/browse/LUCENE-8988 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > > Use LUCENE-8978 to implement the same logic for TopFieldCollector -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-8988) Maximal -- Minimum Based Early Termination For TopFieldCollector
Atri Sharma created LUCENE-8988: --- Summary: Maximal -- Minimum Based Early Termination For TopFieldCollector Key: LUCENE-8988 URL: https://issues.apache.org/jira/browse/LUCENE-8988 Project: Lucene - Core Issue Type: Improvement Reporter: Atri Sharma Use LUCENE-8978 to implement the same logic for TopFieldCollector -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-8989) IndexSearcher Should Handle Rejection of Concurrent Task
Atri Sharma created LUCENE-8989: --- Summary: IndexSearcher Should Handle Rejection of Concurrent Task Key: LUCENE-8989 URL: https://issues.apache.org/jira/browse/LUCENE-8989 Project: Lucene - Core Issue Type: Improvement Reporter: Atri Sharma As discussed in [https://github.com/apache/lucene-solr/pull/815,] IndexSearcher should handle the case when the executor rejects the execution of a task (unavailability of threads?). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8989) IndexSearcher Should Handle Rejection of Concurrent Task
[ https://issues.apache.org/jira/browse/LUCENE-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937679#comment-16937679 ] Atri Sharma commented on LUCENE-8989: - I think the safest bet here is to run the rejected task on the caller thread. Any objections/thoughts? > IndexSearcher Should Handle Rejection of Concurrent Task > > > Key: LUCENE-8989 > URL: https://issues.apache.org/jira/browse/LUCENE-8989 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > As discussed in [https://github.com/apache/lucene-solr/pull/815,] > IndexSearcher should handle the case when the executor rejects the execution > of a task (unavailability of threads?). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8027) Do not cache clauses if they might make the query more than X times slower
[ https://issues.apache.org/jira/browse/LUCENE-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16947060#comment-16947060 ] Atri Sharma commented on LUCENE-8027: - I think this is a great idea -- should really help with costly queries especially when one clause is dominating the others in terms of cost. In terms of how this will work with LUCENE-8213, I believe we should relax the cost parameters a bit when asynchronous caching is enabled i.e. we should not cache a query with async caching enabled when the cost is 15x higher, and not cache when cost difference > 10x for sync queries. > Do not cache clauses if they might make the query more than X times slower > -- > > Key: LUCENE-8027 > URL: https://issues.apache.org/jira/browse/LUCENE-8027 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Attachments: LUCENE-8027.patch > > > Query caching can have a negative impact on tail latencies as the clause that > is cached needs to be entirely consumed. Maybe we could leverage the fact > that we can know the lead cost from any scorer now (LUCENE-7897) in order to > implement heuristics that would look like "do not cache clause X if its cost > is 10x greater than the cost of the entire query". This would make sure that > a fast query can not become absurdly slow just because it had to cache a > costly filter. The filter will need to wait for a more costly query to be > cached, or might never be cached at all. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-8213) Cache costly subqueries asynchronously
[ https://issues.apache.org/jira/browse/LUCENE-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943317#comment-16943317 ] Atri Sharma edited comment on LUCENE-8213 at 10/3/19 4:18 AM: -- Thanks [~hossman] Interestingly, all but one test failures are coming from LatLon queries -- is there anything special about them? was (Author: atris): Thanks [~hossman] Interestingly, all but one test failures are coming from LatLong queries -- is there anything special about them? > Cache costly subqueries asynchronously > -- > > Key: LUCENE-8213 > URL: https://issues.apache.org/jira/browse/LUCENE-8213 > Project: Lucene - Core > Issue Type: Improvement > Components: core/query/scoring >Affects Versions: 7.2.1 >Reporter: Amir Hadadi >Priority: Minor > Labels: performance > Attachments: > 0001-Reproduce-across-segment-caching-of-same-query.patch, > thetaphi_Lucene-Solr-master-Linux_24839.log.txt > > Time Spent: 13h > Remaining Estimate: 0h > > IndexOrDocValuesQuery allows to combine costly range queries with a selective > lead iterator in an optimized way. However, the range query at some point > gets cached by a querying thread in LRUQueryCache, which negates the > optimization of IndexOrDocValuesQuery for that specific query. > It would be nice to see an asynchronous caching implementation in such cases, > so that queries involving IndexOrDocValuesQuery would have consistent > performance characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8989) IndexSearcher Should Handle Rejection of Concurrent Task
[ https://issues.apache.org/jira/browse/LUCENE-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943296#comment-16943296 ] Atri Sharma commented on LUCENE-8989: - bq. Hmm, that is a good point – maybe we should not handle this exception and let it throw? I would still argue that the default case should be to handle the exception and let the query execute gracefully. The case where executor has been erratically shut down is a good point, but for edges like this, we should probably override IndexSearcher's default behaviour. If it helps, we could potentially introduce a toggle which is enabled by default and indicates that IndexSearcher will perform the graceful handling, and if the user wishes the error to propagate, just call a new API method to disable the toggle? > IndexSearcher Should Handle Rejection of Concurrent Task > > > Key: LUCENE-8989 > URL: https://issues.apache.org/jira/browse/LUCENE-8989 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > As discussed in [https://github.com/apache/lucene-solr/pull/815,] > IndexSearcher should handle the case when the executor rejects the execution > of a task (unavailability of threads?). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9114) Add FunctionValues.cost
[ https://issues.apache.org/jira/browse/LUCENE-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048261#comment-17048261 ] Atri Sharma commented on LUCENE-9114: - I strongly believe that this is the right approach and we should be pursuing this. I am actively working on this and will post a patch by Monday morning > Add FunctionValues.cost > --- > > Key: LUCENE-9114 > URL: https://issues.apache.org/jira/browse/LUCENE-9114 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/query >Reporter: David Smiley >Priority: Major > > The FunctionRangeQuery uses FunctionValues.getRangeScorer which returns a > subclass of ValueSourceScorer. VSC's TwoPhaseIterator has a matchCost impl > that returns a constant 100. This is pretty terrible; the cost should vary > based on the complexity of the ValueSource provided to FRQ. ValueSource's > are typically nested a number of levels, so they should aggregate. > BTW there is a parallel concern for FunctionMatchQuery which works with > DoubleValuesSource which doesn't have a cost either, and unsurprisingly there > is a TPI with matchCost 100 there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9114) Add FunctionValues.cost
[ https://issues.apache.org/jira/browse/LUCENE-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048492#comment-17048492 ] Atri Sharma commented on LUCENE-9114: - [~dsmiley] I have raised a PR for the same -- it is a minimalistic change to allow VSS to incorporate the delegated FunctionValues's cost into its cost. Would that help you get unblocked by adding stacked FunctionValues with custom costing functions? > Add FunctionValues.cost > --- > > Key: LUCENE-9114 > URL: https://issues.apache.org/jira/browse/LUCENE-9114 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/query >Reporter: David Smiley >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The FunctionRangeQuery uses FunctionValues.getRangeScorer which returns a > subclass of ValueSourceScorer. VSC's TwoPhaseIterator has a matchCost impl > that returns a constant 100. This is pretty terrible; the cost should vary > based on the complexity of the ValueSource provided to FRQ. ValueSource's > are typically nested a number of levels, so they should aggregate. > BTW there is a parallel concern for FunctionMatchQuery which works with > DoubleValuesSource which doesn't have a cost either, and unsurprisingly there > is a TPI with matchCost 100 there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9114) Add FunctionValues.cost
[ https://issues.apache.org/jira/browse/LUCENE-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned LUCENE-9114: --- Assignee: Atri Sharma > Add FunctionValues.cost > --- > > Key: LUCENE-9114 > URL: https://issues.apache.org/jira/browse/LUCENE-9114 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/query >Reporter: David Smiley >Assignee: Atri Sharma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The FunctionRangeQuery uses FunctionValues.getRangeScorer which returns a > subclass of ValueSourceScorer. VSC's TwoPhaseIterator has a matchCost impl > that returns a constant 100. This is pretty terrible; the cost should vary > based on the complexity of the ValueSource provided to FRQ. ValueSource's > are typically nested a number of levels, so they should aggregate. > BTW there is a parallel concern for FunctionMatchQuery which works with > DoubleValuesSource which doesn't have a cost either, and unsurprisingly there > is a TPI with matchCost 100 there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9262) Better Cost Models for Common FunctionValues Implementations
[ https://issues.apache.org/jira/browse/LUCENE-9262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051786#comment-17051786 ] Atri Sharma commented on LUCENE-9262: - cc [~dsmiley] > Better Cost Models for Common FunctionValues Implementations > > > Key: LUCENE-9262 > URL: https://issues.apache.org/jira/browse/LUCENE-9262 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > > In follow up to LUCENE-9114, we should pursue and have better cost models for > classes like DoubleValuesSource, IntFieldsSource etc -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9114) Add FunctionValues.cost
[ https://issues.apache.org/jira/browse/LUCENE-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved LUCENE-9114. - Resolution: Fixed > Add FunctionValues.cost > --- > > Key: LUCENE-9114 > URL: https://issues.apache.org/jira/browse/LUCENE-9114 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/query >Reporter: David Smiley >Assignee: Atri Sharma >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > The FunctionRangeQuery uses FunctionValues.getRangeScorer which returns a > subclass of ValueSourceScorer. VSC's TwoPhaseIterator has a matchCost impl > that returns a constant 100. This is pretty terrible; the cost should vary > based on the complexity of the ValueSource provided to FRQ. ValueSource's > are typically nested a number of levels, so they should aggregate. > BTW there is a parallel concern for FunctionMatchQuery which works with > DoubleValuesSource which doesn't have a cost either, and unsurprisingly there > is a TPI with matchCost 100 there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9255) ValueSource Has Generic Typing Issues
Atri Sharma created LUCENE-9255: --- Summary: ValueSource Has Generic Typing Issues Key: LUCENE-9255 URL: https://issues.apache.org/jira/browse/LUCENE-9255 Project: Lucene - Core Issue Type: Improvement Reporter: Atri Sharma ValueSource uses a bunch of weakly typed members which raises compiler issues. We need to fix this in ValueSource and all of its subclasses. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9114) Add FunctionValues.cost
[ https://issues.apache.org/jira/browse/LUCENE-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026048#comment-17026048 ] Atri Sharma commented on LUCENE-9114: - [~dsmiley] This looks interesting -- I am happy to hack on this one unless you are planning to. Please let me know. > Add FunctionValues.cost > --- > > Key: LUCENE-9114 > URL: https://issues.apache.org/jira/browse/LUCENE-9114 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/query >Reporter: David Smiley >Priority: Major > > The FunctionRangeQuery uses FunctionValues.getRangeScorer which returns a > subclass of ValueSourceScorer. VSC's TwoPhaseIterator has a matchCost impl > that returns a constant 100. This is pretty terrible; the cost should vary > based on the complexity of the ValueSource provided to FRQ. ValueSource's > are typically nested a number of levels, so they should aggregate. > BTW there is a parallel concern for FunctionMatchQuery which works with > DoubleValuesSource which doesn't have a cost either, and unsurprisingly there > is a TPI with matchCost 100 there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9114) Add FunctionValues.cost
[ https://issues.apache.org/jira/browse/LUCENE-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17026083#comment-17026083 ] Atri Sharma commented on LUCENE-9114: - +1 - I am on it. I am a bit groggy from sleep right now -- will post my thoughts tomorrow. > Add FunctionValues.cost > --- > > Key: LUCENE-9114 > URL: https://issues.apache.org/jira/browse/LUCENE-9114 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/query >Reporter: David Smiley >Priority: Major > > The FunctionRangeQuery uses FunctionValues.getRangeScorer which returns a > subclass of ValueSourceScorer. VSC's TwoPhaseIterator has a matchCost impl > that returns a constant 100. This is pretty terrible; the cost should vary > based on the complexity of the ValueSource provided to FRQ. ValueSource's > are typically nested a number of levels, so they should aggregate. > BTW there is a parallel concern for FunctionMatchQuery which works with > DoubleValuesSource which doesn't have a cost either, and unsurprisingly there > is a TPI with matchCost 100 there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9164) Should not consider ACE a tragedy if IW is closed
[ https://issues.apache.org/jira/browse/LUCENE-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024461#comment-17024461 ] Atri Sharma commented on LUCENE-9164: - I am in dual minds on this -- isnt trying to close an already closed IndexWriter a sign of a potentially fatal bug in the user code? This changes user facing behaviour (unless I am reading the patch wrong) so would want to understand the reasoning for this change. > Should not consider ACE a tragedy if IW is closed > - > > Key: LUCENE-9164 > URL: https://issues.apache.org/jira/browse/LUCENE-9164 > Project: Lucene - Core > Issue Type: Bug > Components: core/index >Affects Versions: master (9.0), 8.5, 8.4.2 >Reporter: Nhat Nguyen >Assignee: Nhat Nguyen >Priority: Major > Attachments: LUCENE-9164.patch, LUCENE-9164.patch > > > If IndexWriter is closed or being closed, AlreadyClosedException is expected. > We should not consider it a tragic event in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072602#comment-17072602 ] Atri Sharma commented on LUCENE-9299: - Can you open a PR instead? Easier to review. I was split with the same decision when working on 8811. IMO, we should be throwing an error when the max clause count is *breached*, which is what the current check accomplishes. Hence this was intentionally done. Keep in mind that this can be potentially a breaking change -- some queries that worked before might stop working all of a sudden post an upgrade. > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072722#comment-17072722 ] Atri Sharma commented on LUCENE-9299: - Ok, I am misreading your patch then -- will give it a run later today. Thanks for picking it up. > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9299) ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses in a query exceeds the limit by 1
[ https://issues.apache.org/jira/browse/LUCENE-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned LUCENE-9299: --- Assignee: Atri Sharma > ClausesCheckVisitor fails to throw TooManyClauses exception when the clauses > in a query exceeds the limit by 1 > -- > > Key: LUCENE-9299 > URL: https://issues.apache.org/jira/browse/LUCENE-9299 > Project: Lucene - Core > Issue Type: Bug >Affects Versions: master (9.0) >Reporter: Stamatis Zampetakis >Assignee: Atri Sharma >Priority: Trivial > Attachments: LUCENE-9299.patch > > > The visitor that was introduced in LUCENE-8811 (obtained by > IndexSearcher#getNumClausesCheckVisitor) that checks that the number of > clauses in query does not exceed a certain limit (defined by > org.apache.lucene.search.IndexSearcher#setMaxClauseCount) misses the case > where the clauses/terms in the query exceed the limit by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9335) Add a bulk scorer for disjunctions that does dynamic pruning
[ https://issues.apache.org/jira/browse/LUCENE-9335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089637#comment-17089637 ] Atri Sharma commented on LUCENE-9335: - +1. I was looking at comparisons with Vespa recently and noticed something similar. Are you planning to work on this? Can take this up if you would like. > Add a bulk scorer for disjunctions that does dynamic pruning > > > Key: LUCENE-9335 > URL: https://issues.apache.org/jira/browse/LUCENE-9335 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > > Lucene often gets benchmarked against other engines, e.g. against Tantivy and > PISA at [https://tantivy-search.github.io/bench/] or against research > prototypes in Table 1 of > [https://cs.uwaterloo.ca/~jimmylin/publications/Grand_etal_ECIR2020_preprint.pdf]. > Given that top-level disjunctions of term queries are commonly used for > benchmarking, it would be nice to optimize this case a bit more, I suspect > that we could make fewer per-document decisions by implementing a BulkScorer > instead of a Scorer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9373) Allow FunctionMatchQuery to customize matchCost of TwoPhaseIterator
[ https://issues.apache.org/jira/browse/LUCENE-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108886#comment-17108886 ] Atri Sharma commented on LUCENE-9373: - +1 > Allow FunctionMatchQuery to customize matchCost of TwoPhaseIterator > --- > > Key: LUCENE-9373 > URL: https://issues.apache.org/jira/browse/LUCENE-9373 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/queries >Reporter: David Smiley >Priority: Major > > FunctionMatchQuery internally has a TwoPhaseIterator using a constant > matchCost. If it were customizable by the query, the user could control this > ordering. I propose an optional matchCost via an overloaded constructor. > Ideally the DoubleValues abstraction would have a matchCost but it doesn't, > and even if it did, the user might just want real control over this at a > query construction/parse level. > See similar LUCENE-9114 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9114) Add FunctionValues.cost
[ https://issues.apache.org/jira/browse/LUCENE-9114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108885#comment-17108885 ] Atri Sharma commented on LUCENE-9114: - [~dsmiley] If you recall, that was one of the ways that I had done in the iteration for this PR :) I agree with allowing the Weight to define an internal cost that ValueSourceScorer.matchCost can delegate to – it can be return a weird value (Float.NEGATIVE_INFINITY) to define that it is not implemented and then it is the matchCost's job to ensure that it does the right thing? > Add FunctionValues.cost > --- > > Key: LUCENE-9114 > URL: https://issues.apache.org/jira/browse/LUCENE-9114 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/query >Reporter: David Smiley >Assignee: Atri Sharma >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > The FunctionRangeQuery uses FunctionValues.getRangeScorer which returns a > subclass of ValueSourceScorer. VSC's TwoPhaseIterator has a matchCost impl > that returns a constant 100. This is pretty terrible; the cost should vary > based on the complexity of the ValueSource provided to FRQ. ValueSource's > are typically nested a number of levels, so they should aggregate. > BTW there is a parallel concern for FunctionMatchQuery which works with > DoubleValuesSource which doesn't have a cost either, and unsurprisingly there > is a TPI with matchCost 100 there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9074) Account for Executor's Queue Length When Planning Slices in IndexSearcher
[ https://issues.apache.org/jira/browse/LUCENE-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110501#comment-17110501 ] Atri Sharma commented on LUCENE-9074: - No, this Jira is completed. Sorry, I forgot to close this after completion. > Account for Executor's Queue Length When Planning Slices in IndexSearcher > - > > Key: LUCENE-9074 > URL: https://issues.apache.org/jira/browse/LUCENE-9074 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9074) Account for Executor's Queue Length When Planning Slices in IndexSearcher
[ https://issues.apache.org/jira/browse/LUCENE-9074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved LUCENE-9074. - Resolution: Fixed > Account for Executor's Queue Length When Planning Slices in IndexSearcher > - > > Key: LUCENE-9074 > URL: https://issues.apache.org/jira/browse/LUCENE-9074 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 10h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13528) Rate limiting in Solr
[ https://issues.apache.org/jira/browse/SOLR-13528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma updated SOLR-13528: --- Fix Version/s: master (9.0) > Rate limiting in Solr > - > > Key: SOLR-13528 > URL: https://issues.apache.org/jira/browse/SOLR-13528 > Project: Solr > Issue Type: New Feature >Reporter: Anshum Gupta >Assignee: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Time Spent: 11.5h > Remaining Estimate: 0h > > In relation to SOLR-13527, Solr also needs a way to throttle update and > search requests based on usage metrics. This is the umbrella JIRA for both > update and search rate limiting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11611) Starting Solr using solr.cmd fails under Windows, when the path contains a parenthesis character
[
https://issues.apache.org/jira/browse/SOLR-11611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216599#comment-17216599
]
Atri Sharma commented on SOLR-11611:
Whats the status of this? Is this a 8.7 blocker?
> Starting Solr using solr.cmd fails under Windows, when the path contains a
> parenthesis character
>
>
> Key: SOLR-11611
> URL: https://issues.apache.org/jira/browse/SOLR-11611
> Project: Solr
> Issue Type: Bug
> Components: scripts and tools, SolrCLI
>Affects Versions: 7.1, 7.4
> Environment: Microsoft Windows [Version 10.0.15063]
> java version "1.8.0_144"
> Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
> Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
>Reporter: Jakob Furrer
>Priority: Blocker
> Fix For: 8.7
>
> Attachments: SOLR-11611.patch, SOLR-11611.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Starting Solr using solr.cli fails in Windows, when the path contains spaces.
> Use the following example to reproduce the error:
> {quote}C:\>c:
> C:\>cd "C:\Program Files (x86)\Company Name\ProductName Solr\bin"
> C:\Program Files (x86)\Company Name\ProductName Solr\bin>dir
> Volume in Laufwerk C: hat keine Bezeichnung.
> Volumeseriennummer: 8207-3B8B
> Verzeichnis von C:\Program Files (x86)\Company Name\ProductName Solr\bin
> 06.11.2017 15:52 .
> 06.11.2017 15:52 ..
> 06.11.2017 15:39 init.d
> 03.11.2017 17:32 8 209 post
> 03.11.2017 17:3275 963 solr
> 06.11.2017 14:2469 407 solr.cmd
>3 Datei(en),153 579 Bytes
>3 Verzeichnis(se), 51 191 619 584 Bytes frei
> C:\Program Files (x86)\Company Name\ProductName Solr\bin>solr.cmd start
> *"\Company" kann syntaktisch an dieser Stelle nicht verarbeitet werden.*
> C:\Program Files (x86)\Company Name\ProductName Solr\bin>{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14354) HttpShardHandler send requests in async
[
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216596#comment-17216596
]
Atri Sharma commented on SOLR-14354:
Where are we with this issue? This is marked as a blocker for 8.7 – is that
still the case?
> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Blocker
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png,
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n
> equals to number of shards) to an executor. So each request will correspond
> to a thread, after sending a request that thread basically do nothing just
> waiting for response from other side. That thread will be swapped out and CPU
> will try to handle another thread (this is called context switch, CPU will
> save the context of the current thread and switch to another one). When some
> data (not all) come back, that thread will be called to parsing these data,
> then it will wait until more data come back. So there will be lots of context
> switching in CPU. That is quite inefficient on using threads.Basically we
> want less threads and most of them must busy all the time, because threads
> are not free as well as context switching. That is the main idea behind
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path;)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without
> any block. Then when the client received the header from other side, it will
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not
> all response) from the data it will call {{onContent(buffer)}} listeners.
> When everything finished it will call {{onComplete}} listeners. One main
> thing that will must notice here is all listeners should finish quick, if the
> listener block, all further data of that request won’t be handled until the
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
> Jetty HttpClient already provides several listeners, one of them is
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
> // Obtain the input stream on the response content
> try (InputStream input = listener.getInputStream()) {
> // Read the response content
> }
> } {code}
> In this case, there will be 2 thread
> * one thread trying to read the response content from InputStream
> * one thread (this is a short-live task) feeding content to above
> InputStream whenever some byte[] is available. Note that if this thread
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can be written into
> something like this
> {code:java}
> handler.sendReq(req, (is) -> {
> executor.submit(() ->
> try (is) {
> // Read the content from InputStream
> }
> )
> }) {code}
> The first diagram will be changed into this
> !image-2020-03-23-10-09-10-221.png!
> Notice that although “sending req to shard1” is wide, it won’t take long time
> since sending req is a very quick operation. With this operation, handling
> threads won’t be spin up until first bytes are sent back. Notice that in this
> approach we still have active threads waiting for more data from InputStream
> h2. 4. Solution 2: Buffering data and handle it inside jetty’s thread.
> Jetty have another listener called BufferingResponseListener. This is how it
> is get used
> {code:java}
> client.newRequest(...).send(new BufferingResponseListener() {
> public void onComplete(Result result) {
> try {
> byte[] response =
[jira] [Commented] (SOLR-14067) Deprecate StatelessScriptUpdateProcessor and disabled by default
[ https://issues.apache.org/jira/browse/SOLR-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216598#comment-17216598 ] Atri Sharma commented on SOLR-14067: Is this a 8.7 blocker? > Deprecate StatelessScriptUpdateProcessor and disabled by default > > > Key: SOLR-14067 > URL: https://issues.apache.org/jira/browse/SOLR-14067 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Priority: Blocker > Fix For: 8.7 > > Time Spent: 10m > Remaining Estimate: 0h > > We should eliminate all scripting capabilities within Solr. Let us start with > the StatelessScriptUpdateProcessor deprecation/removal. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13973) Deprecate Tika
[ https://issues.apache.org/jira/browse/SOLR-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17216597#comment-17216597 ] Atri Sharma commented on SOLR-13973: Do we want to treat this as. a blocker for 8.7? > Deprecate Tika > -- > > Key: SOLR-13973 > URL: https://issues.apache.org/jira/browse/SOLR-13973 > Project: Solr > Issue Type: Improvement >Reporter: Ishan Chattopadhyaya >Priority: Blocker > Fix For: 8.7 > > Time Spent: 10m > Remaining Estimate: 0h > > Solr's primary responsibility should be to focus on search and scalability. > Having to deal with the problems (CVEs) of Velocity, Tika etc. can slow us > down. I propose that we deprecate it going forward. > Tika can be run outside Solr. Going forward, if someone wants to use these, > it should be possible to bring them into third party packages and installed > via package manager. > Plan is to just to throw warnings in logs and add deprecation notes in > reference guide for now. Removal can be done in 9.0. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217027#comment-17217027 ] Atri Sharma commented on SOLR-14588: Thanks for catching it, David! I backported the package-info.json to branch_8x and branch_8_7 – precommit should be fine now? > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Blocker > Fix For: master (9.0), 8.7 > > Time Spent: 13h 50m > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14772) Reproducible failure TestConfigOverlay
[ https://issues.apache.org/jira/browse/SOLR-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved SOLR-14772. Resolution: Fixed > Reproducible failure TestConfigOverlay > -- > > Key: SOLR-14772 > URL: https://issues.apache.org/jira/browse/SOLR-14772 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Reporter: Erick Erickson >Assignee: Atri Sharma >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > ./gradlew :solr:core:test --tests "org.apache.solr.core.TestConfigOverlay" > -Ptests.seed=E6AFE16CC61929A6 -Ptests.file.encoding=US-ASCII > > this fails too: > > ant test -Dtestcase=TestConfigOverlay -Dtests.method=testPaths > -Dtests.seed=A7D552FAB7A88C69 -Dtests.multiplier=2 -Dtests.slow=true > -Dtests.locale=ff-GN -Dtests.timezone=Australia/Adelaide -Dtests.asserts=true > -Dtests.file.encoding=US-ASCII -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9480) investigate slow DataInput.skipBytes
[
https://issues.apache.org/jira/browse/LUCENE-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183774#comment-17183774
]
Atri Sharma commented on LUCENE-9480:
-
+1.
I have scratched my head over the difference between these two, and still dont
have a conclusive answer.
> investigate slow DataInput.skipBytes
>
>
> Key: LUCENE-9480
> URL: https://issues.apache.org/jira/browse/LUCENE-9480
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Robert Muir
>Priority: Major
>
> Currently DataInput has skipBytes(), but IndexInput also adds seek().
> There isn't a clear reason about the differences in the two methods: why
> would you choose one over the other?
> It causes some performance issues: for example the default implementation
> actually reads bytes into a byte array and throws everything away. This is
> really silly for MMapDirectory: skipping bytes should only be a glorified
> {{+=}}.
> So when I look at latest LUCENE-9447 patch, I can't help but think a ton of
> waste is happening:
> * Maybe skipBytes() is only used because the stored fields compressor
> interface happens to take DataInput? Should it take IndexInput instead?
> * Should skipBytes() be overridden by MMapDirectory rather than delegating to
> super? doing real reads and byte array copies isn't free. It should be a
> {{+=}} with single bounds check.
> * Should we revisit having DataInput vs IndexInput at all? Maybe they should
> be collapsed into one thing?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14755) Dedicated Exception Type for Circuit Breakers
Atri Sharma created SOLR-14755: -- Summary: Dedicated Exception Type for Circuit Breakers Key: SOLR-14755 URL: https://issues.apache.org/jira/browse/SOLR-14755 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Atri Sharma Circuit breakers should raise a dedicated exception when triggered. This Jira tracks the effort. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved SOLR-14588. Resolution: Fixed Fixed configuration in [https://github.com/apache/lucene-solr/commit/6a7da3cd508f5b1e445df08aa2f1fa926d586e99] > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Blocker > Fix For: master (9.0), 8.7 > > Time Spent: 13h 50m > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-6930) Provide "Circuit Breakers" For Expensive Solr Queries
[ https://issues.apache.org/jira/browse/SOLR-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved SOLR-6930. --- Resolution: Fixed > Provide "Circuit Breakers" For Expensive Solr Queries > - > > Key: SOLR-6930 > URL: https://issues.apache.org/jira/browse/SOLR-6930 > Project: Solr > Issue Type: Improvement > Components: search >Reporter: Mike Drob >Priority: Major > Attachments: SOLR-6930.patch, SOLR-6930.patch, SOLR-6930.patch, > SOLR-6930.patch, SOLR-6930.patch, SOLR-6930.patch > > > Ref: > http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html > ES currently allows operators to configure "circuit breakers" to preemptively > fail queries that are estimated too large rather than allowing an OOM > Exception to happen. We might be able to do the same thing. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14772) Reproducible failure TestConfigOverlay
[ https://issues.apache.org/jira/browse/SOLR-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183477#comment-17183477 ] Atri Sharma commented on SOLR-14772: I have pushed a change to fix this test. > Reproducible failure TestConfigOverlay > -- > > Key: SOLR-14772 > URL: https://issues.apache.org/jira/browse/SOLR-14772 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Reporter: Erick Erickson >Assignee: Atri Sharma >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > ./gradlew :solr:core:test --tests "org.apache.solr.core.TestConfigOverlay" > -Ptests.seed=E6AFE16CC61929A6 -Ptests.file.encoding=US-ASCII > > this fails too: > > ant test -Dtestcase=TestConfigOverlay -Dtests.method=testPaths > -Dtests.seed=A7D552FAB7A88C69 -Dtests.multiplier=2 -Dtests.slow=true > -Dtests.locale=ff-GN -Dtests.timezone=Australia/Adelaide -Dtests.asserts=true > -Dtests.file.encoding=US-ASCII -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14772) Reproducible failure TestConfigOverlay
[ https://issues.apache.org/jira/browse/SOLR-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183393#comment-17183393 ] Atri Sharma commented on SOLR-14772: Thanks for investigating this one, [~erickerickson]. I have raised a PR for this one – please see and let me know. > Reproducible failure TestConfigOverlay > -- > > Key: SOLR-14772 > URL: https://issues.apache.org/jira/browse/SOLR-14772 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Reporter: Erick Erickson >Assignee: Atri Sharma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > ./gradlew :solr:core:test --tests "org.apache.solr.core.TestConfigOverlay" > -Ptests.seed=E6AFE16CC61929A6 -Ptests.file.encoding=US-ASCII > > this fails too: > > ant test -Dtestcase=TestConfigOverlay -Dtests.method=testPaths > -Dtests.seed=A7D552FAB7A88C69 -Dtests.multiplier=2 -Dtests.slow=true > -Dtests.locale=ff-GN -Dtests.timezone=Australia/Adelaide -Dtests.asserts=true > -Dtests.file.encoding=US-ASCII -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14615) CPU Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved SOLR-14615. Resolution: Fixed > CPU Based Circuit Breaker > - > > Key: SOLR-14615 > URL: https://issues.apache.org/jira/browse/SOLR-14615 > Project: Solr > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 4h 40m > Remaining Estimate: 0h > > We should have a circuit breaker that can monitor and trigger on CPU > utilization -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14354) HttpShardHandler send requests in async
[
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17220531#comment-17220531
]
Atri Sharma commented on SOLR-14354:
Is this no longer a blocker?
> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Blocker
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png,
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png,
> image-2020-10-23-16-45-20-034.png, image-2020-10-23-16-45-21-789.png,
> image-2020-10-23-16-45-37-628.png
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n
> equals to number of shards) to an executor. So each request will correspond
> to a thread, after sending a request that thread basically do nothing just
> waiting for response from other side. That thread will be swapped out and CPU
> will try to handle another thread (this is called context switch, CPU will
> save the context of the current thread and switch to another one). When some
> data (not all) come back, that thread will be called to parsing these data,
> then it will wait until more data come back. So there will be lots of context
> switching in CPU. That is quite inefficient on using threads.Basically we
> want less threads and most of them must busy all the time, because threads
> are not free as well as context switching. That is the main idea behind
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path;)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without
> any block. Then when the client received the header from other side, it will
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not
> all response) from the data it will call {{onContent(buffer)}} listeners.
> When everything finished it will call {{onComplete}} listeners. One main
> thing that will must notice here is all listeners should finish quick, if the
> listener block, all further data of that request won’t be handled until the
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
> Jetty HttpClient already provides several listeners, one of them is
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
> // Obtain the input stream on the response content
> try (InputStream input = listener.getInputStream()) {
> // Read the response content
> }
> } {code}
> In this case, there will be 2 thread
> * one thread trying to read the response content from InputStream
> * one thread (this is a short-live task) feeding content to above
> InputStream whenever some byte[] is available. Note that if this thread
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can be written into
> something like this
> {code:java}
> handler.sendReq(req, (is) -> {
> executor.submit(() ->
> try (is) {
> // Read the content from InputStream
> }
> )
> }) {code}
> The first diagram will be changed into this
> !image-2020-03-23-10-09-10-221.png!
> Notice that although “sending req to shard1” is wide, it won’t take long time
> since sending req is a very quick operation. With this operation, handling
> threads won’t be spin up until first bytes are sent back. Notice that in this
> approach we still have active threads waiting for more data from InputStream
> h2. 4. Solution 2: Buffering data and handle it inside jetty’s thread.
> Jetty have another listener called BufferingResponseListener. This is how it
> is get used
> {code:java}
> client.newRequest(...).send(new BufferingResponseListener() {
> public void onComplete(Result
[jira] [Commented] (LUCENE-9432) Use LinkedList instead of manual array re-sizing for better throughput.
[
https://issues.apache.org/jira/browse/LUCENE-9432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17162174#comment-17162174
]
Atri Sharma commented on LUCENE-9432:
-
I think the first step is to rewrite the code with the Deque interface – it is
just more logical and readable. Post that, we can play around with the
underlying implementation (I am in favour of ArrayDeque).
> Use LinkedList instead of manual array re-sizing for better throughput.
> ---
>
> Key: LUCENE-9432
> URL: https://issues.apache.org/jira/browse/LUCENE-9432
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/codecs
>Reporter: Mohammad Sadiq
>Priority: Minor
> Labels: patch-available, performance, pull-request-available
> Attachments: LUCENE-9432.patch
>
>
> I observed that using {{LinkedList}} instead of manually re-sizing and
> copying {{SegmentTermEnumFrame}}s improves red-line QPS. Does it make sense
> to include this?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14720) Validate Sanctity of Request Type
Atri Sharma created SOLR-14720: -- Summary: Validate Sanctity of Request Type Key: SOLR-14720 URL: https://issues.apache.org/jira/browse/SOLR-14720 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Atri Sharma https://issues.apache.org/jira/browse/SOLR-13528 introduces a mechanism to identify between internal (server) and external (client) requests. Currently, this mechanism works on populating a relevant field in the request's headers. However, a rogue client can impersonate or fabricate a server request. This Jira tracks effort to validate that a client request's context is set correctly. We look to tap into the authentication loop to piggy back on the information provided there. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14722) timeAllowed should track from the very start of the request
[
https://issues.apache.org/jira/browse/SOLR-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17173435#comment-17173435
]
Atri Sharma commented on SOLR-14722:
Huge +1.
> timeAllowed should track from the very start of the request
> ---
>
> Key: SOLR-14722
> URL: https://issues.apache.org/jira/browse/SOLR-14722
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
>
> The "timeAllowed" param starts tracking time from the moment
> {{SolrQueryTimeoutImpl.set}} is called. I think it ought to track from when
> the request actually started, which is {{SolrQueryRequest.getRequestTimer}}.
> Lazy core loading can substantially increase the delta. Additionally, I'd
> like to make some small improvements to SolrQueryTimeoutImpl to make it
> easier to use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14709) Priority Based Model for Slot Borrowing in Request Rate Limiting
Atri Sharma created SOLR-14709: -- Summary: Priority Based Model for Slot Borrowing in Request Rate Limiting Key: SOLR-14709 URL: https://issues.apache.org/jira/browse/SOLR-14709 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Atri Sharma When a request exceeds the quota of its request type, it can borrow a slot from another request rate limiter. Today, the request rate limiter chosen to be checked for slot borrowing is round robin. This should be a priority queue based model. This Jira tracks the same. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14710) Finer Grained Rate Limiting
Atri Sharma created SOLR-14710: -- Summary: Finer Grained Rate Limiting Key: SOLR-14710 URL: https://issues.apache.org/jira/browse/SOLR-14710 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Atri Sharma The default rate limiting model enable rate limiting at a node/JVM level. This Jira tracks a future effort for finer grained ate limiting at core/collection level. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13350) Explore collector managers for multi-threaded search
[ https://issues.apache.org/jira/browse/SOLR-13350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17176774#comment-17176774 ] Atri Sharma commented on SOLR-13350: I have been bringing this up to date and fixing some outstanding comments at: [https://github.com/atris/lucene-solr/tree/solr-13350] > Explore collector managers for multi-threaded search > > > Key: SOLR-13350 > URL: https://issues.apache.org/jira/browse/SOLR-13350 > Project: Solr > Issue Type: New Feature >Reporter: Ishan Chattopadhyaya >Assignee: Ishan Chattopadhyaya >Priority: Major > Attachments: SOLR-13350.patch, SOLR-13350.patch, SOLR-13350.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > AFAICT, SolrIndexSearcher can be used only to search all the segments of an > index in series. However, using CollectorManagers, segments can be searched > concurrently and result in reduced latency. Opening this issue to explore the > effectiveness of using CollectorManagers in SolrIndexSearcher from latency > and throughput perspective. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13528) Rate limiting in Solr
[ https://issues.apache.org/jira/browse/SOLR-13528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175298#comment-17175298 ] Atri Sharma commented on SOLR-13528: Thanks for fixing this, [~ctargett] ! > Rate limiting in Solr > - > > Key: SOLR-13528 > URL: https://issues.apache.org/jira/browse/SOLR-13528 > Project: Solr > Issue Type: New Feature >Reporter: Anshum Gupta >Assignee: Atri Sharma >Priority: Major > Time Spent: 9h 40m > Remaining Estimate: 0h > > In relation to SOLR-13527, Solr also needs a way to throttle update and > search requests based on usage metrics. This is the umbrella JIRA for both > update and search rate limiting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17175453#comment-17175453 ] Atri Sharma commented on SOLR-14588: As discussed offline, this can be done as a refactor before 8.7 – hence leaving this Jira open to track that specific effort. > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Blocker > Fix For: master (9.0), 8.7 > > Time Spent: 13h 50m > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14615) CPU Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned SOLR-14615: -- Assignee: Atri Sharma > CPU Based Circuit Breaker > - > > Key: SOLR-14615 > URL: https://issues.apache.org/jira/browse/SOLR-14615 > Project: Solr > Issue Type: Improvement >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > > We should have a circuit breaker that can monitor and trigger on CPU > utilization -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14609) TestJsonFacetsWithNestedObjects Has Unused Imports
[ https://issues.apache.org/jira/browse/SOLR-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148862#comment-17148862 ] Atri Sharma commented on SOLR-14609: Fixed in https://github.com/apache/lucene-solr/commit/d1c29ae8a9481d87602b54edb608a1cc36be3e8a > TestJsonFacetsWithNestedObjects Has Unused Imports > -- > > Key: SOLR-14609 > URL: https://issues.apache.org/jira/browse/SOLR-14609 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14609) TestJsonFacetsWithNestedObjects Has Unused Imports
Atri Sharma created SOLR-14609: -- Summary: TestJsonFacetsWithNestedObjects Has Unused Imports Key: SOLR-14609 URL: https://issues.apache.org/jira/browse/SOLR-14609 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Atri Sharma -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14609) TestJsonFacetsWithNestedObjects Has Unused Imports
[ https://issues.apache.org/jira/browse/SOLR-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned SOLR-14609: -- Assignee: Atri Sharma > TestJsonFacetsWithNestedObjects Has Unused Imports > -- > > Key: SOLR-14609 > URL: https://issues.apache.org/jira/browse/SOLR-14609 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154882#comment-17154882 ] Atri Sharma commented on SOLR-14588: Thanks for highlighting -- I am taking a look. Can you please point to where I can monitor future Jenkins build? > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Time Spent: 13h 50m > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14621) Deprecate and eventually remove HDFS support
[ https://issues.apache.org/jira/browse/SOLR-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151058#comment-17151058 ] Atri Sharma commented on SOLR-14621: FWIW, I am +1 to this. I suggest that we move this to a a plugin and remove it in a major version > Deprecate and eventually remove HDFS support > > > Key: SOLR-14621 > URL: https://issues.apache.org/jira/browse/SOLR-14621 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Ishan Chattopadhyaya >Priority: Major > > This issue is to deprecate HDFS support from Solr. HDFS clutter should not be > part of Solr core and should either be removed completely or be packaged up > as an optional module. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154897#comment-17154897 ] Atri Sharma commented on SOLR-14588: I beasted the test under high CPU stress -- still could not reproduce the same. I will push a change to add logging tomorrow. > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Time Spent: 13h 50m > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma resolved SOLR-14588. Resolution: Fixed > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: master (9.0) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 13.5h > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14615) CPU Based Circuit Breaker
Atri Sharma created SOLR-14615: -- Summary: CPU Based Circuit Breaker Key: SOLR-14615 URL: https://issues.apache.org/jira/browse/SOLR-14615 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Atri Sharma We should have a circuit breaker that can monitor and trigger on CPU utilization -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
Atri Sharma created SOLR-14588: -- Summary: Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker Key: SOLR-14588 URL: https://issues.apache.org/jira/browse/SOLR-14588 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Atri Sharma This Jira tracks addition of circuit breakers in the search path and implements JVM based circuit breaker which rejects incoming search requests if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned SOLR-14588: -- Assignee: Atri Sharma > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145099#comment-17145099 ] Atri Sharma commented on SOLR-14588: [~ab] Thanks for commenting. I will add documentation and update the CHANGES entry accordingly. > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145057#comment-17145057 ] Atri Sharma commented on SOLR-14588: [~mdrob] Yes, realised that post pushing. Pushing a fix now. > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 7h 40m > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14591) Move JoinQuery in JoinQParserPlugin to its own class
[ https://issues.apache.org/jira/browse/SOLR-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17144033#comment-17144033 ] Atri Sharma commented on SOLR-14591: I I made it public since it seems weird to have an independent class which is package private but yet being inherited. However, I agree with your point. I will push a fix to make the visibility package again now, thanks for highlighting. > Move JoinQuery in JoinQParserPlugin to its own class > > > Key: SOLR-14591 > URL: https://issues.apache.org/jira/browse/SOLR-14591 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Atri Sharma >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Not sure why this isn't being flagged as a warning in master, but it did > cause an error on a Jenkins build, there's no reason NOT to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14591) Move JoinQuery in JoinQParserPlugin to its own class
[ https://issues.apache.org/jira/browse/SOLR-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Atri Sharma reassigned SOLR-14591: -- Assignee: Atri Sharma (was: Erick Erickson) > Move JoinQuery in JoinQParserPlugin to its own class > > > Key: SOLR-14591 > URL: https://issues.apache.org/jira/browse/SOLR-14591 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Atri Sharma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Not sure why this isn't being flagged as a warning in master, but it did > cause an error on a Jenkins build, there's no reason NOT to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14591) Move JoinQuery in JoinQParserPlugin to its own class
[ https://issues.apache.org/jira/browse/SOLR-14591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143933#comment-17143933 ] Atri Sharma commented on SOLR-14591: [~erickerickson] I have raised a PR for this – please take a look and let me know your thoughts. Also, I have taken the liberty of reassigning this issue to myself – hoping that is fine. > Move JoinQuery in JoinQParserPlugin to its own class > > > Key: SOLR-14591 > URL: https://issues.apache.org/jira/browse/SOLR-14591 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Atri Sharma >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Not sure why this isn't being flagged as a warning in master, but it did > cause an error on a Jenkins build, there's no reason NOT to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145999#comment-17145999 ] Atri Sharma commented on SOLR-14588: [~erickerickson] Thank you for proactively fixing the test! Did you get a chance to see the PR I raised late last night (https://github.com/apache/lucene-solr/pull/1615)? This fixes the same by not printing circuit breaker timings when circuit breakers are disabled. Would you feel that this is a longer term strategy to follow since this insulates the circuit breaker response messages breaking existing tests and only affecting ones which want to run with circuit breakers enabled? > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145999#comment-17145999 ] Atri Sharma edited comment on SOLR-14588 at 6/26/20, 4:14 AM: -- [~erickerickson] Thank you for proactively fixing the test! Did you get a chance to see the PR I raised late last night? (https://github.com/apache/lucene-solr/pull/1615) This fixes the same by not printing circuit breaker timings when circuit breakers are disabled. Would you feel that this is a longer term strategy to follow since this insulates the circuit breaker response messages breaking existing tests and only affecting ones which want to run with circuit breakers enabled? was (Author: atris): [~erickerickson] Thank you for proactively fixing the test! Did you get a chance to see the PR I raised late last night? (https://github.com/apache/lucene-solr/pull/1615)This fixes the same by not printing circuit breaker timings when circuit breakers are disabled. Would you feel that this is a longer term strategy to follow since this insulates the circuit breaker response messages breaking existing tests and only affecting ones which want to run with circuit breakers enabled? > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145999#comment-17145999 ] Atri Sharma edited comment on SOLR-14588 at 6/26/20, 4:14 AM: -- [~erickerickson] Thank you for proactively fixing the test! Did you get a chance to see the PR I raised late last night? (https://github.com/apache/lucene-solr/pull/1615)This fixes the same by not printing circuit breaker timings when circuit breakers are disabled. Would you feel that this is a longer term strategy to follow since this insulates the circuit breaker response messages breaking existing tests and only affecting ones which want to run with circuit breakers enabled? was (Author: atris): [~erickerickson] Thank you for proactively fixing the test! Did you get a chance to see the PR I raised late last night (https://github.com/apache/lucene-solr/pull/1615)? This fixes the same by not printing circuit breaker timings when circuit breakers are disabled. Would you feel that this is a longer term strategy to follow since this insulates the circuit breaker response messages breaking existing tests and only affecting ones which want to run with circuit breakers enabled? > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 8h > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14588) Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker
[ https://issues.apache.org/jira/browse/SOLR-14588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146486#comment-17146486 ] Atri Sharma commented on SOLR-14588: [~erickerickson] Thank you so much for covering the test failure yesterday, I definitely owe you one! Per our discussion, I have pushed a commit to only add timing information about circuit breakers when they are enabled, modify DebugTestComponent to expect the original query response and add a new test to check query response with debug enabled and circuit breakers enabled. Please let me know if it looks fine. > Circuit Breakers Infrastructure and Real JVM Based Circuit Breaker > -- > > Key: SOLR-14588 > URL: https://issues.apache.org/jira/browse/SOLR-14588 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Atri Sharma >Assignee: Atri Sharma >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > > This Jira tracks addition of circuit breakers in the search path and > implements JVM based circuit breaker which rejects incoming search requests > if the JVM heap usage exceeds a defined percentage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
