Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5380455302 I fixed the prek complaints and executed "gradlew tidy". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler merged PR #16431: URL: https://github.com/apache/lucene/pull/16431 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5374181698 The PREK check fails, unfortunately due to formatting issues. Maybe "tidy" wasn't executed. This is a precommit hook executed by Git if you commit (biut you need to install it). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5374071588 Looks fine to me, new constant test case is fine. I will fix the merge conflicts and merge it to main branch. Many thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5306260839 Sorry this got stale. Will verify test cases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
github-actions[bot] commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5304896957 This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the [email protected] list. Thank you for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5150718106 Sorry, I got busy in other responsibilities/tasks so couldn't give time to this.. Could you please verify the test-cases once? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5145937278 Can you add a test with a constant lambda function using INCREASING as well as DECREASING as well as NONE and verify it returns same results? This would also make clear that the above text is correct. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5141267432 I think this can be back-ported if possible as it gives considerable performance improvement in script scoring. Function scripts are used in almost everywhere and this fix should provide an improvement. There's no harm if users are able to take advantage of it without any major upgrade. :+1: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5128655636 Do you think this should also be backported or let us get it into Lucene 11 only. Please add a changes entry. I will wait for more time and think about the monotonicy again and try to figure out if everything works. In addition it would be good to add a test that checks witha constent function (`v -> 1.`) and assert same results for all three enum constants. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
msokolov commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676795267
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
thanks for sharing the definitions, OK by me to drop the `NON`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on PR #16431:
URL: https://github.com/apache/lucene/pull/16431#issuecomment-5121440072
I have performed benchmarking on wikimedia dataset using luceneutil and
below are the findings.
I have used a different query dataset than the one it uses as I didn't want
to run tests on Fuzzy queries and other types.
I was focused on Function Queries where results are modified by the function
that the user defines, more specifically monotonically increasing or decreasing.
The benchmark evaluates **20 distinct query permutations** formed by
combining **4 scoring function categories** with **5 query template
categories** across 100 query instances per category (total: 1,936 task
iterations across 5 JVM processes).
### A. Scoring Function Categories
1. **`Inc_*` (Monotonically Increasing Function)**
* **Formula**: $f(x) = x$ on numeric field `lastMod_skipper`.
* **Skipper Bound**: Upper bound evaluated from `skipper.maxValue(0)`.
2. **`DecNeg_*` (Monotonically Decreasing Negation Function)**
* **Formula**: $g(x) = -x$ on numeric field `lastMod_skipper`.
* **Skipper Bound**: Upper bound evaluated from `skipper.minValue(0)`.
3. **`DecInv_*` (Monotonically Decreasing Inverse Function)**
* **Formula**: $g(x) = \frac{100,000}{x}$ on numeric field
`lastMod_skipper`.
* **Skipper Bound**: Upper bound evaluated from `skipper.minValue(0)`.
4. **`TwoFields_*` (Multi-Field Composite Function)**
* **Formula**: Combines values from two fields (`lastMod_skipper +
dayOfYear_skipper`).
### B. Query Template Categories
1. **`HighTerm`**: Single-term query matching high-frequency corpus words
(e.g., `ref`, `title`, `name`, `cite`, `year`).
2. **`MedTerm`**: Single-term query matching medium-frequency technical
words (e.g., `algorithm`, `database`, `vector`, `search`, `lucene`).
3. **`LowTerm`**: Single-term query matching low-frequency rare words (e.g.,
`quantum`, `qubit`, `singularity`, `hologram`).
4. **`AndHighHigh`**: Conjunctive boolean query (`AND`) between two
high-frequency terms (e.g., `body:ref AND body:title`).
5. **`OrHighHigh`**: Disjunctive boolean query (`OR`) between two
high-frequency terms (e.g., `body:ref OR body:title`).
---
## 3. Detailed Benchmark Performance Results
| Row Name (Permutation) | Query Type Description|
Scoring Function | Baseline QPS (`main`) | Candidate QPS (`feature/...`) | QPS
Diff (%) | P100 Max Latency Reduction |
| :| :--|
:--| :-:| :-:|
::| :--:|
| **`DecNeg_HighTerm`** | Single High-Frequency Term Query |
$g(x) = -x$ | 157.67| **215.48**|
**+36.7%** | -20.4% |
| **`Inc_MedTerm`** | Single Medium-Frequency Term Query|
$f(x) = x$| 355.48| **485.73**|
**+36.6%** | -69.4% |
| **`DecInv_OrHighHigh`** | Boolean OR Query (`term1 OR term2`) |
$g(x) = 100k / x$ | 52.03 | **69.62** |
**+33.8%** | -19.1% |
| **`Inc_LowTerm`** | Single Low-Frequency Term Query |
$f(x) = x$| 541.12| **722.21**|
**+33.5%** | **-40.6%** |
| **`Inc_OrHighHigh`**| Boolean OR Query (`term1 OR term2`) |
$f(x) = x$| 42.70 | **55.23** |
**+29.3%** | -18.9% |
| **`TwoFields_AndHighHigh`** | Boolean AND Query (`term1 AND term2`) |
Multi-Field Sum | 57.95 | **72.49** |
**+25.1%** | **-39.0%** |
| **`DecInv_MedTerm`**| Single Medium-Frequency Term Query|
$g(x) = 100k / x$ | 277.27| **344.99**|
**+24.4%** | +1.3% |
| **`DecNeg_OrHighHigh`** | Boolean OR Query (`term1 OR term2`) |
$g(x) = -x$ | 48.72 | **60.49** |
**+24.2%** | -12.6% |
| **`Inc_HighTerm`** | Single High-Frequency Term Query |
$f(x) = x$| 100.48| **124.42**|
**+23.8%** | +28.5% |
| **`DecNeg_AndHighHigh`**| Boolean AND Query (`term1 AND term2`) |
$g(x) = -x$ | 52.39 | **64.34** |
**+22.8%** | **-59.6%** |
| **`TwoFields_HighTerm`**| Single High-Frequency Term Query |
Multi-Field Sum | 125.21| **153.51**|
**+22.6%** | +22.5% |
| **`DecInv_HighTerm`** | Single High-Frequency Term Query |
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676616933
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
I think it would still work if it is constant. In that case it would see the
first doc with the constant value. After that it asks to skip, but provides the
constant value. The code then needs to revisit all docs, because the comparison
to min and max score needs to revisit all docs with exact same score. It can
only exclude docs with larger or less vale (that don't exist).
Maybe add a test. 😜
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676616933
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
I think it would still work if it is constant. In that case it would see the
first doc with the constant value. After that it asks to skip, but provides the
constant value. The code then needs to revisit all docs, because the comparison
to min and max score needs to revisit all docs with exact same score. It can
only exclude docs with larger or less value (that don't exist).
Maybe add a test. 😜
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676527683
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
In case of constant scoring, my view-point is a little different.
Since all docs have the same score, we need to evaluate each and every doc.
We can't prune any block.
In that case, we might need to use, `Monotonicity.NONE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676527683
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
In case of constant scoring, my view-point is a little different.
Since all docs have the same score, we need to evaluate each and every doc.
We can't prune any block.
In that case, we need to use, `Monotonicity.NONE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676519624
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
So I think terms are correct. Constant function is allowed and can be
classified as both increasing or decreasing. It would also work fine in our
code - you can pass both enum constants, because in case of a constant function
the `getMinScore()` and `getMaxScore()` would both return the same value (the
constant), so the result for a block with constant function is same for both
increasing or decreasing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676527683
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
In case of constant scoring, my view-point is a little different.
Since all docs have the same score, we need to evaluate each and every doc.
We can't prune any block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676519624
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
So I think terms are correct. And constant is allowed and can be both. It
would also work fine, because in case of a constant function the
`getMinScore()` and `getMaxScore()` would both return the same value, so the
result for a block with constant function is same for both increasing or
decreasing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676501962
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
Wikipedia says:
> https://github.com/user-attachments/assets/badd7728-fe7d-4fe4-a098-34687263958d";
/>
Because of the `<=` in that definitiion "monitonically increasing means"
that the function result may be constant.
With strictly increasing the definition is:
> https://github.com/user-attachments/assets/7d6f5de9-afae-4878-b5c6-0db2d5adb63e";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676475280
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
I think in those cases where we are uncertain about the monotonicity of a
function, we can use `Monotonicity.NONE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676435977
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -278,28 +289,48 @@ public static DoubleValues similarityToQueryVector(
}
/**
- * Creates a DoubleValuesSource that wraps a generic NumericDocValues field
+ * Creates a DoubleValuesSource that wraps a generic NumericDocValues field.
+ * Assumes no monotonic direction by default (Monotonicity.NONE).
*
* @param field the field to wrap, must have NumericDocValues
* @param decoder a function to convert the long-valued doc values to doubles
*/
public static DoubleValuesSource fromField(String field,
LongToDoubleFunction decoder) {
-return new FieldValuesSource(field, decoder);
+return fromField(field, decoder, Monotonicity.NONE);
+ }
+
+ /**
+ * Creates a DoubleValuesSource that wraps a generic NumericDocValues field
+ *
+ * @param field the field to wrap, must have NumericDocValues
+ * @param decoder a function to convert the long-valued doc values to doubles
+ * @param monotonicity the monotonicity direction of the decoder function
+ */
+ public static DoubleValuesSource fromField(String field,
LongToDoubleFunction decoder, Monotonicity monotonicity) {
+return new FieldValuesSource(field, decoder, monotonicity);
+ }
+
+ /**
+ * @deprecated Use {@link #fromField(String, LongToDoubleFunction,
Monotonicity)}
+ */
+ @Deprecated
Review Comment:
Removed. Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676434717
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
I don't like to have "NON" in the name, this makes it hard to use.
INCREASING should be fine, because unless it is strictly increasing it means
that the values go up. A constant function is fine, you should be able to
chooseany of INCREASING or DECREASING, correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
msokolov commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676398949
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
Also, if the function is a composite function, like `sum`, it is
non-increasing when both of its inputs are, but non-decreasing when both of its
inputs are, but when one is decreasing and the other one increasing, it is
undefined. I don't know how we can really expect this to work with general
functions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
msokolov commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676382886
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -46,6 +47,16 @@
*/
public abstract class DoubleValuesSource implements SegmentCacheable {
+ /** Monotonicity direction of a decoder function */
+ public enum Monotonicity {
+/** Increasing function */
+INCREASING,
Review Comment:
I think we do not need strictly increasing, rather `NONDECREASING` and
`NONINCREASING`, right? what if the function is constant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3676035239
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -278,28 +289,48 @@ public static DoubleValues similarityToQueryVector(
}
/**
- * Creates a DoubleValuesSource that wraps a generic NumericDocValues field
+ * Creates a DoubleValuesSource that wraps a generic NumericDocValues field.
+ * Assumes no monotonic direction by default (Monotonicity.NONE).
*
* @param field the field to wrap, must have NumericDocValues
* @param decoder a function to convert the long-valued doc values to doubles
*/
public static DoubleValuesSource fromField(String field,
LongToDoubleFunction decoder) {
-return new FieldValuesSource(field, decoder);
+return fromField(field, decoder, Monotonicity.NONE);
+ }
+
+ /**
+ * Creates a DoubleValuesSource that wraps a generic NumericDocValues field
+ *
+ * @param field the field to wrap, must have NumericDocValues
+ * @param decoder a function to convert the long-valued doc values to doubles
+ * @param monotonicity the monotonicity direction of the decoder function
+ */
+ public static DoubleValuesSource fromField(String field,
LongToDoubleFunction decoder, Monotonicity monotonicity) {
+return new FieldValuesSource(field, decoder, monotonicity);
+ }
+
+ /**
+ * @deprecated Use {@link #fromField(String, LongToDoubleFunction,
Monotonicity)}
+ */
+ @Deprecated
Review Comment:
This method is new in this pr, no need to add a new deprecated method.
Remove it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5120135501 I guess you are correct, when monotonicity is unknown or un-supplied, we need to assume score to be Float.POSITIVE_INFINITY and go over each Doc in the block. Let me change my code. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431:
URL: https://github.com/apache/lucene/pull/16431#issuecomment-5119983060
I'd suggest to replace the boolean with an enum: `public enum Monotonicity {
INCREAING, DECREASING, UNDEFINED }` (in the matimatical sense it need to be
strictly monotonic, correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431:
URL: https://github.com/apache/lucene/pull/16431#issuecomment-5119902219
I think there's one problem, in your example with a "simple function" inline
(lambda):
```java
DoubleValuesSource valSource = DoubleValuesSource.fromField("val", v ->
1.0 / v, false);
```
The `fromField` function requires `true` or `false`. If you now use `v ->
sin(v)` as lambda, you need to pass somehow "undefined" here. So basically the
thirs parameter needs to be a three-state! E.g., you need to be able to pass
true/false or null.
And on top: the version without the boolean by default assumes no direction
and we undeprecate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on PR #16431:
URL: https://github.com/apache/lucene/pull/16431#issuecomment-5119357376
> Can we have an example and a test for a "homemade" function that followins
decreasing or increasing logic? I want this to be defined together with the
function!
I have attached a test specifying `DoubleValuesSource` where scoring is done
using `Math.sqrt()` which is a monotonically increasing function.
> The caller who constructs a FunctionQuery should not need to kown if the
function heshe uses is increasing!
The caller constructing a `FunctionScoreQuery` **does not need to know or
specify whether the function is increasing or decreasing**:
```java
// Clean, transparent caller API (no boolean flags needed):
Query query = new FunctionScoreQuery(baseQuery, doubleValuesSource);
```
> What happens if you use a ValueSource with an increasing function and you
wrap it with another decreasing function (nesting)?
When nesting functions $f(g(x))$, monotonicity composes deterministically:
| Outer Function | Inner Function | Composed
Result | Outer `getMaxScore()` Delegates To |
| :| :|
:| :---|
| **Increasing** ($\sqrt{u}$) | **Increasing** ($x$)| **Increasing**
| `inner.getMaxScore(upTo)` |
| **Increasing** ($\sqrt{u}$) | **Decreasing** ($-x$) | **Decreasing**
| `inner.getMaxScore(upTo)` (delegated down) |
| **Decreasing** ($-u$) | **Increasing** ($\sqrt{x}$) | **Decreasing**
| `inner.getMinScore(upTo)` |
| **Decreasing** ($-u$) | **Decreasing** ($-x$) | **Increasing**
| `inner.getMinScore(upTo)` |
If any layer in the nesting chain is non-monotonic or lacks skip index
information, it returns `Float.POSITIVE_INFINITY`, making nested expressions
safe by construction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5118192316 > I don't see how you can " guarantee[s] zero correctness bugs " - sounds like a statement from an overambitious AI! > > In particular, if someone implements a scoring function like `sin()` and provides its maximum (== 1); won't that cause scoring bugs? I understood it like this: The default is to use INFINITY when returning if it is maxScore ready. If you implement a sinus function and implement maxScore for it, it would break. But then it's your own problem, because you implemented maxScore support for it. What I did not yet undertsood: Where are the defaults (increasing/decreasing) as in the statement above for `-x`, `x`, `sqrt(x)`? Can we have an example and a test for a "homemade" function that followins decreasing or increasing logic? I want this to be defined together with the function! The caller who constructs a FunctionQuery should not need to kown if the function heshe uses is increasing! What happens if you use a ValueSource with an increasing function and you wrap it with another decreasing function (nesting)? There should be clear paths so it is also safe for the user. Otherwise I would be against such an optimization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5118188025 @msokolov For a non-monotonic function like `sin()`, monotonic flag is NOT set, `DoubleValuesSource.FieldValuesSource.getMaxScore(upTo)` returns `Float.POSITIVE_INFINITY`. Returning `Float.POSITIVE_INFINITY` prevents WAND from skipping any candidate blocks, forcing standard exact evaluation on all documents—**guaranteeing zero correctness regressions for non-monotonic functions.** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
msokolov commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5118008750 I don't see how you can " guarantee[s] zero correctness bugs " - sounds like a statement from an overambitious AI! In particular, if someone implements a scoring function like `sin()` and provides its maximum (== 1); won't that cause scoring bugs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3672129992
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -284,7 +285,18 @@ public static DoubleValues similarityToQueryVector(
* @param decoder a function to convert the long-valued doc values to doubles
*/
public static DoubleValuesSource fromField(String field,
LongToDoubleFunction decoder) {
Review Comment:
Added `@deprecated` in function comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3672058926
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -284,7 +285,18 @@ public static DoubleValues similarityToQueryVector(
* @param decoder a function to convert the long-valued doc values to doubles
*/
public static DoubleValuesSource fromField(String field,
LongToDoubleFunction decoder) {
Review Comment:
Maybe deprecate this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on code in PR #16431:
URL: https://github.com/apache/lucene/pull/16431#discussion_r3672055206
##
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##
@@ -284,7 +285,18 @@ public static DoubleValues similarityToQueryVector(
* @param decoder a function to convert the long-valued doc values to doubles
*/
public static DoubleValuesSource fromField(String field,
LongToDoubleFunction decoder) {
Review Comment:
We should add a note here that the function needs to be increasing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
rajat315315 commented on PR #16431:
URL: https://github.com/apache/lucene/pull/16431#issuecomment-5114543243
Thanks for raising this crucial point regarding monotonicity!
I have updated the PR to handle both **monotonically increasing** and
**monotonically decreasing** functions while maintaining strict safety for
arbitrary expressions:
### 1. 100% Sound & Safe Fallback (`Float.POSITIVE_INFINITY`) for Arbitrary
Expressions
In `DoubleValues.java`, the default interface implementation of
`getMaxScore` is conservative:
```java
public float getMaxScore(int upTo) throws IOException {
return Float.POSITIVE_INFINITY;
}
```
Any `DoubleValuesSource` (such as `ExpressionValueSource`, script functions,
or un-analyzed custom functions) that does **not** override `getMaxScore` will
automatically return `Float.POSITIVE_INFINITY`.
In `FunctionScoreQuery`:
```java
float valueMaxScore = scores.getMaxScore(upTo);
if (Float.isInfinite(valueMaxScore)) {
return Float.POSITIVE_INFINITY;
}
```
When `valueMaxScore` is `Float.POSITIVE_INFINITY`, Block-Max WAND dynamic
block pruning is safely bypassed. This guarantees **zero correctness bugs** for
non-monotonic or arbitrary expression sources.
---
### 2. Support for Both Increasing AND Decreasing Monotonic Functions
Because `DocValuesSkipper` provides **both** `skipper.minValue(0)` and
`skipper.maxValue(0)` for every segment block, I added an `increasing` boolean
parameter to `DoubleValuesSource.fromField()`:
```java
DoubleValuesSource.fromField(String field, LongToDoubleFunction decoder,
boolean increasing)
```
* **Increasing Functions** (`increasing = true`, default for $x, \log(x),
\sqrt{x}$):
$$\text{BlockMaxScore} =
\text{decoder.applyAsDouble}(\text{skipper.maxValue}(0))$$
* **Decreasing Functions** (`increasing = false`, for $-x$, $1/x$,
reciprocal decay, distance decay $e^{-\lambda \cdot x}$):
$$\text{BlockMaxScore} =
\text{decoder.applyAsDouble}(\text{skipper.minValue}(0))$$
For a monotonically decreasing function, the maximum output score occurs at
the **smallest** raw field value (`skipper.minValue(0)`), giving exact and
sound block upper bounds for WAND pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] Enable Block-Max WAND dynamic pruning for FunctionScoreQuery via DocValuesSkipper (+422% QPS) [lucene]
uschindler commented on PR #16431: URL: https://github.com/apache/lucene/pull/16431#issuecomment-5114153921 Hi, there is one big problem, otherwise we would have enabled this for function queries already!: It won't work correctly if the function modifies the score in a way that it is not an increasing function, e.g, a function like `1/x` or `-x`. How can we differentiate between that? Or is it safe for this special case only (a pure DocValuesValueSource as function), and if you use another valuesource (e.g., from expressions) that does not implement the maxScore getter, it won't be optimized? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
