rahulgoswami opened a new pull request, #5356:
URL: https://github.com/apache/solr/pull/5356

   https://issues.apache.org/jira/browse/SOLR-18664
   
   # Description
   
   `{!graph}` does a breadth-first traversal: at each hop it reads the `to` 
field of every collected document and adds those values to the next hop's `from 
IN (...)` frontier query. There is no way to collect a node without also 
expanding it.
   
   In a file/folder hierarchy (`{!graph from=ParentGUID to=contentid 
traversalFilter=IsVisible:true}`), only folders can have children, but every 
file's `contentid` still becomes a frontier term. Each one costs a docValues 
read, a hash insert and a terms-dictionary seek per segment, and can never 
match. `traversalFilter=DocumentType:4` would drop the files from the results. 
The existing leaf-node optimization doesn't help either, because it only skips 
documents with no value in `to`, and here every document has one.
   
   # Solution
   
   A new optional `lookupFilter` local param. Documents reached by the 
traversal (and matching `traversalFilter`) are always collected, but only those 
that also match `lookupFilter` have their edges followed. It applies at every 
level, including the root. Without the param, behaviour is unchanged.
   
   ```
   {!graph from=ParentGUID to=contentid traversalFilter=IsVisible:true 
lookupFilter=DocumentType:4}<root query>
   ```
   
   - `GraphQuery` gets the filter's DocSet once per request (via the 
filterCache) and passes it to the edge collectors. 
`GraphEdgeCollector.collect()` checks it only before extracting edges; 
collection itself is unconditional.
   - `lookupFilter` is part of `GraphQuery`'s `equals`/`hashCode`/`toString`, 
so filterCache and queryResultCache keys stay correct.
   - It is kept separate from `leafNodes`, which also drives `returnOnlyLeaf`.
   - The JSON Facet `graph` domain goes through `GraphQueryParser`, so it 
supports the param too.
   - `GraphPointsCollector`'s public 3-arg constructor is kept.
   - Ref guide: new `lookupFilter` entry in the graph query parser parameters.
   
   Related: SOLR-16202 asks for a filter on which documents are *returned*; 
this controls which are *expanded*.
   
   This change was developed with Claude Code (analysis, implementation, tests 
and benchmark), reviewed by me.
   
   ## Benchmark
   
   Synthetic 1M-doc file-system index: 80k folders, 920k files (~11.5 files per 
folder), depth up to 15. JDK 21, 4 GB heap, standalone, `rows=0`, 
queryResultCache disabled so every query does the full traversal. Each session 
ran in a fresh JVM with 10 warmup + 30 measured iterations per root and arm, 
interleaved. Session order: patched, baseline (unpatched build of the same 
commit), patched.
   
   Median QTime in ms, geometric mean over the roots in each bucket:
   
   | Visible subtree size | Unpatched | Patched, no `lookupFilter` | Patched + 
`lookupFilter=DocumentType:4` | Speedup (same build) |
   |---|---|---|---|---|
   | ~100 | 3.0 | 2.6 | 1.3 | 1.4× |
   | ~1k | 8.6 | 7.6 | 2.3 | 2.4× |
   | ~10k | 53 | 53 | 7.8 | 6.0× |
   | ~50k | 178 | 174 | 24 | 7.0× |
   | ~100k | 363 | 349 | 42 | 8.0× |
   | ~250k | 772 | 656 | 73 | 8.8× |
   | whole tree (829k) | 3,077 | 3,038 | 298 | 10.1× |
   
   - **Correctness:** all 5,760 queries returned the `numFound` computed 
independently from the generated tree, and all variants agree.
   - **Why:** for the whole tree the frontier queries carry 12× fewer terms in 
total (828,781 → 68,600). What remains is collecting the documents themselves.
   - **No regression without the param:** unpatched vs. patched ranged from 
0.98× to 1.18× by bucket. An index-time control that runs identical code on 
both builds showed the same per-bucket differences, so these come from the 
baseline JVM session, not the change.
   - **Index-time alternative:** indexing a folders-only id field and using it 
as `to` performs about the same as `lookupFilter`, and 3–6% faster on the three 
largest subtrees, but needs a schema change and a reindex.
   - **Caveats:** one laptop with client and server sharing CPUs, a hot 
(page-cached) index, synthetic data, string ids only (`GraphPointsCollector` 
has the same gating but wasn't benchmarked). The gain scales with the ratio of 
non-expandable to expandable nodes.
   
   # Tests
   
   - `GraphQueryTest.doGraph()` (runs across the existing string, 
docValues-only string and int/long/float/double point field matrix, so both 
collectors are covered):
     - a node with children that doesn't match `lookupFilter` is returned but 
not expanded;
     - a pure-negative `lookupFilter`;
     - a root that doesn't match only returns the root;
     - combined with `traversalFilter`;
     - combined with `maxDepth` and `returnRoot=false`.
   - `QueryEqualityTest.testGraphQuery`: param substitution, and queries with 
vs. without `lookupFilter` are unequal (cache-key safety).
   - Ran `./gradlew tidy`, `./gradlew check -x test`, `TestJsonFacets` and all 
`org.apache.solr.search.join.*` tests.
   
   # Checklist
   
   - [x] I have reviewed the guidelines for [How to 
Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my 
code conforms to the standards described there to the best of my ability.
   - [x] I have created a Jira issue and added the issue ID to my pull request 
title.
   - [x] I have given Solr maintainers 
[access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
 to contribute to my PR branch. (optional but recommended, not available for 
branches on forks living under an organisation)
   - [x] I have developed this patch against the `main` branch.
   - [x] I have run `./gradlew check`.
   - [x] I have added tests for my changes.
   - [x] I have added documentation for the [Reference 
Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide)
   - [x] I have added a [changelog 
entry](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc) for my 
change
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] SOLR-18664: Add lookupFil... rahulgoswami (via GitHub) via issues

Reply via email to