rahulgoswami opened a new pull request, #5356: URL: https://github.com/apache/solr/pull/5356
https://issues.apache.org/jira/browse/SOLR-18664 # Description `{!graph}` does a breadth-first traversal: at each hop it reads the `to` field of every collected document and adds those values to the next hop's `from IN (...)` frontier query. There is no way to collect a node without also expanding it. In a file/folder hierarchy (`{!graph from=ParentGUID to=contentid traversalFilter=IsVisible:true}`), only folders can have children, but every file's `contentid` still becomes a frontier term. Each one costs a docValues read, a hash insert and a terms-dictionary seek per segment, and can never match. `traversalFilter=DocumentType:4` would drop the files from the results. The existing leaf-node optimization doesn't help either, because it only skips documents with no value in `to`, and here every document has one. # Solution A new optional `lookupFilter` local param. Documents reached by the traversal (and matching `traversalFilter`) are always collected, but only those that also match `lookupFilter` have their edges followed. It applies at every level, including the root. Without the param, behaviour is unchanged. ``` {!graph from=ParentGUID to=contentid traversalFilter=IsVisible:true lookupFilter=DocumentType:4}<root query> ``` - `GraphQuery` gets the filter's DocSet once per request (via the filterCache) and passes it to the edge collectors. `GraphEdgeCollector.collect()` checks it only before extracting edges; collection itself is unconditional. - `lookupFilter` is part of `GraphQuery`'s `equals`/`hashCode`/`toString`, so filterCache and queryResultCache keys stay correct. - It is kept separate from `leafNodes`, which also drives `returnOnlyLeaf`. - The JSON Facet `graph` domain goes through `GraphQueryParser`, so it supports the param too. - `GraphPointsCollector`'s public 3-arg constructor is kept. - Ref guide: new `lookupFilter` entry in the graph query parser parameters. Related: SOLR-16202 asks for a filter on which documents are *returned*; this controls which are *expanded*. This change was developed with Claude Code (analysis, implementation, tests and benchmark), reviewed by me. ## Benchmark Synthetic 1M-doc file-system index: 80k folders, 920k files (~11.5 files per folder), depth up to 15. JDK 21, 4 GB heap, standalone, `rows=0`, queryResultCache disabled so every query does the full traversal. Each session ran in a fresh JVM with 10 warmup + 30 measured iterations per root and arm, interleaved. Session order: patched, baseline (unpatched build of the same commit), patched. Median QTime in ms, geometric mean over the roots in each bucket: | Visible subtree size | Unpatched | Patched, no `lookupFilter` | Patched + `lookupFilter=DocumentType:4` | Speedup (same build) | |---|---|---|---|---| | ~100 | 3.0 | 2.6 | 1.3 | 1.4× | | ~1k | 8.6 | 7.6 | 2.3 | 2.4× | | ~10k | 53 | 53 | 7.8 | 6.0× | | ~50k | 178 | 174 | 24 | 7.0× | | ~100k | 363 | 349 | 42 | 8.0× | | ~250k | 772 | 656 | 73 | 8.8× | | whole tree (829k) | 3,077 | 3,038 | 298 | 10.1× | - **Correctness:** all 5,760 queries returned the `numFound` computed independently from the generated tree, and all variants agree. - **Why:** for the whole tree the frontier queries carry 12× fewer terms in total (828,781 → 68,600). What remains is collecting the documents themselves. - **No regression without the param:** unpatched vs. patched ranged from 0.98× to 1.18× by bucket. An index-time control that runs identical code on both builds showed the same per-bucket differences, so these come from the baseline JVM session, not the change. - **Index-time alternative:** indexing a folders-only id field and using it as `to` performs about the same as `lookupFilter`, and 3–6% faster on the three largest subtrees, but needs a schema change and a reindex. - **Caveats:** one laptop with client and server sharing CPUs, a hot (page-cached) index, synthetic data, string ids only (`GraphPointsCollector` has the same gating but wasn't benchmarked). The gain scales with the ratio of non-expandable to expandable nodes. # Tests - `GraphQueryTest.doGraph()` (runs across the existing string, docValues-only string and int/long/float/double point field matrix, so both collectors are covered): - a node with children that doesn't match `lookupFilter` is returned but not expanded; - a pure-negative `lookupFilter`; - a root that doesn't match only returns the root; - combined with `traversalFilter`; - combined with `maxDepth` and `returnRoot=false`. - `QueryEqualityTest.testGraphQuery`: param substitution, and queries with vs. without `lookupFilter` are unequal (cache-key safety). - Ran `./gradlew tidy`, `./gradlew check -x test`, `TestJsonFacets` and all `org.apache.solr.search.join.*` tests. # Checklist - [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [x] I have created a Jira issue and added the issue ID to my pull request title. - [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation) - [x] I have developed this patch against the `main` branch. - [x] I have run `./gradlew check`. - [x] I have added tests for my changes. - [x] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) - [x] I have added a [changelog entry](https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc) for my change 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
