meticulous-dft opened a new pull request, #3170:
URL: https://github.com/apache/jackrabbit-oak/pull/3170
## Summary
This PR replaces #3158, #3159 and #3160 with one change set, so the next AEM
test build for the POC needs a single deployment. Adobe's first benchmark ran
from Azure West Europe against an Atlas cluster in Azure `eastus2`, about 90 ms
per round trip. At that distance, the number of sequential round trips per
query dominates latency. mongot itself served the benchmark's search commands
in 30 to 40 ms.
| Commit | Author | Was | Change |
|---|---|---|---|
| Let the connector bundle resolve in OSGi | Ming He | #3158 | Embed only
`oak-search`, and deploy the MongoDB driver bundles in the OSGi test runtime,
so `OSGiIT.bundleStates` passes |
| Drop per-query search index lookups and redundant sorts | Ming He | #3159
| Check the search index only when a `$search` returns no hits; count without
the sort and projection; no sort for `jcr:score` descending |
| Count matches inside mongot through the search metadata (2 commits) | Oren
Ovadia | #3160 | For a bare `[$search, $project]` pipeline, take the exact
count from `$searchMeta` with `count: {type: "total"}` instead of a `$count`
pipeline |
| Cache the index statistics Oak reads while planning | Ming He | new |
Serve `numDocs()` and `getDocCountFor()` from a cache, as the Elasticsearch
index does |
The statistics cache addresses the largest remaining cost. Oak's planner
reads `numDocs()` and `getDocCountFor(property)` for every query, and each read
was a `countDocuments` against the index collection: 2 to 3 extra round trips
before the query even started. The counts now come from a cache with a
60-second background refresh and a 10-minute expiry
(`oak.mongot.statsRefreshSeconds`, `oak.mongot.statsExpireSeconds`), the same
defaults as the Elasticsearch index. The cache belongs to the index node, and
the tracker replaces that node whenever the index status or definition changes,
so each indexing cycle that writes to the index starts from fresh counts.
**Round trips per query.** These are aggregate plus `getMore` commands the
server received, for six AEM-shaped query shapes over 2,004 assets. Each cell
shows reading all rows, then in brackets reading all rows plus the exact size.
| Query | Branch tip (#3143) | This PR |
|---|---|---|
| property-exact (1 hit) | 4 (6) | 1 (3) |
| prefix | 5 (7) | 2 (4) |
| sorted-by-date | 6 (11) | 3 (5) |
| tag-filtered | 6 (8) | 3 (5) |
| format-facet | 6 (10) | 3 (6) |
| fulltext-phrase | 6 (10) | 3 (5) |
What remains is the query, one `getMore` per 1,000 results and, for an exact
size, the count query. The other extra round trip comes from Oak itself:
calling `getSize()` and then iterating runs the query twice.
**Notes on the combined count path**
- `$searchMeta` applies only when nothing follows the search. Queries with a
path, type or property `$match` after the search, which includes every AEM
benchmark shape today, keep the `$count` fallback until filters move into the
search stage.
- `count: {type: "total"}` is exact; the threshold applies only to
`lowerBound`.
- Against a missing search index, `$searchMeta` returns a count of 0 rather
than failing. The main query runs first and fails on a missing index, so a size
request never reports 0 for one.
**Reviewer focus**
- `MongotIndex.aggregateCursor`: the search index check now runs only when a
`$search` has no first hit, or while retrying an index that is not yet
queryable.
- `MongotIndex.countMatches`: the metadata count comes first, then the
leaner `$count` fallback.
- `MongotIndexStatistics`: the cache keys (the whole collection, or the
encoded property field) and its refresh and expiry settings.
- `oak-search-mongot/pom.xml` and `oak-it-osgi/test-bundles.xml`: the driver
is no longer embedded, and a runtime must provide MongoDB driver 5.x bundles,
5.4 or later.
## Testing
- Added `MongotIndexStatisticsTest`, which counts the server's `$group`
stages (`countDocuments`) to show that repeated planning reads don't reach
MongoDB; it fails against the uncached statistics.
- Kept the tests from #3159 and #3160, which count the stages the server
runs. The #3159 tests were checked to fail against the code they replace; for
Oren's `$searchMeta` test we rely on his report in #3160. The dropped-index
test guards behavior that must stay the same.
- Manually ran `oak-it-osgi`'s integration tests, and all 14 pass, including
`OSGiIT.bundleStates`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
