krickert commented on PR #16357: URL: https://github.com/apache/lucene/pull/16357#issuecomment-4898725031
**Cross-shard optimistic kNN - Phase 1 measurements @ _k_=10000** | Distribution | Arm | Visits/q | Δ vs full-k | Recall | Δ recall | | ------------ | --------------------------------- | --------- | ----------- | ---------- | ---------- | | Uniform | full-k (baseline) | 1,036,532 | - | 0.9908 | - | | Uniform | static quota (pS=3000) | 347,701 | −66.5% | 0.9865 | −0.004 | | Uniform | adaptive floor (gate=3000, g=0.9) | 96,721 | −90.7% | 0.9530 | −0.038 | | Uniform | adaptive floor (gate=6000, g=0.9) | 125,609 | −87.9% | 0.9654 | −0.025 | | Skewed | full-k (baseline) | 1,503,321 | - | 0.9835 | - | | Skewed | static quota (pS=1012, blind) | 224,643 | −85.1% | **0.2383** | **−0.745** | | Skewed | adaptive floor (gate=1012, g=0.9) | 136,762 | −90.9% | 0.9626 | −0.021 | | Skewed | adaptive floor (gate=1012, g=1.0) | 135,592 | −91.0% | 0.9625 | −0.021 | *Summary: on uniform data the floor is cheaper than the static quota but doesn't match its recall - an open, non-dominant tradeoff. On skewed data the static quota's savings come from missing 3 in 4 true results; the floor is the only arm that is both cheap and correct.* **Progress on the recall/visits tradeoff raised:** Rebuilt the mechanism as a sandbox-only module (no Lucene core changes): `GlobalKnnFloor` / `FloorAwareKnnCollector` / `SharedFloorKnnCollectorManager` in `org.apache.lucene.sandbox.search.knn`, branch [`sandbox/shared-floor-knn`](https://github.com/ai-pipestream/lucene/tree/sandbox/shared-floor-knn). To measure it accurately, a distributed lucene setup in one JVM was set up where every shard returns full `k` and the coordinator merges. To run this, I built `Tier2Bench` - S disjoint single-segment shard indexes searched concurrently in one JVM, sharing one floor, recall scored against exact global ground truth. Pushed to my luceneutil fork: [`feature/shared-floor-harness`, `src/main/knn/tier2/`](https://github.com/ai-pipestream/luceneutil/tree/feature/shared-floor-harness/src/main/knn/tier2) (README has full setup + these tables). **Setup**: ~962 GB for the full 247M-vector index used in T1 (16 shards × ~61GB each; the joined single-index copy is a separate 962GB copy of the same data - 1.88TB combined on disk if you count both). The shards are set up `maxConn=16 beamWidthIndex=100 dot_product`, force-merged single-segment. Uniform = the real 247M-vector Cohere corpus's original 16-shard split (~15.4M docs/shard, ground truth = exact top-10000 over the full 247M). Skewed = a 10M-vector subset partitioned into 16 k-means clusters (size-cv 0.43, query-affinity 0.78 vs. 0.06 uniform - real, strong skew). **Not yet run**: shard-count scaling (S=4/8/16) and the k=100/1000 rows - full sweep plan is in the same folder ([`DISTRIBUTED_FLOOR_TEST_PLAN.md`](https://github.com/ai-pipestream/luceneutil/blob/feature/shared-floor-harness/src/main/knn/tier2/DISTRIBUTED_FLOOR_TEST_PLAN.md)). This is Phase 1 and after a couple of days of putting this data together, I really needed a spot check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
