raghavyadav01 opened a new pull request, #20075: URL: https://github.com/apache/pinot/pull/20075
Adds relevance-ranked text search over the Lucene text index, so a query can rank rows by BM25 instead of only filtering with `TEXT_MATCH`. **`TEXT_SCORE(column, 'query'[, 'options'[, 'stats']])`** is a transform that returns each row's Lucene relevance score (BM25 by default) for a text query, and 0 for rows that don't match. It is implemented for single-column, multi-column and realtime (consuming) text indexes; a column without a text index is rejected. ```sql SELECT id, TEXT_SCORE(title, 'wireless headphones') AS score FROM products WHERE TEXT_MATCH(title, 'wireless headphones') ORDER BY score DESC LIMIT 10 ``` **`TEXT_SCORE_STATS(column, 'query')`** is an aggregation that returns corpus-wide BM25 statistics for the query terms: document count, average field length and per-term document frequency, summed across segments and servers. It ignores the query's filter and is used on its own, not combined with other aggregations. Each segment scores with its own statistics, so scores from different segments are not comparable. Passing the `TEXT_SCORE_STATS` result as the 4th argument of `TEXT_SCORE` makes every segment score with the global IDF and average length. On a product-search relevance benchmark this raised nDCG@10 from 0.271 to 0.312. **Hybrid search.** `HybridSearchTest` fuses `TEXT_SCORE` ranks with vector ranks by reciprocal rank fusion in SQL, as an end-to-end example. **SPI.** `TextIndexReader` and `MultiColumnTextIndexReader` get default `getDocScores(...)` and `getTextScoreStats(...)` methods, so existing readers are unaffected. **Tests** - `TextScoreQueriesTest`: scores against the BM25 formula, term frequency and length effects, ordering, agreement with `TEXT_MATCH`, rejection without a text index. - `TextScoreGlobalStatsQueriesTest`: statistics summed across segments and servers, identical documents scoring identically with global statistics, parser options, filter independence, empty-statistics fallback. - `TextScoreStatsTest` covers merging and serializing the statistics. - `LuceneMutableTextIndexTest` covers realtime scoring. - `HybridSearchTest` is the integration test (8 tests). - The existing Lucene text index tests pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
