kerosin75 commented on issue #16263: URL: https://github.com/apache/lucene/issues/16263#issuecomment-4788049411
Not necessarily. My observation is not that short passages are generally less relevant and should always be ranked lower. There are certainly cases where a short chunk is the best possible answer and deserves the top position. The issue I am describing is different: when multiple passages express essentially the same concept, very short passages often receive significantly higher vector similarity scores than longer passages containing the same information plus additional useful context. For example, if a query is about concept X, a chunk containing only "X" may outrank a chunk explaining "X" in detail, even though many users would consider the latter more useful. Our length normalization does not aim to suppress short passages. It merely reduces the systematic advantage that very short passages appear to receive in some embedding models. In practice, genuinely relevant short passages still rank highly, while longer passages with comparable semantic similarity are no longer pushed unnecessarily far down the result list. I fully agree that this behavior originates in the embedding model rather than in Lucene. My point is simply that the effect becomes visible at retrieval time and can have a measurable impact on ranking quality in real-world retrieval systems. Interestingly, this is not entirely foreign to information retrieval. Traditional ranking functions such as BM25 also apply document-length normalization because raw relevance signals are known to be influenced by document length. Nobody would interpret that as an attempt to "bury" long or short documents; it is simply a correction for a measurable statistical effect. I see the situation here in a similar way. If many embedding models systematically favor very short passages when semantically equivalent longer passages exist, then a mild length-aware adjustment can be viewed as a correction of that bias rather than an attempt to redefine relevance. The goal is not to make longer passages win, but to make passage length less influential when semantic similarity is otherwise comparable. Of course, I understand the concern that applying such adjustments purely at retrieval time may not be compatible with the assumptions made by the ANN index. My main point is therefore not a specific implementation proposal, but the observation that passage length appears to influence retrieval quality in practice, and that modest normalization can significantly improve ranking quality in many production workloads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
