Re: Example / Demo re support for filtering in nearest-neighbor vector search (Lucene 9.1.0)

[Michael Wechner]

Hi Julie

I got it running and it seems to work fine so far :-)

Re an example for the demo package, I guess this would go here

https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html

and I thought of something like "KnnVectorPreFilterQuery.java"

Query preFilterQuery =new TermQuery(new Term(TOPIC_FIELD,"general"));
if (filter !=null) {
    log.info("Filter applied before the vector search: " + preFilterQuery);
}
Query query =new KnnVectorQuery(VECTOR_FIELD, queryVector, k, preFilterQuery);

TopDocs topDocs = searcher.search(query, k);

Does that make sense to you?

Thanks

Michael

Am 11.05.22 um 07:59 schrieb Michael Wechner:

Hi Julie

Cool, thanks!

I try to apply it and if it works could create an example to the demo package.

Will keep you posted :-)

Thanks

Michael

Am 11.05.22 um 02:13 schrieb Julie Tibshirani:

Hello Michael, we don't yet have an example of kNN with filtering in the
demo package. This would be nice to add!

For now maybe looking at the unit tests could give a sense of how to use
it. Here's an example:

https://github.com/apache/lucene/blob/main/lucene/core/src/test/org/apache/lucene/search/TestKnnVectorQuery.java#L115-L127. The idea is that KnnVectorQuery optionally accepts a Query as a filter, and returns the k nearest vectors that also match the filter. Many people refer

to this as "kNN with prefiltering" (as opposed to "postfiltering", where

the filter is applied *after* the kNN search, so in the end you may receive

fewer than k matches).

Let us know if you run into any questions/ issues while trying it out!

Julie

On Mon, May 9, 2022 at 8:08 AM Michael Wechner <michael.wech...@wyona.com>

wrote:

sorry for the URLs below.

I have tested Twilio SendGrid as outgoing server and it just rewrote the

URLs


https://issues.apache.org/jira/browse/SOLR-15947
https://issues.apache.org/jira/browse/LUCENE-10382

and

https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html

which I was not aware of, but disabled the tracking now and hope it will

be ok now.

Thanks

Michael

Am 09.05.22 um 15:12 schrieb Michael Wechner:

Hi

I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
vector search, which is great :-)

I have found

http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D

http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D

and

http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D

but it is not really clear to me how to use it.

Is there some additional documentation / example / demo how I can
combine filtering with vector search?

Thanks

Michael



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org