Re: Example / Demo re support for filtering in nearest-neighbor vector search (Lucene 9.1.0)

[Michael Wechner]

yes, sure, will keep you posted :-)

Thanks

Michael

Am 24.05.22 um 02:10 schrieb Julie Tibshirani:

Michael -- I am not totally clear on what you have in mind. Would you be up
for opening a pull request in GitHub (or JIRA issue) to show the idea, and
we can discuss the details there?

Matt -- thanks for sharing!

Julie

On Sun, May 22, 2022 at 3:29 PM Matt Davis <kryptonics...@gmail.com> wrote:

Thanks Julie.  I was able to implement vector search in Zulia with your
pointers.  The pull request might be helpful to others:
https://github.com/zuliaio/zuliasearch/pull/70

Thanks,
Matt

On Fri, May 20, 2022 at 9:23 AM Michael Wechner <michael.wech...@wyona.com
wrote:

Hi Julie

I got it running and it seems to work fine so far :-)

Re an example for the demo package, I guess this would go here

https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html

and I thought of something like "KnnVectorPreFilterQuery.java"

Query preFilterQuery =new TermQuery(new Term(TOPIC_FIELD,"general"));
if (filter !=null) {
      log.info("Filter applied before the vector search: " +
preFilterQuery);
}
Query query =new KnnVectorQuery(VECTOR_FIELD, queryVector, k,
preFilterQuery);

TopDocs topDocs = searcher.search(query, k);

Does that make sense to you?

Thanks

Michael

Am 11.05.22 um 07:59 schrieb Michael Wechner:

Hi Julie

Cool, thanks!

I try to apply it and if it works could create an example to the demo
package.

Will keep you posted :-)

Thanks

Michael

Am 11.05.22 um 02:13 schrieb Julie Tibshirani:

Hello Michael, we don't yet have an example of kNN with filtering in

the

demo package. This would be nice to add!

For now maybe looking at the unit tests could give a sense of how to

use

it. Here's an example:

https://github.com/apache/lucene/blob/main/lucene/core/src/test/org/apache/lucene/search/TestKnnVectorQuery.java#L115-L127
.

The idea is that KnnVectorQuery optionally accepts a Query as a
filter, and
returns the k nearest vectors that also match the filter. Many people
refer
to this as "kNN with prefiltering" (as opposed to "postfiltering",

where

the filter is applied *after* the kNN search, so in the end you may
receive
fewer than k matches).

Let us know if you run into any questions/ issues while trying it out!

Julie

On Mon, May 9, 2022 at 8:08 AM Michael Wechner
<michael.wech...@wyona.com>
wrote:

sorry for the URLs below.

I have tested Twilio SendGrid as outgoing server and it just rewrote
the
URLs


https://issues.apache.org/jira/browse/SOLR-15947
https://issues.apache.org/jira/browse/LUCENE-10382

and

https://lucene.apache.org/core/9_1_0/demo/org/apache/lucene/demo/knn/package-summary.html

which I was not aware of, but disabled the tracking now and hope it
will
be ok now.

Thanks

Michael

Am 09.05.22 um 15:12 schrieb Michael Wechner:

Hi

I noticed that Lucene 9.1.0 supports filtering in nearest-neighbor
vector search, which is great :-)

I have found

http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pcYPJTPhVT3xqtcUDjPgQX5jI0WYWlJZX8h9NDC6okDRg-3D-3DHvvY_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHnF6KpgvICPKY8k5m30V-2FWrDvoSQLIaimtz2YHrzSMNV98es7-2BeXQS174-2B0EHPnQVtOYqUPojoZgGkqmovRXrz1dJlfs9dtFDGqSjpYaFiMaVBoiDTrpJ-2FUkuanwcx6R8UgvxBCq08DUa7vhRJqir7M-3D
http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84WWONyl36e4Tdd3VZFG-2B7pe76-2Fa6XWGARorEmYO8A-2BeVhPN2B1iPvCEp9XG8WpVE6w-3D-3DHR5k_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHiGuZh3RFDkfg6-2FfNpOG8Tly-2FHwK25rp-2F24-2BPYZV6e5mGIWVQ5bpJ0l9u4lRzO6rySncTjxQZEPOjzZIrDMh-2Fo17VBGBagQ0Gr6G-2BAySO2ZdtDBthWjE-2F7HwxioFG9XrEUBVS79a0mcaaPKM-2BdzT9Cc-3D

and

http://url7093.wyona.com/ls/click?upn=JOH5Fjdv9AA9sbvUyiP84Rv2gN4NEUQqv-2Fn4lbJOY6mZGeN60klU5tyssLGPfHHB3IBl2Fx9C7un4UF2pBgYYcEd15H8F-2FPYEn4LTL-2Bz8fMFeo4z-2BjB3yMGv345VDdnStdESYCXN-2FD-2BQOSZSTNQLbQ-3D-3D9uTB_UMWFA-2BOn91WS4mEQPCWI9gZNzEZlJPmWPGP2CeMD7g4c-2Fpo3g6VPyd4ghH4X9o8sJ-2Bl292KOe2-2F30WmSZB1KHkzFWMImz7LKaiDu0g-2BZTsPclbKiyBoQiJHrZiOk5CuKsixeOFVVfvwAjyEhV-2F5McxrC76Q-2F72ILNowoPMFyMwXdaUF-2FhFh6HF0aWgai16l9zSdZIETAq46vRruPFO9ZqlRY6XiSu-2FBiKe4r5xiM0vA-3D

but it is not really clear to me how to use it.

Is there some additional documentation / example / demo how I can
combine filtering with vector search?

Thanks

Michael

---------------------------------------------------------------------

To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org