Re: knn query parser, number of results and filtering by score

Mirko Sertic Thu, 19 Oct 2023 08:51:34 -0700

I've prepared a testcase. Given the following documents withTESTEMBEDDING_EU_3 is a DenseVectorField with length 3 and euclideandistance function. They are written to a collection made of two shardswith no further routing strategy, so they should be more or less evenlydistributed between the two shards:


{
  id: 'Position1',
  TESTEMBEDDING_EU_3: [0, 0, 0]
}
{
  id: 'Position2',
  TESTEMBEDDING_EU_3: [0.1, 0.1, 0.1]
}
{
  id: 'Position3',
  TESTEMBEDDING_EU_3: [0.2, 0.2, 0.2]
}
{
  id: 'Position4',
  TESTEMBEDDING_EU_3: [0.3, 0.3, 0.3]
}
{
  id: 'Position5',
  TESTEMBEDDING_EU_3: [0.4, 0.4, 0.4]
}
{
  id: 'Position6',
  TESTEMBEDDING_EU_3: [0.5, 0.5, 0.5]
}
{
  id: 'Position7',
  TESTEMBEDDING_EU_3: [0.6, 0.6, 0.6]
}
{
  id: 'Position8',
  TESTEMBEDDING_EU_3: [0.7, 0.7, 0.7]
}
{
  id: 'Position9',
  TESTEMBEDDING_EU_3: [0.8, 0.8, 0.8]
}
{
  id: 'Position10',
  TESTEMBEDDING_EU_3: [0.9, 0.9, 0.9]
}
{
  id: 'Position11',
  TESTEMBEDDING_EU_3: [1.0, 1.0, 1.0]
}

How I'll do a {!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0] query.I'd expect a result with 3 documents, id:Position11 should be an exactmacht, and the nearst neighbors should be id:Position10 andid:Position9. I'd also expect that the explain logging should mark thesetree as part of the topK=3. I get the following search result:


{
  "responseHeader": {
    "zkConnected": true,
    "status": 0,
    "QTime": 35
  },
  "response": {
    "numFound": 6,
    "start": 0,
    "maxScore": 1.0,
    "numFoundExact": true,
    "docs": [
      {
        "id": "Position11",
        "TESTEMBEDDING_3": [
          "1.0",
          "1.0",
          "1.0"
        ],

"[shard]":"http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/";,

        "[explain]": "0.0 = not in top 3\n",
        "score": 1.0
      },
      {
        "id": "Position10",
        "TESTEMBEDDING_3": [
          "0.9",
          "0.9",
          "0.9"
        ],

        "[explain]": "0.0 = not in top 3\n",
        "score": 0.97087383
      },
      {
        "id": "Position9",
        "TESTEMBEDDING_3": [
          "0.8",
          "0.8",
          "0.8"
        ],

"[shard]":"http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/";,

        "[explain]": "0.0 = not in top 3\n",
        "score": 0.89285713
      },
      {
        "id": "Position8",
        "TESTEMBEDDING_3": [
          "0.7",
          "0.7",
          "0.7"
        ],

        "[explain]": "0.0 = not in top 3\n",
        "score": 0.78740156
      },
      {
        "id": "Position7",
        "TESTEMBEDDING_3": [
          "0.6",
          "0.6",
          "0.6"
        ],

        "[explain]": "0.0 = not in top 3\n",
        "score": 0.6756757
      },
      {
        "id": "Position6",
        "TESTEMBEDDING_3": [
          "0.5",
          "0.5",
          "0.5"
        ],

        "[explain]": "0.0 = not in top 3\n",
        "score": 0.5714286
      }
    ]
  },
  "spellcheck": {
    "suggestions": [],
    "collations": []
  },
  "debug": {
    "track": {

"rid":"fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless-465214",

      "EXECUTE_QUERY": {

"http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/":{

          "QTime": "23",
          "ElapsedTime": "27",
          "RequestPurpose": "GET_TOP_IDS,SET_TERM_STATS",
          "NumFound": "3",

"Response": "{responseHeader={zkConnected=true, status=0,QTime=23},response={numFound=3,numFoundExact=true,start=0,maxScore=1.0,docs=[SolrDocument{id=Position11,score=1.0}, SolrDocument{id=Position10, score=0.97087383},SolrDocument{id=Position7, score=0.6756757}]}, sort_values={},spellcheck={suggestions={!knn={numFound=0, startOffset=1, endOffset=5,suggestion=[]}, =testembedding_eu_3={numFound=0, startOffset=7,endOffset=26, suggestion=[]}, topk={numFound=16, startOffset=28,endOffset=32, suggestion=[topp, topf, took, tonk, tops, topik, topo,tock, tope, topa, topi, to k, tork, tolk, topb, topz]}, =3={numFound=0,startOffset=32, endOffset=34, suggestion=[]}, [1={numFound=0,startOffset=35, endOffset=37, suggestion=[]}, ,1={numFound=0,startOffset=39, endOffset=41, suggestion=[]}, ,1={numFound=0,startOffset=43, endOffset=45, suggestion=[]}},collations={collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 topp=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topp},collationInternalRank=0}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 topf=3}[1.0,1.0,1.0], hits=6,misspellingsAndCorrections={topk=topf}, collationInternalRank=1000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 took=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=took},collationInternalRank=2000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 tonk=3}[1.0,1.0,1.0], hits=6,misspellingsAndCorrections={topk=tonk}, collationInternalRank=3000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 tops=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=tops},collationInternalRank=4000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 topik=3}[1.0,1.0,1.0], hits=6,misspellingsAndCorrections={topk=topik}, collationInternalRank=5000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 topo=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topo},collationInternalRank=6000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 tock=3}[1.0,1.0,1.0], hits=6,misspellingsAndCorrections={topk=tock}, collationInternalRank=7000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 tope=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=tope},collationInternalRank=8000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 topa=3}[1.0,1.0,1.0], hits=6,misspellingsAndCorrections={topk=topa}, collationInternalRank=9000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 topi=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topi},collationInternalRank=10000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 to k=3}[1.0,1.0,1.0], hits=6,misspellingsAndCorrections={topk=to k}, collationInternalRank=11000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 tork=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=tork},collationInternalRank=12000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 tolk=3}[1.0,1.0,1.0], hits=6,misspellingsAndCorrections={topk=tolk}, collationInternalRank=13000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 topb=3}[1.0,1.0,1.0], hits=6, misspellingsAndCorrections={topk=topb},collationInternalRank=14000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 topz=3}[1.0,1.0,1.0], hits=6,misspellingsAndCorrections={topk=topz}, collationInternalRank=15000}},originalTerms=[!knn, =testembedding_eu_3, topk, =3, [1, ,1, ,1]},debug={timing={time=23.0, prepare={time=0.0, query={time=0.0},facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0},highlight={time=0.0}, stats={time=0.0}, expand={time=0.0},terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}},process={time=23.0, query={time=0.0}, facet={time=0.0},facet_module={time=0.0}, mlt={time=0.0}, highlight={time=0.0},stats={time=0.0}, expand={time=0.0}, terms={time=0.0},spellcheck={time=23.0}, debug={time=0.0}}}}}"

},

"http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/":{

          "QTime": "24",
          "ElapsedTime": "29",
          "RequestPurpose": "GET_TOP_IDS,SET_TERM_STATS",
          "NumFound": "3",

"Response": "{responseHeader={zkConnected=true, status=0,QTime=24},response={numFound=3,numFoundExact=true,start=0,maxScore=0.89285713,docs=[SolrDocument{id=Position9,score=0.89285713}, SolrDocument{id=Position8, score=0.78740156},SolrDocument{id=Position6, score=0.5714286}]}, sort_values={},spellcheck={suggestions={!knn={numFound=0, startOffset=1, endOffset=5,suggestion=[]}, =testembedding_eu_3={numFound=0, startOffset=7,endOffset=26, suggestion=[]}, topk={numFound=16, startOffset=28,endOffset=32, suggestion=[topp, took, topf, tops, topo, topik, tonk,tope, tock, to k, topi, tork, tolk, top5, topc, topka]}, =3={numFound=0,startOffset=32, endOffset=34, suggestion=[]}, [1={numFound=0,startOffset=35, endOffset=37, suggestion=[]}, ,1={numFound=0,startOffset=39, endOffset=41, suggestion=[]}, ,1={numFound=0,startOffset=43, endOffset=45, suggestion=[]}},collations={collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 topp=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topp},collationInternalRank=0}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 took=3}[1.0,1.0,1.0], hits=5,misspellingsAndCorrections={topk=took}, collationInternalRank=1000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 topf=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topf},collationInternalRank=2000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 tops=3}[1.0,1.0,1.0], hits=5,misspellingsAndCorrections={topk=tops}, collationInternalRank=3000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 topo=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topo},collationInternalRank=4000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 topik=3}[1.0,1.0,1.0], hits=5,misspellingsAndCorrections={topk=topik}, collationInternalRank=5000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 tonk=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=tonk},collationInternalRank=6000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 tope=3}[1.0,1.0,1.0], hits=5,misspellingsAndCorrections={topk=tope}, collationInternalRank=7000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 tock=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=tock},collationInternalRank=8000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 to k=3}[1.0,1.0,1.0], hits=5,misspellingsAndCorrections={topk=to k}, collationInternalRank=9000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 topi=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topi},collationInternalRank=10000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 tork=3}[1.0,1.0,1.0], hits=5,misspellingsAndCorrections={topk=tork}, collationInternalRank=11000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 tolk=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=tolk},collationInternalRank=12000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 top5=3}[1.0,1.0,1.0], hits=5,misspellingsAndCorrections={topk=top5}, collationInternalRank=13000},collation={collationQuery={!knn f=TESTEMBEDDING_EU_3 topc=3}[1.0,1.0,1.0], hits=5, misspellingsAndCorrections={topk=topc},collationInternalRank=14000}, collation={collationQuery={!knnf=TESTEMBEDDING_EU_3 topka=3}[1.0,1.0,1.0], hits=5,misspellingsAndCorrections={topk=topka}, collationInternalRank=15000}},originalTerms=[!knn, =testembedding_eu_3, topk, =3, [1, ,1, ,1]},debug={timing={time=24.0, prepare={time=0.0, query={time=0.0},facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0},highlight={time=0.0}, stats={time=0.0}, expand={time=0.0},terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}},process={time=24.0, query={time=0.0}, facet={time=0.0},facet_module={time=0.0}, mlt={time=0.0}, highlight={time=0.0},stats={time=0.0}, expand={time=0.0}, terms={time=0.0},spellcheck={time=24.0}, debug={time=0.0}}}}}"

        }
      },
      "GET_FIELDS": {

          "QTime": "1",
          "ElapsedTime": "3",
          "RequestPurpose": "GET_FIELDS,GET_DEBUG,SET_TERM_STATS",
          "NumFound": "3",

"Response": "{responseHeader={zkConnected=true, status=0,QTime=1},response={numFound=3,numFoundExact=true,start=0,docs=[SolrDocument{id=Position9,TESTEMBEDDING_3=[0.8, 0.8, 0.8],[shard]=http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/,[explain]=0.0 = not in top 3\n, score=0.89285713},SolrDocument{id=Position8, TESTEMBEDDING_3=[0.7, 0.7, 0.7],[shard]=http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/,[explain]=0.0 = not in top 3\n, score=0.78740156},SolrDocument{id=Position6, TESTEMBEDDING_3=[0.5, 0.5, 0.5],[shard]=http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p17/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_p21/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t13/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard2_replica_t15/,[explain]=0.0 = not in top 3\n, score=0.5714286}]},debug={rawquerystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0],querystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0],parsedquery=KnnVectorQuery(KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3]),parsedquery_toString=KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3],explain={Position9={match=false, value=0.0, description=not in top 3},Position8={match=false, value=0.0, description=not in top 3},Position6={match=false, value=0.0, description=not in top 3}},QParser=KnnQParser, timing={time=1.0, prepare={time=0.0,query={time=0.0}, facet={time=0.0}, facet_module={time=0.0},mlt={time=0.0}, highlight={time=0.0}, stats={time=0.0},expand={time=0.0}, terms={time=0.0}, spellcheck={time=0.0},debug={time=0.0}}, process={time=0.0, query={time=0.0},facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0},highlight={time=0.0}, stats={time=0.0}, expand={time=0.0},terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}}}}}"

},

          "QTime": "1",
          "ElapsedTime": "3",
          "RequestPurpose": "GET_FIELDS,GET_DEBUG,SET_TERM_STATS",
          "NumFound": "3",

"Response": "{responseHeader={zkConnected=true, status=0,QTime=1},response={numFound=3,numFoundExact=true,start=0,docs=[SolrDocument{id=Position10,TESTEMBEDDING_3=[0.9, 0.9, 0.9],[shard]=http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/,[explain]=0.0 = not in top 3\n, score=0.97087383},SolrDocument{id=Position11, TESTEMBEDDING_3=[1.0, 1.0, 1.0],[shard]=http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/,[explain]=0.0 = not in top 3\n, score=1.0}, SolrDocument{id=Position7,TESTEMBEDDING_3=[0.6, 0.6, 0.6],[shard]=http://fusion-integ-solr-search-200gb-0.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p9/|http://fusion-integ-solr-analytics-200gb-1.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t7/|http://fusion-integ-solr-search-200gb-1.fusion-integ-solr-search-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_p11/|http://fusion-integ-solr-analytics-200gb-0.fusion-integ-solr-analytics-200gb-headless:8983/solr/suchpool_atlas_2023_10_08_shard1_replica_t5/,[explain]=0.0 = not in top 3\n, score=0.6756757}]},debug={rawquerystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0],querystring={!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0],parsedquery=KnnVectorQuery(KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3]),parsedquery_toString=KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3],explain={Position10={match=false, value=0.0, description=not in top 3},Position11={match=false, value=0.0, description=not in top 3},Position7={match=false, value=0.0, description=not in top 3}},QParser=KnnQParser, timing={time=1.0, prepare={time=0.0,query={time=0.0}, facet={time=0.0}, facet_module={time=0.0},mlt={time=0.0}, highlight={time=0.0}, stats={time=0.0},expand={time=0.0}, terms={time=0.0}, spellcheck={time=0.0},debug={time=0.0}}, process={time=0.0, query={time=0.0},facet={time=0.0}, facet_module={time=0.0}, mlt={time=0.0},highlight={time=0.0}, stats={time=0.0}, expand={time=0.0},terms={time=0.0}, spellcheck={time=0.0}, debug={time=0.0}}}}}"

        }
      }
    },
    "timing": {
      "time": 49.0,
      "prepare": {
        "time": 0.0,
        "query": {
          "time": 0.0
        },
        "facet": {
          "time": 0.0
        },
        "facet_module": {
          "time": 0.0
        },
        "mlt": {
          "time": 0.0
        },
        "highlight": {
          "time": 0.0
        },
        "stats": {
          "time": 0.0
        },
        "expand": {
          "time": 0.0
        },
        "terms": {
          "time": 0.0
        },
        "spellcheck": {
          "time": 0.0
        },
        "debug": {
          "time": 0.0
        }
      },
      "process": {
        "time": 47.0,
        "query": {
          "time": 0.0
        },
        "facet": {
          "time": 0.0
        },
        "facet_module": {
          "time": 0.0
        },
        "mlt": {
          "time": 0.0
        },
        "highlight": {
          "time": 0.0
        },
        "stats": {
          "time": 0.0
        },
        "expand": {
          "time": 0.0
        },
        "terms": {
          "time": 0.0
        },
        "spellcheck": {
          "time": 47.0
        },
        "debug": {
          "time": 0.0
        }
      }
    },
    "rawquerystring": "{!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0]",
    "querystring": "{!knn f=TESTEMBEDDING_EU_3 topK=3}[1.0,1.0,1.0]",

"parsedquery":"KnnVectorQuery(KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3])", "parsedquery_toString":"KnnVectorQuery:TESTEMBEDDING_EU_3[1.0,...][3]",

    "QParser": "KnnQParser",
    "explain": {
      "Position11": {
        "match": false,
        "value": 0.0,
        "description": "not in top 3"
      },
      "Position10": {
        "match": false,
        "value": 0.0,
        "description": "not in top 3"
      },
      "Position9": {
        "match": false,
        "value": 0.0,
        "description": "not in top 3"
      },
      "Position8": {
        "match": false,
        "value": 0.0,
        "description": "not in top 3"
      },
      "Position7": {
        "match": false,
        "value": 0.0,
        "description": "not in top 3"
      },
      "Position6": {
        "match": false,
        "value": 0.0,
        "description": "not in top 3"
      }
    }
  }
}

I got 6 search results. The top 3 of them are correct, and it seems thattwo shards were queried, the results were combined and sorted correctlyby score. From my point of view is the result set size a functional bug,as I requested the topK=3, but I got 6.

The explain data is completely random. The description is wrong, thematch flag is wrong, and if value should be the score, it is also wrong.This is a bug in the explain implementation, and regarding to this adebug bug.


Mirko

Am 18.10.2023 um 16:56 schrieb Alessandro Benedetti:

Let's distinguish if you are talking about a "debug" bug (the explain
message) or a functional bug.

--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*

e-mail: a.benede...@sease.io


*Sease* - Information Retrieval Applied
Consulting | Training | Open Source

Website: Sease.io <http://sease.io/>
LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
<https://github.com/seaseltd>


On Wed, 18 Oct 2023 at 16:17, Mirko Sertic <mirko.ser...@web.de> wrote:

Am 18.10.2023 um 15:44 schrieb Alessandro Benedetti:

--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*

e-mail: a.benede...@sease.io


*Sease* - Information Retrieval Applied
Consulting | Training | Open Source

Website: Sease.io <http://sease.io/>
LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
<https://github.com/seaseltd>


On Wed, 18 Oct 2023 at 15:32, Mirko Sertic <mirko.ser...@web.de> wrote:

Hi Alessandro,
thank you for clarification! I havent't read something about "topK is
per shard" in the documentation, but maybe I've missed that somewhere.
   From a performance and implementation perspective, this might make
sense. From a users perspective, I'd expect that "topK" is per
collection, and not per shard, so the same behavior as "rows". I
should't have to think about the collection data layout while
formulating queries and interpreting the layouts.
Now, if "topK" is per shard, we assume that good matching results are
evenly distributed among the shards. This might not be always the case.
If we use features like collapse & expand, block/join or explicit
document routing strategies, this might lead to strange results. In this
case, it might be better to retrieve the topK per shard, join them all
together and sort them by score, and take topK from that sorted list to
get the correct overall topK. What do you think about that?

That is what currently *should* happen.
You can select the number of rows you like and get a subset of the
NShards*K results.
Are you observing something different?

Yes, if this does work correct, there should be no "not in top x" in
explain logs, but I see them, so I assume there is a bug somewhere.

We've also found https://issues.apache.org/jira/browse/SOLR-16952,

which

might be correlated.

That seems to me like a bug for the stored fields functionality.

I also tried to understand

https://github.com/apache/lucene/blob/723c05b047b4a66105636162bb3e0daaf2cc1de5/lucene/core/src/java/org/apache/lucene/search/AbstractKnnVectorQuery.java#L337
,

but failed to get the details.

I oberved the explain data further. Every document gets a score, which
is the result of the similarity function. However, some of them are
marked with "not in top 6 docs". So from my point of view those whould
not be part of the search result.

Can you elaborate on this? maybe through a full query and response with

the

debug?
Anyway, the debug score for vector-based search is currently almost
useless, the score is not really explained.
We know this and it's a Lucene level 'problem' (we haven't worked yet on

better way for doing it, is your company interested in sponsoring maybe

:) )

Mirko

Am 17.10.2023 um 18:16 schrieb Alessandro Benedetti:

Hi Mirko,
the topK is per shard.
Then shards * k results are aggregated.
Does it make sense?

In regards to the debugging, it seems a bug, they all should be with a
score and within top-k
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*

e-mail:a.benede...@sease.io


*Sease* - Information Retrieval Applied
Consulting | Training | Open Source

Website: Sease.io<http://sease.io/>
LinkedIn<https://linkedin.com/company/sease-ltd>  | Twitter
<https://twitter.com/seaseltd>  | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ>  | Github
<https://github.com/seaseltd>


On Tue, 17 Oct 2023 at 10:41, Mirko Sertic<mirko.ser...@web.de>

wrote:

Hey!

Thank you for your help!

We are running in cloud mode on GKE. Our index has 2 shards, and every
shard has 2 replicas. The leader is a TLOG, the other replica is a

PULL.

Our main query is basically {!knn f=VECTOR_FIELD topK=10}[VECTOR

DATA].

Thats it.

I am really unsure how to debug this issue, Solr explain only gives

some

basic information. I really do not understand why I am getting more

than

topK=10 search results, and some of them do not match according to the
explain data.

Here is an example debug log of the parsed query:

"rawquerystring" : "{!knn f=TEXTEMBEDDINGS_BESCHREIBUNG_768

topK=6}[-0.72818565,-0.41219202,0.5225889,0.53035444,0.76394194,0.7767894,0.82073385,0.95900697,-0.36803666,-0.21959017,0.7140104,-0.13439204,0.06747756,1.3333143,0.2797215,1.3020324,0.13869889,-0.10991136,0.21142228,-0.36915645,-0.15180133,-0.62384933,-0.51568866,0.23904686,-0.61362225,0.026911864,0.36433217,-0.6443799,0.86989623,-0.0058394694,0.360967,-0.39565668,0.4483029,-0.26269752,0.24516392,0.44815013,0.43040967,0.3587644,0.873287,-0.10215642,-0.25633913,-0.21385787,-0.5121543,-0.103228725,0.3895751,-0.49162507,-0.9432531,-0.5980949,0.63356507,-0.40455475,0.9550473,0.7282631,-0.031743232,0.439844,0.21040638,-0.36207297,0.22014715,-0.5069794,0.4957589,-0.30554834,-0.8615413,-0.4926672,-0.013558676,-0.041132603,0.65787005,0.30416572,0.18375356,0.6744972,-0.6142544,-0.18229552,-0.11830064,1.0248266,0.2171961,1.0895737,-0.027262354,-0.17439449,0.74050045,-0.8744221,-0.16733684,0.93963474,-0.4188998,-1.1815201,-0.4237022,-1.4382845,0.9899152,-0.17618449,0.8811465,-0.1435171,0.1467631,0.20906292,-0.19986387,-0.07172442,-0.5412453,-0.38141707,-0.8519122,0.57226634,0.27108946,0.47601715,-0.50961906,0.7443681,-0.46236074,0.44012043,-0.64782083,0.12072155,0.17669038,0.8894314,-0.26589298,-1.0005188,-0.38552156,-0.17849512,-0.41564974,0.46785104,0.07669882,0.098340474,-0.40940204,-0.22534837,-0.46240783,-0.6026122,-0.150852,-0.056922164,-0.5217692,0.2522323,-0.05158736,0.4368061,0.07538188,-0.15538858,-0.27512375,-7.942749,-0.5104796,0.3684156,0.17436846,0.33592805,0.30256566,0.3045547,-0.8253086,0.39595905,-0.63918144,0.023157855,-0.19350462,-1.1046336,0.24591433,-0.35902688,-0.80178124,0.60646725,-0.30873522,0.3227124,0.14232452,-0.11611875,0.36796737,-0.53184104,0.30364224,0.28488177,-1.0550518,-0.42094457,-0.12320968,-0.9765126,-0.15129066,0.001331389,-0.27246574,0.4749163,0.4177817,0.2973763,-0.29119635,-0.3972071,0.59532064,0.10756115,0.23748569,-0.70517355,1.201706,0.502714,-0.26869884,-0.053134415,-0.66120774,-0.0039092028,-0.64522946,-0.041144915,-0.05124693,-0.9726303,-0.99400455,-0.07065071,-0.78489596,0.80071807,-0.13281979,-0.006903738,0.5823942,-0.2669629,1.6502038,0.7482211,-0.6007372,0.66805166,-0.3942953,-0.5034327,0.27077916,0.15055649,0.10765681,-0.12882604,0.24207312,0.18053184,-0.21453774,0.45982376,-0.5745059,0.2006089,0.50371104,0.8538735,0.97700334,0.05624181,-0.132805,0.033950403,-0.18651533,-0.06351512,-0.7937849,-0.6627913,-0.3048031,0.53663576,0.17241986,0.36505452,-0.09964923,0.13906284,0.5434706,-0.36837682,-0.35080168,0.46206424,-0.20238751,-0.27848598,0.2480285,0.5826801,-0.48960462,0.7367855,1.0053717,0.17819782,0.38643655,-0.16025475,-0.35159552,-0.0024995431,-0.022663444,0.05895573,-0.19539028,-0.36216947,-0.06652315,-0.13567717,-0.68101376,-0.3490555,0.55224353,0.045317963,-0.8309715,-0.24363773,0.4269611,0.24411823,-0.06477823,0.80422884,0.35045695,0.20747428,-0.3475987,0.1297931,0.09917005,0.007867369,0.38570535,0.13428706,1.3826305,-0.8623333,-1.0942854,0.3748387,-0.5336166,-0.35972998,-0.10899716,0.278379,0.94615966,-0.93360096,-0.9554229,-0.28387132,-0.02770327,-0.9298182,0.45993057,-0.65919405,0.1475371,-0.34568012,1.209977,0.2802038,0.45577803,0.12152732,0.87355286,1.5883585,0.6779906,0.17098032,1.0125086,-0.55774075,-0.38352862,0.16628654,0.15557681,0.1861148,0.04379474,-0.7305415,-0.030636007,0.024982145,0.67828315,0.42840484,-0.053252134,-0.4504099,-0.20452605,-0.1081517,-0.14191566,0.07114916,-0.078530826,0.17243765,-0.46307424,-0.008914786,0.26029265,0.52127206,-0.35917446,-0.015462582,-0.28368863,-0.7689151,-0.263665,0.49229744,0.7343957,0.24539919,-0.2738248,-0.12916541,1.5979353,0.2385032,-0.5589315,0.1753772,-0.57815063,0.13694653,-0.20679545,0.78396624,-0.26194802,0.16952795,0.009313789,0.053571712,-0.50364834,0.19213165,0.020542603,0.3242613,-0.21481745,0.46329287,0.1883949,-0.53527766,0.19803612,-0.10915735,-0.53217936,0.5851483,0.8633852,-0.22306518,0.19917397,-0.48938546,-0.3489624,0.26114023,0.5178694,-0.49423835,0.22051883,0.9698861,-0.023483349,0.5058909,-0.19136174,-0.06710764,-0.50730103,-0.5724341,-0.07458467,0.72695184,0.36962357,0.12050099,0.16827017,-0.36094713,-0.22090997,0.5247373,0.5981514,0.11581371,-0.14274465,0.110738344,0.5218388,-0.38720345,0.9173073,-0.8510778,-0.22613077,-0.2748853,0.2863147,-0.3778913,-0.18058139,0.37012923,-0.5144021,-0.5551922,0.16914673,0.78963786,1.1504575,-0.7211657,-0.63688916,-0.075630635,-0.89228654,0.8719732,-0.16148148,-0.093216084,-0.0649338,0.40419587,-0.5091749,0.3257159,-0.38114238,-0.7489807,-0.14795141,0.070623375,1.0296395,-0.18232572,0.045388192,-0.5071435,-0.23788519,-0.60791194,-0.25087252,0.009456612,0.8095627,-0.43995717,-0.21105917,-0.83442384,0.7455656,-0.9289396,-0.26139355,0.33529952,-0.7209875,0.44793245,-0.11075375,0.35380912,-0.49433497,1.061043,0.24260058,1.277706,-1.0634571,-0.028423825,-0.6881516,-0.2748189,0.32650438,0.22231913,-0.6346664,-0.32749617,0.6449113,0.4030831,0.086880766,-0.34941757,-0.32754892,0.44228384,-1.0537914,-0.4920558,0.11018584,-0.4000114,0.19572248,-0.006614799,-0.1560454,-0.6138168,-0.2874727,0.08457174,0.852747,0.020211289,-0.46017483,0.05079806,0.61822194,0.16241436,-0.418595,-0.44193658,-0.23446727,0.08835063,-0.86544657,0.039239086,0.47766128,-0.31095138,-0.545759,-0.09576557,-0.5694708,-0.10043368,-0.2653149,-0.1599367,0.13423382,-0.1830842,0.046266567,-0.08192452,-0.75070643,-0.65771914,0.18250088,-0.36299148,0.33271846,-0.60936797,1.339671,-0.12343848,1.0515145,0.25372154,-0.4882932,-0.06304815,-0.17309606,0.30466563,0.08878487,-0.5156037,-0.33687803,0.5997892,0.1856752,0.5528808,0.5089958,-0.77673227,-0.43565074,-3.0061991,-0.22721101,1.1010107,1.285386,0.39106908,0.06541425,0.18982089,-0.33513412,-0.5287475,0.2362032,0.40598843,0.62325794,-0.8586378,-0.64406306,0.09112385,0.055086333,0.28247252,0.13180174,0.5969885,0.15263812,0.39325213,-0.45515132,-0.47813737,-0.33163494,0.3496457,0.39913225,-0.5750896,0.104484975,-0.69256973,0.18164606,0.09423768,0.045548365,0.51491827,-0.854532,0.40470743,-0.12821743,-0.22693269,-0.31501335,-0.53367645,0.4482267,-0.647523,-0.48177683,-0.45763886,0.31513563,0.115996234,-0.56461513,0.112646066,0.6897073,0.30277768,1.2858859,-0.47804725,0.36647198,0.8737555,0.89239925,-0.26573908,0.13890187,0.8274238,0.6068263,0.21134727,-0.5331567,-0.40894493,-0.20580018,-0.4116305,-8.852631E-4,0.71050125,0.18508287,0.3889124,-0.38231707,-0.7620664,-0.26053023,0.5970558,0.12187529,-0.35657012,-0.20820218,-0.4604194,-0.36569384,0.08593381,0.31331047,-0.12055105,0.12107455,0.45513165,-0.0023682665,0.3980597,0.1632141,0.33392814,-0.050680444,0.13038154,0.13847367,1.3034295,0.2744402,0.124595374,-0.53905815,0.35776177,-0.13275874,0.018385967,-0.16347891,-0.19726902,-0.4961019,0.07333634,0.52688265,0.20600657,0.14087541,-0.21112329,0.5135732,0.4315345,-0.020893982,0.41702232,0.07464256,-0.2845659,-0.13949631,-0.013015106,0.40522456,0.041121256,1.1591781,0.043223336,0.39874303,-0.30360883,-0.27349794,-0.04964445,-0.5729313,0.14069785,-0.33394682,0.9343837,-0.9350466,-0.8602237,-0.29167423,-1.5445148,0.040775828,-0.018334387,0.42681727,-0.1458294,0.49805698,0.27569738,-0.10979689,-0.5429115,-0.020050785,0.31627277,-0.6220587,0.061334964,0.13807471,0.34479836,0.11312207,0.796787,-0.48607716,-0.14010525,-0.61075526,0.6672587,1.6726193,0.4950563,0.41054416,0.2522166,-0.95681673,-1.0119423,-0.4870938,0.48430035,-0.38044646,0.59904295,0.23687701,0.93788594,-0.22263275,0.07065558,-0.4562675,-0.08655307,0.69646436,-0.5812278,-0.30097616,-0.9931224,0.23148312,0.13233739,0.74470276,0.014401446,0.08970716,0.23342986,-0.4664313,0.054693338,0.2778716,0.4019597,-0.97575116,0.058492173,-0.5017976,0.09864781,-0.3473021,-0.4823157,0.13100009,0.4488969,-0.1672722,-1.0326554,-0.18146743,-0.349057,0.61888343,-0.18784277,-0.53318554,-0.2559318,-0.6182706,0.24759102,-0.71451706,0.7098109,0.53027123,0.36208203,0.23936976,-0.10664157,0.3863506,0.42203853,0.1808002,-0.22444467,1.0584201,-0.29779503,0.43873647,0.2835624,-0.76047546,0.37234184,0.2044634,0.5471627,0.4247879,0.4856389,-0.7011678,0.40757468,0.1918484,0.54874665,-0.1279645,0.5678282,0.02382508,0.18181127,-0.42465243,0.20996757,-0.48161522,-0.53782755,-0.1985728,-0.12512843,-0.47403875,0.44477355,-0.05491643,0.81035346,0.3633467,0.5024226,-0.07482373,-0.7337039,0.0675221,0.3952929,0.45677915,-0.29935122,-0.42017272,-0.80567694,0.2868536,-0.40015128,-0.060443062,-0.21671702,0.18321274,-0.20112215,-0.50728095,0.3698101,0.56871736,-0.0904237,-0.43735075,-0.79018253,-0.0739331,0.49475756,0.014863159,-0.47107005,0.14020085,0.28982484,0.32168245,-0.56485337,0.17436136,-0.0050138533,-0.9533212,0.036251593,-0.33072975,-0.87562966,-0.4379939,0.8052754,0.30534235,0.20573251]",

"querystring" : "{!knn f=TEXTEMBEDDINGS_BESCHREIBUNG_768

topK=6}[-0.72818565,-0.41219202,0.5225889,0.53035444,0.76394194,0.7767894,0.82073385,0.95900697,-0.36803666,-0.21959017,0.7140104,-0.13439204,0.06747756,1.3333143,0.2797215,1.3020324,0.13869889,-0.10991136,0.21142228,-0.36915645,-0.15180133,-0.62384933,-0.51568866,0.23904686,-0.61362225,0.026911864,0.36433217,-0.6443799,0.86989623,-0.0058394694,0.360967,-0.39565668,0.4483029,-0.26269752,0.24516392,0.44815013,0.43040967,0.3587644,0.873287,-0.10215642,-0.25633913,-0.21385787,-0.5121543,-0.103228725,0.3895751,-0.49162507,-0.9432531,-0.5980949,0.63356507,-0.40455475,0.9550473,0.7282631,-0.031743232,0.439844,0.21040638,-0.36207297,0.22014715,-0.5069794,0.4957589,-0.30554834,-0.8615413,-0.4926672,-0.013558676,-0.041132603,0.65787005,0.30416572,0.18375356,0.6744972,-0.6142544,-0.18229552,-0.11830064,1.0248266,0.2171961,1.0895737,-0.027262354,-0.17439449,0.74050045,-0.8744221,-0.16733684,0.93963474,-0.4188998,-1.1815201,-0.4237022,-1.4382845,0.9899152,-0.17618449,0.8811465,-0.1435171,0.1467631,0.20906292,-0.19986387,-0.07172442,-0.5412453,-0.38141707,-0.8519122,0.57226634,0.27108946,0.47601715,-0.50961906,0.7443681,-0.46236074,0.44012043,-0.64782083,0.12072155,0.17669038,0.8894314,-0.26589298,-1.0005188,-0.38552156,-0.17849512,-0.41564974,0.46785104,0.07669882,0.098340474,-0.40940204,-0.22534837,-0.46240783,-0.6026122,-0.150852,-0.056922164,-0.5217692,0.2522323,-0.05158736,0.4368061,0.07538188,-0.15538858,-0.27512375,-7.942749,-0.5104796,0.3684156,0.17436846,0.33592805,0.30256566,0.3045547,-0.8253086,0.39595905,-0.63918144,0.023157855,-0.19350462,-1.1046336,0.24591433,-0.35902688,-0.80178124,0.60646725,-0.30873522,0.3227124,0.14232452,-0.11611875,0.36796737,-0.53184104,0.30364224,0.28488177,-1.0550518,-0.42094457,-0.12320968,-0.9765126,-0.15129066,0.001331389,-0.27246574,0.4749163,0.4177817,0.2973763,-0.29119635,-0.3972071,0.59532064,0.10756115,0.23748569,-0.70517355,1.201706,0.502714,-0.26869884,-0.053134415,-0.66120774,-0.0039092028,-0.64522946,-0.041144915,-0.05124693,-0.9726303,-0.99400455,-0.07065071,-0.78489596,0.80071807,-0.13281979,-0.006903738,0.5823942,-0.2669629,1.6502038,0.7482211,-0.6007372,0.66805166,-0.3942953,-0.5034327,0.27077916,0.15055649,0.10765681,-0.12882604,0.24207312,0.18053184,-0.21453774,0.45982376,-0.5745059,0.2006089,0.50371104,0.8538735,0.97700334,0.05624181,-0.132805,0.033950403,-0.18651533,-0.06351512,-0.7937849,-0.6627913,-0.3048031,0.53663576,0.17241986,0.36505452,-0.09964923,0.13906284,0.5434706,-0.36837682,-0.35080168,0.46206424,-0.20238751,-0.27848598,0.2480285,0.5826801,-0.48960462,0.7367855,1.0053717,0.17819782,0.38643655,-0.16025475,-0.35159552,-0.0024995431,-0.022663444,0.05895573,-0.19539028,-0.36216947,-0.06652315,-0.13567717,-0.68101376,-0.3490555,0.55224353,0.045317963,-0.8309715,-0.24363773,0.4269611,0.24411823,-0.06477823,0.80422884,0.35045695,0.20747428,-0.3475987,0.1297931,0.09917005,0.007867369,0.38570535,0.13428706,1.3826305,-0.8623333,-1.0942854,0.3748387,-0.5336166,-0.35972998,-0.10899716,0.278379,0.94615966,-0.93360096,-0.9554229,-0.28387132,-0.02770327,-0.9298182,0.45993057,-0.65919405,0.1475371,-0.34568012,1.209977,0.2802038,0.45577803,0.12152732,0.87355286,1.5883585,0.6779906,0.17098032,1.0125086,-0.55774075,-0.38352862,0.16628654,0.15557681,0.1861148,0.04379474,-0.7305415,-0.030636007,0.024982145,0.67828315,0.42840484,-0.053252134,-0.4504099,-0.20452605,-0.1081517,-0.14191566,0.07114916,-0.078530826,0.17243765,-0.46307424,-0.008914786,0.26029265,0.52127206,-0.35917446,-0.015462582,-0.28368863,-0.7689151,-0.263665,0.49229744,0.7343957,0.24539919,-0.2738248,-0.12916541,1.5979353,0.2385032,-0.5589315,0.1753772,-0.57815063,0.13694653,-0.20679545,0.78396624,-0.26194802,0.16952795,0.009313789,0.053571712,-0.50364834,0.19213165,0.020542603,0.3242613,-0.21481745,0.46329287,0.1883949,-0.53527766,0.19803612,-0.10915735,-0.53217936,0.5851483,0.8633852,-0.22306518,0.19917397,-0.48938546,-0.3489624,0.26114023,0.5178694,-0.49423835,0.22051883,0.9698861,-0.023483349,0.5058909,-0.19136174,-0.06710764,-0.50730103,-0.5724341,-0.07458467,0.72695184,0.36962357,0.12050099,0.16827017,-0.36094713,-0.22090997,0.5247373,0.5981514,0.11581371,-0.14274465,0.110738344,0.5218388,-0.38720345,0.9173073,-0.8510778,-0.22613077,-0.2748853,0.2863147,-0.3778913,-0.18058139,0.37012923,-0.5144021,-0.5551922,0.16914673,0.78963786,1.1504575,-0.7211657,-0.63688916,-0.075630635,-0.89228654,0.8719732,-0.16148148,-0.093216084,-0.0649338,0.40419587,-0.5091749,0.3257159,-0.38114238,-0.7489807,-0.14795141,0.070623375,1.0296395,-0.18232572,0.045388192,-0.5071435,-0.23788519,-0.60791194,-0.25087252,0.009456612,0.8095627,-0.43995717,-0.21105917,-0.83442384,0.7455656,-0.9289396,-0.26139355,0.33529952,-0.7209875,0.44793245,-0.11075375,0.35380912,-0.49433497,1.061043,0.24260058,1.277706,-1.0634571,-0.028423825,-0.6881516,-0.2748189,0.32650438,0.22231913,-0.6346664,-0.32749617,0.6449113,0.4030831,0.086880766,-0.34941757,-0.32754892,0.44228384,-1.0537914,-0.4920558,0.11018584,-0.4000114,0.19572248,-0.006614799,-0.1560454,-0.6138168,-0.2874727,0.08457174,0.852747,0.020211289,-0.46017483,0.05079806,0.61822194,0.16241436,-0.418595,-0.44193658,-0.23446727,0.08835063,-0.86544657,0.039239086,0.47766128,-0.31095138,-0.545759,-0.09576557,-0.5694708,-0.10043368,-0.2653149,-0.1599367,0.13423382,-0.1830842,0.046266567,-0.08192452,-0.75070643,-0.65771914,0.18250088,-0.36299148,0.33271846,-0.60936797,1.339671,-0.12343848,1.0515145,0.25372154,-0.4882932,-0.06304815,-0.17309606,0.30466563,0.08878487,-0.5156037,-0.33687803,0.5997892,0.1856752,0.5528808,0.5089958,-0.77673227,-0.43565074,-3.0061991,-0.22721101,1.1010107,1.285386,0.39106908,0.06541425,0.18982089,-0.33513412,-0.5287475,0.2362032,0.40598843,0.62325794,-0.8586378,-0.64406306,0.09112385,0.055086333,0.28247252,0.13180174,0.5969885,0.15263812,0.39325213,-0.45515132,-0.47813737,-0.33163494,0.3496457,0.39913225,-0.5750896,0.104484975,-0.69256973,0.18164606,0.09423768,0.045548365,0.51491827,-0.854532,0.40470743,-0.12821743,-0.22693269,-0.31501335,-0.53367645,0.4482267,-0.647523,-0.48177683,-0.45763886,0.31513563,0.115996234,-0.56461513,0.112646066,0.6897073,0.30277768,1.2858859,-0.47804725,0.36647198,0.8737555,0.89239925,-0.26573908,0.13890187,0.8274238,0.6068263,0.21134727,-0.5331567,-0.40894493,-0.20580018,-0.4116305,-8.852631E-4,0.71050125,0.18508287,0.3889124,-0.38231707,-0.7620664,-0.26053023,0.5970558,0.12187529,-0.35657012,-0.20820218,-0.4604194,-0.36569384,0.08593381,0.31331047,-0.12055105,0.12107455,0.45513165,-0.0023682665,0.3980597,0.1632141,0.33392814,-0.050680444,0.13038154,0.13847367,1.3034295,0.2744402,0.124595374,-0.53905815,0.35776177,-0.13275874,0.018385967,-0.16347891,-0.19726902,-0.4961019,0.07333634,0.52688265,0.20600657,0.14087541,-0.21112329,0.5135732,0.4315345,-0.020893982,0.41702232,0.07464256,-0.2845659,-0.13949631,-0.013015106,0.40522456,0.041121256,1.1591781,0.043223336,0.39874303,-0.30360883,-0.27349794,-0.04964445,-0.5729313,0.14069785,-0.33394682,0.9343837,-0.9350466,-0.8602237,-0.29167423,-1.5445148,0.040775828,-0.018334387,0.42681727,-0.1458294,0.49805698,0.27569738,-0.10979689,-0.5429115,-0.020050785,0.31627277,-0.6220587,0.061334964,0.13807471,0.34479836,0.11312207,0.796787,-0.48607716,-0.14010525,-0.61075526,0.6672587,1.6726193,0.4950563,0.41054416,0.2522166,-0.95681673,-1.0119423,-0.4870938,0.48430035,-0.38044646,0.59904295,0.23687701,0.93788594,-0.22263275,0.07065558,-0.4562675,-0.08655307,0.69646436,-0.5812278,-0.30097616,-0.9931224,0.23148312,0.13233739,0.74470276,0.014401446,0.08970716,0.23342986,-0.4664313,0.054693338,0.2778716,0.4019597,-0.97575116,0.058492173,-0.5017976,0.09864781,-0.3473021,-0.4823157,0.13100009,0.4488969,-0.1672722,-1.0326554,-0.18146743,-0.349057,0.61888343,-0.18784277,-0.53318554,-0.2559318,-0.6182706,0.24759102,-0.71451706,0.7098109,0.53027123,0.36208203,0.23936976,-0.10664157,0.3863506,0.42203853,0.1808002,-0.22444467,1.0584201,-0.29779503,0.43873647,0.2835624,-0.76047546,0.37234184,0.2044634,0.5471627,0.4247879,0.4856389,-0.7011678,0.40757468,0.1918484,0.54874665,-0.1279645,0.5678282,0.02382508,0.18181127,-0.42465243,0.20996757,-0.48161522,-0.53782755,-0.1985728,-0.12512843,-0.47403875,0.44477355,-0.05491643,0.81035346,0.3633467,0.5024226,-0.07482373,-0.7337039,0.0675221,0.3952929,0.45677915,-0.29935122,-0.42017272,-0.80567694,0.2868536,-0.40015128,-0.060443062,-0.21671702,0.18321274,-0.20112215,-0.50728095,0.3698101,0.56871736,-0.0904237,-0.43735075,-0.79018253,-0.0739331,0.49475756,0.014863159,-0.47107005,0.14020085,0.28982484,0.32168245,-0.56485337,0.17436136,-0.0050138533,-0.9533212,0.036251593,-0.33072975,-0.87562966,-0.4379939,0.8052754,0.30534235,0.20573251]",

"parsedquery" :

"KnnVectorQuery(KnnVectorQuery:TEXTEMBEDDINGS_BESCHREIBUNG_768[-0.72818565,...][6])",

"parsedquery_toString" :
"KnnVectorQuery:TEXTEMBEDDINGS_BESCHREIBUNG_768[-0.72818565,...][6]",
"QParser" : "KnnQParser",

Am 17.10.2023 um 10:05 schrieb Alessandro Benedetti:

What's your full Solr query?
Are you on SolrCloud or single Solr node?
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*

e-mail:a.benede...@sease.io


*Sease* - Information Retrieval Applied
Consulting | Training | Open Source

Website: Sease.io<http://sease.io/>
LinkedIn<https://linkedin.com/company/sease-ltd>   | Twitter
<https://twitter.com/seaseltd>   | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ>   |

Github

<https://github.com/seaseltd>


On Tue, 17 Oct 2023 at 09:45, Mirko Sertic<mirko.ser...@web.de>

   wrote:

To correct me, there was a typo. I meant:

If I specify topK=6, I get numFound=12, but only some of them match

the

top 6


Am 17.10.2023 um 09:31 schrieb Mirko Sertic:

Hi!

To keep you updated, here are some observations regarding the
numFound/resultset size and DenseVectorQueries:

If I specity topK=10, I get numFound=20, but only some of them

match

the

top 10

If I specify topK=8, I get numFound=16, but only some of them match

the

top 8

If I specify topK=6, I get numFound=8, but only some of them match

the

top 6

So the numFound seems always to be the double of topK. Might there

be a

correlation with sharding? Our collection has two shards, so does

this

double the results? I would't expect that, but that might be the

only

thing relating to a constant 2 in our setup.

Mirko


Am 16.10.2023 um 14:46 schrieb Mirko Sertic:

Hi@all

We are using Solr 9.1.1, and are trying usecases with DenseVector

Fields

and knnQueries in mind.

During our tests, we see the following results and are trying to

figure

out what is going on:

a) We use the following main query : {!knn f=VECTOR_FIELD
topK=10}[VECTOR DATA]. We use it as a main query because we want

to

apply the distance function to the document score. However, when I

try

do debug and explain the search results, I am getting more than

topK=10

result documents, some are marked as match = true with "within top

10",

others are marked as match = false with "not in top 10". I'd

expect

that

only matched documents are part of the search result, but there

are

result documents, but only 5 of them are matched. Did I miss

something?

b) The knn query results are the approximate nearest neighbors,

but

they

might not be the best. We'd like to define some kind of cut-off

value

for knn document scores. Is this possible, and what would be a

good

day

to do so? Implement a post-processing filter query with an frange

on

the

score field?

Thank you all,

Mirko

Re: knn query parser, number of results and filtering by score

Reply via email to