>From Ali Alsuliman <[email protected]>: Ali Alsuliman has submitted this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21502?usp=email )
( 10 is the latest approved patch-set. No files were changed between the latest approved patch-set and the submitted one. )Change subject: [ASTERIXDB-3804][COMP] Drop the kmultiplier hint ...................................................................... [ASTERIXDB-3804][COMP] Drop the kmultiplier hint - user model changes: no - storage format changes: no - interface changes: yes Details: compiler.vector.kmultiplier duplicated the ann_distance 5th argument and took precedence over it whenever both were set. Drop the hint, its settable-option allowlist entry and the kMultiplier field it fed on the VTree search operator so the per-query argument is the only channel. Ext-ref: MB-72975 Change-Id: I67d27b8e74f5ee5e8eccc0d799d61ad45854a5ce Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21502 Reviewed-by: Hongyu Shi <[email protected]> Reviewed-by: Ali Alsuliman <[email protected]> Integration-Tests: Jenkins <[email protected]> Tested-by: Jenkins <[email protected]> --- M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/SqlppCompilationProvider.java M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.5.query.sqlpp M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.5.query.sqlpp M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java M asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorDescriptor.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorNodePushable.java 7 files changed, 12 insertions(+), 37 deletions(-) Approvals: Jenkins: Verified; Verified Ali Alsuliman: Looks good to me, approved Hongyu Shi: Looks good to me, but someone else must approve diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/SqlppCompilationProvider.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/SqlppCompilationProvider.java index aa4342e..13840f0 100644 --- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/SqlppCompilationProvider.java +++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/compiler/provider/SqlppCompilationProvider.java @@ -122,7 +122,6 @@ CompilerProperties.COMPILER_HDFS_SPLIT_PARALLELISM_KEY, CompilerProperties.COMPILER_AGGREGATE_DISTINCT_HASH_MEMORY_KEY, CompilerProperties.COMPILER_AGGREGATE_DISTINCT_HASH_KEY, - CompilerProperties.COMPILER_VECTOR_K_MULTIPLIER_KEY, DisjunctivePredicateToJoinRule.REWRITE_OR_AS_JOIN_OPTION, SetAsterixPhysicalOperatorsRule.REWRITE_ATTEMPT_BATCH_ASSIGN, EquivalenceClassUtils.REWRITE_INTERNAL_QUERYUID_PK, SqlppQueryRewriter.SQL_COMPAT_OPTION, diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.5.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.5.query.sqlpp index 3327177..e1092be 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.5.query.sqlpp +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-glove/create-index-vtree-glove.5.query.sqlpp @@ -20,10 +20,9 @@ USE testg; // Selecting a non-PK field forces the lookup-and-rerank plan: final order is exact -// ann_distance over real vectors. kmultiplier 30 + min_probe_fraction 1.0 makes the +// ann_distance over real vectors. k_multiplier 30 (arg 5) + min_probe_fraction 1.0 makes the // candidate set cover the whole subset, so the golden equals brute force by construction. -SET `compiler.vector.kmultiplier` "30"; SELECT VALUE m.label FROM GloveReal m -ORDER BY ann_distance(m.embedding, [-0.31189998984336853, -0.017993999645113945, -0.955560028553009, -0.2546199858188629, 0.3377699851989746, 0.63714998960495, -0.03886299952864647, 0.11513999849557877, -0.049150001257658005, -0.60139000415802, -0.05385199934244156, 0.49226999282836914, -2.186800003051758, 0.5229499936103821, -0.20976999402046204, 0.05969399958848953, -0.3080799877643585, 1.2232999801635742, 0.1712999939918518, -0.025250999256968498, -0.36294999718666077, -0.4676100015640259, 0.033385999500751495, -0.561710000038147, -0.3194600045681, -2.5153000354766846, 0.20243999361991882, -0.010824000462889671, 0.45072999596595764, 0.08113600313663483, -0.17021000385284424, 0.25001001358032227, -0.35822001099586487, 0.023899000138044357, 0.483379989862442, 0.17270000278949738, -0.16419999301433563, 0.016022000461816788, 0.12279000133275986, -0.14080999791622162, -0.28887999057769775, 0.22788000106811523, 0.22682000696659088, 0.1999800056219101, 0.7948499917984009, -0.10678000003099442, -0.16958999633789062, -0.09583199769258499, -0.2455199956893921, 0.1941699981689453, -0.36068999767303467, -0.29128000140190125, -0.7184299826622009, -0.23351000249385834, 0.12636999785900116, 0.04436499997973442, 0.9266200065612793, -0.19019000232219696, -0.5694000124931335, 0.24188999831676483, -0.33375000953674316, 0.6856499910354614, 0.35210999846458435, -0.08330400288105011, 1.9664000272750854, 0.15578000247478485, 0.10346999764442444, -0.06088100001215935, 0.7112200260162354, -0.355430006980896, 0.4097599983215332, 0.18609000742435455, 0.5758600234985352, 0.8240699768066406, 0.010301000438630581, 0.7072100043296814, -0.06193400174379349, 0.37685999274253845, 0.7658299803733826, -0.40865999460220337, 1.7396999597549438, -0.19054999947547913, 0.5143200159072876, -1.0634000301361084, -0.02160000056028366, 0.2547900080680847, 0.025088999420404434, -0.0999630019068718, 0.08318299800157547, 0.0013690999476239085, -0.3424699902534485, 0.10591000318527222, 0.3529700040817261, -0.3697499930858612, 0.10261999815702438, 0.4934200048446655, 0.532230019569397, -1.070199966430664, -0.6526299715042114, 0.36702001094818115], "cosine", 1.0) +ORDER BY ann_distance(m.embedding, [-0.31189998984336853, -0.017993999645113945, -0.955560028553009, -0.2546199858188629, 0.3377699851989746, 0.63714998960495, -0.03886299952864647, 0.11513999849557877, -0.049150001257658005, -0.60139000415802, -0.05385199934244156, 0.49226999282836914, -2.186800003051758, 0.5229499936103821, -0.20976999402046204, 0.05969399958848953, -0.3080799877643585, 1.2232999801635742, 0.1712999939918518, -0.025250999256968498, -0.36294999718666077, -0.4676100015640259, 0.033385999500751495, -0.561710000038147, -0.3194600045681, -2.5153000354766846, 0.20243999361991882, -0.010824000462889671, 0.45072999596595764, 0.08113600313663483, -0.17021000385284424, 0.25001001358032227, -0.35822001099586487, 0.023899000138044357, 0.483379989862442, 0.17270000278949738, -0.16419999301433563, 0.016022000461816788, 0.12279000133275986, -0.14080999791622162, -0.28887999057769775, 0.22788000106811523, 0.22682000696659088, 0.1999800056219101, 0.7948499917984009, -0.10678000003099442, -0.16958999633789062, -0.09583199769258499, -0.2455199956893921, 0.1941699981689453, -0.36068999767303467, -0.29128000140190125, -0.7184299826622009, -0.23351000249385834, 0.12636999785900116, 0.04436499997973442, 0.9266200065612793, -0.19019000232219696, -0.5694000124931335, 0.24188999831676483, -0.33375000953674316, 0.6856499910354614, 0.35210999846458435, -0.08330400288105011, 1.9664000272750854, 0.15578000247478485, 0.10346999764442444, -0.06088100001215935, 0.7112200260162354, -0.355430006980896, 0.4097599983215332, 0.18609000742435455, 0.5758600234985352, 0.8240699768066406, 0.010301000438630581, 0.7072100043296814, -0.06193400174379349, 0.37685999274253845, 0.7658299803733826, -0.40865999460220337, 1.7396999597549438, -0.19054999947547913, 0.5143200159072876, -1.0634000301361084, -0.02160000056028366, 0.2547900080680847, 0.025088999420404434, -0.0999630019068718, 0.08318299800157547, 0.0013690999476239085, -0.3424699902534485, 0.10591000318527222, 0.3529700040817261, -0.3697499930858612, 0.10261999815702438, 0.4934200048446655, 0.532230019569397, -1.070199966430664, -0.6526299715042114, 0.36702001094818115], "cosine", 1.0, 30) LIMIT 5; diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.5.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.5.query.sqlpp index ef0e189..a0855d1 100644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.5.query.sqlpp +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/vector/create-index-vtree-movie/create-index-vtree-movie.5.query.sqlpp @@ -20,10 +20,9 @@ USE testm; // Selecting a non-PK field forces the lookup-and-rerank plan: final order is exact -// ann_distance over real vectors. kmultiplier 30 + min_probe_fraction 1.0 makes the +// ann_distance over real vectors. k_multiplier 30 (arg 5) + min_probe_fraction 1.0 makes the // candidate set cover the whole subset, so the golden equals brute force by construction. -SET `compiler.vector.kmultiplier` "30"; SELECT VALUE m.title FROM MovieReal m -ORDER BY ann_distance(m.embedding, [-0.01857922412455082, -0.0614464208483696, -0.07124938815832138, 0.003173143370077014, -0.026651009917259216, 0.07428602129220963, 0.014454759657382965, 0.0009055715636350214, 0.09926825761795044, -0.05026109889149666, 0.050097666680812836, 0.041439659893512726, 0.009396902285516262, 0.04711673781275749, 0.004123533610254526, -0.03571753948926926, 0.03176308423280716, 0.0560125932097435, 0.04764378443360329, 0.009818535298109055, -0.005663667339831591, 0.02399379573762417, 0.012543700635433197, -0.019701527431607246, -0.04974042996764183, -0.0022816669661551714, 0.005969239864498377, 0.0007606040453538299, -0.0997253879904747, -0.03994547575712204, 0.05498439818620682, 0.06168409809470177, 0.021613294258713722, 0.03297579288482666, 0.05400355905294418, -0.06328761577606201, 0.03684171661734581, 0.07926641404628754, 0.014345908537507057, -0.0059095327742397785, -0.030750686302781105, -0.04278847202658653, 0.04796496033668518, 0.02237524278461933, 0.08090664446353912, -0.13150203227996826, -0.07132289558649063, -0.02492316998541355, 0.03502165526151657, 0.0053969332948327065, -0.021918678656220436, -0.04347515106201172, 0.01707594096660614, 0.07778174430131912, -0.0009688373538665473, -0.009289815090596676, 0.024557504802942276, 0.07663727551698685, 0.04940816015005112, -0.02507856860756874, 0.008993245661258698, -0.008366926573216915, 0.0312025286257267, 0.06300637125968933, 0.04590312018990517, -0.06628154218196869, -0.007432340178638697, 0.11651013046503067, -0.04510491341352463, 0.08607837557792664, -0.045920416712760925, -0.071485236287117, -0.0136015135794878, -0.03322683647274971, -0.014835990034043789, -0.007704234682023525, -0.10948145389556885, 0.007482130546122789, -0.014187892898917198, -0.002135256538167596, 0.006972563453018665, -0.046164482831954956, -0.10869627445936203, -0.018784793093800545, -0.0901474729180336, 0.014426912181079388, -0.03082936257123947, -0.0009643586818128824, -0.02510443888604641, -0.019585473462939262, -0.09764410555362701, -0.026426440104842186, 0.03617390990257263, -0.04059203714132309, -0.08562717586755753, -0.03982330113649368, -0.024080470204353333, 0.019618013873696327, -0.003956600558012724, 0.004903997760266066, 0.029250403866171837, -0.03079192340373993, 0.039038658142089844, -0.08655649423599243, 0.15646182000637054, 0.00026213270029984415, 0.030180705711245537, 0.0030696187168359756, 0.03589688241481781, -0.014934344217181206, 0.011576713994145393, 0.04430369660258293, -0.0005557635449804366, -0.025718053802847862, 0.1800142377614975, 0.054825909435749054, -0.022272437810897827, 0.038649093359708786, -0.044413141906261444, 0.07403233647346497, 0.035068329423666, 0.027785509824752808, -0.09360716491937637, -0.03326297923922539, -0.09805632382631302, -0.0419611856341362, 0.04094903916120529, -1.0503740892722901e-33, -0.009065549820661545, -0.04004768282175064, -0.08231294900178909, 0.05198328197002411, 0.08590659499168396, 0.006291812751442194, -0.009413616731762886, 0.0015937554417178035, -0.046500954777002335, 0.03878068923950195, -0.014843151904642582, 0.005073499400168657, -0.08867146074771881, 0.09745395183563232, 0.01727250963449478, 0.043382544070482254, -0.08794844895601273, -0.0534907728433609, 0.05481107532978058, -0.025791222229599953, -0.06366338580846786, 0.037842266261577606, -0.027113165706396103, 0.0015952001558616757, -0.06755585968494415, 0.0013132041785866022, -0.05610702559351921, -0.11119788885116577, 0.10298287123441696, -0.00596569525077939, -0.046586696058511734, 0.12350807338953018, 0.055325765162706375, -0.020102113485336304, 0.017435625195503235, 0.04402526095509529, -0.01169867254793644, -0.043985188007354736, -0.014159473590552807, 0.08971824496984482, -0.08812947571277618, 0.029233265668153763, -0.016919871792197227, -0.04773721843957901, 0.004967318847775459, 0.016504360362887383, -0.028715811669826508, -0.01689995266497135, -0.028662871569395065, 0.06107230857014656, -0.012691300362348557, -0.02005895785987377, 0.0007857103482820094, 0.0020176509860903025, -0.06752794235944748, 0.018846377730369568, -0.06125511974096298, -0.013897846452891827, -0.0004379384045023471, 0.052370160818099976, 0.0652880147099495, 0.03635825216770172, 0.004397145938128233, -0.030738957226276398, -0.048334185034036636, 0.04099040478467941, 0.018386656418442726, 0.04714670777320862, -0.0236875768750906, 0.09330955147743225, -0.07421127706766129, 0.08887913078069687, -0.011085454374551773, -0.02150590717792511, 0.025899238884449005, 0.040646784007549286, 0.018326999619603157, -0.009661125019192696, -0.09291890263557434, 0.023891927674412727, -0.04675614833831787, -0.024499593302607536, 0.025196222588419914, 0.002284818096086383, -0.0713845044374466, 0.05112673342227936, 0.0011782529763877392, -0.09097935259342194, 0.03155287355184555, 0.07124300301074982, -0.03573298081755638, -0.07202157378196716, 0.024810437113046646, 0.06414418667554855, 0.04424704611301422, -1.3988019106220436e-33, 0.07596112787723541, -0.08598760515451431, 0.023408066481351852, 0.010256262496113777, 0.0975826308131218, 0.03112787753343582, -0.19472666084766388, -0.031074006110429764, -0.024440864101052284, -0.04895230010151863, -0.12436351925134659, -0.015835482627153397, 0.039502210915088654, 0.03140244632959366, 0.07963060587644577, -0.01341002993285656, -0.02321160025894642, -0.043583400547504425, -0.06827471405267715, 0.08362080156803131, 0.048776354640722275, -0.008129852823913097, 0.005035134498029947, -0.03532380238175392, 0.047764550894498825, 0.047842249274253845, -0.026258224621415138, 0.017514122650027275, 0.008043339475989342, 0.08044011890888214, -0.036833275109529495, -0.006690408103168011, 0.040764402598142624, -0.02005578577518463, -0.08742470294237137, 0.0876992866396904, 0.0018844923470169306, -0.02887721173465252, 0.017003506422042847, -0.013857734389603138, -0.00867075752466917, -0.029464416205883026, -0.05655977502465248, 0.060238782316446304, -0.038879845291376114, -0.05127982795238495, -0.03536272794008255, 0.0035438239574432373, 0.04611906409263611, -0.0027573236729949713, -0.0005832789465785027, 0.045093733817338943, -0.05433909595012665, -0.004022092558443546, -0.016827723011374474, 0.005193574819713831, -0.011994755826890469, -0.05103548243641853, -0.06774675101041794, -0.02298033982515335, -0.1271994262933731, 0.10202986747026443, -0.0015871650539338589, 0.08326882869005203, 0.0021731259766966105, -0.035482317209243774, -0.007749130018055439, -0.007956304587423801, -0.12692050635814667, -0.027656463906168938, -0.033358749002218246, -0.008985556662082672, -0.004958901554346085, 0.0425625815987587, 0.06226276978850365, -0.018582263961434364, -0.08583124727010727, -0.01759367622435093, -0.039044685661792755, 0.023986786603927612, 0.014820548705756664, -0.005747810937464237, 0.06648056209087372, 0.09194660931825638, -0.007023005746304989, 0.08919760584831238, -0.022919125854969025, 0.025847693905234337, 0.01645096018910408, 0.001485480577684939, 0.09275486320257187, -0.0025220727548003197, 0.03092137724161148, -0.001075696898624301, -0.029201358556747437, -4.3094267709875567e-08, -0.08215756714344025, -0.02054356038570404, -0.05952959880232811, 0.006801601964980364, -0.004263667855411768, 0.07483506202697754, -0.05195298045873642, 0.012787892483174801, 0.033007457852363586, 0.06424403190612793, -0.0157863087952137, 0.03818659111857414, -0.03128368780016899, -0.018244897946715355, -0.054079264402389526, 0.010515407659113407, 0.053067345172166824, -0.0673329085111618, 0.006953484378755093, 0.005656025372445583, 0.009271008893847466, 0.015381263568997383, 0.0840095579624176, 0.00570611422881484, -0.044348619878292084, 0.0063552213832736015, -0.07655209302902222, -0.0024638352915644646, 0.05700727179646492, 0.09733395278453827, -0.014641156420111656, 0.05327199399471283, -0.11215652525424957, -0.05268767103552818, -0.009384018369019032, -0.02264312468469143, 0.0762319415807724, -0.002632847521454096, -0.043804094195365906, 0.0016813224647194147, 0.019892744719982147, -0.04176023229956627, -0.025662662461400032, -0.04123847186565399, 0.014800482429564, -0.008958661928772926, 0.05388932675123215, -0.054007936269044876, 0.05098652094602585, 0.006724518723785877, -0.008515674620866776, -0.0498877614736557, 0.06435029208660126, 0.012781825847923756, 0.04520608112215996, -0.031236935406923294, -0.03453203663229942, 0.05942486599087715, -0.02799394354224205, -0.060746390372514725, 0.00975852832198143, -0.04912598431110382, 0.09202055633068085, -0.018656374886631966], "l2_squared", 1.0) +ORDER BY ann_distance(m.embedding, [-0.01857922412455082, -0.0614464208483696, -0.07124938815832138, 0.003173143370077014, -0.026651009917259216, 0.07428602129220963, 0.014454759657382965, 0.0009055715636350214, 0.09926825761795044, -0.05026109889149666, 0.050097666680812836, 0.041439659893512726, 0.009396902285516262, 0.04711673781275749, 0.004123533610254526, -0.03571753948926926, 0.03176308423280716, 0.0560125932097435, 0.04764378443360329, 0.009818535298109055, -0.005663667339831591, 0.02399379573762417, 0.012543700635433197, -0.019701527431607246, -0.04974042996764183, -0.0022816669661551714, 0.005969239864498377, 0.0007606040453538299, -0.0997253879904747, -0.03994547575712204, 0.05498439818620682, 0.06168409809470177, 0.021613294258713722, 0.03297579288482666, 0.05400355905294418, -0.06328761577606201, 0.03684171661734581, 0.07926641404628754, 0.014345908537507057, -0.0059095327742397785, -0.030750686302781105, -0.04278847202658653, 0.04796496033668518, 0.02237524278461933, 0.08090664446353912, -0.13150203227996826, -0.07132289558649063, -0.02492316998541355, 0.03502165526151657, 0.0053969332948327065, -0.021918678656220436, -0.04347515106201172, 0.01707594096660614, 0.07778174430131912, -0.0009688373538665473, -0.009289815090596676, 0.024557504802942276, 0.07663727551698685, 0.04940816015005112, -0.02507856860756874, 0.008993245661258698, -0.008366926573216915, 0.0312025286257267, 0.06300637125968933, 0.04590312018990517, -0.06628154218196869, -0.007432340178638697, 0.11651013046503067, -0.04510491341352463, 0.08607837557792664, -0.045920416712760925, -0.071485236287117, -0.0136015135794878, -0.03322683647274971, -0.014835990034043789, -0.007704234682023525, -0.10948145389556885, 0.007482130546122789, -0.014187892898917198, -0.002135256538167596, 0.006972563453018665, -0.046164482831954956, -0.10869627445936203, -0.018784793093800545, -0.0901474729180336, 0.014426912181079388, -0.03082936257123947, -0.0009643586818128824, -0.02510443888604641, -0.019585473462939262, -0.09764410555362701, -0.026426440104842186, 0.03617390990257263, -0.04059203714132309, -0.08562717586755753, -0.03982330113649368, -0.024080470204353333, 0.019618013873696327, -0.003956600558012724, 0.004903997760266066, 0.029250403866171837, -0.03079192340373993, 0.039038658142089844, -0.08655649423599243, 0.15646182000637054, 0.00026213270029984415, 0.030180705711245537, 0.0030696187168359756, 0.03589688241481781, -0.014934344217181206, 0.011576713994145393, 0.04430369660258293, -0.0005557635449804366, -0.025718053802847862, 0.1800142377614975, 0.054825909435749054, -0.022272437810897827, 0.038649093359708786, -0.044413141906261444, 0.07403233647346497, 0.035068329423666, 0.027785509824752808, -0.09360716491937637, -0.03326297923922539, -0.09805632382631302, -0.0419611856341362, 0.04094903916120529, -1.0503740892722901e-33, -0.009065549820661545, -0.04004768282175064, -0.08231294900178909, 0.05198328197002411, 0.08590659499168396, 0.006291812751442194, -0.009413616731762886, 0.0015937554417178035, -0.046500954777002335, 0.03878068923950195, -0.014843151904642582, 0.005073499400168657, -0.08867146074771881, 0.09745395183563232, 0.01727250963449478, 0.043382544070482254, -0.08794844895601273, -0.0534907728433609, 0.05481107532978058, -0.025791222229599953, -0.06366338580846786, 0.037842266261577606, -0.027113165706396103, 0.0015952001558616757, -0.06755585968494415, 0.0013132041785866022, -0.05610702559351921, -0.11119788885116577, 0.10298287123441696, -0.00596569525077939, -0.046586696058511734, 0.12350807338953018, 0.055325765162706375, -0.020102113485336304, 0.017435625195503235, 0.04402526095509529, -0.01169867254793644, -0.043985188007354736, -0.014159473590552807, 0.08971824496984482, -0.08812947571277618, 0.029233265668153763, -0.016919871792197227, -0.04773721843957901, 0.004967318847775459, 0.016504360362887383, -0.028715811669826508, -0.01689995266497135, -0.028662871569395065, 0.06107230857014656, -0.012691300362348557, -0.02005895785987377, 0.0007857103482820094, 0.0020176509860903025, -0.06752794235944748, 0.018846377730369568, -0.06125511974096298, -0.013897846452891827, -0.0004379384045023471, 0.052370160818099976, 0.0652880147099495, 0.03635825216770172, 0.004397145938128233, -0.030738957226276398, -0.048334185034036636, 0.04099040478467941, 0.018386656418442726, 0.04714670777320862, -0.0236875768750906, 0.09330955147743225, -0.07421127706766129, 0.08887913078069687, -0.011085454374551773, -0.02150590717792511, 0.025899238884449005, 0.040646784007549286, 0.018326999619603157, -0.009661125019192696, -0.09291890263557434, 0.023891927674412727, -0.04675614833831787, -0.024499593302607536, 0.025196222588419914, 0.002284818096086383, -0.0713845044374466, 0.05112673342227936, 0.0011782529763877392, -0.09097935259342194, 0.03155287355184555, 0.07124300301074982, -0.03573298081755638, -0.07202157378196716, 0.024810437113046646, 0.06414418667554855, 0.04424704611301422, -1.3988019106220436e-33, 0.07596112787723541, -0.08598760515451431, 0.023408066481351852, 0.010256262496113777, 0.0975826308131218, 0.03112787753343582, -0.19472666084766388, -0.031074006110429764, -0.024440864101052284, -0.04895230010151863, -0.12436351925134659, -0.015835482627153397, 0.039502210915088654, 0.03140244632959366, 0.07963060587644577, -0.01341002993285656, -0.02321160025894642, -0.043583400547504425, -0.06827471405267715, 0.08362080156803131, 0.048776354640722275, -0.008129852823913097, 0.005035134498029947, -0.03532380238175392, 0.047764550894498825, 0.047842249274253845, -0.026258224621415138, 0.017514122650027275, 0.008043339475989342, 0.08044011890888214, -0.036833275109529495, -0.006690408103168011, 0.040764402598142624, -0.02005578577518463, -0.08742470294237137, 0.0876992866396904, 0.0018844923470169306, -0.02887721173465252, 0.017003506422042847, -0.013857734389603138, -0.00867075752466917, -0.029464416205883026, -0.05655977502465248, 0.060238782316446304, -0.038879845291376114, -0.05127982795238495, -0.03536272794008255, 0.0035438239574432373, 0.04611906409263611, -0.0027573236729949713, -0.0005832789465785027, 0.045093733817338943, -0.05433909595012665, -0.004022092558443546, -0.016827723011374474, 0.005193574819713831, -0.011994755826890469, -0.05103548243641853, -0.06774675101041794, -0.02298033982515335, -0.1271994262933731, 0.10202986747026443, -0.0015871650539338589, 0.08326882869005203, 0.0021731259766966105, -0.035482317209243774, -0.007749130018055439, -0.007956304587423801, -0.12692050635814667, -0.027656463906168938, -0.033358749002218246, -0.008985556662082672, -0.004958901554346085, 0.0425625815987587, 0.06226276978850365, -0.018582263961434364, -0.08583124727010727, -0.01759367622435093, -0.039044685661792755, 0.023986786603927612, 0.014820548705756664, -0.005747810937464237, 0.06648056209087372, 0.09194660931825638, -0.007023005746304989, 0.08919760584831238, -0.022919125854969025, 0.025847693905234337, 0.01645096018910408, 0.001485480577684939, 0.09275486320257187, -0.0025220727548003197, 0.03092137724161148, -0.001075696898624301, -0.029201358556747437, -4.3094267709875567e-08, -0.08215756714344025, -0.02054356038570404, -0.05952959880232811, 0.006801601964980364, -0.004263667855411768, 0.07483506202697754, -0.05195298045873642, 0.012787892483174801, 0.033007457852363586, 0.06424403190612793, -0.0157863087952137, 0.03818659111857414, -0.03128368780016899, -0.018244897946715355, -0.054079264402389526, 0.010515407659113407, 0.053067345172166824, -0.0673329085111618, 0.006953484378755093, 0.005656025372445583, 0.009271008893847466, 0.015381263568997383, 0.0840095579624176, 0.00570611422881484, -0.044348619878292084, 0.0063552213832736015, -0.07655209302902222, -0.0024638352915644646, 0.05700727179646492, 0.09733395278453827, -0.014641156420111656, 0.05327199399471283, -0.11215652525424957, -0.05268767103552818, -0.009384018369019032, -0.02264312468469143, 0.0762319415807724, -0.002632847521454096, -0.043804094195365906, 0.0016813224647194147, 0.019892744719982147, -0.04176023229956627, -0.025662662461400032, -0.04123847186565399, 0.014800482429564, -0.008958661928772926, 0.05388932675123215, -0.054007936269044876, 0.05098652094602585, 0.006724518723785877, -0.008515674620866776, -0.0498877614736557, 0.06435029208660126, 0.012781825847923756, 0.04520608112215996, -0.031236935406923294, -0.03453203663229942, 0.05942486599087715, -0.02799394354224205, -0.060746390372514725, 0.00975852832198143, -0.04912598431110382, 0.09202055633068085, -0.018656374886631966], "l2_squared", 1.0, 30) LIMIT 5; diff --git a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java index 49f604f..09a5d4f 100644 --- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java +++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java @@ -308,7 +308,6 @@ public static final String COMPILER_DELTALAKE_FILESPLITS_KEY = Option.COMPILER_DELTALAKE_FILESPLITS.ini(); public static final String COMPILER_PARQUET_FILESPLITS_KEY = Option.COMPILER_PARQUET_FILESPLITS.ini(); public static final String COMPILER_HDFS_SPLIT_PARALLELISM_KEY = Option.COMPILER_HDFS_SPLIT_PARALLELISM.ini(); - public static final String COMPILER_VECTOR_K_MULTIPLIER_KEY = "compiler.vector.kmultiplier"; public CompilerProperties(PropertiesAccessor accessor) { super(accessor); diff --git a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java index 5b0832e..e371bce 100644 --- a/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java +++ b/asterixdb/asterix-metadata/src/main/java/org/apache/asterix/metadata/declared/MetadataProvider.java @@ -37,7 +37,6 @@ import org.apache.asterix.common.api.INamespaceResolver; import org.apache.asterix.common.cluster.PartitioningProperties; -import org.apache.asterix.common.config.CompilerProperties; import org.apache.asterix.common.config.DatasetConfig.DatasetType; import org.apache.asterix.common.config.DatasetConfig.IndexType; import org.apache.asterix.common.config.StorageProperties; @@ -859,19 +858,6 @@ int numSecondaryKeys = isQuantized ? VTreeDataTupleAccessor.Q_NUM_SECONDARY_FIELDS : VTreeDataTupleAccessor.NQ_NUM_SECONDARY_FIELDS; - // Session-level kMultiplier override. A value > 1 signals to the NodePushable to widen - // the per-partition top-k to compensate for cross-partition ranking error. - int kMultiplier = 1; - String kmultStr = (String) getConfig().get(CompilerProperties.COMPILER_VECTOR_K_MULTIPLIER_KEY); - if (kmultStr != null && !kmultStr.trim().isEmpty()) { - try { - int parsed = Integer.parseInt(kmultStr.trim()); - kMultiplier = Math.max(1, parsed); - } catch (NumberFormatException e) { - LOGGER.warn("Invalid compiler.vector.kmultiplier '{}', using default 1", kmultStr); - } - } - AOrderedListVectorBinaryAccessorFactory vectorAccessorFactory = new AOrderedListVectorBinaryAccessorFactory(); // The distance metric is fixed at index creation and baked into the factories the search operator // ships; the storage layer never sees a metric string. @@ -884,7 +870,7 @@ VTreeSearchOperatorDescriptor vectorSearchOp = new VTreeSearchOperatorDescriptor(jobSpec, outputRecDesc, queryFields, indexDataflowHelperFactory, retainInput, searchCallbackFactory, vectorAccessorFactory, distanceFunctionFactory, quantizerFactory, partitionsMap, numPrimaryKeys, numSecondaryKeys, - tupleFilterFactory, kMultiplier, indexEpsilon, indexOnly); + tupleFilterFactory, indexEpsilon, indexOnly); return new Pair<>(vectorSearchOp, partitioningProperties.getConstraints()); } diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorDescriptor.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorDescriptor.java index 6377d04..d9e34aa 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorDescriptor.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorDescriptor.java @@ -80,9 +80,6 @@ // When set, the cursor will only return tuples that pass this filter protected final ITupleFilterFactory tupleFilterFactory; - // Multiplier for candidate limit: K * kMultiplier candidates sent to PK for reranking - protected final int kMultiplier; - /** Epsilon from vector index WITH metadata (ANN / cluster search). */ protected final double indexEpsilon; @@ -99,7 +96,7 @@ ISearchOperationCallbackFactory searchCallbackFactory, IVTreeBinaryAccessorFactory vectorAccessorFactory, IVTreeDistanceFunctionFactory distanceFunctionFactory, IVTreeQuantizerFactory quantizerFactory, int[][] partitionsMap, int numPrimaryKeys, int numSecondaryKeys, ITupleFilterFactory tupleFilterFactory, - int kMultiplier, double indexEpsilon, boolean indexOnly) { + double indexEpsilon, boolean indexOnly) { super(spec, 1, 1); // 1 input, 1 output this.queryFields = queryFields; this.indexHelperFactory = indexHelperFactory; @@ -112,7 +109,6 @@ this.numPrimaryKeys = numPrimaryKeys; this.numSecondaryKeys = numSecondaryKeys; this.tupleFilterFactory = tupleFilterFactory; - this.kMultiplier = kMultiplier; this.indexEpsilon = indexEpsilon; this.indexOnly = indexOnly; this.outRecDescs[0] = outRecDesc; @@ -128,7 +124,7 @@ return new VTreeSearchOperatorNodePushable(ctx, partition, recordDescProvider.getInputRecordDescriptor(getActivityId(), 0), queryFields, indexHelperFactory, retainInput, searchCallbackFactory, tupleProjectorFactory, vectorAccessorFactory, - distanceFunctionFactory, quantizerFactory, partitionsMap, tupleFilterFactory, kMultiplier, indexEpsilon, + distanceFunctionFactory, quantizerFactory, partitionsMap, tupleFilterFactory, indexEpsilon, numPrimaryKeys, indexOnly); } } diff --git a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorNodePushable.java b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorNodePushable.java index f67733d..9740548 100644 --- a/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorNodePushable.java +++ b/hyracks-fullstack/hyracks/hyracks-storage-am-lsm-vtree/src/main/java/org/apache/hyracks/storage/am/lsm/vector/dataflow/VTreeSearchOperatorNodePushable.java @@ -94,9 +94,6 @@ // The actual tuple filter, created from the factory protected ITupleFilter tupleFilter; - // Multiplier for candidate limit: K * kMultiplier candidates sent to PK for reranking - protected final int kMultiplier; - /** Epsilon from vector index metadata (default 0.3 when absent in catalog). */ protected final double indexEpsilon; @@ -126,7 +123,7 @@ ISearchOperationCallbackFactory searchCallbackFactory, ITupleProjectorFactory projectorFactory, IVTreeBinaryAccessorFactory vectorAccessorFactory, IVTreeDistanceFunctionFactory distanceFunctionFactory, IVTreeQuantizerFactory quantizerFactory, int[][] partitionsMap, ITupleFilterFactory tupleFilterFactory, - int kMultiplier, double indexEpsilon, int numPrimaryKeys, boolean indexOnly) throws HyracksDataException { + double indexEpsilon, int numPrimaryKeys, boolean indexOnly) throws HyracksDataException { // Vector search does its filtering in the cursor, so the operator passes no filter fields, // tuple filter, output limit, or search-callback proceed result (see the args below). super(ctx, inputRecDesc, partition, null, // minFilterFieldIndexes @@ -149,7 +146,6 @@ this.distanceFunctionFactory = distanceFunctionFactory; this.quantizerFactory = quantizerFactory; this.tupleFilterFactory = tupleFilterFactory; - this.kMultiplier = kMultiplier; this.indexEpsilon = indexEpsilon; this.numPrimaryKeys = numPrimaryKeys; this.indexOnly = indexOnly; @@ -221,11 +217,12 @@ queryParamsTuple.getFieldStart(QP_FIELD_MIN_PROBE_FRACTION) + 1) : 0.0); - // k_multiplier: the per-query value, unless the session config compiler.vector.kmultiplier is set - // (> 1), which currently wins. + // k_multiplier (ann_distance arg 5): widen the per-partition candidate limit to K * k_multiplier + // tuples sent to the PK for reranking, compensating for cross-partition ranking error. Defaults to + // 1 when the query omits the argument. int queryKMultiplier = queryFields.length > QP_FIELD_K_MULTIPLIER ? Math.max(1, readIntegerQueryParam(QP_FIELD_K_MULTIPLIER)) : 1; - vectorPred.setKMultiplier(kMultiplier > 1 ? kMultiplier : queryKMultiplier); + vectorPred.setKMultiplier(queryKMultiplier); // Tuple filter for INCLUDE field predicates (e.g. year > 2000), applied at cursor level so that K // counts only passing tuples. null when this search has no pushed filter. -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/21502?usp=email To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings?usp=email Gerrit-MessageType: merged Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Change-Id: I67d27b8e74f5ee5e8eccc0d799d61ad45854a5ce Gerrit-Change-Number: 21502 Gerrit-PatchSet: 13 Gerrit-Owner: Hongyu Shi <[email protected]> Gerrit-Reviewer: Ali Alsuliman <[email protected]> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Hongyu Shi <[email protected]> Gerrit-Reviewer: Jenkins <[email protected]>
