github-actions[bot] commented on code in PR #70552:
URL: https://github.com/apache/doris/pull/70552#discussion_r4652843770


##########
be/src/exec/operator/group_commit_block_sink_operator.cpp:
##########
@@ -408,6 +502,9 @@ Status 
GroupCommitBlockSinkOperatorX::sink_impl(RuntimeState* state, Block* inpu
                                                rows_to_keep.data() + 
rows_to_keep.size()));
             block->swap(res_block.to_block());
         }
+        // Check retained rows before buffering/enqueueing: async success only 
guarantees WAL storage,
+        // and an invalid vector discovered by the segment writer cannot be 
repaired by WAL replay.
+        RETURN_IF_ERROR(_validate_ann_vectors(*block));

Review Comment:
   [P1] Preserve request atomicity when a later ANN block is rejected
   
   For a request whose valid prefix exceeds 
`group_commit_memory_rows_for_max_filter_ratio` (10,000 rows by default), 
`_add_block()` has already flushed earlier blocks to the shared queue, and 
async mode has written them to WAL. If a later block has the wrong vector 
dimension, this return fails only the submitting request. `close()` propagates 
the error and the local-state destructor merely calls `remove_load_id()`; that 
removes the dependency without retracting queued/WAL data. The separate 
background scan can therefore finish successfully and commit the valid prefix 
of a request reported as failed. Previously the malformed block reached the 
segment writer and failed that transaction. Validate/stage the whole request 
before exposing its blocks, or provide request-scoped rollback; cancelling the 
shared queue would also affect other accepted loads. The small validator/load 
tests miss the multi-block case beyond the flush threshold.



##########
be/src/exprs/function/array/function_array_distance.h:
##########
@@ -109,74 +111,69 @@ class FunctionArrayDistance : public IFunction {
                                    "Arguments for function {} must be arrays", 
get_name());
         }
 
-        return std::make_shared<DataTypeFloat32>();
+        return make_nullable(std::make_shared<DataTypeFloat32>());
     }
 
-    // All array distance functions has always not nullable return type.
-    // We want to make sure throw exception if input columns contain NULL.
+    // Handle outer NULLs explicitly so their nested array payload is never 
inspected.
     bool use_default_implementation_for_nulls() const override { return false; 
}
 
-    // Validate that neither outer column nor inner array elements contain 
NULL.
-    // Distance functions always throw on NULL input.
-    static void _validate_no_nulls(const ColumnPtr& col, const char* arg_name,
-                                   const String& func_name) {
-        const IColumn* raw = col.get();
-
-        // Unwrap const
-        if (is_column_const(*raw)) {
-            raw = assert_cast<const 
ColumnConst*>(raw)->get_data_column_ptr().get();
-        }
-
-        // Check outer nullable
-        if (const auto* nullable_raw = 
check_and_get_column<ColumnNullable>(raw)) {
-            if (raw->has_null()) {
-                throw doris::Exception(ErrorCode::INVALID_ARGUMENT,
-                                       "{} for function {} cannot be null", 
arg_name, func_name);
+    static Status _validate_array_elements(const ColumnArrayView<TYPE_FLOAT>& 
view, size_t num_rows,
+                                           const char* arg_name) {
+        for (size_t row = 0; row < num_rows; ++row) {
+            if (view.is_null_at(row)) {
+                // The nested payload of a NULL data row is not a vector.
+                continue;
+            }
+            const auto array = view[row];
+            if (array.size() == 0) {
+                continue;
+            }
+            if (std::memchr(array.get_null_map_data(), 1, array.size()) != 
nullptr) {
+                return Status::InvalidArgument("{} for function {} cannot have 
null", arg_name,
+                                               name);
             }
-            raw = nullable_raw->get_nested_column_ptr().get();
-        }
-
-        // Check inner nullable (array elements)
-        const auto& array_col = assert_cast<const ColumnArray&>(*raw);
-        if (is_column_nullable(*array_col.get_data_ptr()) && 
array_col.get_data_ptr()->has_null()) {
-            throw doris::Exception(ErrorCode::INVALID_ARGUMENT,
-                                   "{} for function {} cannot have null", 
arg_name, func_name);
         }
+        return Status::OK();
     }
 
-    Status execute_impl(FunctionContext* context, Block& block, const 
ColumnNumbers& arguments,
+    Status execute_impl(FunctionContext*, Block& block, const ColumnNumbers& 
arguments,
                         uint32_t result, size_t input_rows_count) const 
override {
         const auto& col1 = block.get_by_position(arguments[0]).column;
         const auto& col2 = block.get_by_position(arguments[1]).column;
 
-        // Validate no NULLs (distance functions always throw on NULL input)
-        _validate_no_nulls(col1, "First argument", get_name());
-        _validate_no_nulls(col2, "Second argument", get_name());
-
-        // Create views — handles Const/Nullable unwrapping automatically
         auto view1 = ColumnArrayView<TYPE_FLOAT>::create(col1);
         auto view2 = ColumnArrayView<TYPE_FLOAT>::create(col2);
+        RETURN_IF_ERROR(_validate_array_elements(view1, view1.is_const ? 1 : 
input_rows_count,
+                                                 "First argument"));
+        RETURN_IF_ERROR(_validate_array_elements(view2, view2.is_const ? 1 : 
input_rows_count,
+                                                 "Second argument"));
 
         auto dst = ColumnType::create(input_rows_count);
         auto& dst_data = dst->get_data();
+        auto null_map = ColumnUInt8::create(input_rows_count, 0);
 
         for (size_t row = 0; row < input_rows_count; ++row) {
+            // Missing vectors have no distance, including NULL/NULL and []/[].
+            if (view1.is_null_at(row) || view2.is_null_at(row) || 
view1[row].size() == 0 ||
+                view2[row].size() == 0) {
+                null_map->get_data()[row] = 1;
+                dst_data[row] = 0;
+                continue;
+            }
             auto a1 = view1[row];
             auto a2 = view2[row];
-            const float* p1 = a1.get_data();
-            const float* p2 = a2.get_data();
-            auto dim1 = a1.size();
-            auto dim2 = a2.size();
-
+            const auto dim1 = a1.size();
+            const auto dim2 = a2.size();
             if (dim1 != dim2) [[unlikely]] {
                 return Status::InvalidArgument(
                         "function {} have different input element sizes of 
array: {} and {}",
                         get_name(), dim1, dim2);
             }
-            dst_data[row] = DistanceImpl::distance(p1, p2, dim1);
+            dst_data[row] = DistanceImpl::distance(a1.get_data(), 
a2.get_data(), dim1);
         }
 
-        block.replace_by_position(result, std::move(dst));
+        block.replace_by_position(result,

Review Comment:
   [P1] Honor older FE result metadata during a rolling upgrade
   
   An old FE declares these array-distance results as nonnullable FLOAT, 
including an ordinary nonconstant query such as `SELECT l2_distance(vec,[0,0]) 
FROM t` over valid, nonempty vectors. A newly upgraded BE now unconditionally 
constructs `ColumnNullable(Float32)`. Function construction accepts the 
nested-type match, but `VExpr::execute_column()` checks the physical column 
against the FE-declared `Float32` and returns `return column type mismatch` 
before result serialization. This affects existing queries even while 
`enable_ann_index_empty_array=false`; the default-off write option provides no 
compatibility protection. Add a result-metadata/execution-version compatibility 
path so old FE plans continue to receive the legacy physical type until the 
cluster can use the new nullable contract. Requiring a coordinated FE/BE 
upgrade in the description does not preserve rolling query compatibility.



##########
be/src/storage/index/ann/faiss_ann_index.cpp:
##########
@@ -888,30 +997,71 @@ doris::Status FaissVectorIndex::range_search(const float* 
query_vec, const float
         }
     }
 
+    // The pinned FAISS IDMap::range_search slices SearchParametersHNSW to its 
base
+    // type. Translate the selector here and call the child with the original 
params,
+    // preserving efSearch/bounded_queue even when filtering out segment 
rowids.
+    auto* mapped = dynamic_cast<faiss::IndexIDMap*>(_index.get());
+    faiss::Index* search_index = mapped != nullptr ? mapped->index : 
_index.get();
+    std::unique_ptr<faiss::IDSelectorTranslated> translated_selector;
+    if (mapped != nullptr && search_param->sel != nullptr) {
+        translated_selector =
+                std::make_unique<faiss::IDSelectorTranslated>(mapped->id_map, 
search_param->sel);
+        search_param->sel = translated_selector.get();
+    }
+    // Empty arrays must not reappear when an L2 > radius / IP < radius 
predicate
+    // complements the hits. E.g. [] is absent from both sides of a distance 
comparison.
+    roaring::Roaring sparse_candidates;
+    const roaring::Roaring* range_domain = params.roaring;
+    // Dense files already cover every data row. Avoid collecting all IVF list 
IDs
+    // (and reading every on-disk list) just to reconstruct the same candidate 
domain.
+    if (needs_complement && cast_set<size_t>(_index->ntotal) != 
params.rows_of_segment) {
+        SCOPED_RAW_TIMER(&result.engine_prepare_ns);
+        ScopedIoCtxBinding io_ctx_binding(params.io_ctx);
+        RETURN_IF_ERROR(_init_indexed_rows());
+        sparse_candidates = *params.roaring & *_indexed_rows;
+        range_domain = &sparse_candidates;
+    }
+
     faiss::RangeSearchResult native_search_result(1, true);
     {
         // Engine search: FAISS range_search
         SCOPED_RAW_TIMER(&result.engine_search_ns);
         ScopedIoCtxBinding io_ctx_binding(params.io_ctx);
         try {
+            // FAISS uses strict comparisons: squared L2 < radius and IP > 
radius.
+            // Include equality by moving the bound one float toward the 
outside.
+            // Complementing flips inclusiveness: L2 > 3 must exclude L2 <= 3.
+            const bool include_boundary = params.include_boundary != 
needs_complement;
             if (_metric == AnnIndexMetric::L2) {
-                if (radius <= 0) {
-                    _index->range_search(1, query_vec, 0.0f, 
&native_search_result,
-                                         search_param.get());
-                } else {
-                    _index->range_search(1, query_vec, radius * radius, 
&native_search_result,
-                                         search_param.get());
+                float search_radius = radius > 0 ? radius * radius : 0.0F;
+                // Negative L2 bounds have no matching distances, including at 
zero.
+                if (include_boundary && radius >= 0) {

Review Comment:
   [P1] Preserve the original DOUBLE threshold when adjusting range bounds
   
   With an L2 index and stored vector `[1,0]`, the predicate 
`CAST(l2_distance_approximate(vec,[0,0]) AS DOUBLE) <= CAST(0.99999999 AS 
DOUBLE)` must reject the row: its distance is exactly 1. 
`prepare_ann_range_search()` accepts this cast/literal shape but narrows the 
bound to FLOAT 1. This new inclusive adjustment then searches squared distances 
below `nextafter(1,+inf)`, so it incorrectly includes that row; the previous 
strict bound excluded it. `VExprContext::evaluate_ann_range_search()` installs 
the bitmap as the original comparison result, so the DOUBLE predicate is not 
rechecked. The symmetric IP strict/complement paths also change results near 
rounded bounds. Retain the original threshold and derive operator-aware bounds, 
or use scalar fallback when translation is inexact; cover DOUBLE thresholds 
immediately above/below representable distances as well as exact integer 
boundaries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to